Cloud-printing device software built with AWS Greengrass.
We’ve open-sourced this repo as an example of an AWS IoT Greengrass project. This is our first time
using Greengrass and this project isn’t battle-tested yet, but we hope it will be useful as a
reference for other engineers working on Greengrass projects.
For some background about this repo, have a look at the article we wrote about it: Cloud-printing
for Restaurants with AWS IoT
Greengrass.
If you want to try running this software yourself, read through the Without the Private
Dependencies section first. It explains how to mock the internal
components that aren’t included in this repo.
This repo contains
A PrintOS device is a computer, usually a Raspberry Pi, running at a vendor’s store, connected to
the internet and to a receipt printer. It can also be connected to local PotatOS point-of-sale
devices.
It receives print jobs from the POS devices when patrons place orders in person and from the
internet when patrons place orders online.
See https://jira.agiledigital.com.au/browse/QFXFB-888 for more details. Based on
https://github.com/DataPOS-Labs/print-provision.
This project uses AWS
Greengrass,
an orchestration system for IoT devices. Its main parts are the Greengrass service in AWS and the
Greengrass Core software that runs on the devices.
The software is packaged into Greengrass “components”, which are deployed through the Greengrass
service. On the devices, the Greengrass Core software downloads the components, runs them, restarts
them if they crash, reports their statuses to the Greengrass service and so on.
We use MQTT to send the remote print jobs mainly because it’s the protocol with the best support in
AWS IoT. The main difference between it and HTTP is that MQTT uses a pub/sub model.
├── artifacts/
│ │ The software artifacts for the Greengrass components, one subdir per component. The contents
│ │ are deployed to the IoT devices (the Raspberry Pis).
│ ├── io.datapos.ReceiptPrinter/
│ │ Formats the print jobs and prints them.
│ ├── io.datapos.ReceiptPrinterHTTPInterface/
│ │ Receives print jobs through HTTP from the local network and from
│ │ ReceiptPrinterMQTTInterface.
│ └── io.datapos.ReceiptPrinterMQTTInterface/
│ Receives remote (internet) print jobs from AWS through MQTT.
├── component-artifact-policy.json
│ Used by deploy.sh when it creates the IAM policy that lets the devices get the artifacts from
│ S3.
├── components.drawio
│ A component diagram for the project.
├── components.png
│ A raster of components.drawio.
├── copy-to-pi.sh
│ Copies this dir to your test device (RPi) so you can deploy locally for testing.
├── deploy-local-on-pi.sh
│ Deploy locally for testing. Run this on your test device.
├── deploy.sh
│ Deploy remotely, i.e. through AWS. Use this for production deployments.
├── deployment.yaml
│ Used by deploy.sh. Specifies the components to be deployed, among other things.
├── recipes/
│ │ The config and metadata for the Greengrass components.
│ ├── io.datapos.ReceiptPrinterMQTTInterface.yaml
│ ├── io.datapos.ReceiptPrinterHTTPInterface.yaml
│ └── io.datapos.ReceiptPrinter.yaml
└── repair
A script called when the system watchdog detects an error.
2021-03-04-raspios-buster-armhf-lite.img
.
touch /path/to/mounted/sd/card/boot/ssh
raspberrypi.local
.raspberry
.passwd
to change the password for the pi
user. TODO: Can we do this more securely by/etc/shadow
before the initial boot?
sudo apt update
sudo apt install --yes openjdk-11-jdk
sudo su
curl -fsSL https://deb.nodesource.com/setup_12.x | bash -
apt install --yes nodejs
exit
sudo apt install --yes cmake libssl-dev
Greengrass.jar
. It’s probably--setup-system-service true
so Greengrass will start on boot. The installer will createap-southeast-2
region./greengrass/v2
.--tes-role-name ReceiptPrinterGreengrassV2TokenExchangeRole
or edit thedevice_role
global in deploy.sh
.
sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
-jar ./greengrass-nucleus-latest/lib/Greengrass.jar \
--aws-region ap-southeast-2 \
--thing-name ReceiptPrinterPi \
--thing-group-name ReceiptPrinterGroup \
--tes-role-name ReceiptPrinterGreengrassV2TokenExchangeRole \
--tes-role-alias-name ReceiptPrinterGreengrassCoreTokenExchangeRoleAlias \
--component-default-user ggc_user:ggc_group \
--provision true \
--deploy-dev-tools true \
--setup-system-service true
printClientsTable-[stage]
table.destination
. Make a note of thepassword
you choose as you will need it when you deploy this project to the Pi. If you haveConfigure the hardware watchdog to reboot the Pi if it freezes or its network connection drops
out.
We only configure it to check “that successive intervals see a different value of RX bytes”,
rather than pinging the server, so internet drop-outs won’t interrupt local printing. If the Pi
loses internet connection, but not network connection, rebooting isn’t likely to help.
sudo su
echo 'dtparam=watchdog=on' >> /boot/config.txt
reboot
/home/pi
.
scp repair pi@raspberrypi.local:/home/pi/
Set up the watchdog service.
sudo su
# For me, this installed 5.15-2.
apt install --yes watchdog
# Install the repair script and set its permissions.
cp /home/pi/repair /usr/sbin/repair
chown root:root /usr/sbin/repair
chmod 700 /usr/sbin/repair
# Fail the watchdog if the Greengrass service isn't running.
mkdir -p /etc/watchdog.d
echo '#!/bin/sh' > /etc/watchdog.d/greengrass
echo 'systemctl is-active greengrass' >> /etc/watchdog.d/greengrass
chown root:root /etc/watchdog.d/greengrass
chmod 700 /etc/watchdog.d/greengrass
# Update the config file.
cat >> /etc/watchdog.conf << EOM
# Added for PrintOS.
watchdog-device = /dev/watchdog
# This defaults to 60 on my Pi, but that causes an error.
# See https://www.raspberrypi.org/forums/viewtopic.php?t=244843
watchdog-timeout = 15
test-directory = /etc/watchdog.d
# Run the repair script when the watchdog fails to try to recover.
repair-binary = /usr/sbin/repair
# Kill the repair script if it takes more than 10 seconds.
repair-timeout = 10
# If the repair script reports success (returns 0), but the error hasn't cleared, reboot
# anyway.
repair-maximum = 1
# Wait 60 seconds for the error to clear on its own before repairing and rebooting.
retry-timeout = 60
# Fail the watchdog if 1 min load average goes over 24.
max-load-1 = 24
# Fail the watchdog if the network interface disconnects or stops working.
# IMPORTANT: Use 'ip link' to check that the interface name (wlan0) is right.
interface = wlan0
# Ensure the watchdog daemon will be scheduled in time.
realtime = yes
priority = 1
EOM
# Start the service.
systemctl enable watchdog
systemctl start watchdog
# Check it.
systemctl status watchdog
exit
sudo ifconfig wlan0 down
and waiting a minute to see if the Pi comes backjournalctl -t watchdog
.If you need to install a driver for an Epson TM-T20 printer, see
https://github.com/DataPOS-Labs/print-provision#raspberry-pi-deps.
First, download
PrintOS.jar
from the print-provision repo and put it in artifacts/io.datapos.ReceiptPrinter/
. If you don’t
have access to PrintOS.jar, you can use the mock version instead.
nvm use
in the root dir to switch to the project’s Node.js version.npm install
in health-reporting/
, artifacts/io.datapos.ReceiptPrinterMQTTInterface/
andartifacts/io.datapos.ReceiptPrinterHTTPInterface/
.Then check
sudo /greengrass/v2/bin/greengrass-cli deployment create \
--remove io.datapos.ReceiptPrinter --groupId thinggroup/ReceiptPrinterGroup
sudo /greengrass/v2/bin/greengrass-cli component list
until it’s removed from the--remove
, but I’ve found that--groupId
.recipes/
dir and change the configuration variables for the componentscopy-to-pi.sh
(or some other way).copy-to-pi.sh
assumes its hostname will be raspberrypi.local
, so you’ll need to edit it ifhealth-reporting
, run npm ci
in health-reporting/
on the Pi. This will compilehealth-reporting/node_modules/
in case you accidentally delete it.deploy-local-on-pi.sh
from the project directory on the device.nvm use
in the root dir to switch to the project’s Node.js version.npm ci
in artifacts/io.datapos.ReceiptPrinterMQTTInterface/
andartifacts/io.datapos.ReceiptPrinterHTTPInterface/
.Build the native binaries for the AWS packages that have native code.
If you want to deploy to devices with different architectures, you’ll need a separate deployment
for each. You might be able to follow these steps once for each architecture and then combine thehealth-reporting/node_modules/aws-crt/dist/bin/
dirs, but I haven’t tried it. And I don’t know
how you’d combine the sha256_profile
files.
npm ci
in health-reporting/
on that device. This will compile the AWS packages thathealth-reporting/node_modules/
from the device to your PC. For example,
rm -rf print-provision-greengrass/health-reporting/node_modules
rsync --info=progress2 --archive \
pi@raspberrypi.local:/home/pi/print-provision-greengrass/health-reporting/node_modules \
print-provision-greengrass/health-reporting/
deployment.yaml
:targetArn
field to the ARN of your AWS IoT Thing Group. You can find it atYou can check first with
aws configure
aws sts get-caller-identity
.Choose an S3 bucket to store the components’ artifacts and run deploy.sh [S3 bucket name]
. If
the bucket doesn’t already exist, deploy.sh
will create it. Note that S3 bucket names must be
globally unique.
deploy.sh
will then:
deployment.yaml
to the main
branch, for example, asdeployment-brodburger.yaml
.It can take a while for the deployment to roll out to your device and start running, even for a
local deployment. You can check its progress in the AWS Console or on your device.
To check the progress on a particular device, you can watch the logs from the deployment by running
this on the device:
sudo tail --follow=name /greengrass/v2/logs/greengrass.log
Or you can run (from any machine) aws greengrassv2 list-installed-components
--core-device-thing-name [thing name]
to see the version numbers of the components currently
deployed to it. The thing name will be “ReceiptPrinterPi” if you followed the example above. Or runsudo /greengrass/v2/bin/greengrass-cli component list
on the device itself to get a list with more
useful details.
See the Logging and
monitoring
section in the Greengrass docs for more info.
We use Greengrass’s built-in Log
Manager
component to send logs from the devices to CloudWatch. You can find those logs in the AWS console
under CloudWatch > Log Groups
in these log groups:
/aws/greengrass/UserComponent/[region]/io.datapos.ReceiptPrinter
/aws/greengrass/UserComponent/[region]/io.datapos.ReceiptPrinterHTTPInterface
/aws/greengrass/UserComponent/[region]/io.datapos.ReceiptPrinterMQTTInterface
/aws/greengrass/GreengrassSystemComponent/[region]/System
The devices only send logs to CloudWatch after the log file has been rotated out and in some cases
they will wait a long time before sending new logs. During development, it’s much easier to use SSH
to check the log files directly.
To change the configuration for Log Manager, edit deployment.yaml
.
There’s a bug somewhere that puts the logs from ReceiptPrinterHTTPInterface and
ReceiptPrinterMQTTInterface into the log group for ReceiptPrinter as well. You can use this filter
to clean up the logs in that group.
-"io.datapos.ReceiptPrinterHTTPInterface:" -"io.datapos.ReceiptPrinterMQTTInterface:" -"\"pass\":true"
This filter will hide the repeated health check logging in the logs from ReceiptPrinterHTTPInterface
and ReceiptPrinterMQTTInterface.
-"Health report data:" -"Successfully reported health" -"Reporting health..."
You can check the latest health status reported from a device by the Receipt Printer software in the
AWS console at
AWS IoT > Things > [the device's thing name] > Shadows > mqtt-health or http-health
The mqtt-health
shadow is the status of the ReceiptPrinterMQTTInterface component and http-health
is the status of ReceiptPrinterHTTPInterface. The ReceiptPrinter component doesn’t currently report
its health status because we don’t have the source code for it.
To see older health statuses, search in the logs.
You can also graph the health statuses in Grafana
(staging,
live) by going to the Explore page and using theexternal_service_status_total
metric. For example, this query will graph all failing health status
reports.
external_service_status_total{status != 'Success'}
You can check the status reported to printos-serverless-service for each print job in the AWS
console here. Replace [stage]
with the stage name you used when you deployed
printos-serverless-service.
https://console.aws.amazon.com/dynamodb/home#tables:selected=printJobsTable-[stage];tab=items
You can check the health of the Greengrass Core software and components on each device at
AWS IoT > Greengrass > Core Devices.
Click one of the devices to see the health of each of its components individually.
To set up more advanced monitoring, see Gather system health telemetry
data in the AWS docs.
Greengrass writes logs to the directory /greengrass/v2/logs
on the device, including logs from the
components. You can watch the most relevant logs with
sudo tail --follow=name /greengrass/v2/logs/io.datapos.ReceiptPrinterHTTPInterface.log \
--follow=name /greengrass/v2/logs/io.datapos.ReceiptPrinterMQTTInterface.log \
--follow=name /greengrass/v2/logs/io.datapos.ReceiptPrinter.log \
--follow=name /greengrass/v2/logs/greengrass.log
You can check on the Greengrass Core software with
systemctl status greengrass
List the components installed on the device with their version numbers and statuses with
sudo /greengrass/v2/bin/greengrass-cli component list
Manually remove a component after deploying it locally (e.g. with deploy-local-to-pi.sh
) with
sudo /greengrass/v2/bin/greengrass-cli deployment create \
--remove [component name, e.g. io.datapos.ReceiptPrinter]
If you deployed the component remotely (e.g. with deploy.sh
), you’ll need to add--groupId
thinggroup/[thing group name, e.g. ReceiptPrinterGroup]
.
recipes/
files. You may need to change it in multipledeploy.sh
and deploy-local-to-pi.sh
.deployment.yaml
.main
.For testing, you can configure the ReceiptPrinter component to print to PDF. However, the PDF will
always be blank, so you still need a real receipt printer to test the output.
printer
configuration variable inPDF
, which it is by default.sudo apt install cups cups-bsd
printer-driver-cups-pdf
/etc/cups/cups-pdf.conf
on your device, comment out the line Out ${HOME}/PDF
. That/var/spool/cups-pdf/ggc_user
(ggc_user
is thesudo systemctl restart cups
In this example, https://3qpbp0efwe.execute-api.ap-southeast-2.amazonaws.com/dev/submit
is the/submit
endpoint of your
printos-serverless-service deployment,blueberry
is the password in its DynamoDB and ReceiptPrinterPi
is the AWS IoT Thing Name of your
test device (i.e. your Raspberry Pi).
curl https://3qpbp0efwe.execute-api.ap-southeast-2.amazonaws.com/dev/submit --data "destination=Rec\
eiptPrinterPi&password=blueberry&data=%7B%22mode%22%3A%22tagged%22%2C%22comments%22%3A%22%3Ccenter%\
3E+Powered+by+DataPOS+%3C%2Fcenter%3E+%3Ccenter%3E+Powered+by+DataPOS+%3C%2Fcenter%3E+%3Ccenter%3E+\
%3Ch3%3ETime+Ordered%3A%3C%2Fh3%3E+%3C%2Fcenter%3E+%3Ccenter%3E+%3Ch3%3E+2%2F05%2F21+2%3A23+PM+%3C%\
2Fh3%3E+%3C%2Fcenter%3E+%3Cleft%3EService+Mode%3A+TakeAway%3C%2Fleft%3E+++++%3Cleft%3E+%3Ch3%3E1+Br\
azilian+Rooster%7E%3C%2Fh3%3E+%3C%2Fleft%3E+++++%3Cleft%3E+%3Ch3%3E2+Japanese+Rooster%7E%3C%2Fh3%3E\
+%3C%2Fleft%3E+++++%3Cleft%3E+%3Ch3%3E1+Little+Rooster%7E%3C%2Fh3%3E+%3C%2Fleft%3E+++++%3Cleft%3E+%\
3Ch3%3E1+Manly+Rooster%7E%3C%2Fh3%3E+%3C%2Fleft%3E++++++%3Ccenter%3E%3Ch3%3E%2B+Pineapple%3C%2Fh3%3\
E%3C%2Fcenter%3E++++++%3Ccenter%3E%3Ch3%3E%2B+Bacon%3C%2Fh3%3E%3C%2Fcenter%3E+++++++++%3Cleft%3E+%3\
Ch3%3E3+Hot+Chips%3C%2Fh3%3E+%3C%2Fleft%3E++++++%3Ccenter%3E%3Ch3%3EChicken+Salt%3C%2Fh3%3E%3C%2Fce\
nter%3E+++++%3Cleft%3E+%3Ch3%3E2+Hot+Chips%3C%2Fh3%3E+%3C%2Fleft%3E++++++%3Ccenter%3E%3Ch3%3ERegula\
r+Salt%3C%2Fh3%3E%3C%2Fcenter%3E++++%3Ccenter%3E+%3Ch4%3EOrder+and+Collect%3C%2Fh4%3E+%3C%2Fcenter%\
3E+%3Ccenter%3E+%3Ch5%3EOrder+NO.+Y14%3C%2Fh5%3E+%3C%2Fcenter%3E++++%3Cleft%3EPhone%3A+%2B614001210\
94%3C%2Fleft%3E+++++%3Cleft%3EName%3A+Sharon+Newman%3C%2Fleft%3E++++%3Ccenter%3E+Powered+by+DataPOS\
+%3C%2Fcenter%3E+%22%7D"
This software has an internal dependency that we haven’t made public, PrintOS.jar
. It also expects
to be able to reach two internal services, printos-serverless-service and our public API.
You can still deploy the project and test it yourself, you just won’t be able to actually print any
receipts.
First, follow the instructions in the Setting Up a Raspberry Pi
section to prepare your IoT device. You should still be able to use them if you’re going to use a
standard PC or something other than a Raspberry Pi, but you’ll need to adjust some of the
instructions. And skip over the ones that assume you have access to the internal dependencies.
Then you’ll need to set up some simple mocks.
The easiest way to mock out the internal services is using socat
. If you don’t have it installed,
your package manager probably has a package named socat
. Run these commands on the system you’ll
be deploying to.
socat -v TCP-LISTEN:12345,crlf,reuseaddr,fork SYSTEM:"echo HTTP/1.1 200; echo" &
socat -v TCP-LISTEN:12346,crlf,reuseaddr,fork \
SYSTEM:"echo HTTP/1.1 200; echo set-cookie\: abc; echo" &
Then you’ll need to change some URLs in the config files, so they point to those mock services. Indeployment.yaml
and in each of the files in the recipes/
dir, set printServerUrl
to"http://localhost:12345"
and set dataposApiUrl
to "http://localhost:12346"
.
We’ve provided a script for mocking PrintOS.jar
, mock-PrintOS.jar.py
. Configure the project to
use it by setting mockPrintOSJar
to "true"
in deployment.yaml
and inrecipes/io.datapos.ReceiptPrinter.yaml
. Then install mock-PrintOS.jar.py
‘s dependency by running
this in artifacts/io.datapos.ReceiptPrinter/
:
python3 -m pip install --target ./py_modules requests
Now you should be able to deploy the project by following the instructions in the
Deploying section.
The print jobs normally come from printos-serverless-service, so you’ll need to send them yourself.
You can do this from the AWS Console at AWS IoT >
Test > “Publish to a topic”.
Enter print-job/YourThingName
as the topic name, where YourThingName
is the name of your IoT
device. (“ReceiptPrinterPi” if you followed the example in Setting Up a Raspberry Pi.) Then enter
the following message payload and click Publish.
{
"id": "1",
"data": "{\"mode\":\"tagged\",\"comments\":\"<h3>Example+Receipt</h3><center>Powered+by+DataPOS</center>\"}"
}
If it worked, you should see this in /greengrass/v2/logs/io.datapos.ReceiptPrinter.log
:
Reporting success for job 1. data: {"mode":"tagged","comments":"<h3>Example Receipt</h3><center>Powered by DataPOS</center>"}
Response status: 200, body: {"pass":true}
And similar success messages in io.datapos.ReceiptPrinterMQTTInterface.log
andio.datapos.ReceiptPrinterHTTPInterface.log
.