full stack plane spotting and data analysis

The task at hand is simple. Whenever a particular airplane is visible overhead, send out a tweet with that notice. Don’t repeat yourself with this announcement more than twice an hour, but try not to have too much lag in reporting.

The full stack of hardware and software to do this is not particularly complicated to use once you get it all running, but there are a series of issues and observations along the way that add to the complexity. I’ll walk through the whole thing start to finish, and let you know where I left off.

The basic toolkit is dump1090-mutability for plane spotting, Node-RED for wiring flows for data processing, and Docker to package up the bits that are stable.

Plane spotting

When last I wrote, I had dump1090 running on my MacBook. Since then, I’ve taken the same software and gotten the latest dev release going on a Raspberry Pi 2.

I switched to a smaller, simpler antenna, and the result was positive: instead of having a 40 to 50 nautical mile range, now I’m in the 60 to 65 nm range (about 110 km). It might also help that the Pi 2 is not really doing very much else. That particular Pi 2 setup is using a slightly back-rev version of the Hypriot software for the Pi, so it has all of the modern conveniences plus an instance of Docker on board. The temptation was to Dockerize everything, but I haven’t gone there yet; the available Docker prebuilt releases don’t have the bleeding edge of the dump1090-mutability code, and I want to see how the whole thing runs native for a while before containerizing it.

The snag that slowed down this deployment a little was Google’s decision to enforce an API key for personal use of Google Maps. In response, the very latest dump1090-mutability pulled out its Google Maps support, and went to OpenLayers. This new support gives access to all kinds of base maps, including some aviation-specific charts. (“Not for use in navigation.”)

Another piece of this was bringing up lighttpd (“lighty”), a lightweight HTTP server that replaces the rickety built-in HTTP server that dump1090 had originally shipped with. Configuration following the supplied instructions was simple, and lighttpd looks like a very suitable system to put in place to serve HTTP on embedded-class system that need flexiblity but not a big footprint.

The other component in the Pi setup is an ADS-B feed to the excellent Planefinder site. Planefinder provides a Debian package for setting up the feed, and it runs neatly with only a minimum of overhead. As a bonus you get free use of their apps, which are notable for running in much closer to real time than others I have seen.

The Pi does not yet have a working OpenVPN configuration, so I can’t get to it over the net directly. For that I plan to revisit the SaltStack for Pi configuration I wrote up last September. In the interim, salt has been upgraded, and perhaps it will be a little more stable over bad connections; we’ll see.

Data analysis

dump1090 provides a data feed in a convenient JSON format, available through a URL on a standard configuruation. The ADS-B messages themselves are binary encoded, but the system brings them into JSON so that all of the downstream processing is very easy.

Data handling is so straightforward that you can do it all without needing any special-purpose processing nodes in IBM’s Node-RED. A standard HTTP in node polls the data at the pace that you choose, and a simple function pulls apart the record of all planes into individual records for each plane.

Getting Node-RED running is continuing to get easier over time. I’m doing prototyping work on my Macintosh again, using the new Docker for Mac setup to pick up a Node-RED container to do the work in. Kitematic comes with Docker for Mac, providing a convenient dashboard and interface to Docker Hub.

(Do note, however, that if you got Docker for Mac running a few months ago, you’ll need to go through a rebuild process to go to the latest beta. To recreate or update your containers after Beta 18 upgrade follow the provided instructions.)

Once you have both Node-RED and dump1090 running, connecting the two together is straightforward. I started with this gist from dceejay which provides the basic flow and feeds into a worldmap node that is a general programmable interface for putting things on a map. My updated dump1090 gist for Node-RED adapts the data for a slightly different format between dump1090 forks.

From here in it’s relatively clear sailing. Set up a flow downstream from the plane messages, using a filter node to match the name or ICAO hex code of the airplane that you want to track. Wire it through a delay filter to set the frequency of messages you want to allow through, and a formatting node to pretty up the message in a form suitable for Twitter output.

Incomplete image and next steps.

This is a relatively complex stack, but it all fits neatly on a Raspberry Pi 2, and with a little bit more elbow grease it should be possible to containerize everything so that it Just Works and can be made to reliably work again.

I’d like to have OpenVPN running on the Pi 2, but it isn’t yet, pending rework of Salt to get that to a current version and to port all of the work I did from an earlier version.

I’m picking up planes from 60+ nm away, but I know that with a better antenna, bandpass filter, and amplifier that I can do better. Jared Mauch’s parts list would be the place to go for starters with a very modest budget of a couple of hundred bucks.

Reception is best if you have a clear view of the sky, which suggests an outdoors installation, powering the Pi with power over ethernet.

The system is running on a Pi 2, but reports from Jared and from Phil Karn KA9Q suggest that if you do upgrade the reception that it’s possible to saturate the Pi 2. Fortunately a Pi 3 does even better at the intended task.

I’m still figuring out what the very best way to create a Docker image that handles Node-RED, especially the installation of additional nodes and the management of flows. You’d like to have the whole thing suitably tidy that it could be all loaded in from the get-go.

dump1090-mutability is still mutating! It’s under active development, both at the presentation layer and at the decoder layer. So it would be appropriate to have the attitude for at least one development machine to be sure that you were ready to rebuild from source whenever.

If you only have half of this stack running (the Node-RED side) but no access to radios, you can find an existing dump1090 installation and pull the JSON from there. This is best done only via prior arrangement and only for testing purposes or for specific tasks. It’s easy to crank up Node-RED to poll a URL every second; don’t do that to a poor little Pi.

The planes I want to watch don’t take off very often - that’s the appeal of watching for them.