Locating speeding cameras with Python, Mapbox and OverpassQL

Lets make a bot to display all the speed camera locations in my city using Python and OpenStreetMap data!

Time: 6 hours

Tech: OpenStreetMap, Mapbox API, Facebook Graph API

Languages: Python, NodeJS, OverpassQL

edit: Message the bot here https://m.me/waspeedcamerabot

In Western Australia, the location of fixed and mobile speed cameras is published to reinforce the fact that they are indeed for the prevention for deadly crashes and aren’t profit making machines. The location of mobile cameras for the following week is available for download in .pdf form on the WA police website, in a long list of small-font alphabetically listed streets and suburbs.

I thought it would be great if these locations were all rendered on a map that was delivered straight to my phone every morning. It would be a good exercise in geography programming which is being used increasingly in self-driving cars and the Internet of Things.

The scope of this mini-project:

• Facebook messenger bot (Python/js?)
• Renders a .png map of Perth metropolitan area with monitored roads highlighted/pinned
• Ability to zoom in on post codes (render on the fly)

I decided against an interactive map because the messenger api doesn’t support them and they don’t really work very well in embedded browser windows.

The first and messiest part is extracting the data from the pdf tables. pdfminer was used to process the pdf files. It couldn’t grab the layout properly (probably not the library’s fault), so I decided just to dump all the text and parse it. This took a while because of page breaks but I eventually got an algorithm that seems to work. The result is lovely .json.

Now, what the bot needs to do is:

1. Get all the lat/long points of the road
2. Get the lat/long boundaries of the suburb
3. Remove all lat/long road points that aren’t in the suburb
4. Draw the resulting points as a line

After some searching, I realised the best option was to use the OpenStreetMap community-built database by querying an Overpass API interpreter (which can performs steps 1–3!). Overpass has a complex Query Language which has endless cool applications (self-driving, IoT, cartography). Most of the time on this bot was spent learning the Overpass QL language.

There are 3 main objects in Overpass QL, nodes(lat/long points), ways(one or more nodes), and relations(describe areas/boundaries). Each object has properties, such as a name, types like postbox or swimming pool, public transport info, etc... The data can get quite detailed, like shoptype:cafe - the only limit to this data is how much everyone is willing to contribute :) Overpass can return results within a boundary by using bracket notation, for example a big box around my city from lat:-32.398 to -31.6248 and lon:115.64 to 116.2796 would become (-32.398, 115.64, -31.6248, 116.2796).

To get the suburb boundaries, I downloaded the suburb polygon data from here and estimated each suburb to a square because otherwise Overpass took too long (not good for big regional suburbs but should be fine for innercity stuff).

view code here

Now we have all the data we need for Overpass to perform steps 1–3 all in one query! The following query in words is

retrieve all the lat/long points for a road called "Canning Highway" in the bounds of Perth and a suburb called Bicton

way["name"="Canning Highway"](-32.398, 115.64, -31.6248, 116.2796)(-32.0383, 115.7760, -32.00963, 115.7935)

You can try this query on this online Overpass QL interpreter.

Here is the script I wrote to generate a large query to get all the points between each suburb.

Now we need to render these points on a map. I settled on using mapbox API to render the static images, because of the awesome map customisation (and generous free tier). The first thing I tried was to encode all these points in google’s polyline format, but there were far too many nodes and lines to pass by argument into the mapbox API.

So I had to upload the points as a dataset into mapbox, then create a custom style to colour the selected nodes/ways. Mapbox has a very powerful tool called “mapbox studio”

As you can see, I have created a layer that highlights all the data from the todays_data source red.

Now we can call the static url with our username and published style to receive a beautiful static image!

The last, easy step is to implement this into a messenger bot! I downloaded the example bot off Facebook’s tutorial and deleted all the examples except the send image function. If the user says "today", lets send them an image of the whole of Perth. If they give a number that matches a regex of Perth postcodes, lets send them a static image zoomed in on that suburb.

the relevant code can be viewed here

To prevent Facebook from caching the images, I added a negligible amount of lat/long to change to request slightly.

The bot works well! It is currently running on a raspberry pi 3B+ using ngrok as a reverse proxy (dont tell anyone!)

and thats it! This was real satisfying and I’m sure the Overpass QL experience will come in handy later.

If you want to message the bot, have a go here:

https://www.messenger.com/t/waspeedcamerabot