Experience it yourself at transitmap.io.
Transitmap lets you watch thousands of buses, metros, commuter trains, and even ferries move across a map in real time. Try it on your phone next time you are waiting for a bus in Sweden. In most cases, the vehicle on transitmap will move only a few seconds behind the real one.
Project Goal
My friend Fabian Zeiher and I recently completed the Data Intensive Computing class at KTH [1] . The class mostly covers the typical Big Data stack of systems in an applied manner, including distributed file systems like HDFS, message brokers like Kafka, processing systems like Spark or Flink, Data Lake Houses like Delta Lake, and even graph processing systems like Pregel and GraphX. Part of that class was a freely chosen project with the objective of making use of some of the systems we covered in class.
Fabian and I both enjoy using public transport and it just so happens that a lot of Sweden's public transport agencies have detailed data feeds available in GTFS format, including real time vehicle position data [2] .
Our project idea was simple: Put all of Sweden's public transport on a map, animated and in real time.
And this is exactly what we did. If you didn't before, you can take a look at it at transitmap.io.
How it was built
The first iteration of the project, which was built for the class at KTH, used these components:
- An API client that continuously pulls in new data from the GTFS APIs.
- A Kafka Cluster that combines all of the different real time feeds and acts as a buffer for the rest of our system.
- A Spark Cluster that takes care of processing the data in real time.
- A Web Server that serves the position updates to the client via a Server Sent Events connection.
The data flow through the system looked like this:
However, the biggest challenge in building this application was not the data processing, but actually displaying it in the browser the way we wanted to. Leaflet [3] , the library we used to display the map, does not support animated markers by itself. There are plugins that implement this functionality, but at least at the time, none of them scaled well enough to display the thousands of vehicles in Sweden without severe performance issues.
Instead, we chose to use an efficient HTML canvas overlay on top of the map to implement our own redering layer to animate the vehicles. This custom solution performs well enough to run on a smartphone, even when displaying all of Sweden at once.
Going forward
For Fabian and myself, both of us studends, operating the infrastructure described above in the cloud is not affordable long term.
Because of this, we went back to the drawing board and designed a more lightweight backend system that does not make use of the same heavy weight systems. Instead, we built a custom processing engine in Rust and switched from Kafka to Redis as our message broker.
This new implementation processes the same amount of data, over a hundred million events per day, with even lower latency than before. It also uses less than a single CPU on Google Cloud.
We intend to keep operating and, as time permits, improving transitmap for the foreseeable future. Take a look at the code on GitHub and contribute to it yourself, if you want to.