Open Source Data Infrastructure Meetup — Feb 2024

Apache NiFi, Apache Kafka, Apache Flink, Postgresql, Python, GTFS

NY Open Source Data Meetup Feb 8 2024 Building Real-time Pipelines with FLaNK_ A Case Study with…

It was an amazing event with a great talk A Guide to Product Experimentation (Erin Mikail Staples, LaunchDarkly), this talk has made me want to improve my slide formatting and interactivity. When the video comes out, definitely check this out.

Building Real-time Pipelines: A Case Study with Transit Data Tim Spann
In this session, we will explore the powerful combination of Apache Flink, Apache NiFi, and Apache Kafka for building real-time data processing pipelines. We will present a case study using the FLaNK-Transit projects, which leverages these technologies to process and analyze real-time data from the New York City Metropolitan Transportation Authority (MTA) and other transit systems like Halifax and Sao Paulo. By integrating Flink, NiFi, and Kafka, FLaNK-Transit projects demonstrate how to efficiently collect, transform, and analyze high-volume data streams, enabling timely insights and decision-making.

My talk following using Apache NiFi and Postgresql to ingest, enrich, transform, route and store data from various transit systems in real-time.

We use my GTFS Python feed to process the code.

FLaNK-python-processors/GetGTFSFeed.py at main · tspannhw/FLaNK-python-processors

Building a Library of Python Processors

I ran the demo on Apache NiFi 2.0.0-M2 on my Mac.

Apache NiFi 2.0.0-M2 Out!

A number of people asked about getting data from databases when they change, here is a few examples and deep context.

CDC: Not Cat Data Capture

For real-time local travel, we need to look at things like:

Tracking Aircraft in Real-Time With Open Source

We also need to look at travel advisories of where we should or should not travel.

Building a Travel Advisory app with Apache NiFi in K8

We need to check out planes, airports and the weather. Sometimes these require your own sensors, SDR and antennas.

Real-Time Flight Analytics

Watching Airport Traffic in Real-Time

Tracking Aircraft in Real-Time With Open Source

In the demo, I showed Halifax:

FLaNK for Halifax Canada Transit

I also should you Sao Paulo.

Transit in Sao Paulo, Brasil — FLaNK Style

Then I showed grabbing all the public GTFS feeds available that don’t require a login.

Iteration 1: Building a System to Consume All the Real-Time Transit Data in the World At Once

Iteration 2: Building a System to Consume All the (Unsecured) Real-Time Transit Data in the World

These are all using GTFS format which is Protocol Buffers.

General Transit Feed Specification

General Transit Feed Specification

We did the local MTA for buses.

Finding the Best Way Around

GitHub - tspannhw/FLaNK-MTA: MTA Data Sources

Sometimes the sky turns brown and should we go outside. We will use real-time analytics to check with sensors and feeds.

Wildfires… Air Quality… Time to fire up the sensors… And Start FLaNKing…

Building a Real-Time Data Pipeline: A Comprehensive Tutorial on MiNiFi, NiFi, Kafka, and Flink

If I missed anything, check this out:

Never Get Lost in the Stream

Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data

Data in Motion - How To Be a Streaming Engineer

CoC23_Utilizing Real-Time Transit Data for Travel Optimization

GitHub - tspannhw/CoC2023: Community over Code, Apache NiFi, Apache Kafka, Apache Flink, Python…

MEETUP RESOURCES

Login to Meetup | Meetup

meetups/2024Feb08OSS at main · tspannhw/meetups

Ken Lamken on LinkedIn: #nifi #aiven #ovhcloud #launchdarkly #softwareengineers

Jenn Junod on LinkedIn: #osdi

transit/gtfs-realtime/spec/en/vehicle-positions.md at master · google/transit

transit/gtfs-realtime/spec/en at master · google/transit

MEETUP PHOTOS