Tim’s Quarter in Streaming 2Q — 2023

Apache NiFi, Apache Flink, Apache Kafka, MiNiFi, Apache Pulsar, IoT

Photo by Andy Holmes on Unsplash

May 1 to July 31, 2023

In this quarter, we had some cool things going on.

First, we had the “Meet the NiFi Committers” and NiFi Competition Event in the Cloud.

I created some assets for this:

Slack

Cloudera Sandbox Event Guidelines For Workflow

Meet the Committers Webinar_ Lab Preparation

Best Practices For Workflow

I spoke at a number of events:

big data fest building modern data streaming apps

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and Kafka

Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp

Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data

Implement a Universal Data Distribution Architecture to Manage All Streaming Data

OSSNA Building Modern Data Streaming Apps

I spoke at some meetups:

Login to Meetup | Meetup

Login to Meetup | Meetup

GSJUG: Mastering Data Streaming Pipelines 09May2023

Meetup Streaming Data Pipeline Development

I wrote some articles:

Community Cloudera 2,468 views, 1 article

Best in Flow Event

Datainmotion.dev 26.4K views

Medium 4,873 views, 18 articles

Open Source Streaming Talks in Progress

Watching Airport Traffic in Real-Time

CDC: Not Cat Data Capture

Functions Anywhere — FaaS

Writing critical syslog events to Apache Iceberg for analysis

Ingesting Events into Dockerized IBM DB2 (JDBC) with Apache NiFi

Apache NiFi 1.22 Updates

Harnessing the Power of NiFi: Building a Seamless Flow to Ingest PM2.5

Wildfires… Air Quality… Time to fire up the sensors… And Start FLaNKing…

Seamless Integration: Unleashing the Power of Real-Time Groceries with NiFi, Kafka, Flink and…

Building a Real-Time Data Pipeline: A Comprehensive Tutorial on MiNiFi, NiFi, Kafka, and Flink

Using NiFi As an SMTP (Email) Test Server

Finding the Best Way Around

Garden State Java Users Group Talk on May 9, 2023

Deploy Your NiFi Flow to K8 in AWS

Ingest From Iceberg Tables with Cloudera DataFlow

Building a Travel Advisory app with Apache NiFi in K8

DZone 16.9K views, 3 articles

What Is a Modern Developer? 8049 views

What Is a Modern Developer? - DZone

from https://dzone.com/trendreports/development-at-scale

Streaming Change Data Capture Data Two Ways 4352 views

Streaming Change Data Capture Data Two Ways - DZone

Harnessing the Power of NiFi: Building a Seamless Flow To Ingest PM2.5 Data… 4538 views

Harnessing the Power of NiFi: Building a Seamless Flow To Ingest PM2.5 Data From a MiNiFi Java…

I created some videos:

My Youtube Channel is growing slowly, I need to add more content every week. Any suggestions for things people would want to see?

Data in Motion

I created some demos, code and exaxmples:

GitHub - tspannhw/FLaNK-ADSB: ADSB-Y Flights

GitHub - tspannhw/FLaNK-MTA: MTA Data Sources

GitHub - tspannhw/FLaNK-Py-Stocks: FLaNK Python Stocks to Apache Kafka - Cloudera

GitHub - tspannhw/FLaNK-CDC: CDC with NiFi, Kafka Connect, Flink SQL, Cloudera Data in Motion

GitHub - tspannhw/FLaNK-Edge: An example of FLaNK Edge

GitHub - tspannhw/FLaNK-HBase2HBase: Simple HBase to HBase scan to HBase Cell

GitHub - tspannhw/FLaNK-TravelAdvisory: Travel Advisory - RSS Processing - Apache NiFi - Apache…

GitHub - tspannhw/FLaNK-ParticulateMatterSensor: Clean air, up in smoke…

GitHub - tspannhw/FLaNK-Store: Retail Grocery Store with MiNiFi, NiFi, Kafka, Flink, Kudu, Iceberg…

GitHub - tspannhw/FLaNK-DataFlows: DataFlows

I created a number of weekly newsletter issues:

This quarter had some slow downs due to summer vacation and corporate unplug days.

• 96: 31-july-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/31July2023.md https://medium.com/@tspann/flank-stack-for-31-july-2023-5ccccbe57402 https://dev.to/tspannhw/flank-stack-weekly-for-31-july-2023-2bam https://timwithpulsar.hashnode.dev/flank-stack-weekly-for-31-july-2023 https://www.datainmotion.dev/2023/07/flank-stack-weekly-for-31-july-2023.html https://www.linkedin.com/pulse/flank-stack-weekly-31-july-2023-tim-spann-
• 95: 24-july-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/24July2023.md https://www.datainmotion.dev/2023/07/flank-stack-weekly-for-24-july-2023.html https://timwithpulsar.hashnode.dev/flank-stack-weekly-for-24-july-2023 https://dev.to/tspannhw/flank-stack-weekly-for-24-july-2023-40kf https://medium.com/@tspann/flank-stack-weekly-for-24-july-2023-1b0b6546293c https://www.linkedin.com/pulse/flank-stack-weekly-24-july-2023-tim-spann-
• 94: 17-july-2023:
  https://dev.to/tspannhw/flank-stack-weekly-for-17-july-2023-256c https://github.com/tspannhw/FLiPStackWeekly/blob/main/17July2023.md https://timwithpulsar.hashnode.dev/flank-stack-weekly-for-17-july-2023 https://www.linkedin.com/pulse/flank-stack-weekly-july-17-2023-tim-spann- https://medium.com/@tspann/flank-sta-844eb57a1e85
• 93: 10-July-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/10July2023.md https://dev.to/tspannhw/flank-stack-weekly-for-10-july-2023-2a2h https://www.datainmotion.dev/2023/07/flank-stack-weekly-for-10-july-2023.html https://timwithpulsar.hashnode.dev/flank-stack-for-10-july-2023 https://medium.com/@tspann/flank-stack-for-10-july-2023-386ee6532aaa https://www.linkedin.com/pulse/flank-stack-weekly-10-july-2023-tim-spann-
• 92: 03-July-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/03July2023.md https://medium.com/@tspann/flank-stack-weekly-for-july-3-2023-18af7cd0ab26 https://dev.to/tspannhw/flank-stack-for-4th-of-july-4hf4 https://timwithpulsar.hashnode.dev/flank-stack-for-3-july-2023 https://www.datainmotion.dev/2023/07/flank-stack-for-3rd-july-2023.html https://www.linkedin.com/pulse/flank-stack-weekly-3-july-2023-tim-spann-
• 91: 26-June-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/26June2023.md https://dev.to/tspannhw/flank-stack-weekly-on-26-june-2023-3082 https://www.linkedin.com/pulse/flank-stack-weekly-26-june-2023-tim-spann- https://timwithpulsar.hashnode.dev/flank-stack-weekly-for-26-june-2023 https://www.datainmotion.dev/2023/06/flank-stack-weekly-for-26-june-2023.html https://medium.com/@tspann/flank-stack-weekly-for-26-june-2023-d80be66c6faf
• 90: 20-June-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/20June2023.md https://dev.to/tspannhw/flank-stack-weekly-for-20-june-2023-2cc5 https://timwithpulsar.hashnode.dev/flank-forward-for-20-june-2023 https://www.datainmotion.dev/2023/06/flank-stack-weekly-for-20-june-2023.html https://medium.com/@tspann/flank-stack-weekly-for-20-june-20-8c73cf278c75 https://www.linkedin.com/pulse/flank-stack-weekly-20-june-2023-tim-spann-
• 87: 12-June-2023:
  https://www.datainmotion.dev/2023/06/flipn-flank-stack-weekly-12-june-2023.html https://medium.com/@tspann/flank-stack-weekly-for-12-june-202-27ab5f58b21e https://timwithpulsar.hashnode.dev/flank-stack-for-12-june-2023 https://www.linkedin.com/pulse/flank-stack-weekly-12-june-2023-tim-spann- https://dev.to/tspannhw/flank-stack-weekly-for-12-june-2023-506h https://github.com/tspannhw/FLiPStackWeekly/blob/main/12june2023.md
• 86: 05-June-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/05June2023.md https://dev.to/tspannhw/flank-stack-5-june-2023-1gpo https://medium.com/@tspann/flank-stack-weekly-for-05june2023-553cb6aea41b https://www.datainmotion.dev/2023/06/flank-stack-weekly-for-5-june-2023.html https://www.linkedin.com/pulse/flank-stack-weekly-june-5-2023-tim-spann- https://timwithpulsar.hashnode.dev/flank-stack-5-june-2023
• 85: 30-May-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/29May2023.md https://www.datainmotion.dev/2023/05/flank-stack-weekly-for-29-may-2023.html https://medium.com/@tspann/flank-stack-weekly-29-may-2023-cf53a408bf1e https://timwithpulsar.hashnode.dev/flank-stack-weekly-29-may-2023 https://dev.to/tspannhw/flank-stack-weekly-29-may-2023-3556 https://www.linkedin.com/pulse/flank-stack-weekly-29-may-2023-tim-spann-/?published=t
• 84: 22-May-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/22May2023.md https://www.linkedin.com/pulse/flank-stack-weekly-22-may-2023-tim-spann- https://www.datainmotion.dev/2023/05/flank-stack-for-22-may-2023.html https://dev.to/tspannhw/flank-stack-weekly-for-22-may-2023-1fgc https://timwithpulsar.hashnode.dev/flank-stack-for-22-may-2023 https://medium.com/@tspann/flank-stack-for-22-may-2023-a4fb874ead4d
• 83: 15-May-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/15May2023.md https://timwithpulsar.hashnode.dev/flank-stack-weekly-for-15-may-2023 https://dev.to/tspannhw/flank-stack-for-15-may-2023-5f77 https://medium.com/@tspann/flank-stack-for-15-m-ce9aad3a5d2d https://www.datainmotion.dev/2023/05/flank-stack-for-may-15-2023.html https://www.linkedin.com/pulse/flank-stack-15-may-2023-tim-spann-
• 82: 08-May-2023: https://github.com/tspannhw/FLiPStackWeekly/blob/main/08May2023.md https://www.datainmotion.dev/2023/05/flank-stack-weekly-for-08-may-2023.html https://dev.to/tspannhw/flipn-flank-stack-weekly-may-8-2023-41a1 https://www.linkedin.com/pulse/flank-stack-08may2023-tim-spann- https://timwithpulsar.hashnode.dev/flank-stack-for-08may2023 https://medium.com/@tspann/08-may-2023-90e333eff7d6

There was a new release of Apache NiFi 1.23!

Release Notes

Some interesting thing going on in streaming and Apache Iceberg is everywhere!