Recently, I’ve been in charge to create a streaming data pipeline which handle data coming through Cloud Pub/Sub, process it with some encryption algorithms, then push a pair of key/value data to Cloud Firestore.

The pipeline was made in Java, mainly because of the stability of streaming on apache beam compared to Python and the use of some native AEAD libraries in Java.

Streaming data to Cloud Firestore and BigQuery

The struggle while setting up this job was to find a suitable I/O transforms for Firestore. Here we can get some of the helpful built-in connectors, but nothing for what i’m looking for. …

Today, we will try to inspect events organised by Tech groups on Meetup. Our main goal is to take a look on trending techs / IT fields and get the overall concerns about technologies development and communities expansion.

To reach our objective, we will harness the amazing API given by Meetup to explore its entire interesting data which gives many opportunities such as getting information about categories, groups, events, attendees, reviews, etc...

There's even an SDK in python and a bunch of other languages that you can use easily.

We will use the api calls to retrieve data. Later, we will…

Alaeddine Bouattour

Data Enginner on Google Cloud Platform

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store