Exporting Data From Storage To Memorystore Using Cloud Dataflow
Cloud Dataflow made streaming analytics simpler, cheaper, better.
Recently, I got a chance to do an R&D on a requirement where I would need to read files stored in a Cloud Storage bucket, which would be processed and transformed in the desired format. And the new transformed data would be stored in an in-memory database, i.e., Memorystore for faster access.
Well, honestly, it took several days to figure out the correct approach before finding the correct technologies to implement this.
One of the best services in Google Cloud Platform that I have worked and experimented with is Cloud Dataflow, which is a fully-managed service to execute pipelines within the Google Cloud Platform ecosystem.
Dataflow is a service which is fully dedicated to transforming and enriching data in stream (real time) and batch (historical) modes. It takes a serverless approach where users can focus on programming instead of managing server clusters, can be integrated with Operations (formerly Stackdriver), which lets you monitor and troubleshoot pipelines as they are running.
Memorystore is Google’s implementation of Redis data store with reduced latency but high scalability. Well, caching is a technique used to…
