Key Takeaways from Flink Forward SF 2019

The Apache Flink Community is alive and well in 2019! At FinTech Studios, we are migrating to Flink for a variety of our data enrichment and anomaly detection use cases on our live news data streams.

Flink Forward, in San Francisco on April 1st and 2nd, joined the community, consumers, and committers alike to share how Flink has grown in the last year and where we’re headed over the next. Among the many steps forward, a few high-level takeaways stood out:

• Unification of Batch and Streaming
• Operational maturity and a growing list of managed solutions
• Machine Learning as a first-class citizen
• The emergence of Self-Service Flink Jobs as a Platform

Unified Batch and Streaming

• The Flink community is, like many other communities, tired of maintaining duplicate logic, especially in streaming and batch applications for the same system — the Flink Team is unifying the APIs for data analytics, such as the Table API, so devs can leverage the same code for both needs.
• Other projects, like Apache Beam, also aim to allow one codebase for both streaming and batch. They recently released the first stage of support for targeting Flink as a runner.
• Alongside codebase unification, many are moving away from Lambda Architecture and towards Kappa Architecture. Uber has coined Kappa+ in a move to provide more concrete implementation guidelines and constraints around the types of jobs that fit into the unified offline-realtime paradigm.

Operations for Everyone

• Many in the Flink community, FinTech Studios included, have struggled and built homegrown tools to manage deployments, and in response, many companies have released managed solutions, Ververica, AWS, and Eventadaor.io among them.
• AWS released Flink as a runner for Kinesis Data Analytics; Ververica offers the similar Ververica Platform, which has familiar configuration options to a manual deployment and excellent job lifecycle management; and Google Cloud Platform’s Beam and DataFlow both have managed deployments as well.
• Though there are many managed solutions, many teams are moving to Flink because their internal ops have started to support it at scale.
• Kubernetes seems to dominate both externally and internally managed deployments; it is also the only supported cluster on the Ververica Platform.
• Along with deployment operations, Flink jobs are more compatible than ever with the inclusion of Avro schema migrations in 1.7 and TypeSerializers-based schema migrations in 1.8, making it possible to upgrade a job without having to rebuild the state.

Machine Learning within Flink

• Following Alibaba’s purchase of Ververica, Baba will be contributing the major improvements they’ve made in their internal Flink fork, Blink, over the next year. Among the improvements, a few make training machine learning models much simpler, such as a library of common ML algorithms and improvements on the Table API.
• There have been considerable attempts to incorporate trained machine learning models into production streaming applications, such as using RPC + Py4j to communicate with Python-hosted models. TensorFlow Extended was presented as another option for the productionisation of ML models.
• The use of native machine learning models in Flink is still not an easy task and, though it is improving, microservices + perhaps the AsyncIO API is currently the easiest way for model integration.

Self-Service Flink Jobs as a Platform

• As streaming data starts to account for more and more of business data, the business stakeholders need ways to leverage and work with the streams. Dynamically-defined Flink jobs to analyze and surface streaming data have popped up this year, among them Netflix’s Consolidated Logging (CL) and Cogility’s Cogynt, allowing stakeholders to do complex analytics on streamed data with little-to-no code.
• At Netflix, their CL platform allows users to define filtering with simple SQL, analytics transformations, and dynamic output streams on their entire application logging stream, which handles hundreds of billions of events a day. This platform drives their core services, such as personalization and A/B testing.

Overall, the Flink platform is proving to be incredibly flexible. The Ververica team is committed to the growth of the ecosystem and have one of the most helpful, open cultures out there — I’m not sure I ever heard a member say that a use case was not possible with Flink, you just might have to get a little creative.

FinTech Studios works with tens of millions of news documents a month, NER, data extraction + enrichment, and anomaly detection are just a few of our use cases … and we are hiring!

Reach out to me personally at austin@fintechstudios.com — always open to chat!

If you want to learn more, find all the conference talks here.