Intuit Launches Open Source Stream Processing and AIOps Tools for AI-native Development
As platform engineering evolves in a data-flooded world, becoming an AI-native platform can only be achieved through the successful use of automation and machine learning at scale. Capturing, processing, and making data useful will require new AI tools on cloud native technologies, and the open source community behind it.
At Intuit, we support 350 clusters with over 3500 micro services, processing 5 billion events per day for observability and real-time anomaly detection through AIOps. We’re constantly training the ML models that power 65B predictions per day.
This would never have been possible without adopting cloud native and open source tooling. It also required us to take an approach to building a development platform that provides a simplified experience so that technologists across the organization can focus less on infrastructure and more on application code for customers.
Our team isn’t just an end user of open source software, we maintain and contribute to many tools. When we have hard problems to solve we look to open source software (OSS) but also build solutions that we believe will help others solve hard problems. That’s why we’re excited to share some new contributions to the cloud native and AIOps community that we believe will drive the next phase of platform engineering on all our collective journeys to becoming an AI-native platform.
Today we announced the general availability release of Numaflow 1.0, a massively parallel, real-time data and stream processing engine, available for Kubernetes and as an ArgoCD extension. Read more product details here.
Operational excellence and automation powered by GenAI
GenAI is allowing us to create simpler, more powerful user experiences, dramatically improving the productivity of our developers. When the team set out to tackle a growing challenge with the amount of incoming data to decipher critical metrics needed during deployments, we realized the tools available just couldn’t handle the scale. This is where we focused our energy, to create the next generation of tooling with GenAI that enables real-time analytics on Kubernetes.
Numaflow 1.0 is one part of the Numaproj project that includes two products: Numaflow for parallel stream processing and Numalogic for anomaly detection ML models. It’s also completely open source and generally available now.
Numaflow1.0 also has a metrics plugin for Argo, another open source project that powers Intuit’s progressive delivery and deployments. These two tools (both created, tested, and open sourced by Intuit teams) together generate a single anomaly score tied to Kubernetes events using live streaming metrics. It enables top tier observability that gives developers the right level of critical data to make fast decisions during deployments. Read more about that use case here.
All of this infrastructure is what powers Intuit engineers to deliver app-level experiences that bring the power of GenAI to Intuit customers. Read more about that experience called Intuit Assist here.
Join Intuit at ArgoCon and KubeCon North America
Learn more how Intuit is leveraging AI and platform engineering in these sessions at ArgoCon and KubeCon.
You can also join our open source community on LinkedIn here where we share updates, best practices, and look forward to conversations with the community!
Intuit is a two-time recognized End User of the Year by the Cloud Native Computing Foundation. Intuit’s investment in cloud native technologies to provide modern platforming company-wide has driven a 9x increase in development velocity over the last three years and enables Intuit to fuel product innovation at speed and scale for 100 million consumer and small business customers worldwide with TurboTax, Credit Karma, QuickBooks, and Mailchimp. Intuit is the original creator of the recently graduated Argo project.
