Introducing Javelin Processors
Enterprises adopting Large Language Models (LLMs) often face the challenge of embedding specific controls, activating mechanisms to trigger enterprise systems, and broadening the integration of these systems within their existing technology investments. When these tasks are duplicated across individual applications, it leads to inefficiencies and hampers the consistent application and management of controls and data governance across the enterprise’s tech ecosystem. Moreover, any change in controls necessitates modifications in the application code, posing substantial challenges in production environments. These issues complicate the transition from experimental prototypes to full-scale production.
We are thrilled to announce the launch of Javelin Processors, an extensibility toolkit intended to enable customers and developers to expand the platform for both extensible controls and dynamic chaining of logic. This toolkit enhances Javelin’s core capabilities, allowing it to be tailored precisely to meet the diverse and evolving needs of companies. Developers can leverage this capability to customize the platform’s processing logic to meet specific business requirements and integrate with enterprise tools.
Javelin Processors represent our commitment is to provide a foundation that not only supports robust enterprise integrations and application extensions but also ensures the company-specific governance policies to the use of LLMs. This commitment stems from an understanding of the unique challenges businesses face, including data security, privacy, and the need for rapid adaptation to new AI models and providers.
Challenges of LLM Adoption in Enterprises
In the dynamic landscape of medium to large enterprises, the need for Artificial Intelligence (AI) solutions spans various teams and business lines, each with unique requirements and challenges. These organizations face the pressing need for a platform that can cater to the multifaceted needs of different departments while ensuring that the adoption process is seamless, secure, and aligned with the company’s overarching governance policies.
A common initial approach involves individual teams creating their applications, each embedding specific guardrails or controls according to their discretion.
“Whats good for a Proof-Of-Concept, is bad for Production”
Integrating safety measures and controls directly into each application might appear as a simple fix, yet it presents significant limitations. This approach leads to a fragmented ecosystem where each application implements its own set of controls, lacking a centralized management system. Such decentralization makes it challenging to enforce uniform standards across the organization, especially when organizational policies evolve. Additionally, updating applications individually to reflect changes in policies becomes cumbersome, highlighting the necessity for a separation of concerns to ensure more efficient and cohesive governance.
Moreover, this direct integration approach fails to offer connectivity with existing enterprise tools, creating silos that hinder efficient workflows and collaboration. Without the capability for straightforward integration, organizations cannot leverage their full suite of tools and systems alongside AI applications, further compounding the challenges of maintaining consistency and streamlining operations across the enterprise. This underscores the need for a more flexible and integrated framework that allows for central management and easy integration with the broader ecosystem of enterprise tools.
Introducing Javelin Processors
We built Javelin with Enterprises in mind, and the platform's core foundation is built around the notion of bounded functional units. Each function within the platform is encapsulated into independent processing units, referred to as processors. These processors are chained together to form Directed Acyclic Graphs (DAGs) — creating structured execution pathways known as processor chains. The structure of these DAGs allows for precise control over the execution flow within the platform, enabling a highly customizable and efficient processing environment.
Processor Execution Framework
The processors in Javelin can operate in two distinct modes: asynchronous or synchronous. This operational model is selected during the initialization phase of each processor.
Synchronous Execution: This mode is beneficial for tasks that require immediate processing and feedback. It is often employed for critical operations such as implementing model guardrails, which may involve checking against a trust & safety model or applying content and toxicity filters. While synchronous processing ensures inline execution, care must be used to avoid unnecessary processing overhead in the path of LLM calls. Developers can also configure latency budgets for synchronous processors, ensuring that LLM request processing stays within deterministic pre-established limits.
Asynchronous Execution: In contrast, asynchronous execution allows tasks to be performed without blocking the system, making it ideal for operations in the background. This mode is typically used for triggering events or notifications to enterprise systems, where immediate response is not critical. For example, a custom processor can be written to send a notification to Slack or trigger an event in a Security dashboard when certain data is exchanged with LLMs. Custom processors trigger webhooks, insert data into Data Warehouses, or trigger enterprise workflows.
Understanding Processor Chains
Processor chains are a foundational concept within the Javelin platform, representing a sophisticated mechanism for orchestrating the execution flow for calls.
Chains are essentially sequences of processors — self-contained modules designed to perform specific functions — arranged in a DAG structure.
This arrangement facilitates complex processing sequences and ensures the execution flow is efficient, logical, and adaptable to various enterprise needs.
Each processor within the chain can be developed, tested, and deployed independently, enhancing the platform’s flexibility and responsiveness to changing requirements. Processor chains can be dynamically adjusted to handle varying loads, making them well-suited for enterprises with fluctuating processing demands. The ability to run processors asynchronously also contributes to the platform’s overall efficiency, allowing for non-blocking operations and improved resource utilization.
Request Chains
When a request enters the Javelin platform, it can be directed through a specifically designed processor chain configured to analyze, validate, and preprocess the data before it goes to a model. Typical checks here involve filtering or redacting PII/PHI information.
Response Chains
Once the LLM responds to the application, the response processor chain is triggered to format, enrich, and securely deliver the response back to the application.
By creating an efficient DAG execution framework, we are enabling Enterprises to rapidly build capabilities and continue to be at the forefront of AI adoption without getting pigeon-holed into tools or AI paradigms that may become obsolete given the lightning speed of AI evolution.
Processors can be attached to one or more routes in Javelin. For example, you can designate a specific Processor Chain to execute while triggering OpenAI calls and a completely different Processor Chain to trigger a Huggingface model.
Implementing Custom Processors
We’ve developed a framework that empowers developers to effortlessly create extensions, bypassing the intricate challenges typically associated with constructing advanced systems. This is achieved by offering a deceptively simple user interface. To further streamline the experience, we’ve integrated an accessible and intuitive DAG (Directed Acyclic Graph) canvas as the front end, concealing the sophisticated engineering and innovation behind the scenes.
Implementing a custom processor is as straightforward as implementing a simple interface in your preferred programming language, including Go, Python, or TypeScript. Our platform automatically furnishes a fully managed runtime environment for these extensions and seamlessly integrates them into your processing flows.
Leveraging Javelin APIs or the DAG canvas, you can effortlessly introduce a new processor or adjust the processing sequence, facilitating fully customizable workflows. These workflows are designed to be triggered in real time, offering flexibility and control over your processing tasks.
Javelin provides automatic lifecycle management, latency tracking, and usage statistics for each processor, enabling teams to adopt and track these systems in production fully.
Future-Proofing Your AI Journey
Javelin’s Processor framework empowers enterprises to build capabilities and navigate AI's rapid evolution. It ensures that organizations are not confined to specific tools or paradigms that risk obsolescence, thereby future-proofing their AI journey.
Within the Javelin platform, processor chains offer a robust and adaptable mechanism for orchestrating the lifecycle of LLM requests and responses. These chains guarantee seamless and secure data transmission throughout the enterprise's ecosystem by granting meticulous control over each phase — from the initial receipt to the final output. This method bolsters operational efficiency and reliability and amplifies opportunities for innovation and compliance in an ever-changing enterprise environment.
Experience the Future of Enterprise AI with Javelin
We invite you to explore Javelin’s capabilities and discover how we can transform your AI workflows, ensuring they are more secure, efficient, and aligned with your business objectives. Today we are releasing Javelin Processors to a select set of customers but will be expanding it to everyone soon.
Having developed platforms for some of the world’s largest enterprises, our expertise in navigating the complex AI integration landscape is deep and broad. If you’re pondering the intricacies of adopting Large Language Models (LLMs) or looking to transition your LLM prototypes to full-scale production, we’re here to share our knowledge and insights.
Let’s connect and explore the possibilities together!
