The Next Step in MCP: The Scalable Power of the MCP Mesh

The GenAI world is buzzing, and the latest frequency everyone’s tuning into is Agentic AI. We’re moving beyond simple chatbots to AI systems that can do things — interact with software, access data, and perform complex tasks. At the heart of this shift, a new standard is emerging: the Model Context Protocol (MCP). Hailed as the potential lingua franca for AI interaction with external systems, MCP promises to standardize how AI models request and receive context, use tools, and access resources. It’s for sure an exciting prospect. AI can now (finally) seamlessly integrating with your calendar, project management tools, or databases with a protocol that is backed by major players like Anthropic and OpenAI, and that offers a well-documented, albeit complex, specification for this future.

But let’s pause the hype train for a moment. As promising as MCP is, scratching beneath the surface reveals significant hurdles to widespread, production-ready adoption. If we want MCP-based solutions to be the robust foundation for the next generation of AI applications, we need to talk about scaling it and making it enterprise-ready.

MCP for Enterprise Applications

Imagine a complex enterprise environment. You don’t just have one AI application talking to one tool. You have multiple AI applications needing access to a diverse set of MCP-enabled services — connecting to databases, project trackers (like Jira), document repositories (like SharePoint or Google Drive), and internal APIs. Each connection needs to be secure, reliable, observable, and managed.

Here’s the first blocker: Most current MCP server implementations and demos rely on stdio (standard input/output) for transport. This works fine for local development, but it is unsuitable for web-based applications or scenarios involving multiple users connecting to shared MCP resources over a network. We need robust HTTP transport to become the norm for MCP to break out of the local sandbox.

But thare’s more. In an enterprise applications, if you want to access a resource then Authentication and Authorization (AuthN and AuthZ) are mandatory. Even if MCP strongly advises developers to spend effort in implementing them (which is good), it does not provide implementations of these features out-of-the-box, and such activities adds overhead and complexity that require significant effort.

MCP Mesh: MCP Meets the Service Mesh

Recently, I had the opportunity to attend the Google /dev/cloud day in Milan, and one piece of advice resonated strongly:

There’s no secret sauce specific to making GenAI production-ready. If you understand Software Engineering principles, you already have the essential ingredients.

And I can’t agree more. While the capabilities of large language models are novel, deploying them reliably, securely, and scalably in production isn’t magic — it’s engineering. The challenges we face with GenAI applications — managing dependencies (like MCP tools), ensuring security, maintaining observability, handling traffic, and achieving resilience — are fundamentally the same challenges software engineers have been tackling for years in distributed systems and microservice architectures. Therefore, established practices, patterns, and principles from software engineering are essential prerequisites for building production-ready GenAI systems.

Then I started thinking: Let’s consider a standard enterprise scenario with a complex microservice architecture. If every microservice needs to adhere to the same security rules, integrate with observability tools, and manage network communication reliably, all with minimal impact on the core service code, what established pattern or architecture would we apply? The answer for many is a Service Mesh.

A Service Mesh (like Istio or Linkerd) is an intelligent infrastructure layer that manages communication between microservices. By deploying lightweight proxies (sidecars) alongside each service instance, the mesh handles tasks like traffic management, security policy enforcement, and telemetry collection — abstracting this complexity away from the application logic itself.

Istio architecture diagram

I strongly suggest you to take a look at this talk (IstioCon 2021) to understand how the service mesh pattern helped eBay enforce observability and security across numerous microservices.

Similarly, for large MCP server pools to be production-ready, they require the same level of care as microservices, addressing needs like:

• Observability
• Security
• Flexible release processes (A/B testing, blue/green deployments, etc.)
• Availability
• Discoverability
• Resilience (rate limiting, retries, failovers, etc.)

In large companies, you might have dozens of GenAI applications (chatbots, automation tools, data analysis assistants) needing to talk to potentially dozens or hundreds of different MCP Servers. This raises questions:

• How can GenAI Applications be aware of which MCP servers are available at any given time?
• How can enterprises ensure that all MCP servers adhere to the same policies and best practices?
• What effort is required to take an existing MCP server and integrate it into my server pool?

It seems clear we’re facing an already-solved problem; we need an ‘MCP Mesh’ (perhaps a new term has been coined?).

MCP Mesh

Govern MCP Mesh with the MCP Gateway

While the Service Mesh excels at managing the operational aspects of the backend MCP Servers, a remaining challenge lies on the client side (outside the mesh), specifically for the developers building the GenAI applications. Even with the mesh in place, requiring every individual GenAI application to directly find the right server and connect to potentially dozens or hundreds of different mesh-managed MCP endpoints introduces significant complexity for the application developers themselves. Indeed, MCP clients would still need to embed logic to:

1. Discover and Select: Figure out which specific mesh service address corresponds to the required MCP capability.
2. Manage Connections: Handle the connection lifecycle to various endpoints (managing Server-Sent Events at scale can be challenging).
3. Implement Application-Level Logic: Potentially duplicate logic across applications for things like request formatting or initial validation before sending requests into the mesh.

This naturally leads us to the next logical step in architecting the solution: introducing a dedicated Gateway.

The full MCP Mesh architecture (with MCP Gateway)

An MCP Gateway can act as a specialized centralized entry point and intelligent aggregator specifically designed for MCP interactions. Its primary role is to simplify the experience for GenAI application developers by providing a Single Endpoint to communicate with the MCP Server Pool, reducing the need for managing several connections. Instead, based on the incoming request, the Gateway understands the landscape of available MCP Servers (leveraging the Service Mesh’s discovery capabilities) and intelligently routes the request from applications to the appropriate backend server within the mesh.

In this sense, the Gateway provides a unified view of the full MCP capabilities to different applications or principals, acting like an API Facade tailored for MCP. Indeed, it becomes the ideal place to enforce cross-cutting, application-aware policies, such as authenticating the calling application (or end-user) and authorizing access only to the specific MCP functions that application is permitted to use, before the request even enters the internal mesh network.

Example: The CorpAssist chatbot

Imagine an enterprise chatbot called “CorpAssist” designed to help employees with various tasks.

The Scenario: An employee asks CorpAssist: “Please summarize the Q1 strategy document for ‘Project Titan’ and tell me who the current project lead is.”

CorpAssist needs to interact with two different backend systems via MCP:

1. A Document Repository (like SharePoint or Google Drive) to find and summarize the strategy document (using an MCP Tool or Resource).
2. A Project Management System (like Jira or an internal database) to find the project lead for ‘Project Titan’ (using an MCP Tool).

CorpAssist example

Here’s how the simplified request flows through the MCP Gateway and the MCP Mesh architecture:

1. User Request: The employee types the request into the CorpAssist chat interface.
2. CorpAssist -> MCP Gateway: CorpAssist parses the request and determines it needs external information via MCP (found by listing the available resources and tools exposed by the gateway). It formulates two distinct MCP requests: one for the document summary, another for the project lead. CorpAssist sends both requests to a single, known endpoint: the MCP Gateway (e.g., https://mcp-gateway.internal.company.com). It doesn’t need to know the specific addresses of the underlying SharePoint or Jira MCP servers.
3. MCP Gateway Processing: The Gateway verifies CorpAssist’s identity (e.g., via a service token) and checks if CorpAssist (and perhaps the specific employee making the request) has the general permissions required to access document summaries and project lead information based on centrally managed policies.
4. Intelligent Routing: The Gateway inspects the requests and determines: The “summarize document” MCP request must go to the Document Repository MCP Server. The “find project lead” MCP request must go to the Project Management MCP Server.
5. Service Mesh Communication (Gateway -> Mesh -> MCP Servers): The Gateway, via its own sidecar proxy, initiates secure (mTLS) connections through the Service Mesh towards the target services. The Service Mesh Control Plane directs the traffic, potentially load balancing if multiple instances of the Document or Project Management MCP Servers exist. The requests arrive at the sidecar proxies running alongside the target MCP Servers (one for the Document Repository server, one for the Project Management server). These sidecars handle the secure mTLS connection termination, check if the requester has sufficient permissions to ask for the specific resource/tool, enforce any finer-grained policies (like rate limits on specific tools), and record logs and trace data for observability. Once these checks are done, the sidecars forward the requests to their respective MCP Server processes locally.
6. MCP Server Execution: The Document Repository MCP Server receives the request, interacts with its backend (e.g., SharePoint API) to find the ‘Project Titan Q1 strategy doc’, performs the summarization, and packages the summary into an MCP response. Similarly, the Project Management MCP Server receives its request, interacts with its backend (e.g., Jira API) to look up ‘Project Titan’, find the current lead, and packages the name into an MCP response.
7. Response Journey (MCP Servers -> Mesh -> Gateway -> CorpAssist): The MCP responses travel back through their respective sidecars (logging egress details), across the secure Service Mesh network, and arrive back at the MCP Gateway. The Gateway may perform final checks before sending a consolidated result back to the CorpAssist application.
8. CorpAssist -> User: CorpAssist receives the summary and the project lead’s name from the Gateway and presents the complete answer to the employee in the chat interface.

Final Thoughts: Engineering the Agentic Future

The Model Context Protocol (MCP) represents a significant step towards standardized interaction between AI models and the vast landscape of digital tools and resources. The excitement around GenAI is warranted, but it doesn’t negate the need for sound engineering. The path to production-ready, scalable agentic AI built on MCP lies in applying the proven principles and patterns of modern software engineering.

This isn’t about finding a magical “GenAI” solution; it’s about leveraging the robust tools and practices we already have to build the next generation of intelligent applications.

The MCP Mesh concept aims to make a step in this direction by offering a concrete implementation framework applying gateway and service mesh patterns to manage MCP interactions securely and ready for production.

Hope you found this article useful.

Danilo Trombino | LinkedIn