Netty at Layer: Powering Client API
Modern communication platforms deal with an ever growing number of protocols. In order to keep pace, network technology stacks must be built to remain flexible and extensible.
To achieve this flexibility, companies such as Apple, Facebook, Twitter, and others, turn to Netty, an open source Java NIO framework that enables the development of high performing protocol servers and clients. Netty has also found a home here at Layer, where it helps us handle the flow of millions of messages every day through our Client API.
What Netty brings us
Layer is a distributed system with highly dependent services, where each one may spend significant time waiting for responses from other dependent upstream services. A non-blocking, event-driven I/O library like Netty allows these services to concurrently process requests, while taking full advantage of our server hardware.
Netty is also protocol agnostic, and adding a new protocol to Netty is as simple as creating the appropriate event handlers. This extensibility has led to many community-driven protocol implementations including HTTP/2, WebSockets, SPDY, and MQTT [1].
Additionally Netty provides:
- A unified API with which to build network services
- A powerful threading model
- Reduced resource consumption and zero copy support
- A vibrant community and wide industry adoption
All in all it’s an excellent framework on which to base our stack.
Integrating Netty into Layer’s Client API
Entry server
Layer’s entry server is built atop Netty to make use of performance goodies and HTTP/2.
Our HTTP/1.1 pipeline contains handlers for SslHandler (TLS support to Connection), HTTP codecs (Encoder and Decoder), HTTP message aggregation (HTTP Object Aggregator), CORS handler, and app specific handlers such as nonces, sessions, WebSocketServerHandler, etc.
We’ve adopted best practices from the community [2] such as:
- OpenSSL based SslEngine to speed-up connections.
- epoll based native socket transport in Linux (for higher performance and produces less garbage)
- PooledByteBufAllocator (to reduce allocation/deallocation time)
- Shared stateless channel handler across channels to reduce GC (nonce and session handlers are stateless and both are annotated as @ChannelHandler.Shareable)
The system’s NoncesHandler uses Memcache to cache nonces for identity-token validation and session creation.
REST API and WebSockets
Additionally we take advantage of Netty’s RFC 6455 compliant WebSocket support to expose a REST API and bi-directional WebSockets for real-time notifications.
Each REST API request comes with the Authorization header containing a session-token. The entry server caches this session-token in read-through SessionCache (a cache miss will trigger an event to fetch session info from Authentication service), and each session-token is then validated and authenticated to process the request (see our Client API documentation for more details on authentication challenges).
API requests for conversations and messages are passed through to a user recon engine — a process in which a user’s conversations and messages are reconciled into a given state (such as deleted, updated, seen, etc.).
The WebSocket API allows a client to send requests to the server. Each WebSocket upgrade request comes with a session-token for authentication and sub-protocol negotiation for the connection handshake. Upon successful WebSocket upgrade, the WebSocket connection is added in WebSocketConnectionManager’s DefaultChannelGroup. Each WebSocket connection is identified with a session-token and connection-id.
Request/Response/Change/Signal packets are wrapped as TextWebSocketFrame. Request packets are decoded as String and routed to a given user recon engine through the ‘in’ exchange/queue in RabbitMQ server. Consumer of the ‘in’ exchange/queue in a user recon engine will consume the request (message) and process the request (with the help of invoking HTTP requests to messaging fabric and DB inserts/updates etc)
A user recon engine can emit the response/changes to the ‘out’ exchange/queue in RabbitMQ server. Consumer of the ‘out’ exchange/queue in the entry server will consume the response/changes, meaning message headers shall contain session-token and connection-id to look up the WebSocket connection from WebSocketConnectionManager and then notification (response) will be pushed to that connection using the Netty Channel’s writeAndFlush.
Customer use cases
- Bidirectional WebSocket for a typing indicator : Clients can send a typing indicator using WebSocket to indicate that the user is currently writing a message in a conversation. This can be sent on every key press in the input and will send a user_typing ephemeral event to all the participants in a conversation.
- Bidirectional WebSocket for desktop applications with real-time collaboration such as gaming, trading, chat, etc
Sample code can be found in our Layer for Web github repo.
Future work
We are far from done with our implementation of Netty. Some of the upcoming work we are looking to tackle, now that the Client API has been released:
- RFC 7540 compliant HTTP/2 implementation (to use single persistent connection for REST and server push, to avoid head-of-line blocking issue, and to leverage performance improvements)
- WebSocket extensions for compression and multiplexing (separate logical WebSocket connections to share an underlying transport connection)
- Netty metrics
- BinaryWebSocketFrame with serialization bindings for request/response
- WebSocket rate limiting
