WebSocket API: Sec-WebSocket-Protocol (Subprotocol) header support

Fundamentally, WebSocket is a low-level protocol which allows you to send a data frame over HTTP after the handshaking. To leverage it correctly, the client and the server should use an application-level protocol describing how the message would look like, what we call “subprotocol”.
During the handshake, the client would ask the server what kind of subprotocols are supported by sending Sec-WebSocket-Protocol header.

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

Then the server would select a subprotocol to use and let the client know which one would be used by sending Sec-WebSocket-Protocol header.

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat

However, if you ever tried to use subprotocol with WebSocket API in API Gateway in the past, you found that it was not possible. Why Amazon API Gateway didn’t consider this?

By design, WebSocket API is subprotocol-agnostic not to lock you down to the specific protocol(s). It is a fair criticism to say it is biased for JSON message since routeSelectionExpression supports JSON path only, but if you’re willing to route within your integration, you can use any data format with $default route.
WebSocket API is conceptually abstracted as a Serverless stateful service which bridges to the stateless world via routes. Strictly speaking, you own WebSocket API, not WebSocket server itself. There is a flip side: You don’t have a control for the WebSocket server and it does not wish to agree any subprotocol.

https://tools.ietf.org/html/rfc6455#section-4.2.2

If the client’s handshake did not contain such a header field or if the server does not agree to any of the client’s requested subprotocols, the only acceptable value is null. The absence of such a field is equivalent to the null value (meaning that if the server does not wish to agree to one of the suggested subprotocols, it MUST NOT send back a |Sec-WebSocket-Protocol| header field in its response).

So it had been subprotocol-ignorant rather than subprotocol-agnostic. What if you want to take a control for the subprotocol selection only? You understand that the subprotocol handling itself must be done by yourself, but you want to control Sec-WebSocket-Protocol header. Since the handshake is handled by API Gateway, API Gateway should allow to respond with the header.

Since $connect route handles the handshake, you should be able to access Sec-WebSocket-Protocol header in the request from your integration, then return a selected protocol as Sec-WebSocket-Protocol header in the integration response. Then the header in the $connect route response should be sent to the client. The response part was the showstopper since it didn’t allow you to return the header.

When you tested with wscat (wscat has -s, --subprotocol <protocol> option to send the header during the handshake), you had get this error.

wscat -c 'wss://YOUR_API_ID.execute-api.YOUR_REGION.amazonaws.com/beta' -s 'myprotocol'
error: Server sent no subprotocol

Now, the showstopper was gone as of today.

https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-api-gateway-allows-subprotocols-on-websocket-api-connection/

Return Sec-WebSocket-Protocol header from the $connect

This is pretty much explained in the public document. You need to configure $connect route and it should return the header. Let’s imagine you’re using Lambda proxy integration. Here is an example of Lambda function:

exports.handler = async (event) => {
    if (event.headers != undefined) {
        const headers = toLowerCaseProperties(event.headers);
        
        if (headers['sec-websocket-protocol'] != undefined) {
            const subprotocolHeader = headers['sec-websocket-protocol'];
            const subprotocols = subprotocolHeader.split(',');
            
            if (subprotocols.indexOf('myprotocol') >= 0) {
                const response = {
                    statusCode: 200,
                    headers: {
                        "Sec-WebSocket-Protocol" : "myprotocol"
                    }
                };
                return response;
            }
        }
    }
    
    const response = {
        statusCode: 400
    };
        
    return response;
};
function toLowerCaseProperties(obj) {
    var wrapper = {};
    for (var key in obj) {
        wrapper[key.toLowerCase()] = obj[key];
    }
    return wrapper;
}

This function accepts ‘myprotocol’ only, unless it returns 400 for other protocol. Now deploy your API.

Test with wscat

When "myprotocol" isn't included, the server will respond with 400 and the connection won’t be established. When it is available, the server will accept the connection.

I hope this unblock your use case to use WebSocket API.