C++ TLS handshake on sockets

There are numerous articles on the internet about the TLS handshake algorithm. However, many of them only provide a superficial description. In this article, we will explore a C++ implementation of a specific cipher. This source code relies solely on sockets for networking and utilizes the OpenSSL library exclusively for encryption (RSA, AES, SHA).

We will examine the cipher TLS_RSA_WITH_AES_128_CBC_SHA on TLSv1.2 (RFC5246), which entails:

• The RSA key exchanging strategy
• The AES-128 algorithm for encrypting application data, with added security using the CBC method
• SHA1 hashing for generating application data checksums

You can find the complete source code here: https://github.com/xreptoid/tls-handshake.

Basic steps of TLS handshake

This is a Wireshark log capturing our TLS handshake implementation’s actions:

In fact, the TLS handshake process can be divided into two stages:

1. Hello stage: The client receives the server’s public certificate for secure handshaking (Certificates packet).
2. Key exchange stage: The client sends data for generating the AES key, using the server’s public certificate (RSA) to encrypt it and protect it from prying eyes (Client Key Exchange packet).

Other packets in stages 1 and 2 are used to ensure that both sides employ the same algorithm. Since we can choose to ignore the server packets (which we will), it is crucial to correctly create all packets on our side. If any client packet contains incorrect data, the server will not respond, making it difficult to debug if you are implementing the TLS handshake algorithm from scratch.

Selecting a host

The host credentials are hardcoded in main.cpp#L8. This example makes a request to api.binance.com using a specific host with the IP address 99.84.56.223. api.binance.com is hosted on CloudFront, which periodically rotates its hosts in DNS and occasionally changes IP addresses entirely. Therefore, it is important to ensure that the address is still active when testing, and select a new one if the current address is unavailable.

Obtaining the IP address from the hostname:

$ host api.binance.com

api.binance.com is an alias for d3h36i1mno13q3.cloudfront.net.
d3h36i1mno13q3.cloudfront.net has address 99.84.56.223

Our selected CloudFront host is active and supports api.binance.com on it:

$ curl https://api.binance.com/api/v3/time \
    --connect-to api.binance.com:443:99.84.56.223:443 \
    --http1.1 --tls-max 1.2

{"serverTime":1681509580156}

The host is compatible with our chosen cipher:

$ nmap --script ssl-enum-ciphers -p 443 99.84.56.223

...
TLSv1.2:
|     ciphers:
|       ...
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       ...

Hello stage

We need to send the initial packet called Client Hello to the server. In our example, there are two important components in this packet:

1. client_hello value — a random 32-byte buffer that will be used for AES-key generation.
2. hostname (api.binance.com) — some servers, including CloudFront, require specifying the hostname you want to connect to, rather than just the IP address to which you are already connected. Since CloudFront hosts multiple services on a fixed set of hosts, it needs to know which service’s certificate it should return.

You can find the complete packet here: tls_context.cpp#L13.

After sending the hello, we immediately receive three packets: Server Hello, Certificates, and Hello Done. Some fields that we need for AES-key generation are:

1. server_random

We need to extract the server_random from the Server Hello packet.

// Server Hello 
0x16           // ContentType = Handshake
0x03 0x03      // TLSv1.2
0x12 0x34      // 2-bytes size of the packet
0x02           // HandshakeType = Server Hello
0x12 0x34 0x56 // 3-bytes size of the next data
0x03 0x03      // TLSv1.2 
<32 bytes of server random>

Thus, the server_random is located at ServerHello[11:11+32].

2. Server’s public RSA-certificate

The server sends a chain of certificates in the Certificates packet. We only need the first one.

The data for the first certificate starts from the 12th byte.

We read 3 bytes after the 12th byte of the Certificates packet — this represents the length of the server’s public certificate. Next, we should copy this number of bytes starting from the 15th byte — this corresponds to the certificate itself.

For a better understanding of packet structures, you can use tools like Wireshark that group data with labels. Additionally, I recommend the following resources:

https://wiki.osdev.org/TLS_Handshake — provides a more detailed explanation of packets and some logical steps.

https://tls12.xargs.org/ — a byte-by-byte explanation. Note that this example uses a different TLS handshake key exchange strategy, so the Client Key Exchange packet will vary.

AES-key calculating

The client generates a premaster_secret: x03 x03 <46 random bytes>. With the RSA key exchange, both sides only need client_random, server_random, and premaster_secret for AES-key calculation.

We need to calculate a buffer called master_secret. A Pseudo-random function (PRF) is used for this purpose. Here is the PRF definition from the RFC:

PRF(secret, seed) = HMAC_hash(secret, A(1) + seed) +
                    HMAC_hash(secret, A(2) + seed) +
                    HMAC_hash(secret, A(3) + seed) + ...
where + indicates concatenation.
A() is defined as:
A(0) = seed
A(i) = HMAC_hash(secret, A(i-1))

// tls_context.cpp
bytes_t prf(const bytes_t& secret, const bytes_t& seed, int len) {
    bytes_t res;
    std::vector<bytes_t> a;
    a.push_back(seed);
    while (res.size() < len) {
        a.push_back(hmac_sha256(a.back(), secret));
        res += hmac_sha256(a.back() + seed, secret);
    }
    return subbytes(res, 0, len);
}

The master_secret is calculated using the PRF in the following manner:

master_secret = PRF(
    premaster_secret,
    "master secret" + client_random + server_random
)[:48]

As per the RFC, keys are mapped by the master_secret in this way:

// AES-key for client->server encrypting
client_write_key = master_secret[40:40+16]

// AES-key for server->client encrypting
// We use this for decrypting incoming packets.
server_write_key = master_secret[56:56+16]

Keys Exchanging stage

We send three packets to the server simultaneously:

1. Client Key Exchange

This message contains the RSA-encrypted premaster_secret, encrypted using the server’s public RSA certificate. The server needs this information to generate AES keys on its side.

// crpyto.cpp
bytes_t rsa_encrypt(const bytes_t& key, const bytes_t& buffer) {
    std::uint8_t* key_data = key.data();
    X509* x509 = d2i_X509(NULL, &key_data, key.size());
    EVP_PKEY* pkey = X509_get_pubkey(x509);
    RSA* rsa = EVP_PKEY_get0_RSA(pkey);
    std::uint8_t encrypted_buf[1000]; // FIXME
    int encrypted_size = RSA_public_encrypt(
            buffer.size(),
            buffer.data(),
            encrypted_buf,
            rsa,
            RSA_PKCS1_PADDING
    );
    return bytes_t(encrypted_buf, encrypted_buf + encrypted_size);
}

// tls_context.cpp
bytes_t TLSContext::get_client_key_exchange_packet() {
    // ...
    bytes_t premaster_secret_encrypted = rsa_encrypt(*server_public_key, *premaster_secret);
    bytes_t packet = number2bytes(premaster_secret_encrypted.size(), 2) + premaster_secret_encrypted;
    // 0x10 - Client Key exchange
    packet = bytes_t{0x10} + number2bytes(packet.size(), 3) + packet;
    // 0x16 - Handshake message; 0x0303 - TLSv1.2
    packet = bytes_t{0x16} + bytes_t{0x03, 0x03} + number2bytes(packet.size(), 2) + packet;
    // ...
    return packet;
}

0x16           // ContentType = Handshake
0x03 0x03      // TLSv1.2
0x12 0x34      // 2-bytes size of the message
0x10           // HandshakeType = Client Key Exchange
0x12 0x34 0x56 // 3-bytes size of the next data
0x12 0x34      // 2-bytes size of encrypted premaster_secret
<premaster_secret>

2. Change Cipher Spec

This is simple. We set the flag (spec=1), indicating that all subsequent client messages will be encrypted from the next packet onwards.

// Change Cipher Spec
0x14      // ContentType = Change Cipher Spec
0x03 0x03 // TLSv1.2
0x00 0x01 // 2-bytes size of the message (=1)
0x01      // spec=1

3. Client Handshake Finished

In the Client Handshake Finished message, you must accurately calculate a hash of the previous TLS handshake messages. In our case, we do this using the SHA-256 hash. Afterward, you should calculate the PRF in a specific way using this hash to obtain the value called verify_data.

You should concatenate the bodies of the following packets in the given order: Client Hello, Server Hello, Certificates, Server Hello Done, and Client Key Exchange. Technically, this means bytes from position 5 to the end of all packets with ContentType=Handshake(0x16) .

This value should be encrypted in the same manner as it will be after the handshake. Note that if you send incorrect packets to the server, you will not receive any messages from it.

// tls_context.cpp
bytes_t TLSContext::get_verify_data_packet() {
    // ...
    bytes_t handshake_sum;
    for (const auto& packet: handshake_packets) {
        handshake_sum += packet;
    }
    auto handshake_hash = sha256(handshake_sum);
    auto verify_data =
        bytes_t{0x14, 0x00, 0x00, 0x0C}
        + prf(
            master_secret,
            make_bytes("client finished") + handshake_hash,
            12);
    return encrypt_packet(0x16, verify_data);
}

Since the encryption logic is the same as for application data, I will describe it in the next section.

Application data — After the TLS handshake

At this point, the connection is secure.

AES CBC employs an Initialization Vector (IV) for additional security. Essentially, this means generating a 16-byte random buffer before encryption and inserting this buffer before the encrypted data in the packet.

In our implementation, we handle padding on our side and disable padding in the OpenSSL AES logic. You can find the AES encryption implementation here: crypto.cpp#L65.

// tls_context.cpp
bytes_t TLSContext::encrypt_packet(
    std::uint8_t content_type, const bytes_t& data
) {
    bytes_t seq = number2bytes(i_seq++, 8);
    bytes_t rechdr = {content_type, 0x03, 0x03};
    bytes_t datalen = number2bytes(data.size(), 2);
    bytes_t hash = hmac_sha1(
        seq + rechdr + datalen + data,
        keys->client_write_mac_key);
    bytes_t data_with_meta = data + hash;

    std::uint8_t padding = 16 - data_with_meta.size() % 16; // 1..16
    std::uint8_t padding_val = padding - 1; // 0x00..0x0F
    for (int i = 0; i < padding; ++i) {
        data_with_meta += {padding_val};
    }

    bytes_t enc_iv = generate_bytes(16);
    auto data_encrypted =
         enc_iv
         + aes128_encrypt(keys->client_write_key, enc_iv, data_with_meta);
    bytes_t packet = rechdr;
    packet += number2bytes(data_encrypted.size(), 2);
    packet += data_encrypted;
    return packet;
}

bytes_t TLSContext::encrypt_packet(const bytes_t& data) {
    return encrypt_packet(0x17, data);
}

bytes_t TLSContext::decrypt_server_packet(const bytes_t& packet) {
    bytes_t enc_iv = subbytes(packet, 0, 16);
    bytes_t enc_data = subbytes(packet, 16, packet.size() - 16);
    auto data_with_meta = aes128_decrypt(
        keys->server_write_key, enc_iv, enc_data);
    if (data_with_meta.empty()) {
        return data_with_meta;
    }
    auto padding = static_cast<std::uint8_t>(data_with_meta.back()) + 1;
    return subbytes(data_with_meta, 0, 
        data_with_meta.size() - padding - 20); // hash size = 20 bytes
}

Application data uses ContentType=0x17. The same logic was applied for Client Handshake Finished packet with ContentType=0x16 .

Examine tls_connection.cpp for connect() function, which contains the full TLS handshake schema, and send()/recv() implementations.

// main.cpp
auto ctx = TLSContext();
if (hostname.has_value()) {
    ctx.set_hostname(*hostname);
}
auto con = TLSConnection(&ctx, host, port);
con.connect();
std::cout << "connected" << std::endl;

std::string req = "GET /api/v3/time HTTP/1.1\r\nHost: api.binance.com\r\nAccept: */*\r\n\r\n";
con.send(req);
auto resp = con.recv()[0];
std::cout << resp << std::endl;

You should see something like this:

connected
HTTP/1.1 200 OK
Content-Type: application/json;charset=UTF-8
Content-Length: 28
Connection: keep-alive
Date: Sat, 15 Apr 2023 11:09:24 GMT
Server: nginx
x-mbx-uuid: 02748b96-893b-4830-a348-95509d5f2397
x-mbx-used-weight: 1
x-mbx-used-weight-1m: 1
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'
X-Content-Security-Policy: default-src 'self'
X-WebKit-CSP: default-src 'self'
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, HEAD, OPTIONS
X-Cache: Miss from cloudfront
Via: 1.1 33a8c80e33219ff09d001534e1f845c4.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: NRT20-C3
X-Amz-Cf-Id: -C-Nu6-mOq8M6kJgokkErIlVv7_fD8X8p5lwBNgHDE1rv-XoZTkYlg==

{"serverTime":1681556964107}