TCP, TLS and Handshakes

Adam Sullivan
5 min readOct 6, 2022

--

In order for data transfer to happen smoothly, reliably, and securely over the Internet, there needs to be a way for different layers of a network to cooperate. Conceptually we can think of the Internet as being a system of layers, with each layer processing and sharing data, while also communicating to other layers of the system. Luckily, protocols have been established to implement steps within and between the various layers to make this sort of cooperation possible. Whether it’s the physical infrastructure of devices in our offices, or more abstract layers such as applications running on our phones, these protocols oversee a wide array of tasks. In this blog post I’ll aim to parse out the roles of the TCP and TLS protocols, when they’re used and how they’re connected in hopes of clarifying these concepts for myself and anyone interested.

The typical use case for these protocols would occur in a client-server model where the client (usually a browser), sends a request to the server (generally multiple devices), which provides a response to the client. The Transmission Control Protocol or TCP is one such protocol that works at the transport layer of the Internet and provides numerous benefits such as flow control and congestion avoidance. TCP is responsible for establishing the initial connection between a sender and receiver and allows for reliability on top of the unreliable network protocols of the layers below. TCP also looks after each piece of data and ensures it arrives in the correct order. Using headers stored in the TCP Protocol Data Unit, or segment, we’ll see there are a couple key fields which offer functionality.

Photo by CloudFare at https://cf-assets.www.cloudflare.com/slt3lc6tev37/5aYOr5erfyNBq20X5djTco/3c859532c91f25d961b2884bf521c1eb/tls-ssl-handshake.png

Let’s take a look at some of the more important fields in the header section of a TCP segment. We’ll then examine how these are related to the all-important TCP Handshake. Two fields found in a segment are the Source Port and Destination Port which are key components in multiplexing, allowing multiple signals from different applications to share bandwidth and be transmitted over one line. Segments have a Checksum field in IPv4 which is part of the error detection aspect of TCP reliability (Note that in IPv6 checksums are not included due to their redundancy with checksums at other layers). Sequence Number and Acknowledgement Number fields endow segments with the capability of in-order delivery mentioned above. The Window Size field is useful for flow control. TCP has the ability to prevent the sender from transmitting too much data to the receiver thanks to flow control. Lastly, the flag fields are used in the TCP handshake.

The TCP three-way handshake is used to establish an initial connection between application processes. Due to the connection-oriented nature of TCP, this is an important first step, as no information will be passed between sender and receiver until after the initial connection is established. During the handshake, the sender and receiver exchange flags through segments. The sender will first send a segment with the “SYN” message to the receiver with the SYN flag set to 1. To acknowledge the receipt of this flag the receiver sends back a “SYN ACK” message with both these flags set to 1. The sender will finally send an “ACK” message and upon sending this, the sender can now start sending data. The receiver must wait until receiving this last “ACK” message before it too can start sending data to the sender. These flags are used to manage the connection state between application processes (therefore they also play a role in terminating connection through what’s known as the TCP four-way handshake). The rounds of messages sent back and forth during the TCP handshake as well as some of the features used such as flow control and congestion avoidance, means that there is some expense to using TCP. However, the reliability provided, in-order delivery and data retransmission can be worth the cost especially in cases like networked applications where we depend on not losing data.

TLS or Transport Layer Security (formerly SSL), like TCP, has a handshake of its own. In fact, these two protocols are used in tandem, and TLS is not used with UDP. TLS is an important part of Internet security due to the unsecure nature of HTTP requests and responses being sent in plain text. There are three important services that TLS offers: Encryption, Authentication, and Integrity.

The TLS handshake is where encrypted keys are exchanged so that each party may decode messages securely. The handshake begins with the client sending a message to the server, similar to the TCP handshake. The server then sends its own SERVERHELLO message, its certificate containing the public key, and other information which determines the protocol version and cipher suites to be used for encryption. The server will also send a SERVERHELLODONE message to the client. Asymmetric key exchange happens after this message is received by the client and then symmetric keys can be exchanged and used for the duration of communications between client and server.

The certificate mentioned above is part of a chain of trust used for TLS Authentication to verify identity. Using a combination of certificate names, signatures and public keys, a hierarchy of Certificate Authorities create multiple layers of security. The last service that TLS is known for is integrity. When TLS encapsulates application data it makes use of a Message Authentication Code, also known as a tag, to check if any messages have been meddled with. As with TCP, the services that TLS provides, don’t come without some overhead cost. However, using TLS allows for use of the HTTPS protocol, which is considered very secure, especially when used with TCP.

In an HTTPS request/response cycle, TCP operates at the Transport layer while TLS operates between the Transport layer and Application layer. The TCP three-way handshake takes place first, followed by the TLS handshake. While both TCP and TLS provide useful services, these services cause latency because of the time taken to establish connections. Combining TCP and TLS is a reliable and secure way to create networked applications.

--

--