Transport layer security explained

Overview of transport layer security protocols and mechanisms

Anupama Pathirage
8 min readMay 12, 2023

Transport Layer Security (TLS) is a cryptographic protocol that provides secure communication over the Internet by encrypting data sent between two endpoints. TLS is the successor to SSL (Secure Sockets Layer), and is often used interchangeably with SSL, although TLS is the more modern and secure protocol.

Photo by Towfiqu barbhuiya on Unsplash

How is it used with HTTP?

HTTP (Hypertext Transfer Protocol) is the protocol used for communication between web servers and web clients, such as web browsers. When HTTP is used over TLS, it is referred to as HTTPS (HTTP Secure), and the communication is encrypted and authenticated using the TLS protocol. HTTPS is commonly used for secure communication over the Internet, particularly for sensitive transactions such as online banking and e-commerce.

Once the TLS connection is established, data transmitted between the client and server is encrypted and authenticated using the agreed-upon cryptographic parameters, providing a secure channel for communication. This helps to protect against eavesdropping, data tampering, and other types of attacks that can compromise the security and confidentiality of the data being transmitted.

When a client, such as a web browser, initiates a connection with a server over HTTPS, the TLS protocol performs a series of steps to establish a secure connection between the client and the server. This typically involves exchanging public keys, verifying the authenticity of the server, and negotiating a set of cryptographic parameters that will be used to encrypt and decrypt the data transmitted between the client and the server.

There are two different approaches to securing communication over SSL/TLS.

  • One-way SSL
  • Two-way SSL (mutual SSL)

One-way SSL

In one-way SSL, only the server is required to present a certificate to the client to authenticate itself. The client does not need to present a certificate to the server for authentication.

In one-way SSL, the client establishes a connection to the server using the HTTPS protocol, and the server presents its digital certificate to the client. The client then verifies the server’s certificate against a list of trusted certificates to ensure that the server is authentic and that the communication is secure.

One-way SSL is commonly used in scenarios where the client does not need to be identified or authenticated, such as when accessing public websites or retrieving information from a server. One-way SSL provides a level of security that protects against eavesdropping and data tampering, but it does not provide any assurance that the client is trustworthy.

One-way SSL

Two-way SSL (Mutual SSL or mTLS)

In Two-way SSL, both the server and the client are required to present digital certificates to authenticate themselves to each other.

The client establishes a connection to the server using the HTTPS protocol, and the server presents its digital certificate to the client. The client then sends its own digital certificate to the server, and the server verifies the client’s certificate against a list of trusted certificates to ensure that the client is authentic and that the communication is secure.

Two-way SSL is commonly used in scenarios where both the client and the server need to be identified or authenticated, such as in online banking, e-commerce, or other applications that involve sensitive data. By requiring mutual authentication, two-way SSL provides an additional layer of security and helps prevent man-in-the-middle attacks.

Two-way SSL (mTLS)

The Certificate and Private Key Files

In SSL/TLS communication, a certificate file and a key file are two components used to establish a secure connection between a client and a server.

A certificate file contains a digital certificate issued by a trusted certificate authority (CA). The certificate includes the public key of the server, along with information about the identity of the server, such as its domain name. When a client connects to a server, the server presents its digital certificate to the client, which can then verify the identity of the server and the authenticity of the connection.

A key file on the other hand, contains the private key that corresponds to the public key in the server’s digital certificate. The private key is used to decrypt data that has been encrypted with the public key, and to sign data to verify the identity of the server. The private key must be kept secure and should only be accessible to authorized parties.

Keystores and truststores

Keystores and truststores are repositories that contain cryptographic artifacts like digital certificates and private keys that are used for cryptographic protocols such as TLS.

What is a keystore?

A keystore contains personal certificates, plus the corresponding private keys that are used to identify the owner of the certificate. You only need this if you are a server, or if the server requires client authentication. (i.e. mutual SSL)

What is truststore?

TrustStore stores certificates from third parties your application communicate, or certificates signed by CA which can be used to identify third party.

While it is possible to validate certificates on the fly without a truststore, it is not recommended. When a client receives a server’s certificate during the SSL/TLS handshake, it needs to verify the certificate’s authenticity before continuing with the connection. This involves checking that the certificate was issued by a trusted CA and that the certificate has not been revoked or tampered with.

Without a truststore, the client has no way to verify the authenticity of the certificate, and any attempt to validate the certificate on the fly could be vulnerable to attacks such as Man-in-the-Middle (MitM). Therefore, it is recommended to use a truststore to ensure secure SSL/TLS communication.

Laptops and other computing devices usually have a default truststore that is pre-installed with a list of trusted root CAs. The list of trusted root CAs in the truststore varies depending on the operating system and other factors, but typically includes well-known CAs.

When a client application (such as a web browser or email client) attempts to establish a secure connection with a server, it checks the server’s certificate against the list of trusted root CAs in the truststore. If the server’s certificate was issued by a trusted CA, the client will accept the certificate and proceed with the secure connection. If the server’s certificate was not issued by a trusted CA, the client will typically display a warning or error message, indicating that the certificate is not trusted.

If a CA is not listed in the truststore, you can add it manually. It is important to note that adding a CA to the truststore manually should be done with caution, as it can potentially open up security vulnerabilities. Only add a CA to the truststore if you are confident that it is trustworthy and necessary for your specific use case.

Different store types

  1. Java KeyStore (JKS) — This is a proprietary format used by Java-based applications, such as Tomcat, JBoss, and Apache HTTP Server.
  2. PKCS#12 — This is a standard format that is supported by many applications and operating systems, including Microsoft Windows, macOS, and iOS. It can also be used in Java-based applications with the help of third-party libraries.
  3. OpenSSL — This is a widely used open-source toolkit for SSL/TLS communication, and it includes support for creating and managing keystore files in various formats.
  4. Windows Certificate Store — This is provided by the Windows operating system, which is used to store digital certificates and private keys. It can be accessed using the Microsoft Management Console (MMC) or the Certificate Manager utility.
  5. Android KeyStore — This is used by Android devices to store cryptographic keys and certificates. It is managed by the Android KeyChain API, which provides a secure way for applications to access the keys and certificates stored in the keystone.

What are Certificate Authorities (CA)?

CAs are trusted entities that issue digital certificates to authenticate the identity of a website or organization. When a website or organization requests a digital certificate from a CA, the CA verifies their identity and issues a certificate containing their public key. The digital certificate is then installed on the website’s server, allowing it to encrypt data and establish secure connections with clients.

They help prevent fraudulent websites and phishing attacks that attempt to trick users into revealing sensitive information.

There are many commercial CAs that offer digital certificate services, as well as several public CAs that provide free certificates. Some well-known commercial CAs include DigiCert, GlobalSign, and Symantec, while some popular public CAs include Let’s Encrypt, Comodo, and StartCom. It’s important to choose a reputable and trusted CA when obtaining digital certificates to ensure the security and authenticity of online transactions.

What is a self-signed certificate?

A self-signed certificate is a digital certificate that is signed by the same entity whose identity it certifies rather than by a trusted third-party Certificate Authority (CA). In other words, a self-signed certificate is not issued by a trusted CA and is not recognized as valid by web browsers or other applications that use SSL/TLS for secure communication.

Self-signed certificates are often used in testing, development, and other situations where SSL/TLS encryption is needed but a trusted certificate is not required. They can be generated and installed on a web server or other device, providing encryption and allowing secure communication between the server and clients.

However, when a client attempts to establish a secure connection to a server with a self-signed certificate, the client’s web browser or application will typically display a warning or error message, indicating that the certificate is not trusted. The user must then manually verify and accept the certificate in order to proceed with the secure connection.

It’s important to note that while self-signed certificates provide encryption, they do not provide authentication. Without a trusted third-party CA to verify the identity of the server, there is no way to ensure that the server is who it claims to be, and a malicious actor could potentially intercept or manipulate the communication. Therefore, self-signed certificates should not be used in production environments or for public-facing websites.

How to create a self-signed certificate?

To create a self-signed certificate locally, you can use OpenSSL which is an open-source cryptographic library.

openssl req -x509 -out localhost.crt -keyout localhost.key \
-newkey rsa:2048 -nodes -sha256 \
-subj '/CN=localhost' -extensions EXT -config <( \
printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

--

--

Anupama Pathirage

Open Source Contributor | Developer — Ballerina Language| Director of Engineering — WSO2 | Travel 🏝 . Photography 📸 | 🇱🇰 | Twitter: https://bit.ly/356icnr