How the Web Works — what happens when you type a URL and press “Enter”
The Internet is a very amazing part of our lives that came into existence fairly recently. In short, it is a global network that provides data, information, and communication through interconnected networks, guided and wireless technologies, and communication protocols. In the case of websites, it gives us the ability to connect to servers in huge data centers all over the world to access information and content stored on them.
The accessibility of the World Wide Web, one of the services offered over the Internet, is amazing in that anyone of almost any age can access it by simply typing some address into a web browser like Internet Explorer or Chrome and pressing “Enter”. This makes it very easy to take all the intricacies and complexity of the internet for granted and some people probably cannot imagine a world without it because they grew up with it. In this blog post, I will attempt to explain, from what I understand, what actually happens when a user types an internet Uniform Resource Locator (URL) like www.holbertonschool.com into a web browser and presses “Enter”.
The Process
The basic outline of steps when typing in a URL is listed below. I will go into more detail in the sections below.
- The website Uniform Resource Locator (URL) is typed into the web browser
- The browser performs a Domain Name System, (DNS) record lookup with a set series of steps in order to find the respective Internet Protocol (IP) address of the server that hosts the website
- A Transmission Control Protocol (TCP) connection is opened between browser and host
- An Hypertext Transfer Protocol (HTTP) request is sent to the website host server
- The server, which contains tools such as load balancers, web servers, application servers, firewalls, and databases, handles the request sent by the browser
- The server sends back an HTTP response with the requested web page attached
- The browser displays the Hypertext Markup Language (HTML) and sends out additional HTTP requests for the CSS, Javascript, and other plugins.
Step 1 — The URL is typed into the web browser
A URL is a syntactical way to identify a unique resource over the web. It has the following form and consists of four distinct parts:
protocol://hostname:port/path_to_resource_file
- Protocol — the protocol used by the browser and server (usually HTTP or HTTP secured [HTTPS])
- Hostname — the DNS domain name or IP address of the server hosting the content
- Port — The TCP port number that the server is listening to for HTTP requests sent by the browser
- Path to Resource File — The name/location of a specific document that you are trying to access
In our example, the URL www.holbertonschool.com contains no specific protocol, port, or file path, so the browser will likely use HTTP as the protocol, a port 80, which is default if no port is specified, and request the website at its root (/).
The hostname itself is made up of various levels of domain names. In www.holbertonschool.com, there are three domains in total. The www makes up the subdomain, holbertonschool is the primary domain, and .com is the Top Level Domain (TLD).
The browser can use all of these domains to identify the IP address of the server that hosts the website. The web uses a protocol called Transmission Control Protocol/Internet Procotol (TCP/IP) to communicate with servers. At a high level, TCP is a protocol that allows two hosts to communicate with each other and send reliable, ordered, and error-checked packets of data through receiving and sending locations called ports. An IP address is like the address to any device connected to the Internet.
Step 2 — The Browser Performs a DNS Record Lookup for the IP address
In order to know what server to ask for web content, the browser must know what IP address to contact that will serve the content that the user has specified. An IP address is a series of numbers which uniquely identifies all devices and things connected to the Internet and allows direct communication to them. There are two types of IP addresses: IPv4 and IPv6 which differ in their format. IPv4 is a series of 4 numbers from 0 to 255 separated by . such as 127.0.0.1 which is the local address of the computer you are currently on. IPv6 is a series of 8 groups of hexadecimal numbers separated by colons which solves the issue of saturated IP addresses since it allows trillion upon trillion times the number of IP addresses offered by IPv4. In our example, the browser must find the IP address of the server that hosts the holbertonschool website.
The reason that plain English domains exist for websites is because of the difficulty of remembering sequences of numbers. It is much easier to remember words and phrases. Thus, the Domain Name System (DNS) exists because of this translation between domain names and IP address numbers.
The browser does this through a DNS record lookup that goes through the following steps. If at any point, the IP address is found, the steps below it are simply skipped and the IP is sent back to the web browser.
- Browser cache — the browser maintains a collection of DNS records for websites that you have recently visited. The browser searches this first.
- Operating System cache — the operating system also maintains a collection of DNS records that can be searched for the information it needs. The browser communicates with the OS.
- Internet Service Provider (ISP) cache — the operating system communicates with the Resolver server of your ISP. Your ISP maintains its own DNS Resolver server which can communicate with other servers associated with various domains all over the world for the specific information you requested.
- If the IP address is not found, it does not exist and an error is returned
In the last step, the operating system contacts the resolver which communicates with the ISP DNS server to initiate a query to find the IP address by contacting various other DNS servers called Name Servers which search for the website based on the domains that the user typed into the browser (ie. www.holbertonschool.com in this case). The ISP DNS root server gives the resolver the address of the root name server (in this case, .com) and the resolver communicates with the root name server which finds the various name servers associated with the domain entered and reserved by the domain registrar (in this case, the name server is holbertonschool.com). The name server will find the IP address of www.holbertonschool.com in its records and give it to the resolver which then returns it to the browser.
Step 3 — A TCP Connection is Opened
The browser now has the IP address and is able to build a connection with the server that is associated with this IP address. TCP governs the reliability of the transmission of data and communicates with the website host server asking whether there are ports open to receive and send packets of data. In this case, since no port is specified in the URL, the browser will ask by default if port 80 is open for connections by sending a synchronization packet. If port 80 is not being used, the website host server will send back a synchronization/acknowledgement packet in which case, the browser will send back its own acknowledgement packet to the website host server which starts the TCP connection for data transfer. If any other ports are specified in the URL, these will be checked via the TCP protocol.
The TCP and IP protocols typically go hand-in-hand such that a full IP address usually specifies the actual IP address followed by a port number separated with a colon. For example, if the localhost was to be communicated at port 80, the full form of the IP address would look like the following: 127.0.0.1:80
Step 4 — An HTTP Request is Sent to the Website Host Server
The browser will then send a translation of the URL into the specified protocol, in this case, an HTTP GET request for the root page of the holbertonschool.com domain. This HTTP request message contains information in the form of various headers that define how the information and connections are to be handled by the servers.
When the server receives this request message, it interprets this request, maps the request to a file or program on the server itself, and sends back a response. This response can even be an error if something went wrong or if the file or location does not exist on the server.
The issues of security have led to the need for more secure connections. A common one that is used by many websites nowadays is HTTPS which is the secure version of HTTP. In this system, Secure Sockets Layer (SSL) protocol is used to build a secure connection through a Public Key Infrastructure (PKI) system. This consists of a public key and a private key to encrypt and decrypt all communications. Anything encrypted using the public key can only be decrypted using the associated private key. When an HTTPS connection is requested, the website will send its SSL certificate to the browser which contains the public key for encryption and initiates an “SSL handshake”. The information traveling through the TCP connection in this way will be encrypted using the public key and the information sent back will be decrypted using the private key contained in the website host servers. All of this communication occurs over TCP port 443, so if www.holbertonschool.com’s servers are capable of HTTPS, connectons to port 80 will be redirected to port 443.
Step 5 — The Server Handles the Request Sent by the Browser
The website host server IP address contains the location of the web infrastructure that will handle the request. The goal of efficient web infrastructures is to build redundancy so that any errors or downtime will not completely halt all processing operations.
The initial connection might be a connection to a server that has a load balancer (like HAProxy) installed on it which uses algorithms to systematically direct traffic to the various other web servers that serve the content itself. Common algorithms include round robin, least connections, and IP hash.
- Round robin — distributes traffic to groups of servers sequentially
- Least connections — distributes traffic to servers based on which have the fewest connections at the time
- IP hash — IP address is translated to a hash to determine which server handles the request (pseudo-random)
The load balancer is a single-point-of-failure in this case and thus, it might be beneficial to configure it with an active-passive copy of itself so that if the server in charge of the load balancer fails, another server containing this load balancer will be started up to continue distributing traffic while the primary load balancer server reboots or is fixed.
After the load balancer transmits the request to one of the servers to handle it, the message is received by the web server (like NGINX or Apache) and processed. The web server is in charge of serving static web content of the codebase, like the HTML and CSS pages themselves. If there is dynamic content that is requested such as PHP or Python code that specifically requests certain attributes or data, this goes to an application server which connects to a database (like a relational MySQL database) to draw information based on what is requested. The database might benefit from a Master-Slave cluster setup so that it is easier to update the server and pull data from it without conflict since only one server instance can have write privileges and the rest of the instances are synced to this server. This information is sent back to the application server which then sends it to the web server to incorporate into its response.
All this occurs assuming that a firewall (like ufw on Ubuntu) allows incoming and outgoing traffic through port 80 of the server side of the TCP connection. Firewalls are computer programs or hardware that can be configured to block incoming and outgoing connections from a network. The software can be installed on any server in the process. In this case, if a firewall is installed on the server carrying the load balancer or the servers containing the pages that were requested, if it is configured to allow incoming and outgoing traffic on port 80, the HTTP request and its subsequent interactions with the web server occur unhindered. If, however, the firewall is configured to block incoming and outgoing connections on port 80, the TCP connection step of this process would have failed and there would have been no connection between the browser and the server when the URL was typed. This is also true if only incoming traffic is blocked. If only outgoing traffic is blocked, the server attempts to send back the HTTP response after processing, but will be unsuccessful in doing so.
Step 6 — The Server Sends an HTTP Response
The server sends back an HTTP response after it interprets and handles the browser’s request. This response contains the status code followed by a series of optional headers that define and contain information about the content that is returned. After the status code and the headers, the actual response body/content that was requested is contained. If an HTML page was requested, the body will contain this content. The www.holbertonschool.com HTML content will be included in the response body like in the image blow.
Step 7 — The Browser Displays the HTML Content
The browser is a program that is built to be able to interpret HTML content. Thus, it will receive the HTML content in the HTTP response body and display it to the user. Additionally, if there are additional plugins or styling referred to in the HTML code, the browser will send several other HTTP GET requests to the website host server for the files containing these styling codes and plugins. This is how the browser pulls the CSS and Javascript content that might be associated with the website function as well as the images, videos, and application plugins that might be called for in the HTML code. After all content is transferred, a complete web page will be displayed in the web browser!
The world wide web is a very intriguing and interesting thing and learning about what actually happens underneath all of the supposed simplicity is a fun journey. This blog post only scratched the surface, but hopefully it helps in communicating that such a seemingly simple process is actually beautifully complex and intricate. Thanks for reading!
