Data Center Networking : Part I
Understanding CLOS topologies
Something I wanted to do for a long time is to write about Data Center networking and how it evolved in the modern age. Again will target 4-5 mins read time and hopefully I will be able to do some justice to the question at hand.
What are Data Centers and why are they special
As applications evolved in the early 2000s there arose an idea of putting clusters of PCs together and harness the compute and storage powers of such clusters for applications like scientific computing, financial analysis etc. With time such cluster became increasingly large and hundreds of thousands of PCs per location became a common thing.
Having a single location where all the compute, storage and networking reside provides a lot of benefits. The main benefit is security policies could be applied and following established security standards become easy. Redundancy can be improved and a device or equipment failure scenarios mitigated. Also the temperature of the data center can be maintained such that the life of the devices could be extended to their maximum.
The question then was how to efficiently connect these PCs and also provide optimized network performance, resiliency and performance.
Modern Data Centers
To better manage the complexity of handling the Data center operations a lot of innovations have come about over the last decade or so. But let us just focus on a smaller topic of network connections since focusing on all other data center operations will require writing an entire book.
The data center networking architecture goes back to 1930s to a man named Charles Clos. Clos came up with a topology or a way of arranging the telephone nodes of the 30’s in a way to optimize their placements in a way today called as a “leaf-n-spine” topology. This leaf-n-spine topology forms the basis of the data center today. It not only increases resiliency but also provides scalability and performance.
The above figure shows a leaf-n-spine architecture example [1]. The servers in the data center (grey boxes at the bottom) are connected to the Leaf Switches. The servers are usually arranged in racks and each server then connects to a switch that usually sits on top of the rack so that is what it is called : “top of the rack switch or TOR”. This switch is further connected to a spine switch which is usually a high performance switch. Usually there are 3 layers, but the figure shows only two. The three layers are Access (TOR switches), Distribution and core layer. In large data centers there could even be 5 layer hierarchy.
The most important thing to note about this architecture is that any server that needs to talk to any other server needs just TWO hops to reach the other server. There are redundant paths (2 or more) to reach from one switch to the other switch in the data center as seen from the figure above. This multiple paths along with deterministic 2 hops architecture forms the basis of the data center architecture. This is all we need to know to continue the discussion.
Let us just revisit the CLOS architecture and its benefits again briefly:
- Having equidistant endpoints (2 hop) means the latency is almost similar across all communications. This helps in the performance and scaling the network.
- The existence of multiple paths between the servers leads to the the formation of Equal cost multipath (ECMP). This forms a basis of routing strategy and allows for load balancing in data centers.
- Multiple links can be combined to increase bandwidth and resiliency. This is know as port-channels or aggregated ethernet interfaces.
- As you can visualize in the picture above, the server is connected to say Leaf1 with one link and Leaf2 with another link. If we use both the links together and combine them this forms a vPC port channel. So if the leaf switch goes down the communication can still continue using the other leaf switch.
Next part
In the next part we will go into depth of network traffic in data centers and cover:
- TCP and its impact
- Multipaths and TCP
- TCP : flowlets and flowers
- Future open ended questions.
