7 steps for organizing your server rack space
In this article we talk about proper placement of equipment in a rack, in other words, we take a systematic look at the operation of a server rack: from drawing up a plan and installation to wiring labeling. The entire narrative is based primarily on my experience as a data center engineer, and, for good or bad, I try to follow these very rules and practices.
Introduction. Why you need order in the rack
A server rack with a well-organized cable management and thoughtful arrangement of equipment is not only for aesthetic pleasure. Of course, a properly arranged rack looks spectacular, but first of all, purely utility tasks are solved:
- Improved cooling
- Extend the life of your equipment
- Shorter time to locate failure points
- Fewer setup and replacement times
- Decrease in the number of accidents, including those caused by personnel
As a consequence of the above, quality rack management reduces costs, increases PUE, reduces downtime… And there is no shame in showing potential customers.
At the same time you should understand that rack management is a complicated process that demands a certain approach. Each action needs to be thought through before being put into practice. In the future, if it turns out to be ineffective this or that solution, working equipment may not allow to remake in another way. And of course, you need to keep order all the time — a mess arises very quickly.
So, where do you start with managing your rack?
Step 1: Make a plan
Let’s assume that you are starting from a relative zero — you already have space in the data center and you have been allocated empty racks (or space for them).Power and cooling is provided by the data center. There is also an understanding of what equipment and how much will be installed in the racks.
As mentioned above, you should place the equipment thoughtfully, first of all, because the IT infrastructure in the rack is supposed to work non-stop for a long time, and later you may not be able to make changes in the installation without affecting the performance. And that means down time and dissatisfied customers. Therefore, before you start installing racks and equipment in them, make an approximate plan of how your hardware will be placed in the rack. You can do it with the help of Microsoft Visio, Lucidchart or even in Excel spreadsheet. You should also pay attention to draw.io.
The location of the equipment depends primarily on the rack dimensions. Server racks have certain standards: the standard width (19 inches), the height is measured in units (U).
1 unit is 1.75 inches, and the taller the rack, the more units it has. A 48-unit rack can hold 48 units of rack form factor equipment if each unit is one unit (1U) high. There is also equipment in 2U, 3U, 4U… Probably more, but I haven’t come across any. The depth of the rack can vary — usually from 600 to 1100 mm. Usually the frames on which the equipment is attached can be moved, thereby slightly reducing or increasing the mounting depth, but you should not rely on it entirely, so find out in advance if your equipment can fit into the rack in depth.
So, prepare a layout of your equipment, taking into account its size and number, as well as the dimensions of the rack. Don’t chase the maximum density of equipment based on the thought “I paid for 48 units, so I’ll use all 48 units!” First, it has been my experience that the rack will most likely run out of power limit first, not space, and second, chasing packing will reduce the quality of your cable management and cooling.
You should also take into account where the power cables to the PDU (Power Distribution Unit) come from (top or bottom), you should also think about PDU placement, type and number — usually two vertical PDUs are placed on the left and right side of the rack, as shown in the picture below:
Schleifenbauer, Wikipedia.org
Don’t forget about your power redundancy requirements. Inside the data center I work with, a 2N scheme is built: 2 complete parallel power supply systems, in which each element is duplicated. In the case of a rack, this means that it has two independent inputs to two PDUs, and the equipment is connected in such a way that a complete shutdown of one does not affect the operation of the infrastructure in any way.
A few additional recommendations:
- Plan to place equipment with open optical connectors on top of the rack, this will reduce dust contamination.
- Do not aim for the densest possible placement of equipment per unit. Leave units for unscheduled future scaling and for horizontal organizers.
- Use wide racks, with room for vertical socket blocks and extra room for vertical wiring.
- The plan should include not only networking and computing equipment, but also auxiliary equipment if it takes up rack space: shelves, UPS (Uninterruptible Power Supply), organizers, cameras, and empty units.
- Strive for total detail circuit in this case is not necessary — draw the laying of power cables and patch cords are not necessary, just enough to describe the general principles of cable management in the explanatory note. Vertical PDU, by the way, draw too, not everyone.
- It is better to place heavy equipment at the bottom of the rack cabinet.
- Speaking of gravity. Racks have a concept of carrying capacity — how many much weight a particular rack can support. It is worthwhile to estimate the weight of your equipment and compare it to the carrying capacity of your server racks.
You should end up with a diagram of the rack similar to the one below, with an explanatory note with the general principles of cable management, for example: power cables are laid on the left side of the rack, patch cords on the right, red patches are plugged into the first network card, blue ones into the second, wires are only secured in a certain way… And so on.
Also in the explanatory note it is obligatory to specify where which device will be connected by network and power.
The explanatory note plays a more important role than it might seem at first glance. A server rack can be assembled well once, but without a description of the installation rules, every new action in the rack will cause confusion and clutter.
Step 2: Make a list of supplies
To install the equipment and connect it, you will need a certain assortment of materials. Make this list in advance, and take care of getting all the supplies before you begin the installation. Here’s a list to get you started:
- Hardware fasteners. What attaches your hardware to the rack rails. Most models are screwed through a screw-nut-washer fastener (pictured below), but some equipment models are mounted only with their own fasteners (and there are rails that require no fasteners at all) — check with your supplier.
- Cable Wire Box. For patch cords of non-standard length.
- Prefabricated cables in different lengths: Ethernet patch cords and others, such as optical cables. Using prefabricated cables instead of twisted pair will save you a lot of time.
- RJ-45 connectors for twisted pair crimping.
- A tool for crimping twisted pairs and a tester to test the resulting cables.
- Power cables of different lengths, with a suitable socket connector and the proper cross-section. Remember — the higher the power of the server power supply unit, the larger the cross-section of the power cable must be! The socket connectors are selected according to the connectors in the equipment and in the PDU. Below are photos and names of the most common connectors:
- Labeling accessories. Experience has shown that the plastic clips for marking wires (pictured below) are not very convenient, so I recommend using only paper.
I use a Brady printer with vinyl labels for labeling cables and a Brother label printer for everything else.
- Accessories for laying wires: horizontal and vertical organizers, ties, patch panels and conduits.
- Tools for work: screwdrivers, screwdrivers, wire cutters, scissors, and so on.
- Working place to store all the above: room, shelving, workbench, anti-static mat, etc.
A few additional recommendations:
- It makes little sense to save money on cabling — compared to other equipment their cost is low and their service life, on the contrary, is long. You can’t save much, but it’s easier to get into trouble because of cheap, low-quality cable.
- Instead of plastic strap clamps, consider buying reusable Velcro, they are much more convenient.
- The rails and mounts on which the device is placed in the rack can fail, get lost, or come faulty. It might make sense to buy a few spare kits. You need a reserve in order not to waste time — yes, you will change the defect under warranty, but the replacement will take some time, it is important not to allow downtime.
Step 3: Installing the rack
After making a plan and purchasing all supplies, you can wait for the racks to arrive. There is nothing complicated about installation, just follow the instructions, work with at least three people, open only those plugs and covers that you need for the installation of cables in the rack. The fewer extraneous openings — the more effectively the airflow will cool the equipment. By the way, it is better to use racks with unit numbers signed with paint — as on the photo below. Stickers tear off or fade with time, and the numbers stamped on the metal are hard to see.
Here we can also talk about the installation and wiring of the PDU, as well as the organization of the cable conduits.
It is most convenient to install the PDU in a rack in the first place. In the rack, the server socket blocks are either placed on the sides on standard mounts (vertical PDUs), or simply screwed into the rack as normal equipment (horizontal PDUs).
The conduits should not be placed too high to avoid heating — all the warm air will be under the ceiling. 20–30 centimeters (~10 inches) from the roof of the racks will be optimal.
Additional recommendations:
- Do not forget to ground the racks.
- Immediately after installing the PDUs, you can install organizers to vertically organize the power cables and network patchcords in the server rack; they will not interfere with the installation of the equipment.
- If necessary, adjust the rack mounting depth by sliding the rails.
Step 4: Installing the hardware
Perhaps the main part of our ballet. I propose to use a certain order of hardware installation:
PDUs and vertical organizers are installed first, followed by networking equipment, UPSs (if necessary), and then the servers.
Servers are installed from the first unit to the last, i.e. from bottom to top.
It is necessary to fasten the equipment completely without missing any holes in the rails.
Do not skew rails and equipment, make sure the units on the front and rear mount frames match. Unit 14 in the front is 14 in the back, etc.
Servers should slide into the rack easily and smoothly. About halfway down the rail, most server models will stand on the safety stop. Slide the safety latch and slide the server all the way into the rack.
At the end it is necessary to fix the server still in the rack.
Be sure to pay attention to the airflow when installing the equipment — it goes from the cold aisle to the hot aisle through your equipment. In fact, your equipment makes hot air out of cold air (unexpected, right?).
So, inside most equipment, there are coolers inside the rack that blow air inside the device enclosure and cool it down. The air flow they create also has a certain direction. It is important that the equipment coolers blow air correctly — from the cold aisle to the hot aisle.
Additional recommendations:
- After installation, all equipment should be secured in place.
- After installation, cover any units where no equipment has been installed with plugs on the cold aisle side.
- Equipment in a rack without regular mounts, such as a server without rails just placed on top of another, is not a good story for workloads.
Step 5: Wiring and labeling
Once the equipment is installed in the rack, it needs to be wired and labeled. In some cases it is better to combine the wiring and labeling process with the installation, i.e. the engineer acts in this sequence:
a) installs the server,
b) connects it for power and networking,
c) marks the server and cables,
d) proceeds to install the next server.
The connection process does not cause any special difficulties — just plug it in until it clicks and check that the pings went and the load of electricity is normal. Always check the tightness of the plug to the socket! A loose contact provokes heat, which in turn increases the likelihood of fire.
There are some nuances with the marking. Either way, the following types of markings should be applied:
On the server:
Server number, on both sides, mandatory.
MAC-address, numbers of power supply units, numbers of baskets in disks, serial numbers of disks installed in server slots — optionally.
On other equipment: switches, PDUs, UPSs:
Also numbers for identification.
On cables:
Power cables should be labeled with the powered server (switch, UPSs, etc.), its unit number, the number of the power supply to which the cable comes, and the PDU number and outlet number from which the cable comes. Naturally, stickers must be placed on both ends of the cable — at the socket and at the server.
Network cables should also have stickers indicating the connection server, its unit location, and the number of the network card where the wire comes in. Also present should be the number of the network switch, the unit of its location and the port from which the cable comes.
Experience shows that labeling is the most boring job of the entire rack management process, but you can’t do without it. There are literally tons of wires in any data center, and in most cases you can’t tell which one leads to which one. And in case of client’s request (or even more seriously — in case of emergency) there is no time to look through the wires in search of the right one. To quickly identify wires it is necessary to use markings.
The picture above shows an example of a cable flag layout for labeling a network patch cord. The top side shows what the patch cord is connected to — the server number, its location, and the port number of the NIC. The bottom side shows the data from which the wire leads — the switch, its location, and the port number.
Labeling can be done in several ways: vinyl-based stickers, self-adhesive paper stickers, heat shrink tubing, cable ties with tags and thermal labels. I currently use thermal labels and vinyl stickers, these are the most convenient solutions.
Additional Recommendations:
- Think through the naming system for the labels in advance and document it in an explanatory note when you make your plan to avoid confusion and confusion. It should be clear what is labeled and by what acronym. Abbreviations for designating equipment should be unique, so as not to confuse them with each other.
- Labeling is a mandatory element of rack management. It should be implemented in the process of IT infrastructure deployment from the very beginning. And if it is forgotten at the beginning, and then remembered in the middle, it will take three times as much effort to label the already working system.
Step 6: Maintenance
Once the equipment is installed, connected, and the admins have run all the dialogs and put the equipment into operation, most of the network engineer’s work is done, but there is still some work to be done. In a nutshell, racks and servers need to be periodically cleaned, thermal paste changed, the temperature and humidity checked. Everything else is done as needed, I’m talking about replacing components (RAM, power supplies, etc.). If something has failed and needs intervention — the equipment will give the appropriate alarm, then the replacement will be carried out. There is a concept of scheduled replacement — in this case there is a notification from colleagues from technical support.
In general, when it comes to equipment in the data center, you can follow the general rule: if the equipment works, there are no abnormal situations and it is not time for scheduled maintenance — do not touch the equipment.
In my opinion, it is reasonable to clean racks and servers in it with an industrial vacuum cleaner at least once every six months. The operation is very simple — you go up to the rack and with a soft nozzle gently sweep over each “gland”. Then repeat on the other side of the rack. It is important that at the moment of cleaning, admins should watch the state of servers to signal in case of abnormal changes, in this sense, cleaning is a team sport.
When taking equipment out of the rack, it is a good idea to do a purge and replace the thermal paste — at least once a year. Blowing can be done either with a pneumatic can cleaner or with an electric compressor.
Particular attention should be paid to the power supply unit. There is no need to disassemble it — you can lose the warranty on the equipment, but it is worth blowing it from the outside.
Additional recommendations:
Use quality thermal compound, apply a thin layer to the processors, using the standard spatula tool.
Step 7: Audit
It makes sense to incorporate the practice of periodic checks of machine room order, say, once a month, into the production process. In addition to identifying errors in the installation of equipment and cabling, regular checks of the condition of your racks will, for example, allow you to quickly determine the possibility of further scaling of your infrastructure. The term “audit” in this case should not be understood as ordering expensive expertise from specialized companies (although it is sometimes useful), but simply a specific procedure within the company to collect, analyze and evaluate changes in the hardware infrastructure.
Do not think that a simple inspection of racks can not detect any errors. When installing servers “in bulk” a few dozen at a time, an engineer can make incredibly stupid miscalculations — probably the number of cases in the world when a server with two PSUs is connected to one power beam, already counted in thousands. To see such a mistake, sometimes just a close look is enough.
Conclusion
The above 7 steps are a good start to learning the art of organizing the hardware part of the IT infrastructure. These theses are not a one-size-fits-all solution, but they give a good general idea of rack operation as a process.
Today, more and more companies are moving to the cloud, refusing the policy of building their own data centers or organizing colocation. As a consequence, the number of people whose job is to physically interact with hardware is decreasing. Nevertheless, this article is written primarily for data center engineers and trainees in this field.