Why need HPC office immersion cooled?
Ensuring IT load online /MEP /Facility /Hardware /Software
In the realm of digital infrastructure, hyperscale data centers are the backbone of networks, supporting the immense data needs of technology giants. Known for their efficiency and scalability, the rise of colocation services reflects the market’s shift towards flexibility and cost-effectiveness. To enhance energy efficiency, the industry is increasingly adopting immersion cooling, direct liquid cooling (DLC), and monitor-driven systems. The application of digital twin technology optimizes energy usage, cooling performance, and overall efficiency, particularly in managing the dynamic loads of air-cooled racks, which are more complex to control than water-cooled systems.
With computing power increasing at an extraordinary rate, data centers must handle higher power densities. Upgrading transformers and switchgear to accommodate increased loads is crucial. Hot aisle/cold aisle containment improves cooling efficiency, and high-density power distribution systems, such as busbars and overhead distribution, provide more power to server racks in smaller spaces. The emergence of higher power density servers, the consolidation of more processing power into smaller footprints, and the adoption of 48V direct current (DC) power distribution systems enhance efficiency and reduce current losses. Reducing water consumption is also key, with DLC or immersion cooling reducing reliance on chilled-water-based air conditioning.
In the dynamic field of digital infrastructure, high-bandwidth cabling infrastructure is essential for supporting high-speed network protocols like 100GbE and 400GbE. Effective network segmentation enhances security and optimizes traffic flow, while high-speed network technology and software-defined networking (SDN) boost the efficiency of data center network traffic. Trends such as server virtualization enable the independent decoupling and scaling of computing, storage, and network resources. Solid-state drives (SSDs) reduce data access times, while hybrid storage solutions combine the high performance of SSDs with the cost-effectiveness of HDDs.
Comprehensive systems for monitoring power, cooling, and environmental conditions are now integrated with automation capabilities. These systems optimize airflow and enable predictive maintenance, particularly in high-volume deployments. Sustainability and safety are paramount in data center design. Selecting energy-efficient equipment and renewable energy sources is crucial. Ensuring that cooling technology is paired with appropriate fire suppression systems and safety protocols is vital for the long-term operation of the data center.
As technology advances and computing demands increase, data center mechanical design continues to evolve. To support high-density servers and storage systems, floor load requirements have increased to over 1,000 pounds per square foot. This is due to the heavier new-generation servers and denser rack configurations, especially with water-cooled racks that include more pipes and metal components, with some cabinets weighing up to 1.5 tons. To address these challenges, designers use BIM 3D modeling to ensure even load distribution. This involves using steel frames, iron plates, and cement reinforcements to maximize floor load capacity.
Location selection for data centers also considers climate factors to improve efficiency and reduce environmental impact. Cool-climate locations can reduce reliance on mechanical cooling, while water-scarce areas may adopt water-saving cooling technologies. Space optimization is crucial. Horizontal optimization saves space for air pretreatment equipment by eliminating unnecessary air cooling and reducing land costs. Vertical optimization, such as higher ceilings and hot aisle/cold aisle containment systems, along with modular cooling systems and improved airflow management, effectively dissipates heat in high-density environments. In office buildings and basements, water piping transfers energy across passageways.
Although the fire extinguishing system is fixed in the building, there are no changes from the new technology deployed. Gas fire extinguishing systems cool temperatures and reduce oxygen concentration quickly. With water cooling, whether DLC or immersion, fire risks can be controlled below the ignition point through effective heat exchange and comprehensive disaster verification. Despite challenges with oil and water resources, lessons from the automotive industry, such as low accident fire risks and reliable oil lubrication and insulation, are applicable. With rising environmental awareness of ESG and GWP, gas fire extinguishers are shifting from Halon/FM200 to lower GWP materials like NOVEC1230, even rarely used.
There are debates on various cooling liquids with different GWPs available. As reflecting the fire system’s evolution, the cooling architecture decision is not merely thermal dissipation but involves multi-faceted trade-offs.
As technology advances, data centers must handle higher power densities, requiring efficient power distribution systems. This involves using high-current busbars or overhead distribution systems and upgrading transformers and switchgear to accommodate increased loads. Implementing N+1 redundancy is critical to ensure the reliability of power supplies and cooling systems, emphasizing the need for efficient power usage to minimize redundant power backup.
The application of busbars facilitates indoor power transmission, saving manpower for cable maintenance and simplifying complex routing. Modular transformers provide greater power distribution flexibility and scalability, while low-voltage transformers and switchboards are continually improved to handle higher power loads, such as the popular indoor 23kV armored switch boxes. Deploying high-efficiency transformers with amorphous metal cores minimizes energy losses, achieving 98% to 99% efficiency. High-density power distribution systems, like indoor busbars and overhead distribution, provide more power to server racks in limited spaces, handling current ratings of 2000 to 4000 amps per phase. High-density PDUs offer more power sockets (40–60 kVA) in smaller footprints, supporting high-density server configurations.
The emerging trend of 48V DC power distribution systems improves efficiency compared to traditional 12V systems. The adoption of smart circuit breakers, large-capacity busbar systems, and remote monitoring and management functions of PDUs aim to achieve refined power consumption control and optimization. Advanced monitoring systems, including PUE meters, help track power consumption in real-time, identifying inefficiencies and optimization opportunities. Low-smoke, halogen-free cables are favored for their flame-retardant properties and reduced smoke emissions in fires. Fiber-optic cabling supports high-bandwidth data transmission within data center networks. IEC sockets, known for their higher current ratings and secure connections, are increasingly used in data centers. Standard NEMA plugs and receptacles (e.g., NEMA L5–30) are common in North America, while the popularity of 3-phase IEC 60A devices has led to updated power distribution needs.
Cooling system design requires flexibility to accommodate various technologies, such as direct liquid cooling (DLC) or immersion cooling. Reducing water consumption is crucial, and deploying DLC or immersion cooling can lessen reliance on water-based air conditioning. In suitable climates, water-saving technologies like evaporative cooling are worth exploring. Free cooling using outside air may require dedicated air intake and filtration systems.
As IT equipment demand grows, power and cooling infrastructure must scale accordingly. Hot aisle/cold aisle containment can improve cooling efficiency. Datacenter piping design has evolved to focus on water conservation and efficiency, improving power usage effectiveness (PUE). Choosing corrosion-resistant materials for pipelines is essential. For example, CPVC is economical and easy to construct but less reliable. PPR requires special tools for heating thermoplastic piping, balancing reliability and cost. Stainless steel is durable for high-pressure or high-temperature applications, especially in earthquake-prone areas where it maintains high ductility.
Pump System Design Variable speed drives (VSD) are widely used in pumps to adjust their speed according to cooling needs, resulting in significant energy savings of 30–50%. However, they require corresponding engineering design experience. Redundant pump systems with N+1 redundancy and ring loops are often used in critical cooling circuits to ensure operational continuity in the event of a failure. Variable flow systems use valves and controls to adjust water flow based on cooling needs. Pressure-independent linear flow control valves and electric proportional valves save energy in air conditioners and balance loads between cabinets in high-density computing environments. Temperature sensors monitor water temperature at various points to adjust flow rates accordingly. Differential pressure sensors provide fast response and are cost-effective indicators in flow systems. Using pressure impedance of equipment to obtain flow curves can save the high cost of flow meters. Thermometers provide energy monitoring, though they respond slowly. Temperature is a common heat exchange language for different liquids and flow rates. For example, antifreeze is required outdoors while high-performance pure water can be used indoors both running at different flow rates and temperature rises for the same energy transmission.
Implementing a leak detection system identifies and resolves leaks promptly, minimizing water loss and machine room risks. Flanges or welds are preferred for joining data center piping due to their reliability and leak-proof properties. Mechanical connectors and high-pressure hoses, commonly used at the end of units, require careful selection and maintenance to ensure long-term performance. Water Usage Efficiency (WUE) reducing water consumption is crucial. Closed-loop systems stabilize optimal temperature control and dynamically deliver flow control cooling tower water usage and pumping energy consumption, significantly contributing to reducing WUE. DLC and immersion cooling using closed-loop systems can reduce water consumption by 70–80% compared to traditional air conditioning, as the cold water cycle eliminates the need for a chiller and wet bulb temperature sensitive.
Data hall operations are undergoing significant transformation to enhance user experience, efficiency, and reliability. Self-service portals now offer remote monitoring, power usage tracking, and streamlined service requests, empowering users with control over IT equipment. Enhanced reporting mechanisms provide detailed performance analyses for data-driven decisions. Hot-swapping technology allows seamless component replacement without downtime, maintaining operational continuity. Standardizing server models simplifies maintenance and maximizes uptime, while automation of tasks like server provisioning and configuration frees up human resources for strategic initiatives. Remote monitoring tools enable early issue identification and swift resolution, improving Mean Time Between Failures (MTBF). Redundancy in power supplies, cooling units, and network connections ensures high availability. Efficient cooling practices, such as hot aisle/cold aisle containment, reduce energy consumption and prevent overheating. High-density racks and virtualization optimize space and resource utilization. N+1 redundancy in cooling systems and continuous monitoring of temperature and humidity levels enhance reliability, while a robust disaster recovery plan minimizes downtime during failures.
Data centers and IT rooms are facing greater power demands and challenges. Although resource infrastructure such as water, electricity, and air conditioning can be upgraded, in-situ upgrades are also a possibility due to limitations in network light and geographical location, such as It is to expand the computer room resources of general buildings. Deploying high-performance computing (HPC) systems in general office buildings does face many challenges, which involve location selection, mechanical, electrical, piping, operation and maintenance, and other aspects. First, office buildings may lack dedicated space or reinforced floors to support the weight and heat of the HPC system, so a structurally proper location within the building to house the HPC racks needs to consider floor load requirements. Typically be achieved by full-area cement reinforcement due to idle floor load in operation working areas. Secondly, traditional HVAC systems may not be able to handle the high heat output of HPC systems, which necessitates the exploration of cabinet precision air conditioning and immersion equipment to maintain existing maintenance patterns while increasing IT loads, and extra maintenance complexity from DLC parts. In addition, the existing power capacity of the office building may not be enough to meet the high power requirements of the HPC system. It becomes necessary to squeeze every little efficiency and explore high-density power distribution systems. High-density power busbars and power shelves can maximize density while providing excess performance, and even less energy loss in cooled environments. In terms of piping, traditional office buildings may lack dedicated piping infrastructure. The feasibility of installing additional piping for closed-loop systems within the building, such as hot tapping works, circulation loop piping, and backup cooling towers needs to be implemented to ensure loads are online. Finally, it is important to conduct a comprehensive analysis to evaluate the feasibility of deploying HPC versus other solutions in office buildings from a cost-benefit perspective, especially in denser and rent-saving cases. Additionally, soundproofing measures need to be considered to reduce disruption to other office occupants and ensure compliance with relevant regulations and safety standards. These challenges require comprehensive consideration and careful planning to ensure smooth deployment and effective operation of HPC systems.
For traditional air conditioning systems, their capacity limitations and low efficiency are major pain points as they may not be able to handle the high heat output of HPC systems. Strategies to address this include upgrading existing HVAC systems to handle increased cooling loads, which may require adding additional cooling units or increasing chiller capacity. Additionally, implementing targeted cooling solutions can provide centralized cooling directly to the HPC racks, thereby offloading the central HVAC system. For DLC, key challenges include infrastructure transformation and lack of expertise. Office buildings may require additional ducting to support closed-loop DLC systems, and building personnel may not have the expertise required to operate and maintain these systems. A comprehensive feasibility study is required to determine the possibility of installing additional ducting and work with the building management staff. Consult with experts and consider deploying modular DLC units with fewer plumbing modifications, and work with a managed services provider that specializes in DLC systems for installation, maintenance, and ongoing support in complex water-cooled system maintenance despite additional Operating costs will be incurred. Challenges with immersion cooling systems include higher upfront costs and safety concerns when handling dielectric fluids. Conducting a cost-benefit analysis can help evaluate the long-term cost-effectiveness of immersion cooling compared to other options, taking into account energy savings and potential efficiency gains. Provide employees with appropriate safety training for handling dielectric fluids and specific immersion cooling system safety procedures, and consider starting with a pilot deployment of a smaller immersion cooling system to evaluate its effectiveness and feasibility in an office environment. In addition to these specific cooling technology challenges, there are also general issues that need to be addressed, such as space constraints and noise control. Office buildings may not have dedicated space to place additional cooling equipment, so options to utilize existing unused space within the building need to be explored. Although DLC significantly reduces air conditioning requirements, there is still a need for air movement and the associated energy consumption. In addition, HPC systems and their cooling units can generate significant noise, so soundproofing measures need to be implemented within designated HPC spaces to reduce disruption to other office occupants, a benefit that comes with immersion foam cooling. By being immersed in liquid, the operating noise is significantly reduced by half, achieving the quality of an office environment, and even reducing the management costs of temperature and humidity precision air conditioning.