ITRI TODAY Fall issue 2024

ITRI Cools AI Servers with Kilowatt-Level Cooling Solution

At the 2024 World Government Summit, Nvidia CEO Jensen Huang quipped that if AI chips don’t keep evolving, humanity’s thirst for computational power might deplete several galaxies’ worth of energy. In this hypothetical scenario, at least one sun could be consumed entirely just to cool AI servers. According to the International Energy Agency, by 2026, AI data centers will demand 1,000 terawatts (TW) of power, with the cooling alone consuming as much electricity as Germany uses in a year.

The enormous computational demands for training generative AI models have led to data center chips ballooning in scale, power consumption, and heat output, pushing air cooling technology to its limit. Furthermore, heat dissipation technology for AI chips must keep pace not just because chips are simply getting hotter, but also because 3D packaging technology is driving thermal density through the roof.

To tackle these challenges, ITRI has advanced the vapor chamber lid (VC lid) technology, using water cooling to reduce temperatures at the source. The VC lid is a highly efficient heat-spreading component—a square copper plate that covers AI chips like a fever cooling patch.

Dr. Shih-Chieh Chang, General Director of the Electronic and Optoelectronic System Research Laboratories at ITRI, said, “Compared to air cooling, water cooling technology like the VC lid can directly contact the chip surface and form a small vacuum space, removing heat much faster.” In this vacuum, he explained, pure water vaporizes at just 50 to 60 degrees Celsius. The steam then condenses back into liquid water upon touching the cool copper plate above. The nanostructures on the VC lid promote capillary action, effectively spreading water across its surface and removing heat through rapid evaporation and condensation within the chip.

With Nvidia’s H100 and H200 chips generating heat at the 700-watt level and the latest B200 chip exceeding 1,000 watts, however, an AI server would need more than a single copper plate to keep the temperature in check—entire AI systems must now take a dip in cooling liquid.

To keep up with AI’s relentless pace, ITRI leveraged its successful VC lid technology to create a two-phase immersion cooling system, boosting AI chip cooling capacity beyond 1,000 watts. While conventional air cooling technology’s power usage efficiency (PUE) is 1.6, the two-phase immersion cooling system achieves a PUE of just 1.05, reducing power consumption to a mere 30% of air cooling’s demands.

To further turbocharge cooling solutions, ITRI has partnered with Intel to establish a joint lab for high-performance computing cooling certification. This collaboration focuses on testing cooling liquids, thermal properties, surface tension, and the long-term durability of components, accelerating the development of advanced cooling solutions for domestic data centers and aligning with international standards.

Taiwan leads the world in advanced AI chip manufacturing and packaging technology, and its server cooling technology must remain at the forefront. To combat the ever-escalating thermal density, ITRI will continue to advance AI cooling technology, driving the industry toward higher efficiency and lower energy consumption, and moving closer to achieving global net-zero emission goals.