Good but slow: The story of immersion cooling

Despite its superior efficiency, why didn't immersion cooling take off for AI workloads? Well, it's too much work and too slow to market.

Given the surge of AI data centres being built, one would imagine immersion cooling taking off. Instead, the latest AI servers all use direct-to-chip cooling.

In terms of efficiency in heat removal, we've long assumed that: Immersion cooling > Direct-to-chip > Air-cooling.

So, why do data centre operators favour direct-to-chip?

The real problem

The problem can be summed up as: Scalability and speed-to-market. I'll explain in a moment.

In immersion cooling, servers are submerged entirely in a thermally conductive but electrically insulating liquid for cooling.

There are two main approaches:

• Single phase: Hardware is submerged in a dielectric liquid. As it absorbs heat from the equipment, the heated liquid is circulated for cooling.
• Two phase: Coolant with low boiling point vaporises when heated, removing heat, and then condenses back to liquid as it cools.

Just how efficient is immersion cooling?

From my research, blockchain technology firm BitFury achieved a self-reported PUE of 1.02 with two-phase cooling in its Georgia data centre.

Using a unique single-pump design and single-phase cooling, SMC saved close to 50% of total energy compared to air-cooled infrastructure for H100 GPUs in its S'pore data centre*.

*Validated by MLCommons

Good but slow

The problem is that immersion cooling is too slow for the AI race, where millions of GPUs are being deployed annually - and the number continues to rise.

Immersion cooling is a lot more work than direct-to-chip cooling; you can't just dunk the servers into the liquid, dielectric fluid notwithstanding.

Some steps required:

• Remove all server fans.
• Build immersion enclosures.
• Switch to metal-based thermal paste.
• Update firmware - keep running when fan speed = 0.
• Strip out plastic parts - or they'll melt in oil-based fluid.
• Switch to immersion-ready fibre optic components.

Let's just say it's a lot of work.

In fact, Meta wrote a well-regarded technical paper on immersion cooling and the work to be done.

Towards greener AI

Is immersion cooling destined to remain a niche solution in the data centre?

I think there is room for it beyond state-of-the-art AI training, in situations where speed to market is less of a priority.

• HPC workloads.
• Tier 2 AI training.
• AI inference.

Ultimately, it does give us more energy-efficient deployments for greener data centres.

What do you think?