Why I ran out of memory, and why it matters

When I first set up a local AI inference device, I thought its onboard 128GB of RAM and 1TB of storage were a lot. They were, when you consider that my work machine has “only” 64GB of RAM, and that my 2TB NVMe drive was mostly empty. The box, I reckoned, had plenty of both.

That changed after I downloaded my first dozen AI models and started using the Linux machine for AI coding. The storage filled up quickly. Then I hit my first out-of-memory (OOM) error, which prevented me from  SSH’ing in. There was simply too much going on at once – and I had to optimise my usage to prevent a recurrence.

That experience kept coming to mind as I recorded the podcast series Why Memory and Storage Matter with Micron Technology to explore the impact of AI. Because AI is no longer confined to the four walls of a data centre. It is showing up in factories, cars and the devices we use every day. And that means memory and storage matter there, too.

Memory is becoming more strategic and specialised

As with every conversation with experts and thought leaders, there was much I learned. In the first episode, I sat down with Koen De Backer, Corporate Vice President of smart manufacturing and artificial intelligence at Micron Technology, and tech content creator Daren Yong. The topic? How AI is “getting real” in our world.

It was Daren who highlighted the different types of memory. The memory used in cars, he observed, is different from that used in factories or high-performance computers. I’ve always known that not all memory is the same, but only in the context of PC memory modules and performance. Yet it makes perfect sense. I hadn’t fully considered how factors such as reliability, operating conditions and workload shape the memory used in different environments.

And we can expect more specialty memory to come. As Koen noted, memory used to be a commodity. What we have seen in the last couple of years is the emergence of more specialised forms of high-speed memory. And with GPUs and other specialised AI accelerators springing up, one can expect an even greater variety to emerge.

So what does the future of memory look like? According to Koen: “Depending on your end application, and which AI workload you’re talking about, you see more and more specialty memory coming up.”

The bottleneck is the opportunity

So how is AI being used today? Koen pointed to manufacturing, where AI is already making a huge impact. Consider the wafer fab, which he describes as one of the most advanced manufacturing plants in the world. A single wafer takes more than three months and more than 1,500 production steps to make. And a single fab generates about 78TB of compressed data a day from its equipment alone.

No engineer can realistically comb through all that data manually. That’s where AI comes in. But as Koen put it, that cuts both ways: “Generative AI is really fuelled by the models, and the capabilities of the models are linked to the size of the data it is trained on, and that size of data is humongous.”

Daren made the same point but from the other direction. “It’s not just processing the AI that’s enough. You need to store it, and you need to pass information quickly. So that’s the bottleneck. That’s the opportunity.”

Where will this lead? Daren sees AI potentially powering a path towards a world of abundance. That may be some way off. But closer to home, Koen, notes that the tools have evolved to the point where he can develop things himself.

Having built about half a dozen apps since I picked up AI coding, I completely agree.

Watch the podcast here.