> ## Content Index
> Fetch the complete content index at: https://www.techstories.co/llms.txt
> Use this file to discover other available public pages before exploring further.

# 100,000-strong GPU clusters are next
- URL: https://www.techstories.co/100-000-strong-gpu-clusters-are-next/
- Published: 2024-06-21T02:16:00.000Z
- Updated: 2024-06-22T02:19:08.000Z
- Description: LLMs show no sign of plateauing, says OpenAI's CTO Mira Murati.
- Author: Paul Mah
- Tags: AI

LLMs show no sign of plateauing, says OpenAI's CTO Mira Murati - so we can expect ever-larger AI models.

Mira spoke at a fireside chat at Qualtrics X4 which I attended in May, and recent developments show that we are indeed on the trajectory she painted.

Here are some comments she made that caught my interest:

1. **Expect another leap in performance**

Even the researchers at OpenAI were astounded by the new capabilities that GPT-3, then GPT-4 delivered. She expects GPT-5 to offer yet another "step change" in new capabilities.

1. **How you are helping train AI**

As observers have long pointed out, OpenAI isn't offering a free version of ChatGPT for purely altruistic reasons. By Mira's admission, inputs from users are used for reinforcement learning, helping make AI more steerable and useful".

1. **100,000 GPU AI models coming soon**

Mira says there is no sign that the scaling paradigm - which posits that AI models will become more powerful with more compute and data - will taper off soon.

- To add to point 3, we know that LLama 3 was trained on 24,000 GPU clusters and that Meta is currently working to build [GPU clusters](https://www.cdotrends.com/story/4064/how-meta-trains-its-llms-scale?ref=techstories.co) in the "hundreds of thousands".
- Elon Musk has separately told investors he wants to deploy a 100,000 H100 GPU data centre for his xAI startup, which recently closed a US$6 billion fundraising round.

The next generation of AI models will be trained on supercomputers with GPUs numbering in the 100,000s - likely leaving the supercomputers of just a few years ago in the dust.

Where will the AI race end? Or should I ask instead: How will the AI race end?

Read my full article [here](https://www.cdotrends.com/story/4065/openais-mira-murati-llms-show-no-sign-plateauing?ref=techstories.co).