Grok 5 features
Grok 5, xAI's new flagship model, features a 6-trillion parameter architecture. Only a subset of the most relevant parameters activate per query, allowing such a large model to run efficiently and generate answers quickly.
Combined with live X data, a native multi-agent layer, and a context window of 1.5 million tokens, which lets the model hold long documents, extended conversations, and full video timelines in memory at once, Grok 5 is positioned to become the most advanced text-generation model by xAI.
Grok 5 architecture and training
Architecture. Grok 5 is built on a Mixture-of-Experts (MoE) architecture, a design that enables massive scale without proportional increases in computation cost. In an MoE system, the model consists of many specialized neural subnetworks, called experts, but only a small subset is activated for each query.
The design philosophy behind Grok 5 follows scaling laws in machine learning, where increasing compute, data, and parameters tends to improve model capability. Large-scale training runs like Grok 5’s are important because they:
1. Improve the model's ability to succesfully complete tasks it hasn't encountered during training
2. Gain new abilities, particularly more advanced planing
3. Improve the models ability to retain information in context-intensive tasks
Training. Grok 5 is trained on Colossus 2, xAI’s next-generation AI supercomputing cluster located in Memphis, Tennessee.
- The initial Colossus system was built in just 122 days
- But it was scaled to ~850k GPUs making it the biggest training cluster in existance
- It is designed to reach ~1.5 gigawatts of power capacity
The infrastructure enables xAI to train extremely large models with multi-trillion parameter count on its mission to become the first company to achieve AGI.
Grok 5 vs Grok 4
Architecture. Grok 4 is a dense transformer model, while Grok 5 is a Mixture-of-Experts (MoE) model. Grok 5 architecture reduces compute cost per request while improving the model's reasoning quality.
Context window. Grok 4 supports a 256,000-token context window, while Grok 5 expands this to approximately ~1.5 million tokens. This allows Grok 5 to handle much larger inputs.
Training. Grok 4 was trained on xAI’s Colossus supercomputer with up to 200,000 NVIDIA H100 GPUs. Grok 5, however, is trained on Colossus 2, which is a much larger system. Although xAI hasn't revealed the exact number, it is estimated to be in the range of 500,000–850,000 GPUs. The training data now includes image, audio, and video datasets, enabling training of a multimodal LLM that can understand different types of files.
Features. Grok 5 builds on the features introduced in Grok 4. These include always-on reasoning, native tool use, and multi-agent reasoning. In Grok 5, reasoning will be adaptive and not controlled by the user via a UI toggle. The model will also improve its ability to understand images, audio, and video input while improving its ability to use multiple sub-agents to complete difficult tasks.
.svg){kind=icon}
