Million-Token Context Window: Why AI Agents Need It

Inside MiniMax M3 and the sparse-attention breakthrough that came from an intern's desk

A million-token context window sounds like a spec sheet number until you watch an AI agent actually try to work without one. Ask it to read a long contract, call three tools, remember what the first tool returned, and still make sense of the whole conversation an hour later — and a short context window collapses under its own forgetfulness. That’s the problem Olive Song, RL Lead at MiniMax, and Thomas Wolf, co-founder and Chief Science Officer of Hugging Face, unpacked in a recent conversation recorded at the AI Engineer World’s Fair in San Francisco.

Their discussion centers on MiniMax M3, an open-source model that pairs a functional million-token context window with coding, agentic and multimodal capabilities in a single system. It’s a useful lens on where large language models are actually headed — not toward bigger benchmarks, but toward models that can hold an entire working session in their head.

Why Context Length Is the New Bottleneck for AI Agents

It’s easy to treat context length as a technical footnote. Song makes the opposite case: for an agent that has to work across long conversations, tool responses and complex environments, a short context window isn’t an inconvenience — it’s a hard ceiling on what the agent can do at all.

Compare that to where the field started. GPT-2 shipped with a context window of just 1,024 tokens — barely enough for a couple of pages of text. Getting from there to models that can genuinely hold a million-token context window in working memory required rethinking how attention itself is computed, not just scaling up the same architecture. That’s exactly the kind of shift founders leaning on AI copilots for daily workflows should be watching, because the ceiling their tools hit today is often a context-length problem in disguise.

Inside MiniMax M3: Sparse Attention and an Intern’s Breakthrough

One of the more disarming moments in the conversation: the sparse-attention architecture that makes M3’s long context computationally viable wasn’t designed by a senior researcher with a decade of papers behind them. It came from an intern.

Standard attention mechanisms compare every token to every other token, and the cost rises sharply as context grows. Sparse attention instead lets the model focus computation on the parts of the context that actually matter for a given step, rather than exhaustively scoring the whole sequence. That’s the difference between a context window that’s technically large and one that’s functionally usable at a million tokens.

Wolf and Song also describe how research projects take shape inside MiniMax more broadly: small teams pursue ideas with real autonomy, and a junior researcher’s architecture can end up shipping in a flagship model rather than sitting in a research note. For an industry that sometimes assumes breakthroughs only come from the most senior person in the room, it’s a reminder worth sitting with.

Multimodal From Day One — and What Happens When Training Goes Wrong

MiniMax didn’t bolt vision and video understanding onto a finished text model. M3 was trained as multimodal from its very first training step, so the model learns text, images and video as one system rather than stitched-together specialists. That choice raises the stakes when something goes wrong mid-training — a failure doesn’t just cost a text checkpoint, it can compromise every modality at once. Song and Wolf discuss what those failure modes look like and how the team recovers, a rare, candid look at the messier side of frontier model training that companies don’t usually put on stage.

Scaling Past a Trillion Parameters While Staying Open

M3 moves beyond one trillion parameters, and MiniMax builds AI products used by roughly 300 million people — numbers that would justify keeping the underlying research firmly behind closed doors. Instead, the company plans to keep open-sourcing its models. Song frames community feedback as a genuine input into how the models improve, not a marketing line: real usage at that scale surfaces failure cases no internal test suite would catch.

Scale used to be the argument for closing a model down. MiniMax is using it as the argument for keeping the model open.

For founders building on open-source foundation models, that combination — frontier scale plus an open license — is worth watching closely. It changes the calculus on vendor lock-in, and it’s the kind of shift that shows up quietly in conversations about who actually owns the tools AI-driven teams depend on.

Agents Building Agents: MiniMax’s Internal Research Harnesses

Perhaps the most striking part of the conversation is what M3 is already doing inside MiniMax. The company has built internal agent harnesses — structured environments that let AI agents run experiments, analyze results and accelerate the company’s own research. Agents that can understand presentations and long-form video are already part of that loop.

The result is a kind of recursive progress: M3 is no longer just the model being discussed on stage, it’s actively helping the team build its successor, M3.1. Wolf and Song point to multi-agent systems — multiple specialized agents coordinating rather than one model doing everything — as the next frontier once long context and multimodality are solid foundations.

That said, agents accelerating their own research only work if the outputs stay trustworthy. Long context and high-volume automation raise the same question that’s dogged AI-generated content everywhere: how do you know the output is grounded rather than confidently wrong? It’s the same tension explored in recent reporting on AI-generated inaccuracies slipping past review — a caution worth carrying into any agent workflow, not just MiniMax’s.

What Founders and Builders Can Take From This

Three things stand out. First, context length isn’t a vanity metric — it’s the difference between an agent that can hold a real task in mind and one that quietly forgets its own instructions halfway through. If you’re evaluating AI tools for your business, ask what happens to quality as a conversation or document gets long, not just how the tool performs on a short demo.

Second, MiniMax’s bet on staying open while operating at frontier scale is worth watching as a market signal. It suggests founders don’t have to choose between capability and flexibility when picking a foundation model — a choice that used to feel much more binary.

Third, and maybe most interesting for anyone running a lean team: internal agent harnesses that let a model help build its own successor point toward where the real leverage in AI development is moving. That’s less about hiring more researchers and more about building the right scaffolding around the models you already have.

▶

Sources

• Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax — AI Engineer channel interview recorded at the AI Engineer World's Fair, San Francisco; published 2026-09-04