Fern
Mirendil has introduced Fern, an LLM supporting a massive 10-million-token context window designed specifically for high-volume agentic coding tasks.
Deep dive
- Scale: Supports up to 10M tokens, roughly equivalent to 23MB of code.
- Architecture: The model is designed for agentic tasks like analyzing full training traces and auditing data shards.
- Target Use-cases: Debugging logs, end-to-end agent trace analysis, and research-heavy coding environments.
- Status: Currently in early access/post-training phases.
Decoder
- RAG (Retrieval-Augmented Generation): A technique that connects an LLM to external data sources to retrieve relevant information before generating a response, used to mitigate limited context windows.
Original article
Introducing Fern
Fern is Mirendil’s first model. It is pretrained from scratch and supports a context window of up to 10M tokens. Its first version, Fern 0.1, was finished with a short round of post-training. New versions ship frequently.
The Mirendil grove
Fern 0.1 · Latest
Our most efficient model, built for focused, high-volume agentic tasks.
- Base model: Mirendil-Base
- Size: 0.1–0.5T params
- Context: Up to 10M tokens
- Input: Text and code
- Built for: High-volume agentic tasks
- Access: Coming soon
More capable, for harder and longer-running work.
- Base model: Mirendil-Base-Large
- Size: 0.5–2T params
Our most capable model, for the most demanding work.
- Base model: Mirendil-Base-XL
- Size: 2–10T params
Context is intelligence
A language model’s capabilities grow with compute, and there are two ways to spend more of it. One is the model’s parameter count. The other is its context length. A model with more parameters holds more static knowledge and can carry out more sophisticated reasoning steps. A model with a larger context window can use more task-specific information: draw on more of its own past experience, keep track of more specifications and instructions, integrate evidence from more documents and more code, and look at more data directly.
Fern’s architecture, training and serving can support up to 10M tokens, about 23 MB of code, at once. That’s about half a million lines of code, a codebase on the scale of the Megatron-LM library, dozens to hundreds of complete agent traces, or days of an agent’s work, all held in context at the same time.
Below, we first show that more context helps even with next-token prediction, the foundation of language model training and of every capability built on it. We then run a basic test of effective access to the long context with a standard benchmark (RULER). More comprehensive long-context results are coming soon.
Early post-training results
Fern is built for the focused, high-volume agentic work of anyone building AI: auditing training data, debugging a training run from its logs, and reading agent traces end to end. Fern 0.1 is Mirendil-Base after a short round of post-training. Here’s how it performs on agentic benchmarks at this point.
What Fern is for
- Works on larger repos: A 10M-token context window, so it can work across larger codebases.
- Agentic coding: Runs commands, reads the output and keeps going until the tests pass.
- Research code: Specialized for the code behind AI models: training loops, data pipelines and evaluations.
- Debugging from logs: Works through long logs and stack traces to find where a run went wrong.
- Analyzing full traces: Reads an entire agent or training trace end to end, so it can explain what happened and why.
- Auditing training data: Scans a whole pretraining shard for duplicates, benchmark contamination and junk.
1 Bits per byte is a model’s cross-entropy loss per byte of text, so models with different tokenizers can be compared fairly. Lower is better.
Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi