Knowledge Agents: Beat Frontier Models with Better Structure

Anthropic recently had to pull Mythos/Fable due to an edict from the US government. While Mythos was a step up from Opus, I’ve been actively moving smaller in terms of my agentic models—and matching the quality of output of some of the largest frontier models.

The use cases have spanned from hard “hedge fund level” market analysis, financial management, and AI personal assistants to even helping a few friends in difficult medical situations. I’ve called this pattern “knowledge agents” with a generic template available to everyone here. They literally inject the right knowledge into the AI agent plugged into it. Anyone can do this, with or without my template.

As my README proudly declares:

This methodology was developed and battle-tested on a markets knowledge agent, meant to replicate James Wang’s thought process in markets: ~10,000 pages of scanned financial market reference materials + ~100 web articles, producing 381 concept documents and 54 thesis documents with hybrid BM25 + semantic search. This was further tested on other specialized knowledge areas—including company-specific policy docs (for a “corporate knowledge agent”) and rare research areas (women’s sexual health)—to great effect. The generalized version here captures a domain-agnostic methodology so it can be applied to any subject.

These were the first, but at this point I have twelve of these specialist “knowledge agents” that handle queries from other agents. Or, obviously, from me. When I’m coding new things that require specialist knowledge, I often start Claude Code in a knowledge agent folder instead of making a new folder and have it benefit from the expert knowledge within it to plan. Especially for specialized machine learning algorithms or economic models, I get far better results this way than with a “subject-agnostic” model—even a really big frontier model.

In general, I have used Claude Opus in these knowledge agent “harnesses.” As such, it’s pairing the really big model with injected knowledge from the harness. However, I’ve found that I get very, very good results even with far smaller models. The LLM is merely the “engine”—all of the expert knowledge is provided from my knowledge agent system, which surfaces the relevant knowledge at the right time.

Relevant, of course, is key. As most of you know, you can’t just drag 10,000 pages of documents into your chat window. Even if you could, you’d get a mess of irrelevant information drowning the LLM.

This has allowed me to move many of my agents from Anthropic’s Claude to a locally run open-weight Qwen model. It’s a tiny fraction of the size of Claude Opus and is able to run on hardware I have plugged in at home.

How does it work?

The simple answer, as said, is that it injects the right, specific knowledge into the AI agent at the right time. Let’s talk about the forms that knowledge takes in LLMs.

First, a significant portion of frontier models’ huge footprint is “knowledge.” While that’s very useful if you’re casually asking about some random topic, it’s entirely irrelevant if you either already have the data you want to reference or the data isn’t publicly available anyway. The latter is quite common in fields that are specialist, secretive, or proprietary. A lot of the massive size of the model to cover every random subject matter is a huge waste.

The second form is data provided in the context window—in other words, your prompt/query. Injected knowledge in the context window does not make it impossible to have hallucinations, but it is mechanistically different from parametric knowledge. And, in general, if you’re injecting relevant knowledge, it’s right there and more likely to be used.

This base concept is RAG (retrieval-augmented generation). The difficulty is not “give relevant information.” It’s how to actually do it and surface the right thing for a good answer—even with extremely difficult questions.

Embedding

A naive text search is often going to miss things. If I search up a concept about a “poodle,” things that relate to “dog” might be highly relevant. To not be awful, we need to have, at minimum, base-level related concepts come up that don’t need literal matches. In my knowledge agents, I use both literal search and search using embeddings.

For calculating embeddings, I now use a local embedding model (BGE-M3), but OpenAI has an easy one to use with just an API key (text-embedding-3-small).

Structure

The question here is how do we chunk our data? How do we break it up into chunks that are relevant? For my purposes, having referenced summarized subdocuments for certain purposes ended up working. I have three types of documents:

1. Source extractions—these are the raw sources in markdown.
2. Concepts—these are the “encyclopedia entries” for our canonical knowledge base.
3. Theses—these are more opinionated, cross-cutting syntheses of multiple sources.
4. PRIMER.md—this is the “summary” and self-updating guide that helps orient the agent on startup.

Multi-Pass, if Necessary

I have found that to get the best results, we need to set even more tokens on fire. Even with perfect concept and thesis extraction, sometimes we have a really, really hard topic that requires multiple searches. The concept here, which is written into the instructions for the agent, is it must do multiple passes. I landed on three searches; in general, that gives enough breadth without drowning the agent.

So how well does this work?

There is no objective measure for these things. I used a three-frontier model panel to score the answers. I threw a fairly simple question about lessons from the Great Depression and Japan for monetary policy. These are topics that are definitely in Claude Opus and similarly sized models.

In a way, our results do tell a story as to why Anthropic is riding high right now. For such a hard query, Claude Opus 4.8 did remarkably well. As you can see, the knowledge agent basically didn’t really help it. In fact, on the easy query, the knowledge agent actually hurt it, probably because its built-in parametric knowledge gave more relevant information.

Nevertheless, the real story is how they did with the harness. The harness basically equalized the models, including Qwen 3.6 27B. Remember, that model is literally small enough to run at home with consumer hardware.

Plugging a real weakness in LLMs

Knowledge-based AI (KBAI)—also known as symbolic AI—was our prior generation. Unlike today’s AI, it never got an answer wrong or hallucinated, because it had a strict process of drawing from a knowledge base and applying rules. Today’s AI is much more free. I usually give the analogy of a student who has memorized facts but isn’t able to apply them broadly and another student who really doesn’t remember many facts but is able to “get the gist.” KBAI is the former, and modern deep learning is the latter.

This structure is attempting to plug the hole, because we can never totally remove hallucinations without destroying the utility of modern AI. It is, in a way, a hybrid system—which is what inspired my naming of them: “knowledge agents.”

It’s remarkable how far small models have come, especially with software improvements. It’s great that open-weight models have closed the gap a lot more. And it’s liberating to know that even if Anthropic and OpenAI close off access to their models, we can still get similar results, just with more work.