Domain specific harnesses

The idea used to be a domain specific language model. It was theoretically sound: build a tokenizer that could represent in-domain tokens effectively, pre-train or fine-tune model weights on in-domain language, and build a specialised model.

But domain specific large language models never really took off. There was always the trade-off, loss of generality, which would hurt the system. Small language models built for a specific task, like classification fared better, but most demand is for a single model

Now, we see that different workloads perform better with certain combinations of model and harness. There are a number of modular less opinionated harnesses that can be forked (like pi), and numerous enterprise harnesses are already open source (codex, qwen code, opencode, gemini cli, grok build, and so on), so the building blocks are there.

The harness manages memory, controls how the model retrieves information, and how it plans work and uses tools. A model is stateless and the harness supplies the instructions, permissions, context, and tools it needs. We no longer rely on specialisation centred around model weights, and move it further into the environment around the model.

Memory handles how an agent codifies and persists information. Memory is a clear target for domain-specific applications as the hierarchy of information per-domains can be specified as a graph or ontology.

Compaction has to handle the model’s working context by maintaining a durable state. This includes retrieval, instructions, task state, checkpoints, and links to the memory system. Specifically for us, what it preserves, how it represents it, and what it is allowed to discard. If i’m working in IE a legal domain, I will not discard raw file references- I will persist an artifact that allows the model to explicitly look up the original text. Whereas if i’m working in another domain where a conceptual summary is more important i’ll allow the agent to do so.

Tools provide the model with ways to observe or change the outside world: search, file access, code execution, APIs, databases, and so on. They can encode domain-specific inputs and outputs, operations and validation loops. This allows us to enforce the correct use of domain-specific data, or force the agent to cross-reference a database before making a change, and so on.

The counter-argument to the domain-specific harness is that each frontier model provider post-trains their model on their harness. That is, which tools to call, when to call them, what arguments they expect, how validators respond, how to recover from errors, when to use memory, and so on. And there will always be some juice to be gained from adjusting model weights to its environment. But given the current general capability of models, the importance of this is diminishing, just as with domain-specific fine tuning, and a retry loop with a comprehensive error message will mean that most models never get a tool call wrong a second time.