Why the world's best AI startups write bad prompts (& how to fix this)

Most prompts are bad because prompt evolution tends to be accretive: we only add, never remove, over time. This leads to spaghetti prompts, with contradictions and ambiguity. This has real business impact.

We need to treat prompt changes as product changes (because agent behaviour is product), and treat prompts as code (modularised, MECE, and all your other favourite acronyms; refactored if need be & actively maintained).

Well structured prompts enable teams to move faster, prevent regressions, and have better agents. I propose a very simple structure at the end.

Prompting decisions are product decisions, and using structure to make unambiguous, maintainable prompts is critical for making great agents. This is a guide on how to do that.

Background

I think I have one of the best jobs in the world. I lead Applied AI Engineering for the OpenAI startups team across EMEA & APAC, and that means that every week I get to see behind the scenes of the best AI startups globally. And I get pretty hands on in how I work with their engineers to improve their agents: on everything from prompts to evals to finetuning.

These startups are advanced. Some have ARR in the hundreds of millions. Some have their own data annotation teams. Some train their own models.

So it came as a surprise that often, when I look behind the curtains, they have prompts that just don't make sense. This is not about being beautifully written prose, or nicely formatted; it's about logical errors that lead to the mistakes their agents make.

This is not a critique of those startups - indeed, they are more successful than any company I have ever built, and their teams are full of the best engineers globally. They are a true pleasure to work with.

But they're leaving huge gains on the table. After only a couple of days re-writing agents together, I've seen some startups speed up their agents by 50%; others increase 7 day retention by 40%; and still others reduce costs by 30%. These results hold across the LLMs they use, from every provider. When you're talking millions of ARR & LLM spend, this is pretty material.

Don't believe me? See this Loveable engineer's post about how he decreased their LLM spend by $20M per year… because his mum caught inconsistencies & duplication in their prompt.

In fact it's because they are so incredible, that I'm writing this. Because clearly even when you are genuinely world class, our current paradigm for prompting leads to suboptimal results.

So I thought I would try to scale my impact beyond the startups I can work with directly by writing this. First, I'll cover why the world's best startups write bad prompts; then, I'll cover my prompting philosophy; finally, I'll propose a prompt template.

Bad prompts

Given this is so common, there are clearly universal tendencies that lead to bad prompts. The two key issues are contradictions and ambiguity.

Our current process for prompting is accretive & leads to contradictions

Most prompts evolve like this: the first engineer building an agent writes a simple prose prompt. As the startup grows, and the agent is required to do more things, they add to the prompt. Errors occur, so they add a few lines to fix those.

The prompt only gets longer.

And no-one reviews the entire prompt end to end. Almost always, that leads to contradictions in the prompt, because as you add new content, old content saying something else is kept.

Implicit knowledge leads to ambiguity

Even if an engineer does review a prompt end to end, they often don't truly read it. When you read & interpret a sentence, you don't only use the words on the page. You use all of the knowledge you already have to make sense of those words. And that's a problem, because you often know what you want the sentence to say; and you read that, rather than what it actually does say.

This is the problem of specificity: the prompt doesn't actually say what we want the agent to do, because we haven't unambiguously specified it. For example, let's say I tell my agent to "never refer to competitors in [its] output". This makes sense to that startup’s engineer who spends every day thinking about my startup and its competitors. But to an agent without that implicit knowledge, that is incredibly vague - who are the competitors? What about partial competitors we also collaborate with sometimes? We haven't specified what we actually want.

Conditional prompts compound this

The above problems are compounded by conditional prompts, where additional prompt content is injected depending on the scenario. Different engineers work on various parts in separate files. And that means that even if each team/engineer reviews their prompt for contradictions & specificity, no-one reviews the whole.

The outcome? We get spaghetti prompts. Thousands of lines, with interaction effects between many paragraphs, and it's almost impossible to review them because by the final sentence, most humans have totally forgotten the first line. Or got bored and stopped.

My principles: start treating prompting as product, and prompting as code.

Prompt decisions are product decisions

Agents are at the core of product experience. In chat based products, they are the entire product. The formatting of the agent's output is the UI. For example, should it output bullet points? Or markdown?; should it always reply in English? or in the language of the user's message?

Agent behaviour is a product decision: should it err on the side of responding fast? Or do comprehensive research, making the user wait? Should it call a tool asking for the user to approve something? Or just get on with it? It's all product.

Prompting as code

We use human readable language to get a computer to do what we want. We can borrow principles from software engineering to manage prompts that grow over time.

• Structure with MECE prompt sections: MECE stands for mutually exclusive (ME), collectively exhaustive (CE). Collectively Exhaustive means your prompt sections comprehensively cover the behaviour you want. Mutually Exclusive means each prompt section should be self contained, with no overlap.
• Aim for the specificity of programming: Think of it like code. If this, then that. Specify behaviours for core branches of user requests.
• Separation of backend and frontend: Separate Behaviour (how the agent acts/uses tools) from Output (what the user actually sees).
• Refactor every so often: Dedicate time to paying down your prompt debt.

Template of a good prompt: Background, Behaviour, Output

You'll notice this is hierarchically organised, like a tree. This helps it be MECE, and means you can find sections much faster. If you're having issues with your agent's output, it's unambiguous where that content sits.

A quick note on evals

Evals force you to decide on what you want. When you make an eval, you have to specify the desired behaviour. And half the time you realise you literally just never specified that desire in your prompt, it was latent context in your brain, not explicit in the prompt, OR you actually hadn't clarified it even to yourself.

Benefits of structured prompts

• Fewer contradictions: With MECE sections, reviewing the prompt is easy. Each section is self-contained.
• Faster & safer iteration: Separating concerns lets you iterate incredibly fast without worrying about interaction effects.
• Faster search: Hierarchically organised prompts make finding relevant sections much faster.
• Everyone's an engineer: It's easier for non-engineers to contribute when the structure is clear and self-contained.
• Faster Model Upgrades: Testing model defaults or removing outdated instructions becomes a one-line change.

FAQs

Isn't this just a waste of time if our prompt works now?

No, the payback period is pretty fast. It will save a LOT of time when your agent starts making very weird and complex mistakes, because those are the hardest issues to solve in spaghetti prompts.

As models get smarter doesn't this become irrelevant?

No. Higher intelligence cannot automatically solve ambiguity and contradictions in your preferences. Only you can. You can often leave the "how" less specified, but you still need to specify what you want.

Can't we just automate this process and have LLMs write the prompts?

Somewhere, you need to specify your preferences unambiguously, whether in the final prompt, or in the 'rewriter' prompt. There's no free lunch; you need to get clarity for yourself.