If coding is solved, what now?: Measuring the sloppiness of code

LLMs have become almost perfect at generating code, but that isn’t the end of the story. Just because the code is formally correct doesn’t mean that it is not introducing unnecessary abstractions, creating duplicates, or just making bad decisions overall. This is not a groundbreaking observation, most people who have vibe-coded a project, have realized that each additional feature can sometimes lead to an explosion of lines of code (LOC).

This results in a loss of human agency, because in projects that are adding millions of LOC per month, it is hard for humans to keep up. Some people might say that that is not an issue at all, because they trust their agents to deal with it. I have bad news for you, agents can't really deal with the slop either.

Coming from a physics background, I always had an experimental/quantitative approach to solving problems. When I started at Earendil, with the task of figuring out how to measure code sloppiness, my natural instinct was to first take a deep dive into the literature and then check what other companies were doing.

To be frank, with the exception of a few insightful research papers, I was disappointed at how “vibes based” the industry seems at the moment. In my research and on X, I was constantly bombarded with messages such as “End-to-end coding agents”, “AI that doesn't just suggest code—it ships it” or “Human-level evaluation without human-level cost”. Which like all good tales, have a grain of truth in them.

LLMs are able to write almost perfectly correct code. This is because of the scalability and the verifiability of code. It is pretty straightforward to let LLMs generate code and then let that code be checked by hidden tests, which results in a clear reward signal. In stark contrast to that, checking the ‘sloppiness’ of this code often requires human intuition and taste, and is an extremely difficult task in general. I think the best way to illustrate why that is, is by going through possible ways of measuring slop.

AI as a judge: This is probably the most common way of evaluating code quality in the industry and from my observations it rarely works. The more sophisticated approach, namely trying to give the judge model two solutions A and B, and then letting it decide which solution it prefers, has the downside of the model changing its preference when you rename the solutions. Asking LLMs to judge the code they write is not a substitute for a proper evaluation.

Human judges the AI: If we ignore the fact that there is huge diversity in the quality of software-engineers, this would be the best solution to assure that the code stays human readable. With the downside being that this is not scalable for training AI or having large benchmarks with multiple model providers and harnesses.

The simplest method: In my research and tests simply taking the change in the number of LOCs has been a surprisingly effective metric for sloppiness, with the ironic caveat that if we started optimizing for it, it would cease to be a meaningful measure.

The next two measures were introduced to me by the paper SlopCodeBench, and seemed promising because they were able to separate legacy code bases from LLM-slop quite well.

Verbosity: Tries to measure the amount of duplicated and unnecessary verbose lines.

Verbosity = |AST-Grep flagged lines ∪ clone lines| / LOC

Erosion: Tries to measure how much of a codebase's mass is concentrated in a few large and complex functions.

mass(f) = CC(f) * sqrt(SLOC(f))

Here f is the function, SLOC is the source lines of code and CC(f) is the cyclomatic complexity of the function.

Erosion = Σ(mass(f) for CC(f) > 10) / Σ(mass(f) for all f)

The erosion is then simply the fraction between the mass of the functions with a cyclomatic complexity larger than 10, by the mass of all functions.

If we look at the average verbosity and erosion of the code generated during the SlopCodeBench evaluation and compare that to a set of established repos there is a stark difference between them. On average the verbosity in the repos is 0.15 ± 0.06 and in the agents code is 0.33 ± 0.10. For erosion the repos achieve 0.31 ± 0.17 and the agents 0.68 ± 0.20. The agent's code is on average roughly twice as verbose and eroded as human code.

To come back to the point of why agents can’t (really) deal with the slop themselves, we need to look at the evaluation of SlopCodeBench. In contrast to other coding benchmarks, they create multiple rounds of instruction and test iterations, where in between checkpoints the context of the models is erased. The result of that is that bad coding decisions accumulate over time and for the strict solve rate, where all tests have to be passed at all checkpoints, even state of the art models achieve 0% pass rate. Which should be a warning sign to everyone who happily adds tens of thousands or even hundreds of thousands of LOC a day.

In exploring these metrics I hope you now have a clearer picture of why it is challenging to evaluate code sloppiness and why human intuition and taste are still either implicitly or explicitly baked into the evaluation.

There are some promising other directions I want to explore, such as coupledness of functions, code churn, cohesion and so on. If you are working on evals and would like to talk, I would be happy to do that.