Bug Finding Considered Harmful?

…is a question I’ve been asking for a week now, while developing a new property-based testing (PBT) tool in the meantime if you’re one to appreciate irony. As you know, a title with a question usually means the answer is no; but I wouldn’t expect I would ever be in a position to ask the question in the 6 years of working on this, so I thought it’s worth a discussion. Let me step back a bit though.

Property-based testing (PBT) is an “advanced” method of software testing; instead of testing a particular behavior as we classically do, we essentially define a “blueprint”, traditionally called a property. A property is a general fact about the system under test, like “running this program with ANY input should not lead to crash”, “addition of ANY TWO NUMBERS should be commutative”, or “my physics simulation should ALWAYS conserve momentum”. We can express these properties quite easily by writing functions from states or inputs to booleans; we then test them by randomly sampling the input space defined by inputs to those functions.

The job of a testing researcher is, well, finding bugs, locating their root causes, and essentially helping the maintainers fix those bugs. Of course, these are essentially the way we work, not the value proposition; in my opinion the value proposition of testing research is to help build reliable software systems. PBT is only one of the approaches here, fuzzing, deterministic simulation testing, formal verification… all serve the same purpose with different trade-offs in different situations and domains.

A critical tension of testing research is the separation between the tester and the maintainer, the tester invariably approaches the system from outside, as a separate entity than the maintainer. The tester neither understands the system as well as the maintainer, nor are they incentivized or able to do so. I believe this tension is reaching a critical point with the AI age, and I am interested in understanding where we stand.

Barbarians at the Gate: Bug Reports Considered Harmful?

In the last month I apparently filed 500+ bug reports. I'm honestly quite proud of it; it’s the dream of a testing researcher to find novel bugs in well-tested systems. It's like you’re an athlete breaking a world record, at least a much more nerdy and much less impressive version of it. I tried to be very gracious when filing the bugs, I didn’t let an agent dump all of them but rather personally triaged as much as possible; but I couldn’t let out the feeling that if billions of people haven’t hit these cases, maybe they were just not that important in the first place.

The fact that I might be partially responsible for the delay of Postgres19 is quite frightening; although the concept of “fuzzer fatigue” isn’t really new, Z3 prover was a popular target for fuzzers, which at the time led to some heated discussions, SQLite maintainers operate at an incredible speed of fixing some bugs in 20 literal minutes, but I am sure they are unsurprisingly exhausted.

The single best argument in favor of bug finding is security, if we don’t find the bugs, the attackers will, and they will do bad things with them; this works particularly well for classical fuzzing. But especially with more recent work that I’m a part of where we test functional correctness properties of the systems, I think the value proposition shifts.

Don’t get me wrong, bugs are bugs, in all the bug reports I submitted it was only once that we discussed with the maintainers if the behavior was actually a bug; the question is more practical, is it worth fixing all the bugs in a software even if no practical user will ever experience degradation because of the issue?

Even more so; is it worth spending valuable maintainer time on fixing a 10 year old bug nobody has ever hit in exchange for building a new capability that the users will actually use?

The position I came to conclusion is, to the damnation of my own effort in finding old bugs in existing systems, that testing is most valuable when it can go hand in hand with development. Testing that comes after the fact can be valuable for security vulnerability and silent failures; and we indeed cannot know what types of bugs we will encounter when we are testing a system, so when in doubt, we must test. We should, however, be selective in the bugs we report; even more so, the maintainers should be selective in the bugs they choose to fix. The ideological position that we must simply fix every single bug does not seem feasible to me at this moment.

I came to this conclusion through the asymmetry in bug identification vs bug resolution. The reason I was able to manually go through 500+ bugs is because bugs are quite verifiable. Given a brief snippet, I can review the snippet, the expected behavior, the actual behavior; and with very high confidence present a report to the maintainers. Not only the fix for these simple snippets can range from a single line to multiple hundred lines, but also there are many possible fixes for a given bug. It is sometimes discussed if software has a finite or infinite number of bugs, which I believe is the wrong question. I believe software there is a finite set of fixes to a given software to fix all the bugs that violate a given property, however it is improbable that those are the fixes we actually apply. Hence, given a greedy approach where we apply the “simplest” fix for all bugs where we fix only the neighborhood of the test case we currently inspect, we might actually live in a world where we have to do infinitely many bug fixes. The fixes, therefore, must be carefully crafted, forward looking, and essentially much less verifiable than the bugs themselves.

In the context of this asymmetry, I posit that bug finding will continue to overwhelm bug fixing, so it is left to us to account for this asymmetry so that we can continue to build software. Even in the context of formal verification where we can technically enumerate all bugs that invalidate a stated property, we will simply find more and more properties of a system to test for; correctness to me seems like a never ending pursuit of excellence.

I make these arguments in the context of in vitro testing, testing done after the fact as a method of hardening the system; on the other hand, I believe in vivo testing is more useful than ever, because we of course want to build correct systems given that we do not need to sacrifice usefulness, expressivity, performance… Especially as we get better verification capabilities, leveraging those capabilities to speed up the development instead of slowing it down could, and perhaps should be our norm.