Supervisory Engineering Work

Automation is not new in software development. We’ve been automating parts of our process for decades - build pipelines, testing, deployments. With AI, we’re now able to automate work that previously required humans with specialised skills to do by hand - specifically, the writing of the code. In other words, AI is enabling us to automate a larger portion of the job that software engineers used to do themselves.

As part of my Masters research on the impact of AI on software engineering, I wanted to understand how AI might shift engineers’ focus across the common tasks that developers generally spend their time on when building software. What I found was a creation-to-verification shift, with engineers shifting their focus towards verification-based tasks such as testing and reviewing. I also saw evidence of a new kind of work emerging - supervisory engineering work - that I defined as directing AI, evaluating its output, and correcting it when it’s wrong.

In this post, I’ll make the case that supervisory engineering work is a new form of a much older concept - supervisory control - one that’s been studied for decades and can tell us a lot about what this kind of job means for the people doing it.

It’s not the same job

I often see people arguing that AI has done the reverse of “killing software engineering” - that it has instead meant we need more software than ever before, which in turn means there’s more demand for software engineers too. As they say, Jevons Paradox appears to be playing out exactly as predicted.

That may well be true, but I think it misses an important point: with so much of it now automated, it isn’t the same job any more. We may still call it “software engineering”, but the activities it’s composed of, and how we spend our time on them, are materially different to what they were even just a couple of years ago. New activities require new skills, new training, new definitions of what good looks like, and new ways of hiring and growing people - to name a few. There’s a lot to work through.

This may help explain why this moment feels so disorienting for the software engineering community. For many, supervising AI is not the job they trained for, expected to do, or enjoy doing - and for some, it challenges the very identity they’ve built their careers around. Yet, if they hope to remain employed as software engineers, they’re being left little choice but to adapt.

I believe it’s time for our industry to recognise that the role of a software engineer has changed. We need to agree on what the job actually involves now and what it takes to do it well, and then adjust our expectations to match. For example, what does it really mean to hold engineers accountable for code they’re less and less likely to have written, or even read?

Supervisory control: the job automation leaves behind

That research falls under a field called human factors (or ergonomics), dedicated to exploring how humans and systems work together. The International Ergonomics Association formally defines it as:

The scientific discipline concerned with the understanding of interactions among humans and other elements of a system, and the profession that applies theory, principles, data and methods to design in order to optimize human well-being and overall system performance.

Human factors emerged during the Second World War, when airplanes were becoming faster and more complex than pilots could handle, which unfortunately resulted in planes crashing. Initially this was chalked up to “pilot error” but later research found “design error” to be an important factor. This in turn caused a change in the thinking: instead of training people to fit the machine, why not fit the machine to the person?

The concept of supervisory control followed a couple of decades later, when computers were put to use between the human operator and the machine. Thomas Sheridan and his colleagues at MIT started working on it in the late 1960s and continued for several decades. In 1992, Sheridan defined it as:

In the strictest sense, supervisory control means that one or more human operators are intermittently programming and continually receiving information from a computer that itself closes an autonomous control loop through artificial effectors to the controlled process or task environment.

The main difference between direct and supervisory control is that in the latter, there’s a computer that closes the loop instead of a human operator. Before AI, writing software was much more like direct control - we closed the loop ourselves. We interpreted requirements, made changes to the code, received feedback and acted on it, over and over. Now, coding agents can increasingly close that loop themselves - writing code, running tests and fixing their own mistakes - while we move towards supervisory control: telling the agent what we want, keeping an eye on how it’s going and taking over when we need to.

So what does the supervisor actually do? Sheridan and Verplank described five roles - plan, command, monitor, intervene and trust - and when I put them alongside the components of supervisory engineering work from my research, they line up surprisingly well:

Levels of automation

In the same report, Sheridan and Verplank described ten levels of automation in “man-computer decision-making”.

By 1995, this list had been narrowed down to five distinct levels of automation in a paper by Mica Endsley and Esin Kiris. These five levels are easier to map to what we’ve experienced over the last four years:

The out-of-the-loop performance problem

In the same paper, Endsley and Kiris explored what they called the “out-of-the-loop performance problem” - what happens when automation fails and the human has to take over. They hypothesised that automation could erode both manual skills and situation awareness, making it harder for people to spot when something has gone wrong, or to do the task themselves should they need to. In this context, situation awareness is defined as:

The perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.

Their experiments revealed several interesting findings. They found evidence of what they expected to see: situation awareness dropped as the level of automation increased. More specifically, while their perception of the system held steady, comprehension of the system dropped - in other words, operators could see what had changed but not what it meant.

It’s easy to see the parallels with software development here. When you write the code yourself, you build up a detailed mental model of the system because you had to actively create it. Conversely, when all you do is read code, it’s much harder to reach that same level of understanding because reading is a far more passive mode of learning.

A different job requires a different system around it

The biggest takeaway is that supervisory control - or supervisory engineering work, in our context - is a different job, and a different job requires a different system around it. We’re shifting from a form of direct control to supervisory control, from cutting code ourselves to running agents to do it for us, but the way we build software is still largely designed around humans writing the code.

Lisanne Bainbridge saw this coming in her 1983 paper, Ironies of Automation, in which she described a job reduced to monitoring as “very boring but very responsible”, and “difficult for the individuals involved to come to terms with”.

Redesigning the job and the system

So folks, we’ve been here before, and if you’re feeling disoriented, you’re not alone and it isn’t a personal failing. But thankfully, we have decades of human factors, supervisory control and automation research to help guide us through this rather unsettling time.

As an industry, our next step must be to start redesigning the job and the system, in tandem. By this, I mean we should:

• Deliberately choose the level of automation to operate at, and for which types of work - deciding this helps us understand what we expect the human to do.
• Design system signals to support situation awareness - build our harnesses and software factories to surface just the right signals, at the right time, in the right way and at the right level of fidelity to support our comprehension of the state of the system without needing to “review harder”.
• Describe intervention requirements and mechanisms - explicitly set expectations about when and how humans should step in (from small adjustments to taking over) and what they’re accountable for, then ensure the system is designed to support that.
• Prepare people for the job - invest in the skills and training that supervision demands, especially for the rare moments when they need to step in.