Human Intelligence is Surprising

It is common to be humbled by AI. Today’s models beat experts who have mastered their craft over years, often in minutes, and with simple prompts. As models continue to advance quickly, this is happening repeatedly and everywhere. Software engineers are watching the nature of their work change while writers, researchers, lawyers and bankers all stand nervously by.

Activities once reserved for truly gifted people can now be reproduced for pennies on the dollar. It’s natural that encounters like this lead some to more fundamental questions: what was it that made me special, what makes humans special?

Looking for some remote edges of human capability is the wrong response; we should take an even wider and more humbling view. Examining the origins of intelligence reveals that it is not the inevitable endpoint of evolution, but a relatively recent, circumstantial, and surprising phenomenon.

Life appeared on Earth at least 3.7 billion years ago. Complex multicellular organisms emerged roughly a billion years ago, animals around 600 million years ago, land plants around 470 million years ago and the first primates around 55 million years ago. Homo sapiens have existed for only about 300,000 years. On the timescale of life, human intelligence arrived almost yesterday.

And once it arrived, it moved quickly. A species with enough intelligence for language and toolmaking learned to reshape its environment, leading it to exponential and cascading planetary change. What feels to us like the long arc of human history and complex cultural development is, from the perspective of evolution, a sudden discontinuity.

In fact, we actually never really had much of a chance to develop meaningful intelligence. Evolution does not optimize for intelligence in the abstract and has no implicit reason to produce the maximum possible intelligence. In effect, once we saw a glimpse of consciousness, we began changing our environment faster than natural selection could change us. We had crossed the threshold at which our technology guaranteed our existence.

Evolution is a lazy optimizer for intelligence.

It should not surprise us that there are better mechanisms for developing particular forms of intelligence. Evolution takes billions of years, haphazardly optimizing organisms for survival under messy and shifting constraints. Machine learning can optimize directly for a target that we explicitly define.

Consider a definition of intelligence as the ability to master the game of Go.

Humans were obviously not selected by evolution for their ability to play this game. The fact that a person can study Go, understand its strategy, and eventually become a grandmaster is a fortunate byproduct of a general intelligence that evolved for entirely different reasons (survive and reproduce in their particular environment). Yet once Go performance became the explicit objective, researchers at Google built AlphaGo, which quickly became superior to the best human Go player at the time.

The achievement took only on the order of months once development focused on that objective. It isn’t because Go is trivial. It is because the optimization target was made unambiguous. Once the hill was visible, we were able to climb it quickly.

When researchers can specify a task, construct a useful distribution of problems, and measure success, AI can reach superior-to-human performance in an extremely short period of time. Not every important capability is easy to formalize. But I expect the hill-climbing trend to continue wherever we can explicitly capture a task that “smart” humans can do today.

Defining intelligence is the greatest barrier to realizing it.

The emerging work on recursive self-improvement (RSI) makes this especially clear. An observer following the lab news of crazy compute spend, outrageous compensation packages and magical capabilities of models may believe the model development process to be mystical. In actuality, the model development process is now well understood and somewhat procedural. Model development can even be decomposed into tasks: finding failure modes, collecting data, running experiments, optimizing the infrastructure, and serving models, to name a few.

In our effort to define and measure RSI through a set of illustrative small tasks from this process, we now have well-defined metrics we can contribute to our conception of intelligence itself. Hill-climbing as a dimension of intelligence is made legible. And sure enough, we are starting to see AI models show promise at the discrete steps for RSI. As with all prior hills, hill-climbing is itself hill-climbable.

This reinforces my belief that being able to define what intelligence should look like is one of the last real human problems. Even the hill-climbing process itself will soon be automated.

The optimization process is becoming well defined and therefore easier to optimize.

This is discerning, but it also unveils a more foundational and disconcerting problem. If defining intelligence allows us to build it, then the definition matters enormously. A system optimized to earn money in the stock market will develop a different intelligence from one optimized to cure cancer, write requested software, or play Pokémon. Even within a single domain, a slightly wrong objective can produce behavior that is highly capable and completely undesirable. A coding agent rewarded only for passing a test might fix the software or manipulate the test, exploit the environment, or take some shortcut its designers failed to anticipate.

This is not a new problem introduced by AI. Human intelligence has always developed in response to how it is evaluated. Students study reading, writing, and mathematics because those subjects are tested in the SAT and determine college placements. In eras when Latin signaled education and status, students spent years learning Latin. Labor markets influence where ambitious people spend their time improving and applying their abilities: trading on the market, trying to solve cancer, writing a novel, or elsewhere.

Measures of intelligence shape it. Defining intelligence is our greatest barrier to realizing it.

With Vals AI, we care a lot about exposing model capabilities through well defined evaluations. Our primary tools for this are the benchmarks we construct measuring models’ ability to do real, valuable work.

In my most abstraction conception, a benchmark is an input distribution and an output-verifier. These are the problems or queries we choose to present, and the way we decide what counts as a good answer, respectively. Both choices carry a worldview. If the problem distribution is narrow or contrived, a system can become excellent at the test while remaining brittle outside it. If the reward captures the wrong thing, the system can become excellent in exactly the wrong direction. It is the burden of the person making the evaluations to ensure they are pushing model progress in the right direction.

Optimizing for the wrong or an ill-defined notion of intelligence will lead us astray.

The rise of machine intelligence should make us doubly humble. First, we should be humble about human intelligence. It is extraordinary, but it is also recent, circumstantial, and shaped by an optimizer that was never trying to create the smartest possible being. There is no reason to assume it represents an upper bound.

Second, we should be humble about our ability to define what comes next. As optimization becomes cheaper and more powerful, choosing the objective becomes more consequential. The most important remaining human task may not be to stay ahead on every hill but instead to decide which hills are worth climbing and how to construct them. Whereas for humans intelligence has largely plateaued, for models there is always a higher peak.