Agents as Webs of Beliefs

In this post I’ll sketch out an informal model of intelligent agents as webs of beliefs (or belief webs for short). The belief webs framework pulls together ideas from active inference, agent foundations and machine learning. In doing so it aims to unify beliefs, goals and actions as three facets of a single phenomenon.

Beliefs are held together by local consistency constraints

The core premise of belief webs is that an agent’s beliefs are typically locally consistent with nearby beliefs but not necessarily globally consistent with all its other beliefs (except, perhaps, in the limit of ideal rationality). This poses a problem for frameworks which describe agents in terms of a single probability distribution.

Two frameworks which are capable of handling global inconsistency are Richardson’s probabilistic dependency graphs (PDGs) and Garrabrant induction. We can roughly analogize the nodes in PDGs to the propositions in Garrabrant inductors; I’ll call them “base-level beliefs”. The central type of base-level belief I think about is beliefs about sensory inputs.

There’s then a second layer of structure in both PDGs (namely hyperedges) and Garrabrant induction (namely traders) which imposes local constraints on base-level beliefs. I think of hyperedges/traders as steps towards formalizing the concept of “concepts”. For example, if you see the front half of a cat starting to emerge from around the corner, a “cat” hyperedge/trader might make predictions about what you’ll see next, which shape your base-level beliefs.

However, having exactly two layers of structure seems rather artificial. In active inference/predictive processing, minds are viewed as hierarchical generative models, with each layer of the hierarchy forming new concepts with reference to lower-level concepts. The success of deep learning suggests that there’s something fundamentally important about this kind of hierarchical concept formation.

So you can think of the term “belief webs” as a (still vague) pointer towards a framework which is capable of handling both internal inconsistency and also hierarchical concept formation. The core difficulty I currently see is in figuring out what it means for a proposition formulated in terms of high-level concepts to be true or false, given that those concepts don’t have binary truth-values themselves.

Actions are beliefs

PDGs and Garrabrant inductors are epistemic processes, not agents. However, Abram Demski’s post on FixDT provides a very interesting suggestion for how to think of an epistemic process as choosing actions. Abram points out that an agent’s beliefs can sometimes affect the world directly (not just via influencing their actions). Indeed, many real-world scenarios (including most social interactions) are affected by our thoughts, not just our deliberate actions.

In one sense, this is a complication: now we need to optimize both our thoughts and actions to achieve our goals. But in another sense it’s a simplification, because it unifies beliefs and actions: we can simply consider actions as the subset of beliefs where (we expect that) holding that belief makes it come true due to the intervention of an external actuator. For example, I can think of my nervous system (or a neuralink interface) as watching out for the belief “I will move my arm” and then moving my arm in response.

I’ll call this the self-predictive model of actions (in contrast with the implicit standard “argmax model” where rational agents directly implement the action with highest expected utility). The self-predictive model is closely related to the idea of action as prediction from active inference. However, a key difference between active inference and Garrabrant induction is that only the latter can handle paradoxes of self-reference (using the probabilistic logic approach pioneered in this paper). So Garrabrant inductors should be capable of believing sentences like “if I believe X, then X will come true” without causing difficulties. This provides an epistemic grounding for which predictions you should think of as actions, which then generalizes to things we don’t usually think of as actions.

Indeed, in the self-predictive model of actions, much of the hard work of being an effective agent is managing your beliefs. Under the self-predictive model, there’s a gap between believing that an action is a good idea and actually taking it, because you also need to believe that you’re the kind of agent who acts on good ideas. This helps explain the central role of the ego in psychology, and why people are often so sensitive to negative feedback—since taking that feedback on board in the wrong way could harm their ability to act coherently. Instead, people typically spend a huge amount of time building up and maintaining their identity as a good, productive, trustworthy agent. Such an identity is a kind of belief, but the process of forming it can’t be described in terms of Bayesian updating, because it often involves choosing between multiple different self-fulfilling beliefs.

The self-predictive model also helps explain internal conflict (a phenomenon which the argmax model struggles with). If identifying the best action were sufficient for taking it, then we wouldn’t procrastinate or self-sabotage nearly as much. But our reasoning processes are only one input into our overall expectations about what actions we’ll take. Other inputs include our identities (which thereby serve as commitment mechanisms), predictions that we’ll continue long-standing habits (even when we know they’re bad for us), emotional memories and traumas, and learned (or evolved) instincts and heuristics.

Goals are beliefs

I’ve talked a lot about how to choose actions, but not much about the goals behind those choices. In Abram’s original FixDT framework, utility functions live separately from beliefs: FixDT agents search for all fixed points of their beliefs (including beliefs about the actions they’ll take) and then select the highest-utility fixed point. However, this reintroduces the ugly argmax that we’d gotten rid of by treating actions as beliefs.

But what alternative way is there to navigate towards a good fixed point? The main alternative Abram discusses is the active inference approach of interpreting goals as beliefs. Specifically, goals in active inference are beliefs which are fixed at an artificially high credence. We can then infer from those credences that we will probably take actions consistent with achieving those goals.

I think this doesn’t work in its current form, but is pointing in the right direction. Suppose I start off with the following set of consistent beliefs:

• P(win race next month | train this month) = 0.36
• P(win race next month | ¬train this month) = 0.04
• P(win race next month) = 0.12
• P(train this month) = 0.25

I’ll treat P(train this month) as an action in the sense described in the previous section—that is, my credence in it is self-fulfilling. Now suppose that I represent my goal as a credence in winning the race that’s higher than 0.12, and update my other credences accordingly.

I don’t think the active inference literature has a solution to this problem, but to me it seems like the natural response is to avoid fixing goals at all, and rather think in terms of “forces” that are trying to pull credences in goals upwards (which I’ll call drives). Conversely, we should think about credences in empirical beliefs as being pulled into place by the force of empirical evidence (which I’ll call anchors). A belief web equilibrates when those two types of forces—drives and anchors—balance.

You might be concerned that we’re still in a setting where our desires can move our empirical beliefs. However, we can define a fully rational agent as the limiting case where action probabilities are moved using drives of arbitrarily small magnitude (relative to the strength of anchoring to empirical evidence). We might then be able to define such an agent’s utility function in terms of the choices it would make in that limit.

I also call these forces “drives” rather than “preferences” because my current guess is that insofar as belief webs are a good model of humans, drives correspond to evolutionarily hardcoded desires, which affect higher-level goals only by propagating through the belief web. In other words, what makes a high-level outcome a goal is just the fact that the credence we assign it is sensitive to drives applied at lower levels.

Open problems for belief webs

I find the picture I’ve sketched out above extremely elegant. It takes the beautiful unification provided by active inference (that actions are self-fulfilling predictions, and goals are optimistic beliefs) and at least gestures towards putting it on firmer foundations. The big open problem is of course how to make all of these claims more precise and rigorous.

In order to pin down such a formal framework, it feels necessary to grapple with a few core conceptual questions:

1. Can belief webs reach the best equilibrium in principle? By default it seems like they might just get stuck in local equilibria: unlike FixDT they don’t have a mechanism to “jump” into the best equilibrium.
2. It seems like belief webs implicitly implement EDT, which struggles to evaluate hypotheticals without interference from existing beliefs. But might they have emergent FDT/UDT-like properties in a way that gets the best of both worlds?
3. I’ve played pretty fast and loose with the notion of self-reference in belief webs. I’m uncertain whether this will basically turn out to be fine, or if there are important nuances that I’ve missed.
4. As mentioned in the first section: what does it mean for a proposition formulated in terms of high-level concepts to be true or false, given that those concepts don’t have binary truth-values themselves?

My longer-term hope is that belief webs will allow us to think of individual agents as an emergent phenomenon, rather than something we need to bake in when reasoning about intelligent agency. You could potentially consider all intelligent beings to be part of a huge, highly non-equilibrated belief web. An “agent” could then just be a densely-connected region of that belief web which trusts updates from within that region much more than updates from outside it.