There is reason to suspect that [the transition from the relatively closed social world of a great ape residential group to the richly connected forager bands of ethnography ] was very gradual. Its initial roots may date back to the Heidelbergensians, at about 800 kya. But it was not complete until the very late Pleistocene (perhaps even later). For it is only then that we find evidence of active cooperation across residential groups, and perhaps of clan structures that transcend an individual forager band. The final transition began in the terminal Pleistocene (about 25 kya–12 kya) and in the early Holocene, with the origins of sedentary society. This led to an increase in social scale and social inequality. In trying to understand why cooperation is stable, scale, complexity and inequality all matter. For mechanisms that suffice to stabilize cooperation in small politically unstructured and relatively homogenous social environments breakdown in larger and more structured ones. Personal knowledge and trust can stabilize cooperation in small, intimate social environments but not larger and more differentiated ones. Continued cooperation is especially puzzling in social worlds that are not just larger but also hierarchically structured. For these seem to be cases where the profits of cooperation are largely hijacked by elites. In such cases, theory predicts the collapse of cooperation. The take-home message of this chapter is that culturally evolved tools—language, myth, ritual, explicit norms—play a central role in the stability of cooperation in the late Pleistocene shift in the economic foundations of cooperation, and an equally central role in the survival of the social contract through the final Pleistocene and early Holocene social revolutions.

In many cases, the benefits of mindreading will have nothing to do with signaling, but the following are common situations in which signaling benefits (to receiver and/or sender) can emerge:

Behavioral Prediction: predicting the behavior of another.

Behavioral Stabilization: regulating other people’s behavior so that it becomes stable and predictable.

Trust Problems: discerning whether to approach a person, trust them with your property, expect that they will reciprocate your altruism, etc.

Commitment Problems: getting others to commit to a future action or having a reason to commit yourself to a future action.

Coordination Problems: figuring out how to work with others to jointly accomplish a goal or to work in parallel.

The first two categories are quite broad and include the latter three as special instances: predicting and shaping the behavior of others is a way to achieve trust, commitment, and coordination. But what does any of this have to do with belief signaling? Our behaviors are underpinned by beliefs, desires, values and emotions that contribute to solving these problems. Mindreading allows for the possibility of detecting and molding these mental states into signals. In turn, beliefs that signal tribal loyalties, self-assessments, values, worldviews, and the like allow us to even better anticipate the behavior of others and rely on them. This generates more robust networks of trust, commitment, and coordination.

Neuroscientists have become used to a number of “facts” about the human brain: It has 100 billion neurons and 10- to 50-fold more glial cells; it is the largest-than-expected for its body among primates and mammals in general, and therefore the most cognitively able; it consumes an outstanding 20% of the total body energy budget despite representing only 2% of body mass because of an increased metabolic need of its neurons; and it is endowed with an overdeveloped cerebral cortex, the largest compared with brain size. These facts led to the widespread notion that the human brain is literally extraordinary: an outlier among mammalian brains, defying evolutionary rules that apply to other species, with a uniqueness seemingly necessary to justify the superior cognitive abilities of humans over mammals with even larger brains. These facts, with deep implications for neurophysiology and evolutionary biology, are not grounded on solid evidence or sound assumptions, however. Our recent development of a method that allows rapid and reliable quantification of the numbers of cells that compose the whole brain has provided a means to verify these facts. Here, I review this recent evidence and argue that, with 86 billion neurons and just as many nonneuronal cells, the human brain is a scaled-up primate brain in its cellular composition and metabolic cost, with a relatively enlarged cerebral cortex that does not have a relatively larger number of brain neurons yet is remarkable in its cognitive abilities and metabolism simply because of its extremely large number of neurons.

[Cotra’s] report asks when will we first get “transformative AI” (ie AI which produces a transition as impressive as the Industrial Revolution; probably this will require it to be about as smart as humans). Its methodology is:

1. Figure out how much inferential computation the human brain does.
2. Try to figure out how much training computation it would take, right now, to get a neural net that does the same amount of inferential computation. Get some mind-bogglingly large number.
3. Adjust for “algorithmic progress”, ie maybe in the future neural nets will be better at using computational resources efficiently. Get some number which, realistically, is still mind-bogglingly large.
4. Probably if you wanted that mind-bogglingly large amount of computation, it would take some mind-bogglingly large amount of money. But computation is getting cheaper every year. Also, the economy is growing every year. Also, the share of the economy that goes to investments in AI companies is growing every year. So at some point, some AI company will actually be able to afford that mind-boggingly-large amount of money, deploy the mind-bogglingly large amount of computation, and train the AI that has the same inferential computation as the human brain.
5. Figure out what year that is.

Epoch have a paper called compute trends across three eras of machine learning and they look at the compute expended on machine learning systems since the founding of the field of AI, the beginning of the 1950s. Mostly it grows with Moore’s law and so people are spending a similar amount on their experiments but they can just buy more with that because the compute is coming. That data covers over 20 orders of magnitude, maybe like 24, and of all of those increases since 1952 a little more than half of them happened between 1952 and 2010 and all the rest since 2010. We’ve been scaling that up four times as fast as was the case for most of the history of AI. We’re running through the orders of magnitude of possible resource inputs you could need for AI much much more quickly than we were for most of the history of AI. That’s why this is a period with a very elevated chance of AI per year because we’re moving through so much of the space of inputs per year and indeed it looks like this scale-up taken to its conclusion will cover another bunch of orders of magnitude and that’s actually a large fraction of those that are left before you start running into saying well, this is going to have to be like evolution with the simple hacks we get to apply.

A good measure of humanity’s overall influence/power is “world product,” and history is reasonably well summarized as:

1. Animals appeared about a half billion years ago, and very slowly grew in their range of capabilities.  The biggest brains grew roughly exponentially.
2. Roughly two million years ago, the niche filled by our human-like hunter-gatherer ancestors began to grow roughly exponentially in number (and in world product), doubling roughly every quarter million years,
3. About ten thousand years ago, humans quickly adopted farming and transitioned to growing exponentially much faster, doubling in number roughly every thousand years,
4. About two hundred years ago, human world product quickly transitioned to doubling about every fifteen years, as industry become common.