In many cases, the benefits of mindreading will have nothing to do with signaling, but the following are common situations in which signaling benefits (to receiver and/or sender) can emerge:

Behavioral Prediction: predicting the behavior of another.

Behavioral Stabilization: regulating other people’s behavior so that it becomes stable and predictable.

Trust Problems: discerning whether to approach a person, trust them with your property, expect that they will reciprocate your altruism, etc.

Commitment Problems: getting others to commit to a future action or having a reason to commit yourself to a future action.

Coordination Problems: figuring out how to work with others to jointly accomplish a goal or to work in parallel.

The first two categories are quite broad and include the latter three as special instances: predicting and shaping the behavior of others is a way to achieve trust, commitment, and coordination. But what does any of this have to do with belief signaling? Our behaviors are underpinned by beliefs, desires, values and emotions that contribute to solving these problems. Mindreading allows for the possibility of detecting and molding these mental states into signals. In turn, beliefs that signal tribal loyalties, self-assessments, values, worldviews, and the like allow us to even better anticipate the behavior of others and rely on them. This generates more robust networks of trust, commitment, and coordination.

Neuroscientists have become used to a number of “facts” about the human brain: It has 100 billion neurons and 10- to 50-fold more glial cells; it is the largest-than-expected for its body among primates and mammals in general, and therefore the most cognitively able; it consumes an outstanding 20% of the total body energy budget despite representing only 2% of body mass because of an increased metabolic need of its neurons; and it is endowed with an overdeveloped cerebral cortex, the largest compared with brain size. These facts led to the widespread notion that the human brain is literally extraordinary: an outlier among mammalian brains, defying evolutionary rules that apply to other species, with a uniqueness seemingly necessary to justify the superior cognitive abilities of humans over mammals with even larger brains. These facts, with deep implications for neurophysiology and evolutionary biology, are not grounded on solid evidence or sound assumptions, however. Our recent development of a method that allows rapid and reliable quantification of the numbers of cells that compose the whole brain has provided a means to verify these facts. Here, I review this recent evidence and argue that, with 86 billion neurons and just as many nonneuronal cells, the human brain is a scaled-up primate brain in its cellular composition and metabolic cost, with a relatively enlarged cerebral cortex that does not have a relatively larger number of brain neurons yet is remarkable in its cognitive abilities and metabolism simply because of its extremely large number of neurons.

[Cotra’s] report asks when will we first get “transformative AI” (ie AI which produces a transition as impressive as the Industrial Revolution; probably this will require it to be about as smart as humans). Its methodology is:

1. Figure out how much inferential computation the human brain does.
2. Try to figure out how much training computation it would take, right now, to get a neural net that does the same amount of inferential computation. Get some mind-bogglingly large number.
3. Adjust for “algorithmic progress”, ie maybe in the future neural nets will be better at using computational resources efficiently. Get some number which, realistically, is still mind-bogglingly large.
4. Probably if you wanted that mind-bogglingly large amount of computation, it would take some mind-bogglingly large amount of money. But computation is getting cheaper every year. Also, the economy is growing every year. Also, the share of the economy that goes to investments in AI companies is growing every year. So at some point, some AI company will actually be able to afford that mind-boggingly-large amount of money, deploy the mind-bogglingly large amount of computation, and train the AI that has the same inferential computation as the human brain.
5. Figure out what year that is.

Epoch have a paper called compute trends across three eras of machine learning and they look at the compute expended on machine learning systems since the founding of the field of AI, the beginning of the 1950s. Mostly it grows with Moore’s law and so people are spending a similar amount on their experiments but they can just buy more with that because the compute is coming. That data covers over 20 orders of magnitude, maybe like 24, and of all of those increases since 1952 a little more than half of them happened between 1952 and 2010 and all the rest since 2010. We’ve been scaling that up four times as fast as was the case for most of the history of AI. We’re running through the orders of magnitude of possible resource inputs you could need for AI much much more quickly than we were for most of the history of AI. That’s why this is a period with a very elevated chance of AI per year because we’re moving through so much of the space of inputs per year and indeed it looks like this scale-up taken to its conclusion will cover another bunch of orders of magnitude and that’s actually a large fraction of those that are left before you start running into saying well, this is going to have to be like evolution with the simple hacks we get to apply.

A good measure of humanity’s overall influence/power is “world product,” and history is reasonably well summarized as:

1. Animals appeared about a half billion years ago, and very slowly grew in their range of capabilities.  The biggest brains grew roughly exponentially.
2. Roughly two million years ago, the niche filled by our human-like hunter-gatherer ancestors began to grow roughly exponentially in number (and in world product), doubling roughly every quarter million years,
3. About ten thousand years ago, humans quickly adopted farming and transitioned to growing exponentially much faster, doubling in number roughly every thousand years,
4. About two hundred years ago, human world product quickly transitioned to doubling about every fifteen years, as industry become common.

A theory of human cognitive evolution needs to integrate the biological and social-scientific perspectives on human nature. Niche construction and its partial transformation into bona fide inheritance is the key to this integration. Some of the apparatus of hominid social life has become part of inherited hominid developmental resources. Hominids do not just inherit genes: they inherit epistemic resources that scaffold the development of life skills that are characteristic of their parents and of their immediate group, and which quite often distinguish them phenotypically from other hominids. Thus niche construction is a mechanism that supports developmental flexibility: a child becomes a skilled hunter rather than a fisherman because he inherits this set of developmental resources. Human genes have become adapted to sharing the job of directing development with an array of other resources. Moreover, since these new developmental resources are made and incorporated into inheritance systems more quickly than new genetic resources, one effect os a potential acceleration of hominid evolution. Expanded inheritance can then act as a means both for the evolutionary fragmentation of hominid lineages and as a means by which evolutionary change is accelerated.

Early in hominid evolution, it’s very likely that biological inheritance was much as it now is in chimps. To a reasonable approximation, a chimp inherits only genes from its parents. Though social learning is important in chimp life, and there is some meme-like flow of information from mothers to children, that flow is diffuse and short-lived. There is no evidence of deep behavioral traditions in chimp life, nor of cumulative downstream niche construction. There is no sign that group selection is allowing cooperation to take off. There are certainly some limited forms of cooperation: males cooperate to defend territory against other groups of chimps, to hunt, and to form coalitions against other males. Females too form coalitions. But there is little evidence of effective suppression of free-riding and defection. In contrast, over time in hominid evolution:

1. Group selection became very important, and underwrote the evolution of a cooperation explosion, the effects of which include language, division of labor, and resource sharing.
2. Cooperation itself accentuates niche construction: it becomes more powerful; more downstream, and more like genetic inheritance.
3. As this transformation proceeds, elements of culture become elements of biology, as they become part of a developmental matrix which is transmitted from one generation to the next.
4. Once information transmission became reliable and precise, downstream niche construction became cumulative, and Tomasello’s Ratchet began to work. That Ratchet required both cognitive and social preconditions, but once these were met, the Ratchet began to turn. Different human groups became more markedly differentiated, for their phenotypes come to reflect not just their current environmental differences but also the differences in their lineages’ learning history.
5. The geographic expansion of the hominid range, the cumulative transformation of hominid lifeways, and the intensification of climatic variability select for flexible response.