Excellent topic, Paul. Thinking about superintelligence is difficult. When a human attempts to envision an entity that is smarter than a human it almost seems paradoxical. Yet, I think it is possible.
Here are three types of existing entities that sometimes display superintelligence (and occasionally subhuman intelligence):
(1) Group mind
(2) Corporation (A type of group mind)
(3) Market
Your point about the importance of nested abstractions is on the right track, I think. Mathematics is a domain which has some of the most deeply nested abstractions. There are theorems and proofs in mathematics that most humans cannot understand even with intensive study.
One of the remarkable accomplishments of mathematics is the proof of the classification of finite simple groups.
[Begin excerpt from Wikipedia]
The proof consists of tens of thousands of pages in several hundred journal articles written by about 100 authors, published mostly between 1955 and 2004.
[End excerpt from Wikipedia]
I would argue that the mathematicians who created this proof were acting together as a group mind. This group mind was coordinated via text written in journals. This group mind may be viewed as an entity with superhuman intelligence.
Mark Zuckerberg was asked about superhuman intelligence during a recent interview with Lex Fridman on YouTube. He suggested that corporations and markets are able to display superhuman intelligence on occasion:
[Begin excerpt from Mark Zuckerberg]
We do seem to have organizations and structures in the world that exhibit greater than human intelligence already. So, one example is a company. You know, it acts as an entity. It has, you know, a singular brand. Um, obviously it's a collection of people.
But I certainly hope that, you know, Meta with tens of thousands of people make smarter decisions than one person, but I think that that would be pretty bad if it didn't.
[End excerpt from Mark Zuckerberg]
[Begin excerpt from Mark Zuckerberg]
Think about something like the stock market . . . It takes inputs. It's a distributed system. It's like the cybernetic organism that, you know, probably millions of people around the world are basically voting every day by choosing what to invest in.
But it's basically this, this organism or structure that is smarter than any individual that we use to allocate capital as efficiently as possible around the world.
[End excerpt from Mark Zuckerberg]
Interestingly, some current strategies for increasing the capabilities of LLMs (large language models) involve generating multiple answers from the model and trying to combine the inputs via voting or some other mechanism. This might be viewed as a crude bootstrap mechanism that is analogous to a group mind.
Future AI systems might use multiple agents with internal auction markets to allocate computational cycles and computer memory.
