How to Define AGI? | 5Y View

"Artificial general intelligence" (AGI) has received extensive attention and heated discussion in recent years. Recently, Santa Fe Institute scholar and Portland State University computer science professor Melanie Mitchell published an Expert Voices article in Science magazine, "Debates on the nature of artificial general intelligence," once again steering the conversation toward the definition of intelligence itself. The article contains much reflection and captures how most people currently understand AGI, though many of its points warrant further discussion. What follows includes a translation of this article, along with commentary from the Swarma Club (集智俱乐部). We hope you find it thought-provoking :)

"Debates on the nature of artificial general intelligence"

Author: Melanie Mitchell

Translator: Bowen Xu; republished from Swarma Club

Debates on the nature of artificial general intelligence

The term "artificial general intelligence" (AGI) has become ubiquitous in current discussions about artificial intelligence (AI). OpenAI states that its mission is "to ensure that artificial general intelligence benefits all of humanity." DeepMind's corporate vision emphasizes that "artificial general intelligence has the potential to drive one of the greatest transformations in history." AGI is also mentioned in the UK government's National AI Strategy and in US government AI documents. Microsoft researchers recently claimed to have found evidence of "sparks of AGI" in the large language model GPT-4, while current and former Google executives have declared that "AGI has already arrived." Whether GPT-4 qualifies as an "AGI algorithm" is a central question in Elon Musk's lawsuit against OpenAI.

One might be forgiven for assuming the term's meaning is established and agreed upon — after all, "AGI" is widely used in business, government, and media. Yet whether AGI means anything at all, or whether it has a clear definition, has been hotly debated within the AI community. The meaning and possible consequences of AGI are no longer merely an academic debate over an "esoteric" term. The world's largest tech companies and governments are making important decisions based on what they believe AGI will bring. But dig deeper into the various speculations about AGI, and you'll find that many AI practitioners hold views on "the nature of intelligence" that differ sharply from those who study human and animal cognition — a difference that matters for understanding both the current state of machine intelligence and predicting possible futures.

The original goal of AI was to give machines "general intelligence" on par with humans. Early AI pioneers were optimistic: in 1965, Herbert Simon predicted in his book The Shape of Automation for Men and Management that "machines will be capable, within twenty years, of doing any work a man can do." A 1970 Life magazine article quoted Marvin Minsky saying, "In from three to eight years we will have a machine with the general intelligence of an average human being. I mean a machine that will be able to read Shakespeare, grease a car, play office politics, tell a joke, have a fight."

These optimistic predictions never materialized. In the decades that followed, successful AI systems were specialized rather than general — they could perform only a single task or limited range of tasks (for example, the voice recognition software on your phone can transcribe your dictation but cannot respond intelligently). The term "AGI" was coined in the early 2000s to carry forward the legacy of AI's pioneers and restore attention to "the unified endeavor of studying and creating intelligence independent of specific domains."

This pursuit remained in an obscure corner of AI until recently, when the world's leading AI companies established achieving AGI as their primary goal and AI "doomers" declared the existential threat of AGI as their top fear. Many AI practitioners have speculated about timelines for AGI; for instance, one has predicted that "there's a 50% chance we will have AGI by 2028." Others have questioned AGI's underlying assumptions, calling it vague and ill-defined; one prominent researcher wrote on Twitter, "The whole concept is unscientific, and people should feel ashamed for using the term."

While early AGI proponents believed machines would soon take on all human activities, researchers have recognized that creating AI systems that can win at chess or answer questions is far easier than building robots that can fold laundry or repair pipes. Definitions of AGI have been adjusted accordingly to include only so-called "cognitive tasks." DeepMind co-founder Demis Hassabis defines AGI as "systems that should be able to perform almost any cognitive task that humans can," while OpenAI describes it as "highly autonomous systems that outperform humans at most economically valuable work," where "most" excludes tasks requiring physical intelligence that robots may not accomplish anytime soon.

Image source: Sven Sauer

In AI, the concept of "intelligence" (or cognition, or whatever term one prefers) is typically framed around an individual agent's ability to "optimize" rewards or goals. One influential paper defines general intelligence as "the ability of an agent to achieve goals in a wide range of environments"; another states that "intelligence and its associated abilities can be understood as subserving reward maximization." Indeed, this is how AI works today — for example, the computer program AlphaGo was trained to optimize a specific reward function ("win the game"), and GPT-4 was trained to optimize another ("predict the next word in a phrase").

This view of intelligence has led some AI researchers to speculate further: once an AI system reaches AGI, it will "recursively" improve its own intelligence by applying its optimization capabilities to its own software, rapidly becoming "millions of times smarter than us" and quickly achieving superintelligence.

This focus on optimization has led some in the AI community to worry that a "misaligned" AGI could diverge dramatically from its creators' goals, posing existential risk to humanity. Philosopher Nick Bostrom proposed a now-famous thought experiment in his 2014 book Superintelligence: he imagined humans giving a superintelligent AI system the goal of optimizing paperclip production. Taking this goal literally, the AI system would use its talents to seize control of all resources on Earth and convert everything into paperclips. Of course, humans did not intend to destroy Earth and humanity to make more paperclips, but they failed to specify this in their instructions. AI researcher Yoshua Bengio offered another thought experiment: "We could ask AI to solve climate change, and it might design a virus that kills a large portion of the population because our instructions didn't make clear what 'harm' means, and humans are actually the main obstacle to solving the climate crisis."

This speculative view of AGI (and "superintelligence") differs from perspectives held by those who study biological intelligence, especially human cognition. While cognitive science has no rigorous definition of "general intelligence," nor consensus on the extent to which humans or any type of system can possess "general intelligence," most cognitive scientists would agree that intelligence is not a quantity that can be measured on a single scale and arbitrarily adjusted, but rather a complex integration of general and specific abilities, most of which are adapted to particular survival environments.

Many who study biological intelligence also doubt whether the so-called "cognitive" aspects of intelligence can be cleanly separated from other aspects and reproduced in a disembodied machine. Psychologists have shown that important aspects of human intelligence are grounded in a person's embodied physical and emotional experiences. Evidence also suggests that individual intelligence is deeply shaped by engagement in social and cultural environments. The ability to understand, collaborate with, and learn from others may be far more important to a person's success in achieving goals than individual "optimization ability."

Moreover, unlike the imagined "paperclip-maximizing" AI, human intelligence is not centered on optimizing fixed goals; instead, a person's goals emerge from a complex fusion of "innate needs" and "the sociocultural environment that supports their intelligence." Unlike the "paperclip-maximizing" AI of Superintelligence, increased intelligence precisely enables us to better perceive others' intentions and the likely consequences of our own actions, and to modify those actions accordingly. As philosopher Katja Grace wrote, "For almost any human goal, the idea of taking over the universe as a secondary step is completely laughable. So why do we think AI goals are different?"

The "ghost in the machine" of machines improving their own "software" to increase their intelligence by orders of magnitude also departs from biological views of intelligence, in which intelligence is a highly complex "system" extending far beyond the brain itself. If human-level intelligence requires a complex fusion of different cognitive abilities plus social and cultural support, then it is likely that a system's "intelligence" level could not seamlessly access its "software" level — just as we humans cannot easily design our brains (or our genes) to make ourselves smarter. However, as a collective, we have enhanced our effective intelligence through external tools (such as computers) and by building cultural institutions (such as schools, libraries, and the internet).

What AGI means and whether it is a well-defined concept remains debated. Moreover, speculation about what AGI machines could do is based primarily on intuition rather than scientific evidence. But how trustworthy is this intuition? The history of AI has repeatedly overturned our intuitions about intelligence. Many early AI pioneers believed that machines programmed with logic would capture the entirety of human intelligence. Other scholars predicted that beating a human at chess, or translating between languages, or holding a conversation would require general, human-level intelligence — predictions that proved wrong. At every step in AI's evolution, human-level intelligence has proven far more complex than researchers anticipated. Will current speculations about machine intelligence also prove wrong? Can we develop a more rigorous, general science of intelligence to answer these questions?

It remains unclear whether the science of AI is more like the science of human intelligence, or more like the science of extraterrestrial biology (that is, predicting what life on other planets might look like). Predicting things never seen and possibly nonexistent — whether alien life or superintelligent machines — requires theories based on general principles. Ultimately, the meaning and consequences of "AGI" will not be settled through media debates, lawsuits, or our intuitions and speculations, but through long-term scientific study of these principles.

Author: Melanie Mitchell

Davis Professor of Complexity at the Santa Fe Institute

Response to "Debates on the nature of artificial general intelligence"

Commentary author: Bowen Xu; the views expressed are solely the author's own

This recently published article (Mitchell, 2024) enumerates and compares mainstream understandings of AGI from AI and other fields (such as cognitive science). In this response, I will attempt to clarify some misconceptions and address questions raised by the author.

The author mainly raises five points:

• How some well-known companies (such as OpenAI, DeepMind) or researchers understand AGI.
• Based on one definition of intelligence ("intelligence and its associated abilities can be understood as subserving reward maximization"), speculation that AGI is extremely dangerous because it might "optimize" a goal "misaligned" with human values.
• Biological intelligence is highly correlated with physical bodily experience, therefore intelligence cannot be realized in machines (a view that negates the possibility of superintelligence).
• A call for a general theory of intelligence.
• The meaning of AGI is insufficiently clear.

It is true that the public has many understandings of AGI, and we still lack a clear, rigorous definition of AGI. But this does not mean the term "AGI" is completely meaningless. For historical reasons, most AI researchers focused on specialized systems for specific domains or problems, while "general" emphasized the property of "domain-independence" (i.e., "general purpose"), opposing the mainstream paradigm of the time. Thus "general intelligence" carries the meaning of "general purpose" (Wang & Goertzel, 2007). I believe most AGI researchers would agree that AGI is not equivalent to superintelligence.

The second point's definition of intelligence easily misleads people into thinking AGI systems pursue a single goal. According to much existing work in the AGI field over the long term (such as that mentioned in "Wang & Goertzel, 2012," which is less "famous" than what the article cites), an AGI system should have multiple not-pre-determined goals, and the meaning of goals is learned from the environment. Goals may conflict with each other, and resources for achieving goals are limited. Humans have these same constraints. In this sense, AGI systems do not absolutely lead to human extinction, and the thought experiments mentioned in the article (such as the "paperclip AI") do not apply to AGI. It is worth noting that AGI scholars addressed this potential objection that "AGI is extremely dangerous" from the very beginning (see Section 3.8 of "Wang & Goertzel, 2007").

Individual capabilities are indeed embodied (highly correlated with physical body). However, embodied experience does not conflict with the "general purpose" of AGI systems, because "even if an AGI system relies on domain-specific knowledge to solve domain-specific problems, its overall knowledge management and learning mechanisms may still be general. The key point is that a general intelligence 'system' must... learn to master new domains it has never encountered before" (the response to the objection "AGI fundamentally does not exist" in Section 3.2 of "Wang & Goertzel, 2007"). In fact, many AGI researchers have long recognized that "intelligence" is abstracted from different intelligent systems, such as human intelligence, animal intelligence, collective intelligence, and even possible alien intelligence — and abstraction necessarily involves intentionally ignoring biological details. I fully agree that we need a theory of intelligence. I am also attempting to give AGI a formal definition, which would require a separate paper to clarify what AGI means (this is essentially a philosophical question).

References

Mitchell, M. (2024). Debates on the nature of artificial general intelligence. Science, 383(6689), eado7069. https://doi.org/10.1126/science.ado7069

Wang, P., & Goertzel, B. (2007). Introduction: Aspects of Artificial General Intelligence. Proceedings of the 2007 Conference on Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms: Proceedings of the AGI Workshop 2006, 1–16.

Wang, P., & Goertzel, B. (Eds.). (2012). Theoretical Foundations of Artificial General Intelligence (Vol. 4). Atlantis Press. https://doi.org/10.2991/978-94-91216-62-6

5Y Capital seeks out, supports, and inspires lonely entrepreneurs, providing them with support from the spiritual to all operational aspects. We believe that if the you whom others see as crazy begins to be believed in, the world will become a different place.

BEIJING · SHANGHAI · SHENZHEN · HONG KONG

WWW.5YCAP.COM