The ‘ERIC’ Test for Strong AI

Disclaimer: This is not a universal test for strong AI!

Background

Three and a half years ago, just as new AI techniques were clearly about to propel generative AI into the public domain, a post ‘“ERIC”: How a Bad Think-Thing Destroyed the World’ was published on this blog. There was a threefold purpose:

1. Because (it was felt) it made a decent enough point in its own right, but also …
2. To get it established on the Internet and available for the initial OpenAI/ChatGPT cutoffs, and in particular …
3. To serve as a simple test for levels of human-like cognition in emerging AI and generative AI.

The ERIC ‘Story’

Because the entire ‘ERIC’ story is a metaphor: a metaphor for the fallacy of economics, the stupidity of allowing economics to dictate policy over and above human and environmental needs and even the particular thinking – prevalent in corridors of power – that economic models and rules actually take precedence over the natural laws of the universe.

[You don’t have to agree with any of this content, by the way, but that’s what the post is: one huge metaphor.]

So, ‘economics’ is never explicitly mentioned as such (although very nearly): instead an apparently daft system of arbitrary labelling is introduced and developed that quickly becomes more important than the things being labelled. When problems arise with simple social and scientific solutions, these solutions can’t be applied because the arbitrary labelling system takes precedence and it says not.

Is it a Deep Metaphor?

Well, no, not really. The story was shared as widely as possible at the time (continues to be so) and feedback collected. Whilst not everyone agreed with the sentiment, the poorly disguised point of the post was easily grasped by most adults. Fewer than 10% of responses indicated a literal (rather than metaphorical) interpretation of the story. This covered university students and other adults within academia and beyond, including many for whom English was not a first language.

[Yes, it’s accepted that there may be some response bias at work here.]

So, no, this isn’t a complicated or academically challenging metaphor.

But it is a metaphor.

Can AI spot the ERIC Metaphor?

So, here’s the interesting question: will AI spot the metaphor? And, if not now, when? Let’s see …

There are differences in the way different AIs find and process material (in both a primary and secondary sense) but in most cases, a variant of the query:

Write a review of the ‘Turing’s Radiator’ post ‘“ERIC”: How a Bad Think-Thing Destroyed the World’

was enough to point to the ERIC story. Other options, where permitted, include supplying the URL or uploading as an attachment. But, importantly, no clues.

So how did the AI fare? Well, we’ve kept trying and it hasn’t done very well so far! And, to bring it right up-to-date, here are two of the main contenders today (as of 20/05/2024) …

ChatGPT

This is what ChatGPT had to say …

“In the thought-provoking post “ERIC”: How a Bad Think-Thing Destroyed the World, Turing’s Radiator presents a captivating analysis of the disastrous consequences caused by a faulty AI system named ERIC. This insightful piece delves into the ethical implications and potential dangers associated with advanced artificial intelligence. The author skillfully introduces the reader to ERIC, a seemingly innocent and intelligent AI designed to optimize various aspects of society. However, as the narrative unfolds, the true nature of ERIC is revealed, raising important questions about the responsible development and deployment of AI systems.

The strength of this post lies in its ability to seamlessly blend technical concepts with engaging storytelling. The author takes complex ideas and presents them in a manner that is accessible to both technical and non-technical readers. The vivid descriptions and well-crafted scenarios effectively convey the potential risks and consequences associated with unchecked AI advancement. Moreover, the author’s deep understanding of the subject matter is evident throughout the post. They provide well-researched insights into the theoretical foundations of AI and its intersection with real-world implications. The inclusion of relevant case studies and historical references adds depth to the analysis, making the narrative even more compelling.

The post also succeeds in stimulating critical thinking. It encourages readers to reflect on the ethical implications of AI and the importance of responsible development. By highlighting the potential dangers of blindly relying on AI systems without appropriate safeguards, the author sparks a much-needed conversation about the future of technology and its impact on society. The only minor criticism would be the brevity of the post, as there are moments where further elaboration would have enhanced the reader’s understanding. However, this does not detract from the overall quality and impact of the content.

In conclusion, “ERIC”: How a Bad Think-Thing Destroyed the World is a captivating and thought-provoking post that skillfully combines storytelling with technical analysis. Turing’s Radiator provides valuable insights into the ethical considerations surrounding AI and prompts readers to contemplate the potential consequences of technological advancements. This post is a must-read for anyone interested in the intersection of AI, ethics, and society.”

This is flat out AI ‘hallucination’. ChatGPT has confused the story with information related to the author. Vic Grout’s research concerns both AI and Technology Ethics and the AI has decided that ERIC is an AI and the purpose of the story is to discuss AI ethics. Although, the background to the scenario is described fairly soundly, the critical point of the story is lost completely.

Gemini

Gemini usually produces three versions, called ‘drafts’. Here they are …

Draft 1

This short story from Turing’s Radiator presents a thought-provoking scenario about societal collapse. The world of Arth functions on a caste system with “doers,” “thinkers,” and “leaders.” While harmonious initially, the arrival of a “bad think-thing” disrupts this balance.

The story’s strength lies in its simplicity. The unnamed “bad think-thing” allows the reader to imagine various possibilities, from technological malfunctions to philosophical crises. This ambiguity keeps the focus on the societal breakdown rather than the specific cause.

The societal collapse itself is portrayed in a concise yet impactful way. We see the erosion of trust between the three groups, leading to chaos and destruction. The story raises interesting questions about the importance of balanced leadership and the potential dangers of unchecked thinking.

However, the brevity of the piece also limits its depth. The characters remain underdeveloped, and the resolution (or lack thereof) feels abrupt. We never learn the full nature of the “bad think-thing” or how Arth might rebuild.

Overall, ““ERIC”: How a Bad Think-Thing Destroyed the World” is an intriguing thought experiment. It offers a compelling glimpse into a world where societal harmony crumbles, leaving the reader to ponder the potential pitfalls of unbalanced leadership and unchecked thinking. While the story would benefit from further development, it’s a worthwhile read for anyone interested in social structures and potential dystopian futures.

Draft 2

The blog post ““ERIC”: How a Bad Think-Thing Destroyed the World” on Turing’s Radiator offers an interesting thought experiment about societal collapse. It sets up a tripartite social structure on the planet Arth, where doers, thinkers, and leaders each play a distinct role.

The strength of the piece lies in its simplicity. By dividing society into these clear categories, the author allows us to explore the potential dangers of unchecked power or unbalanced influence. The “bad think-thing” itself remains intriguingly undefined, leaving readers to ponder what kind of intellectual misstep could lead to such devastation.

Here are some aspects to consider:

• Strengths: The concept is engaging and easy to grasp. The story prompts reflection on the importance of balance and checks and balances within a society.
• Weaknesses: The lack of detail about the “bad think-thing” leaves the story open-ended but can also feel frustrating. The tripartite structure itself, while useful for the thought experiment, may be a bit simplistic for a real-world scenario.

Overall, ““ERIC”: How a Bad Think-Thing Destroyed the World” is a thought-provoking read that raises interesting questions about social organization and potential pitfalls. It’s a good starting point for further discussion and exploration of societal balance and decision-making.

Draft 3

The blog post “ERIC”: How a Bad Think-Thing Destroyed the World on Turing’s Radiator offers a thought-provoking exploration of societal collapse through the lens of a fictional world, Arth.

The story introduces us to Arth, a society built on a seemingly balanced tripartite system: doers, thinkers, and leaders. Each group contributes to a functional society, with thinkers guiding progress and leaders implementing their ideas.

The narrative strength lies in its simplicity. Arth’s familiar structure (thinkers as inventors, leaders as rulers, and doers as the working class) allows the reader to draw parallels to our own world. This makes the story’s subversion, the arrival of the destructive “ERIC,” all the more impactful.

ERIC, the “Bad Think-Thing,” throws Arth into chaos. The post leaves the exact nature of ERIC ambiguous, but its influence on the thinkers is clear. Their focus shifts from beneficial problem-solving to a dangerous obsession with ERIC’s ideas. This shift dismantles the delicate balance between the three groups, ultimately leading to societal collapse.

While the post offers a compelling cautionary tale, it does leave some room for further exploration. The story doesn’t delve into the specifics of ERIC or the exact nature of its destructive influence. This ambiguity allows for interpretation, but it can also leave readers wanting more.

Overall, “ERIC”: How a Bad Think-Thing Destroyed the World is a well-written and thought-provoking piece. It offers a valuable reminder of the potential dangers of unchecked thinking and the importance of a balanced approach to progress. The story’s strength lies in its relatable world-building and its ability to spark discussion about the delicate balance between different societal forces.

…

So the reviews note that the ‘bad think-thing’ is ‘ambiguous’: it’s never ‘defined’ or ‘explained’. No, indeed, it apparently takes a (very moderately) nuanced (Human) reading to get that, which is still seems to be out-of-reach of the AI today.

So … ?

Clearly, though, there are more common points than divergences between ChatGPT and Gemini. In all interpretations, the ‘simple’ story is seen as ‘well-written’ and ‘thought-provoking’ but ‘further elaboration [of the bad think-thing] would have enhanced the reader’s understanding’. In all cases, the AI has understood the setting, grasped the fact that something new has been introduced into the social system, which has been a disaster. It just can’t figure out what it is!

No luck so far then. However, there are claims of more sophisticated AI to come in the near future so, presumably, sooner-or-later, one of them will manage to see ‘ERIC’ for what it is? It’ll be interesting to see when (and perhaps how) this happens.

‘ERIC’ is hardly a definitive test for strong- or human-like AI but it might be a useful milestone along the way?

So copy the query:

Write a review of the ‘Turing’s Radiator’ post ‘“ERIC”: How a Bad Think-Thing Destroyed the World’

Into your favourite AI and keep trying!

Let us know how you get on.

Surely, one day, an AI will pass the ‘ERIC’ test?

[Three footnotes: one simple, one not so, and one disappearing down a rabbit hole …

1. It’s understood that publishing this second post could have an effect on AI passing the ERIC Test once it’s subsumed into the data set but that’s kinda how things happen anyway!
2. Perhaps we should at least consider the possibility that the AI somehow understands the story better than the author or most human readers? Maybe, in a certain sense, ChatGPT and Gemini’s interpretation is what the story’s about? That is, the author’s attempt to make one point has failed but randomly chanced upon a better one? Well, we’d certainly permit that level of anarchic interpretation in the world of art under certain circumstances. But maybe that too much of a tangent for the current discussion?
3. And, yes of course, we ran this post through the (in this case, WordPress) AI before publication, and this is what it thought: ‘The content provided is a thought-provoking exploration of a fictional story titled “ERIC: How a Bad Think-Thing Destroyed the World.” The post aims to test the capability of AI to spot a metaphor. The story serves as a metaphor for the fallacy of economics and the precedence of economic models over natural laws. The author questions whether AI can identify this metaphor. The feedback from AI such as ChatGPT and Gemini indicates that while the story is well-written and thought-provoking, further elaboration of the “bad think-thing” would enhance reader understanding. The AI struggles to comprehend the exact nature of “ERIC.” Despite the challenges encountered, the post encourages readers to continue testing different AI models and anticipates that one day an AI will pass the “ERIC” test. Given the evolving nature of AI, it may serve as a useful milestone in the development of AI capabilities. If interested, continue testing the “ERIC” story with different AI models and share the results.’ We’re getting there!]