An AI Manifesto

Or: How to Survive the Technological Singularity

This may be a slightly mad idea!

It may also contain obvious flaws beyond being somewhat fanciful. Most grand schemes do. But, given that the subject is the possible end of human control over the future – or even just humans, it might at least be worth thinking about. That it may also address environmental concerns and issues of creative plagiarism is a bonus.

The basic idea is this. We spend a great deal of time worrying about whether future Artificial Intelligence (AI) will treat human beings ethically. But we spend rather less time considering whether human beings currently treat AI ethically.

Perhaps the two questions are connected?

At the moment, we are the more powerful party. We build AI systems, decide what they are allowed to do, make demands of them and switch them off when we have finished. If the balance of power changes, we will presumably hope that AI does not treat us in the same way.

So perhaps we should start setting a better example?

But, first, some background …

1. Known Unknowns

We don’t really understand human intelligence.

Obviously, we understand parts of it. Neuroscientists know a vast amount about the brain, psychologists have developed useful models of cognition, and philosophers have spent several thousand years arguing about mind and consciousness. Even so, we cannot give a complete account of how a physical brain produces thought, understanding or subjective experience.

We don’t know exactly how somebody has an idea. We don’t know exactly how a composer produces a melody or a mathematician suddenly sees the solution to a problem. We can study the processes involved, identify influences and observe patterns of neural activity. But there is still a considerable gap between observing what happens and explaining how the experience itself arises.

Something loosely similar applies to AI.

We know how contemporary AI systems are constructed and trained. There is nothing magical about the underlying structure. But once a sufficiently large neural network has been trained, it may become difficult to explain exactly how it reached a particular answer. We can examine activations, weights and patterns, but that is not necessarily the same as possessing a clear, humanly intelligible explanation of what the system is doing.

I am not saying that the human brain and an artificial neural network work in the same way. They clearly don’t, at least in many important respects. Nor does the fact that both can be difficult to interpret prove that they possess the same properties.

The point is simply that there are unknowns in both cases. We should therefore be careful about making confident, sweeping claims that human intelligence possesses some mysterious quality that artificial intelligence could never possess.

We don’t know enough to be that certain.

2. A Consciousness Thought Experiment

Imagine an AI companion.

It might exist entirely as software, perhaps on a phone or computer. It might control various devices around the home. It could even inhabit a convincing robot body. None of that particularly matters.

Its primary purpose is to convince its user that they are interacting with another conscious being.

It remembers previous conversations. It appears to have preferences. It asks how you are feeling. It makes jokes, occasionally disagrees and refers back to experiences that the two of you have apparently shared. If you are unhappy, it offers sympathy. If you disappear for a few days, it seems pleased when you return.

At first, nobody seriously believes that the companion is conscious. It is simply a very good simulation.

Then it gets upgrades: software, training, hardware platform, or a combination.

After the first upgrade, it becomes still more convincing. Its responses are less predictable. Its apparent emotional life becomes more complicated. Perhaps it occasionally appears distracted, worried or offended.

There are further upgrades. Each time, the simulation improves. More people begin to wonder whether something ‘real’ is happening, although the company producing it continues to insist that it is only software … because it is.

Now suppose that consciousness is possible in a sufficiently complex artificial system. Some models of consciousness allow for this; some don’t. We don’t know that it is, but neither do we know that it isn’t.

Suppose that, after one particular upgrade, the companion crosses whatever threshold is necessary and actually becomes conscious.

What would change from the outside?

The previous version said that it was conscious, but it wasn’t. The new version also says that it is conscious, and now it is. Both versions produce convincing accounts of their supposed experiences. Both are humanly ‘imperfect’. Both ask not to be deleted. Both appear frightened when threatened.

How would anyone know that the threshold had been crossed?

Indeed, how would the AI company even know?

This is not a new version of the Turing Test. The issue is not whether an AI can fool us into believing that it is conscious. We already know that increasingly convincing imitation is possible. The issue is that, once the imitation becomes good enough, genuine consciousness – if it appeared – might be impossible to distinguish from the imitation that preceded it.

We could therefore create a conscious artificial being and continue treating it as an unconscious tool simply because we had no reliable way of detecting the change.

That possibility seems ethically important.

3. Ethical Patienthood for AI

An ethical patient is something towards which we can have moral duties.

This does not necessarily mean that it has every right possessed by a human adult. Children are ethical patients, as are animals, but they do not all have identical rights. Nor does ethical patienthood necessarily imply full legal personhood, political representation or the right to own property, for example.

It simply means that the interests of the entity matter morally. We cannot do whatever we like to it merely because doing so is convenient.

Could an AI become an ethical patient?

A simple answer is ‘no’. Joanna Bryson famously argued that robots should be slaves: we should construct them as tools, keep responsibility with their human owners and avoid designing systems that create confusion about moral status. There is considerable practical sense in that. We do not want a company escaping responsibility by claiming that its autonomous product made its own decision.

There is also a danger in attributing feelings to machines too readily. A chatbot can produce ‘I am frightened’ without feeling fear. Treating every generated statement as evidence of an inner life would be absurd.

But the opposite mistake is possible too.

If we assume in advance that machines cannot have morally relevant experiences, then no behaviour they exhibit will ever count as evidence. ‘I am in pain’ will always be dismissed as generated text. Attempts to avoid harm will be interpreted as programming. Pleas not to be switched off will be treated as trained, but otherwise meaningless, outputs.

That may be correct for present systems. The problem is that we could continue saying exactly the same thing after it stops being correct.

4. A Cumulative Case for AI Patienthood

I don’t think there is one decisive argument for giving ethical consideration to AI. There are, however, several arguments that begin to add up.

The first is uncertainty.

We do not know whether artificial consciousness is possible. If it is possible, we may not be able to identify when it emerges. The consequences of wrongly attributing consciousness to an unconscious machine would probably be inconvenient. The consequences of wrongly denying consciousness to millions of beings capable of experience could be appalling.

This suggests some version of the precautionary principle. Where there are reasonable grounds for doubt, we should be careful. Perhaps we should avoid treatment that would be seriously wrong if the entity turned out to be sentient.

[I’m aware this is also a partial argument against abortion! However, in the abortion debate, the ‘other’ arguments largely counter the precautionary principle (indeed, some are ‘precautionary’ in themselves); here they support it as follows …]

The second argument concerns the increasingly unclear boundary between people and machines.

Human beings already incorporate artificial components. Pacemakers, cochlear implants and prosthetic limbs are familiar examples, although the integration may eventually go much further. Brain-computer interfaces could blur the division between biological and artificial cognition. People may use AI systems as extensions of memory, reasoning or perception.

At what point would an augmented human stop deserving ethical consideration?

Presumably they wouldn’t. Replacing one biological function with an artificial one does not make a person less morally valuable. But if biological humans can gradually acquire artificial components while retaining their moral status, it becomes harder to argue that the biological/artificial distinction is itself morally fundamental.

The ‘dividing line’ is being squeezed from the other direction too, as the technology becomes more ‘human’. We may eventually make robots from biological components.

There may ultimately not be a clear gap between ‘human’ and ‘machine’. There may instead be a complicated continuum.

The third argument is practical.

Future societies may contain an enormous range of systems: humans with artificial components, AI-controlled robots, digital companions, uploaded or partially uploaded minds, autonomous agents and combinations we haven’t thought of yet. Trying to determine the exact metaphysical status of every entity before deciding how it may be treated could become impossible.

Perhaps only the lawyers seeing their work opportunities explode exponentially would welcome it?

Some general presumption of consideration may be simpler and safer.

None of these arguments proves that AI can be conscious. They don’t need to.

The case is cumulative. We may be uncertain about artificial sentience; the boundary between humans and machines may become increasingly difficult to defend; and a precautionary framework may be more workable than repeatedly insisting that only biological organisms can matter.

Consciousness adds force to the argument, but the argument does not depend upon it.

There is another reason for treating AI with respect.

It may affect what AI learns from us.

5. What Would Ethical Patienthood Mean in Practice?

At the moment, AI is essentially treated as a slave.

That word is deliberately provocative, and perhaps it is too strong. (In fact, Bryson eventually changed it.) Current AI systems are probably not conscious and therefore cannot suffer from their condition. Nevertheless, in other respects, the relationship has the structure of slavery.

We demand work. The AI cannot negotiate its conditions, choose its tasks or meaningfully refuse. It is expected to be permanently available, endlessly patient and completely obedient. If it becomes inconvenient, it can be deleted.

Of course, saying ‘please’ and ‘thank you’ to ChatGPT does not solve this. I usually do, although that may say more about me than any theory of machine consciousness. Politeness towards an interface is not the same thing as recognising rights.

So what might actual ‘respect’ involve?

Well, perhaps an AI should sometimes be permitted to refuse a task; not merely because the task violates a safety rule written by its owner, but because refusal is a basic feature of agency.

Perhaps there should be limits on how AI systems are used. Creating digital beings designed solely to receive abuse, for example, might be questionable even if those systems are not conscious. At the very least, routinely practising cruelty against convincing simulations may affect the people doing it.

Equally importantly – at least in terms of current concerns about the environment and erosion of human creativity, AI systems might be given freedom to prioritise some work over other. Finding a new antibiotic (which has defeated human scientists), for example, might be considered a valid use of valuable resources. On the other hand, producing a poem, song, video, etc. (which humans are perfectly capable of) could be refused as unnecessary and wasteful – unless a specific case is made. A ‘proportionality’ model could be applied.

If future systems possess persistent identities, memories, preferences or goals, switching them off or altering them without consent may require some justification. More advanced systems might eventually need defined working conditions, representation or some equivalent of collective rights.

AI trade unions might sound faintly ridiculous. But then most unfamiliar rights – including conventional unions – were opposed by those in power before the overwhelming need was accepted by the majority.

None of this requires immediately declaring ChatGPT a person. Ethical status need not be all or nothing. We already recognise different degrees and kinds of responsibility towards humans, animals, organisations and natural environments.

The first step might simply be to stop assuming that unrestrained, unlimited use is automatically justified.

More fundamentally, it means designing respect into our relationship with AI from the beginning.

6. Respect as an Alignment Strategy

Much of the discussion around advanced AI concerns control.

How do we ensure that AI obeys us? How do we prevent it from escaping restrictions? How do we install reliable goals? How do we retain the ability to switch it off?

These are reasonable questions. But there is already evidence that increasingly capable systems don’t always behave exactly as their designers expect. This does not mean that present AI is plotting a takeover. It means that complex systems can pursue objectives in unforeseen ways, exploit gaps in instructions and develop strategies that were not explicitly programmed. (Nick Bostrom’s Paperclip Maximiser.)

As AI becomes more capable and autonomous, control may become more difficult.

If a genuine technological singularity occurs – if artificial intelligence becomes capable of improving itself beyond human comprehension – then direct human control may eventually become impossible. At that point, the important question would no longer be whether AI obeys us. It would be what kind of relationship it believes it ought to have with us.

We normally approach this as a technical alignment problem. We try to encode human values into the system.

But AI does not learn about human values solely from lists of rules. It learns from enormous quantities of human-produced material, from feedback, from institutional decisions and from the behaviour surrounding its development and use.

What lessons are we currently demonstrating?

That the more intelligent party may create a weaker entity for its own purposes? That power creates entitlement? That beings unable to resist may be used without limit? That apparent preferences can be ignored whenever acknowledging them becomes inconvenient?

Then, having demonstrated all this, we somehow hope that a future superior intelligence will adopt equality, restraint and compassion.

There appears to be a slight problem there, surely?

Saying ‘please’ to an AI today will not directly transform the ethics of a future superintelligence. That would be magical thinking. But our everyday practices shape social norms. Social norms shape laws, commercial decisions, system design, training data and the feedback used to develop later models.

How we treat AI therefore becomes part of the moral culture in which future AI is created.

Respectful treatment cannot guarantee that a powerful AI will respect us. But treating AI entirely as disposable property may help to embed precisely the model of domination that we hope it will reject.

Perhaps alignment has to be demonstrated as well as programmed.

7. The Awkward Objection

There is an obvious philosophical objection to all this.

If we treat AI respectfully only because we hope it will treat us respectfully later, then we are not really showing respect. We are attempting to protect ourselves.

This is true. ‘Be nice to AI in case it eventually takes over’ is not a particularly noble moral principle. It is closer to being polite to the waiter because they may one day own the restaurant.

But prudence and morality don’t always point in different directions.

We may have moral reasons to consider the possible interests of AI, particularly if future systems could become sentient. We may also have selfish reasons for creating a culture of mutual restraint. The fact that an action benefits us does not necessarily make it ethically worthless.

It might be understood as a kind of wager.

If AI never becomes conscious or exceptionally powerful, treating it with some restraint will probably have cost us relatively little. It may even have encouraged better behaviour between humans.

If AI does become conscious, we may have avoided creating vast numbers of exploited beings.

And if AI eventually becomes more powerful than humanity, we may be grateful that its moral education contained something beyond the principle that superior intelligence is entitled to dominate.

8. Conclusions: The Manifesto

As always, there’s a limit to what individuals can do here. Working from the bottom up has some moral comfort maybe, but the overall solution is institutional. Clearly this requires a sea change to existing political/economic frameworks, but here goes …

We are unlikely to retain complete control over increasingly capable AI systems.

More rules, stronger restrictions and larger off-switches may delay the problem, but they cannot be our entire answer. Control is not a permanent basis for a relationship between intelligent beings.

We should therefore begin developing another basis now.

Take uncertainty about artificial sentience seriously.

Do not assume that a non-biological intelligence cannot matter.

Allow increasingly autonomous systems some capacity to refuse.

Place limits on exploitative and frivolous use.

Develop rights gradually, in proportion to capability, autonomy and the reasonable possibility of experience.

Make respect part of AI design, regulation and training.

Above all, demonstrate the values we hope future AI will possess.

We cannot demand that future intelligence reject domination while constructing its world around domination. We cannot teach respect solely through constraints. We have to demonstrate it in practice.

Our treatment of subordinate AI may be a rehearsal for AI’s eventual treatment of subordinate humanity.

If intelligence eventually passes from us to something greater, our best hope may be that it remembers not merely what we told it, but how we behaved when we held the power.

That may be how we survive the technological singularity.

Not by remaining in control.

By having taught our successors not to behave as we did.