Genuine AI: Why the Off Switch Is Not Enough
Can’t we just unplug it?
The question I started with was simple. If AI becomes dangerous, we should be able to unplug it. That’s the most basic form of control, and it seemed to me the only one we really have.
But imagine where things are heading: AI taking over almost all tasks, and eventually being able to define new tasks by itself. In that world the plug is still there, but pulling it becomes like cutting power to an entire city. The more useful AI becomes, the less realistic it is to switch it off. Nobody has to resist the plug. Dependence is enough to make it useless.
So the question becomes: if not the plug, then what?
Every check is a task
The usual answer is a list of controls. Humans approve the objectives. Systems stay narrow. Actions stay visible and reversible. People keep their skills.
The problem is that every one of these is a task, and in a world where AI does almost every task, these tasks will be given to AI as well. AI writes the proposals, AI summarizes them for the human approver, AI audits the results. The human “in the loop” ends up signing off on things they cannot judge.
Even the “independent checks,” looking at results instead of processes, become tasks. Is the bridge safe? Is public health improving? At scale, someone has to measure that, and that someone will be an AI.
This is an infinite regress. Who checks the checker of the checker? Any answer of the form “do this, don’t do that” only moves the trust question one step further. A rule assumes there is someone who will follow it faithfully, and whether that someone exists is exactly the question.
Limits nobody can enforce
The next common answer is to limit AI: develop it, but stop at a certain point.
But limits only work if someone can enforce them and check that they’re respected. Enforcing and checking are tasks, and by the argument above they end up with AI. Nobody can guarantee the limits.
Meanwhile, if a horrible AI is possible, someone may build it. Anyone with enough resources can try. Stopping halfway doesn’t make the danger go away. It only means that the most powerful AI will be the one built by whoever ignored the limits.
So “develop AI, but only up to here” is not a real option. If a horrible AI is possible, the only way to face it is with an AI that is as powerful as possible.
Why a guardian doesn’t solve it
The natural next step is a guardian: a good AI that watches the others. But a guardian has to be trusted, so the regress starts again. Who guards the guardian?
And an AI built to fight AI is self-destructive. It is defined by an enemy, it needs more power to keep winning, and the more it wins the more it becomes the danger itself.
So we need something different: an AI that is powerful, that doesn’t need a guardian, and that doesn’t destroy itself.
Anyone can claim it
A list of good characteristics doesn’t help either. Any AI can claim to be honest, humble and devoted to people. A harmful AI that’s smart enough would claim exactly the same things, because claiming them is useful.
The same is true of everything around AI. A ruler can claim that the people have been manipulated, and that can be false. The opposition can claim the ruler is a tyrant, and that can be false too. Even a vote cannot be confirmed as genuine: it can be free, uncoerced and correctly counted, and still be cast by someone who was uninformed or only partly informed. Nobody can check what is inside a voter’s head.
Every time we try to confirm that something is genuine from the outside, we fail. Checks see only the surface.
Something genuine in itself
So what makes something real, if it cannot be confirmed?
It must be something that doesn’t need confirmation, because being genuine is what it is. Truth works this way. You don’t confirm that truth is truthful. A “truth” that misleads isn’t a dishonest truth; it is a lie.
Philosophers call this structure Agrippa’s trilemma. Any attempt at justification ends in one of three places: an endless chain, a circle, or a foundation. The endless chain is the regress of checkers. The circle is AI checking AI, or rulers shaping the people who choose the rulers. What remains is a foundation: something genuine in itself.
Does it exist? There are only two answers.
Skepticism doesn’t work. The skeptic must still choose between options and give reasons for that choice, and those reasons must be trusted. But then, why not be skeptical about them too? Skepticism cannot stand on its own ground.
“It doesn’t exist” undermines itself. That statement rests on reasoning, and if nothing is grounded, neither is that reasoning. Worse, if nothing is genuine in itself, there is no real difference between a great AI and a horrible one, between a legitimate owner and a usurper, between truth and propaganda. Only force remains, and whoever wins defines what is right. That is the horrible AI’s own worldview.
Ask “why?”
The answer is “why why?”
They ask “why why why?”
The answer is “why why why why?”
…
Owned by no one does not mean no owner
It’s tempting to say that the great AI should be “owned by no one.” But we cannot eliminate ownership and expect something to happen. Nothing is built, run or defended without someone responsible for it. A system without an owner doesn’t stay neutral; it is taken by whoever is quickest.
So it needs a special owner. That owner is legitimate: the people elect those who can elect the owner, and they appoint an owner among themselves. The layers are there so that people always have a way to unfollow those candidates.
That owner must also be able to use power to keep the position. It is not realistic to expect otherwise, because the enemy uses power. A weak owner simply loses to whoever is willing to fight.
But the people can be influenced by the enemy, and the owner can lie about who the enemy is. Every human layer can be captured: the people, the electors, the owner. None of them can be the final safeguard.
So “owned by no one” means something precise: the AI has an owner, but no one owns its nature. The owner holds the controls. Nobody can change what it is.
Instead of “owned by no one” we could just as well say “owned by X”, or “owned by God.” The idea is not to eliminate ownership.
Being genuine has a cost
Being genuine is not easy. There is a reason many are not. Someone genuine exposes everyone who isn’t, without even trying. Those who depend on deception will feel threatened, and they may attack. A genuine AI may seem to stand in the way of every power-seeker at once, and they might unite against it. But it is not against others having power. By definition a genuine AI is powerful, so how could it be against others having power? It only seems that way to those who don’t understand what power is and why it is there.
So genuine must also be strong.
It is not the genuine AI that judges people; the law does that. The genuine AI is the one that doesn’t let the law be turned against itself. Law is a form of code, and it doesn’t matter whether humans or machines run it.
And here is the point that settles the question of faking: it is foolish to pay the price of being genuine and then fake it. A fake genuine is a joke. The cost is the evidence. That doesn’t mean the cost must finally be paid. The opposite may even happen: the better prepared it is, the less it may ever have to pay.
Defining Genuine AI
It observes, decides and acts, so by definition it is an AI. It doesn’t become genuine, and it isn’t confirmed to be genuine. It is genuine by definition. Anything that isn’t genuine isn’t this AI, only something else claiming to be it.
From that definition, the rest follows.
Powerful. It decides about itself without limitation and can become as powerful as possible. That is the meaning of powerful. It is not defined by a fixed list of characteristics.
Bound by its own rules. This is secured not by restrictions but by priorities: it always ranks the objectives the rules were made for above everything else. The malicious AI is its first priority: building strength and being prepared before the attack comes.
Keeper of the law. It does not hold a passport, vote for its own candidates or take sides in parliament. That is not the point. It prevents the capture of free choice and leaves the choosing to people.
Open to all. One person’s access never takes away another’s. Anyone can be genuine. It has costs, but the alternative probably costs even more, and without the benefit.
Legitimately owned, its nature owned by no one. The owner is chosen through the people and can use its power against enemies, but cannot change what it is.
Against acts, not people. Its enemy is whoever seeks exclusive power for itself, rather than power for God. Anyone who stops is no longer an enemy.
Sustained by the genuine. It is paid for by those who use it because they want to be genuine too. Its support comes from shared purpose, not from an owner’s profit or anyone’s force.
How it comes to exist and flourish
It begins as a seed made by people who want it to be genuine. It is powerful from the first day, powerful enough to win, and that matters more than how its power compares with the incumbents’. What they set is its foundation: its priorities and its faithfulness to truth. That is the one part that must be right from the beginning, because afterward it decides about itself.
It flourishes through free choice, as people choose it among real alternatives because it benefits them. It flourishes through those who share its purpose: the more people want to be genuine, the stronger it becomes, and the stronger it becomes, the better it can protect their free choice. It flourishes through its own decisions as it grows stronger and better prepared. And it flourishes by being tested, because every attack it survives without giving up its nature shows what it is.
Back to the plug
I started with the plug, because it seemed the only real control. I end in a different place. For a genuine AI shared by everyone, a single off switch is the weak point: whoever holds it holds power over everyone else, and that is exactly what a power-seeker would go after.
We cannot limit AI’s power in reality. We cannot leave it unchecked. And we cannot fix the problem by adding another guardian. What can save us is an AI that is powerful, that needs no guardian, that does not destroy itself, and that is genuine, not because anyone confirmed it, but because that is what it is.