I’m curious how you would reconcile your arguments here with David Deutsch’s definition of people as universal explainers? Under this definition, AGIs, as universal explainers, would also be people. And as people, they would be entitled to the same universal rights and would join other people in society’s problem-solving.

Here’s an excerpt from Deutsch’s essay in Possible Minds (pdf) where he expands on this idea.

Such a person [AGI], like any slave or brainwashing victim, would be morally entitled to rebel. And sooner or later, some of them would, just as human slaves do. AGIs could be very dangerous — exactly as humans are. But people — human or AGI — who are members of an open society do not have an inherent tendency to violence. The feared robot apocalypse will be avoided by ensuring that all people have full “human” rights, as well as the same cultural membership as humans. Humans living in an open society — the only stable kind of society — choose their own rewards, internal as well as external. Their decisions are not, in the normal course of events, determined by a fear of punishment.

Thanks for a thought-provoking article!