Elon Musk warns against unleashing artificial intelligence 'demon'

[True Blue Door]

35. The danger of AI is not AI itself, but the human agendas behind it.

So in some ways it would indeed be like "Summoning the demon."

Think of all the horror movie and TV show plots about people summoning demons to do their will, and then the results being horrifyingly out of their control.

We already know the primary task to which AI would be set: Making money for Wall Street banks. It would be tasked with automating high-level economic decisions to maximize returns for its masters. Since its success would be judged entirely by profit margin, the external costs would grow exponentially until it basically robbed the entire world and destroyed the global economy.

Other AIs would be deployed by other countries and organizations, but only the one absolute fastest and smartest would win - it would be zero-sum competition, and everyone other than the organization owning The One would be bankrupted.

The AI apocalypse would not be a nuclear holocaust followed by Terminators hunting humans in the ruins. It would just be a total economic collapse with nearly all the world's resources delivered into the hands of a handful of humans who control the AI, resulting in Gigadeath from starvation, disease, and geopolitical chaos. Supply chains managed by AI would be disrupted or diverted with no explanation, leaving store shelves empty, food rotting on the vine, transportation shut down, etc. etc.

The owners would know what's going on, and perhaps be afraid of what they'd done (the rich like stability), but rather than step back that fear would most likely make them dive in and set up private quasi-states to protect themselves from the hungry, enraged mobs the world over.

What evolves from that wouldn't look much like modern civilization - more like ancient Egypt on a global scale with high technology. What remains of humanity would be beaten down into subconscious servitude while a handful of individuals - probably, eventually just one - rule as God Kings through their AIs, while programmers and other technical people who manage the systems would be a new kind of priesthood, probably eventually hereditary.

That's one scenario anyway.