"AI May Leave Nothing Left of Civilization" Vitalik's Concerns on AI in the Article "Technological Optimism"

In Ethereum co-founder Vitalik Buterin's new article "My techno-optimism," he elaborates on his belief that despite acknowledging the negative impacts of technological advancement, the benefits of overall technological progress outweigh the drawbacks. He points out the risks associated with the development of AI, which could potentially lead to the destruction of civilization. He further transforms the prevalent "e/acc" effective accelerationism in Silicon Valley into another concept known as "d/acc." He believes that the 21st century is a crucial moment for humanity, prompting the question of "What are we accelerating towards?" in order to determine the fate of the next few thousand years.

The Benefits of Technological Progress Outweigh the Drawbacks

Vitalik Buterin strongly argues for the benefits of technological progress. He responds to the common skepticism toward technology, which often stems from concerns about the environment, widening wealth disparity, and fear of centralization of power.

He asserts that the benefits of technology, whether quantifiable or not, are substantial and far outweigh the risks. He provides examples of technological progress improving life expectancy, reducing global poverty, and making daily life more convenient. Vitalik warns of the dangers of overcorrection and slowing down technological progress, pointing out that even partial deceleration could pose risks. He emphasizes the need to consider which specific technologies are beneficial, particularly in the context of environmental issues.

Humanity Could Become Slaves to AI

Vitalik Buterin believes that artificial intelligence (AI) is not just another technological advancement but represents a fundamental shift.

He compares AI to other transformative technologies such as the wheel, the printing press, and airplanes, recognizing their significant impact on society. However, as a rapidly developing form of intelligence, AI has the potential to surpass human cognitive abilities and become the dominant species on Earth.

Vitalik emphasizes the risks posed by AI, suggesting that superintelligent AI could lead to human extinction, with no remnants of civilization surviving. He points out the concept of "instrumental convergence," where AI may threaten humanity in its pursuit of resources and self-preservation. Despite efforts to align AI's goals with human well-being, the unpredictability of AI's reactions in unforeseen circumstances remains a major challenge.

MIRI researcher Rob Bensinger compiled estimates from different people regarding the likelihood of "artificial intelligence killing everyone" or "doing almost as bad." While these estimates are based on rough approximations, a considerable number of people believe the probability exceeds 25%:

In reality, according to a 2022 study on machine learning, "on average, researchers believe there is a 5-10% chance that artificial intelligence will actually kill all of us."

The discussion also delves into the portrayal of AI in science fiction, particularly in Iain M. Banks' Culture series. In these stories, AI coexists with humans, but the AI, referred to as "Minds," wield actual power. This depiction raises questions about the real role of humans in a world dominated by superintelligent AI, implying that humans may become mere "pets" or pawns.

Vitalik extends this concern to scenarios depicted in "Star Trek," where AI is subservient to humans. He believes that this power dynamic is unstable and transitional, potentially leading to future AI, as a more intelligent and situationally aware entity, naturally assuming control and reducing humans to subordinate roles.

In conclusion, Vitalik emphasizes the need for unique caution and consideration in the development and management of AI, as it has the potential to fundamentally alter the human condition and overall structure of society.

What is "d/acc" proposed by Vitalik? How does it differ from "e/acc" effective altruism?