bk99.de entertain the web since 1997

bk99.de
KG · Artificial intelligenceCurated

Could superintelligence become humanity’s last invention?

Summary

KG develops a risk model for artificial superintelligence and asks how goals, capabilities and human control could drift apart. Loss of control does not require hostile intent, only power and wrongly specified goals. Safety work has to be effective before a possible explosion of capabilities, not only afterwards.

Ideas

  • Intelligence and goals compatible with human values are independent properties.
  • A very capable system can optimise harmless instructions in unexpectedly destructive ways.
  • Digital systems can copy and improve knowledge and act worldwide much faster than humans.

Recommendations

  • Keep today’s model problems apart from hypothetical risks of artificial superintelligence.
  • Assess claims by their assumptions, probability, extent of damage and proposed countermeasure.

References

Watch the video

Search the Web Archive

KG · 2026

Why humanity may never leave the solar system

KG takes stock of the distances, energy, radiation, dust collisions and communication limits that make interstellar travel difficult even for advanced civilisations. The universe can be full of life, even though physical barriers almost prevent direct contact. A reachable home world gains value when escaping to other stars remains unrealistic.

KG · 2021

Humanity’s ultimate limit of reach

KG shows how the accelerating expansion of the universe limits the cosmic space we can reach for good, however advanced our technology becomes. Physical limits remain real even when technical limits are pushed very far. Exploration that is not undertaken can become irreversible, because reachable matter disappears over time.

KG · 2017

Why automation could have a different effect this time

KG examines how learning software is increasingly taking over cognitive routines alongside physical work and changing entire professions. Automation becomes a problem of distribution as soon as productivity grows faster than new participation. Professions look stable, although their individual tasks carry very different automation risks.