Words&Dirt

“Mankind can build a Great Thing. Sometimes we do it. But then we have to live with the consequences of greatness. What does a Great Thing tell us about ourselves? Not that we are great, but that our Great Things are so rare, and so much abused. So many in our dreams, so few to loom like towers in the light of day.”

––Hieroglyph: Stories & Visions for a Better Future, “Tall Tower,” by Bruce Sterling, pg. 506

“The AI makers––those I’ve spoken with, busily beavering away to make intelligent systems, don’t think what they’re doing is dangerous. Most, however, have a deep-seated sense that machine intelligence will replace human intelligence. But they don’t speculate about how that will come about.

AI makers tend to believe intelligent systems will only do what they’re programmed to do. But Omohundro says they’ll do that, but lots more, too, and we can know with some precision how advanced AI systems will behave. Some of that behavior is unexpected and creative. It’s embedded in a concept that’s so alarmingly simple that it took insight like Omohundro’s to spot it: for a sufficiently intelligent system, avoiding vulnerabilities is as powerful a motivator as explicitly constructed goals and subgoals.

We must beware the unintended consequences of the goals we program into intelligent systems, and also beware the consequences of what we leave out.”

––Our Final Invention: Artificial Intelligence and the End of the Human Era, by James Barrat, pg. 98