Words&Dirt

“‘The reasons orphans should be adopted before adolescence is that they should be loved, and have someone to love, before they embark on that necessary phase of adolescence: namely deceitfulness,’ Larch argued in the letter.  ‘A teen-ager discovers that deceit is almost as seductive as sex, and much more easily accomplished.  It may be especially easy to deceive loved ones––the people who love you are the least willing to acknowledge your deceit.  But if you love no one, and feel that no one loves you, there’s no one with the power to sting you by pointing out to you that you’re lying.  If an orphan is not adopted by the time he reaches this alarming period of adolescence, he may continue to deceive himself, and others forever.

‘For a terrible time of life a teen-ager deceives himself; he believes he can trick the world.  He believes he is invulnerable.  An adolescent who is an orphan at this phase is in danger of never growing up.'”

––The Cider House Rules, by John Irving, pg. 102

“There is a cottage industry of people who propose the One Simple Principle that will make AI do what we want.  None of them will work.  We act not for the sake of happiness or pleasure alone.  What we value is highly complex.  Evolution gave you a thousand shards of desire.  (To see what a mess this makes of your neurobiology, read the first two chapters of Neuroscience of Preference and Choice).

This is also why moral philosophers have spent thousands of years failing to find a simple set of principles that, if enacted, would create a world we want.  Every time someone proposes a small set of moral principles, somebody else shows where the holes are.  Leave something out, even something that seems trivial, and things can go disastrously wrong:

Consider the incredibly important human value of ‘boredom’––our desire not to do ‘the same thing’ over and over and over again.  You can imagine a mind that contained almost the whole specification of human value, almost all the morals and metamorals, but left you just this one thing––

––and so it spent until the end of time, and until the farthest reaches of its light cone, replaying a single highly optimized experience, over and over and over again.

Or imagine a mind that contained almost the whole specification of which sort of feelings humans most enjoy––but not the idea that those feelings had important external referents.  So that the mind just went around feeling like it had made an important discovery, feeling it had found the perfect lover, feeling it had helped a friend, but not actually doing any of those things, having become its own experience machine.  And if the mind pursued those feelings and their referents, it would be a good future and true; but because this one dimension of value was left out, the future became something dull.  Boring and repetitive, because although this mind felt that it was encountering experiences of incredible novelty, this feeling was in no wise true.

Or the converse problem: an agent that contains all the aspects of human value, except the valuation of subjective experience.  So that the result is a nonsentient optimizer that goes around making genuine discoveries, but the discoveries are not savored and enjoyed, because there is no one there to do so…

Value isn’t just complicated, it’s fragile.  There is more than one dimension of human value, where if just that one thing is lost, the Future becomes null.  Not every single blow will shatter all value––but more than one possible ‘single blow’ will do so.

You can see where this is going.  Since we’ve never decoded an entire human value system, we don’t know what values to give an AI.  We don’t know what wish to make.  If we created superhuman AI tomorrow, we could only give it a disastrously incomplete value system, and then it would go on to do things we don’t want, because it would be doing what we wished for instead of what we wanted.

Right now, we only know how to build AIs that optimize for something other than what we want.  We only know how to build dangerous AIs.  Worse, we’re learning how to make AIs safe much more slowly than we’re learning how to make AIs powerful, because we’re devoting more resources to the problems of AI capability than we are to the problems of AI safety.

The clock is ticking.  AI is coming.  And we are not ready.”

––Facing the Intelligence Explosion, by Luke Muehlhauser, loc. 967