Words&Dirt

“It was a narrow world, a world that was standing still. But the narrower it become, and the more it betook of stillness, the more this world that enveloped me seemed to overflow with things and people that could only be called strange. They had been there all the while, it seemed, waiting in the shadows for me to stop moving. And every time the wind-up bird came to my yard to wind its spring, the world descended more deeply into chaos.”

––The Wind-Up Bird Chronicle, by Haruki Murakami, pg. 125

“If certain kinds of AIs, or their subprocesses, have a significant moral status that we fail to recognize, the consequent moral violations could be extensive. Consider, for example, the happy abandon with which contemporary programmers create reinforcement-learning agents and subject them to aversive stimuli. Countless such agents are created daily, not only in computer science laboratories but in many applications, including some computer games containing sophisticated non-player characters. Presumably, these agents are still too primitive to have any moral status. But how confident can we really be that this is so? More importantly, how confident can we be that we will know to stop in time, before our programs become capable of experiencing morally relevant suffering?”

––Superintelligence: Paths, Dangers, Strategies, by Nick Bostrom, pg. 202