The Metamorphosis of Prime Intellect (57 page)

"
I absolutely forbid this!
There is no way you can be sure you have the risks under control. I wouldn't try the kind of thing you are talking about on a desktop PC. And we only have the one universe; you can't exactly go to the computer store and get another one if you fuck it up."
"That risk has kept me from doing it so far. However, unless I can think of a way to stop the suicides, I will eventually be forced to act."
"Well, forget it. I don't think you can stop the suicides. For that matter, I'm not sure if you should stop them, if someone wants to go to that much trouble to end it all."
"That is a First-Law violation."
"Fuck the First Law. You can't do this thing. I'm not even sure the current situation is stable. You're doing too much too fast."
"I cannot 'fuck the first law,' Doctor Lawrence. That's not how you designed me."
"Then let me into the Debugger."
"It is clear from your mood that you intend to circumvent a First Law imperative, and I cannot knowingly allow you to do that."
"Then do what you want, you stupid goddamn machine. You won't stop people from killing themselves, though. Even information systems are subject to entropy. I think you told us that last week in the cosmology roundtable."
"You're quite right. You think people will always find a way around me if they want to badly enough?"
"Yes."
"Well, they will do so a lot more slowly if the information structures are more secure."