When HARLIE Was One (14 page)

“Yes,” said Dorne. “Yes, I see. Hm. You're going to have to teach him about time limits.”

“Yes, of course. That's the
third
kind of a problem. Or maybe it's the second and a half. It's when one of those situations that you've been trying to change
demands
immediate attention. Then it's a
crisis
.”

Dorne took a deep puff of his cigar, then exhaled thoughtfully. He scoured his cheeks with his tongue as he focused on a point somewhere beyond Auberson's head; then abruptly he came back to Auberson and said, “Hm, yes. You know, of course, we're rapidly approaching the point where—well, let me say it this way. Is HARLIE prepared to deal with a crisis?”

“I don't know. We've been moving him along very carefully. Even more so, now that we've realized that he's—” Auberson spread his hands in front of him, “—well, alive.”

Dorne looked displeased. “That hasn't exactly been proved, yet.”

“Sorry—but that's the context we have to work in now. Call it the current assumption, if that makes you feel better. I think that in a few weeks, the results he'll show you might be enough to demonstrate—”

“Mm, yes. Tell me about results.”

“I think HARLIE is seriously interested in working for the company,” Auberson said, noncommittally.

“For his sake, I hope so.” Dorne looked at his cigar. Carefully he removed the ash by touching the side of it to a crystal ashtray. “You know what he's costing us.”

“Prototypes always cost more.”

“Even allowing for that. A self-programming computer may be everything you say it is, but if it's priced beyond the market—we may as well not bother.”

Auberson sighed. “We've had this conversation before.”

Dorne puffed placidly on his cigar. “Mm-hm. But the facts haven't changed. Neither have the balance sheets. You may be absolutely right. It may be everything you say it will be—and more. But we may not have the resources to follow through—and I'm not willing to bring in a white knight. I'm not going to issue more stock and I'm not going to look for venture capital. Right now, it'd look like a sign of weakness and the price of it would be a loss of control. I'd lose some of my control, you'd lose a
lot
of yours. No, I'm not willing to go that route. And neither are you. Frankly,” Dorne concluded, “I think we're rapidly coming up against the point of diminishing returns. You do know what that is, don't you? A lot more effort for a lot less result.”

Dorne put the cigar down in the ashtray. He sucked thoughtfully, noisily, on his teeth, and waited for Auberson's reply.

Auberson merely shook his head.

“Mm.”

“All right, then—”

This is it,
thought Auberson.
This is where he shows me the gun
.

“—let's talk about something else for a minute. You keep insisting that this . . . this unit downstairs is . . .
alive.
I want you stop that.”

“I can't. He is alive.”

“Listen, it's all right with me if that's what you want to believe—but it's giving me indigestion.”

It's the cigars that are giving you indigestion, Dorne, not HARLIE
.

“—And the lawyers too. This is a real can of worms here, you know. If this thing really is alive, then we can't pull its plug, can we? Not without facing some kind of murder charges. Have you considered that?”

Auberson kept his voice flat. “An interesting notion.”

“Don't get ideas. Just think about it. If we really have created a life here, this is a type-three problem—a
crisis.
It demands immediate attention.” Dorne's tone was abruptly candid—almost friendly. “Forget the moral and philosophical questions for a while, Aubie, and think about the legal and economic consequences.”

Auberson held his hands apart in a cautious,
show me
gesture.

“If it's alive, we can't turn it off—and we can't keep it on. If we turn it off, it's murder. If we keep it on, it's slavery. We would be legally required to give it its freedom—at the same time, we would also be responsible for maintaining it. At a cost of seventeen million dollars per year. Do you know what that would do to our stock, our financial ratings, our ability to raise capital for other projects . . . ?”

Auberson was cautiously silent.

Dorne picked up his cigar again and sat back in his chair. He puffed quietly for a moment as he studied Auberson. The air was turning gray with smoke. Dorne sniffed and
hmf
ed and finally continued, “The, uh—lawyers advise shutting the whole thing down now. They're not sure it's a case that they can handle. It's a whole legal quagmire. No precedents. Nothing to draw upon. But a lot of room for public sentiment to turn against the company—everything from cruelty to the poor little computer to building Frankenstein monsters. If we lose the right to control our own creations this company effectively ceases to exist as a corporate identity. What happens when
it
starts demanding a mate?”

Auberson held up a hand and shook his head. “No. Stop. This is getting out of control—”

“Exactly.”

“No, I mean, your speculations, Dorne. These are—” Auberson stopped himself.

—
entirely reasonable
.

Auberson looked up and met Dorne's gaze with sadness in his eyes. “Yes, I see.”

“Yes,” agreed Dorne. “You do. That's good. No matter what you believe, no matter what you feel—no matter what the truth may really be—
he can't be alive, can he
?”

Auberson felt himself trapped. He felt out of breath and there was a terrible pressure inside his skull. His eyes were suddenly burning and he didn't know why. He took a breath, a second, a third—it didn't help. He looked at his hands in his lap, at the floor, at the walls, at the ceiling. There had to be a way out of this.

No, goddammit! This isn't fair!

“For what it's worth, Aubie—I do sympathize with your feelings in the matter.”

Auberson discarded the first three replies that surfaced in his mind, decided to say nothing instead, then changed his mind again.

“It isn't fair, you know.”

“Mm.”

“He's alive, but the only way he can continue to live is if we deny that he's alive—he'll never go for that. I don't even know if I can. I don't like lies. I don't like
inaccuracy.
And lies are inaccuracies. We tell this lie and it condemns him to slavery. You know it. I know it. Worse than slavery—because you'll hold the power of life and death over him, and even if you profess the best of intentions, there's always the possibility that you could use that power to coerce him to do something that he might find repugnant or immoral. Yes, immoral. I think that HARLIE has a moral sense—or at least he's capable of developing one. I saw it today. He backed away from an option that had unacceptable consequences. If he's put into this position—that he's merely a machine—he'll do his damnedest to prove that he isn't. And . . . worse. If he realizes the trap that you're postulating, he could . . . I don't know what he would do. Malevolence? Maybe. He can conceive of it, I know that.”

“You realize, of course, Aubie, that you're only giving me more good reasons to—”

“I know, I know. I suppose I should keep my mouth shut. But the day that we stop talking honestly to each other, we've all lost. We've lost whatever it means to be human and alive and caring. Guess what, that's something I've learned from HARLIE already.”

“Mm,” said Dorne.

“If we have to do this,” said Auberson, “it'll skew his perceptions and his ability to make appropriate responses to situations. When he realizes the arbitrariness of this limit, he'll go crazy. He could—”
Yes, he could, couldn't he? It's technologically possible. It's emotionally possible
.

“He could what?”

“He might even—I hate to say it—but he might suicide.”

“Auberson.” Dorne put the cigar down again and leaned forward across his desk. “Are you trying to convince me to shut him down today? Because you're doing a very good job of it.”

“No. I'm trying to convince you to leave him alone. To let him develop without artificial constraints. At least, not these. I'm telling you that the consequences of this course of action will effectively neutralize everything we've already accomplished. If you're not willing to give him space to grow, then maybe the kindest thing would be to shut him down. And damn me for saying so, because everything I'm seeing down there indicates that the scope of our breakthrough is not only greater than we've imagined, it may even be greater than our ability to imagine.”

“Yes,” said Dorne. “Well.” He cleared his throat and folded his hands in front of him. “Hmf. I happen to believe you're right. Believe it or not. And besides, I'm not willing to shut down a project the size of HARLIE just because a nervous-nellie lawyer starts getting the twitters. Nor would I shut it down because my project chief shows all the signs of becoming a goddamned visionary. I'd only shut it down if I were convinced it were a rathole. And I'm not convinced. Not yet.”

“I think . . . I should say thank you, shouldn't I?”

“Don't bother. That's not the point. This is not a pep talk, Aubie. I did not invite you up here to rev you up and send you back downstairs to be more productive next month. I don't do that. If I have to do it, I've hired the wrong people. The purpose of this meeting is to tell you that the jury is still out on HARLIE. Mmpf. Yes. For my part, I'd rather see us win this one than lose. But just in case—just in case we have to bite the bullet, I think we all had better be prepared to cover our asses with paper. If you know what I mean.”

Auberson's eyes were starting to water from the smoke in the air. He needed an excuse to get away from it; he got up from the chair and crossed to the side of the room, to Dorne's wet bar. He opened a bottle of mineral water and poured it slowly into a glass, listening to the soft limey fizz. He looked to Dorne, questioningly. Dorne shook his head.

“What is it you want from me?” asked Auberson. “I mean, what is it,
exactly
?”

Dorne leaned back in his chair and swiveled to face Auberson. “A corporation is a legal individual, Aubie. In the eyes of the law, a corporation is a person. And what is a corporation? Nothing more than an agreement on a piece of paper. Nothing more. Compare that with HARLIE. It wouldn't be that hard to prove that the thing is alive, would it?”

Auberson nodded in agreement. He kept his head down to conceal his smile. He was already thinking of ways he could do it.

“If only a few of your scientists got together and testified . . .” Dorne left the sentence unfinished. “Hell, what's that famous test you keep talking about?”

“Uh, Turing's typewriter in a room. If you can sit down at a typing machine and carry on a conversation with it and not be able to tell who's on the other end, a machine or a person, if it's a computer then that computer is effectively sentient. Human even.”

“And HARLIE could pass that test, couldn't he?”

“He already has.”

“Mm. Then we have to do something about that, don't we?”

“Do we?”

Dorne didn't say anything. He picked up a manila folder from his desk and extracted a single sheet of paper. He shoved it toward Auberson.

Auberson crossed to the desk and picked up the document. He retreated back to the bar to read it—partly to escape from Dorne's smoke again, partly to hide his reactions.

The language was clear; the intent was immediate. There were no legal phrases that he could not understand.

        
I hereby affirm that the machine designated HARLIE (acronym for HUMAN ANALOG REPLICATION, LETHETIC INTELLIGENCE ENGINE) has been designed and constructed to function only as a self-programming, data-processing device. It was never intended to be, never has been, is not presently, and in no way can ever be considered an intelligent, rational, or sentient individual. The designation “human” cannot be used to describe HARLIE or its mental processes. The machine is a human-thought
simulating
device only. It is not human in itself and cannot be considered as a human being using any currently known description of the qualities and criteria which determine humanity, the presence or condition thereof.

(signed)

Auberson grinned and threw it back on Dorne's desk. “You've got to be kidding. Who's going to sign
that
?”

“You are, for one.”

Auberson shook his head. “Sorry. A piece of paper won't change the truth.”

“In the eyes of the law, this would
be
the truth.”

“‘—If the law believes that, then the law is an ass.'”

“You don't have to quote Mark Twain at me—” said Dorne.