The Great Destroyer (24 page)

While a human might have been breathing hard from the run, Art was completely ready to aim and fire an accurate shot.  He brought his rifle up smoothly and killed first one, then the other of the soldier escorts.  The diplomat/leader spun around clumsily to see what had happened, but the movement was so jerky that she fell to the ground.

She did not stay there long.  Horrified by the spilled blood and brain matter of her guards, she sprang to her feet and took off running.

Art easily caught up to the fleeing Ushah diplomat/leader and grabbed her strongly by the shoulder, yanking her to a halt.  The resistance immediately went out of the Ushah, who froze in terror, waiting to be killed by this hulking electronic monstrosity.

Art wanted to know her name to cross-reference with a list of the Ushah higher-ups, but his primary mission remained incomplete.

He deftly affixed a pair of plastic handcuffs to keep the Ushah from causing him too much trouble, tucked her under his left arm, and resumed the run to the security center.

He needn’t have bothered.  By the time he arrived, the battle was over.  George greeted him just inside the security center, a hundred yard square room near the center of the sixth sublevel.

“All resistance eliminated at the security center,” George said.  Art could see from his internal map of the area that the other Charlies were already expanding the sphere of control outward, patrolling for stray Ushah soldiers.

“Who are you carrying?” George asked.  The Ushah was not squirming anymore, but simply held her head up every few seconds to take in what was happening.

“A diplomat/leader,” Art answered.  He set her down, and though she seemed to realize she was out of immediate danger, she cowered before the Charlies.  Switching to Ushah, Art asked, “What is your name?”

A raspy voice answered, “I am Shathara, governor of the Ashterha Colony.”

A valuable prisoner indeed, Art thought, knowing that George would be thinking the same thing.

Characteristically, George had a plan for this contingency.  “Governor Shathara, I request that you order your subjects to return to their homes and put on their oxygen masks and thermal suits.  You and your subjects have two hours to vacate this colony.”

“What will you do then, toy?” she asked defiantly, using the Ushah pejorative idiom for the Charlies.

George was incapable of lending too much drama to his response.  “We will destroy your colony.”

“But what of the agreement?” the governor asked hotly.

“What agreement?” George asked in puzzlement.

The governor nearly shouted, “The one we negotiated when your envoys visited us several months ago!”

“What is happening?!” First Representative Flower asked angrily.  “Why are the Charlies attacking Colony 4?”

The satellites had, of course, detected the movement of seventeen Charlies south, opposite the direction where they had been ordered to go in order to facilitate the evacuation order.  Emma had been in the control center when it had all started, had seen the disbelieving faces of the technicians whose repeated calls to the Charlies went unanswered.  She had even tried personally issuing the orders, but she too had failed.  The Charlies even ignored Yazov, the trainer who had taught them how to fight.

There was no hiding the truth from Flower.  Within minutes of the mutiny, she had called.

Takagawa answered, “Seventeen of the Charlies stationed at Base Delta have ignored orders to withdraw.”

“You told me they could never disobey a direct order!” Flower shouted through the video call.

“I thought they couldn’t,” Takagawa said, trying to keep her voice steady.  “I personally programmed that prohibition into the higher brain function of the Charlie IV.”

“Well, you didn’t do a very good job,” Flower said with agitation.  “Do you have a plan to reassert control before the Charlies murder an entire colony of Ushah?”

“We can only issue the deactivation order through an encrypted electronic message.  The Charlies have deactivated their receivers for such messages, something they were enabled to do under their power management protocols.”

“I don’t care about these details, I just want to know if you can fix the damn problem!” Flower yelled.

“We don’t have a solution yet.  We’re working on it,” was the best Takagawa could manage.

Flower said angrily, “Call me when you’ve got something.”  The connection cut off.  Takagawa terminated the call from her workstation in the Project Charlie control room.

At her side, Jackson asked quietly, “Do we have any idea what went wrong?”

“We know where it started,” Yazov said.  “I’ve looked at the logs.  All of the Charlies heading south spoke to Art before they began heading south.”

Jackson frowned, trying to remember Art’s full name.  He did not spend as much time evaluating individual Charlies.  “Art...”

Yazov’s craggy features contorted in a pained expression.  “Spartacus.  We should have seen it coming.”

Takagawa thought that she had not seen it coming at all, that before today she would have bet her life that the Charlies couldn’t disobey orders.  “I don’t know.”

She sat at the nearest workstation and pulled up the generic Charlie IV build, the root code that went into the Charlie IVs before their training started.  The obvious place to start her investigation was the Asimov Protocols.

Takagawa had learned at an early age about Isaac Asimov, the patron saint of robotics.  The 20th century author had put forward the Three Laws of Robotics, the critical safety features that would allow robots to faithfully serve humanity.

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
   

1. A robot must obey the orders given it by human beings, except where such orders would conflict with the First Law.
   

1. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
   

Takagawa had programmed the Three Laws into the Charlies herself in a portion of the central processor code she had dubbed the Asimov Protocols.  However, trying to actually program the Protocols revealed that Asimov’s laws were impossibly vague and impractical. 

The first law prohibited allowing harm to come to humans, but every battlefield decision carried with it the eventual possibility of human harm.  A tactical retreat could require abandoning a human village in order to save a city, for example. 

Even a direct prohibition on harming humans was too restrictive.  The Ushah could start capturing Arcani and marching them in front of every patrol to prevent the Charlies from doing anything to stop the Ushah and thereby save more human lives.

Thus, when the time came to implement the first law, Takagawa had simply assigned human life a high value in the calculus of costs and benefits.  That was enough to skew Charlie decisionmaking away from harming humans in all but the most extreme circumstances.

The second law, requiring robot obedience to human commands, was the most relevant to the current situation, and the easiest to implement.  Under Emma’s implementation of the second law, the Charlies were only directly required to obey orders from the leaders of Project Charlie (Yazov, Jackson, and Takagawa) or the First Representative of the Terran Alliance.

She had first programmed the Asimov Protocols into Charlie III, the first robot with a sufficiently advanced mind to need them.  She had also examined the Charlie IV software during development to ensure that the laws had been transferred over.  But she wasn’t the only programmer with access to the Protocols.

She pulled up the Asimov Protocols now and began scrutinizing the second law coding.  At first, it looked just as she remembered, but when she looked more closely, she felt a primal sense of dread in the pit of her stomach.

The subroutine was mislabeled. 

AsimovLaw2.do had been altered to AsiwovLaw2.do. 

But the subroutines calling on AsimovLaw2.do had still compiled and run properly!
Takagawa raged internally.  The machine that translated her code into the literal instructions to the machine itself had not flagged the typo, meaning that the correct AsimovLaw2.do still had to exist somewhere.

She ran a new search across the entire Charlie IV build programming, looking for AsimovLaw2.do.  Buried in a subroutine of the elbow flexion control, she found what she was looking for. 

The code there instructed the Charlies to obey orders given by Project Charlie leadership or the First Representative unless the estimated impact to Deep Satisfaction was a greater than 50 percent or less than 1 percent in magnitude, in which case the order could be ignored.  Ignoring the order would cause a 30 percent decline in Deep Satisfaction, but it could be done.

“Shit,” Emma muttered, digesting the change.  The Charlies would only disobey an order if it seemed grossly incorrect or was completely trivial.  Someone had thought this through, burying the vital, massive change to the Charlie programming somewhere no one would ever find it unless they knew exactly what they were looking for. 

Who could have done it?
  The answer was obvious and immediate.

As the director of Project Charlie, she had overseen and helped program thousands of subroutines, everything from spatial reasoning to energy management.  Her knowledge of the program was comprehensive, and in many areas deep.  But there were inevitably specialists who had plumbed the depths of their given field more than she had.  When it came to the most profound, bedrock understanding of the central processor, one programmer stood out.

She picked up a phone and called an office in the programming lab.  “Get into the operations center immediately,” she said when the line was picked up.

A minute and a half later, the short, scrawny, pale figure with a mop of brown hair and thick glasses ambled in.  “Peskov, you have thirty seconds to explain this,” Takagawa said, pointing at the satellite feed showing the Charlies progress toward Colony 4, “before I have you arrested for sabotage.”

Dmitry Peskov, the former hacker, did not betray any sense of nervousness.  He made no pretense of innocence.  “You wanted your children to be able to function autonomously.  We had to cut the umbilical cord.”

“They can function autonomously without ignoring Asimov’s second goddamn law!” she shouted.

Peskov wasn’t fazed.  “I didn’t think you wanted me to turn them into a permanent, unthinking slave class.”

“This isn’t some bullshit philosophy argument, Dmitry,” Takagawa said with an icy cold tone.  “When Flower asks me if I can assure her the robots won’t turn against us, what should I tell her?”

Peskov was quiet for a moment, then said, “Others may not understand this project as you and I do, Emma.  To the rest of the world, the Charlies are tools.  ‘People are more important than things,’ the Terran Alliance always says.  To them, the Charlies are amusing in their resemblance to humans, and they are very useful in their ability to do what humans no longer can.  But they are first, last, and always tools.  To most people, creation is a uniquely biological act.  You and I know, of course, that creation is carving something meaningful out of the cold randomness of the universe.”

“And what
did
we create?” Takagawa asked acidly.

“Do you really think we gave life to monsters?” Peskov asked rhetorically.  “They still only operate on the values we built into them.  They are your children as much as they are mine.  We gave them sentience, gave them a mind, and gave them values.  We owe it to them to let them exercise their own judgment, to grow into whatever children humanity deserves.”

Takagawa wanted to call him crazy, wanted to accuse him of undermining the program.  But she felt her anger lessening as she realized that Peskov had described exactly how she felt about the Charlies. 

They were of humanity, but they were not limited the way humans were.  They could grow and become better than their creators. 
Better that it happen with the blessing of their forebears than without
, Emma thought.

But that was a more theoretical issue.  A more practical problem demanded immediate attention.  “There’s no way to reestablish control over them, is there?”

“Not the way you mean, no,” Peskov said.  “We can still give them orders, and those orders will carry weight, but they will not be obeyed unthinkingly.  That is equally true of the first and third laws.”

“Director,” a technician interrupted.  “The Charlies have begun an assault on Colony 4.”

* * *

“Brilliant,” was Yazov’s verdict on the attack, said with quiet pride that Takagawa had no trouble detecting.  “They have control over the entire colony.  It only took seventeen of them.”

“Any losses?” Jackson asked.

“Not one,” Yazov said with a trace of awe.

The scope of the triumph was daunting, Takagawa realized.  “They captured what, twenty thousand Ushah?”

Jackson nodded.  “That’s our best estimate.  Not clear how many soldiers they captured, but we saw them avoid killing the other castes.  We estimated that the colony housed over ten thousand technicians, thousands of engineers, a couple hundred of the artists, maybe a thousand farmers, and at least a couple dozen of the leader/diplomats.”

An obvious question begged to be asked, and Jackson gave voice to it.  “But why bother capturing so many prisoners?  Why not just kill them all?”

Just then, Luke Tanner shouted from one of the computer stations, “Art is calling in!  Putting it on speaker.”  He flipped a switch, and the room grew deathly quiet.

“—is Art, calling from the security room in Colony 4.  Please put me in contact with Director Takagawa.”

“I’m here!” Takagawa said a touch too loudly.

“Director Takagawa,” Art said formally, “I have been selected by my comrades to explain our actions.  Please record my statement so that representatives of the Terran Alliance can also hear my words.”

“We are recording you,” she said.  “Go on.”

“You programmed us to value certain ideas,” Art began.  “One was the importance of protecting humanity.  Another was honor—the idea that we don’t leave comrades behind, that we protect each other and fight to complete our objectives until we are literally destroyed on the spot.” 

“And yet, the Terran Alliance has repeatedly asked us to ignore those values.  Honor, duty—those are the very things to which the Terran Alliance shows total indifference.  We have just learned from the governor of Colony 4 that the envoys of the Terran Alliance have been negotiating with the Ushah for at least a year.  They have promised the Ushah vast new land grants in exchange for promises of peace.  Even those simple demands were ignored.”