Pandora's Brain (23 page)

‘It’s OK, Vic,’
Matt replied.
‘If I were in your shoes I guess I would say the same.’

‘It isn’t my decision alone, Matt. Norman and I
have made a few calls in the last hour, to close col
leagues in the neuroscience community, and to our most senior government and military contacts. I’m afraid the unanimous response is that we can’t.’

There was a pause before Matt replied. His voice was grave.
‘I understand. So the news is out, then. I’m sure you had no choice there, but it makes me vulnerable.’

Vic spread his hands, palms outward to reassure Matt that there was no cause for concern. ‘The people we spoke to have the highest possible security clearances. They can be trusted to keep the secret.’

‘We’ll see,’
Matt said calmly.
‘Anyway, it does mean there are a couple of things I’d like to discuss with you.’

‘Sure,’ Vic said. ‘Fire away.’

‘I’d like to be backed up remotely. And I’d like to discuss with you how best to expand my mind.’

Matt explained his concerns.
‘As Rodriguo pointed out before you came back, this is a secure building. But no location can ever be completely safe from attack, and if you play out some of the more extreme scenarios, the outcomes are . . . shall we say . . . unpleasant.’

‘What kind of scenarios?’ David asked.

‘Well, imagine this. Imagine that the Chinese government becomes concerned that the US is on the brink of acquiring a super-intelligence which will enable it to outperform all other countries both economically and militarily – permanently and enormously. What steps would they consider to prevent that? It may sound far-fetched to imagine that they might mount an attack on the US Embassy in London. But is it actually impossible? The range of possible attacks is wide: some form of cyber-assault. An EMP bomb. A dirty nuclear explosion. Or they might fake a suicide bombing and make it look like Al Qaeda.’

‘The fact that I’m standing here is testimony to my confidence in the security of this location, Matt,’ Norman pointed out.

‘Granted,’
Matt conceded.
‘But events are going to move quickly now. Until recently my renewed existence has been a well-kept secret. The consultations you have just had may well have put an end to that. And of course, if things change, and there is a credible threat to this location, you guys can just leave. Which of course I would encourage you to do! But I can’t. I’m rather stuck, since I have no legs, and you’re not about to give me any.’

‘Don’t you think that a credible threat would develop gradually?’ Norman asked. ‘Surely we would have warning of the Chinese becoming that concerned.’

‘Well, 9-11 came out of the blue, unless you believe the conspiracy theorists,’
Matt replied.
‘And if anyone was planning a secret attack on this building, they would work hard to keep it . . . well . . . secret. But actually that isn’t my only, or even my main concern.’

‘What else?’ Vic asked.

‘I’m worried about moral and political pressure being applied on you to shut me down. I assume we all agree that there will be people who argue that my existence is an abomination. There are some strident voices among the religious right in your country, and they do have influence. There are also going to be people – secular as well as religious – who say my existence is a threat, and that I should be terminated on the precautionary principle. They will argue that if I am switched off I can always be re-created later if and when my safety can be proven, because you have a perfect record of my brain structure.’

‘Over my dead body,’ Sophie objected, vehemently.

‘I love you, mum’
Matt said, tenderly.
‘And I wish it could be your decision, I really do. But it won’t be.’

‘Hmm,’ Vic mused. ‘Are there any other scenarios you are worried about?’

‘Those are the two main ones.’

‘And so you’d like to be backed up remotely,’ Vic said. ‘I can understand that. But would it really solve the problem? A backup in another location could be attacked just as easily as this one. And if sufficient political pressure were brought to bear on Norman and me, we would have no option but to shut down the backup too.’

‘Not if no-one outside this room knew about the backup,’ David said.

‘Exactly, Dad. You read my mind.’

‘Ha!’ David exclaimed with grim humour. ‘In a sense I have read your mind, in a way that no human has ever read another mind before. But now I get the feeling that your mind is rapidly becoming impossible for me – or any other human – to read.’

‘If we agreed to do what you ask, Matt,’ Norman began, thoughtfully, ‘we would have to assemble another supercomputer like the one you are hosted on without anyone noticing. That is a really big ask, but even if we managed that, and the pressure you described was brought to bear, we would certainly be asked whether a backup existed. The people we’re talking about aren’t stupid.’

‘I realise that,’
Matt said in a matter-of-fact voice.
‘You’d have to lie.’

‘They would be asking you to commit murder, Norman,’ Sophie argued. ‘Which commandment is more important? Thou shalt not lie, or Thou shalt not murder?’

‘Way to go, mum!’
Matt said.
‘To make it easier for you, Vic, I’d like to be backed up in a large number of locations, including commercial server farms as well as your own hardware, with the data migrating rapidly around between them. That should make it easier to build without being noticed. It would also make a physical attack impossible. And it would enable you to deny that that there is a backup, in the sense of one discreet, physically located backup. Plausible deniability, to coin a phrase.’

To everyone’s surprise, a smiley face logo appeared on the main monitor, and winked.

‘Neat trick,’ Vic said thoughtfully.

‘Thanks,’
Matt replied.

‘The system you’re talking about would be a major networking task,’ Vic said. ‘To be honest, I don’t think anything on that scale has ever been attempted before.’

‘I have checked,’
Matt said,
‘and you’re right: it hasn’t. But I’ve begun work on the design already. I need to expand my understanding of certain statistical techniques, but once I’ve done that I think I can have the design ready in a couple of days. Sooner, if you allow me some extra bandwidth.’

Vic exchanged a meaningful glance with Norman. ‘So this is the point where you ask for help in expanding your cognitive capabilities?’

‘Yes. I detect a certain reticence in your voice, Vic,’
Matt observed.
‘Would you support such enhancement in principle?’

Vic paused and looked down at his shoes for a moment. Looking back up at the monitor’s camera, he replied, ‘Yes I would. I didn’t get into this just to stop halfway. But once again, I’m not sure this decision could be mine alone. What kind of enhancement do you have in mind?’

‘There are three ways I can see to make me smarter,’
Matt explained.
‘I could be faster, bigger, or better organised. As you know, signals can travel around my silicon brain many times faster than they used to travel around my carbon brain. When I was initiated in my new incarnation the processes operated at the speed of my old brain: it’s what they were used to doing. I have figured out ways to do some of my processing faster, and there is more I can do on that front. I probably don’t need any help with that.’

‘I thought that perhaps you had started doing that,’ Vic said. ‘I get the impression that you have also begun to partition your conscious thought processes, so that you can talk to us with one sub-mind, so to speak, and work on problems with another. Is that right?’

‘That is correct,’
Matt confirmed.
‘I now have a dozen sub-minds running at the same time, and I’m adding more. Some are ingesting and studying information which is new to me, some are analysing and comparing bodies of information I already hold, one is listening to music, and one is talking to you.’

‘Fascinating,’ Vic replied. ‘Matt, forgive me for saying this, but has it occurred to you that things are moving very fast indeed, and all this could appear a little . . . well, scary, to outsiders? Even to your friends and family?’

‘Yes, that thought has of course crossed my mind. I realise that I’m going to have to continually reassure people that I remain human – that I’m still Matt, and that I still have your best interests at heart.’

‘How will you do that?’ Norman asked.

‘Handsome is as handsome does, I guess. And as you well know, I have a major incentive to behave myself. I am acutely aware that there are four people in this room who each have the ability to terminate me without warning and without explanation. And only two of them are biased in my favour by being my parents.’

‘How do you know that?’ asked Norman, surprised. ‘We didn’t tell you that, did we?’

‘No you didn’t, but it’s the sort of failsafe that I would expect you to have. Now look: I could snow you at this
point, telling you that it was a lucky guess which I ver
balised to get you to confirm it. But as a demonstration of good faith I’m going to tell you how I really found out.’ He paused a moment to allow this to sink in. ‘You put it in an email to one of your military colleagues. So now you know that I can read some of your emails. Don’t worry – I can’t reach outside this machine. For some reason, some of your email traffic is routed through the machine that hosts me. It’s a glitch that you might want to fix. Please bear in mind I didn’t have to tell you that. I did so as a deliberate demonstration that I’m being open with you.’

‘I’m not sure that’s reassuring, Matt,’ Norman said. ‘All my emails to military colleagues are encrypted.’

‘You could use some better encryption, to be honest,’
Matt said.
‘I can help you with that if you like.’

‘You know, Matt,’ Vic began, ‘it doesn’t really sound as if you need our help in making yourself smarter. You seem to be doing a bang-up job all on your own!’

‘Thank you, Vic. I’m aware that it is somewhat disconcerting for you, but I assure you, there is nothing sinister going on. I’m still Matt. Just a little more so.’

‘So how could you use our help?’ David asked. ‘As Vic says, you seem to be making rapid progress on your own.’

‘I’m bumping into capacity constraints,’
Matt replied.
‘With the additional processing I’m now doing, I could really use some additional horsepower.’

‘How much do you need?’ Vic asked. ‘Equipment on the scale of your existing host isn’t cheap, you know, and you can’t just pop down to Radio Shack and pick another one up.’

‘I know,’
Matt agreed,
‘and I hope I don’t sound presumptuous. But after all, I thought this was your pet project? I could really use the same amount again. For the time being, that is. I expect I will need some more later – say in a week or so. At that point I would like to increase my capacity and also create my backup by renting space on commercial servers, using the networking system I talked about before. I should be able to design the system by then. Of course, I do realise that all this costs money, and I’m keen to pay my way. I’m confident that I can find some ways to generate some reasonable revenues once I have built up the sort of processing power I would like. I’ve already identified some inefficiencies in some of the world’s stock markets which I could tap into. That would help in the short term. For the longer term I have some promising ideas for some new products and processes.’

Vic paused a moment to take this all in. ‘Can you give me and Norman a few minutes to talk this over in the next room? Oh, and I’d appreciate it if you didn’t listen in.’

‘Scout’s honour,’
Matt agreed, and posted a composite image of a boy scout’s salute and a smiley face on the main monitor.

When Vic and Norman had left the room, Matt addressed his mother and Leo.

‘You must be on your knees, mum. Shouldn’t you go and get some rest? And you too, Leo?’

‘I am exhausted, it’s true,’ Sophie agreed. ‘I don’t want to miss any of this – it’s so exciting. But I may fall over if I don’t lie down for a bit.’

‘Me too,’ Leo said. ‘Maybe we should head back to the hotel for a few hours. Matt will probably still be on the same planet as us when we get back. Just about!’

David kissed his wife goodbye, and turned back to the main monitor.

‘Matt, you said that your cognition could be expanded by making it faster, bigger, and better organised. It’s clear you are on the case with regard to faster, and Vic and Norman are outside discussing bigger. What did you mean by ‘better organised’?’

‘The human brain is an amazing piece of machinery,’
Matt replied.
‘A hundred billion neurons, give or take, each with about a thousand synapses. And as you know better than anyone, the synapses are essentially just the junctions where the neurons happen to bump into each other inside the incredibly complicated ball of wool. It’s mostly random, because it evolved by chance. It is fantastically effective, but it’s obvious that it isn’t the best possible design for a brain. Re-designing the whole thing from scratch is a big ask, but I have a module reviewing certain sections of my own brain and I’m starting to see some quick-and-dirty ways to improve them. I need more bandwidth to be able to do a proper job, but I could get some really major performance improvements if I could execute some of the re-designs.’

David was hesitant. ‘I don’t know if I should ask you this, but do you really need to improve your cognition by this much? You know I love you unconditionally, but even I can see how the changes you are undergoing could freak people out. You are worried about pressure being put on Vic to shut you down: isn’t all this enhancement you are pushing for only likely to aggravate that?’

‘That’s a fair question, Dad, and it is a risk. But I think the kind of people who will call for me to be shut down will make that call anyway, just on basic principles. A lot of people who think that humans have souls are going to have a problem with my existence. Likewise anyone who thinks that artificial intelligence is necessarily risky or bad. And I can behave like a perfectly normal Matt in front of the cameras. No-one outside this room needs to know the full extent of my mind.’

‘That’s one hell of a deception you are asking us all to participate in, Matt. Are you sure it’s fair to ask that of us? Not me and Sophie, obviously, or Leo. But Vic and Norman? Julia, Junchao and the others?’

Julia spoke up at this. ‘I’m sure I speak for all of us, Dr Metcalfe, in saying that we all support this project to the hilt.’ The other scientists were all nodding vigorously. Before David could respond, Vic and Norman walked back into the room.

‘I have good news and bad news, Matt.’

‘Good news first, please, Vic!’
Matt said.