Authors: Mark Russinovich
Jump Trading was one of the earliest companies to migrate to electronic trading on the old New York Stock Exchange. Known for its cutting-edge algorithmic trading, the company had established itself as one of the founders of the new digital trading world.
With a Ph.D. in computational mathematics, Russo had worked in creating the algos, as they were commonly known, that generated the company’s profits. He’d enjoyed the work, but in his view, too much of what he devised had been vetoed as too risky. Jump, he’d discovered, was too conservative for his taste. He couldn’t understand the persistent aversion to a higher level of risk, which made possible far greater profits. He should have been a very wealthy man by now, rather than one with just a few million. The challenge, and profit sharing, Mitri Growth offered had been the career change he was searching for.
The founder of Mitri Growth wanted cutting-edge code to exploit the company’s recent, expensively acquired proximity advantage, but more than that, he’d challenged Russo to discover new ways to leverage capital out of the Exchange. The assignment was entirely possible, and Russo was eager to discover the next clever means to achieve his mission. The best part had been the founder’s willingness to run with Russo’s instincts in crafting algos.
Traditionally, stock trading took place in a pit. Sellers stood there, offering stock at a certain price using hand gestures; buyers either bought or didn’t. The price was constantly fluctuating in the pit, in sight of everyone. With the introduction of computers, all that had changed. Stocks were no longer bought and sold at a public location by traders. Now the work was done by machines. As late as 2005, 80 percent of all stock and equity trades were still executed at the New York Stock Exchange, but computers allowed those trades to complete not just more quickly but also remotely. The pit could be anywhere. The consequence was that by 2009, just 25 percent of all trades originated at the Exchange; the rest occurred within alternative trading systems known as ATSes.
That was the primary reason for creating the New Jersey hub, and for giving key traders such as Mitri Growth favored access. The Exchange needed this not just to stay profitable, but remain relevant as well. Already, similar Exchange hubs were opening or under construction around the world. Forty global “liquidity hubs,” as the Exchange preferred to call them, were planned. A major hub in Basildon, east of London, was already operational and linked.
Despite public statements to the contrary, the key to all the NYSE expansion was the high-frequency trader, or HFT. Initially, computers had introduced greater efficiency into an aging system, but it wasn’t long before the bright code writers known in the industry as “quants” began figuring out ways to take advantage of a computer’s ability to process enormous amounts of information at inhuman speeds. Once they inserted the code authorizing a machine to buy and sell when specific conditions existed, without human interaction, it functioned like a moneymaking robot. High-frequency traders now accounted for most of the action reported on the Exchange.
As in sports competitions, when it came to high-frequency trading, speed made up for shortcomings. If one performed enough transactions fast enough, one didn’t necessarily require the best code. Volume and speed compensated for minor missteps. Still, those with superior code, preferred access, and the most powerful engines made the most money.
At heart, HFTs were profitable because the computers knew the trading price of a stock anyplace in the world at the same instant and simultaneously compared it to the options price. Then, with lightning speed, they bought and sold on any detected difference before the Exchange’s trading computers could adjust for price fluctuations. One of Russo’s young designers had crafted an elegant bit of code that gave Mitri Growth the ability to predict the options price just ahead of its competitors, based on dozens of inputs and trends from across securities and exchanges. That was the algo they’d launched just after midnight with such high expectations.
The unspoken truth about HFTs was that they worked very much like a Las Vegas or Atlantic City casino, which takes a piece of all the action. It didn’t matter to Mitri Growth if the market went up or down. It could ride a stock up, or short it on the way down. What counted was the action, because Mitri Growth’s algos were structured to make money either way. It was not unusual for an HFT company with as few as thirty employees to earn a net profit of $1 billion. That was Mitri Growth’s target with Russo’s new algo program. But, as in a poker game that required a high stake to compete, money could be lost as quickly as it was won.
And that’s what Russo was seeing—if the downturn was really happening.
Just then, Baker walked up. Tall and prematurely balding, his chief assistant had elected to trim his hair and grow a goatee to compensate. “Well?” Russo asked.
“We’ve ruled out the test code.”
“So the new algo isn’t performing in production the way it did in simulation.”
“It doesn’t seem to be.” Before launching a new algorithm, Mitri Growth fed it current market data to see how it would have reacted in the past. Though not a perfect predictor of future success, it was the best validation the team could perform before letting a new version out to compete with everyone in the real world. Still, a slight unanticipated pattern and coded protections could cause the algo to become unstable in practice.
“So what’s different now?” Russo asked.
The senior programmer shook his head. “We have no idea.”
“So you’re telling me these trades are real?”
“I’m afraid so.” Baker cleared his throat. “We have to shut down, Jon. Then regroup. It’s going to take days to figure this out and fix it.”
“All right!” Russo snapped. “Take us off.” He buried his face into his hands and slowly exhaled. He had to tell the founder. “How much?” he asked without looking at the screen, struggling to control himself.
“Twenty-three million. Hey, it could have been a lot worse.”
9
TRADING PLATFORMS IT SECURITY
WALL STREET
NEW YORK CITY
11:13
A.M.
From the day they started with this project, Jeff and Frank had enjoyed playing hacker. It was one of the more satisfying aspects of their job, especially when they succeeded. “This is the New York Stock Exchange,” Frank had said when Jeff told him about the engagement in their D.C. office. “Do you think we can do it?”
“My bet is that we can. No matter how much a company depends on computers, no matter how big it is or how solid its reputation, its software and network are so complicated, the demands to make the process responsive to the market so great, that there are cracks everywhere. If we probe long enough, we’ll get in.”
“That’s a little unsettling. This is a major cog in the world financial system we’re talking about.”
“Yes, it is.”
They launched the pentest by casing the network from their low-privileged workstation. Jeff ran his own tools to develop a map of the systems in the network, looking to obtain as much information as possible from his position as an outsider. Once that step was completed, he ran other tools, attempting to connect to the systems at the ports used by standard system software and applications. He observed and carefully examined the responses he received. Even error codes returned when his attempts were refused revealed information, if nothing other than what software version was running, along with a few configuration details.
While Jeff was doing that, Frank trawled the Exchange’s intranet directory, following links to the connected Web sites and scanning documents for tidbits of useful intelligence relating to the jump servers. He located a year-old document for the Universal Trading Platform, or UTP, which contained lists of names and user accounts for the team that deployed trading software to the New Jersey engines.
The UTP was designed to support all trading scenarios with submillisecond response time known as latency. The platform was integral to the Exchange’s functionality and capable of being expanded as necessary. It also allowed outside parties “easy integration” within the NYSE Euronext global marketplace, which meant traders could pursue an endless variety of strategic initiatives of every type.
Frank was amazed at the lax approach to a system so essential to the world’s financial security. He had anticipated that the system would be accessible only to NYSE Euronext’s most trusted software engineers. Instead, many of the major traders had all but unfettered access. It was like a bank allowing its biggest customers to play around with its software to make things easy on themselves.
The consequence was that high-frequency traders typically tested new algos, live, on the Exchange, in secret. More than once, they were believed to have nearly caused a catastrophe. For one week, a mysterious computer program had placed orders, then canceled them before they were executed. Those algos made orders in twenty-five-millisecond bursts involving some five hundred stocks. In so doing, the program occupied 10 percent of the bandwidth allocated for the Exchange, certainly shutting out legitimate traders, just to test software in real time. That seemed to Jeff and Frank an unacceptable risk, but it was routinely permitted.
They’d conducted their reconnaissance exactly as a hacker would, constructing a schematic of the Exchange network. This included Web sites, server software, antivirus systems, user accounts, and their roles. Both of them noted potential points of vulnerability from time to time but this phase of their operation was primarily about collecting intelligence.
As they’d anticipated, the Exchange network was segmented into two zones. The first zone was standard issue to most companies and considered both insecure and untrustworthy. It constituted the public face of the Exchange, offering the usual applications anyone visiting a company on the Internet expected to find. It was also where the workstations and servers supporting the business operations of the Exchange operated. The second zone, where the actual trading engine functioned, was buried within the interior of the site and locked down. For security reasons, it was not linked to the Internet.
The two zones were connected through dedicated computers called jump servers. Those servers substituted for the more traditional internal firewall. A jump server was designed to act as the secure conduit between the two zones. In other words, though anyone could access the public zone from their personal computing device, to enter the secure zone, one had to pass through a jump server, the sole gateway to the core systems.
One inherent advantage of the jump server was that all the tools required for network management were maintained within a single system. This made maintenance and updating a straightforward process, performed in a single location. Access permissions were tightly controlled, and all operations performed on it were continuously audited and monitored as well. And it could be thoroughly locked down.
But it was much like keeping all one’s eggs in a single basket. This system had the advantage of isolating a vital gateway, which made it easier to control, but the disadvantage of presenting a single target for hackers to penetrate. If the jump server remained secure, it was a wall against intruders; if it failed, it served as a highway for them. It posed as their greatest challenge, but as a consequence, it was also their target.
Jeff’s tool had identified servers in the Exchange running Payment Dynamo, and on the US-CERT Web site, he learned that a slew of security bugs had been recently patched with an update from the vendor, Payment Data Corp. The bugs were only the latest of a string of holes found over the last year in this particular package, a product that was not unique to the New York Stock Exchange; it was used for many applications within a wide range of financial institutions. For all that, neither Jeff nor Frank had been surprised at its poor design. They saw the same thing time and again. Like fancy chrome-plated door locks easily bypassed, this package offered no sophisticated security. The designers had focused on its utility, as what it did made the sale, not how well it was secured.
When the recent patches were released, FirstReact, the cybersecurity research firm that had reported the vulnerabilities, began selling exploit code for them at a hefty price. This practice, while controversial, was common. FirstReact specialized in discovering holes in software, as well as in writing exploits for those vulnerabilities and ones others had reported. Their customers were willing to pay a premium to gain protection against a hacker discovering the flaw and exploiting it.
Companies purchased these via subscriptions, ostensibly both to check for their exposure by trying the exploits out on their own networks, and develop and deploy mitigations specific to their environment. Because many of the vulnerabilities were unpatched when FirstReact sold them, they were “zero days,” and could be used to spread malware and perform targeted attacks if they fell into the wrong hands. For that reason, FirstReact had a policy to sell them only to publicly traded companies and government agencies from a list of U.S.-friendly countries. But the assumption that knowledge of both the bugs and means to exploit them wouldn’t leak was flawed. The fact was that some of the buyers, typically government agencies, used them to infiltrate foreign governments for espionage and to cyberattack criminal and terrorist organizations.
Jeff viewed zero day bugs to be the digital equivalent of nuclear weapons and believed the only way to make sure they didn’t fall into the wrong hands was to strictly limit knowledge of them.
In this case, Payment Dynamo’s vendor had released patches just a week earlier, so while the bugs weren’t zero days, there was a chance that the Exchange hadn’t yet rolled out the fix. So that it could stay competitive, Red Zoya was one of the companies that paid the FirstReact subscription fees, so Jeff was in possession of the exploit codes to match the vulnerabilities and had used them to break into the fourth Payment Dynamo server he tried them against.