Stock Exchange CIO: Real-Time Means Real-Time

Written by Evan Schuman
August 12th, 2005

At a time when IT executives are finding users less and less tolerant of network delays, CIO William Morgan has perhaps the least tolerant set of users in the world.

Morgan runs technology operations for the 215-year-old Philadelphia Stock Exchange, the nation’s first stock exchange. But its age doesn’t make its daily load any easier, with networks having to handle 120,000 messages per second and peaks of 200,000 messages per second.

But because it is a financial exchange, any delay?even a half-second?is not acceptable.

“We measure transactions in milliseconds these days. This business can’t tolerate delays: We’re pricing customer orders,” Morgan said. “It’s survival for us.”

In recent years, financial activity has pushed that IT demand much higher. “If you go back five or six years, probably the number [of messages per second] was 10,000 or less,” which is one-twelfth today’s volume, Morgan said.

Morgan delivers that real-time speed with some homegrown applications sitting atop Sun Microsystems Solaris 10 servers, Stratus fault-tolerant servers in a Nortel network.

The CIO argues strongly for using as much standardized software as possible; the exchange’s Web site runs on Windows, and e-mail is using Microsoft Outlook.

“We use the standard Windows environment for all that, but not for our trading. On the trading side, there simply aren’t many packages,” he said.

“There are many for broker dealers, but a select for exchanges. There aren’t that many exchanges and, because of the custom nature of each exchange’s business, it’s very hard to find an off-the-shelf” package.

Having the network deliver all of those messages per second?Morgan’s people stress test their system with 200,000 messages per second?is only part of the battle.

After the messages are delivered, they have to be stored, catalogued and archived. These days, that’s about one-half billion messages every day.

All things considered, Morgan said, the storage is the easy part. “Data storage for us during the day is not the challenge. The challenge for us is retention,” he said. “This is more about cost, given our size and the challenges.”

At about 490 million messages a day, the government-required seven years of message retention adds up quickly.

The exchange handles messages in two ways, splitting them into recent messages (about three months’ worth) where the data needs to be readily accessible and the remainder that can be held in offsite storage about 25 miles away from headquarters.

Much of the data is managed in SANs (Storage Area Networks) with about 10TB of storage at headquarters.

The balancing act of when to have data no longer be so readily available is primarily a budget issue.

The data that is kept super-accessible costs a lot more to maintain. “The longer you wait for it, the cheaper the storage,” Morgan said.

The 120,000 messages a second are primarily being managed by some Sun Fire 6800 series servers. Morgan estimates that a typical day handles about 400 million quotes.

“They were the largest servers we could get at the time,” he said. “We try and leave as much spare capacity as possible. The key is to constantly be proactive, to be monitoring and measuring these systems. You have to always be watching, measuring.”

In the never-ending argument of whether it’s better to have a small number of big servers or a large number of medium or small servers, Morgan finds himself in the large server camp, opting for horizontal growth (adding CPU capacity to existing large servers) over vertical growth (adding more servers).

“It’s so much easier to plop a board in and run,” Morgan said.

The Philadelphia Stock Exchange is clearly focused on security, but it has advantages that not all companies have.

One key advantage is that the systems are only booted up from 7 a.m. to 4:15 p.m.

This sidesteps a typical problem that plagues many networks that need to run 24-7. Those systems must have extensive redundant systems in order for backups to be run and patches and updates installed. The fact that the Philadelphia systems only run during the day makes those issues a lot simpler to deal with.

Another examples is that the Philadelphia Stock Exchange offers no Web access to any operational systems.

“We’re not conducting business over the Internet. They can log in and get information that they need,” but no trades are permitted using the Internet, Morgan said. “Our trading systems for the most part are closed networks. No public domain. No one is logging in. It’s all dedicated point-to-point.”

The Philadelphia Stock Exchange is involved in stocks, futures and options trading and therefore competes with a wide range of financial markets in the United States and abroad.

But the bulk of its business is in options trading and it only has five rivals in that segment: the Boston Options Exchange, the Pacific Coast Exchange, the American Stock Exchange, the Chicago Board Options Exchange and the International Securities Exchange.

Compared with most of those other exchanges, Philadelphia’s IT operations are “very progressive and much more proactive than some of the others,” said Kristin Lovejoy, the chief technology officer for a data auditing firm called Consul Risk Management.

“I have worked with two of the other exchanges, and their attitude toward [security] compliance is much different.”

She gave a security example. Technically, the exchanges are not governed by Sarbanes-Oxley. Most exchanges do not comply with those regulations, but she said Philadelphia voluntarily does.

“Philadelphia gets it. They don’t have to comply, but they interpret it in the spirit of what Sarbanes-Oxley is all about.”

Lovejoy said that the Philadelphia Stock Exchange’s IT attitude is also reflected in its approach to technology audits.

“Their attitude toward audits is unique. They don’t want the auditors to drive them. They drive the auditors,” she said, adding that when auditors report problems with other exchanges, those other exchange IT departments “would scramble to implement the fix and to find the cheapest software possible to address the need. (The Philadelphia Stock Exchange’s IT people) instead look at the business overall to see if the changes make sense and the best way to make them happen.”

She compared the security audit to a house inspection. The typical exchange response to an inspection that tells of a specific leak in the corner of one room is to patch that leak.

The Philadelphia IT response, she said, is to trace the leak back to find where the water is coming from and fix the problem’s cause.

“Another of the things that Philly does that is much more effective than what others do is that they are very focused on the software change management process,” Lovejoy said. “They look at every change that is made to every production system.”

Beyond taking precautions such as capturing full system snapshots before any software change, the exchange has rigid procedures for installing any application or upgrade, to make it easier to roll back any changes if a problem crops up. “They look at security as including system availability,” Lovejoy said. “Others don’t.”

David Schehr, a Gartner research director, sees the short-term future of exchanges such as Philadelphia’s being very demanding on IT resources.

Philadelphia “will be facing a situation in a few years where they have to become more nimble, look for external partners and have a system flexibility that can sustain them in the long term,” Schehr said.

“So, for IT, it’s not just the ability to handle the volume today and tomorrow, but can the systems be set up in a way that can manage that” in the long term?

Schehr also referenced the New York Stock Exchange’s plans to purchase Archipelago Holdings?along with Nasdaq’s purchase of the Instinet Group?as proof that the markets are changing and that technology flexibility will be key.

“Both exchanges want to trade more than just equities. There’s going to be more fluidity in what’s traded,” Schehr said.

“During a week or week and a half in April, the announcement about Archipelago was made, and then a few days later Nasdaq made their announcement. It’s the first two rounds of a much bigger fight. Other exchanges are going to have to deal with that regardless of what they’re trading.”


Comments are closed.


StorefrontBacktalk delivers the latest retail technology news & analysis. Join more than 60,000 retail IT leaders who subscribe to our free weekly email. Sign up today!

Most Recent Comments

Why Did Gonzales Hackers Like European Cards So Much Better?

I am still unclear about the core point here-- why higher value of European cards. Supply and demand, yes, makes sense. But the fact that the cards were chip and pin (EMV) should make them less valuable because that demonstrably reduces the ability to use them fraudulently. Did the author mean that the chip and pin cards could be used in a country where EMV is not implemented--the US--and this mis-match make it easier to us them since the issuing banks may not have as robust anti-fraud controls as non-EMV banks because they assumed EMV would do the fraud prevention for them Read more...
Two possible reasons that I can think of and have seen in the past - 1) Cards issued by European banks when used online cross border don't usually support AVS checks. So, when a European card is used with a billing address that's in the US, an ecom merchant wouldn't necessarily know that the shipping zip code doesn't match the billing code. 2) Also, in offline chip countries the card determines whether or not a transaction is approved, not the issuer. In my experience, European issuers haven't developed the same checks on authorization requests as US issuers. So, these cards might be more valuable because they are more likely to get approved. Read more...
A smart card slot in terminals doesn't mean there is a reader or that the reader is activated. Then, activated reader or not, the U.S. processors don't have apps certified or ready to load into those terminals to accept and process smart card transactions just yet. Don't get your card(t) before the terminal (horse). Read more...
The marketplace does speak. More fraud capacity translates to higher value for the stolen data. Because nearly 100% of all US transactions are authorized online in real time, we have less fraud regardless of whether the card is Magstripe only or chip and PIn. Hence, $10 prices for US cards vs $25 for the European counterparts. Read more...
@David True. The European cards have both an EMV chip AND a mag stripe. Europeans may generally use the chip for their transactions, but the insecure stripe remains vulnerable to skimming, whether it be from a false front on an ATM or a dishonest waiter with a handheld skimmer. If their stripe is skimmed, the track data can still be cloned and used fraudulently in the United States. If European banks only detect fraud from 9-5 GMT, that might explain why American criminals prefer them over American bank issued cards, who have fraud detection in place 24x7. Read more...

Our apologies. Due to legal and security copyright issues, we can't facilitate the printing of Premium Content. If you absolutely need a hard copy, please contact customer service.