Exchange Outages Are Going Mainstream: What Robinhood Can Learn From Crypto

Both crypto exchanges and popular online trading platforms including Schwab, TD Ameritrade and Robinhood have a rising number of young investors who, working from home during the coronavirus pandemic, spend some of their work hours trading for their own personal accounts.

However, these platforms have another thing in common: outages in the midst of high volume.

A spokesperson from TD Ameritrade acknowledged “high levels of slowness” some users experienced on its web and mobile platforms but did not offer an explanation of the cause. As of press time, Robinhood and Schwab did not respond to inquiries from CoinDesk.

Robinhood apparently is not alone during a time when a growing number of new and young investors are betting their money on different markets, including cryptocurrencies, by using online brokers amid the coronavirus pandemic.

Like traditional platforms, crypto exchanges have been troubled by outages for a long time, even after they pledge to take more steps to improve stability and reduce outages. These mainstream companies may be able to learn something from the experience of crypto exchanges.

The need for redundancy

After suffering a severe service outage in late August, Deribit, the most popular cryptocurrency options exchange, told CoinDesk it is working to enhance its platform to avoid this happening again.

“Our platform uses redundant load balancers to connect to multiple nodes, gateways to the platform, connecting to a single master node,” Luuk Strijers, chief commercial officer at Deribit, told CoinDesk via Telegram on Aug. 27. “Today we experienced a hardware failure in this master node.”

The problem was resolved when engineers successfully activated one of the regular nodes as the exchange’s new master node. The company will work on speeding up this procedure, Strijers said.

Strijers added that Deribit is in the process of setting up a disaster recovery facility in Zurich to act as immediate failover if multiple nodes were impacted. This, he said, should dispel doubts around the exchange’s redundancy provision.

Setting up a server location in Zurich does not mean the company will have to adopt any new know-your-customer (KYC) and anti-money laundering (AML) requirements in Switzerland, Strijers clarified. (The Dutch exchange’s infrastructure is hosted in the U.K. but its operations are now in Panama as part of DRB Panama Inc., a wholly owned subsidiary of the Dutch entity, created in early February.)

When code becomes a problem

It is not the first time a crypto exchange has sworn that some fundamental improvement it makes will avert new outages.

Earlier this year, Binance, the world’s largest crypto exchange by trading volume, suspended trading for more than six hours due to a “system messaging error.” Coinbase angered its users in May after it was forced to shut down its service due to a traffic spike.

Dave Weisberger, co-founder and CEO of execution provider CoinRoutes, told CoinDesk in a phone interview there are two main causes of technical outages at crypto exchanges.

One is a hardware failure, which was the problem that occurred at Deribit; the solution is to build a redundancy system. By now, most exchanges have built fully redundant systems, according to Weisberger, and as a result any outages caused by hardware failures are usually short-lived.

The other cause, which is more common, is a change in a new piece of code that was not thoroughly tested. Bugs in the new code can be triggered at a later time by an unplanned situation such as a surge in trading volumes, resulting in an outage.

Traffic: When too much of a good thing becomes a bad thing

Derivatives exchange FTX’s support team also told CoinDesk via email that to reduce the risk of outages, their work has been concentrated on making sure enough spare capacity will be available to support the exchange’s operation during busy periods.

Tushar Jain, managing partner at Multicoin Capital, told CoinDesk via Twitter that reducing outages caused by sudden traffic increases on exchanges is “doable,” but it will require time and money.

“Building software which scales to serve so many users is really hard and the operational work to make sure servers stay up and running is quite difficult,” he said. “There are many examples of software companies having trouble scaling to serve extremely high demand. Twitter’s “Fail Whale” is probably the most memorable example.”

Circuit breakers as a solution?

Circuit breakers, which were first implemented on stock exchanges after the “Black Monday” crash in 1987, are automatic stoppages put in place when prices fall below specified levels. They are designed to save the market from a complete meltdown.

Deribit already has an index circuit breaker on its platform which is triggered at +/-1.5% index price move per second to “avoid massive sell-offs, and allow market participants to get up to speed with the market during highly volatile periods,” according to Strijers.

“In the past, multiple derivatives exchanges have experienced flash crashes that have caused a cascade of liquidations and massive sell-offs,” he said. “Reasons have been various: an external market manipulation or internal error. To avoid this from happening, Deribit introduced a form of circuit breaker.”

Deribit’s circuit breaker was triggered a few times during the night of March 13 when bitcoin prices dipped to a 12-month low.

However, with hundreds of crypto exchanges available, the introduction of circuit breakers could hinder an individual exchange’s performance when its service is down for a period of time. During Binance’s outage earlier this year, for example, rival exchanges including OKEx and Bitstamp saw big jumps in trading orders. 

FTX told CoinDesk it is not currently considering what it describes as “hard circuit breakers,” which would limit its users’ abilities to trade at some prices in the long term.

“These make a lot less sense,” the exchange wrote. “Rather than acting as a sanity check, they restrict users’ ability to trade and enforce artificial pricing.”

Jain, who previously was an advocate of circuit breakers, told CoinDesk he now doesn’t think this measure would solve exchanges’ outages. While prices for bitcoin and ether have been “relatively stable” recently, he said, outages still occur on exchanges.

Jain even interprets it as a positive sign: In a more stable market, outages caused by traffic spikes mean that more people are using crypto exchanges.

“I think this goes to demonstrate the level of demand in the crypto markets right now,” he said. “The last time I remember exchanges having problems like this was in early to mid-2017 when their servers just couldn’t keep up with user growth.”

Crypto exchanges’ unregulated downside

Some of the larger crypto exchanges may implement needed changes but without the threat of penalties from regulators if problems aren’t fixed, fundamental improvements are less likely to occur anytime soon.

“The fact is that when you’re not penalized for these sorts of things, then you don’t spend as much money trying to fix it or prevent it,” Weisberger said.

Weisberger pointed out another similarity between mainstream trading platforms and the crypto exchanges: the ethos of the Silicon Valley or, rather, the whole of the tech industry. The people behind these platforms prioritize issues like liquidity and transaction fees rather than reducing outages simply because the financial cost outweighs the benefit.

“Is an uptime requirement of 99.999% something that the same type of people who invented Robinhood are going to aspire to?” Weisberger said. “The answer is no. They say they aspire to it but that’s very expensive. … As a result, there are outages.”

Robinhood, which is more heavily regulated than the crypto exchanges, is now reportedly under investigation by the U.S. Securities and Exchange Commission and the Financial Industry Regulatory Authority for its handling of an outage in March.

If it gives them incentive to keep outages from repeating, regulatory oversight may end up being an asset for mainstream online trading.