Knight Capital: how a reused feature flag lost $460 million

On August 1, 2012, Knight Capital, the firm behind about a tenth of all U.S. stock trading, deployed new code to seven of its eight order-routing servers. The eighth still had code the company had retired in 2003, behind a feature flag the new code reused. In 45 minutes it executed 4 million trades and left Knight with a loss of more than $460 million. The best postmortem of it was written by the regulator, and every lesson in it still applies to anyone who ships with feature flags.

TL;DR

• Knight's router SMARS got new code for a NYSE program starting August 1. A technician copied it to 7 of 8 servers, by hand, and nobody checked.
• The new code reused the flag that used to switch on Power Peg, a function retired in 2003 but never deleted. On server 8, "yes" still meant Power Peg.
• Power Peg's stop condition had been moved in 2005 and never retested, so it sent orders without end: 4 million executions, 397 million shares, 154 stocks.
• 97 emails saying "Power Peg disabled" arrived before the open. They were not alerts, so nobody acted on them. The live fix, uninstalling the new code, made it worse.
• The SEC fined Knight $12 million in its first Market Access Rule case. Knight needed a $400 million rescue and merged with GETCO within months.

What happened at Knight Capital on August 1, 2012

The primary source is the SEC's administrative order, Release 34-70694, published October 16, 2013. It is ten pages long, numbered by paragraph, and almost entirely free of adjectives. The timeline below comes from it and from Knight's own filings on EDGAR (all times Eastern).

The order gives no exact minute for when staff intervened or when the orders stopped, only "approximately 45 minutes". The final count was more than $460 million realized, which the New York Times put at about $10 million a minute.

What is SMARS, and what was Power Peg?

SMARS is Knight's order router, which the SEC describes as "an automated, high speed, algorithmic router that sends orders into the market". It takes a customer's parent order and breaks it into child orders that go to exchanges. SMARS alone handled about 1 % of U.S. listed equity trading.

In 2012 the NYSE was about to start its Retail Liquidity Program (RLP) on August 1, and Knight wrote new SMARS code to take part. That code replaced something unused: Power Peg, a function Knight had discontinued in 2003 but which was, in the order's words, still "present and callable".

The SEC does not say what Power Peg was for, beyond one detail that matters. It had a cumulative-quantity counter, the check that says "the parent order is filled, stop sending children". In 2005 Knight moved that counter to an earlier point in the code and did not retest Power Peg (¶14). Nobody needed to. It was dead code. For seven years that was true.

How a reused feature flag woke up dead code

Here is the sentence at the centre of the incident, SEC ¶13:

The new RLP code also repurposed a flag that was formerly used to activate the Power Peg code.

The plan was that once Power Peg was gone, the old flag set to "yes" would switch on RLP. The deployment went out over several days, by one technician, with no second person reviewing and no written procedure requiring one (¶15):

one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment

So on August 1 the same flag meant two different things on two sets of servers. A simplified sketch of the logic, based on the order's description, not Knight's code:

# illustrative sketch of SEC ¶13-16, not Knight's source
if order.flag == "yes":
    run_rlp(order)         # servers 1-7: the new code
    # run_power_peg(order) # server 8: the old code was still here,
                           # and its stop condition was moved in 2005

On servers 1 to 7, RLP orders were handled correctly. Orders that reached server 8 with the flag set started Power Peg, which sent child orders "continuously … in rapid sequence … without regard to the number of share executions" (¶16). 212 parent orders turned into millions of child orders and 4 million executions.

The damage was not only Knight's. For 75 stocks, Knight's executions were more than 20 % of the trading volume and moved the price by more than 5 %. For 37 of them it was more than half the volume and more than 10 % of the price (¶18).

Why didn't anyone stop it for 45 minutes?

This was the first question on Hacker News the next day. salman89: "I don't understand how it went on for 45 minutes without human intervention. Do they really not have a live person at the very least monitoring trades?" The SEC order answers it in four parts.

The warning went to an inbox. From 8:01 a.m. an internal system sent emails about SMARS that "identified an error described as 'Power Peg disabled.' Knight's system sent 97 of these e-mail messages" (¶19). They went to a group of staff, were not designed as alerts, and nobody acted on them.

The limit was wired to nothing. Unmatched executions piled into a holding account, the "33 Account", which had a $2 million gross limit. That limit was not connected to any automated control (¶23–25). The risk monitor, PMON, was watched by humans, did not show the limits and lagged when volume was high.

There was no off switch. Nothing compared the orders leaving SMARS with the orders entering it, and "Knight also did not have procedures in place to halt SMARS's operations in response to its own aberrant activity" (¶21).

The fix made it worse. With no written incident procedures, the team, in a live market, did the reasonable-sounding thing and rolled back:

Knight uninstalled the new RLP code from the seven servers where it had been deployed correctly. This action worsened the problem

With the new code gone, the flag now triggered Power Peg on all eight servers (¶27). A rollback is only safe when the old version is safe, and here the old version contained the bug.

Root cause: git blame for Knight Capital

It is tempting to blame the technician who missed a server. The SEC order does not name him, and neither did I in the episode. Take him out and the same failure is one typo away on any other day. The causes that were actually decisions:

Any one of the first four alone might have been survivable. Together, with no kill switch, they had nothing left to stop them.

Blast radius: the fine, the rescue and the merger

Knight's shares fell 32 % on Wednesday and 63 % on Thursday, to $2.58 (NYT). On August 2 Knight said its "capital base has been severely impacted". On August 6 investors put in $400 million of convertible preferred stock, convertible into about 267 million shares, roughly $1.50 each. On December 19 Knight agreed to merge with GETCO, at $3.75 a share; the combined company, KCG Holdings, closed on July 1, 2013.

The SEC's action came fourteen months after the incident: $12 million, an independent consultant, and the first enforcement of the Market Access Rule, Rule 15c3-5, which since 2010 requires broker-dealers to have risk controls on the orders they send.

Feature flags and dead code: lessons from Knight Capital

Knight's mistakes are not about trading. Every one of them fits a web app with a feature-flag service and a deploy script.

• A new feature gets a new flag. Never give an old flag a new meaning. A flag name is an API; if any running code still reads it, it keeps its old meaning there.
• Delete dead code, don't disable it. Code behind a flag that is "always off" is still deployable, callable code. Remove the function and the flag together.
• Verify the deploy, not the intention. Before turning a feature on, check that every host runs the same build. An illustrative check:

  # illustrative: every host must report the same build before the flag flips
  for h in host-{1..8}; do ssh "$h" cat /srv/app/BUILD_ID; done | sort -u | wc -l   # expect 1

• Alerts page someone. Emails don't. 97 messages that said exactly what was wrong were not an alert because nobody had designed them as one.
• Limits must act on their own. A limit that a human has to notice is a report. Wire it to something that stops traffic.
• Every system that sends anything needs an off switch. Orders, emails, payments, webhooks. Knowing how to halt it should not require understanding it in the middle of an incident.
• Know what a rollback rolls back to. If the old version is also broken, rolling back spreads the bug.

Verdict: REVERT

I stamped it REVERT. Knight never published its own postmortem; the regulator wrote it fourteen months later, and the fix was a fine and a merger. What I'd ship on Monday instead: new feature, new flag; dead code gets deleted, not disabled; anything that sends orders gets an off switch.

FAQ

What caused the Knight Capital glitch?
New code reached only 7 of 8 servers. It reused a flag that on the eighth server still activated Power Peg, retired code whose stop condition had been broken since 2005.

How much did Knight Capital lose?
Knight reported about $440 million pre-tax on August 2, 2012. The SEC later put the realized loss at more than $460 million.

What was Power Peg?
A function in Knight's SMARS router, discontinued in 2003 but left in the code. The SEC order describes only its counter, the check that stopped it once an order was filled.

What happened to Knight Capital after 2012?
It took a $400 million rescue on August 6, agreed to merge with GETCO in December, and became KCG Holdings on July 1, 2013.

Sources

• SEC Administrative Order, Release No. 34-70694 (Oct 16, 2013): https://www.sec.gov/litigation/admin/2013/34-70694.pdf
• SEC press release 2013-222: https://www.sec.gov/newsroom/press-releases/2013-222
• Knight Capital 8-K, Aug 2, 2012: https://www.sec.gov/Archives/edgar/data/1060749/000119312512332176/d391111dex991.htm
• Knight Capital 8-K, Aug 6, 2012: https://www.sec.gov/Archives/edgar/data/1060749/000119312512336167/d392288d8k.htm
• Knight and GETCO merger, Dec 19, 2012: https://www.sec.gov/Archives/edgar/data/1060749/000089882212000673/ex991.htm
• NYT DealBook, Aug 2, 2012: https://archive.nytimes.com/dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/
• Hacker News, Aug 2, 2012: https://news.ycombinator.com/item?id=4329101
• Hacker News, 2022 discussion: https://news.ycombinator.com/item?id=31239033

This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.