In April we opened a second region, in Frankfurt, for customers whose data has to stay in the EU. The infrastructure was code and the launch checklist had sixty items, all ticked. On the first morning a Frankfurt customer asked support why their invoices had a pay in instalments button. We had not finished that feature, and we had not announced it.
Our feature flags live in a flag service, one per region. Each flag has a default, which is what the service returns when no rule matches, and rules that turn it on or off for particular accounts or percentages. The new region's flag service was filled by a script that copied every flag definition from the first region. It copied definitions and not rules, deliberately, because our rules name account ids that do not exist in Frankfurt.
What nobody had looked at was how often a definition's default disagreed with what customers actually got. Over two years developers had created flags with a default of on, because that was convenient while working locally, and then added a production rule switching the flag off for everyone. In the first region the rule won. In the new region there was no rule, so the default won.
Thirty one of our three hundred and twenty flags were like that. Eight were half built features. One switched off a fraud check, a setting added for a load test and never meant to be seen by a customer.
We turned all thirty one off in Frankfurt within the hour and confirmed that no instalment agreement had been completed. Then we changed what a default means. It is now the value for an environment nobody has configured, so it has to be the safe one, and new flags default to off. Local development reads its values from a separate file, never from the default. A weekly report lists every flag whose default disagrees with what production serves, and its owner has to change one or the other. Region setup copies rules as well as definitions, mapping accounts where they differ. And before any launch, a job lists every flag whose effective value differs between regions, and support reads it.
Launching a region is the one occasion when a system runs on nothing but its defaults. We had spent two years overriding ours and had never once read what they said.
– Sergey Shinder
Top comments (0)