DEV Community

Cover image for Parity Against a Moving Baseline
Anton Brilliantov
Anton Brilliantov

Posted on

Parity Against a Moving Baseline

#architecture #testing #microservices #debugging

Cover reading Parity Against a Moving Baseline, with the word Parity on a vermilion slab, the figure 9 parity runs in one day, and four bars of cascade rows for the same input, 2, 258, 2 and 8, with the 258 bar red

Comparing a new implementation against an old one is only a test while the old one stands still.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about checking a migration while both sides keep moving - and about response codes that describe the wire rather than the storage. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

Where the boundary was drawn

The migration moved computation out of the monolith before it moved data. For a stretch of about seven weeks that produced a split that looks reasonable in a diagram and is very hard to check:

  the monolith                              the extracted service
  ├── works out the composition             ├── owns the identity of a rule set
  ├── asks the service for identity  ────►  ├── answers over the wire
  └── mirrors the row into its own DB       └── knows nothing about the mirror

                    no shared transaction between them
Enter fullscreen mode Exit fullscreen mode

Two panels: the monolith works out the composition, asks for identity and mirrors the row; the extracted service owns the identity, answers over the wire and never sees the mirror; between them a dashed red line with a bolt reading no shared tx, and a stamp: 4 tasks, one shape, 18 edits

One operation - work out what the composition should be, and save it - now spans two processes and two databases with no shared transaction. Over 49 days that boundary produced four tasks of exactly one shape: the monolith computed the right thing and something else got stored. In the two weeks right after ownership moved, the old system took 18 more edits on this subject alone.

The series system map with the PHP monolith, the rule-set-markup service, the domain-rule-map service, the Go API gateway and the acceptance repository lit, and a red note on the monolith: the reference, being fixed

The response code describes the wire

The first thing a parity check leans on is the response. Three cases from this period, all of them real:

  • 502 after commit - the row was written; the caller retried a completed write.
  • 200 completed - the row was silently deleted; the run counted it as success.
  • 504 at 20.26 s - the write went through, and the cascade kept working 8.7 s after the answer.

Three rows from response code to storage: 502 after commit, the row was written; 200 completed, the row was deleted; 504 at 20.26 s, the write went through; a green note reads assert the read-back, not the response

None of these is a lie by the transport. A code honestly reports that the call arrived, the body parsed and the request's own transaction closed. With two storages and no shared transaction, that is all it reports. So the rule that went into the acceptance instructions and has stayed there:

Assert on the persisted read-back, not on the write response. A green answer is not a green database.

The baseline was being fixed at the same time

Now the harder problem. The new write path was accepted by comparing it against the old one - byte for byte, field for field. That is a sound method exactly once: when the reference is frozen.

Ours was not. On one day the same input went through nine iterations, and the size of the cascade the two sides produced went:

  run 1 ─────► 2 rows
  run 4 ─────► 258 rows        same input
  run 6 ─────► 2 rows
  run 9 ─────► 8 rows          settled — for that day
Enter fullscreen mode Exit fullscreen mode

Animated bars of cascade rows for the same input across one day: run 1 at 2, run 4 at 258, run 6 at 2, run 9 at 8, with a note that the reference changed its definition and a stamp reading same input

And the diff itself, one object, both sides in the same run:

  the old implementation                        the new implementation
  {                                             {
    "rule_set": { "id": 1806235194000891392 },    "rule_set": { "id": "1806235194000891392" },
    "scope_rule_set_id": "9412008365512340992",   "scope_rule_set_id": "9412008365512340992",
    "priority": null,                             "priority": 0,
  }                                             }
Enter fullscreen mode Exit fullscreen mode

A dark diff card for one object, old lines marked minus and new lines plus: the rule set id a number on one side and a string on the other, the scope id the same, priority null against 0; notes read a number here, a string there, and zero and absent

Two differences, two entirely different causes. The identifier being a number in one field and a string in another is the old side being inconsistent with itself. The null against 0 is the contract: on the wire, zero and absent are the same thing, so a real value of zero cannot be told from no value at all.

A machine can produce that diff in a second. What it cannot produce is the sentence "the right-hand side is correct here and the left-hand side is correct there".

Every one of those runs was honest work. The swing came from the reference itself changing definition mid-day: what counts as a sibling in the cascade, whether inactive objects belong in it, what the scope of a client-level write is. And the reference was internally inconsistent too: the same identifier came back as a number in one field of one response and as a string in another.

Three times out of nine, the new implementation turned out to be the correct one. Which is the real lesson of the day: when you name a reference, you are also declaring it correct - and that declaration needs to be true.

A green number tile reading 3 of 9 runs where the new side was correct, next to the words a reference is also a claim that it is correct, stamped the reference was wrong

A green suite that could not fail

The same period had a quieter version of the same mistake. The write path had a switch selecting which implementation runs. Production ran the new one. The test environment's config file still pointed at the old one.

Two halves: the test environment with engine equal to old, green every run, against production with engine equal to new, never touched by the suite, stamped proved nothing

So the suite was green, permanently, and green meant nothing: it exercised a code path production had stopped using. One flag, set differently in two places, for weeks. A switch that selects an implementation belongs in the test matrix, not in an environment file.

And then the comparison stopped existing

The end of the parity story is worth telling in full, because it is not a failure - it is a cost.

Partway through, the owner decided the old write endpoint would be removed entirely: one door, one write path, no dual entry. Correct decision, and it simplified everything downstream. It also erased, in one line:

Animated list of four erased items, each crossed out in turn: 9 parity iterations from one day, the cumulative contract diff field by field, every stopwatch number of the old path, the class the two disagree; a stamp reads erased and a note: missing, the date it takes effect

  • nine iterations of parity work from a single day;
  • the cumulative field-by-field contract diff between the two answers;
  • every timing measured on the old path;
  • and the whole defect class "the two implementations disagree", which stopped being a defect class at all.

What was missing was not the decision. It was the date it would take effect, agreed in advance. A comparison is worth its cost only while both sides are going to keep existing; the moment one is scheduled for removal, the parity work has a known expiry and can be scoped to it.

The real bill: finding out what was wanted

Add it up and the shape of the period is not "we wrote buggy code":

Five number tiles of the real bill: 23 requirements with two readings, 7 reversals already built, 15 briefs discarded in a day, about 13 of 27 iterations spent finding out, and 5 of 6 reversals where the code matched its spec

  • 23 requirements that had two defensible readings;
  • 7 reversals of decisions that were already implemented;
  • ~13 of 27 iterations spent working out what was wanted rather than fixing code (an estimate over intervals, not a measurement);
  • 15 iterations of briefs written and discarded within a day;
  • and in 5 reversals out of 6, the code was correct against the specification it was written against.

That last number is the one I keep. The implementation was rarely the problem. The requirement moved, and nothing moved with it.

A torn paper receipt for a reference that kept moving: 9 parity runs in one day, erased; 15 briefs in one day, discarded; about 13 of 27 iterations finding out the ask; code matched its spec in 5 of 6, stamped the spec moved

What keeps them in step

None of the remedy is exotic. It is a loop, run per iteration rather than per release:

Four green tiles in a loop - freeze it in words, write the check first, run it on everything, re-sync spec, check and code - with a dashed return arrow and a note: and write when a comparison expires

  1. Freeze the requirement in words, with the owner of each value named.
  2. Write the check before the code, in two halves: what proves the value, and which reads do not prove it.
  3. Run it against the live system, on everything rather than on a sample.
  4. Re-sync deliberately: when the requirement moves, the check and the specification move in the same change, or the requirement hasn't actually moved.

The one addition the multi-system case forces: when a comparison is part of the method, write down when it expires. Which side is going away, and on what date.

What automation settled, and what it handed back

Nine iterations in a single day are only possible because the run itself is cheap - generated, executed and folded by an automated executor. Worth splitting precisely what that bought.

Two tiles: mechanical and automated - the matrix per object, all 392 objects, a field-by-field diff, again on every build; a decision and still ours - which side is right, who owns the value, when it expires, broke or moved

Mechanical, and automated end to end: building the request matrix per object, running all 392 rather than a sample, folding both answers into a field-by-field diff, repeating the whole pass on the next build, attaching the trace contents to every failure so the report outlives the retention window.

A decision, and still ours: which of the two sides is entitled to be right; who owns a value the two disagree about; when the comparison expires because one side is being removed; the observation that a switch was set differently in the test environment and in production; and the distinction that mattered most - did the build break, or did the requirement move?

The last one is the whole reason this period cost what it did. To an executor, a requirement that changed overnight and a regression look identical: yesterday's expectation, today's mismatch. It will report both with the same confidence and the same formatting. Deciding between them is not a gap in the tooling - it is the job the tooling exists to make room for.

So the honest framing of the multiplier: fast iteration does not settle what "correct" means, it multiplies whatever answer is currently written down. Nine cheap runs against a definition that changed by evening produce nine artefacts and no verdict. Freeze the definition, name the owner of each value, give the comparison an expiry date - and the same automation stops producing noise and starts producing evidence.

A four-panel strip: what we did, 9 runs of parity against the old path in one day; what broke, 2 to 258 rows for the same input; why, the reference was being fixed; what now, an expiry written for every comparison

Anatomy of a service split — Part 4. Next: the cascade that does not fit inside a response - a constant deadline against a cost that grows with the number of children, and a write that succeeds after the caller has already been told it timed out.

If you do this better, tell me how you keep a reference frozen while it is still being changed. If you have been through this, what did your green suite turn out not to be testing? If you see it differently, say where parity against an old implementation earns its cost. How is it solved on your side, and what broke there?

Top comments (0)