DEV Community

Cover image for A Service That Cannot Explain Itself Can Only Be Guessed At
Anton Brilliantov
Anton Brilliantov

Posted on

A Service That Cannot Explain Itself Can Only Be Guessed At

#architecture #testing #observability #microservices

Cover reading A Service That Cannot Explain Itself Can Only Be Guessed At, with the word Explain on a vermilion slab, the figure 210 of 276 had no active set, and two panels: 3 objects, all pass, against 210 of 276, empty

If a service cannot state what its answer should be, the only way to check it is to guess.

👋 Hi, I'm Anton - a software engineer working mostly in PHP/Symfony and Go, currently carving a live PHP monolith into Go services. This series is about one write path that left the monolith and what it took to accept it as working. This part is about the thing that made checking it possible at all: read endpoints whose only job is to describe state. Running notes live on my GitHub: github.com/brilliant-almazov.

This is how I do it right now, with the price attached - maybe you already do it better, maybe you'd do it another way.

The path in one paragraph

A user saves a set of matching rules; every domain under that account gets marked by them. Four participants: the Go API gateway as the only door, the rule-set-markup service which takes the write and folds the cascade, the domain-rule-map service which owns rule-set identity and recalculates the map, and PostgreSQL behind them - the monolith's database for the rule data, the identity service's own for identity.

We knew our input exactly: the rules the user sent. What nobody could state was the expected output: given this input and this account, what should the resulting map contain, and how would we know the answer was complete rather than merely non-empty?

The series system map with the acceptance repository, the Go API gateway, the rule-set-markup service and the domain-rule-map service lit, the UI and the PHP monolith dimmed, and a note on the acceptance repository: four reads, one sweep

What a three-object test hides

The first weeks of checking ran against a playground of three objects. Everything passed. The runs were honest, the assertions were real, and the conclusion was worthless - three objects cannot show you anything that depends on the shape of the data.

Then a run went across everything the environment held:

  playground              3 objects        everything passes
  full sweep            276 objects        active-set registry EMPTY for 210 of 276
  later sweeps          392 objects        392 of 392 covered, 129 map rows of 2 329
                                           carrying neither rule set nor priority (5.5 %)
Enter fullscreen mode Exit fullscreen mode

Three bars of coverage: a playground of 3 objects, a full sweep of 276 objects with a note that the registry was empty for 210 of 276, and later sweeps with 392 of 392 covered

Animated grid of 276 cells filling red one by one up to 210 objects with an empty registry, next to the counters 210 empty registry and 276 objects swept

Nothing was fixed between the first two lines. The system was in exactly the same state; what changed is that it could now be asked. 210 of 276 is not a defect that appeared - it is a defect that was already there, in production data, invisible because no endpoint would tell us.

The endpoints that made the difference

Four reads, none of them user-facing features, all of them ordinary functionality the services already had internally:

  • the input snapshot for an account - which domains are in scope at all, and how many.
  • the rule set in effect, with its rules - what is supposed to be applied.
  • the pointer to the active set - which set the engine believes is current.
  • the stored map, page by page - what was actually written, per row.

Four read tiles - input snapshot, rule set in effect, active pointer, stored map paged - each with an arrow down into one green tile: expected result, computed locally, then compared to the map and the engine

With those four, checking stops being an opinion. You read the snapshot and the rules, compute the expected result locally, then compare it to the stored map and to what the engine answers live:

  snapshot            8 693 domains in scope
  rule set               87 rules
  expected, computed     87 marked
  stored map             87 marked
  engine answer          87 marked
  ───────────────────────────────────
  mismatches              0            ← a verdict, not an impression
Enter fullscreen mode Exit fullscreen mode

A dark terminal card for one account: snapshot 8 693 domains in scope, rule set 87 rules, expected 87 marked, stored map 87 marked, engine answer 87 marked, and the highlighted line mismatches 0, with a note: a verdict, not a guess

This is the whole argument of this part. The expected result is not something a checker invents; it is something the system has to be able to state. If the only endpoint is "do the work", then every verdict about it is a guess dressed as a test.

Two ways this goes wrong when the surface is missing

Both of these are mine, both cost real days.

A green result that never ran. A probe sent a single-value field to an endpoint that requires a list of ids. The endpoint answered 422; the probe took the empty decision list as "nothing disagreed" and printed PASS 24, FAIL 3. Two builds were accepted on a check that had performed zero comparisons. A false FAIL creates work; a false PASS silently removes the check.

A dark log card of the probe: a single value sent where a list of ids is required, a 422 answer, an empty decision list, comparisons made 0, and the highlighted line PASS 24 FAIL 3, with a note: two builds accepted

The rule that came out of it is one line: an empty selection is a FAIL, not a PASS, and every probe carries a counter of comparisons actually made, non-zero by construction.

Two halves: on the dark side what the report said, PASS 24, FAIL 3, two builds accepted; on the red side what happened, 0 comparisons and a 422 answer; a stamp reads empty = fail

The right-looking read of the wrong thing. The set attached to a configuration was read from a listing endpoint that quietly ignores the filter it was given - a request for one account returned a hundred rows belonging to other accounts. The probe printed a confident, entirely fictional finding. Five days later the same misreading produced two phantom defects, one of which was filed as a comment on another team's ticket before being withdrawn.

That one costs a section in every spec since, in two halves:

  • positive definition - which endpoint returns this value, by contract;
  • negative definition - which reads do not prove it, and why.

Two tiles: the positive definition, the endpoint that returns the value by contract, one object asked and its own value returned, stamped proves it; the negative definition, a listing that ignores the filter, one account asked and a hundred rows of others returned, stamped proves nothing

This is functionality, not scaffolding

The tempting way to read all this is "add some debug endpoints". That framing is why they never get built.

Every one of those four reads is capability the service already has. Resolving the active pointer, listing rules, paging the map - it does all of it internally on every request. The only question is whether it is exposed. A service that keeps it inside is not simpler; it is a service whose behaviour can only be inferred.

Where the endpoint lives is a detail, and the detail is smaller than it looks: it can sit behind the same gateway as everything else, under an operator role, under normal rights, or on a separate admin surface. What matters is that the capability is presentable, and that it is treated as part of the product rather than as something added while chasing a bug. The same reads that let a checker compute an expectation are what an operator wants during an incident and what a support answer needs when someone asks why one account looks different.

Two design notes that came out of using them in anger:

  • Read endpoints must page and count honestly. A page of five rows out of eighty-seven that says "there is more" while giving no cursor to continue with is not a diagnostic - it's another guess. That was its own defect and it is still open.
  • Absence must be answerable. An account with no data should return an empty, successful answer, not an error. Modelling "nothing here" as a failure means every sweep has to guess which failures are real - we had objects answering three different codes for one state.

Four guarantees of a read endpoint: paging that continues, open, 5 of 87 rows and no cursor; counts that mean something, in place; absence as absence, open, one state answered with three codes; state per object, in place; a stamp reads two still open

And the reason to treat these as requirements rather than nice-to-haves: everything the sweep found came out of these reads, not out of running the feature.

Animated number tiles appearing one by one: 210 of 276 with no active-set pointer, 129 of 2 329 rows without a set, 5.5 %, 3 to 392 objects under the sweep, 0 mismatches on one account, 2 phantom defects withdrawn

Today an executor, tomorrow the pipeline

Right now the sweep is driven by an automated executor: it reads the four endpoints, computes the expectation, compares, and writes a report with the numbers and the trace contents attached. That is a stage, not a destination. The same four reads and the same comparison belong in a pipeline against a stand - a scheduled run, or one run per deploy of the services involved.

The four read endpoints feeding three consumers: a person in an incident, an executor today, and a dashed pipeline tomorrow

Conceptually nothing changes between those columns; only the thing holding the clipboard does. Which is exactly why it is worth building the surface first and arguing about the runner later - the surface is the durable part, and it is useful to a human during an incident whether or not anything automated ever consumes it.

What it costs

Being honest about the price, because there is one:

  • Endpoints have to be maintained. A read that describes internal state drifts with the internals; a stale describe-endpoint is worse than none, because it is believed.
  • They widen the surface. Rights, rate limits, and "who may see how many rows of whose data" all have to be answered - the same answers as for any other endpoint, but they must actually be given.
  • Computing the expectation locally is real work. It means writing down the rule of the system in a second place, which is a second thing to keep true.

Against that: the sweep of 276 that found 210 empty registries took minutes once the reads existed. Before they existed, the same finding was unavailable at any price.

A torn paper receipt for checks without a surface: probe on a 422, 2 builds, 0 compared; listing with no filter, five days, 2 phantoms; manual sweep, two hours, one pass; price 2 builds, with a stamp reading guessed

The multiplier line

An executor turned a two-hour manual sweep into a repeatable one, and made "run it again on the next build and print the difference" cost nothing. It could not invent the four reads, and it could not tell a genuine mismatch from a probe reading the wrong endpoint - it printed both with equal confidence. What made the automation worth anything was the surface underneath it and the definition of where each value legitimately comes from. Fast is only useful once correct is defined.

A four-panel strip: what we did, 3 objects in the playground, all passed; what broke, 210 of 276 with an empty registry; why, no reads, nobody could state the answer; what now, 0 mismatches, read by read

Anatomy of a service split — Part 2. Next: the part before all of this - the analysis that was not done. One value with four homes and no owner, and how long it takes to find that out when nobody wrote down who is allowed to write it.

If you do this better, tell me what your services expose for checking their own state. If you have been through this, what did your first full sweep turn up that the sample never did? If you see it differently, say where a describe-your-state endpoint costs more than it returns. How is it solved on your side, and what broke there?

Top comments (0)