There’s a lot of talk about pull requests being reviewed and deployed automatically, with no human in the loop. This week I tried to answer the question that idea leaves open for me: which PRs?
Which PRs can skip human review
The answer depends on two things: what guardrails a project already has, and how critical it is.
Before even considering fully automated review, I’d expect a project to have at least:
- a type checker, for dynamically typed languages;
- linters and formatters;
- checks for cognitive and cyclomatic complexity;
- a good test pyramid: unit, integration, component, smoke and end-to-end tests;
- a combination of static tools and agent skills that keeps a minimum of organization in the code;
- a CI/CD pipeline that works and is fast;
Which pages still need an editor?
Even with all of that, projects differ. To think about criticality, I borrowed the idea of service tiers from OpsLevel, a service catalog platform:
- Tier 0: business-critical. Downtime can stop essential operations. Needs high availability and fast recovery;
- Tier 1: downtime affects important business flows.
- Tier 2: degradation affects secondary features. Longer recovery periods are tolerable;
- Tier 3: internal services, auxiliary tools or non-critical workloads. Lower availability and support requirements;
Tiers 0 and 1 are clearly out of the question for 100% automated review and deploy. Tiers 2, and especially 3, can be handled that way without major consequences. Something broke? Roll back, fix it and release it again.
OpsLevel’s tiers only describe availability, though. A bug doesn’t have to take a service down to cause a disaster. So I defined a second classification, for the impact of errors:
- Tier 0: errors can cause significant financial loss, expose sensitive data or violate regulations;
- Tier 1: errors affect important business flows, but the impact is limited or recoverable;
- Tier 2: errors have low impact and can be fixed without relevant consequences;
Given the basic guardrails above, my view is that services in OpsLevel Tiers 2 and 3 that are also Error Tier 2 could have review fully automated. The others, no.
I like these classifications because they make me ask concrete questions. For example, what would make me comfortable with a fully automated review and release for an OpsLevel Tier 1 service? Performance tests? Performance benchmarks? Concurrency tests?
All of this gets much harder in large services or monoliths. When everything is split into separate repositories, it’s far easier to apply the tiers and decide what can be reviewed and deployed automatically. Inside a monolith, especially a messy one with no modules organized around the business (which is the rule), it’s a very different conversation.
Modern monoliths are like this house of cards: somehow, it holds.
One more guardrail applies even to small services: checking that existing tests unrelated to the change weren’t modified. Without it, a change can pass CI because the test that would have caught it was edited. Maybe the code of test cases should always be reviewed by a person.
Paying down debt with parallel agents
This week I upgraded several services to Django’s long-term support releases. One went two steps, from 3.2 to 4.2 and then to 5.2. Another went straight to 5.2.
While doing the upgrades, I found a gap in our test suite and started closing it by adding component tests to the services I was touching. Doing both at the same time proved pretty manageable with tmux, several Claude Code instances and git worktrees. Each worktree is a separate checkout of the same repository on its own branch, so every agent had its own folder to work in, and tmux kept all the sessions open side by side in one terminal. While one instance worked on an upgrade, another wrote tests, and I moved between them reviewing.
For me, that’s a very good example of how AI is an extraordinary tool for paying down technical debt and keeping the lights on.
New releases of these services can now rely on the new tests, which removes part of the testing bottleneck.
By the way, I didn't have much context one some of those services, so building the tests was a very effective and productive way to learning how things work under the hood.
Parallel effort, shared payoff.
Still wondering
- What would make a fully automated review and release safe for a Tier 1 service? Performance tests, benchmarks, concurrency tests, or something else? Canary release + feature switches and or flags?
- How do you apply tiers inside a monolith that has no business-oriented modules?
- Should changes to test code always get human review, even when everything else is automated?
Let's see what the future holds.
Top comments (3)
The second classification is what makes the tiers usable for review, because a bug that exposes data or charges the wrong amount does not need to take anything down. Inside a monolith without business modules, though, the error tier cannot come from where a file lives. One helper can serve checkout and an internal export at the same time, and its folder says nothing about which of the two matters.
One way to get the tier without reorganising the code is to let the tests assign it. Tag each component or end to end test with the error tier of the flow it exercises, run the suite once with coverage recording which test ran each line (in coverage.py that is
dynamic_context = test_function, and Django test methods qualify because their names start with test), and keep the map from line to tests. A pull request then gets the most critical tier of any test that executes a line it changed. The same three line change comes out Tier 2 in a function only the export test reaches, and Tier 0 in a function the checkout test runs through, even when both sit in the same file.Two things follow, and the second is an answer to your third question. A changed line that no test executes gets no tier from the map, so it should default to Tier 0, since nothing in the suite would catch a mistake there. And an edit to an existing test changes the map itself, so the map would be grading a change to its own rules. That makes sending test edits to a person a mechanical rule rather than a cautious one.
Do the component tests you added this week each follow one business flow, or does a single test walk through several?
Good insights @izgorodin
It's one feature/workflow per component test. Regarding this capability of identifying which test relies in which LOC x LOC changed by the pull request, are you doing it in any project you are working on? if yes, are you using any specific tools for doing it or did you built it yourself?
I ask specially because this is really useful in monolith, which usually have slow CI suites, so I guess doing such cov calculation can be pretty slow as well. Some sort of caching perhaps? anyway, I would like to know more about it.
Léo, the coverage run does not need to happen on every pull request. Recording which test ran each line is the slow part, so it can run on main after each merge, and the pull request only looks up the lines of its diff in the stored map, which is a query rather than a test run. coverage.py keeps that map in its .coverage file, a SQLite database, and CoverageData.contexts_by_lineno returns the tests for each line of a file. For choosing which tests to run rather than grading the change, pytest-testmon keeps a similar map, though it needs the suite to run under pytest.
A map built on main lags in two ways. Lines a pull request adds are not in it at all, and sending all of them to the Tier 0 default would grade almost every pull request Tier 0. A new line inside an existing function can take the tests of that function's other lines, so a line falls to Tier 0 only when nothing around it is mapped, as in a new function. The other lag is the unsafe one: a merged change makes the checkout flow start running a helper that only the export test reached before, and until the map is rebuilt, a pull request that edits that helper is still graded Tier 2. Rebuilding on every merge to main, rather than nightly, keeps that window to the length of one rebuild.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.