DEV Community

Sherotree
Sherotree

Posted on

Ask the agent for a Playwright assertion, not another manual QA checklist

#frontend #ai #testing #playwright

After a UI PR, agents (and humans) love dumping a fresh “manual QA checklist” into the description: click here, hover there, check mobile, verify empty state. Those bullets rot by the next sprint. Nobody re-runs them. Regressions still ship.

I’ve started asking for something sharper: one Playwright assertion (or whatever E2E runner the repo already uses) that fails without the fix and passes with it. Same interaction. Same locator style. No new checklist theater.

What “one assertion” means here

  • Cover the changed flow only — one interaction, not the whole suite
  • Reuse existing test helpers / page objects / fixtures
  • Fail without the fix; pass with it (prove it once locally or in CI)
  • Match the locator strategy already in the folder (getByRole, data-testid, whatever neighbors use)
  • Land in the same PR as the UI change

If the agent can’t point at an existing helper, that’s a signal the flow isn’t tested yet — still better than a checklist you’ll never open again.

Prompt shape that gets a real assert 

Add one Playwright assertion for the checkout zip-before-city fix.

Constraints:
- Extend `e2e/checkout.spec.ts` (or the nearest existing file)
- Reuse helpers from `e2e/helpers/checkout.ts` — do not invent a second page object
- One interaction: blur city with empty zip → expect the existing error copy
- No new test runner, no `page.waitForTimeout`, no screenshot-only checks
- Show the failing assertion output before the fix, then the green run after

If you need a new framework or a second suite, stop and explain why.
Enter fullscreen mode Exit fullscreen mode

The “failing output first” line matters. Agents will happily invent a green path that never exercised the bug. A red run without the fix is the receipt.

What I reject in review

Reject / send back

  • “Please manually verify…” as the only QA section
  • A brand-new test runner “because Playwright is cleaner” when Cypress (or the reverse) already owns CI
  • Flaky waitForTimeout / sleep-based waits
  • Screenshot diffs with no assert on the behavior you fixed
  • A 40-step checklist pasted from a template

Accept

  • One focused spec touching the changed flow
  • Locators matching neighbors in the same folder
  • Assertion on the user-visible outcome (error text, button state, URL)
  • Helper reuse instead of a parallel abstraction

Why this beats another checklist

Manual QA bullets feel thorough in the PR. They also:

  • Drift from the product the week after merge
  • Don’t fail CI when someone reintroduces the bug
  • Hide whether the agent actually understood the interaction

An assertion is boring and durable. The checklist is theater that ages into noise.

Tiny habit

Next time an agent (or a teammate) offers a manual QA list for a UI fix, paste the constraints above and ask for one Playwright assertion instead. If the draft invents a second framework or leans on sleep waits, stop — narrow the prompt and try again. The regression you care about usually fits in one expect.

Top comments (0)