When I started learning about test automation, I used to think: If we can automate a test, why not automate it?
But the more I look at real automation projects, the more I feel that not every test needs to be automated.
A test that we run repeatedly for every release makes sense to automate. But a test that changes often, is used only once, or needs exploratory thinking may not be worth maintaining as automation.
So the question becomes: Can we automate this?
but also: Will this automation give us useful feedback?
While working with X360 AI Tech, One thing we focus on is what happens after the automated test runs.
If a test fails, we don't want the result to stop at a red mark.
We can look at what actually happened during the execution - the browser actions, assertions, and network activity - along with the live execution video, logs, and execution history. This gives us the context around the failure instead of looking at the failed status alone.
Then the failure-analysis part helps us investigate that execution and understand what needs attention.
For example, if a checkout test fails, the important question isn't simply: Did the test fail?
We want to look at where it failed, what happened before the failure, and what the execution evidence is telling us.
That distinction matters because a failed test doesn't automatically mean there is a product bug.
It could be a changed application behaviour, a test issue, or something else in the execution environment.
That's what changed my thinking about automation.
The goal isn't to automate every test.
It's to automate the tests that are worth repeating and when something fails, have enough context to understand what actually happened.
That's where I think automation becomes much more useful.
Top comments (6)
Đúng vậy — automation là công cụ, không phải mục đích. Tôi từng thấy team viết hàng trăm UI test cho flow đăng nhập/đăng ký, chạy 45 phút mỗi lần merge, flaky rate 30%, cuối cùng dev tắt báo cáo đi vì "chán nhìn". Chi phí maintain gấp 3-4 lần giá trị test mang lại.
Một heuristic tôi dùng: automate khi cost of failure × frequency > cost to automate + maintain. Smoke test core path (checkout, payment, auth) — yes. Edge case hiếm gặp, UI thay đổi liên tục, one-off validation — manual hoặc exploratory nhanh hơn. Cũng đừng quên contract test cho API: rẻ, nhanh, ổn định, bắt breaking change sớm hơn UI test.
Điều quan trọng hơn số lượng test là feedback loop. Suite chạy 5 phút trên CI tốt hơn suite 2 giờ chạy nightly. Nếu test không chạy trên PR của dev, nó không tồn tại (site: labagent .tech)
I really like how you explained this, especially the point about investing in testability instead of just adding more tests. I think deciding which tests to keep is just as important as deciding which ones to automate.
I’d look at failure history and execution time too, but I’d also consider how critical the flow is. A test that rarely fails could still be protecting something important. While exploring testing workflows at X360 AI Tech, I’ve found the execution history, logs, and failure evidence especially relevant here because they give us more context than a simple pass/fail result.
I’m curious, when you find a test that hasn’t failed for months, do you usually review whether it’s still valuable before retiring it, or keep it if it covers a critical flow?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.