TestFly Part 16: Smart triage and fuzzy self-healing, flaky-test diagnosis without tokens

TestFly 1.0.6 adds a local heuristic layer that runs before any AI is consulted and costs no LLM tokens. SmartTriageEngine inspects the exception hierarchy and classifies a failure as SYSTEM_FLAKY, APPLICATION_BUG or NEEDS_INVESTIGATION in microseconds. FuzzyHealingEngine repairs a broken locator in under 5ms using plain Levenshtein distance against DOM candidates. Neither makes a network call or needs tokens or an API key.

This layer is separate from the LLM-based self-healing in Part 4 (testfly-mcp). That path is the last resort for truly ambiguous locators. The heuristic layer described here is the first resort, and it handles the 80% of healing and triage cases that don't need a model at all. The PageKnowledge / Auto-POM system connects the pieces: TestFly learns a page's elements once and reuses that knowledge locally on every later run.

Why flaky-test triage needs a heuristic layer

Every CI pipeline eventually collects red builds that nobody trusts. Re-running the test is cheap. Working out why it failed takes the time: an infrastructure hiccup, a real regression, or something a person needs to look at more closely. Before TestFly 1.0.6, triage was either manual or meant sending every failure's stack trace to an LLM and paying tokens to hear "this looks like a timeout."

SmartTriageEngine is wired into TestExecutionListener and does this triage synchronously, at no cost, as part of the failure-reporting path:

package io.testfly.flakiness;

public final class SmartTriageEngine {

    public enum TriageResult {
        APPLICATION_BUG,
        SYSTEM_FLAKY,
        NEEDS_INVESTIGATION
    }

    public static TriageResult triageFailure(Throwable exception, boolean wasRetriedAndPassed) {
        if (exception == null) return TriageResult.NEEDS_INVESTIGATION;

        // Rule 1: Immediate indicator of flakiness
        if (wasRetriedAndPassed) {
            return TriageResult.SYSTEM_FLAKY;
        }

        int flakyScore = 0;
        Throwable rootCause = getRootCause(exception);

        // Rule 2: Exception heuristics
        flakyScore += switch (rootCause) {
            case TimeoutException t -> 80;
            case StaleElementReferenceException s -> 80;
            case ConnectException c -> 90;
            case AssertionError a -> -50; // almost always an application bug
            case WebDriverException w -> 40;
            case null, default -> 0;
        };

        if (flakyScore >= 70) return TriageResult.SYSTEM_FLAKY;
        else if (flakyScore <= 0) return TriageResult.APPLICATION_BUG;
        else return TriageResult.NEEDS_INVESTIGATION;
    }
}

There is no model call, prompt or round trip. A Java 21 pattern-matching switch runs over the root cause of the exception chain, which may be wrapped several layers deep. A ConnectException scores 90 and is flagged SYSTEM_FLAKY right away. An AssertionError scores -50 and is flagged APPLICATION_BUG, because a failing assertion almost always points to a real product bug and not to infrastructure noise. Anything that clears neither threshold goes to NEEDS_INVESTIGATION, which means "a person should look at this" instead of a guess that only sounds confident.

TestExecutionListener calls this for every failed test and writes the verdict into the report:

if (result.getThrowable() != null) {
    SmartTriageEngine.TriageResult triage =
        SmartTriageEngine.triageFailure(result.getThrowable(), false);

    String triageMsg = "[Triage] " + triage;
    if (triage == SmartTriageEngine.TriageResult.SYSTEM_FLAKY) {
        System.out.println("⚠️ " + triageMsg + " -> This is likely an infrastructure/flaky issue.");
    } else if (triage == SmartTriageEngine.TriageResult.APPLICATION_BUG) {
        System.out.println("🐞 " + triageMsg + " -> This is likely an application bug (Assertion).");
    }
    StepLogger.step(triageMsg, StepStatus.WARN);
}

A test that passed on retry is flagged SYSTEM_FLAKY immediately. Rule 1 skips the exception scoring entirely, because passing on retry is a stronger flakiness signal than any exception type.

How fast fuzzy self-healing is

FuzzyHealingEngine is the fallback that runs before TestFly asks an LLM to find an element again. It is plain Java with few allocations. It computes the Levenshtein distance between clues from the locator and the attributes and text of interactive DOM candidates, scores each candidate, and picks the best match above a threshold.

package io.testfly.healing;

public final class FuzzyHealingEngine {

    private static final String INTERACTIVE_ELEMENTS =
        "button, a, input, [role='button'], [role='link'], [tabindex='0']";
    private static final int THRESHOLD_SCORE = 5;

    public static WebElement tryFuzzyHeal(WebDriver driver, By originalLocator) {
        String desc = originalLocator.toString().toLowerCase();
        List<String> clues = extractClues(desc);
        if (clues.isEmpty()) return null;

        List<WebElement> candidates = driver.findElements(By.cssSelector(INTERACTIVE_ELEMENTS));

        WebElement bestMatch = null;
        int highestScore = 0;
        for (WebElement el : candidates) {
            int score = scoreElement(el, clues);
            if (score > highestScore) {
                highestScore = score;
                bestMatch = el;
            }
        }
        return highestScore >= THRESHOLD_SCORE ? bestMatch : null;
    }
}

Clue extraction removes locator syntax noise (cssselector, xpath, contains, text, data, testid) and keeps alphanumeric tokens of 3 or more characters, all in a single regex pass. Scoring weights exact attribute matches above fuzzy text matches: an id match is worth 5 points, a placeholder or value match 4, and a CSS class match 3. When an element's text doesn't contain the clue outright, the engine allows one typo for short words (5 characters or fewer) and two for longer ones. It uses the same getLevenshteinDistance() implementation (classic single-array DP, O(n·m)) that the attribute scoring relies on:

public static int getLevenshteinDistance(String a, String b) {
    int[] costs = new int[b.length() + 1];
    for (int j = 0; j < costs.length; j++) costs[j] = j;
    for (int i = 1; i <= a.length(); i++) {
        costs[0] = i;
        int nw = i - 1;
        for (int j = 1; j <= b.length(); j++) {
            int cj = Math.min(1 + Math.min(costs[j], costs[j - 1]),
                    a.charAt(i - 1) == b.charAt(j - 1) ? nw : nw + 1);
            nw = costs[j];
            costs[j] = cj;
        }
    }
    return costs[b.length()];
}

Only elements that score at least THRESHOLD_SCORE = 5 can replace the broken locator, so one weak signal, such as a single 3-point class match, is not enough. The pass stays under 5ms on a typical DOM because it takes no screenshot, serializes no markup and makes no network hop. It only compares strings over the candidate list that findElements() already returned. TestFly escalates to the LLM-based self-healing from Part 4 only when this engine returns null.

What Auto-POM / PageKnowledge does

The third piece is PageKnowledgeStore. It saves a LearnedPageModel per URL path to .testfly/page-knowledge.json. Once TestFly, or a person through recorded runs, discovers an element on a page, every later run can reuse it instead of discovering or healing it again. That includes CI runs, as long as the file is restored.

public static Optional<LearnedPageModel> getPageModel(String url) {
    load();
    String key = normalizeUrl(url);
    return Optional.ofNullable(MODELS.get(key));
}

public static void recordElement(String url, LearnedElement element) {
    if (element == null || element.name().isBlank()) return;
    load();
    LearnedPageModel model = getOrCreate(url, null);
    model.putElement(element);
    save();
}

normalizeUrl reduces URLs to their path, so /checkout?order=123 and /checkout?order=456 share one learned model. If the store is empty on first load, KnowledgeLearner.learnFromActionCache() fills it from the existing action history, so the cache has data even on a fresh checkout.

LocalIntentResolver uses this knowledge. Given a natural-language goal and the current URL, it tries to resolve the goal from PageKnowledge alone before it considers an LLM call:

public static ActionPlan resolve(String url, String goal) {
    Optional<LearnedPageModel> modelOpt = PageKnowledgeStore.getPageModel(url);
    if (modelOpt.isEmpty()) return null;

    LearnedPageModel model = modelOpt.get();

    ActionPlan formPlan = tryResolveFormInput(model, url, goal);
    if (formPlan != null) return formPlan;

    return tryResolveSingleElement(model, url, goal);
}

It also handles a simple bilingual login pattern locally. It matches the keywords "user"/"kullanıcı"/"email"/"posta" and "pass"/"şifre"/"parola" in the goal text, along with quoted literal values, and builds a TYPE → TYPE → CLICK action plan directly from the learned username, password and submit elements without model inference. Only goals that the local resolver can't match against the learned model go on to the LLM agent path.

How the token-free layer fits with Part 4's LLM self-healing

The fallbacks form three nested rings, cheapest first:

1. PageKnowledge / LocalIntentResolver: if the page has been learned and the goal matches a known element or form pattern, it is resolved locally. There is no DOM re-scan beyond the cache and no LLM.
2. FuzzyHealingEngine: if a known locator breaks (a renamed class or a shifted DOM), live interactive elements are scored by Levenshtein distance and attribute overlap. This takes under 5ms and uses no LLM.
3. LLM-based self-healing (testfly-mcp, Part 4): reached only when steps 1 and 2 both fail to produce a confident match. Token cost and network latency appear only here, and this step is kept for genuinely ambiguous cases.

SmartTriageEngine works next to these rings. It classifies a failure after it happens and does not try to prevent it. It tells you quickly, and at no cost, whether the failure deserves a retry, an investigation or a bug ticket.

FAQ

Does SmartTriageEngine call an LLM to classify failures?

No. It is plain Java. It follows the exception's cause chain to the root, matches the root's type in a fixed switch expression, and adds or subtracts fixed point values. The only outside signal it uses is whether the test passed on retry.

What score does FuzzyHealingEngine require before it heals a locator?

A total of at least THRESHOLD_SCORE = 5 across all extracted clues. Exact id matches add 5, placeholder/value matches 4, class matches 3, and fuzzy text matches (within one or two Levenshtein edits, depending on clue length) add 4. Below 5, the engine returns null and the caller escalates.

Does Auto-POM require every page to be mapped by hand first?

No. PageKnowledgeStore starts empty and grows through recordElement() as elements are discovered or healed during real runs. When the store is empty, KnowledgeLearner.learnFromActionCache() also seeds it from previously recorded actions, so there is no separate "page object generation" step.

If FuzzyHealingEngine fails, does TestFly fail the test right away?

No. A null return from tryFuzzyHeal tells TestFly to escalate to the LLM-based self-healing described in Part 4. The heuristic layer is a cheaper first attempt and does not replace the AI fallback.