TestFly Part 12: Playwright-style assertions in Selenium with not(), ARIA snapshots and strict mode

TestFly 1.1.0 closes most of the gap with Playwright for Java's assertion API without leaving Selenium. LocatorAssert.not() negates the next check. hasRole(), hasAccessibleName() and hasAccessibleDescription() read the real accessibility tree instead of guessing from DOM attributes. matchesAriaSnapshot() compares a subtree against a Playwright-style YAML template. The opt-in locators.strict setting makes an ambiguous locator fail loudly, where before it would silently click the wrong element.

Why Selenium assertions need not()

In classic Selenium/TestNG code a negative assertion usually means one of two bad options: wrap the positive assertion in a try/catch and invert the result, or write a separate polling loop that waits for a condition to stay false. Either way the intent of the test ends up buried in boilerplate.

TestFly's LocatorAssert now has the same not() as Playwright for Java. It negates only the assertion that comes directly after it in the chain:

import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

import static io.testfly.assertion.Assertions.assertThat;

public class CheckoutErrorTest extends BaseTest {

    @Test
    public void testErrorBannerClearsAfterRetry() {
        open("/checkout");
        find("#pay-btn").click();

        assertThat(find("#error-banner")).hasText("Payment declined");

        find("#retry-btn").click();

        // Only the assertion right after not() is negated
        assertThat(find("#error-banner")).not().isVisible();
        assertThat(find("#spinner")).not().isAttached();
    }
}

As in Playwright, most negated assertions also pass when no matching element exists, so not().isVisible() passes whether the banner is hidden or removed from the DOM. If the element has to exist and only be in a different state, assert presence first: assertThat(status).isVisible().not().hasText("Error"). You cannot combine not() with the verified actions click(), fill() or clear().

Deterministic accessibility assertions

TestFly already had hasAriaLabel-style attribute checks. Version 1.1.0 adds assertions that read the computed accessibility tree the way a screen reader does. They run through one bundled probe script, with no axe-core injection and no LLM call:

@Test
public void testSubmitButtonIsAccessible() {
    open("/signup");

    assertThat(find("#submit"))
        .hasRole(Role.BUTTON)
        .hasAccessibleName("Create account");

    assertThat(find("#email"))
        .hasAccessibleDescription(Pattern.compile("(?i)required"));
}

hasRole(Role | String) checks the explicit role attribute or the implicit role of the HTML tag. hasAccessibleName and hasAccessibleDescription take either an exact String (whitespace collapsed, with the same matching Locator.withName(...) uses) or a Pattern for a partial or case-insensitive match. Only the 36 standard ARIA Role values and visible text are reported. Decorative images and hidden content are skipped, and the probe does not enter Shadow DOM.

How ARIA snapshots work

matchesAriaSnapshot() is TestFly's version of Playwright's toMatchAriaSnapshot(). It captures the accessibility subtree under a locator and compares it with a YAML template. The comparison is partial and ordered, so it does not need a full serialization of the subtree to match.

@Test
public void testNavigationLandmarkStructure() {
    open("/dashboard");

    assertThat(By.tagName("nav")).matchesAriaSnapshot("""
        - link "Home"
        - link "Docs"
        - button "Account" [expanded=false]
        """);
}

The matching rules follow Playwright:

A malformed template fails at once with IllegalArgumentException, without waiting for the timeout. When the assertion fails, the error contains both the template and the captured snapshot, so you can see the difference without re-running anything. To write a template in the first place, print what TestFly currently sees:

System.out.println(find("nav").ariaSnapshot());
// or, for the whole page:
System.out.println(ariaSnapshot());

Then paste the printed YAML into the test and trim it down to the nodes that matter.

How strict mode catches silent multi-element bugs

By default, when a locator resolves to several visible elements, TestFly does what classic Selenium does and silently clicks or reads the first one. That is how a renamed CSS class or an A/B-tested duplicate banner produces a test that passes against the wrong element for weeks before anyone notices.

You turn strict mode on per project:

# testfly.yml
locators:
  strict: true

With locators.strict: true, an action on a locator that matches more than one visible element, and that was not narrowed with first(), nth() or last(), throws immediately:

Strict mode violation: css=".card .btn-buy" resolved to 3 visible elements;
use first()/nth()/last() or narrow the locator. Candidates: <button.btn-buy#card-1>, <button.btn-buy#card-2>, <button.btn-buy#card-3>

isVisible() and isEnabled() report the same violation instead of silently checking the first match. count(), elements() and assertThat(...) (for counting or enumerating) are not affected, because multiple matches are expected there. When strict mode flags a locator, narrow it explicitly:

find(".card .btn-buy").nth(1).click();          // second card, explicit
find(".card").filter(withText("Pro Plan"))
             .find(".btn-buy").click();          // narrowed by content instead

Hidden duplicates, such as a closed mobile nav next to the desktop one or a hidden modal clone, are filtered out before the check runs, so responsive layouts don't cause false strict-mode violations.

FAQ

Does not() replace the existing Selenium assertion methods?

No. not() is a modifier on LocatorAssert. It negates whichever assertion comes right after it in the same chain, and all existing positive assertions stay as they are.

Do the accessibility assertions require axe-core or an external library?

No. hasRole, hasAccessibleName, hasAccessibleDescription and matchesAriaSnapshot use one bundled probe script that reads the browser's computed accessibility tree directly. There is no axe-core injection and no LLM call.

Will enabling locators.strict break existing tests?

It can, and that is intended. Any locator that used to match more than one visible element silently now fails explicitly. The fix is the one Playwright recommends: narrow the locator with first(), nth(), last() or a more specific selector.

Does strict mode affect count() or elements()?

No. Those methods exist to work with multiple matches, so strict mode does not apply to them. It applies only to actions (click, fill, …) and to the boolean checks isVisible()/isEnabled().

Originally published on hakangul.lovable.app.