TestFly 1.0.6 and 1.1.0 close three cross-browser reliability gaps.
First, browser.bidi opens a real WebDriver BiDi connection to a locally launched Chrome, Edge or Firefox. ConsoleErrorCollector then reads console.error calls and uncaught exceptions from BiDi log events instead of a JS shim, which catches errors Firefox used to hide. ConsoleErrorCollector.isBiDiActive() tells you which capture path ran.
Second, RemoteDriverProvider now resolves Edge and Safari on Selenium Grid and honors browser.matrix. Before, it only knew chrome/firefox and always read browser.name.
Third, JVM shutdown hooks and DriverManager.quitAllTrackedDrivers() quit every tracked browser exactly once when a run is aborted with Ctrl+C or System.exit. That covers browsers on every thread, including threads other than the one running the hook.
How BiDi catches console errors that Firefox used to miss
TestFly's captureConsoleErrors has always read browser console errors. Firefox exposes no browser logs over classic WebDriver, so on Firefox this meant injecting a JavaScript shim that only sees errors after it runs on the current page. The shim could not see an error thrown while the page was still loading, or one on a page reached by following a link.
browser.bidi (default false) solves this at the protocol level. When a Chrome, Edge or Firefox session is launched locally, TestFly requests a real WebDriver BiDi connection and subscribes to BiDi log events from the moment the test gets its browser:
# testfly.yml
browser:
name: firefox
bidi: true
captureConsoleErrors: true
The capability request is one line and is added only when BiDi is turned on:
// BiDiCapability.java
static void applyIfConfigured(MutableCapabilities options, TestFlyConfig config) {
if (config != null && config.getBrowser() != null && config.getBrowser().isBidi()) {
options.setCapability("webSocketUrl", true);
}
}
ConsoleErrorCollector then opens one LogInspector per session over that connection. The inspectors are kept in a WeakHashMap<WebDriver, BiDiCapture>, so a session is not held in memory after its driver is gone:
private static final Map<WebDriver, BiDiCapture> BIDI =
Collections.synchronizedMap(new WeakHashMap<>());
The subscription starts at beginTest(driver), so errors from a previous test are dropped even when a per-suite browser is reused. There is no JS shim and no race with page load.
Safari and remote drivers (Selenium Grid, BrowserStack, Sauce Labs) don't get a BiDi connection in TestFly today. For those, TestFly logs one warning per session and falls back to the usual capture path without throwing.
To check which path ran without reading logs:
if (ConsoleErrorCollector.isBiDiActive()) {
// BiDi log events are driving capture for this session
}
// ConsoleErrorCollector.java
@TestFlyApi(since = "1.1.0")
public static boolean isBiDiActive() {
try {
return BIDI.containsKey(DriverManager.getDriver());
} catch (RuntimeException e) {
return false;
}
}
Knowing which browser features you lose
A locally started Chrome or Edge gives TestFly the Chrome DevTools Protocol (CDP). CDP powers network mocking, device emulation, geolocation overrides and the screencast-based video recorder. Firefox, Safari, Selenium Grid and cloud sessions don't expose CDP, and neither does a remote Chrome, so these features would degrade silently unless something reported it.
BrowserFeatures, new in 1.1.0, names the four CDP-dependent features and describes the fallback for each:
public enum Feature {
NETWORK_MOCK("Network interception requires a local Chrome/Edge session... "
+ "routes are ignored and requests reach the real network."),
CDP_RECORDING("Video recording uses periodic screenshots instead of the "
+ "DevTools screencast, so frames are less frequent."),
DEVICE_EMULATION("Device emulation only resizes the window and overrides "
+ "navigator.userAgent in the current page..."),
GEOLOCATION("The location is overridden through JavaScript in the current "
+ "page only and is lost on navigation.");
}
BrowserFeatures.supports(driver, feature) answers the question directly (driver instanceof ChromiumDriver for three of the four features, HasDevTools for recording). warnOnce(driver, feature) logs the fallback exactly once per browser session and feature, and names the browser:
BrowserFeatures.warnOnce(driver, BrowserFeatures.Feature.GEOLOCATION);
GeoLocation already calls this before it falls back to its JS override on Firefox. The video recording fallback used to be silent. Now every degraded feature logs a warning.
Running Edge and Safari on Selenium Grid
Before this release, RemoteDriverProvider only built ChromeOptions or FirefoxOptions and always read browser.name directly. A browser.matrix run on a Grid therefore used whatever browser.name said and ignored the matrix browser, and Edge and Safari on Grid were not supported at all.
Browser resolution now goes through one shared method that both the local and remote paths call. BrowserContext (the matrix override) takes precedence over YAML:
// DriverProviderFactory.java
public static String resolveBrowserName(TestFlyConfig config) {
String contextBrowser = BrowserContext.get();
if (contextBrowser != null && !contextBrowser.isBlank()) {
return contextBrowser;
}
String name = config.getBrowser().getName();
if (name == null || name.isBlank()) {
throw new IllegalStateException(
"No browser selected: set browser.name, or run through browser.matrix.");
}
return name;
}
RemoteDriverProvider.buildOptions(config, browser) is a public method you can test on its own. It now covers all four browsers that Grid supports:
// RemoteDriverProvider.java
switch (browser.toLowerCase(Locale.ROOT)) {
case "chrome" -> options = chromium(new ChromeOptions(), browserConfig.isHeadless(), arguments);
case "edge" -> options = chromium(new EdgeOptions(), browserConfig.isHeadless(), arguments);
case "firefox" -> { /* FirefoxOptions, -headless */ }
case "safari" -> {
if (browserConfig.isHeadless()) {
System.err.println("[TestFly] Safari does not support headless mode via Selenium; ignoring headless=true.");
}
options = new SafariOptions();
}
default -> throw new IllegalStateException(
"Unsupported browser for remote execution: " + browser + ". Supported: chrome, edge, firefox, safari.");
}
options.setCapability("unhandledPromptBehavior", "ignore");
A matrix config that runs the same suite on Chrome, Edge and Safari on a Grid now works without changes:
# testfly.yml
execution:
mode: remote
gridUrl: http://grid.local:4444/wd/hub
browser:
matrix: [chrome, edge, safari]
Two related fixes shipped alongside. The first (do not reuse a per-suite driver for another matrix browser) stops DriverManager from handing a Chrome per-suite driver to a thread that should be testing Edge; DRIVER_BROWSER_CONTEXT is now tracked per thread and checked before reuse. The second (quit the browser when post-launch session setup fails) makes every built-in provider quit the session before rethrowing when timeout configuration fails right after launch, so a setup failure no longer leaves a browser process behind.
Making sure no browser is left running
DriverManager has registered a JVM shutdown hook from early on. Until this release, though, the hook only quit per-suite drivers and whatever driver was bound to the hook's own thread, and that is never the thread that ran a per-test driver. If you aborted a parallel per-test run with Ctrl+C or System.exit, the worker-thread browsers kept running after the JVM exited.
The hook now calls quitAllTrackedDrivers(). It takes every driver that still holds a session permit plus every registered per-suite driver, across all threads, and quits each one exactly once:
static void quitAllTrackedDrivers() {
java.util.Set<WebDriver> drivers =
java.util.Collections.newSetFromMap(new java.util.IdentityHashMap<>());
drivers.addAll(PERMIT_HOLDERS);
drivers.addAll(SUITE_DRIVERS);
for (WebDriver driver : drivers) {
try {
driver.quit();
} catch (Throwable ignored) {
} finally {
SUITE_DRIVERS.remove(driver);
releasePermitOf(driver);
}
}
}
PERMIT_HOLDERS is the set of drivers that currently hold a session-slot permit. Exactly one code path per driver adds and removes entries, which is also why execution.maxActiveSessions can't leak slots. Normal suite teardown still calls DriverManager.quitAllSuiteDrivers() and then quitDriver() from SuiteExecutionListener, TestFlyExtension (JUnit 5) and CucumberHooks. The shutdown hook is the safety net for abnormal exits and runs the same cleanup logic no matter which thread owned the browser.
This builds on the original "Guaranteed Driver Cleanup" item from 1.0.6 (shutdown hooks plus @AfterAll listener execution). The hook has existed since 1.0.6; this release makes it reach every thread's browser and not only the suite-level ones.
FAQ
Does browser.bidi work with Selenium Grid or BrowserStack/Sauce Labs?
No. BiDi only applies to a locally launched Chrome, Edge or Firefox. On Safari or any remote driver, TestFly logs one warning and falls back to the previous capture path (CDP logs on Chrome, the JS shim elsewhere). Nothing fails.
How do I check which console-error capture path is active for the current test?
Call ConsoleErrorCollector.isBiDiActive(). It returns true only when the current thread's driver has an open BiDi log subscription. It never throws; it returns false if there is no driver or the check fails.
Can I run the same suite on Chrome, Edge and Safari against one Selenium Grid?
Yes, with execution.mode: remote, execution.gridUrl and browser.matrix: [chrome, edge, safari]. RemoteDriverProvider resolves the matrix browser before browser.name and builds Grid options for all four supported browsers, including Safari's headless restriction.
If I kill a parallel run with Ctrl+C, are all browser processes gone?
Yes. The JVM shutdown hook now collects every driver that holds a session permit and every per-suite driver, on every thread, and quits each one exactly once, regardless of which thread the hook runs on.
Top comments (0)