27 localizations, 20+ devices, 4 browsers. Multiply them and you get 2,160 combinations. Nobody runs that, and no team I've worked with has ever tried.
Those three numbers are real. They come from the Abbott Laboratories engagement — LibreView, the cloud platform behind the FreeStyle Libre glucose monitoring system — which we've been on since 2019. Same project: 1,600+ test cases, 1,500 automated scripts, 10 parallel threads, 90% test coverage, regression down from 2.5 weeks to 1 day, smoke from 7 days to 1 day, and 70% of translations covered by automated tests. Not 100%. Seventy.
The 2,160 is a fake number because the two axes are independent. A locale is a data-and-format variant of one build. A device is a runtime. They fail differently, they cost differently, and the moment you treat them as one matrix you start testing both of them badly.
Two matrices, and why multiplying them is the mistake
A locale bug reproduces on any device. If a German decimal separator breaks a price field, it breaks on a Pixel 8, on Safari 18, and on your laptop. A device bug reproduces in any locale. If a camera permission prompt kills the app on one Android OEM, it kills it in English too.
That gives you the split:
- Sweep locales on one reference runtime — one browser, one OS version, one screen size. Locale is test data.
- Sweep devices in one reference locale. Device is a runtime.
- Then pick a handful of intersection cells on purpose: the longest-expanding language on the smallest screen, an RTL locale on the oldest supported OS, one CJK locale on a low-DPI device. Five to ten cells, named in a document, not generated by a cross product.
The cases show this shape as an end state, not as a design doc — I can't tell you anyone drew it on a whiteboard. At Abbott, locale switching runs inside the automated regression suite while device coverage lives in a separate cloud-device matrix. At Allego, where we've worked since 2013, 27 localizations sit behind 2,400+ web automation tests, and the device axis isn't expressed as a device list at all — it's 5 iOS versions and 5 Android versions, with 1,400+ iOS and 1,100+ Android tests, 5,000+ automated scenarios in total and 95% coverage. Two axes, two different units of measurement, one suite.
The split is my argument. The numbers are the evidence that nobody is running the product of the two.
Sizing the device matrix from evidence, not a catalogue
The device list should come from your own analytics and your own defect history, and it should grow by one rule: add a device only when it has produced a defect class no device already on the list produced. "Popular in our market" is not that rule. "We should probably have a Samsung" is not that rule.
The platform distributions tell you where the risk actually concentrates, and they are wildly asymmetric. Apple measured 79% of all iPhones running iOS 26 on 7 June 2026, and 86% of devices introduced in the last four years. Android has no equivalent head: StatCounter's August 2026 data puts Android 16 at 25.76%, Android 15 at 16.91%, Android 13 at 14.75% — you need six versions, back to Android 11, to reach roughly 89% of devices. A "latest two versions" policy is a strong promise on iOS and a weak one on Android, and that asymmetry should show up in the matrix as different version counts per platform, not as a device count.
What the engagements actually landed on, at very different scales:
- Abbott: from 5 test devices before the engagement to 20+ with varying OS versions and screen resolutions, plus 4 browsers.
- Dootrix (a UK EV-charging app, since 2023): 20+ devices with parallel execution pipelines on different real devices, 400+ test cases, 100+ automated mobile scenarios and 50+ backend ones.
- Grover (tech rental, since 2018): 5+ devices spanning OS versions, screen resolutions and browsers, with 600+ automated tests and 90% coverage.
- Learningbank: 4 browsers — Chrome, Firefox, Edge, Safari — and nothing else, across ~500 test cases, 100+ shipped features and below 5% post-release defects.
And the counterargument, which matters: Xola runs 160 devices. That is eight times Abbott's list. It also runs 70+ parallel threads, 6 CI pipelines across 6 apps, 3,200+ automation scripts, 8,300+ checks, and an automation build that dropped from an hour to about 20 minutes. A 160-device matrix is viable because the infrastructure absorbs it. Matrix size is a function of parallelism and pipeline throughput, not of diligence. Copy the 160 without the 70 threads and you get a suite that runs overnight and gets muted by the second week.
Worth noting what the Xola case does not report: a locale axis. Six apps, 2,400+ bugs, 90% coverage, 5–10 production releases per 5 working days, and localization never appears as a dimension. That is a legitimate product decision, and it is what a single-axis matrix looks like when you're honest about it.
What actually breaks when you localise
Translation quality is the smallest part of localization testing and the part everyone budgets for. The defects live in data and formatting:
- Numbers: decimal separator (1,234.56 vs 1.234,56 vs 1 234,56), digit grouping (Indian lakh/crore grouping is not a thousands separator), and non-Latin digit shapes.
- Units: metric vs imperial, and in a medical or scientific product, unit conversion that changes the meaning of a value rather than its presentation.
- Dates and times: field order, 12- vs 24-hour clocks, week start day, non-Gregorian calendars, and time zones with DST transitions your fixtures never hit.
- Currency: symbol position, currency-specific decimal places (JPY has none), and rounding.
- Validation: postcode, address shape, and phone number rules that silently reject valid input for a country you just launched in.
- Collation and search: sort order and case-folding differ per locale; a "sorted list" assertion written for English is wrong in Swedish.
- Plurals: CLDR defines six plural categories — zero, one, two, few, many, other. A string built with if (n === 1) is broken in Russian, Welsh and Arabic before anyone translates it.
- Layout: string expansion. Microsoft's own guidance is to lengthen text by 40% as a heuristic when the source is English, with extreme cases at 200% or even 400%, and short strings expanding proportionally more than long ones. That is what breaks buttons, tab bars and table headers.
- Concatenation: any string assembled from fragments at runtime is a bug waiting for a language with different word order.
The cheapest way to find most of that is pseudo-localization, before a single real translation exists. Android ships two pseudolocales: en-XA, which accents and expands text and brackets each message unit — so hardcoded strings appear unaccented and concatenated strings appear as multiple brackets — and ar-XB, which reverses text direction to expose RTL mirroring failures.
One more thing that moves under you: format truth is a library version. CLDR 48 shipped on 30 October 2025 with 104 locales at modern coverage plus 305 regional variants, and was integrated into ICU 78. If your assertions hardcode formatted output, a CLDR bump turns into a wall of red tests that are not product bugs. Pin the ICU/CLDR version in CI and treat upgrades as a scheduled event.
Abbott's product is where this class of defect is expensive — glucose data with unit and format correctness at the centre of it — and its localization automation went from unsupported to 70% of translations covered. The case reports coverage figures, not a defect taxonomy; the failure list above is craft knowledge, not Abbott's incident log.
Automating locale coverage — the part that pays
Do not run full regression per locale. Parameterise a small, high-value set of scenarios — signup, checkout, the one screen that displays money, the one that displays dates — across every locale, and run everything else once.
And assert on the right things. Not translated strings (that's the TMS's job and it makes your suite fail on every copy edit), but on format correctness, missing-key markers, and clipped text. Derive expectations from ICU rather than fixtures:
// expectations come from the same CLDR data the product uses,
// so adding a locale adds a row, not a fixture file
const cases = [
{ locale: 'de-DE', amount: 1234.5, currency: 'EUR' },
{ locale: 'en-IN', amount: 1234567, currency: 'INR' },
{ locale: 'ja-JP', amount: 1234.5, currency: 'JPY' },
{ locale: 'ar-EG', amount: 1234.5, currency: 'EGP' },
];
for (const { locale, amount, currency } of cases) {
test(price formatting — ${locale}, async ({ page }) => {
await page.goto(/checkout?locale=${locale});
const expected = new Intl.NumberFormat(locale, { style: 'currency', currency })
.format(amount);
await expect(page.getByTestId('total')).toHaveText(expected);
});
}
Two assertions catch most of the rest, and neither knows any language:
``` // 1. untranslated or missing keys leaking into the UI
const body = await page.locator('body').innerText();
expect(body).not.toMatch(/[a-z0-9_]+.[a-z0-9_.]+/); // raw i18n keys
expect(body).not.toMatch(/{{|%[sd]\b/); // unresolved placeholders
// 2. text clipped by expansion — run this hard in en-XA
const clipped = await page.$$eval('button, [role="tab"], h1, h2, label', els =>
els.filter(e => e.scrollWidth > e.clientWidth + 1).map(e => e.textContent.trim())
);
expect(clipped, clipped in ${locale}).toEqual([]);```
Then make locale a CI dimension rather than a suite:
strategy:
fail-fast: false
matrix:
locale: [en-XA, ar-XB, de-DE, ja-JP, tr-TR, en-IN, ar-EG]
en-XA and ar-XB belong in that list permanently — they run before translations exist and they catch the layout and RTL failures the real locales will only confirm later. The translation management system stays the source of truth for keys; the suite's job is to prove the keys resolve and the formats are right.
What this buys, at Abbott: per-country smoke testing dropped by more than 85%, running on a keyword-driven Robot Framework suite in CI with 10 parallel threads across 27 country versions. That figure is real, and the attribution is not clean — automation, CI and staffing (8 automation plus 11 manual QA engineers) all changed together. Don't credit locale parameterisation with the whole drop.
At the other end of the scale, Grover automates 10%+ of translations across 4 localizations with 1 full-stack QA engineer and 1 manual QA engineer, alongside 90% overall coverage, regression in 1–2 days and smoke in 3–4 hours. Ten percent of translations, deliberately. That is what the locale axis looks like on a two-person team, and it is not a failure state.
Real devices, cloud farms, emulators
Emulation is adequate for what locale testing needs: layout, text expansion, RTL mirroring, format rendering. All of that is deterministic on a simulated runtime.
It is not adequate for OS behaviour. Permission prompts, biometric auth, camera and Bluetooth, backgrounding and process death, push delivery, and network transitions between Wi-Fi, LTE and nothing at all — none of those are faithfully reproducible off hardware. Dootrix is the clean example: an EV-charging app with live charging sessions, Stripe payments and Charles Proxy in the stack, running parallel execution on real devices. You cannot fake "the session was mid-charge when the phone lost signal in an underground car park."
Cloud farms sit between the two. Abbott, Grover and Dootrix all run BrowserStack; that is the pragmatic default when the matrix is 20+ devices and the team is not buying a device lab.
The counter-case is Learningbank, where mobile coverage runs through Appetize emulation and the engagement reports a 95% defect detection rate, 1,000+ bugs found, 300+ of them critical/blocking/major, and below 5% post-release defects across 100+ features — with two manual QA engineers and no automation at all. That is a browser-based corporate learning platform. It does not generalise to anything hardware-adjacent, and it is not an argument that emulators are enough.
The tier follows the feature, not a blanket policy: layout and locale on emulators, hardware and OS behaviour on real devices, and the split written down so nobody has to relitigate it per sprint.
Regional scenarios are not localization
Grover operates in five countries — the US, Germany, Austria, the Netherlands and Spain — with four localizations. Countries outnumber locales, and that gap is the whole point. Austria and Germany share a language and do not share tax handling, payment habits or consumer-law text.
What varies by market is scenarios, not strings: payment methods (iDEAL in the Netherlands, SEPA direct debit, cards, BNPL), tax and price display rules, regulatory and consent text, KYC and age checks, address and phone validation, delivery and returns flows, and network conditions. These are the ones that break revenue, and they are invisible to a translation-coverage metric. A suite can be at 100% locale coverage and have never once attempted a Dutch payment.
No source here itemises region-specific defects, so treat this as reasoning rather than measurement — but note that Dootrix's stack includes the Stripe API as a first-class integration, which is where market-specific behaviour actually lives.
Practically: keep a third, small list — one end-to-end revenue scenario per country, run per release, in that country's locale and payment configuration. Not a matrix. A list.
What this costs
Both matrices are ongoing maintenance. Every locale you add adds fixtures, test data and a column in every parameterised run. Every OS version you add invalidates selectors, waits and screenshot baselines, and OS versions arrive whether or not you have capacity.
The headcount behind these numbers is the part that never makes it into a strategy deck: Abbott, 19 QA engineers (8 automation, 11 manual). Allego, 7. Xola, 6 full-stack. Dootrix, 3. Grover, 2. Learningbank, 2. Abbott's 27 locales and 20+ devices are sustained by 19 people; Grover's 4 locales and 5+ devices by 2. The matrix is sized to the team, in both directions.
If you are small, cut in this order: locales per run first (sweep a subset per release, full set nightly or weekly), then device count, and only then scenario depth. Depth is what finds defects. Breadth is what makes a dashboard look thorough.
Sizing it, concretely
- Treat the locale axis as test data. Parameterise a small set of scenarios, assert on formats and markers rather than translations, and run en-XA and ar-XB continuously from before the first translation lands.
- Build the device axis from your own analytics and defect history. Different version counts per platform, because iOS and Android distributions are not shaped the same. Add a device only when it produces a defect class nothing else on the list produces.
- Name the intersection cells — five to ten, written down, chosen for longest string, RTL, smallest screen, oldest OS. Never generated.
- Keep a separate one-scenario-per-country revenue list. It is not localization and it does not belong in either matrix.
- Publish a coverage number below 100% and say what it excludes. Abbott's is 70% of translations, on a medical data product, with a one-day regression. That is a decision, not a shortfall.
And note that the coverage percentages above are not comparable to each other — Abbott's 90%, Allego's 95%, Grover's 90% and Learningbank's >50% are four different definitions of "covered". Yours will be a fifth. Define it once, then argue about the size of the matrix with your headcount in front of you, because that is the constraint that decides it.
Top comments (0)