The EAS build that demanded four packages that never existed

The build log said the lockfile was missing four packages:

npm error Missing: @opentelemetry/[email protected] from lock file
npm error Missing: @opentelemetry/[email protected] from lock file
npm error Missing: @opentelemetry/[email protected] from lock file
npm error Missing: @opentelemetry/[email protected] from lock file

Four identical lines. Here's the thing: nothing in the project asks for @opentelemetry/[email protected]. Not the app, not a dependency I could find, not the lockfile itself. The same repo installed cleanly on my machine with npm install. Only EAS — which installs with npm ci — refused it.

This is the story of two days spent learning more about npm's bundled-dependency handling than anyone should, ending at a conclusion I want on the record: the failure was real, the lockfile was honest, and the packages being demanded genuinely did not need to exist.

TL;DR — npm ci validates a lockfile instead of resolving; different npm versions record subtly different lockfile shapes around bundleDependencies, so a lock written by one npm can fail validation under another. The deep cause: the published @aws-amplify/{data,graphql-api}-construct tarballs are internally incoherent — they bundle @opentelemetry/[email protected] next to bundled siblings that exact-pin [email protected]. Some npm versions synthesize the four phantom lockfile entries that paper over this; others demand them. Root overrides cannot reach bundled deps — don't burn a day trying. The fix: pin the construct packages to a coherent pair and regenerate the lock under the same node major CI uses. Then guard it: npm ci --dry-run in a pre-push hook, because this bites again.

(Part 21 of Building CannyCart, a voice-first shopping app I'm building in public. Self-contained — no earlier context needed.)

Act 1: the innocent-looking upgrade

It started as a routine Expo point upgrade that left the repo with two copies of Expo — the root node_modules held one version, a nested tree held another, and expo-doctor listed duplicate after duplicate. The cause was tilde drift in a workspaces monorepo: "expo": "~57.0.4" permits 57.0.7, and depending on install order npm can materialize both, nested. The fix for that was a clean lockfile regeneration, and doctor went 18/18.

Then I pushed, EAS ran npm ci, and got the four Missing lines above. The local tree was healthy; only CI's validator objected. That asymmetry is the whole mystery: npm install resolves, npm ci validates — they are different programs with different opinions, and a lockfile can satisfy one and not the other.

Act 2: the theories that failed

Reproducing locally was step one: npm ci --dry-run fails in seconds with exactly EAS's error, under both npm 10 and (via npx npm@11 ci --dry-run) npm 11. That made iteration cheap. What didn't work:

1. "Just reinstall." A second full npm install + regen produced a byte-different lockfile with the same validation failure. The missing entries never appeared.
2. A scoped override — "@aws-amplify/data-construct": {"@opentelemetry/core": "2.0.0"} — changed nothing under the constructs.
3. A global override — "@opentelemetry/core": "2.0.0" — changed the root copy only, and quietly downgraded a package my own code depends on. Reverted.
4. Pinning the construct packages alone — closer, but the nested entries still recorded [email protected].

Every failure had the same signature: whatever I did to package.json, the entries under the construct packages didn't move. Which was the clue.

Act 3: open the tarball

The paths in the error all pointed under two packages — @aws-amplify/data-construct and @aws-amplify/graphql-api-construct, deep in the Amplify backend toolchain. Both use bundleDependencies: their published tarballs ship their own node_modules inside. Bundled deps aren't resolved by npm at all — they're extracted verbatim from the tarball. Which explains every failed theory at once: overrides cannot reach bundled dependencies, because there is nothing to resolve. The tarball is the truth.

So I downloaded the tarballs and looked. Inside each construct package:

• a bundled @opentelemetry/core at 2.8.0
• bundled @opentelemetry/resources and @opentelemetry/sdk-trace-base at 2.0.0 — each of which declares an exact dependency on [email protected]

The published artifact is internally incoherent: it ships siblings that demand a version of core it doesn't ship. No install of mine created this; it came off the registry this way.

Act 4: so why did it ever build?

The previous, working lockfile — the one committed before the upgrade — contained four entries I had never consciously created:

…data-construct/node_modules/@opentelemetry/resources/node_modules/@opentelemetry/core   2.0.0  inBundle
…data-construct/node_modules/@opentelemetry/sdk-trace-base/node_modules/@opentelemetry/core  2.0.0  inBundle
(same pair under graphql-api-construct)

Synthetic entries. The older npm that wrote that lockfile had noticed the incoherence and recorded four phantom deep placements that satisfy the exact-pins — marked inBundle even though the tarball ships no such files. The newer npm regenerating the lock doesn't synthesize them; the newer validator demands them. The lockfile was honest both times — it just spoke two dialects, and CI only accepted one.

That's the resolution of the mystery in one sentence: the four "missing packages" were bookkeeping fictions that one arborist version invents and another insists on.

The fix, and the interim surrender

Mid-investigation there was a pragmatic surrender, and I'll own it: for a few days every build installed with npm install instead of npm ci — the pre-install hook deleted the lockfile so EAS fell back to resolving. It shipped builds, and it was wrong as a resting state: CI without a validated lockfile is CI that can silently drift.

The real fix, in three parts:

1. Pin the construct packages (@aws-amplify/data-construct, @aws-amplify/graphql-api-construct) to a known-coherent pair in root overrides — the one dial that does affect which tarball gets bundled in.
2. Regenerate the lockfile under the same node major CI uses (node 20 / npm 10, via nvm) — the lockfile dialect problem dissolves when writer and validator match.
3. Go back to npm ci and guard it: a pre-push hook running npm ci --dry-run — seconds, and it is exactly what EAS will do.

Epilogue: it struck twice more

The guard wasn't paranoia. The same class of failure returned twice — once out of nowhere, once immediately after adding a new server-side dependency — and both times the pre-push dry-run caught it at my desk instead of twenty minutes into a cloud build. A war story that ends "and then it never happened again" is usually missing an epilogue; this one ends "and now it can't reach CI," which is the only ending that counts.

What I took away

• npm install resolves; npm ci validates. Different programs, different opinions. A green local install proves nothing about CI. Run npm ci --dry-run before pushing — it's free.
• Lockfiles have dialects. Different npm/arborist versions record different shapes, especially around bundleDependencies. Regenerate locks under CI's node version, always.
• overrides cannot reach bundled dependencies. If the broken thing is inside someone's shipped node_modules, no amount of resolution config will touch it. Check for inBundle in the lock before spending a day on overrides.
• Published packages can be internally incoherent. When nothing you did explains the error, download the tarball and look inside. The registry is not a source of guaranteed-consistent artifacts.
• Time-box the workaround, then do the real fix. npm install in CI kept releases moving and would have rotted forever if left.

Next up

Part 22 is gentler: making a phone-first app tablet-worthy without a redesign — the content-column pass that capped every screen to a centered column in seven small phases, and why the phones never noticed.

What's the deepest you've had to dig below your own code to explain a CI failure — and did the artifact at the bottom turn out to be broken as published?