DEV Community

Cover image for Carry the scheduler's memory between CI runners
VX
VX

Posted on Edited on Originally published at vznjs.github.io

Carry the scheduler's memory between CI runners

A fresh CI runner doesn't know which tasks are slow, so it can start
them last. One small file carries task times from runner to runner, and
the slow tasks start first.

Runner Wall time (min of 3)
Cold, no file 12.09 s
Cold, with the file 10.68 s
Full local history 10.64 s

8 packages with a build → check → pack chain (0.5 s each) and 3 slow e2e tasks (10 s each) · sleep tasks · linux x64, 4 cores, 4 workers · vx 0.0.634 · bun private/vx-bench/timings-file-bench.ts

Every task runs in all three. Only the order changes. The file saves
1.4 s and matches a machine with its full history. The floor is 10.5 s.

Turn it on

import { scheduleHistoryPlugin } from '@vzn/vx-schedule-history'

export default {
  plugins: [scheduleHistoryPlugin({ file: '.vx-timings.json' })],
}
Enter fullscreen mode Exit fullscreen mode

@vzn/vx-schedule-history starts the longest chain first. vx reads the
file before it orders the run, and writes each task's median time after:

{
  "version": 1,
  "tasks": {
    "app#build": 41200,
    "app#test": 18350,
    "docs#build": 29010
  }
}
Enter fullscreen mode Exit fullscreen mode

Carry it between runners

- uses: actions/cache@v4
  with:
    path: .vx-timings.json
    key: vx-timings-${{ github.run_id }}
    restore-keys: vx-timings-
Enter fullscreen mode Exit fullscreen mode

Add the file to .gitignore, or a task with ** inputs gets a new key
each time it changes. vx warns if git doesn't ignore it.

diagram

When it can't help

If one task is the whole wall time, order doesn't matter. vx's own CI
had one 194 s test suite: 194.07 s with the file, 194.60 s without.

A fully cached run doesn't rewrite the file, so it pays nothing extra.

More at vznjs.github.io/vx.


Originally published on the vx blog. vx is open-source (MIT) task orchestration with a build cache: vznjs.github.io/vx · GitHub.

Written with AI assistance.

Top comments (0)