Zipp benchmarks

Headline

Zipp/Node, all 30 rows, geometric mean

0.728×

95% bootstrap interval 0.723 to 0.730

Zipp/Bun, all 30 rows

0.594×

0.591 to 0.598

Zipp/Deno, all 30 rows

0.460×

0.458 to 0.464

Rows where Zipp's median beats Node's

21 / 30

9 rows are slower; they are listed below

Capture: 2 September 2026, engine commit 8229b3fc, a clean profile-guided-optimization build recorded as zipp 0.0.12 (the README labels the same binary 0.0.13; this page uses the identity the artifact itself recorded). Native Windows x86-64, four engines, 15 repetitions per row, exact output on every observation.

Methodology

• Cold process time. Every observation is a fresh process timed with a wall clock around spawn and exit. A paired empty launch of the same engine runs immediately before each full launch; the headline uses the cold time, and the launch-adjusted time is reported as a diagnostic.
• No warm-up and no --jitless games. Every engine runs with its production JIT and an empty environment; any inherited runtime control such as NODE_OPTIONS or ZIPP_NOJIT marks a run diagnostic-only.
• Counterbalanced schedule. A seeded engine order rotates once per repetition so each engine occupies every position; the per-repetition order is recorded in the artifact.
• Intervals. 95% percentile bootstrap with 10,000 replicates, described in the artifact as descriptive, not a hypothesis test. Suite geometric means use a cluster bootstrap by repetition. "Win" below means a Bonferroni-adjusted exact one-sided paired sign test at 0.05 across all comparisons (39 for the normal suite, 51 for hostile).
• Correctness first. Standard output is compared byte for byte against the reference on every repetition. A single mismatch disqualifies the capture.
• Publication gate. Full Node, Bun, Deno, Zipp order with Node as baseline; the complete frozen suite; at least 15 repetitions and 10,000 bootstrap samples; a clean, HEAD-matching, provenance-stamped PGO binary.

Startup

Median empty-process launch, the paired baseline for every row:

Normal suite: 13 rows

Ten headline workloads plus three architecture diagnostics that are deliberately kept out of the headline geometric mean. Cold medians in milliseconds; the interval is the 95% bootstrap on the Zipp/Node ratio; an asterisk marks a Bonferroni-adjusted sign-test win over Node.

Note what parse-large-js measures: a hand-written userland tokenizer over two megabytes of synthetic source. It does not benchmark Zipp's own parser. The diagnostic rows isolate the inline-cache cliff at nine receiver shapes and the sparse-array gap-size axes; they are useful for engineering and would flatter the headline, which is why they are excluded from it.

Hostile suite: 17 rows

The hostile corpus asks how gracefully Zipp performs when JavaScript stops looking like an ideal reducer: six baseline-and-stressor pairs plus five standalone applications, including a React-style reconciler, a warm HTTP router, a bytecode interpreter and the unmodified [email protected] package.

Ordinary geometric mean 0.829× Node [0.820, 0.833], 0.647× Bun, 0.419× Deno; category-balanced 0.862× Node. The suite's aspirational gate (category-balanced at most 1.05×, no category above 1.15×, no row above 1.50×) fails on both the category and the row bounds.

Where Zipp loses

Nine of thirty rows have a Node point gap: reactish-reconcile 1.578×, allocation-survival 1.559×, warm-router 1.520×, shapes-megamorphic 1.245×, shapes-stable 1.241×, calls-closures 1.118×, async-promise-chain 1.118×, and two rows whose intervals cross parity, async-lived 1.005× and json-large 1.005×. The pattern is consistent: long-lived allocation, megamorphic property access and promise-heavy code are where V8's optimizing tiers still win.

Against Bun, Zipp loses json-large (1.408×), throw-catch (1.636×), allocation-survival (1.198×), reactish-reconcile (1.041×), parse-large-js (1.019×) and async-promise-chain (1.009×). Bun is the absolute fastest engine on json-large, markdown-render and parse-large-js; Deno is fastest on typedarray-math. Against Deno, Zipp loses throw-catch (1.470×) and async-promise-chain (1.041×).

WebAssembly, Python and GPU

None of the numbers above apply to the WebAssembly build, which is an interpreter. The only WebAssembly comparison in the repository, against QuickJS-NG v0.16.2, is marked not publishable: only five hostile rows were comparable, QuickJS-NG led four of them, and Zipp led warm-router at 0.476×. Module size is a settled fact rather than a benchmark: 5,558,860 bytes raw and 1,248,649 Brotli for Zipp's v0.0.12 artifact against 1,528,293 and 417,087 for QuickJS-NG, 3.59× raw and 2.96× on the wire.

Five WebAssembly interpreter cliffs were closed on 5 September 2026 with paired before-and-after measurements on one machine: a 64K non-ASCII charCodeAt loop 4,462 → 3.2 ms, a tokenizer 9,828 → 8.2 ms, a slice loop 449 → 4.3 ms, join with 300K retained strings 8,423 → 138 ms, and a new Function property loop 20.9 → 14.8 ms.

There are no published Python-frontend benchmark tables beyond the three micro-timings on the Python page, and no GPU performance numbers at all; the GPU work is validated for correctness only.

The 11 September capture, and why it is slower

A newer publishable capture exists at commit 14770703 (zipp 0.0.15, 11 September 2026): headline ten 1.003× Node [0.995, 1.009], all 13 rows 0.677×, hostile ordinary 0.875×, with bytecode-vm at 3.59× Node. A paired A/B against the audited baseline e6e0f65d puts that baseline within 1% of the new build on every row, so the regression opened between 8229b3fc and e6e0f65d. The suspect is the v0.0.15 change that made specification-order method calls the default: the fused CallMethod lowering was observably wrong for a getter or a proxy trap and is now used only where argument shapes provably cannot observe the order. No headline is claimed for 14770703; the trade was correctness for speed, and the ledger says so.

1. 28 July 2026
   1.90× Node, headline ten

   First attributable default capture; interpreter-only smoke at 6.42×.

2. 3 August 2026
   PGO adopted at −13.3%

   1.749× to 1.451× in one build change; two more waves reach 1.330× the same day.

3. 19 August 2026
   Nursery on by default: 1.212×
4. 20 August 2026
   First capture under 1.20×

   typedarray-math reaches Node parity.

5. 24 August 2026
   0.9695×: Node parity crossed

   Wave 19 decomposed the three rows that were 60% of the remaining gap.

6. 27 August 2026
   Hostile suite under parity at 0.961×
7. 29 August 2026
   A 2.16× regression, reversed

   A de-fused call lowering; root-caused and reverted the same day.

8. 2 September 2026
   0.878× headline, 0.728× all-30: the public capture
9. 11 September 2026
   1.003× headline after strict call order

   Correctness kept; speed to be recovered.

Reproduce it

& 'C:\Program Files\Git\bin\bash.exe' tools/pgo.sh      # profile-guided build of target/release/zipp.exe
New-Item -ItemType Directory -Force target\bench-results | Out-Null
python tools\bench.py --zipp target\release\zipp.exe --engines node,bun,deno,zipp `
  --reps 15 --bootstrap-samples 10000 --json target\bench-results\real13.json
python tools\bench_hostile.py --zipp target\release\zipp.exe `
  --reps 15 --bootstrap-samples 10000 --json target\bench-results\hostile17.json
python -m unittest tools.test_bench

• bench/real13_8229b3fc_pgo_2026-09-02.json, the normal-suite capture.
• bench/hostile/head_clean_8229b3fc_pgo_2026-09-02.json, the hostile capture.
• bench/real13_14770703_pgo_2026-09-11.json and the hostile pair, the later capture.
• bench/README.md, the suite map and publication gate; bench/hostile/README.md and manifest.json.
• PERF_ROADMAP.md, the checkable roadmap, and the archived performance ledger, 881 KB of experiments including the refuted ones.