JavaScript Test262 conformance
Test262 is the official ECMAScript conformance suite. Zipp runs the whole pinned corpus, unmodified, in both sloppy and strict mode, with a scorer that cannot raise the pass rate by skipping tests. This page explains exactly what the numbers mean, what the one expected failure is, and how the result is gated in CI.
The result
| Suite | Pass | Fail | Skip | Executions |
|---|---|---|---|---|
| Core corpus, unmodified upstream | 96,229 | 1 | 0 | 96,230 |
test/intl402/DateTimeFormat, unmodified | 490 | 0 | 0 | 490 |
The unmodified pass rate is 99.999%. There is no corrected profile: no test file is patched, and the one failure stays in the denominator, listed with its reason in a checked-in expected-failure manifest.
The corpus is tc39/test262 at commit d53ebc8b1931d9f1541e43d59650776e4b766248 (13 October 2026), walking test/ including the staging directory and excluding the separate ECMA-402 suite under test/intl402/, which is opt-in. Nothing is excluded by feature flag: Temporal, decorators, iterator helpers, ShadowRealm, resizable buffers and every other feature in the pinned corpus are in scope.
How executions are counted
96,230 is a count of executions, not files. Following the suite's own interpretation guide, a test that carries none of the onlyStrict, noStrict, raw or module flags runs twice, once as sloppy code and once with "use strict"; prepended. raw and module tests run once; onlyStrict and noStrict run once each in their mode.
Skips stay in the denominator. The runner's own comment says why: a change that made 500 tests unreadable must not be able to raise the reported pass rate. A skip can only arise from a read error or a missing harness include, and any skip fails the gate. Timeouts (20 seconds locally, 120 in CI) score as failures, never as skips.
The one expected failure
One execution fails against the unmodified corpus, and it is not an engine defect: annexB/language/function-code/block-decl-func-skip-arguments.js in sloppy mode. Its final assertion is ES2017 wording that ES2018 deleted, tracked upstream as tc39/test262 issue 5113, and current Node fails it too; Zipp's own regression tests cover the current semantics. Rather than patch the file or exclude it, tools/test262-expected-failures.txt lists it with that reason. The test still runs every time, and the gate fails if it starts passing.
The previous pin, 4249661, was also reported on a corrected profile, with five reviewable patches to five test files. Four of them are no longer needed, because upstream itself removed or fixed the inconsistent staging/sm/Error tests and excluded immutable buffers from the TypedArray slice alias-copy test. The fifth is the Annex B test above. The corrected profile, its patches and its separate runner have been retired, so there is now one number, and it is the unmodified one.
Moving to this pin was not free. Before the move, the engine failed 237 executions of the new corpus: 218 for the new stage-4 Iterator.prototype.chunks, windows and includes methods, 12 for the 2^53−1 limit bound in take() and drop(), 4 for Promise.try returning an already-resolved promise unwrapped, 2 for the source-phase-imports module-source prototype chain, and the Annex B test. All but the Annex B test are fixed in 1676c9e4. Zipp's own new tests also found that IteratorClose ignored an overridden generator return, fixed in eb93d408.
The runner
tools/run_test262.py is a 623-line Python runner and scorer. Given --expected-failures, it requires the run's failures to equal the manifest exactly: an unexpected failure, an entry that now passes, or any skip exits nonzero. Its unit tests live in tools/test_run_test262.py.
- The harness is passed as a separate script (
zipp js --script-goal <test> <harness>), never concatenated. Concatenation had made the harness strict and broke nineteen SpiderMonkey lenient-and-strict executions. - Negative tests must exit with code exactly 1 and print a diagnostic matching the expected error type. A Rust panic is a failure. Evidence that the body ran, such as
$DONOTEVALUATEoutput, is a failure. - Async tests must print
Test262:AsyncTestComplete, neverTest262:AsyncTestFailure, and exit 0. Positive tests must exit 0 with noTest262Errorin output, which catches assertions swallowed in promise reactions. - Module tests run the original file under
zipp mjsso self-imports resolve; script tests run from a temporary file in the test's own directory so relative fixture imports resolve. - Files are read byte-faithfully, undoing any
core.autocrlfmangling, becauseFunction.prototype.toStringmust reproduce the original line terminators. - Engine identity (
--version --json, including source commit and dirty flag) and corpus identity are printed before any test runs.
The CI gate
The test262 job in .github/workflows/security.yml checks out tc39/test262 at the pin, builds the release CLI with Rust 1.92.0, asserts that the binary's recorded source commit equals HEAD and is not dirty, runs the tooling unit tests, runs the unmodified corpus once with two workers, a 120-second timeout and --expected-failures tools/test262-expected-failures.txt, then runs the DateTimeFormat shard. Any unexpected failure, stale expectation or skip in the corpus run, or any DateTimeFormat failure, fails the job.
The release workflow calls this job at the exact tagged commit and will not publish without it. A green main at some other revision is not a release gate. The evidence artifact (test262.json, failures.txt, datetimeformat.json and datetimeformat-failures.txt) is retained for thirty days. The figures above are from a local run on this pin; hosted CI has not yet run against it.
Evidence
- tools/test262-expected-failures.txt, the one-entry manifest with its reason.
- tools/run_test262.py, the runner and scorer.
- tools/CLDR-DATA.md, the provenance of the generated DateTimeFormat CLDR data.
- For the previous pin,
4249661: the hosted run at 1539eb4b, the machine-readable summary of 14 September 2026, and the narrative log with a per-failure analysis. - The correctness audits of 12 September 2026.
How the number got here
The most important entry in this history is the first honest one. On 25 July 2026 the runner was found to be scoring a single execution mode and reporting 100%; fixed, it reported 96.97%. Everything after that is measured against a denominator that only ever got stricter.
- Runner added
tools: add test262 conformance runner, one day after the pivot to JavaScript. - 96.97%: the real pass rate
Three scorer defects fixed. 93,122 of 96,029 executions; the previous single-mode "100%" retracted in the commit subject.
- 97.72%
Regex literals validated at compile time, 712 more executions.
- 99.0% with a new in-house front end
Parse-negative failures fell from 607 files to 80.
- 99.88%, Intl402 96.9%
Intl.DurationFormat, the IANA time-zone database, and non-ISO Temporal calendars. - 99.997%
30 failures to 5 to 3; 22 of the 30 were repo or runner defects. Corpus refreshed to a newer pin with a different execution count.
- A 24-row drift, root-caused
Traced to a call-lowering change and reversed; 95,939 of 95,942 restored on 30 August.
- New pin: 95,665 of 95,680
First run on corpus 4249661; two engine defects found by JIT-only failures and fixed the same day.
- 95,671 unmodified, 95,680 corrected, 488 DateTimeFormat
An Annex B hidden defect fixed behind an obsolete expectation, CLDR 48 DateTimeFormat data generated, the corrected profile documented and gated.
- New pin: 96,229 of 96,230 unmodified, 490 DateTimeFormat
Corpus d53ebc8: 237 failures before the move, 218 of them the new iterator
chunks,windowsandincludes; all but the Annex B test fixed in1676c9e4, and the corrected profile retired.
Conformance is also verified locally in four modes: default, ZIPP_NOJIT=1, ZIPP_JIT_THRESHOLD=1 and ZIPP_NO_NURSERY=1, because default conformance alone cannot certify the native helper paths. The CI gate runs the default profile.
What this result does not claim
- It is not complete ECMA-402 conformance. DateTimeFormat is one shard, and the other Intl services ship English locale data only. The last full-suite Intl402 figure, 6,502 of 6,714 on an older corpus, is from 29 July 2026.
- It is not a claim about other engines. The expected-failure manifest records that current Node fails the same Annex B assertion, but no engine-versus-engine conformance table is published.
- It is a result at one pinned commit. A newer corpus will have a different execution count and may add tests Zipp fails; the number will be re-reported against the new pin, not carried forward.