JavaScript Test262 conformance
Test262 is the official ECMAScript conformance suite. Zipp runs the whole pinned corpus, in both sloppy and strict mode, with a scorer that cannot raise the pass rate by skipping tests. This page explains exactly what the numbers mean, what "corrected" means, and how the result is gated in CI.
The result
| Profile | Pass | Fail | Skip | Executions |
|---|---|---|---|---|
| Core corpus, five documented corrections | 95,680 | 0 | 0 | 95,680 |
| Core corpus, unmodified upstream | 95,671 | 9 | 0 | 95,680 |
test/intl402/DateTimeFormat, unmodified | 488 | 0 | 0 | 488 (244 files) |
The unmodified pass rate is 99.991%. The corrected-core result is 100%, and the two are always reported side by side; a patched result is never presented as unmodified upstream conformance.
The corpus is tc39/test262 at commit 4249661388e5d3f92a85186213da140a6481490f, walking test/ including the staging directory and excluding the separate ECMA-402 suite under test/intl402/, which is opt-in. Nothing is excluded by feature flag: Temporal, decorators, iterator helpers, ShadowRealm, resizable buffers and every other feature in the pinned corpus are in scope.
How executions are counted
95,680 is a count of executions, not files. Following the suite's own interpretation guide, a test that carries none of the onlyStrict, noStrict, raw or module flags runs twice, once as sloppy code and once with "use strict"; prepended. raw and module tests run once; onlyStrict and noStrict run once each in their mode.
Skips stay in the denominator. The runner's own comment says why: a change that made 500 tests unreadable must not be able to raise the reported pass rate. A skip can only arise from a read error or a missing harness include, and any skip fails the gate. Timeouts (20 seconds locally, 120 in CI) score as failures, never as skips.
What "corrected" means
The nine executions that fail against the unmodified corpus are not engine defects; they are places where the pinned test files contradict themselves or a later edition of the specification. Rather than hide them behind an exclusion list, Zipp keeps five reviewable unified patches under tools/test262-corrections/, each with the original and corrected SHA-256, a prose reason and a specification link. The patches touch five test files, never the harness, and remove no files, modes or cases.
| Test file | Executions | Why |
|---|---|---|
staging/sm/Error/constructor-proto.js, prototype-properties.js, prototype.js | 6 | The pinned harness/nativeErrors.js now lists Error itself, so the tests demand Reflect.getPrototypeOf(Error) === Error, contradicting their own earlier assertions. Node 24.19 fails all three unchanged. |
built-ins/TypedArray/prototype/slice/speciesctor-return-same-buffer-with-offset.js | 2 | The harness added an immutable-buffer factory; the test expects a successful write, but the immutable ArrayBuffer proposal requires a mutable destination. Zipp correctly throws TypeError. |
annexB/language/function-code/block-decl-func-skip-arguments.js | 1 | Carries ES2017 wording deleted in ES2018; Node 24.19 fails the same final assertion. Tracked upstream as tc39/test262 issue 5113. |
Two further failures, both modes of a German-language string test in staging, were fixed in the engine with no test patch. That is why the expected-failure manifest has nine entries rather than eleven.
The runner
tools/run_test262.py is a 566-line Python runner and scorer; tools/run_test262_dual.py drives the unmodified and corrected suites together and refuses to score unless the checkout is at the pinned revision, no newline conversion has occurred, the set of changed paths equals the manifest exactly, and there are no untracked files. Both runners have their own unit tests.
- The harness is passed as a separate script (
zipp js --script-goal <test> <harness>), never concatenated. Concatenation had made the harness strict and broke nineteen SpiderMonkey lenient-and-strict executions. - Negative tests must exit with code exactly 1 and print a diagnostic matching the expected error type. A Rust panic is a failure. Evidence that the body ran, such as
$DONOTEVALUATEoutput, is a failure. - Async tests must print
Test262:AsyncTestComplete, neverTest262:AsyncTestFailure, and exit 0. Positive tests must exit 0 with noTest262Errorin output, which catches assertions swallowed in promise reactions. - Module tests run the original file under
zipp mjsso self-imports resolve; script tests run from a temporary file in the test's own directory so relative fixture imports resolve. - Files are read byte-faithfully, undoing any
core.autocrlfmangling, becauseFunction.prototype.toStringmust reproduce the original line terminators. - Engine identity (
--version --json, including source commit and dirty flag) and corpus identity are printed before any test runs.
The CI gate
The test262 job in .github/workflows/security.yml checks out tc39/test262 at the pin, builds the release CLI with Rust 1.92.0, asserts that the binary's recorded source commit equals HEAD and is not dirty, runs the tooling unit tests, runs the dual suite with two workers and a 120-second timeout, then runs the DateTimeFormat shard. Any unexpected failure, stale expectation or skip in the unmodified run, any failure at all in the corrected run, or any DateTimeFormat failure fails the job.
The release workflow calls this job at the exact tagged commit and will not publish without it. A green main at some other revision is not a release gate. The evidence artifact (upstream.json, corrected.json, both failure lists, comparison.json and datetimeformat.json) is retained for thirty days, and the durable summary lives in the repository.
Evidence
- Hosted run at 1539eb4b, the clean-build confirmation of the table above.
- docs/validation/2026-09-14-test262-datetimeformat.json, the machine-readable summary with engine identity, corpus pin, correction hashes and report hashes.
- docs/validation/2026-09-14-ci-readiness.md, the narrative log with a per-failure analysis.
- tools/test262-corrections/, the manifest and the five patches.
- tools/test262-expected-failures.txt, the nine-entry manifest with reasons.
- tools/run_test262.py and tools/run_test262_dual.py.
- The correctness audits of 12 September 2026.
How the number got here
The most important entry in this history is the first honest one. On 25 July 2026 the runner was found to be scoring a single execution mode and reporting 100%; fixed, it reported 96.97%. Everything after that is measured against a denominator that only ever got stricter.
- Runner added
tools: add test262 conformance runner, one day after the pivot to JavaScript. - 96.97%: the real pass rate
Three scorer defects fixed. 93,122 of 96,029 executions; the previous single-mode "100%" retracted in the commit subject.
- 97.72%
Regex literals validated at compile time, 712 more executions.
- 99.0% with a new in-house front end
Parse-negative failures fell from 607 files to 80.
- 99.88%, Intl402 96.9%
Intl.DurationFormat, the IANA time-zone database, and non-ISO Temporal calendars. - 99.997%
30 failures to 5 to 3; 22 of the 30 were repo or runner defects. Corpus refreshed to a newer pin with a different execution count.
- A 24-row drift, root-caused
Traced to a call-lowering change and reversed; 95,939 of 95,942 restored on 30 August.
- New pin: 95,665 of 95,680
First run on corpus 4249661; two engine defects found by JIT-only failures and fixed the same day.
- 95,671 unmodified, 95,680 corrected, 488 DateTimeFormat
An Annex B hidden defect fixed behind an obsolete expectation, CLDR 48 DateTimeFormat data generated, the corrected profile documented and gated.
Conformance is also verified locally in four modes: default, ZIPP_NOJIT=1, ZIPP_JIT_THRESHOLD=1 and ZIPP_NO_NURSERY=1, because default conformance alone cannot certify the native helper paths. The CI gate runs the default profile.
What this result does not claim
- It is not complete ECMA-402 conformance. DateTimeFormat is one 244-file shard, and the other Intl services ship English locale data only. The last full-suite Intl402 figure, 6,502 of 6,714 on an older corpus, is from 29 July 2026.
- It is not a claim about other engines. The corrections document that Node 24.19 fails four of the same executions, but no engine-versus-engine conformance table is published.
- It is a result at one pinned commit. A newer corpus will have a different execution count and may add tests Zipp fails; the number will be re-reported against the new pin, not carried forward.