CrewRig

ADR 0014 — CI test-wiring guard

Status: Accepted (issue #534; realizes spec 0076)

Context

The repository ships 48 scripts/tests/test-*.sh scripts, each wired into CI by an explicit per-test invocation in the check-components job. Nothing enforced that a new test script actually ran anywhere: at the time spec 0076 was picked up, 27 test scripts executed in zero workflows. A test that exists but runs nowhere is a silent false-negative — the class of regression that hid for months until issue #530 surfaced one by wiring it. Spec 0076 requires CI to fail whenever a scripts/tests/test-*.sh is neither executed by a workflow nor listed in an explicit, reasoned exemption allowlist, while preserving the ordered GitHub-Actions-to-GitLab command-parity guarantee (spec 0049).

Decision

Add a registration guard, scripts/check-test-wiring.sh, that enumerates every scripts/tests/test-*.sh and classifies each as WIRED (its full invocation token scripts/tests/<name> appears in a non-comment command position of any .github/workflows/*.yml or ci/ci-capabilities.yml) or EXEMPT (listed in ci/test-wiring-exemptions.txt with a recorded reason). A test that is neither fails the check by name (R1/R2); a reasonless exemption fails (R3); an exemption naming an absent test file fails (R5). The guard is a structural sibling of check-core-paths.sh — a check-*.sh with a test-*.sh regression twin (scripts/tests/test-check-test-wiring.sh) — and is itself run as a step in the existing check-components job on all three CI surfaces, so it cannot silently stop running (R6). The 27 orphans found at realization are each resolved to one terminal state (R4): hermetic passes are wired into check-components; environment-bound and genuinely-failing tests are exempted with a reason. The guard matches the full path (never the bare basename) and ignores #-commented lines, so a filename mentioned only in prose or a disabled run: | line never reads as a false "wired".

Alternatives considered

  • Auto-discovering glob runner — a job that discovers and runs every scripts/tests/test-*.sh in a loop. Rejected: it collapses the per-test CI granularity and the ordered command-parity model (ci/ci-capabilities.yml command: lists aligned step-by-step by check-ci-parity.sh) into an opaque loop that hides which test runs where, and it offers no place to record why a test is deliberately not run. The registration guard keeps every test an explicit, individually-traceable CI step and makes the never-run set an auditable allowlist. This is the owner-chosen strategy fixed by spec 0076.
  • A dedicated check-test-wiring capability and its own GHA/GitLab job, rather than a step inside check-components. Rejected: it enlarges the parity surface check-ci-parity.sh must reconcile and breaks the established precedent where check-*.sh guards run as steps within check-components.
  • A YAML exemption file (ci/test-wiring-exemptions.yml). Rejected: a <name><TAB>reason line format mirrors the .crewrig/core-paths.txt parser the guard already reuses, needs no yq dependency, and keeps per-entry reasons trivially greppable.

Consequences

  • Every new scripts/tests/test-*.sh must be wired into a workflow or exempted with a reason in the same PR, or CI fails naming it. Wiring lands symmetrically in .github/workflows/build.yml and ci/ci-capabilities.yml, with .gitlab-ci.yml regenerated by scripts/build-ci.sh, so parity stays green.
  • The exemption allowlist carries two kinds of entry: environment-dependent tests (cannot run in the hermetic check-components job) and quarantined tests (a genuine pre-existing failure whose fix is out of scope for spec 0076 and is tracked by a follow-up issue). The guard treats both identically; the reason string records the distinction.
  • Hygiene — remove the exemption when a quarantined test is fixed. R5 only catches an exemption whose test file was deleted. When a quarantined (bug-exempted) test is later fixed and wired into a workflow, its exemption entry — still naming an existing file — is redundant: the test is both wired and exempted, which the guard tolerates but which is dead weight that hides the fact that the underlying bug is resolved. The fixer MUST delete the stale line from ci/test-wiring-exemptions.txt as part of the fix. This obligation is stated in the allowlist header and here.
  • Rollback is a plain commit revert: the guard is additive, runs no destructive action, and the only downstream coordination is closing any follow-up issues opened for the quarantined tests if the guard is withdrawn.