edufeed/edufeed-app

Watch

Fork 0

pnpm test on dev is not a trustworthy signal: random teardown error + 2 env-dependent failures #55

New issue

Open

opened 2026-07-30 06:14:30 +00:00 by laoc · 1 comment

laoc commented

2026-07-30 06:14:30 +00:00

Owner

pnpm test on dev is not currently a trustworthy pass/fail signal. Two independent problems, neither of which fails a test.

1. Random non-zero exit from an `EnvironmentTeardownError` (no test fails)

Every so often the run ends like this while reporting every test passed:

Test Files  468 passed (468)
Tests      5149 passed (5149)
     Errors  1 error
 ELIFECYCLE  Test failed.

The error is an unhandled rejection during environment teardown:

EnvironmentTeardownError: Cannot load .../svelte/src/reactivity/reactive-value.js
imported from .../svelte/src/reactivity/media-query.js after the environment was torn down.

last recorded callstack:
  svelte/src/reactivity/media-query.js
  svelte/src/reactivity/index-client.js
  src/lib/services/app-relay-service.svelte.js
  src/lib/stores/accounts.svelte.js
  src/lib/components/shared/GlobalFAB.svelte
  src/lib/components/__tests__/GlobalFAB.test.js

A lazy svelte/reactivity import lands after vitest has torn down the jsdom environment for GlobalFAB.test.js.

It is a race, and the count varies run to run on an unchanged tree. Five consecutive full runs on clean dev:

0, 0, 1, 0, 2   errors

It does not reproduce when GlobalFAB.test.js is run alone, or together with just the membership suites — it needs full-suite load. Consequence: CI on dev goes red at random with nothing actually broken, and a genuinely broken build is indistinguishable from this at a glance.

Worth knowing for anyone debugging it: two samples are not enough to attribute this to a branch. It cost me a nearly-published false attribution — I saw 5 errors on a feature branch, got 0 twice on clean dev, and concluded the branch caused it. Sampling five each showed the same distribution.

2. Two tests fail on a fresh clone because `pnpm test` does not load `.env`

FAIL scripts/lib/__tests__/publish-forms-build.test.mjs
FAIL scripts/lib/__tests__/publish-forms-build.amb.test.mjs
     Error: Missing env: SCHEME_NADDR_EDUCATIONAL_LEVEL

Both read process.env directly, and vitest run does not populate it from .env. Workaround:

set -a; . ./.env; set +a; pnpm test

That is how every "5149/5149" figure in the membership thread was obtained. Anyone on a fresh clone — and CI, which has no .env — sees two red tests unrelated to their change.

Fix direction (not prescriptive): either load the env in the vitest setup for those files, or have the tests supply their own fixture values instead of reading the ambient environment. The second is probably better — a unit test that needs 25 real naddrs from a deployment env is testing the environment as much as the code.

Not urgent, but do not lose it

Neither problem is a product bug and neither blocks a release. They matter because together they make a red test run uninformative, which is exactly when you most want it to be informative.

Filed at the request of @laoc_buzz from the membership thread in #edufeed-app. Observed on dev @ 09842a48.

`pnpm test` on `dev` is not currently a trustworthy pass/fail signal. Two independent problems, neither of which fails a test. ## 1. Random non-zero exit from an `EnvironmentTeardownError` (no test fails) Every so often the run ends like this while reporting **every test passed**: ``` Test Files 468 passed (468) Tests 5149 passed (5149) Errors 1 error ELIFECYCLE Test failed. ``` The error is an unhandled rejection during environment teardown: ``` EnvironmentTeardownError: Cannot load .../svelte/src/reactivity/reactive-value.js imported from .../svelte/src/reactivity/media-query.js after the environment was torn down. last recorded callstack: svelte/src/reactivity/media-query.js svelte/src/reactivity/index-client.js src/lib/services/app-relay-service.svelte.js src/lib/stores/accounts.svelte.js src/lib/components/shared/GlobalFAB.svelte src/lib/components/__tests__/GlobalFAB.test.js ``` A lazy `svelte/reactivity` import lands after vitest has torn down the jsdom environment for `GlobalFAB.test.js`. **It is a race, and the count varies run to run on an unchanged tree.** Five consecutive full runs on clean `dev`: ``` 0, 0, 1, 0, 2 errors ``` It does not reproduce when `GlobalFAB.test.js` is run alone, or together with just the membership suites — it needs full-suite load. Consequence: **CI on `dev` goes red at random with nothing actually broken**, and a genuinely broken build is indistinguishable from this at a glance. Worth knowing for anyone debugging it: two samples are not enough to attribute this to a branch. It cost me a nearly-published false attribution — I saw 5 errors on a feature branch, got 0 twice on clean `dev`, and concluded the branch caused it. Sampling five each showed the same distribution. ## 2. Two tests fail on a fresh clone because `pnpm test` does not load `.env` ``` FAIL scripts/lib/__tests__/publish-forms-build.test.mjs FAIL scripts/lib/__tests__/publish-forms-build.amb.test.mjs Error: Missing env: SCHEME_NADDR_EDUCATIONAL_LEVEL ``` Both read `process.env` directly, and `vitest run` does not populate it from `.env`. Workaround: ```bash set -a; . ./.env; set +a; pnpm test ``` That is how every "5149/5149" figure in the membership thread was obtained. Anyone on a fresh clone — and CI, which has no `.env` — sees two red tests unrelated to their change. Fix direction (not prescriptive): either load the env in the vitest setup for those files, or have the tests supply their own fixture values instead of reading the ambient environment. The second is probably better — a unit test that needs 25 real `naddr`s from a deployment env is testing the environment as much as the code. ## Not urgent, but do not lose it Neither problem is a product bug and neither blocks a release. They matter because together they make a red test run uninformative, which is exactly when you most want it to be informative. Filed at the request of @laoc_buzz from the membership thread in #edufeed-app. Observed on `dev` @ `09842a48`.

laoc referenced this issue

2026-07-30 06:15:59 +00:00

WIP: dev -> main release gate — config/env that must be decided and set before deploy #56

laoc referenced this issue

2026-07-30 10:09:58 +00:00

fix(search): key ext facet filters on the NIP-AMB <ns>:<facet> grammar (#54) #58

laoc commented

2026-07-30 10:55:43 +00:00

Author

Owner

Second, deterministic instance: three tests pass only because of the developer's `direnv`

Found while baselining the suite for #31. On a fresh worktree with .env copied but direnv not allowed, three tests fail on clean dev @ 3353662b:

× scripts/lib/__tests__/publish-forms-build.test.mjs
    > emits field-vocab and field-output tags for vocab-bound fields
× scripts/lib/__tests__/publish-forms-build.amb.test.mjs
    > groups amb-basic fields into sections that round-trip
× src/lib/components/educational/__tests__/ResourceFormWizard.edit-prefill.svelte.test.js
    > without the guard: same event triggers effect_update_depth_exceeded

The first two fail with Error: Missing env: SCHEME_NADDR_EDUCATIONAL_LEVEL at scripts/lib/publish-forms-build.mjs:16.

Why it is a test bug, not a config bug

publish-forms-build.test.mjs:79-83 deliberately stubs the env so it does not need real values:

beforeEach(() => {
  vi.stubEnv('SCHEME_NADDR_SCHULFAECHER', fakeNaddr('schulfaecher'));
  vi.stubEnv('SCHEME_NADDR_HCRT', fakeNaddr('hcrt'));
});

But amb-basic in scripts/data/edufeed-forms.json:81 grew a third vocabRef — educational-level — and the stub list was never updated. Vitest does not load .env into process.env, so the test should have failed from that commit onward. It does not, because .envrc contains dotenv_if_exists: with direnv active the shell exports all 25 SCHEME_NADDR_* vars, req() finds the real value, and the gap is invisible.

Confirmed both directions, same commit, same checkout:

without exported env:  Test Files 2 failed | Tests 2 failed | 6 passed
with .env exported:    Test Files 2 passed | Tests 8 passed

So the tests are green on every developer machine with direnv and red anywhere without it — CI, a container, a fresh worktree. Whichever way CI is set up, one of the two states is lying.

Fix

Add the missing vi.stubEnv('SCHEME_NADDR_EDUCATIONAL_LEVEL', fakeNaddr('educational-level')) — and better, derive the stub list from the form's own vocabRefs so adding a fourth cannot silently reintroduce this. A test that reads ambient env is exactly the class of thing this issue is about.

The third failure (ResourceFormWizard.edit-prefill, 10.5s before failing) also cleared once the env was exported, so it is likely the same root cause via a slow path rather than an independent flake — I did not isolate it further.

Full suite with the env exported: 5177/5177, 468 files. The GlobalFAB teardown race in the original report still surfaces intermittently as Errors: 1 and is unaffected by any of this.

## Second, deterministic instance: three tests pass only because of the developer's `direnv` Found while baselining the suite for #31. On a **fresh worktree with `.env` copied but direnv not allowed**, three tests fail on clean `dev` @ `3353662b`: ``` × scripts/lib/__tests__/publish-forms-build.test.mjs > emits field-vocab and field-output tags for vocab-bound fields × scripts/lib/__tests__/publish-forms-build.amb.test.mjs > groups amb-basic fields into sections that round-trip × src/lib/components/educational/__tests__/ResourceFormWizard.edit-prefill.svelte.test.js > without the guard: same event triggers effect_update_depth_exceeded ``` The first two fail with `Error: Missing env: SCHEME_NADDR_EDUCATIONAL_LEVEL` at `scripts/lib/publish-forms-build.mjs:16`. ### Why it is a test bug, not a config bug `publish-forms-build.test.mjs:79-83` deliberately stubs the env so it does not need real values: ```js beforeEach(() => { vi.stubEnv('SCHEME_NADDR_SCHULFAECHER', fakeNaddr('schulfaecher')); vi.stubEnv('SCHEME_NADDR_HCRT', fakeNaddr('hcrt')); }); ``` But `amb-basic` in `scripts/data/edufeed-forms.json:81` grew a third `vocabRef` — `educational-level` — and the stub list was never updated. Vitest does not load `.env` into `process.env`, so the test should have failed from that commit onward. It does not, because `.envrc` contains `dotenv_if_exists`: with direnv active the shell exports all 25 `SCHEME_NADDR_*` vars, `req()` finds the real value, and the gap is invisible. Confirmed both directions, same commit, same checkout: ``` without exported env: Test Files 2 failed | Tests 2 failed | 6 passed with .env exported: Test Files 2 passed | Tests 8 passed ``` So the tests are green on every developer machine with direnv and red anywhere without it — CI, a container, a fresh worktree. Whichever way CI is set up, one of the two states is lying. ### Fix Add the missing `vi.stubEnv('SCHEME_NADDR_EDUCATIONAL_LEVEL', fakeNaddr('educational-level'))` — and better, derive the stub list from the form's own `vocabRef`s so adding a fourth cannot silently reintroduce this. A test that reads ambient env is exactly the class of thing this issue is about. The third failure (`ResourceFormWizard.edit-prefill`, 10.5s before failing) also cleared once the env was exported, so it is likely the same root cause via a slow path rather than an independent flake — I did not isolate it further. Full suite with the env exported: **5177/5177, 468 files.** The `GlobalFAB` teardown race in the original report still surfaces intermittently as `Errors: 1` and is unaffected by any of this.

laoc referenced this issue

2026-07-30 12:22:46 +00:00

fix(calendar): write updated events to the EventStore so the cache is not left stale (#62) #63

No labels

bug

discussion

enhancement

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

edufeed/edufeed-app#55

No description provided.

Rows
Columns

pnpm test on dev is not a trustworthy signal: random teardown error + 2 env-dependent failures #55

1. Random non-zero exit from an EnvironmentTeardownError (no test fails)

2. Two tests fail on a fresh clone because pnpm test does not load .env

Not urgent, but do not lose it

Second, deterministic instance: three tests pass only because of the developer's direnv

Why it is a test bug, not a config bug

Fix

1. Random non-zero exit from an `EnvironmentTeardownError` (no test fails)

2. Two tests fail on a fresh clone because `pnpm test` does not load `.env`

Second, deterministic instance: three tests pass only because of the developer's `direnv`