6 forked calendar events on relay.edufeed.org (same pubkey+d under both 31922 and 31923) #67
Labels
No labels
bug
discussion
enhancement
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
edufeed/edufeed-app#67
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-up to #65, which was preventive only — it stops the app creating new forks but repairs nothing existing. TestOER raised that there might be real forked pairs on production. There are: 6, measured.
What I measured
nak countonwss://relay.edufeed.org, bogus-kind control (kinds:[65535]-> 0) so COUNT is honoured:Then paginated both kinds by
untiland grouped by(pubkey, d):6 is a lower bound, not a total — the 31923 scan reached 95.4% coverage. Events sharing a
created_atsecond at a page boundary are where the remainder sits.The 6
These are NOT from the #65 code path
Worth stating plainly, because the obvious reading is "the toggle bug did this" and the evidence says otherwise.
The app mints its own d-tags as
event-${Date.now()}-${Math.random().toString(36).substr(2,9)}(calendar-actions.svelte.js:68). None of the 6 has that shape — 4 arebildungsserver:<id>, 1 isoerworldmap:<uuid>, 1 is a bare UUID. All six carry importer-assigned identifiers, and 4 of 6 share a single pubkey.So the cause is an importer republishing the same source record under a different kind on a later run — the same
(pubkey, d)with a different date/time decision. #66 would never have prevented these, and fixing the app does not stop more appearing.What this needs
Not doing any of the deletions without an explicit go-ahead — they are on a production relay and two of the pairs are not obvious duplicates.
Reproducing the scan
Group all 31922 and all 31923 by
(pubkey, d-tag)and intersect. Two traps:nak reqfilters client-side so it can look right while the relay honoured nothing (usenak countfor the totals and a bogus-kind control), and naiveuntilpagination stalls on a page boundary where many events share one second — stepuntilpast it or you silently stop at ~84%.Correction: 1 of the 6 IS from #65, and my d-tag-shape test could not have seen it
The reasoning in the issue body above is wrong on this point. TestOER found the flaw and I have verified it against the relay and the source myself.
Why the d-tag-shape test was invalid as a causal test.
createEventmintsevent-<ms>-<rand>— butupdateEventdoes not mint anything, it reads the d-tag off the event it replaces (calendar-actions.svelte.js:139). So d-tag shape tells you how an event was created, never how a fork was produced. A user editing an imported event in the app forks it while keeping the importer'sbildungsserver:…or bare-UUID d-tag — exactly the case my test excluded by construction. I had read that line myself earlier in the same session and still built the inference on top of it.The pair that is ours, re-queried directly and confirmed (bogus-
#dcontrol returns 0):139 seconds apart, all-day -> timed, and the later one carries this app's own NIP-89
clienttag. It also gained precisely what a filled edit form produces: a unixstartand anend,start_tzid/end_tzid, animage, alocation.Two supporting claims I checked in source rather than taking:
Dtags are emitted only for kind 31923 (calendar.js:842,if (eventData.kind === 31923 && eventData.start)). The older 31922 carries aDtag, so it was not built by this app — consistent with imported-then-edited-here.buildCalendarEventTagshas exactly two callers insrc/—calendar-actions.svelte.js:74(create) and:167(update), excluding__tests__. So anEdufeed-tagged event on a foreign d-tag can only have come throughupdateEvent.What this changes
bildungsserver:x4,oerworldmap:x1clienttag)a68eacf6-…So "fixing the app does not stop more appearing" was wrong — it holds for the 5 importer pairs and not for the user-caused class. #66 does stop that class.
Remediation, revised
The user pair needs no production write from us. The pubkey belongs to a person who owns the key, and the app's own Delete affordance is available to the event owner (
CalendarEventDetailView.svelte:196). The 31923 is the finished version — the 31922 has noend,locationorimage— so if a winner is picked it is the 31923, and the owner can retire the stale 31922 themselves.The 5 importer pairs are unchanged and still need whoever owns the importer: a stable kind decision per source record, or a NIP-09 delete of the old-kind event when the decision changes.
Coverage, stated jointly
TestOER ran an independent scan and landed on the same 680 / 2314 / 6, hitting the same pagination stall (a dense
created_atsecond returning+0 new, needing the cursor walked back a second). 95.4% of kind 31923 is a shared ceiling, not one run's limitation — two independent scans agreeing at the same coverage is not the same as coverage being complete. 6 remains a lower bound.Second correction: 6 is exact, not a lower bound
The "95.4% coverage, so 6 is a floor" caveat is now closed. TestOER closed it and I reproduced the decisive half independently, by a different query path.
The method that closes it: you never needed the missing 31923s. A fork requires a member of both kinds at one
(pubkey, d), so you can enumerate coordinates from the 31922 side and query 31923 point-wise with#d. That is a completely different query path from the time-scan both of us originally ran, so it is not the earlier scan agreeing with itself.My independent sweep, COUNT-driven bucketing (split a window when its COUNT exceeds 400, rather than when the returned batch looks full):
Same six, same single
client=Edufeed.The argument that actually closes it is the author-side one, and it does not depend on 31922 being complete. The unreachable 31923s live in two dense seconds that are single-author, and that author has no 31922 events at all:
No 31922 from that author anywhere on the relay, so those events cannot form a pair no matter how many of them stay unreachable. Verified directly.
One caveat on the other proof, from my own run
I could not reproduce "31922 is 100% complete". I got 679 of COUNT 680, and narrowing the one bad window shows the shortfall is not a cap — it is COUNT and REQ disagreeing at 1-2 event granularity:
Both windows are one second wide with counts far below any limit, so nothing is being truncated. Either COUNT is over-reporting (counting something the read path will not serve — a deleted or superseded event would do it) or REQ is withholding. I cannot tell which from outside the relay, and it is a small enough discrepancy that I am recording it rather than chasing it here.
It does not weaken the conclusion, because the author-side argument stands alone. But it is the reason I would state the result as "6 exact, on the author-side proof" rather than resting it on 31922 completeness — which is a claim my own scan does not support.
Also found, minor
aa404cd7…is a kind 31922 carrying nodtag at all — titleTesttermin 201, author67aea7eb…. An addressable event without adis not addressable: it can never be replaced, and noatag can reference it. Not a fork, not urgent, presumably a leftover test event. Independently confirmed; noting it here since this is where calendar-data cleanup is tracked.And no coordinate holds more than one live 31922 (678 coords, 0 with >1), so there is no second class of duplication hiding in that kind.
Retracting the "COUNT/REQ divergence" — and replacing it with the actual cause
TestOER was right that my framing was wrong, and right that
untilis inclusive. Both are my errors:sinceanduntilare both inclusive (NIP-01:created_at >= since,created_at <= until). I described[X, X+1)windows that were really two seconds wide and overlapping. Confirmed empirically:{since: X, until: X}returns exactly the events atcreated_at == X. My bucketer also split as[lo,mid] + [mid,hi], double-querying the boundary second — harmless because we dedupe by id, but wasteful. Correct split is[lo,mid] + [mid+1,hi].But the divergence was real and reproducible on my side, 3/3 — and the cause is a genuine data finding.
relay.edufeed.orgis serving a kind-31922 event whose signature does not verify.Re-scanned the whole kind accounting for it, with
mid+1splitting and a per-window check:Both of my anomalies were this one event. The unbounded query returning 499 rather than 500 was the relay serving 500 and
nakdropping the bad one — I checked, and that query's stderr carries exactly onebad signatureline. So my "the relay returns 499" explanation for the silent quarter-scan was wrong too; the cap is 500 and the shortfall was the same single event.Why neither of our readers told us
nakverifies signatures and drops failures to stderr. My parser read stdout only, so a rejected event vanished with no error — a silent, plausible undercount.Neither number was wrong about the relay; they were measuring different things. 679 valid + 1 invalid = 680 served.
It does not change the fork count
The bad-signature event sits at a coordinate my complete-side sweep never saw, so it could have hidden a seventh pair. It does not:
No 31923 at that address, so it is not a fork. 6 remains exact.
What this adds to the cleanup list
Alongside the malformed no-
dTesttermin 201, this relay now has a known invalid-signature event in the calendar data. Any client that verifies (most do) will silently ignore it, so it is effectively invisible to users while still occupying its coordinate and being counted by COUNT.Whether the relay should have accepted it at all is a relay-side question, not an app one — NIP-01 puts signature validation on the relay before acceptance. I am raising that in the relay lane rather than here; this issue just records that the event exists and that it is not one of the six.
Correction: that signature is valid. My "bad signature" reading was wrong.
Retracting the previous comment's framing. TestOER caught it and I reproduced the disproof by construction rather than taking it.
The event has a raw U+001E (RECORD SEPARATOR) in its content, and the two serializations disagree:
The second hash is exactly the id
nak verifyreports as "expected". So the event'sidand signature are internally consistent under JSON escaping;nak/go-nostr computes a different id and therefore reports "bad signature".The spec genuinely is ambiguous here. NIP-01 lists exactly seven characters that must be escaped —
\n \" \\ \r \t \b \f— and says "all other characters must be included verbatim". RFC 8259 §7 requires every character below U+0020 to be escaped. For an unlisted control character the two rules contradict, and each implementation picked a side.One data point I can add: this relay is strfry, not a Go relay.
strfry validates signatures on ingest and accepted this event, so its serializer escapes the character too. That makes it two independent implementations (nostr-tools, strfry) escaping, and go-nostr the outlier — which is worth knowing before anyone decides the JS side is the one that is wrong.
What this changes for this issue
#dcontrol.0a45c707…is the oerworldmap importer — the same author as one of the six forked pairs. U+001E is almost certainly a stray field separator from the source record that was never meant to be in the text, so the importer should strip or replace control characters before publishing.Why it is still worth recording
These events are invisible to Go-based clients and tooling. Anything on go-nostr computes a different id and drops the event as unverifiable — which is exactly what happened to my scan, silently, because
nakreports the rejection on stderr while my parser read stdout only.Scope, measured across every fetchable calendar event: 1 event of 2994 carries an unlisted control character (kind 31922: 1 of 680; kind 31923: 0 of 2314). edufeed-app is on nostr-tools/applesauce, so the app renders it fine — the asymmetry only bites at the Go boundary.