Relay provenance, cross-relay ranking, passages fan-out, find_community #2

Merged
laoc merged 21 commits from feat/relay-provenance-ranking into main 2026-11-09 08:39:37 +00:00
Owner

Added

  • resolve_author candidates include picture.
  • Results carry relay; naddrs are encoded with the source relay as hint;
    addressable events are deduplicated across relays (newest wins).
  • search_content/search_resources rank across relays (snippet + lexical
    title/description match), limit after the merge, zero-match hits
    last (never dropped); new lexical field.
  • search_passages now searches every selected relay that has a passage
    index (not just the first) and merges the hits, sorted by score desc,
    before the per-document cap and limit; new relaysSearched and
    relaysIncomplete (relays whose indexer rejected the request) fields,
    alongside the existing relaySearched (kept for compatibility). Each
    metadataOnlyMatches naddr hints the relay its hit actually came from.
  • search_calendar_events returns NIP-52 time semantics the model no longer
    has to know: effectiveEnd (Unix seconds when the event is over — an
    end-less 31923 ends at its start, an end-less 31922 is a one-day event,
    an end at or before start counts as missing), allDay, lastDay (the
    inclusive last day of an all-day event), and endTzid inherited from
    startTzid.
  • The edufeed educational attributes ("Calendar events in edufeed" wiki) are
    exposed as attributes (registrationRequired, price with free,
    attendanceMode, educationalLevel with labels), plus namedParticipants
    (participant tags) and communities (h tags), and filterable with the
    new free, registrationRequired, attendanceMode and educationalLevel
    parameters (relay facets on nope-relay's calendar collection).
  • Every calendar event carries author {pubkey, name, bot} from the
    relay's kind-0 index (NIP-24 bot marks mirrors and crawlers), cached per
    author for 10 minutes.
  • search_calendar_events covers all four NIP-52 kinds: kinds: [31924]
    returns calendars (calendars: title, description, eventRefs) and
    kinds: [31925] RSVPs (rsvps: status, freeBusy — dropped when
    declined, eventRef, eventId, eventAuthor, note); an RSVP without a
    valid status is ignored. New references (#a) and participants
    (#p) filters answer "who RSVP'd to event X", "which calendars list X"
    and "events naming participant Y". Events expose calendarRefs, the
    kind-31924 calendars they ask to be included in.
  • Events expose their D day tags as days; an RSVP whose a does not
    reference a 31922/31923 coordinate is ignored; an empty
    eventAttendanceMode is no statement.
  • New find_community tool: free-text matches a topic against Communikey
    community definitions (kind 10222) by name/description (lexical score,
    same ranking helper as search_content), for "welche Community beschäftigt
    sich mit …" questions. Per the Communikey spec ("Communities = npubs"),
    name/picture are resolved from the community pubkey's own kind-0
    profile (one batched query alongside the 10222 list) — description is
    the 10222's own override tag, falling back to the kind-0 about.
    Returns {pubkey, npub, name, description, picture?, relay?} per match —
    pubkey can be passed as the community parameter of
    search_content/search_calendar_events. Reads from the new
    COMMUNITY_RELAYS env var (comma-separated; unset disables the tool), a
    5-minute in-memory stale-while-revalidate cache per process.

Fixed

  • Attribute read rules now match edufeed-app's calendar-attributes.js
    exactly: the first VALID tag of a name wins (an invalid one no longer
    shadows a later valid one), the attendance mode is trimmed and read by
    its last path segment, an empty educationalLevel:id is skipped, and an
    empty title falls back to the deprecated name.
  • Geohash queries dropped most results: the client re-applied #g locally
    as an exact match while the relay answers it as a prefix, so only events
    whose geohash equalled the queried prefix survived. #g is now relay-only.
  • CALENDAR_HIDDEN_PUBKEYS (hex, npub or kind-30000 naddr entries, as in
    edufeed-app) hides bulk calendar sources from search_calendar_events by
    default; the response reports hiddenSources and includeHiddenSources: true returns them. A bot flag alone hides nothing.

Changed

  • License classification reads ALL license:id tags of a learning resource
    (falling back to the plain license tag) and follows the shared
    amb-mcp/amb-indexer contract: DINI AG KIM license-type concepts
    (https://w3id.org/kim/license/…) are composed as a set into a CC variant
    (e.g. LEHRE LADEN [attribution, shareAlike] → BY-SA (KIM), open;
    [attribution, nonCommercial] → BY-NC (KIM), cc; noLicense → other),
    and GPL/LGPL/AGPL/GFDL/MIT/Apache-2.0/BSD URIs are now open (full
    access) instead of unknown. The de.wikipedia "Urheberrecht (Deutschland)"
    marker is labelled "Alle Rechte vorbehalten" (still metadata-only). A CC
    URI on the event still takes precedence. licenseFilter: "open" /
    "cc-any", accessLevel and get_resource's openLicense follow suit;
    the relay-side license.id prefilter lists include the KIM terms and the
    stored free-licence spellings.

Live check

Live run against wss://amb-relay.edufeed.org + wss://oersi.edufeed.org + wss://sodix.edufeed.org (search_content) and wss://relay.edufeed.org (find_community), via scripts/rank-eval.ts. Both acceptance checks pass — top-1 matches the expected result for each:

  • "Statistik Vorkurs": 1. Vorkurs Mathematik "11 Stochastik" (oersi, score=0.718, lexical=0.750) · 2. Künstliche Intelligenz und Maschinelles Lernen mit Entscheidungsbäumen (sodix, score=0.700, lexical=0.000)
  • "Unterrichtsentwurf Nachhaltigkeit": 1. Unterrichtsentwurf Ölverschmutzung im Wattenmeer (oersi, score=0.700, lexical=0.625) · 2. Unterrichtsentwurf Mikroplastik im Meer (oersi, score=0.087, lexical=0.625)
  • "OER Lizenzen Creative Commons": 1. OER – wie funktioniert das und wofür brauche ich das? (sodix, score=0.750, lexical=0.625) · 2. OER erstellen: Lizenzwahl (oersi, score=0.700, lexical=0.625)
  • "Unaufmerksamkeit Studierende Seminar": 1. Der „Filter“ (amb-relay, score=0.703, lexical=0.250) · 2. „Was tun, wenn's stockt im Seminar?" (amb-relay, score=0.075, lexical=0.417)
  • "gendersensibler Unterricht": 1. Didaktische Materialsammlung zur Bekämpfung von Geschlechterstereotypen (sodix, score=0.700, lexical=0.250) · 2. Geschlechtergrundlagen (amb-relay, score=0.700, lexical=0.125)
  • find_community "e-teaching": 1. e-teaching.org (top, as expected)
  • find_community "Religion": 8 matches, top is @Fortbildung_Religion

No change to rank.ts was needed — both brief-mandated first-place checks already passed.

Deploy note

COMMUNITY_RELAYS=wss://relay.edufeed.org must be set on the instances that should expose find_community; nothing else changes without it.

🤖 Generated with Claude Code

### Added - `resolve_author` candidates include `picture`. - Results carry `relay`; naddrs are encoded with the source relay as hint; addressable events are deduplicated across relays (newest wins). - search_content/search_resources rank across relays (snippet + lexical title/description match), `limit` after the merge, zero-match hits last (never dropped); new `lexical` field. - search_passages now searches every selected relay that has a passage index (not just the first) and merges the hits, sorted by score desc, before the per-document cap and `limit`; new `relaysSearched` and `relaysIncomplete` (relays whose indexer rejected the request) fields, alongside the existing `relaySearched` (kept for compatibility). Each `metadataOnlyMatches` naddr hints the relay its hit actually came from. - `search_calendar_events` returns NIP-52 time semantics the model no longer has to know: `effectiveEnd` (Unix seconds when the event is over — an end-less 31923 ends at its start, an end-less 31922 is a one-day event, an `end` at or before `start` counts as missing), `allDay`, `lastDay` (the inclusive last day of an all-day event), and `endTzid` inherited from `startTzid`. - The edufeed educational attributes ("Calendar events in edufeed" wiki) are exposed as `attributes` (`registrationRequired`, `price` with `free`, `attendanceMode`, `educationalLevel` with labels), plus `namedParticipants` (`participant` tags) and `communities` (`h` tags), and filterable with the new `free`, `registrationRequired`, `attendanceMode` and `educationalLevel` parameters (relay facets on nope-relay's calendar collection). - Every calendar event carries `author` `{pubkey, name, bot}` from the relay's kind-0 index (NIP-24 `bot` marks mirrors and crawlers), cached per author for 10 minutes. - `search_calendar_events` covers all four NIP-52 kinds: `kinds: [31924]` returns calendars (`calendars`: title, description, `eventRefs`) and `kinds: [31925]` RSVPs (`rsvps`: status, `freeBusy` — dropped when declined, `eventRef`, `eventId`, `eventAuthor`, note); an RSVP without a valid status is ignored. New `references` (`#a`) and `participants` (`#p`) filters answer "who RSVP'd to event X", "which calendars list X" and "events naming participant Y". Events expose `calendarRefs`, the kind-31924 calendars they ask to be included in. - Events expose their `D` day tags as `days`; an RSVP whose `a` does not reference a 31922/31923 coordinate is ignored; an empty `eventAttendanceMode` is no statement. - New `find_community` tool: free-text matches a topic against Communikey community definitions (kind 10222) by name/description (lexical score, same ranking helper as search_content), for "welche Community beschäftigt sich mit …" questions. Per the Communikey spec ("Communities = npubs"), `name`/`picture` are resolved from the community pubkey's own kind-0 profile (one batched query alongside the 10222 list) — `description` is the 10222's own override tag, falling back to the kind-0 `about`. Returns `{pubkey, npub, name, description, picture?, relay?}` per match — `pubkey` can be passed as the `community` parameter of `search_content`/`search_calendar_events`. Reads from the new `COMMUNITY_RELAYS` env var (comma-separated; unset disables the tool), a 5-minute in-memory stale-while-revalidate cache per process. ### Fixed - Attribute read rules now match edufeed-app's `calendar-attributes.js` exactly: the first VALID tag of a name wins (an invalid one no longer shadows a later valid one), the attendance mode is trimmed and read by its last path segment, an empty `educationalLevel:id` is skipped, and an empty `title` falls back to the deprecated `name`. - Geohash queries dropped most results: the client re-applied `#g` locally as an exact match while the relay answers it as a prefix, so only events whose geohash equalled the queried prefix survived. `#g` is now relay-only. - `CALENDAR_HIDDEN_PUBKEYS` (hex, npub or kind-30000 naddr entries, as in edufeed-app) hides bulk calendar sources from `search_calendar_events` by default; the response reports `hiddenSources` and `includeHiddenSources: true` returns them. A `bot` flag alone hides nothing. ### Changed - License classification reads ALL `license:id` tags of a learning resource (falling back to the plain `license` tag) and follows the shared amb-mcp/amb-indexer contract: DINI AG KIM license-type concepts (`https://w3id.org/kim/license/…`) are composed as a set into a CC variant (e.g. LEHRE LADEN `[attribution, shareAlike]` → `BY-SA (KIM)`, open; `[attribution, nonCommercial]` → `BY-NC (KIM)`, cc; `noLicense` → other), and GPL/LGPL/AGPL/GFDL/MIT/Apache-2.0/BSD URIs are now `open` (full access) instead of `unknown`. The de.wikipedia "Urheberrecht (Deutschland)" marker is labelled "Alle Rechte vorbehalten" (still metadata-only). A CC URI on the event still takes precedence. `licenseFilter: "open"` / `"cc-any"`, `accessLevel` and `get_resource`'s `openLicense` follow suit; the relay-side `license.id` prefilter lists include the KIM terms and the stored free-licence spellings. ## Live check Live run against `wss://amb-relay.edufeed.org` + `wss://oersi.edufeed.org` + `wss://sodix.edufeed.org` (search_content) and `wss://relay.edufeed.org` (find_community), via `scripts/rank-eval.ts`. Both acceptance checks pass — top-1 matches the expected result for each: - **"Statistik Vorkurs"**: 1. Vorkurs Mathematik "11 Stochastik" (oersi, score=0.718, lexical=0.750) · 2. Künstliche Intelligenz und Maschinelles Lernen mit Entscheidungsbäumen (sodix, score=0.700, lexical=0.000) - **"Unterrichtsentwurf Nachhaltigkeit"**: 1. Unterrichtsentwurf Ölverschmutzung im Wattenmeer (oersi, score=0.700, lexical=0.625) · 2. Unterrichtsentwurf Mikroplastik im Meer (oersi, score=0.087, lexical=0.625) - **"OER Lizenzen Creative Commons"**: 1. OER – wie funktioniert das und wofür brauche ich das? (sodix, score=0.750, lexical=0.625) · 2. OER erstellen: Lizenzwahl (oersi, score=0.700, lexical=0.625) - **"Unaufmerksamkeit Studierende Seminar"**: 1. Der „Filter“ (amb-relay, score=0.703, lexical=0.250) · 2. „Was tun, wenn's stockt im Seminar?" (amb-relay, score=0.075, lexical=0.417) - **"gendersensibler Unterricht"**: 1. Didaktische Materialsammlung zur Bekämpfung von Geschlechterstereotypen (sodix, score=0.700, lexical=0.250) · 2. Geschlechtergrundlagen (amb-relay, score=0.700, lexical=0.125) - **find_community "e-teaching"**: 1. e-teaching.org (top, as expected) - **find_community "Religion"**: 8 matches, top is `@Fortbildung_Religion` No change to `rank.ts` was needed — both brief-mandated first-place checks already passed. ## Deploy note `COMMUNITY_RELAYS=wss://relay.edufeed.org` must be set on the instances that should expose `find_community`; nothing else changes without it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
laoc added 10 commits 2026-11-07 21:57:03 +00:00
queryEvents and queryRelays share one collector (src/relay/provenance.ts):
regular kinds dedupe by id, addressable kinds (30000-39999) by
kind:pubkey:d with the newer created_at replacing the older in place.
The kept event's relay is stored in a WeakMap; the subscribeManyEose path
learns it from nostr-tools' receivedEvent hook. encodeNaddrAndUrl takes
the relay as hint, and resource, content and calendar items gain
`relay?: string`.

test/relay/diagnostics.test.ts: the fakeEvent fixture gets a distinct d
tag, since two d-less 30142s from one pubkey are now one coordinate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rankResults (src/content/rank.ts) adds a lexical title/description match next to the snippet score, stable-sorts by (score ?? 0) + lexical so relay order breaks ties, drops hits matching nothing and applies limit once after the merge. Field-only searches keep arrival order. The relay collector now normalizes relay URLs so queryEvents and queryRelays spell relay the same.

Adjusted tests: one asserting the old arrival order (search_content typed results), fixtures whose names never matched their query (now dropped), and provenance URLs now normalized with a trailing slash.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Controller ruling: rankResults no longer drops hits; zero-match items sort last. lexicalScore adds a 6-char prefix signal for tokens of 7+ chars (half an exact match). Field filters are stripped with the relay tokenizer port (tokenizeSearch/isFieldFilterToken, now exported), tokenize NFC-normalizes first.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
search_passages used to query searchChunks on only the first selected
relay that had an indexer configured. It now fans the chunk search out
to every selected relay with an indexer (Promise.allSettled), merges
the hits sorted by score desc (stable, so relay/rerank order breaks
ties), and runs the existing two-per-document cap and limit on the
merged set. New relaysSearched/relaysIncomplete (rejected) fields are
added next to the existing relaySearched compat field; metadataOnlyMatches
naddrs now hint the relay each hit actually came from. If every relay
fails (including the old single-relay "no indexer" case) the failure
still propagates instead of a silent empty result.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When every indexed relay's searchChunks call failed, only the first
relay's rejection reason was re-thrown — the other failures vanished
and nothing named which relays were tried. A single relay (the old
"no indexer configured" fallback) still throws its original reason
unchanged so the wrapper's instanceof SpellError handling keeps
working; several relays failing together now throw an AggregateError
naming every relay that was tried.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Communikey community definitions, free-text matched by name/description
(lexical score, same helper as search_content). Returns {pubkey, npub,
name, description, picture?, relay?} per match; pubkey can be passed as
the community parameter of search_content/search_calendar_events.

Wired behind the new COMMUNITY_RELAYS env var (comma list, unset = tool
absent); registerTools registers find_community only when a
communityClient is passed. In-memory cache per process, 5 min,
stale-while-revalidate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The brief had find_community read name/picture straight off the kind
10222 tags, but per the Communikey spec ("Communities = npubs") those
live on the community pubkey's own kind-0 profile — the same
resolution edufeed-app does with useProfileMap over community
pubkeys. No live 10222 event carries name/picture tags either.

parseCommunity(event, profileEvent?) now takes the optional kind-0
alongside the 10222: name = display_name || name (npub fallback
unchanged), picture = kind-0 picture, description = the 10222's own
override tag falling back to kind-0 about. fetchCommunities batches
one kind-0 query for all community pubkeys per cache refresh (newest
profile per pubkey wins), so a cache hit still makes zero further
queryEvents calls and a refresh makes exactly two.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
find_community review fixes:

- getCommunities no longer trusts an empty queryEvents result as good
  data: a first-ever fetch that comes back empty leaves the cache
  unset so the next call retries immediately, and a stale background
  refresh only overwrites the cache when it found something (or the
  cache was already empty) - an all-relays-down outage can no longer
  blank 5 minutes of good data. The background refresh chain also
  gets a .catch() so a rejection can't become an unhandled rejection.
- Kind 10222 is replaceable: dedupe newest-per-pubkey before parsing,
  same as the existing kind-0 profile dedup.
- Wire COMMUNITY_RELAYS/communityClient in src/stdio.ts (was only in
  http.ts/index.ts), so the README's "all transports" is actually
  true.
- total now counts matches before the limit slice, not the returned
  page size.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Runs search_content for the five brief queries over amb-relay + oersi +
sodix, and find_community for "e-teaching"/"Religion" when
COMMUNITY_RELAYS is set. Live check passed both acceptance criteria
("Vorkurs Mathematik 11 Stochastik" first for "Statistik Vorkurs",
"e-teaching.org" first for "e-teaching") — no rank.ts change needed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Non-string profile fields are ignored (a {"name":42} no longer throws in
tokenize and poisons the cache); 2-char all-caps tokens (KI, VR, IT) are
kept for matching; relaysIncomplete/warning name relays that did not finish.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Tiebreak is each item's position within its own relay, so a browse or
an all-tie list no longer returns only the fastest relay's items.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- searchChunks aborts after INDEXER_TIMEOUT_MS (default 15000): a hung
  indexer lands in relaysIncomplete instead of stalling the call
- released passages carry `relay`; relaySearched is the first relay that answered
- the same chunk_id from two relays counts once
- all relays failing with SpellErrors throws one indexer_error SpellError
  naming each relay (AggregateError only for non-Spell failures)
- description: scope materialization runs against the primary relay only

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
laoc merged commit 296e5da016 into main 2026-11-09 08:39:37 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
edufeed/nope-mcp!2
No description provided.