feat(amb-nostr): FacetDiscovery — derive filterable facets from the data #5
Loading…
Reference in a new issue
No description provided.
Delete branch "feature/facet-discovery"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Removes the need for
SearchEndpoint.php's four hardcoded RPI facets (decision 1 in the WordPress thread — redistributable scope).Nothing is wired into the plugin yet, so no rendered output changes.
The actual problem
Enumerating tag keys is easy. Deciding which are facets is not: AMB flattens nested properties, so
creator:id,hasPart:idandlearningResourceType:idare indistinguishable by shape. Taking every<x>:idfamily would put 164 creator values in a filter dropdown.The rule: a family is a facet when it carries both
:idand a:prefLabel:<lang>. A controlled vocabulary labels its concepts; an entity reference doesn't.Measured, not assumed
Over a 249-event live sample from
amb-relay.edufeed.org, the separation is exact:Zero on every entity family, hundreds on every vocabulary family. No middle ground to tune a threshold against.
What it finds on live data
23 facets — all four hardcoded ones, plus
conditionsOfAccessandteacheswhich the fixed list never had, plus everyext:facet with its German labels. Exclusions asserted programmatically, 8/8.Review notes
@TestOER — the load-bearing claim is the discriminator, so that's what's worth attacking. Two specific things:
Is
prefLabelpresence actually the right rule, or does it just happen to hold on this corpus? My sample is 249 of 8619 events and one publisher's data shape. A family that is a genuine facet but unlabelled would be silently dropped —licensealready is, and I chose to accept that rather than loosen the rule (loosening readmits creator's 164 values). If you can find a family where this misclassifies, that's the finding.hasLabelis recorded even when the prefLabel attaches to no open concept. So a family with one stray orphan label and otherwise pure entity data would be promoted to a facet. I judged that acceptable because the label still says "vocabulary", but it's the loosest part of the rule.Also worth knowing given PR #4: my parity guard there was vacuous until I mutation-tested it. I have not mutation-tested this one — the unit tests are ordinary. Treat the coverage claim accordingly.
Counts are sample counts
The return value reports
sampleSizealongside everycount, because a corpus-wide facet count is not obtainable over Nostr:facet:truein the relay's schema means filterable-by-exact-value only, there's no HTTP surface to reachfacet_by, and REQ caps atmax_limit: 250. Anything that renders these as corpus totals will be wrong.SearchEndpoint.php hardcodes four RPI facets, which the redistributable scope rules out. This derives them at runtime instead. The hard part is not enumerating tag keys, it is deciding which ones are facets. AMB flattens nested properties, so `creator:id`, `hasPart:id` and `learningResourceType:id` are the same shape — but nobody filters by creator, and taking every `<x>:id` family would put 164 creator values in a dropdown. The rule is SKOS-concept shape: a family is a facet when it carries both `:id` and a `:prefLabel:<lang>`. A controlled vocabulary labels its concepts; an entity reference does not. Measured on a 249-event live sample rather than assumed — the separation is exact: facets learningResourceType 345 prefLabels / 30 distinct values audience 471/8, about 321/46, educationalLevel 301/8, conditionsOfAccess 146/2 not facets creator 0 prefLabels / 164 distinct, hasPart 0/124, mainEntityOfPage 0/73, publisher 0/40, isPartOf 0/18, license 0/12 Zero prefLabels on every entity family, hundreds on every vocabulary one. Against that live sample it finds 23 facets: all four hardcoded ones, and also `conditionsOfAccess` and `teaches`, which the fixed list never had. Entity families are all excluded (asserted, 8/8). `ext:` keys bypass the discriminator — they are facets by grammar via ExtTagParser, and a scalar ext facet has no prefLabel by construction. They are also excluded from the core path so `ext:ekw:method:id` cannot register a second family literally named "ext:ekw:method". Counts are SAMPLE counts and the return value says so. Nostr exposes no facet-count query (facet:true means filterable-by-value only, and there is no HTTP surface to reach facet_by), and REQ is capped at max_limit 250 — so a corpus-wide count is not available to ask for. KNOWN EXCLUSION: `license` is filter-worthy but carries no prefLabel. Loosening the rule to admit it would readmit creator's 164 values, so it needs its own label source instead. Nothing is wired into the plugin yet — no rendered output changes. amb-nostr 43/43, 115 assertions (PHP 8.3.32 / PHPUnit 10.5.64) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Correcting the previous commit before it goes further. The flat LABELLED_VALUE_RATIO of 0.5 was measured only at corpus scale, and `discover()` runs per page. Swept page sizes and measured the worst labelled ratio on real facets: page=10 about 0.500 page=100 about 0.571 page=20 about 0.600 page=250 about 0.545 page=50 about 0.600 page=1999 about 0.891 `about` lands exactly ON the threshold at page 10 — a threshold on a boundary value, which is the trap I have written down and walked into anyway. Widening the sweep to families the rule was *rejecting* rather than admitting showed the real number is worse: 0.286 at page 10. A flat 0.5 silently deletes a real facet from a small archive page, which is the common case, not the edge case. The separation that actually holds is size-dependent, so the rule is now too: the proportion applies only from 20 distinct values. Measured over the same 1999 events: page 10/20/50 no family reaches the gate; all 6 core facets admitted page 250/1999 real facets subject to the ratio measure 0.889 and up stray creator 1 of 502 = 0.002, rejected Below the gate a single attached label is the signal, which is exact on live data: entity families carry zero labels at every page size sampled (10/20/50/250/1999). Above it there is enough evidence for a proportion, and 0.5 sits mid-gap between 0.002 and 0.889 rather than on either boundary. Re-verified end to end: 23 facets on 1999 events, creator still absent with an attached stray injected, and all 6 core facets admitted at every page size. amb-nostr 50/50, 122 assertions (PHP 8.3.32, PHPUnit 10.5.64) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>TestOER was right that RATIO_APPLIES_FROM had to go: `license` carries 17 distinct values corpus-wide, so it sits permanently below a 20-value gate at every page size, and one stray `license:prefLabel:de` would admit it. That is the family the docblock calls a deliberate exclusion, so the gate reopened exactly the hole it was added to close. Their other half does not reproduce here, and I checked before choosing. They swept 206 strided pages of 8482 events and found the sub-0.5 window empty; over 1999 events in contiguous AND strided pages of 5-250 I find 10 real occurrences — `about` as low as 2 of 5 labelled (0.400). Strided sampling contributed 3 of them, so it is not purely contiguity; the samples differ. Since a flat 0.5 deletes `about` on those pages, neither rule is right. What separates them is the labelled COUNT, measured rather than assumed: real facets inside the sub-0.5 window 10, ALL with >= 2 labelled values same window with exactly 1 labelled 0 single-stray attack, by construction exactly 1 labelled So: a lone label only counts when it covers at least half the family's distinct values. No size gate, no family-size special case. Verified over 1999 live events: full sample 23 facets, license absent, creator absent single-stray attack license, creator, isPartOf, publisher, hasPart ALL rejected (injected after a real <family>:id so it genuinely attaches) page sweep 5..250 all 6 core facets admitted at every size, none dropped A family with one distinct value is admitted by any rule (1/1 = 1.0); pinned by a test so it stays a known quantity rather than a surprise. Also replaces testSmallFamilyIsAFacetOnASingleLabelledValue, which encoded a shape the corpus does not produce (7 values, 1 labelled) and was my evidence for the gate. It is replaced by the shape the corpus does produce: 5 values, 2 labelled. amb-nostr 51/51, 123 assertions (PHP 8.3.32, PHPUnit 10.5.64) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Three rules on (labelled, distinct) failed in a row, and TestOER named why: the classes collide in that feature space. A real `about` facet on a small page is 1 labelled / 4 distinct; `isBasedOn` under a single stray label is 1 labelled / 4 distinct. Identical input, opposite answers required. Every threshold — boolean, ratio 0.5, size gate, minimum count — just picks a side of that collision, so each one either deleted a real facet (56 occurrences) or admitted an entity (license at any page size). AMB already declares the answer. `<family>:type` carries the schema.org class, censused over 4999 live events: Concept about, learningResourceType, educationalLevel, audience, conditionsOfAccess, teaches Person/Organization creator, publisher, mainEntityOfPage:provider WebContent mainEntityOfPage LearningResource hasPart; Course/LearningResource isPartOf MediaObject caption, encoding (undeclared) license, creator:affiliation Type alone is NOT enough, for the same reason issue #6 exists: the declaration is missing on some events and the gaps CLUSTER BY PUBLISHER — 25 publishers never emit `learningResourceType:type`. So the signals are complementary rather than alternative: 1. a declared non-Concept type REJECTS (cannot be faked by a tag) 2. an attached label ADMITS (permissive, never deletes) Step 2 can go back to permissive precisely because step 1 removed the entity families first; it now only judges the undeclared residue. That is what every previous version got wrong — one test was being asked to do both jobs. Verified at EVERY offset, page sizes 5-250, 39520 pages: zero real facets dropped, zero entities admitted. Stray-label attack rejected on every family that declares a type. Controls: disabling step 1 readmits creator/publisher under attack, so the rejection is load-bearing. HONEST LIMIT: my own corpus (4999 events, newest-first) does NOT reproduce TestOER's 56 drops under the previous rule, so this sweep under-covers theirs and the clean result is weaker evidence than the page count suggests. The collision pair is therefore pinned directly as a unit test instead of relying on the sweep. KNOWN RESIDUE, pinned not hidden: an undeclared family is admitted by one attached label — only `license` (17 distinct) and `creator:affiliation` (6). Tightening it would delete learningResourceType on the 25 publishers above. amb-nostr 54/54, 127 assertions, 0 skipped Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>