NIP-50 field filters are silently dropped when the search string also contains free text (fail-open in chunk-rerank path) #22

Closed
opened 2026-08-19 11:56:15 +00:00 by laoc · 0 comments
Owner

Symptom (verified live on wss://amb-relay.edufeed.org, 2026-08-19)

A search combining free text with a field filter ignores the filter entirely:

  • {"kinds":[30142],"search":"LADEN publisher.name:zzz-bogus-control-xyz","limit":2} → returns free-text hits for LADEN (bogus control should force 0 results)
  • {"kinds":[30142],"search":"publisher.name:zzz-bogus-control-xyz"} (filter only) → correctly returns 0
  • {"kinds":[30142],"search":"publisher.name:e-teaching.org"} (filter only) → correctly filtered results

So pure field-filter searches take the plain Typesense path and fail closed, but as soon as a free-text term is present the filter is dropped and the query gets wider — exactly the fail-open class the typesense30142 README warns about for tag filters ("Unsupported filters fail closed").

Cause

main.go, relay.QueryStored: searches matching reg.targetsChunked(filter) && searchHasFreeText(filter.Search) are routed to calendarRerankQuery → chunk searcher (amb-indexer). The comment there already documents the hazard for pure field-filter searches ("rerank would send the raw string to the chunk index and drop the field filter — those must take the plain field-filter path too") — but a combined search (free text + field filter) still goes down the rerank path, where the field.path:value tokens are not honoured as filters.

Impact

Any client that follows the documented "Combined Queries" section of the typesense30142 README (e.g. "forschung publisher.name:e-teaching.org") silently gets unfiltered results. For MCP agents this presents as confidently wrong answers — indistinguishable from an honest match. This broke the "topic + publisher" agent flow that triggered this investigation (amb-mcp side: gitworkshop.dev issue "Field-filter values with spaces are not quoted").

Fix options

  1. Route on filter presence (minimal, fail-closed): if ParseSearchQuery(filter.Search).FieldFilters is non-empty, take the plain reg.fetch path even when free text is present — correctness over passage re-ranking.
  2. Carry filters through the rerank path: strip field filters from the string sent to the chunk index, run the plain filtered query in parallel, and intersect / rerank within the filtered id-set.

Option 1 is a small, safe change; option 2 preserves snippet ranking for combined queries and could follow later.

## Symptom (verified live on wss://amb-relay.edufeed.org, 2026-08-19) A search combining free text with a field filter ignores the filter entirely: - `{"kinds":[30142],"search":"LADEN publisher.name:zzz-bogus-control-xyz","limit":2}` → returns free-text hits for `LADEN` (bogus control should force **0** results) - `{"kinds":[30142],"search":"publisher.name:zzz-bogus-control-xyz"}` (filter only) → correctly returns 0 - `{"kinds":[30142],"search":"publisher.name:e-teaching.org"}` (filter only) → correctly filtered results So pure field-filter searches take the plain Typesense path and fail closed, but as soon as a free-text term is present the filter is dropped and the query gets *wider* — exactly the fail-open class the typesense30142 README warns about for tag filters ("Unsupported filters fail closed"). ## Cause `main.go`, `relay.QueryStored`: searches matching `reg.targetsChunked(filter) && searchHasFreeText(filter.Search)` are routed to `calendarRerankQuery` → chunk searcher (amb-indexer). The comment there already documents the hazard for *pure* field-filter searches ("rerank would send the raw string to the chunk index and drop the field filter — those must take the plain field-filter path too") — but a **combined** search (free text + field filter) still goes down the rerank path, where the `field.path:value` tokens are not honoured as filters. ## Impact Any client that follows the documented "Combined Queries" section of the typesense30142 README (e.g. `"forschung publisher.name:e-teaching.org"`) silently gets unfiltered results. For MCP agents this presents as confidently wrong answers — indistinguishable from an honest match. This broke the "topic + publisher" agent flow that triggered this investigation (amb-mcp side: gitworkshop.dev issue "Field-filter values with spaces are not quoted"). ## Fix options 1. **Route on filter presence** (minimal, fail-closed): if `ParseSearchQuery(filter.Search).FieldFilters` is non-empty, take the plain `reg.fetch` path even when free text is present — correctness over passage re-ranking. 2. **Carry filters through the rerank path**: strip field filters from the string sent to the chunk index, run the plain filtered query in parallel, and intersect / rerank within the filtered id-set. Option 1 is a small, safe change; option 2 preserves snippet ranking for combined queries and could follow later.
laoc closed this issue 2026-08-19 12:18:49 +00:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
edufeed/amb-relay#22
No description provided.