From e3f14ef8baf288b7f4e85eb976b361f62e6dad80 Mon Sep 17 00:00:00 2001 From: Krispire Admin Date: Sun, 10 May 2026 20:47:29 +0530 Subject: [PATCH] chore: update project context --- PROJECT_CONTEXT.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index 96b57a0..5de2ba7 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -315,11 +315,35 @@ The BFF includes a CLI tool (`cmd/indexer/main.go`) for managing Typesense index For diacritic-insensitive search, every searchable text field has a `_ascii` shadow field with all IAST diacritics stripped (`ā→a`, `ṛ→r`, `ṣ→s`, `ṭ→t`, `ñ→n`, etc.). This allows queries like "Krsna" to match "Kṛṣṇa", "Yashodanandana" to match "Yaśodanandana". +### Token Separators + +The `verses` collection is configured with `token_separators: ["-"]` so hyphenated Sanskrit transliterations are split into individual tokens at index time. For example, `sarva-dharman` is indexed as the two tokens `sarva` and `dharman`. This means a query like `"sarva dharman"` matches verses where the term appears as `sarva-dharman` in the original transliteration, without requiring the user to type the hyphen. + ### Search Configuration -- **`prefix=true`** — partial word matching (e.g., "sarv" matches "sarva") +- **`prefix=true`** — partial word matching (e.g., "sarv" matches "sarva", "parit" matches "parityajya") - **`num_typos=2`** — up to 2 character substitutions allowed per word +- **`exhaustive_search=true`** — finds all matches across multi-field queries; without this, multi-field searches with mixed-length fields (like a verse's short `transliteration` vs. its long `purport`) would miss valid matches because Typesense's default early-termination heuristics drop short-field hits when longer fields dominate the candidate set +- **`query_by_weights`** — per-collection weights to boost identifier/primary fields over secondary content (see below) - **HTML stripping** — all paragraph content has HTML tags removed before indexing +- **`_ascii` shadow fields** — every searchable text field is duplicated with diacritics stripped, and both the original and `_ascii` field are included in `query_by` + +### Per-Collection Field Strategy + +Each collection defines its own `query_by` field list and matching `query_by_weights`, ranking identifier/primary fields highest and secondary content lowest: + +| Collection | `query_by` | Weights | +|---|---|---| +| `verses` | `reference, transliteration, transliteration_ascii, translation, translation_ascii` | `10,8,8,4,4` | +| `prose` | `title, title_ascii, content, content_ascii` | `10,10,4,4` | +| `letters` | `recipient_display, recipient_display_ascii, content, content_ascii` | `8,8,4,4` | +| `transcriptions` | `reference, name, name_ascii, content, content_ascii` | `10,8,8,4,4` | +| `glossary` | `term, term_ascii, definition, definition_ascii` | `10,10,4,4` | +| `essays` / `journals` / `magazines` / `other_documents` | `title, title_ascii, content, content_ascii` | `10,10,4,4` | + +**Verses — purport is intentionally excluded from default search.** The verse `query_by` list does **not** include `purport` or `purport_ascii`. If purport were searchable by default, verses where a phrase appears only in someone's commentary (e.g. a purport that quotes another verse) would frequently outrank the actual verse containing that phrase. For example, searching `"sarva dharman parityajya"` should rank **BG 18.66** (the verse itself) above any commentary that cites it. Excluding `purport` from `query_by` ensures the canonical verse always wins. + +The same identifier-first pattern applies across collections: references, names, terms, and titles are weighted highest; primary text (translation, definition, recipient) sits in the middle; long-form body content (purport, content) is either weighted lowest or excluded entirely. ### CLI Usage @@ -497,6 +521,12 @@ Migration status (Prisma → BFF): --- +## Known Issues + +- **Verse purport content is not searchable by default.** Because `purport` and `purport_ascii` are excluded from the verses `query_by` list (see *Per-Collection Field Strategy* above), phrases that appear only inside a purport — and not in the verse's `reference`, `transliteration`, or `translation` — will not match a normal verse search. Surfacing those matches would require a separate "deep search" toggle that adds `purport` and `purport_ascii` (with lower weights) to `query_by`. This is **not yet implemented**. + +--- + ## Design Principles - **Single source of truth** — the BFF is the sole data provider for all content consumed by the frontend and future mobile apps. No client should bypass the BFF to access MongoDB, S3, or any other backing service directly.