chore: update project context

PROJECT_CONTEXT.md

@@ -317,13 +317,36 @@ For diacritic-insensitive search, every searchable text field has a `_ascii` sha
 
 ### Token Separators
 
-The `verses` collection is configured with `token_separators: ["-"]` so hyphenated Sanskrit transliterations are split into individual tokens at index time. For example, `sarva-dharman` is indexed as the two tokens `sarva` and `dharman`. This means a query like `"sarva dharman"` matches verses where the term appears as `sarva-dharman` in the original transliteration, without requiring the user to type the hyphen.
+**Every** collection is configured with `token_separators: ["-"]` so hyphenated Sanskrit/Vedic terms are split into individual tokens at index time. For example, `sarva-dharman` is indexed as the two tokens `sarva` and `dharman`; `Dīkṣā-guru` indexes as `dīkṣā` + `guru`. This means queries like `"sarva dharman"` or `"diksa guru"` match cleanly even though the source text uses a hyphen.
+
+A shared `hyphenSeparators` variable is defined once in `cmd/indexer/main.go` and applied to every schema. Earlier only `verses` had this configured — the fix was to extend it to glossary, prose, letters, transcriptions, essays, journals, magazines, and other_documents (a full `--full` reindex is required after enabling it on a collection because tokenisation happens at index time).
+
+### Stopwords
+
+A server-side Typesense stopwords set named `common` strips conversational tokens before matching. The set is provisioned out-of-band via `PUT /stopwords/common` (one-time server config; not part of the indexer):
+
+```
+letter, letters, to, of, the, a, an, is, are, for, by, from,
+in, on, with, and, or,
+essay, essays, transcription, transcriptions,
+magazine, magazines, journal, journals,
+verse, verses, lecture, lectures, conversation, conversations,
+document, documents
+```
+
+The search handler passes `stopwords=common` on every Typesense call (both `multiSearch` and `singleSearch`). Without this, a query like "letter to tusta krsna" would let "letter" and "to" match in the body content of many letters, drowning out the specific recipient match. With stopwords, the query effectively becomes "tusta krsna" and the actual letters to Tusta Krsna rise to the top.
 
 ### Search Configuration
 
+All Typesense search calls (both `multiSearch` and `singleSearch` in `internal/handler/search.go`) use this set of parameters:
+
 - **`prefix=true`** — partial word matching (e.g., "sarv" matches "sarva", "parit" matches "parityajya")
 - **`num_typos=2`** — up to 2 character substitutions allowed per word
 - **`exhaustive_search=true`** — finds all matches across multi-field queries; without this, multi-field searches with mixed-length fields (like a verse's short `transliteration` vs. its long `purport`) would miss valid matches because Typesense's default early-termination heuristics drop short-field hits when longer fields dominate the candidate set
+- **`prioritize_exact_match=true`** — prioritises documents whose field value matches the query exactly over partial / typo matches
+- **`prioritize_token_position=true`** — matches earlier in a field score higher; this is what differentiates documents whose scores would otherwise tie (e.g. "karma" magazine results where every doc had `tokens_matched=1`)
+- **`stopwords=common`** — strips conversational tokens (see *Stopwords* above)
+- **`text_match_type`** — left at Typesense's default `max_score`. Earlier experiments with `sum_score` (for boosting letters matching both recipient + content) caused regressions on the "All" tab because `sum_score` favours documents with matches across more fields, so transcriptions with the verse reference in their `name` field would outrank the verses themselves on a query like "bg 18.66". Stopwords alone solved the letter-ranking problem without this trade-off.
 - **`query_by_weights`** — per-collection weights to boost identifier/primary fields over secondary content (see below)
 - **HTML stripping** — all paragraph content has HTML tags removed before indexing
 - **`_ascii` shadow fields** — every searchable text field is duplicated with diacritics stripped, and both the original and `_ascii` field are included in `query_by`
@@ -358,6 +381,12 @@ go run cmd/indexer/main.go --only=verses,letters
 go run cmd/indexer/main.go
 ```
 
+### Indexer Robustness
+
+- **Typesense connection timeout: 5 minutes.** The indexer uses `typesense.WithConnectionTimeout(5*time.Minute)` (vs the library default of 5s) because the magazines and other_documents collections ship multi-MB batches that exceeded the default. Without this, those collections silently ended up empty in Typesense — the indexer logged `context deadline exceeded` per batch but moved on.
+- **Reduced batch size for heavy collections.** `magazines` and `other_documents` use `importBatch(..., 25)` instead of the default 500 because each document contains very large paragraph arrays (entire magazine issues / chapters concatenated). All other collections still use batch size 500.
+- **Importing post-fix.** If you have empty magazine / other_documents collections from before this fix, simply run `--full` (or `--only=magazines,other_documents`) to rebuild.
+
 ---
 
 ## Data Models
@@ -525,6 +554,10 @@ Migration status (Prisma → BFF):
 
 - **Verse purport content is not searchable by default.** Because `purport` and `purport_ascii` are excluded from the verses `query_by` list (see *Per-Collection Field Strategy* above), phrases that appear only inside a purport — and not in the verse's `reference`, `transliteration`, or `translation` — will not match a normal verse search. Surfacing those matches would require a separate "deep search" toggle that adds `purport` and `purport_ascii` (with lower weights) to `query_by`. This is **not yet implemented**.
 
+- **Stopwords set is an external Typesense resource.** The `common` stopwords set is configured server-side via `PUT /stopwords/common` and is **not** managed by the indexer or the BFF code. If Typesense is rebuilt from scratch or restored from a snapshot that predates the stopwords config, the set must be reprovisioned manually. The BFF passes `stopwords=common` on every search; if the set doesn't exist, Typesense simply ignores the parameter (rather than erroring), so symptoms would be “searches like 'letter to tusta krsna' suddenly behave worse,” not hard failures.
+
+- **Indexer is destructive on `--full` or `--only=`.** Both flags drop the named collection(s) before recreating them. There is currently no incremental indexer — if a single document changes in MongoDB, the whole collection has to be reindexed. Acceptable today (largest collection is verses at ~21k docs, takes seconds), but worth keeping in mind.
+
 ---
 
 ## Design Principles