krispire/spsays-bff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

search added

Browse Source
This commit is contained in:
Krispire Admin
2026-05-10 20:41:40 +05:30
parent b58c8d00fc
commit 76a144e1ea
8 changed files with 1787 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT_CONTEXT.md
+103 -13
View File
@@ -2,7 +2,7 @@
## What this project is
A Go-based Backend for Frontend (BFF) API server for [prabhupadasays.org](https://prabhupadasays.org). It is the **sole data provider** for the production Next.js frontend (and future mobile apps). The knowledgebase frontend has removed all direct Prisma, MongoDB, and S3 dependencies — everything flows through this BFF. It aggregates content from MongoDB, user data from PostgreSQL, search from Meilisearch, caching from DragonflyDB, and media from S3 into a unified API surface.
A Go-based Backend for Frontend (BFF) API server for [prabhupadasays.org](https://prabhupadasays.org). It is the **sole data provider** for the production Next.js frontend (and future mobile apps). The knowledgebase frontend has removed all direct Prisma, MongoDB, and S3 dependencies — everything flows through this BFF. It aggregates content from MongoDB, user data from PostgreSQL, search from Typesense, caching from DragonflyDB, and media from S3 into a unified API surface.
---
@@ -13,7 +13,7 @@ A Go-based Backend for Frontend (BFF) API server for [prabhupadasays.org](https:
- **Zap** — structured logging (uber-go/zap)
- **MongoDB** — content database (books, verses, letters, essays, glossary, disciples, transcriptions)
- **PostgreSQL** — relational data (users, bookmarks, collections, mentions, URL shortener)
- **Meilisearch** — typo-tolerant full-text search across all content
- **Typesense** — typo-tolerant full-text search engine (https://typesense.org)
- **DragonflyDB** — Redis-compatible cache (API responses, rate limiting, sessions)
- **AWS S3** — media storage (avatars, images, audio files)
- **FCM + APNs** — push notifications for Android and iOS
@@ -37,7 +37,7 @@ A Go-based Backend for Frontend (BFF) API server for [prabhupadasays.org](https:
└──────────────┘ │ │ │
│ ┌─────────────────────┼──────────┐ │
│ ▼ ▼ ▼ ▼ ▼ │
│ Mongo Postgres Meili S3 Dragonfly│
│ Mongo Postgres Typesense S3 Dragonfly│
│ (content)(users, (search)(media)(cache)│
│ mentions, │
│ URL short, │
@@ -52,15 +52,20 @@ A Go-based Backend for Frontend (BFF) API server for [prabhupadasays.org](https:
```
spsays-bff/
├── cmd/
│ └── api/
│ └── main.go → Entry point (config, logger, server, graceful shutdown)
│ ├── api/
│ │ └── main.go → Entry point (config, logger, server, graceful shutdown)
│ └── indexer/
│ └── main.go → CLI tool for indexing MongoDB content into Typesense
│ • --full drop all collections and reindex
│ • --only=verses,letters index specific collections only
├── internal/
│ ├── config/
│ │ └── config.go → Environment config (all services: Mongo, Postgres, Meili, S3, etc.)
│ │ └── config.go → Environment config (all services: Mongo, Postgres, Typesense, S3, etc.)
│ ├── server/
│ │ └── server.go → Chi router setup, global middleware, route registration
│ ├── handler/ → HTTP handlers (one file per domain)
│ │ ├── health.go → Health check endpoint
│ │ ├── search.go → Federated search handler (multi-collection, typo-tolerant)
│ │ ├── book.go → Book routes (SB, CC, KB, BG, generic verse & prose)
│ │ ├── letter.go → Letter routes (by year, recipient, location)
│ │ ├── essay.go → Essay routes
@@ -71,7 +76,7 @@ spsays-bff/
│ │ ├── other_document.go → Other-document routes
│ │ └── transcription.go → Transcription routes (by year, place, type, slug)
│ ├── service/ → Business logic layer (orchestrates repositories)
│ ├── repository/ → Data access layer (Mongo, Postgres, Meili, S3, Dragonfly)
│ ├── repository/ → Data access layer (Mongo, Postgres, Typesense, S3, Dragonfly)
│ │ ├── mongo.go → MongoDB connection + database handle
│ │ ├── book_repo.go → Book / verse queries
│ │ ├── letter_repo.go → Letter queries
@@ -81,7 +86,8 @@ spsays-bff/
│ │ ├── journal_repo.go → Journal queries
│ │ ├── magazine_repo.go → Magazine queries
│ │ ├── other_document_repo.go → Other-document queries
│ │ └── transcription_repo.go → Transcription queries (with contentJson fallback)
│ │ ├── transcription_repo.go → Transcription queries (with contentJson fallback)
│ │ └── typesense.go → Typesense client wrapper (search, indexing helpers)
│ ├── middleware/
│ │ ├── logger.go → Zap-based request logging (method, path, status, duration)
│ │ ├── auth.go → JWT Bearer token authentication + context user extraction
@@ -227,6 +233,46 @@ All routes are prefixed with `/api/v1`.
| `GET` | `/api/v1/transcriptions/adjacent/{slug}` | Prev/next within same year (`?year=N`) |
| `GET` | `/api/v1/transcriptions/{slug}` | Single transcription (full content + audio URL) |
### Search
| Method | Route | Description |
|--------|-------|-------------|
| `GET` | `/api/v1/search?q=...&type=...&page=1&limit=20` | Federated typo-tolerant search across all content |
**Query params:**
- `q` — search query (required)
- `type` — content type filter, comma-separated. Valid values:
- `all` (default) — search all 9 collections
- `verses`, `prose`, `letters`, `transcriptions`, `glossary`, `essays`, `journals`, `magazines`, `other_documents`
- Multiple: `verses,letters,transcriptions`
- `page` — page number (default 1)
- `limit` — results per page (default 20, max 100)
**Response:**
```json
{
"success": true,
"data": {
"query": "surrender unto",
"type": "all",
"results": [
{
"collection": "verses",
"id": "...",
"title": "BG 18.66",
"snippet": "...just <mark>surrender unto</mark> Me...",
"url": "/books/bg/18/66",
"meta": { "book": "bg", "reference": "BG 18.66" }
}
],
"total": 150,
"page": 1,
"limit": 20,
"total_pages": 8
}
}
```
### Authenticated (JWT required) — planned
| Method | Route | Description |
@@ -247,6 +293,49 @@ All routes are prefixed with `/api/v1`.
---
## Typesense Indexing
The BFF includes a CLI tool (`cmd/indexer/main.go`) for managing Typesense indexes.
### Collections (9 total, ~35,500 documents)
| Collection | MongoDB Source | Documents | Searchable Fields |
|---|---|---|---|
| `verses` | `verses` | 21,127 | translation, purport, transliteration, reference (+ ASCII variants) |
| `prose` | `prose_chapters` | 627 | title, content (+ ASCII variants) |
| `letters` | `letters` | 6,866 | content, recipient_display (+ ASCII variants) |
| `transcriptions` | `transcriptions` | 3,787 | content, name, reference (+ ASCII variants) |
| `glossary` | `glossary` | 2,749 | term, definition (+ ASCII variants) |
| `essays` | `essays` | 23 | title, content (+ ASCII variants) |
| `journals` | `journal_chapters` | 300 | title, content (+ ASCII variants) |
| `magazines` | `magazine_issues` | 59 | title, content (+ ASCII variants) |
| `other_documents` | `other_document_chapters` | 28 | title, content (+ ASCII variants) |
### ASCII Shadow Fields
For diacritic-insensitive search, every searchable text field has a `_ascii` shadow field with all IAST diacritics stripped (`ā→a`, `ṛ→r`, `ṣ→s`, `ṭ→t`, `ñ→n`, etc.). This allows queries like "Krsna" to match "Kṛṣṇa", "Yashodanandana" to match "Yaśodanandana".
### Search Configuration
- **`prefix=true`** — partial word matching (e.g., "sarv" matches "sarva")
- **`num_typos=2`** — up to 2 character substitutions allowed per word
- **HTML stripping** — all paragraph content has HTML tags removed before indexing
### CLI Usage
```/dev/null/shell.sh#L1-12
# Full reindex (drops all collections and rebuilds)
go run cmd/indexer/main.go --full
# Index specific collections only (drops + recreates them)
go run cmd/indexer/main.go --only=verses,letters
# Index all, but skip collections that already exist
go run cmd/indexer/main.go
```
---
## Data Models
### Content Models (MongoDB)
@@ -333,9 +422,9 @@ MONGO_DB=spsays
# PostgreSQL (users, relations)
POSTGRES_URI=postgres://...
# Meilisearch
MEILISEARCH_HOST=http://localhost:7700
MEILISEARCH_API_KEY=
# Typesense
TYPESENSE_URL=https://search.prabhupadasays.org
TYPESENSE_API_KEY=
# DragonflyDB / Redis
REDIS_URI=redis://localhost:6379
@@ -402,7 +491,7 @@ make tidy
Migration status (Prisma → BFF):
1. ✅ Done — BFF API routes matching all former Prisma queries
2. ✅ Done — Frontend migrated from Prisma to BFF API calls (Prisma, MongoDB driver, and S3 SDK removed from knowledgebase)
3. Add Meilisearch indexing + search endpoints
3. ✅ Done — Typesense indexing + federated search endpoints (`cmd/indexer/main.go`, `/api/v1/search`)
4. Add PostgreSQL for user features (auth, bookmarks, collections, mentions)
5. Add mobile app support (same API, push notifications)
@@ -417,4 +506,5 @@ Migration status (Prisma → BFF):
- **Service layer** — business logic between handlers and repositories (orchestration, validation, caching)
- **Handler layer** — thin HTTP handlers that parse requests, call services, return responses
- **No ORM** — use native drivers (`mongo-go-driver`, `pgx`) for full control
- **Explicit book routing** — SB, CC, and KB have dedicated route groups to avoid URL ambiguity with generic `{code}/{chapter}/{verse}` patterns
- **Explicit book routing** — SB, CC, and KB have dedicated route groups to avoid URL ambiguity with generic `{code}/{chapter}/{verse}` patterns
- **Search gateway** — search is provided by Typesense, indexed from MongoDB via the indexer CLI; the BFF acts as the search gateway with multi-collection federated queries.