krispire/spsays-bff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
26f0e298b518a3493ab759608b75507775269382
spsays-bff/PROJECT_CONTEXT.md
T

29 KiB
Raw Blame History

Prabhupāda Says — BFF (Backend for Frontend)

What this project is

A Go-based Backend for Frontend (BFF) API server for prabhupadasays.org. It is the sole data provider for the production Next.js frontend (and future mobile apps). The knowledgebase frontend has removed all direct Prisma, MongoDB, and S3 dependencies — everything flows through this BFF. It aggregates content from MongoDB, user data from PostgreSQL, search from Typesense, caching from DragonflyDB, and media from S3 into a unified API surface.

Tech Stack

  • Go 1.23 — API server
  • Chi v5 — lightweight HTTP router with middleware support
  • Zap — structured logging (uber-go/zap)
  • MongoDB — content database (books, verses, letters, essays, glossary, disciples, transcriptions)
  • PostgreSQL — relational data (users, bookmarks, collections, mentions, URL shortener)
  • Typesense — typo-tolerant full-text search engine (https://typesense.org)
  • DragonflyDB — Redis-compatible cache (API responses, rate limiting, sessions)
  • AWS S3 — media storage (avatars, images, audio files)
  • FCM + APNs — push notifications for Android and iOS
  • JWT — stateless authentication

Architecture

Note: The BFF is now the ONLY data path for the frontend. The knowledgebase has no Prisma bypass, no direct MongoDB connections, and no direct S3 access. All content is served exclusively through this API.

┌──────────────┐     ┌─────────────────────────────────────────┐
│  Next.js     │────▶│  Go BFF API (:8080)                      │
│  (frontend)  │     │                                          │
│  (sole data  │     │  cmd/api/main.go → server.go → handlers │
│   path ↑)    │     │                                          │
├──────────────┤     │  ┌──────────┐  ┌───────────────┐        │
│  Mobile Apps │────▶│  │ Handlers │──│ Services      │        │
│  (future)    │     │  └──────────┘  └───────┬───────┘        │
└──────────────┘     │                        │                │
                     │  ┌─────────────────────┼──────────┐     │
                     │  ▼         ▼           ▼     ▼    ▼     │
                     │ Mongo  Postgres  Typesense  S3  Dragonfly│
                     │ (content)(users,   (search)(media)(cache)│
                     │          mentions,                       │
                     │          URL short,                      │
                     │          relations)                      │
                     └─────────────────────────────────────────┘

Project Structure

spsays-bff/
├── cmd/
│   ├── api/
│   │   └── main.go              → Entry point (config, logger, server, graceful shutdown)
│   └── indexer/
│       └── main.go              → CLI tool for indexing MongoDB content into Typesense
│                                   • --full           drop all collections and reindex
│                                   • --only=verses,letters  index specific collections only
├── internal/
│   ├── config/
│   │   └── config.go            → Environment config (all services: Mongo, Postgres, Typesense, S3, etc.)
│   ├── server/
│   │   └── server.go            → Chi router setup, global middleware, route registration
│   ├── handler/                  → HTTP handlers (one file per domain)
│   │   ├── health.go            → Health check endpoint
│   │   ├── search.go            → Federated search handler (multi-collection, typo-tolerant)
│   │   ├── book.go              → Book routes (SB, CC, KB, BG, generic verse & prose)
│   │   ├── letter.go            → Letter routes (by year, recipient, location)
│   │   ├── essay.go             → Essay routes
│   │   ├── glossary.go          → Glossary routes (letters, terms)
│   │   ├── disciple.go          → Disciple routes (filter, search, aggregations)
│   │   ├── journal.go           → Journal routes
│   │   ├── magazine.go          → Magazine routes
│   │   ├── other_document.go    → Other-document routes
│   │   └── transcription.go     → Transcription routes (by year, place, type, slug)
│   ├── service/                  → Business logic layer (orchestrates repositories)
│   ├── repository/               → Data access layer (Mongo, Postgres, Typesense, S3, Dragonfly)
│   │   ├── mongo.go             → MongoDB connection + database handle
│   │   ├── book_repo.go         → Book / verse queries
│   │   ├── letter_repo.go       → Letter queries
│   │   ├── essay_repo.go        → Essay queries
│   │   ├── glossary_repo.go     → Glossary queries
│   │   ├── disciple_repo.go     → Disciple queries
│   │   ├── journal_repo.go      → Journal queries
│   │   ├── magazine_repo.go     → Magazine queries
│   │   ├── other_document_repo.go → Other-document queries
│   │   ├── transcription_repo.go → Transcription queries (with contentJson fallback)
│   │   └── typesense.go         → Typesense client wrapper (search, indexing helpers)
│   ├── middleware/
│   │   ├── logger.go            → Zap-based request logging (method, path, status, duration)
│   │   ├── auth.go              → JWT Bearer token authentication + context user extraction
│   │   └── ratelimit.go         → In-memory token bucket rate limiter (→ DragonflyDB in production)
│   └── model/
│       ├── content.go           → Content models (Book, Verse, Letter, Essay, GlossaryEntry,
│       │                           Disciple, Transcription, ContentBlock, Synonym)
│       └── user.go              → User models (User, Bookmark, Collection, Mention, ShortURL)
├── pkg/                          → Public packages (importable by other projects)
│   ├── response/
│   │   └── response.go          → Standardized API response helpers (JSON, Error, Envelope, Meta)
│   ├── validator/
│   │   └── validator.go         → Field validation (Required, MaxLength, Email, In, etc.)
│   ├── push/                     → Push notification sender (FCM + APNs) — placeholder
│   └── shorturl/                 → URL shortener logic — placeholder
├── migrations/                   → PostgreSQL migrations (goose or golang-migrate)
├── go.mod                        → Module: github.com/prabhupadasays/spsays-bff
├── go.sum
├── Makefile                      → build, run, dev, test, clean, lint, tidy
├── .env.example                  → Template for all environment variables
└── .gitignore

API Routes

All routes are prefixed with /api/v1.

Health

Method Route Description
GET /health Server health check

Books

Method Route Description
GET /api/v1/books List all books with full metadata (chapters, cantos, parts, volumes, etc.)
GET /api/v1/books/sb SB book metadata
GET /api/v1/books/sb/{canto} SB canto chapter listing
GET /api/v1/books/sb/{canto}/{chapter} SB chapter verse index
GET /api/v1/books/sb/{canto}/{chapter}/verses SB paginated verses
GET /api/v1/books/sb/{canto}/{chapter}/{verse} SB single verse
GET /api/v1/books/cc CC book metadata
GET /api/v1/books/cc/{lila} CC līlā chapter listing
GET /api/v1/books/cc/{lila}/{chapter} CC chapter verse index
GET /api/v1/books/cc/{lila}/{chapter}/verses CC paginated verses
GET /api/v1/books/cc/{lila}/{chapter}/{verse} CC single verse
GET /api/v1/books/kb KB book metadata
GET /api/v1/books/kb/{volume} KB volume chapter listing
GET /api/v1/books/kb/{volume}/{slug} KB single chapter
GET /api/v1/books/bg BG book metadata
GET /api/v1/books/bg/{chapter} BG chapter verse index
GET /api/v1/books/bg/{chapter}/verses BG paginated verses
GET /api/v1/books/bg/{chapter}/{verse} BG single verse
GET /api/v1/books/{code} Any other book metadata
GET /api/v1/books/{code}/chapters Prose book chapter listing (no paragraphs)
GET /api/v1/books/{code}/chapters/{slug} Prose book single chapter
GET /api/v1/books/{code}/{chapter} Verse book chapter index
GET /api/v1/books/{code}/{chapter}/verses Verse book paginated verses
GET /api/v1/books/{code}/{chapter}/{verse} Verse book single verse

Letters

Method Route Description
GET /api/v1/letters Total letter count
GET /api/v1/letters/years Year → count aggregation
GET /api/v1/letters/recipients Recipient → count aggregation
GET /api/v1/letters/locations Location → count aggregation
GET /api/v1/letters/year/{year} Letters for a year
GET /api/v1/letters/recipient/{name} Letters for a recipient
GET /api/v1/letters/location/{name} Letters from a location
GET /api/v1/letters/{slug} Single letter

Essays

Method Route Description
GET /api/v1/essays All essay summaries
GET /api/v1/essays/{slug} Single essay

Glossary

Method Route Description
GET /api/v1/glossary All glossary entries
GET /api/v1/glossary/letters Letter → count mapping
GET /api/v1/glossary/letter/{letter} Entries for a letter (with diacritical grouping)
GET /api/v1/glossary/letter/{letter}/terms Term names only for sidebar
GET /api/v1/glossary/term/{term} Single entry by term

Disciples

Method Route Description
GET /api/v1/disciples Filtered list (?letter=&suffix=)
GET /api/v1/disciples/count Total + filtered counts
GET /api/v1/disciples/letters Letter → count aggregation
GET /api/v1/disciples/suffixes Suffix → count aggregation
GET /api/v1/disciples/search Search by name (?q=)

Journals

Method Route Description
GET /api/v1/journals All journal metadata
GET /api/v1/journals/{book} Single journal meta
GET /api/v1/journals/{book}/chapters Journal chapters
GET /api/v1/journals/{book}/chapters/{slug} Single chapter

Magazines

Method Route Description
GET /api/v1/magazines All magazine metadata
GET /api/v1/magazines/{book} Single magazine meta
GET /api/v1/magazines/{book}/issues Magazine issues
GET /api/v1/magazines/{book}/issues/{slug} Single issue

Other Documents

Method Route Description
GET /api/v1/other-documents All document metadata
GET /api/v1/other-documents/{book} Single document meta
GET /api/v1/other-documents/{book}/chapters Document chapters
GET /api/v1/other-documents/{book}/chapters/{slug} Single chapter

Transcriptions

Method Route Description
GET /api/v1/transcriptions Total transcription count
GET /api/v1/transcriptions/years Year → count aggregation
GET /api/v1/transcriptions/places Place → count aggregation
GET /api/v1/transcriptions/types Type distribution ({lectures, conversations})
GET /api/v1/transcriptions/year/{year} Transcriptions for a year (summaries)
GET /api/v1/transcriptions/place/{place} Transcriptions for a place (summaries)
GET /api/v1/transcriptions/type/{type} Transcriptions by type (lecture or conversation)
GET /api/v1/transcriptions/adjacent/{slug} Prev/next within same year (?year=N)
GET /api/v1/transcriptions/{slug} Single transcription (full content + audio URL)

Search

Method Route Description
GET /api/v1/search?q=...&type=...&page=1&limit=20 Federated typo-tolerant search across all content

Query params:

  • q — search query (required)
  • type — content type filter, comma-separated. Valid values:
    • all (default) — search all 9 collections
    • verses, prose, letters, transcriptions, glossary, essays, journals, magazines, other_documents
    • Multiple: verses,letters,transcriptions
  • page — page number (default 1)
  • limit — results per page (default 20, max 100)

Response:

{
  "success": true,
  "data": {
    "query": "surrender unto",
    "type": "all",
    "results": [
      {
        "collection": "verses",
        "id": "...",
        "title": "BG 18.66",
        "snippet": "...just <mark>surrender unto</mark> Me...",
        "url": "/books/bg/18/66",
        "meta": { "book": "bg", "reference": "BG 18.66" }
      }
    ],
    "total": 150,
    "page": 1,
    "limit": 20,
    "total_pages": 8
  }
}

Authenticated (JWT required) — planned

Method Route Description
POST /api/v1/auth/register User registration
POST /api/v1/auth/login User login (returns JWT)
POST /api/v1/auth/refresh Refresh JWT token
GET /api/v1/users/me Get current user profile
PUT /api/v1/users/me Update current user profile
GET /api/v1/collections List user's collections
POST /api/v1/collections Create a collection
POST /api/v1/collections/:id/items Add item to collection
DELETE /api/v1/collections/:id/items/:itemId Remove item from collection
GET /api/v1/mentions List user's mentions
POST /api/v1/mentions Create a mention
POST /api/v1/s Create a short URL
POST /api/v1/notifications/register Register device for push notifications

Typesense Indexing

The BFF includes a CLI tool (cmd/indexer/main.go) for managing Typesense indexes.

Collections (9 total, ~35,500 documents)

Collection MongoDB Source Documents Searchable Fields
verses verses 21,127 translation, purport, transliteration, reference (+ ASCII variants)
prose prose_chapters 627 title, content (+ ASCII variants)
letters letters 6,866 content, recipient_display (+ ASCII variants)
transcriptions transcriptions 3,787 content, name, reference (+ ASCII variants)
glossary glossary 2,749 term, definition (+ ASCII variants)
essays essays 23 title, content (+ ASCII variants)
journals journal_chapters 300 title, content (+ ASCII variants)
magazines magazine_issues 59 title, content (+ ASCII variants)
other_documents other_document_chapters 28 title, content (+ ASCII variants)

ASCII Shadow Fields

For diacritic-insensitive search, every searchable text field has a _ascii shadow field with all IAST diacritics stripped (ā→a, ṛ→r, ṣ→s, ṭ→t, ñ→n, etc.). This allows queries like "Krsna" to match "Kṛṣṇa", "Yashodanandana" to match "Yaśodanandana".

Token Separators

Every collection is configured with token_separators: ["-"] so hyphenated Sanskrit/Vedic terms are split into individual tokens at index time. For example, sarva-dharman is indexed as the two tokens sarva and dharman; Dīkṣā-guru indexes as dīkṣā + guru. This means queries like "sarva dharman" or "diksa guru" match cleanly even though the source text uses a hyphen.

A shared hyphenSeparators variable is defined once in cmd/indexer/main.go and applied to every schema. Earlier only verses had this configured — the fix was to extend it to glossary, prose, letters, transcriptions, essays, journals, magazines, and other_documents (a full --full reindex is required after enabling it on a collection because tokenisation happens at index time).

Stopwords

A server-side Typesense stopwords set named common strips conversational tokens before matching. The set is provisioned out-of-band via PUT /stopwords/common (one-time server config; not part of the indexer):

letter, letters, to, of, the, a, an, is, are, for, by, from,
in, on, with, and, or,
essay, essays, transcription, transcriptions,
magazine, magazines, journal, journals,
verse, verses, lecture, lectures, conversation, conversations,
document, documents

The search handler passes stopwords=common on every Typesense call (both multiSearch and singleSearch). Without this, a query like "letter to tusta krsna" would let "letter" and "to" match in the body content of many letters, drowning out the specific recipient match. With stopwords, the query effectively becomes "tusta krsna" and the actual letters to Tusta Krsna rise to the top.

Search Configuration

All Typesense search calls (both multiSearch and singleSearch in internal/handler/search.go) use this set of parameters:

  • prefix=true — partial word matching (e.g., "sarv" matches "sarva", "parit" matches "parityajya")
  • num_typos=2 — up to 2 character substitutions allowed per word
  • exhaustive_search=true — finds all matches across multi-field queries; without this, multi-field searches with mixed-length fields (like a verse's short transliteration vs. its long purport) would miss valid matches because Typesense's default early-termination heuristics drop short-field hits when longer fields dominate the candidate set
  • prioritize_exact_match=true — prioritises documents whose field value matches the query exactly over partial / typo matches
  • prioritize_token_position=true — matches earlier in a field score higher; this is what differentiates documents whose scores would otherwise tie (e.g. "karma" magazine results where every doc had tokens_matched=1)
  • stopwords=common — strips conversational tokens (see Stopwords above)
  • text_match_type — left at Typesense's default max_score. Earlier experiments with sum_score (for boosting letters matching both recipient + content) caused regressions on the "All" tab because sum_score favours documents with matches across more fields, so transcriptions with the verse reference in their name field would outrank the verses themselves on a query like "bg 18.66". Stopwords alone solved the letter-ranking problem without this trade-off.
  • query_by_weights — per-collection weights to boost identifier/primary fields over secondary content (see below)
  • HTML stripping — all paragraph content has HTML tags removed before indexing
  • _ascii shadow fields — every searchable text field is duplicated with diacritics stripped, and both the original and _ascii field are included in query_by

Per-Collection Field Strategy

Each collection defines its own query_by field list and matching query_by_weights, ranking identifier/primary fields highest and secondary content lowest:

Collection query_by Weights
verses reference, transliteration, transliteration_ascii, translation, translation_ascii 10,8,8,4,4
prose title, title_ascii, content, content_ascii 10,10,4,4
letters recipient_display, recipient_display_ascii, content, content_ascii 8,8,4,4
transcriptions reference, name, name_ascii, content, content_ascii 10,8,8,4,4
glossary term, term_ascii, definition, definition_ascii 10,10,4,4
essays / journals / magazines / other_documents title, title_ascii, content, content_ascii 10,10,4,4

Verses — purport is intentionally excluded from default search. The verse query_by list does not include purport or purport_ascii. If purport were searchable by default, verses where a phrase appears only in someone's commentary (e.g. a purport that quotes another verse) would frequently outrank the actual verse containing that phrase. For example, searching "sarva dharman parityajya" should rank BG 18.66 (the verse itself) above any commentary that cites it. Excluding purport from query_by ensures the canonical verse always wins.

The same identifier-first pattern applies across collections: references, names, terms, and titles are weighted highest; primary text (translation, definition, recipient) sits in the middle; long-form body content (purport, content) is either weighted lowest or excluded entirely.

CLI Usage

# Full reindex (drops all collections and rebuilds)
go run cmd/indexer/main.go --full

# Index specific collections only (drops + recreates them)
go run cmd/indexer/main.go --only=verses,letters

# Index all, but skip collections that already exist
go run cmd/indexer/main.go

Indexer Robustness

  • Typesense connection timeout: 5 minutes. The indexer uses typesense.WithConnectionTimeout(5*time.Minute) (vs the library default of 5s) because the magazines and other_documents collections ship multi-MB batches that exceeded the default. Without this, those collections silently ended up empty in Typesense — the indexer logged context deadline exceeded per batch but moved on.
  • Reduced batch size for heavy collections. magazines and other_documents use importBatch(..., 25) instead of the default 500 because each document contains very large paragraph arrays (entire magazine issues / chapters concatenated). All other collections still use batch size 500.
  • Importing post-fix. If you have empty magazine / other_documents collections from before this fix, simply run --full (or --only=magazines,other_documents) to rebuild.

Data Models

Content Models (MongoDB)

Model Collection Description
Book book_meta Book metadata (title, author, chapters, etc.)
Verse verses Individual verses (BG, SB, CC, ISO, NOI, BS, MM, NBS)
Letter letters 6,800+ letters (1947–1977)
Essay essays 23 standalone essays
GlossaryEntry glossary 2,749 Sanskrit/Vedic terms
Disciple disciples 4,747 initiated disciples
Transcription transcriptions 3,787 lectures, conversations, philosophy discussions, SB dictations — with audio URLs

User Models (PostgreSQL)

Model Table Description
User users User accounts (email, name, avatar, role)
Bookmark bookmarks Saved content references with optional notes
Collection collections User-created collections of bookmarks
Mention mentions Content shared/tagged by users
ShortURL short_urls Shortened URLs with click tracking

API Response Format

All responses follow a consistent envelope:

{
  "success": true,
  "data": { ... },
  "meta": {
    "page": 1,
    "limit": 20,
    "total": 150,
    "total_pages": 8
  }
}

Error responses:

{
  "success": false,
  "error": {
    "code": 400,
    "message": "validation failed"
  }
}

Middleware Stack

Middleware Description
RequestID Generates unique request ID (chi built-in)
RealIP Extracts real client IP from proxy headers
Logger Zap-based structured request logging
Recoverer Panic recovery (chi built-in)
Compress Gzip compression (level 5)
CORS Cross-origin resource sharing (frontend URL whitelisted)
Auth JWT Bearer token validation (applied to protected routes)
RateLimit Token bucket rate limiter (in-memory → DragonflyDB in production)

Environment Variables

# Server
APP_ENV=development          # "development" | "production"
PORT=8080

# MongoDB (content)
MONGO_URI=mongodb://...      # required
MONGO_DB=spsays

# PostgreSQL (users, relations)
POSTGRES_URI=postgres://...

# Typesense
TYPESENSE_URL=https://search.prabhupadasays.org
TYPESENSE_API_KEY=

# DragonflyDB / Redis
REDIS_URI=redis://localhost:6379

# S3
S3_REGION=
S3_BUCKET=
S3_ACCESS_KEY=
S3_SECRET_KEY=
S3_ENDPOINT=              # optional — for MinIO or S3-compatible services

# Push Notifications
FCM_CREDENTIALS_FILE=     # path to Firebase service account JSON
APNS_KEY_FILE=            # path to APNs auth key (.p8)
APNS_KEY_ID=
APNS_TEAM_ID=
APNS_BUNDLE_ID=

# JWT
JWT_SECRET=               # required in production
JWT_EXPIRATION=24h

# Frontend (CORS)
FRONTEND_URL=http://localhost:3000

How to Run

# Setup
cp .env.example .env       # then fill in values

# Development (with live reload — requires air)
make dev

# Build + run
make run

# Build only
make build

# Run tests
make test

# Lint (requires golangci-lint)
make lint

# Tidy deps
make tidy

Relationship to Other Projects

Project Path Relationship
knowledgebase ../knowledgebase/ Next.js frontend — depends entirely on this BFF for all data (books, letters, essays, glossary, disciples, journals, magazines, other documents, transcriptions). Has NO direct database (Prisma/MongoDB) or S3 connections. The BFF must be running for the frontend to function.
extractor ../extractor/ Python/TS extraction scripts + raw data + seed scripts
spsays-bff ./ (this project) Go BFF API server — single source of truth for all content served to clients

Migration status (Prisma → BFF):

  1. ✅ Done — BFF API routes matching all former Prisma queries
  2. ✅ Done — Frontend migrated from Prisma to BFF API calls (Prisma, MongoDB driver, and S3 SDK removed from knowledgebase)
  3. ✅ Done — Typesense indexing + federated search endpoints (cmd/indexer/main.go, /api/v1/search)
  4. Add PostgreSQL for user features (auth, bookmarks, collections, mentions)
  5. Add mobile app support (same API, push notifications)

Known Issues

  • Verse purport content is not searchable by default. Because purport and purport_ascii are excluded from the verses query_by list (see Per-Collection Field Strategy above), phrases that appear only inside a purport — and not in the verse's reference, transliteration, or translation — will not match a normal verse search. Surfacing those matches would require a separate "deep search" toggle that adds purport and purport_ascii (with lower weights) to query_by. This is not yet implemented.

  • Stopwords set is an external Typesense resource. The common stopwords set is configured server-side via PUT /stopwords/common and is not managed by the indexer or the BFF code. If Typesense is rebuilt from scratch or restored from a snapshot that predates the stopwords config, the set must be reprovisioned manually. The BFF passes stopwords=common on every search; if the set doesn't exist, Typesense simply ignores the parameter (rather than erroring), so symptoms would be “searches like 'letter to tusta krsna' suddenly behave worse,” not hard failures.

  • Indexer is destructive on --full or --only=. Both flags drop the named collection(s) before recreating them. There is currently no incremental indexer — if a single document changes in MongoDB, the whole collection has to be reindexed. Acceptable today (largest collection is verses at ~21k docs, takes seconds), but worth keeping in mind.

Design Principles

  • Single source of truth — the BFF is the sole data provider for all content consumed by the frontend and future mobile apps. No client should bypass the BFF to access MongoDB, S3, or any other backing service directly.
  • internal/ — private to this module (handlers, services, repositories, middleware, models, config)
  • pkg/ — public packages that could be imported by other projects (response helpers, validators, push sender)
  • Repository pattern — data access isolated behind interfaces (easy to swap Mongo ↔ Postgres, mock for tests)
  • Service layer — business logic between handlers and repositories (orchestration, validation, caching)
  • Handler layer — thin HTTP handlers that parse requests, call services, return responses
  • No ORM — use native drivers (mongo-go-driver, pgx) for full control
  • Explicit book routing — SB, CC, and KB have dedicated route groups to avoid URL ambiguity with generic {code}/{chapter}/{verse} patterns
  • Search gateway — search is provided by Typesense, indexed from MongoDB via the indexer CLI; the BFF acts as the search gateway with multi-collection federated queries.
Reference in New Issue View Git Blame Copy Permalink