krispire/spsays-bff
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
76a144e1ea72718972992761f7b4bc4a67b9ae36
spsays-bff/PROJECT_CONTEXT.md
T
Krispire Admin 76a144e1ea search added
2026-05-10 20:41:40 +05:30

22 KiB
Raw Blame History

Prabhupāda Says — BFF (Backend for Frontend)

What this project is

A Go-based Backend for Frontend (BFF) API server for prabhupadasays.org. It is the sole data provider for the production Next.js frontend (and future mobile apps). The knowledgebase frontend has removed all direct Prisma, MongoDB, and S3 dependencies — everything flows through this BFF. It aggregates content from MongoDB, user data from PostgreSQL, search from Typesense, caching from DragonflyDB, and media from S3 into a unified API surface.

Tech Stack

  • Go 1.23 — API server
  • Chi v5 — lightweight HTTP router with middleware support
  • Zap — structured logging (uber-go/zap)
  • MongoDB — content database (books, verses, letters, essays, glossary, disciples, transcriptions)
  • PostgreSQL — relational data (users, bookmarks, collections, mentions, URL shortener)
  • Typesense — typo-tolerant full-text search engine (https://typesense.org)
  • DragonflyDB — Redis-compatible cache (API responses, rate limiting, sessions)
  • AWS S3 — media storage (avatars, images, audio files)
  • FCM + APNs — push notifications for Android and iOS
  • JWT — stateless authentication

Architecture

Note: The BFF is now the ONLY data path for the frontend. The knowledgebase has no Prisma bypass, no direct MongoDB connections, and no direct S3 access. All content is served exclusively through this API.

┌──────────────┐     ┌─────────────────────────────────────────┐
│  Next.js     │────▶│  Go BFF API (:8080)                      │
│  (frontend)  │     │                                          │
│  (sole data  │     │  cmd/api/main.go → server.go → handlers │
│   path ↑)    │     │                                          │
├──────────────┤     │  ┌──────────┐  ┌───────────────┐        │
│  Mobile Apps │────▶│  │ Handlers │──│ Services      │        │
│  (future)    │     │  └──────────┘  └───────┬───────┘        │
└──────────────┘     │                        │                │
                     │  ┌─────────────────────┼──────────┐     │
                     │  ▼         ▼           ▼     ▼    ▼     │
                     │ Mongo  Postgres  Typesense  S3  Dragonfly│
                     │ (content)(users,   (search)(media)(cache)│
                     │          mentions,                       │
                     │          URL short,                      │
                     │          relations)                      │
                     └─────────────────────────────────────────┘

Project Structure

spsays-bff/
├── cmd/
│   ├── api/
│   │   └── main.go              → Entry point (config, logger, server, graceful shutdown)
│   └── indexer/
│       └── main.go              → CLI tool for indexing MongoDB content into Typesense
│                                   • --full           drop all collections and reindex
│                                   • --only=verses,letters  index specific collections only
├── internal/
│   ├── config/
│   │   └── config.go            → Environment config (all services: Mongo, Postgres, Typesense, S3, etc.)
│   ├── server/
│   │   └── server.go            → Chi router setup, global middleware, route registration
│   ├── handler/                  → HTTP handlers (one file per domain)
│   │   ├── health.go            → Health check endpoint
│   │   ├── search.go            → Federated search handler (multi-collection, typo-tolerant)
│   │   ├── book.go              → Book routes (SB, CC, KB, BG, generic verse & prose)
│   │   ├── letter.go            → Letter routes (by year, recipient, location)
│   │   ├── essay.go             → Essay routes
│   │   ├── glossary.go          → Glossary routes (letters, terms)
│   │   ├── disciple.go          → Disciple routes (filter, search, aggregations)
│   │   ├── journal.go           → Journal routes
│   │   ├── magazine.go          → Magazine routes
│   │   ├── other_document.go    → Other-document routes
│   │   └── transcription.go     → Transcription routes (by year, place, type, slug)
│   ├── service/                  → Business logic layer (orchestrates repositories)
│   ├── repository/               → Data access layer (Mongo, Postgres, Typesense, S3, Dragonfly)
│   │   ├── mongo.go             → MongoDB connection + database handle
│   │   ├── book_repo.go         → Book / verse queries
│   │   ├── letter_repo.go       → Letter queries
│   │   ├── essay_repo.go        → Essay queries
│   │   ├── glossary_repo.go     → Glossary queries
│   │   ├── disciple_repo.go     → Disciple queries
│   │   ├── journal_repo.go      → Journal queries
│   │   ├── magazine_repo.go     → Magazine queries
│   │   ├── other_document_repo.go → Other-document queries
│   │   ├── transcription_repo.go → Transcription queries (with contentJson fallback)
│   │   └── typesense.go         → Typesense client wrapper (search, indexing helpers)
│   ├── middleware/
│   │   ├── logger.go            → Zap-based request logging (method, path, status, duration)
│   │   ├── auth.go              → JWT Bearer token authentication + context user extraction
│   │   └── ratelimit.go         → In-memory token bucket rate limiter (→ DragonflyDB in production)
│   └── model/
│       ├── content.go           → Content models (Book, Verse, Letter, Essay, GlossaryEntry,
│       │                           Disciple, Transcription, ContentBlock, Synonym)
│       └── user.go              → User models (User, Bookmark, Collection, Mention, ShortURL)
├── pkg/                          → Public packages (importable by other projects)
│   ├── response/
│   │   └── response.go          → Standardized API response helpers (JSON, Error, Envelope, Meta)
│   ├── validator/
│   │   └── validator.go         → Field validation (Required, MaxLength, Email, In, etc.)
│   ├── push/                     → Push notification sender (FCM + APNs) — placeholder
│   └── shorturl/                 → URL shortener logic — placeholder
├── migrations/                   → PostgreSQL migrations (goose or golang-migrate)
├── go.mod                        → Module: github.com/prabhupadasays/spsays-bff
├── go.sum
├── Makefile                      → build, run, dev, test, clean, lint, tidy
├── .env.example                  → Template for all environment variables
└── .gitignore

API Routes

All routes are prefixed with /api/v1.

Health

Method Route Description
GET /health Server health check

Books

Method Route Description
GET /api/v1/books List all books with full metadata (chapters, cantos, parts, volumes, etc.)
GET /api/v1/books/sb SB book metadata
GET /api/v1/books/sb/{canto} SB canto chapter listing
GET /api/v1/books/sb/{canto}/{chapter} SB chapter verse index
GET /api/v1/books/sb/{canto}/{chapter}/verses SB paginated verses
GET /api/v1/books/sb/{canto}/{chapter}/{verse} SB single verse
GET /api/v1/books/cc CC book metadata
GET /api/v1/books/cc/{lila} CC līlā chapter listing
GET /api/v1/books/cc/{lila}/{chapter} CC chapter verse index
GET /api/v1/books/cc/{lila}/{chapter}/verses CC paginated verses
GET /api/v1/books/cc/{lila}/{chapter}/{verse} CC single verse
GET /api/v1/books/kb KB book metadata
GET /api/v1/books/kb/{volume} KB volume chapter listing
GET /api/v1/books/kb/{volume}/{slug} KB single chapter
GET /api/v1/books/bg BG book metadata
GET /api/v1/books/bg/{chapter} BG chapter verse index
GET /api/v1/books/bg/{chapter}/verses BG paginated verses
GET /api/v1/books/bg/{chapter}/{verse} BG single verse
GET /api/v1/books/{code} Any other book metadata
GET /api/v1/books/{code}/chapters Prose book chapter listing (no paragraphs)
GET /api/v1/books/{code}/chapters/{slug} Prose book single chapter
GET /api/v1/books/{code}/{chapter} Verse book chapter index
GET /api/v1/books/{code}/{chapter}/verses Verse book paginated verses
GET /api/v1/books/{code}/{chapter}/{verse} Verse book single verse

Letters

Method Route Description
GET /api/v1/letters Total letter count
GET /api/v1/letters/years Year → count aggregation
GET /api/v1/letters/recipients Recipient → count aggregation
GET /api/v1/letters/locations Location → count aggregation
GET /api/v1/letters/year/{year} Letters for a year
GET /api/v1/letters/recipient/{name} Letters for a recipient
GET /api/v1/letters/location/{name} Letters from a location
GET /api/v1/letters/{slug} Single letter

Essays

Method Route Description
GET /api/v1/essays All essay summaries
GET /api/v1/essays/{slug} Single essay

Glossary

Method Route Description
GET /api/v1/glossary All glossary entries
GET /api/v1/glossary/letters Letter → count mapping
GET /api/v1/glossary/letter/{letter} Entries for a letter (with diacritical grouping)
GET /api/v1/glossary/letter/{letter}/terms Term names only for sidebar
GET /api/v1/glossary/term/{term} Single entry by term

Disciples

Method Route Description
GET /api/v1/disciples Filtered list (?letter=&suffix=)
GET /api/v1/disciples/count Total + filtered counts
GET /api/v1/disciples/letters Letter → count aggregation
GET /api/v1/disciples/suffixes Suffix → count aggregation
GET /api/v1/disciples/search Search by name (?q=)

Journals

Method Route Description
GET /api/v1/journals All journal metadata
GET /api/v1/journals/{book} Single journal meta
GET /api/v1/journals/{book}/chapters Journal chapters
GET /api/v1/journals/{book}/chapters/{slug} Single chapter

Magazines

Method Route Description
GET /api/v1/magazines All magazine metadata
GET /api/v1/magazines/{book} Single magazine meta
GET /api/v1/magazines/{book}/issues Magazine issues
GET /api/v1/magazines/{book}/issues/{slug} Single issue

Other Documents

Method Route Description
GET /api/v1/other-documents All document metadata
GET /api/v1/other-documents/{book} Single document meta
GET /api/v1/other-documents/{book}/chapters Document chapters
GET /api/v1/other-documents/{book}/chapters/{slug} Single chapter

Transcriptions

Method Route Description
GET /api/v1/transcriptions Total transcription count
GET /api/v1/transcriptions/years Year → count aggregation
GET /api/v1/transcriptions/places Place → count aggregation
GET /api/v1/transcriptions/types Type distribution ({lectures, conversations})
GET /api/v1/transcriptions/year/{year} Transcriptions for a year (summaries)
GET /api/v1/transcriptions/place/{place} Transcriptions for a place (summaries)
GET /api/v1/transcriptions/type/{type} Transcriptions by type (lecture or conversation)
GET /api/v1/transcriptions/adjacent/{slug} Prev/next within same year (?year=N)
GET /api/v1/transcriptions/{slug} Single transcription (full content + audio URL)

Search

Method Route Description
GET /api/v1/search?q=...&type=...&page=1&limit=20 Federated typo-tolerant search across all content

Query params:

  • q — search query (required)
  • type — content type filter, comma-separated. Valid values:
    • all (default) — search all 9 collections
    • verses, prose, letters, transcriptions, glossary, essays, journals, magazines, other_documents
    • Multiple: verses,letters,transcriptions
  • page — page number (default 1)
  • limit — results per page (default 20, max 100)

Response:

{
  "success": true,
  "data": {
    "query": "surrender unto",
    "type": "all",
    "results": [
      {
        "collection": "verses",
        "id": "...",
        "title": "BG 18.66",
        "snippet": "...just <mark>surrender unto</mark> Me...",
        "url": "/books/bg/18/66",
        "meta": { "book": "bg", "reference": "BG 18.66" }
      }
    ],
    "total": 150,
    "page": 1,
    "limit": 20,
    "total_pages": 8
  }
}

Authenticated (JWT required) — planned

Method Route Description
POST /api/v1/auth/register User registration
POST /api/v1/auth/login User login (returns JWT)
POST /api/v1/auth/refresh Refresh JWT token
GET /api/v1/users/me Get current user profile
PUT /api/v1/users/me Update current user profile
GET /api/v1/collections List user's collections
POST /api/v1/collections Create a collection
POST /api/v1/collections/:id/items Add item to collection
DELETE /api/v1/collections/:id/items/:itemId Remove item from collection
GET /api/v1/mentions List user's mentions
POST /api/v1/mentions Create a mention
POST /api/v1/s Create a short URL
POST /api/v1/notifications/register Register device for push notifications

Typesense Indexing

The BFF includes a CLI tool (cmd/indexer/main.go) for managing Typesense indexes.

Collections (9 total, ~35,500 documents)

Collection MongoDB Source Documents Searchable Fields
verses verses 21,127 translation, purport, transliteration, reference (+ ASCII variants)
prose prose_chapters 627 title, content (+ ASCII variants)
letters letters 6,866 content, recipient_display (+ ASCII variants)
transcriptions transcriptions 3,787 content, name, reference (+ ASCII variants)
glossary glossary 2,749 term, definition (+ ASCII variants)
essays essays 23 title, content (+ ASCII variants)
journals journal_chapters 300 title, content (+ ASCII variants)
magazines magazine_issues 59 title, content (+ ASCII variants)
other_documents other_document_chapters 28 title, content (+ ASCII variants)

ASCII Shadow Fields

For diacritic-insensitive search, every searchable text field has a _ascii shadow field with all IAST diacritics stripped (ā→a, ṛ→r, ṣ→s, ṭ→t, ñ→n, etc.). This allows queries like "Krsna" to match "Kṛṣṇa", "Yashodanandana" to match "Yaśodanandana".

Search Configuration

  • prefix=true — partial word matching (e.g., "sarv" matches "sarva")
  • num_typos=2 — up to 2 character substitutions allowed per word
  • HTML stripping — all paragraph content has HTML tags removed before indexing

CLI Usage

# Full reindex (drops all collections and rebuilds)
go run cmd/indexer/main.go --full

# Index specific collections only (drops + recreates them)
go run cmd/indexer/main.go --only=verses,letters

# Index all, but skip collections that already exist
go run cmd/indexer/main.go

Data Models

Content Models (MongoDB)

Model Collection Description
Book book_meta Book metadata (title, author, chapters, etc.)
Verse verses Individual verses (BG, SB, CC, ISO, NOI, BS, MM, NBS)
Letter letters 6,800+ letters (1947–1977)
Essay essays 23 standalone essays
GlossaryEntry glossary 2,749 Sanskrit/Vedic terms
Disciple disciples 4,747 initiated disciples
Transcription transcriptions 3,787 lectures, conversations, philosophy discussions, SB dictations — with audio URLs

User Models (PostgreSQL)

Model Table Description
User users User accounts (email, name, avatar, role)
Bookmark bookmarks Saved content references with optional notes
Collection collections User-created collections of bookmarks
Mention mentions Content shared/tagged by users
ShortURL short_urls Shortened URLs with click tracking

API Response Format

All responses follow a consistent envelope:

{
  "success": true,
  "data": { ... },
  "meta": {
    "page": 1,
    "limit": 20,
    "total": 150,
    "total_pages": 8
  }
}

Error responses:

{
  "success": false,
  "error": {
    "code": 400,
    "message": "validation failed"
  }
}

Middleware Stack

Middleware Description
RequestID Generates unique request ID (chi built-in)
RealIP Extracts real client IP from proxy headers
Logger Zap-based structured request logging
Recoverer Panic recovery (chi built-in)
Compress Gzip compression (level 5)
CORS Cross-origin resource sharing (frontend URL whitelisted)
Auth JWT Bearer token validation (applied to protected routes)
RateLimit Token bucket rate limiter (in-memory → DragonflyDB in production)

Environment Variables

# Server
APP_ENV=development          # "development" | "production"
PORT=8080

# MongoDB (content)
MONGO_URI=mongodb://...      # required
MONGO_DB=spsays

# PostgreSQL (users, relations)
POSTGRES_URI=postgres://...

# Typesense
TYPESENSE_URL=https://search.prabhupadasays.org
TYPESENSE_API_KEY=

# DragonflyDB / Redis
REDIS_URI=redis://localhost:6379

# S3
S3_REGION=
S3_BUCKET=
S3_ACCESS_KEY=
S3_SECRET_KEY=
S3_ENDPOINT=              # optional — for MinIO or S3-compatible services

# Push Notifications
FCM_CREDENTIALS_FILE=     # path to Firebase service account JSON
APNS_KEY_FILE=            # path to APNs auth key (.p8)
APNS_KEY_ID=
APNS_TEAM_ID=
APNS_BUNDLE_ID=

# JWT
JWT_SECRET=               # required in production
JWT_EXPIRATION=24h

# Frontend (CORS)
FRONTEND_URL=http://localhost:3000

How to Run

# Setup
cp .env.example .env       # then fill in values

# Development (with live reload — requires air)
make dev

# Build + run
make run

# Build only
make build

# Run tests
make test

# Lint (requires golangci-lint)
make lint

# Tidy deps
make tidy

Relationship to Other Projects

Project Path Relationship
knowledgebase ../knowledgebase/ Next.js frontend — depends entirely on this BFF for all data (books, letters, essays, glossary, disciples, journals, magazines, other documents, transcriptions). Has NO direct database (Prisma/MongoDB) or S3 connections. The BFF must be running for the frontend to function.
extractor ../extractor/ Python/TS extraction scripts + raw data + seed scripts
spsays-bff ./ (this project) Go BFF API server — single source of truth for all content served to clients

Migration status (Prisma → BFF):

  1. ✅ Done — BFF API routes matching all former Prisma queries
  2. ✅ Done — Frontend migrated from Prisma to BFF API calls (Prisma, MongoDB driver, and S3 SDK removed from knowledgebase)
  3. ✅ Done — Typesense indexing + federated search endpoints (cmd/indexer/main.go, /api/v1/search)
  4. Add PostgreSQL for user features (auth, bookmarks, collections, mentions)
  5. Add mobile app support (same API, push notifications)

Design Principles

  • Single source of truth — the BFF is the sole data provider for all content consumed by the frontend and future mobile apps. No client should bypass the BFF to access MongoDB, S3, or any other backing service directly.
  • internal/ — private to this module (handlers, services, repositories, middleware, models, config)
  • pkg/ — public packages that could be imported by other projects (response helpers, validators, push sender)
  • Repository pattern — data access isolated behind interfaces (easy to swap Mongo ↔ Postgres, mock for tests)
  • Service layer — business logic between handlers and repositories (orchestration, validation, caching)
  • Handler layer — thin HTTP handlers that parse requests, call services, return responses
  • No ORM — use native drivers (mongo-go-driver, pgx) for full control
  • Explicit book routing — SB, CC, and KB have dedicated route groups to avoid URL ambiguity with generic {code}/{chapter}/{verse} patterns
  • Search gateway — search is provided by Typesense, indexed from MongoDB via the indexer CLI; the BFF acts as the search gateway with multi-collection federated queries.
Reference in New Issue View Git Blame Copy Permalink