18 KiB
18 KiB
Prabhupāda Says — Project Context
What this project is
A Next.js app that converts legacy RTF files containing Vedic scriptures (Bhagavad-gītā, Śrīmad-Bhāgavatam, Caitanya-caritāmṛta, Kṛṣṇa Book, letters, magazines, etc.) into structured, browsable content with proper Unicode Sanskrit.
The original RTF files use custom font encodings (Indevr, ScaGoudy, Inbenr/Inbeni, RM Devanagari, Balaram, etc.) where ASCII characters map to Devanagari/Bengali glyphs and IAST diacritical marks.
Tech Stack
- Next.js 16 with Turbopack, React 19, TypeScript
- Tailwind CSS v4 with
@tailwindcss/postcss - shadcn/ui (base-nova style, neutral base color, lucide icons) — components: Button, Card, Badge, Tooltip, Input
- next-themes for dark/light mode
- Prisma v6 with MongoDB (provider
prisma-client-js) - MongoDB native driver (
mongodbpackage) for seeding (Prisma requires replica set for writes) - Python 3 with BeautifulSoup4 +
indic-transliterationfor HTML extraction scripts - Database: MongoDB at
DATABASE_URLenv variable with?authSource=admin
Project Structure
spsays-training/
├── app/
│ ├── page.tsx → "Prabhupāda Says" landing page (Books, Glossary, RTFs)
│ ├── layout.tsx → ThemeProvider + TooltipProvider wrapper
│ ├── globals.css → shadcn/Tailwind v4 CSS with dark mode
│ ├── books/
│ │ ├── page.tsx → Books index (BG, SB, CC, SB1962, KB)
│ │ ├── chapter-viewer.tsx → Shared client component for verse-based books
│ │ ├── extracted-verse-count.tsx → (i) tooltip for verse counts
│ │ ├── bg/ → Bhagavad-gītā pages
│ │ │ ├── page.tsx → Chapter list
│ │ │ └── [chapter]/
│ │ │ ├── page.tsx → Redirects to first verse
│ │ │ └── [verse]/page.tsx → Single verse viewer
│ │ ├── sb/ → Śrīmad-Bhāgavatam pages
│ │ │ ├── page.tsx → Canto list
│ │ │ └── [canto]/
│ │ │ ├── page.tsx → Chapter list
│ │ │ └── [chapter]/
│ │ │ ├── page.tsx → Redirects to first verse
│ │ │ └── [verse]/page.tsx → Single verse viewer
│ │ ├── cc/ → Caitanya-caritāmṛta pages (same pattern, [lila]/[chapter]/[verse])
│ │ ├── sb1962/ → 1962 SB First Canto (same pattern as BG, [chapter]/[verse])
│ │ └── kb/ → Kṛṣṇa Book (prose, no verses)
│ │ ├── page.tsx → Volume list
│ │ └── [volume]/
│ │ ├── page.tsx → Chapter list + intro segments
│ │ └── [slug]/page.tsx → Reading page with sidebar
│ ├── glossary/
│ │ ├── page.tsx → Server component wrapper
│ │ └── glossary-client.tsx → Searchable client component
│ ├── rtfs/
│ │ ├── page.tsx → RTF file listing (gated by ALLOW_RTF env)
│ │ └── [slug]/page.tsx → RTF iframe viewer (gated by ALLOW_RTF env)
│ └── api/
│ ├── books/
│ │ ├── route.ts → GET all book metas
│ │ └── [book]/
│ │ ├── route.ts → GET single book meta
│ │ └── verses/route.ts → GET verses (?chapter=&verse=&division=&indexOnly=&page=&limit=)
│ └── glossary/route.ts → GET all glossary entries
├── components/
│ ├── ui/ → shadcn components (button, card, badge, tooltip, input)
│ ├── page-header.tsx → Shared header with ThemeToggle
│ ├── theme-provider.tsx → next-themes wrapper
│ └── theme-toggle.tsx → Sun/moon toggle
├── lib/
│ ├── db.ts → Prisma client singleton
│ ├── api.ts → Data access layer (getAllBooks, getBookMeta, getChapterVerses, etc.)
│ ├── utils.ts → cn() helper
│ └── jsons/books/ → Extracted JSON data
│ ├── bg/ → BG chapters/verses/*.json + intro.json
│ ├── sb/ → SB cantos/chapters/verses/*.json + intro.json
│ ├── cc/ → CC lilas/chapters/verses/*.json + intro.json
│ ├── sb-1962/ → 1962 SB chapters/verses/*.json + intro.json
│ ├── kb/ → Kṛṣṇa Book
│ │ ├── intro.json → Book metadata + chapter listing (89 chapters)
│ │ ├── vol-1/intro/ → 5 intro segments (flap, dedication, George Harrison, preface, intro)
│ │ ├── vol-1/chapters/ → 54 chapter JSONs
│ │ └── vol-2/chapters/ → 35 chapter JSONs
│ └── glossary/index.json → 2,749 glossary entries
├── prisma/
│ └── schema.prisma → Verse, BookMeta, ProseChapter, GlossaryEntry models
├── scripts/
│ ├── bg_extract.py → BG extraction from bg.html
│ ├── sb_extract.py → SB extraction from sb.html
│ ├── cc_extract.py → CC extraction from cc.html
│ ├── sb1962_extract.py → 1962 SB extraction from others-before-1977.html
│ ├── match_sb1962.py → Match 1962 SB transliterations with standard SB IAST
│ ├── glossary_extract.py → Glossary extraction from glossary.html
│ ├── kb_extract_intro.py → KB book-level intro + chapter listing
│ ├── kb_extract_vol1_intro.py → KB Vol 1 front matter segments
│ ├── kb_extract_vol1_chapters.py → KB Vol 1 chapters
│ ├── kb_extract_vol2_chapters.py → KB Vol 2 chapters
│ ├── fix-bengali-encoding.py → Fix Bengali font encoding in cc.html
│ ├── inject-fonts.py → Inject @font-face rules into converted HTML
│ ├── seed-db.ts → Main MongoDB seeder (all books + glossary)
│ └── seed-kb-vol2.ts → KB Vol 2 seeder (uses $push workaround)
└── public/
├── rtfs/ → Original RTF files (~430MB, 17 files)
├── html/ → HTML converted from RTF (textutil + @font-face injection)
└── fonts/ → Custom TTF fonts (Balaram, ScaGoudy, Indevr, Inbenr, etc.)
Database Schema (Prisma + MongoDB)
type Synonym {
word String
meanings String[]
}
model Verse {
id String @id @default(auto()) @map("_id") @db.ObjectId
book String // "bg" | "sb" | "cc" | "sb1962"
division String? // null for BG/SB1962; canto for SB; lila for CC
chapter Int
verse Int
verseEnd Int?
reference String
bookTitle String
devanagari String
bengali String? // only CC has Bengali
transliteration String
synonyms Synonym[]
translation String
purport String[]
@@unique([book, division, chapter, verse])
@@index([book])
@@index([book, division])
@@index([book, division, chapter])
@@map("verses")
}
model BookMeta {
id String @id @default(auto()) @map("_id") @db.ObjectId
book String @unique // "bg" | "sb" | "cc" | "sb1962" | "kb"
data Json
@@map("book_meta")
}
model ProseChapter {
id String @id @default(auto()) @map("_id") @db.ObjectId
book String // "kb"
volume Int
chapter Int? // null for intro segments
title String
type String // "chapter" | "intro-segment"
slug String
order Int
paragraphs String[]
@@unique([book, volume, slug])
@@index([book])
@@index([book, volume])
@@index([book, volume, type])
@@map("prose_chapters")
}
model GlossaryEntry {
id String @id @default(auto()) @map("_id") @db.ObjectId
term String
definition String
definitionHtml String
@@unique([term])
@@map("glossary")
}
Books Extracted
| Book | Code | Type | Verses/Chapters | Division |
|---|---|---|---|---|
| Bhagavad-gītā As It Is | bg |
verse | 653 verses, 18 chapters | — |
| Śrīmad-Bhāgavatam | sb |
verse | 8,260 verses, 213 chapters | 10 cantos |
| Śrī Caitanya-caritāmṛta | cc |
verse | 11,348 verses, 62 chapters | 3 līlās (adi/madhya/antya) |
| SB First Canto (1962 Edition) | sb1962 |
verse | 762 verses, 19 chapters | — |
| Kṛṣṇa Book | kb |
prose | 89 chapters (54+35), 2 volumes | 2 volumes |
| Glossary | — | glossary | 2,749 entries | — |
Font Encoding System
ScaGoudy → IAST mapping (used in BG, SB, CC, glossary)
ä→ā Ä→Ā å→ṛ Å→Ṛ ñ→ṣ Ñ→Ṣ ö→ṭ Ö→Ṭ ë→ṇ Ë→Ṇ ò→ḍ Ò→Ḍ
ç→ś Ç→Ś ì→ṅ ï→ñ ü→ū Ü→Ū é→ī É→Ī à→ṁ ù→ḥ è→ṝ û→ṁl ÿ→ḷ
Bengali fonts (Inbenr/Inbeni in CC)
Custom-encoded fonts where ASCII codepoints render as Bengali script. The cc.html had Apple Symbol encoding corruption — textutil mapped bytes through Symbol encoding instead of PUA. Fixed by fix-bengali-encoding.py which remaps 356,396 characters back to PUA range U+F0xx.
Devanagari generation
IAST transliteration → Devanagari via indic-transliteration library. Key fixes:
join_virama_words()— joins virāma + consonant AND virāma + vowel (fixed 12,000+ rendering issues in SB)normalize_devanagari_anusvara()— ङ् → ं before velars, ञ् → ं before palatals
Verse Navigation (verse-based books)
- One verse per page with path-based URLs:
/books/bg/1/5,/books/sb/6/6/1,/books/cc/adi/1/3 - ChapterViewer (
app/books/chapter-viewer.tsx): Client component with:- Sidebar showing ALL verse numbers (from lightweight index query)
- Prev/Next buttons at top and bottom
- Keyboard navigation (← →)
- Client-side verse caching + prefetching adjacent verses
- Click any verse in sidebar → fetches from API and navigates
- API route supports:
?verse=N(single),?indexOnly=true(lightweight index),?page=&limit=(paginated) - Server component SSRs the first verse, client handles navigation
KB (Kṛṣṇa Book) — Prose Book
Different from verse books — chapters are prose paragraphs, no verse structure:
ProseChaptermodel withparagraphs: String[]- 5 intro segments per volume (flap, dedication, George Harrison intro, preface, introduction)
- Sanskrit verses within prose are grouped into
<div style="text-align:center"><i>...</i></div>blocks with<br>tags - Reading page has left sidebar with chapter index, prev/next navigation
Data Access Layer (lib/api.ts)
// Book meta
getAllBooks(): Promise<BookMeta[]>
getBookMeta(book: string): Promise<BookMeta | null>
// Verse index (lightweight — for sidebar)
getChapterVerseIndex(book, division, chapter): Promise<VerseIndexEntry[]>
// Single verse
getSingleVerse(book, division, chapter, verse): Promise<VerseRecord | null>
// Paginated verses
getChapterVerses(book, division, chapter, page?, limit?): Promise<PaginatedVerses>
// Prose chapters (for KB)
getProseChapters(book, volume, type?): Promise<ProseChapterRecord[]>
getProseChapter(book, volume, slug): Promise<ProseChapterRecord | null>
// Glossary
getAllGlossaryEntries(): Promise<GlossaryRecord[]>
The data access layer includes normalizeProse() which handles mixed storage (some chapters stored with native arrays, some with stringified JSON due to BSON corruption workaround). It also includes sanitizeHtml() which strips <script>, <style>, <iframe>, event handlers from paragraphs.
Known Issues
-
MongoDB BSON wire corruption: The MongoDB server (7.0.31) behind a proxy/tunnel occasionally corrupts large BSON documents during transmission. Workaround: seed using
insertOne+$pushfor paragraphs one at a time. TheinsertManyand largeinsertOneoperations fail non-deterministically with "Invalid UTF-8 string" or "Unrecognized BSON type" errors. Retrying usually works. -
SB verse numbering: The HTML had 5 duplicate
p20verse reference tags. Fixed by using thep21TEXT N header as the authoritative verse number (corrects the wrongp20reference). -
SB 1962 chapter 15: Verses 41-45 have different numbering between the 1962 edition and standard SB. The matching script skips these 5 verses (keeps original plain English transliteration).
-
RTF routes gated:
/rtfsand/rtfs/[slug]only accessible whenALLOW_RTF=trueenv variable is set. -
scripts/excluded from tsconfig: Added to"exclude"array to prevent TypeScript build errors from seed scripts. -
MongoDB requires no replica set for reads but requires replica set for Prisma writes. All writes use the native MongoDB driver instead of Prisma.
Environment Variables
DATABASE_URL=mongodb://user:pass@host:port/spsays?authSource=admin
ALLOW_RTF=true # optional — enables RTF viewer routes
How to Run
# Development
npm run dev # Start dev server
# Prisma
npx prisma generate # Regenerate Prisma client after schema changes
# Extraction scripts (Python 3 + beautifulsoup4 + indic-transliteration)
python3 scripts/bg_extract.py # Extract BG → lib/jsons/books/bg/
python3 scripts/sb_extract.py # Extract SB → lib/jsons/books/sb/
python3 scripts/cc_extract.py # Extract CC → lib/jsons/books/cc/
python3 scripts/glossary_extract.py # Extract glossary → lib/jsons/glossary/
python3 scripts/sb1962_extract.py # Extract 1962 SB → lib/jsons/books/sb-1962/
python3 scripts/match_sb1962.py # Match 1962 SB with standard SB IAST
python3 scripts/kb_extract_intro.py # Extract KB intro → lib/jsons/books/kb/intro.json
python3 scripts/kb_extract_vol1_intro.py # KB Vol 1 front matter
python3 scripts/kb_extract_vol1_chapters.py # KB Vol 1 chapters
python3 scripts/kb_extract_vol2_chapters.py # KB Vol 2 chapters
# Database seeding (TypeScript + tsx)
npx tsx scripts/seed-db.ts # Seed all data to MongoDB
npx tsx scripts/seed-kb-vol2.ts # Seed KB Vol 2 (separate due to BSON issues)
# Build
npx next build # Production build
URL Routes
| Route | Description |
|---|---|
/ |
Landing page — Books, Glossary, RTF Viewer cards |
/books |
All books index |
/books/bg |
BG chapter list |
/books/bg/[chapter] |
Redirects to first verse |
/books/bg/[chapter]/[verse] |
Single verse viewer |
/books/sb |
SB canto list |
/books/sb/[canto] |
Chapter list for a canto |
/books/sb/[canto]/[chapter] |
Redirects to first verse |
/books/sb/[canto]/[chapter]/[verse] |
Single verse viewer |
/books/cc |
CC līlā list |
/books/cc/[lila] |
Chapter list for a līlā |
/books/cc/[lila]/[chapter] |
Redirects to first verse |
/books/cc/[lila]/[chapter]/[verse] |
Single verse viewer |
/books/sb1962 |
1962 SB chapter list |
/books/sb1962/[chapter] |
Redirects to first verse |
/books/sb1962/[chapter]/[verse] |
Single verse viewer |
/books/kb |
KB volume list |
/books/kb/[volume] |
Chapter list + intro segments |
/books/kb/[volume]/[slug] |
Prose reading page with sidebar |
/glossary |
Searchable glossary (2,749 entries) |
/rtfs |
RTF file listing (requires ALLOW_RTF=true) |
/rtfs/[slug] |
RTF iframe viewer (requires ALLOW_RTF=true) |
/api/books |
GET all book metas |
/api/books/[book] |
GET single book meta |
/api/books/[book]/verses |
GET verses with query params |
/api/glossary |
GET all glossary entries |
Verse JSON Schema (BG/SB/CC/SB1962)
{
"book_title": "Śrīmad-Bhāgavatam",
"canto": 1,
"chapter": 1,
"verse": 1,
"verse_end": null,
"reference": "SB 1.1.1",
"devanagari": "ॐ नमो भगवते वासुदेवाय\nजन्माद्यस्य यतो ...",
"bengali": null,
"transliteration": "om namo bhagavate vāsudevāya\njanmādy asya yato ...",
"synonyms": [
{ "word": "om", "meanings": ["O my Lord"] },
{ "word": "namaḥ", "meanings": ["offering my obeisances"] }
],
"translation": "<b>I offer my obeisances unto Lord Śrī Kṛṣṇa...</b>",
"purport": ["Obeisances unto the Personality of Godhead...", "paragraph 2..."]
}
Prose Chapter JSON Schema (KB)
{
"book": "kb",
"volume": 1,
"chapter": 1,
"title": "Advent of Lord Kṛṣṇa",
"type": "chapter",
"paragraphs": [
"<b>1 /Advent of Lord Kṛṣṇa</b>",
"Once the world was overburdened...",
"<div style=\"text-align:center\"><i>verse line 1<br>verse line 2</i></div>",
"<i>Thus ends the Bhaktivedanta purport of the First Chapter of </i>Kṛṣṇa..."
]
}
Intro Segment JSON Schema (KB)
{
"slug": "words-from-apple",
"title": "Words from Apple — by George Harrison",
"type": "intro-segment",
"book": "kb",
"volume": 1,
"order": 3,
"paragraphs": ["<b>Words From Apple</b>", "Everybody is looking for KṚṢṆA.", "..."]
}
Glossary JSON Schema
{
"title": "Glossary of Vedic Terms",
"description": "A comprehensive glossary of Sanskrit and Vedic terms...",
"totalEntries": 2749,
"letters": ["A", "B", "C", ...],
"entries": [
{
"term": "Ācārya",
"definition": "a spiritual master who teaches by his own example...",
"definition_html": "a spiritual master who teaches by his own example..."
}
]
}