krispire/spsays-training
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ae6a75278754b6895c9be934b45be4ee7d66b6cf
spsays-training/PROJECT_CONTEXT.md
T

24 KiB
Raw Blame History

Prabhupāda Says — Project Context

What this project is

A Next.js app that converts legacy RTF files containing Vedic scriptures and teachings by His Divine Grace A.C. Bhaktivedanta Swami Prabhupāda into structured, browsable content with proper Unicode Sanskrit.

The original RTF files use custom font encodings (Indevr, ScaGoudy, Inbenr/Inbeni, RM Devanagari, Balaram, etc.) where ASCII characters map to Devanagari/Bengali glyphs and IAST diacritical marks. The extraction pipeline decodes these into proper Unicode, generates Devanagari from IAST, and stores everything in MongoDB for serving via a Next.js frontend.

Tech Stack

  • Next.js 16 with Turbopack, React 19, TypeScript
  • Tailwind CSS v4 with @tailwindcss/postcss
  • shadcn/ui (base-nova style, lucide icons) — components: Button, Card, Badge, Tooltip, Input, Drawer
  • next-themes for dark/light/system mode
  • vaul for mobile drawer navigation
  • Prisma v6 with MongoDB (provider prisma-client-js)
  • MongoDB native driver (mongodb package) for seeding (Prisma requires replica set for writes)
  • Python 3 with BeautifulSoup4 + indic-transliteration for HTML extraction scripts
  • Database: MongoDB at DATABASE_URL env variable with ?authSource=admin

Project Structure

spsays-training/
├── app/
│   ├── page.tsx                → Landing page with Logo, links to Books/Essays/Glossary/RTFs
│   ├── layout.tsx              → ThemeProvider + TooltipProvider + PasswordGate wrapper
│   ├── globals.css             → Vintage color palette + reading settings CSS
│   ├── books/
│   │   ├── page.tsx            → Books index (searchable grid, diacritic-aware search)
│   │   ├── books-grid.tsx      → Client component for searchable book grid
│   │   ├── chapter-viewer.tsx  → Shared client component for verse-based books (BG/SB/CC/SB1962)
│   │   ├── extracted-verse-count.tsx → (i) tooltip for verse counts
│   │   ├── bg/                 → Bhagavad-gītā (18 chapters, 653 verses)
│   │   ├── sb/                 → Śrīmad-Bhāgavatam (10 cantos, 8258 verses)
│   │   ├── cc/                 → Caitanya-caritāmṛta (3 līlās, 11348 verses)
│   │   ├── sb1962/             → SB First Canto 1962 Edition (19 chapters, 762 verses)
│   │   ├── kb/                 → Kṛṣṇa Book (89 chapters, 2 volumes)
│   │   ├── nod/                → The Nectar of Devotion (51 chapters, 4 parts)
│   │   ├── tlc/                → Teachings of Lord Caitanya (32 chapters, 2 parts)
│   │   ├── iso/                → Śrī Īśopaniṣad (18 mantras + invocation)
│   │   ├── noi/                → The Nectar of Instruction (11 texts)
│   │   ├── ej/                 → Easy Journey to Other Planets (2 chapters)
│   │   ├── ek/                 → Elevation to Kṛṣṇa Consciousness (6 chapters)
│   │   ├── mg/                 → Kṛṣṇa Consciousness: The Matchless Gift (7 chapters)
│   │   ├── bbd/                → Beyond Birth and Death (5 chapters)
│   │   ├── owk/                → On the Way to Kṛṣṇa (5 chapters)
│   │   ├── rv/                 → Rāja-vidyā: The King of Knowledge (8 chapters)
│   │   ├── poy/                → The Perfection of Yoga (11 chapters)
│   │   ├── pqpa/               → Perfect Questions Perfect Answers (9 chapters)
│   │   ├── ttp/                → Transcendental Teachings of Prahlāda Mahārāja (5 chapters)
│   │   ├── ochkm/              → On Chanting the Hare Kṛṣṇa Mantra (4 chapters)
│   │   ├── hkh/                → The Kṛṣṇa Consciousness Handbook (19 chapters)
│   │   ├── ssr/                → The Science of Self-Realization (8 chapters, 50 sub-chapters)
│   │   └── ds/                 → Dialectic Spiritualism (13 chapters, 36 sub-chapters)
│   ├── essays/
│   │   ├── page.tsx            → Essays listing page
│   │   └── [slug]/page.tsx     → Individual essay reader
│   ├── glossary/
│   │   ├── page.tsx            → Server component wrapper
│   │   └── glossary-client.tsx → Searchable client component (2749 entries)
│   ├── rtfs/
│   │   ├── page.tsx            → RTF file listing (gated by ALLOW_RTF env)
│   │   └── [slug]/page.tsx     → RTF iframe viewer
│   └── api/
│       ├── books/
│       │   ├── route.ts        → GET all book metas
│       │   └── [book]/
│       │       ├── route.ts    → GET single book meta
│       │       └── verses/route.ts → GET verses (?chapter=&verse=&division=&indexOnly=&page=&limit=)
│       ├── glossary/route.ts   → GET all glossary entries
│       └── verify-access/route.ts → GET (check if password enabled) + POST (validate key)
├── components/
│   ├── ui/                     → shadcn components (button, card, badge, tooltip, input, drawer)
│   ├── page-header.tsx         → Shared header with ReadingSettings
│   ├── reading-settings.tsx    → Theme/font/size/justify accessibility settings panel
│   ├── mobile-nav.tsx          → Left-slide drawer for mobile sidebar navigation
│   ├── logo.tsx                → Theme-aware logo component (light/dark variants)
│   ├── password-gate.tsx       → Client-side password protection (runtime check)
│   └── theme-provider.tsx      → next-themes wrapper
├── lib/
│   ├── db.ts                   → Prisma client singleton
│   ├── api.ts                  → Data access layer (getAllBooks, getBookMeta, getChapterVerses, etc.)
│   ├── utils.ts                → cn() helper
│   ├── nod.ts                  → NoD data access (Prisma)
│   ├── tlc.ts                  → TLC data access (Prisma)
│   ├── iso.ts                  → Īśopaniṣad data access (Prisma)
│   ├── noi.ts                  → NoI data access (Prisma)
│   ├── ej.ts                   → Easy Journey data access (Prisma)
│   ├── ek.ts                   → Elevation to KC data access (Prisma)
│   ├── mg.ts                   → Matchless Gift data access (Prisma)
│   ├── bbd.ts                  → Beyond Birth and Death data access (Prisma)
│   ├── owk.ts                  → On the Way to Kṛṣṇa data access (Prisma)
│   ├── rv.ts                   → Rāja-vidyā data access (Prisma)
│   ├── poy.ts                  → Perfection of Yoga data access (Prisma)
│   ├── pqpa.ts                 → Perfect Questions data access (Prisma)
│   ├── ttp.ts                  → Prahlāda Mahārāja data access (Prisma)
│   ├── ochkm.ts                → On Chanting data access (Prisma)
│   ├── hkh.ts                  → KC Handbook data access (Prisma)
│   ├── ssr.ts                  → Science of Self-Realization data access (Prisma, hierarchical)
│   ├── ds.ts                   → Dialectic Spiritualism data access (Prisma, hierarchical)
│   ├── te.ts                   → Two Essays data access (Prisma, uses essays collection)
│   └── jsons/books/            → Extracted JSON data (all books)
│       ├── bg/, sb/, cc/, sb-1962/, kb/   → Original verse/prose books
│       ├── nod/, tlc/, iso/, noi/         → Extracted from other.html
│       ├── ej/, ek/, mg/, bbd/, owk/      → Small prose books
│       ├── rv/, poy/, pqpa/, ttp/, ochkm/ → More prose books
│       ├── hkh/, ssr/, ds/                → Large structured books
│       ├── te/                            → Two Essays
│       └── glossary/index.json            → 2,749 glossary entries
├── prisma/
│   └── schema.prisma           → Verse, BookMeta, ProseChapter, Essay, GlossaryEntry models
├── scripts/
│   ├── bg_extract.py           → BG extraction from bg.html
│   ├── sb_extract.py           → SB extraction from sb.html
│   ├── cc_extract.py           → CC extraction from cc.html
│   ├── sb1962_extract.py       → 1962 SB extraction
│   ├── nod_extract.py          → NoD extraction from other.html
│   ├── tlc_extract.py          → TLC extraction from other.html
│   ├── iso_extract.py          → Īśopaniṣad extraction from other.html
│   ├── noi_extract.py          → NoI extraction from other.html
│   ├── ej_extract.py           → Easy Journey extraction
│   ├── ek_extract.py           → Elevation to KC extraction
│   ├── mg_extract.py           → Matchless Gift extraction
│   ├── bbd_extract.py          → Beyond Birth and Death extraction
│   ├── owk_extract.py          → On the Way to Kṛṣṇa extraction
│   ├── rv_extract.py           → Rāja-vidyā extraction
│   ├── poy_extract.py          → Perfection of Yoga extraction
│   ├── pqpa_extract.py         → Perfect Questions extraction
│   ├── ttp_extract.py          → Prahlāda Mahārāja extraction
│   ├── ochkm_extract.py        → On Chanting extraction
│   ├── hkh_extract.py          → KC Handbook extraction
│   ├── ssr_extract.py          → Science of Self-Realization extraction (hierarchical sub-chapters)
│   ├── ds_extract.py           → Dialectic Spiritualism extraction (hierarchical sub-chapters)
│   ├── te_extract.py           → Two Essays extraction
│   ├── glossary_extract.py     → Glossary extraction
│   ├── seed-db.ts              → Main MongoDB seeder (BG, SB, CC, SB1962, KB, glossary)
│   ├── seed-nod.ts … seed-ds.ts → Individual book seeders
│   ├── seed-te.ts              → Two Essays seeder (uses essays collection)
│   ├── kb_extract_*.py         → KB extraction scripts (intro, vol1, vol2)
│   ├── inject-fonts.py         → Inject @font-face rules into converted HTML
│   └── fix-bengali-encoding.py → Fix Bengali font encoding in cc.html
└── public/
    ├── rtfs/                   → Original RTF files (~430MB, 17 files)
    ├── html/                   → HTML converted from RTF (textutil + @font-face injection)
    ├── fonts/                  → Custom TTF fonts (Balaram, ScaGoudy, Indevr, Inbenr, RM Devanagari, etc.)
    ├── logo_dark.png           → Logo for light theme (with "says.org")
    ├── logo_light.png          → Logo for dark theme (subtle script)
    └── robots.txt              → Disallow all crawlers

Database Schema (Prisma + MongoDB)

type Synonym {
  word     String
  meanings String[]
}

model Verse {
  id              String    @id @default(auto()) @map("_id") @db.ObjectId
  book            String    // "bg" | "sb" | "cc" | "sb1962" | "iso" | "noi"
  division        String?   // null for BG/SB1962/ISO/NOI; canto for SB; lila for CC
  chapter         Int
  verse           Int
  verseEnd        Int?
  reference       String
  bookTitle       String
  devanagari      String
  bengali         String?   // only CC has Bengali
  transliteration String
  synonyms        Synonym[]
  translation     String
  purport         String[]

  @@unique([book, division, chapter, verse])
  @@index([book])
  @@index([book, division])
  @@index([book, division, chapter])
  @@map("verses")
}

model BookMeta {
  id   String @id @default(auto()) @map("_id") @db.ObjectId
  book String @unique
  data Json

  @@map("book_meta")
}

model ProseChapter {
  id         String   @id @default(auto()) @map("_id") @db.ObjectId
  book       String
  volume     Int      // 0 = no volumes (sentinel for non-volume books)
  chapter    Int?     // null for intro segments
  title      String
  type       String   // "chapter" | "intro-segment" | "conclusion"
  slug       String
  order      Int
  paragraphs String[]

  @@unique([book, volume, slug])
  @@index([book])
  @@index([book, volume])
  @@index([book, volume, type])
  @@map("prose_chapters")
}

model Essay {
  id         String   @id @default(auto()) @map("_id") @db.ObjectId
  slug       String   @unique
  title      String
  author     String
  year       Int?
  group      String?  // e.g., "two-essays"
  order      Int
  paragraphs String[]

  @@index([group])
  @@map("essays")
}

model GlossaryEntry {
  id             String @id @default(auto()) @map("_id") @db.ObjectId
  term           String
  definition     String
  definitionHtml String

  @@unique([term])
  @@map("glossary")
}

Books Extracted

Verse-based Books (stored in verses collection)

Book Code Verses Chapters Division
Bhagavad-gītā As It Is bg 653 18 —
Śrīmad-Bhāgavatam sb 8,258 213 10 cantos
Śrī Caitanya-caritāmṛta cc 11,348 62 3 līlās
SB First Canto (1962 Edition) sb1962 762 19 —
Śrī Īśopaniṣad iso 19 — Invocation + 18 mantras
The Nectar of Instruction noi 11 — 11 texts

Prose Books (stored in prose_chapters collection)

Book Code Chapters Source
Kṛṣṇa Book kb 89 (2 volumes) other.html
The Nectar of Devotion nod 51 (4 parts) other.html
Teachings of Lord Caitanya tlc 32 (2 parts) other.html (1968 ed., plain English)
Easy Journey to Other Planets ej 2 other.html
Elevation to Kṛṣṇa Consciousness ek 6 other.html
Kṛṣṇa Consciousness: The Matchless Gift mg 7 other.html
Beyond Birth and Death bbd 5 other.html
On the Way to Kṛṣṇa owk 5 other.html
Rāja-vidyā: The King of Knowledge rv 8 other.html
The Perfection of Yoga poy 11 other.html
Perfect Questions Perfect Answers pqpa 9 other.html
Transcendental Teachings of Prahlāda Mahārāja ttp 5 other.html
On Chanting the Hare Kṛṣṇa Mantra ochkm 4 other.html
The Kṛṣṇa Consciousness Handbook hkh 19 other.html
The Science of Self-Realization ssr 50 sub-chapters (8 parents) other.html
Dialectic Spiritualism ds 36 sub-chapters (13 parents) other.html

Essays (stored in essays collection)

Essay Slug Group
Krishna The Reservoir Of Pleasure krishna-reservoir-of-pleasure two-essays
Who Is Crazy? who-is-crazy two-essays

Other

Content Collection
Glossary (2,749 entries) glossary

Font Encoding System

ScaGoudy → IAST mapping (used in BG, SB, CC, ISO, NOI, NoD, and most 1969+ books)

ä→ā  Ä→Ā  å→ṛ  Å→Ṛ  ñ→ṣ  Ñ→Ṣ  ö→ṭ  Ö→Ṭ  ë→ṇ  Ë→Ṇ  ò→ḍ  Ò→Ḍ
ç→ś  Ç→Ś  ì→ṅ  ï→ñ  ü→ū  Ü→Ū  é→ī  É→Ī  à→ṁ  ù→ḥ  è→ṝ  û→ṁl  ÿ→ḷ

Bengali fonts (Inbenr/Inbeni in CC)

Custom-encoded fonts where ASCII codepoints render as Bengali script. Fixed by fix-bengali-encoding.py.

RM Devanagari font (used in ISO, NOI)

Custom-encoded Devanagari font (p51 CSS class). The text appears as garbled ASCII without the font. Skipped during extraction — Unicode Devanagari is generated from IAST transliteration instead.

Devanagari generation

IAST transliteration → Devanagari via indic-transliteration library. Key fixes:

  • join_virama_words() — joins virāma + consonant AND virāma + vowel
  • normalize_devanagari_anusvara() — ङ् → ं before velars, ञ् → ं before palatals

Books without ScaGoudy encoding

  • TLC (1968 Edition) — plain English transliterations ("Krishna" not "Kṛṣṇa")
  • Two Essays (1966) — plain English
  • All other books from other.html use ScaGoudy encoding

Book Types and URL Patterns

Verse-based books (use ChapterViewer client component)

Route Pattern Books
/books/bg/[chapter]/[verse] BG
/books/sb/[canto]/[chapter]/[verse] SB
/books/cc/[lila]/[chapter]/[verse] CC
/books/sb1962/[chapter]/[verse] SB1962
/books/iso/[mantra] Īśopaniṣad (verse + intro segments)
/books/noi/[text] Nectar of Instruction (verse + intro segments)

Prose books (use per-book data layers + reading pages)

Route Pattern Books
/books/kb/[volume]/[slug] Kṛṣṇa Book
/books/{code}/[chapter] NoD, TLC, EJ, EK, MG, BBD, OWK, RV, POY, PQPA, TTP, OCHKM, HKH

Hierarchical prose books (sub-chapters as separate pages)

Route Pattern Books
/books/ssr/[chapter] Science of Self-Realization (slugs: "1-1", "1-2", "intro-foreword", etc.)
/books/ds/[chapter] Dialectic Spiritualism (slugs: "1-1" = Socrates, "1-2" = Plato, etc.)

Essays

Route Content
/essays All essays listing
/essays/[slug] Individual essay reader

Other routes

Route Content
/ Landing page
/books Searchable book grid
/glossary Searchable glossary (2,749 entries)
/rtfs RTF viewer (gated by ALLOW_RTF=true)
/api/verify-access Password protection endpoint

Volume Convention

Books without volumes use volume: 0 as a sentinel value in the prose_chapters collection. This satisfies the Prisma schema's @@unique([book, volume, slug]) constraint without requiring schema changes.

  • KB uses volume: 1 and volume: 2
  • All other prose books use volume: 0

Hierarchical Books (SSR, DS)

SSR and DS have parent chapters with sub-chapters stored as individual prose_chapters docs:

  • Sub-chapter slugs: "1-1", "1-2", "2-1", etc.
  • Parent chapter info stored in intro.json metadata: 
    {
  "chapters": [
    {
      "number": 1,
      "title": "Learning the Science of the Soul",
      "subChapters": [
        { "slug": "1-1", "title": "Seeing the Aim of Human Life" },
        { "slug": "1-2", "title": "Your Original Consciousness Is Kṛṣṇa Consciousness" }
      ]
    }
  ]
}
  • Sidebar UI shows parent headings (non-clickable) with sub-chapters indented below
  • Prev/next navigation skips parent headings

UI Features

Reading Settings (accessibility)

  • Theme: Light / Dark / System toggle
  • Serif Font: Toggle to switch reading content to Georgia/serif
  • Font Size: Slider (XS → XXL) using CSS zoom on main and aside
  • Justify Text: Toggle for justified text alignment on reading content
  • Settings persist in localStorage under spsays_reading_prefs
  • Applied via CSS classes on <html> element

Mobile Navigation

  • Left-slide drawer (vaul) triggered by a menu button next to the Previous button
  • Only visible below lg breakpoint
  • Shows the full sidebar navigation grouped by section
  • Implemented in all 24+ reader pages via shared MobileNav component

Search

  • Books page: diacritic-aware search (typing "Krsna" matches "Kṛṣṇa")
  • Glossary page: full-text search across terms and definitions
  • Uses Unicode NFD normalization + manual IAST character stripping

Color Palette

  • Light mode ("Aged Parchment"): Warm cream backgrounds, deep sepia text, saffron/ochre primary
  • Dark mode ("Ancient Manuscript"): Deep warm brown backgrounds, parchment text, golden saffron primary

Password Protection

  • Client-side gate component (PasswordGate) wrapping the entire app
  • Runtime check via GET /api/verify-access (checks if ACCESS_KEY env var is set)
  • Validation via POST /api/verify-access (compares submitted key)
  • Stores valid key in sessionStorage
  • Bypassed when ACCESS_KEY is not set

Environment Variables

DATABASE_URL=mongodb://user:pass@host:port/spsays?authSource=admin
ALLOW_RTF=true           # optional — enables RTF viewer routes
ACCESS_KEY="secret"      # optional — enables password protection (quote if contains #)

How to Run

# Development
npm run dev

# Prisma
npx prisma generate                  # Regenerate Prisma client after schema changes

# Extraction scripts (Python 3 + beautifulsoup4 + indic-transliteration)
python3 scripts/bg_extract.py        # Extract BG
python3 scripts/sb_extract.py        # Extract SB
python3 scripts/cc_extract.py        # Extract CC
python3 scripts/nod_extract.py       # Extract NoD from other.html
python3 scripts/iso_extract.py       # Extract Īśopaniṣad from other.html
python3 scripts/ssr_extract.py       # Extract SSR from other.html
python3 scripts/ds_extract.py        # Extract DS from other.html
# ... (similar for all other books)

# Database seeding
npx tsx scripts/seed-db.ts           # Seed BG, SB, CC, SB1962, KB, glossary
npx tsx scripts/seed-nod.ts          # Seed NoD
npx tsx scripts/seed-iso.ts          # Seed Īśopaniṣad
npx tsx scripts/seed-te.ts           # Seed Two Essays
# ... (similar for all other books)

# Build
npx next build

Verse JSON Schema (BG/SB/CC/SB1962/ISO/NOI)

{
  "book_title": "Śrīmad-Bhāgavatam",
  "canto": 1,
  "chapter": 1,
  "verse": 1,
  "verse_end": null,
  "reference": "SB 1.1.1",
  "devanagari": "ॐ नमो भगवते वासुदेवाय\nजन्माद्यस्य यतो ...",
  "bengali": null,
  "transliteration": "om namo bhagavate vāsudevāya\njanmādy asya yato ...",
  "synonyms": [
    { "word": "om", "meanings": ["O my Lord"] },
    { "word": "namaḥ", "meanings": ["offering my obeisances"] }
  ],
  "translation": "<b>I offer my obeisances unto Lord Śrī Kṛṣṇa...</b>",
  "purport": ["paragraph 1...", "paragraph 2..."]
}

Prose Chapter JSON Schema

{
  "book": "ek",
  "chapter": 1,
  "title": "Choosing Human and Animal Lives",
  "type": "chapter",
  "paragraphs": ["<b>heading</b>", "body text...", "<div style=\"text-align:center\"><i>verse</i></div>"]
}

Intro Segment JSON Schema

{
  "slug": "preface",
  "title": "Preface",
  "type": "intro-segment",
  "book": "ek",
  "order": 2,
  "paragraphs": ["paragraph 1...", "paragraph 2..."]
}

Essay JSON Schema

{
  "slug": "krishna-reservoir-of-pleasure",
  "title": "Krishna The Reservoir Of Pleasure",
  "author": "His Divine Grace A.C. Bhaktivedanta Swami Prabhupāda",
  "year": 1966,
  "group": "two-essays",
  "order": 1,
  "paragraphs": ["paragraph 1...", "paragraph 2..."]
}

Glossary JSON Schema

{
  "title": "Glossary of Vedic Terms",
  "description": "A comprehensive glossary...",
  "totalEntries": 2749,
  "letters": ["A", "B", "C", ...],
  "entries": [
    {
      "term": "Ācārya",
      "definition": "a spiritual master who teaches by his own example...",
      "definition_html": "a spiritual master who teaches by his own example..."
    }
  ]
}

Known Issues

  1. MongoDB BSON wire corruption: The MongoDB server occasionally corrupts large BSON documents. Workaround: seed using insertOne + $push for paragraphs one at a time.

  2. SB verse numbering: 5 duplicate verse reference tags in the HTML. Fixed by using the TEXT N header as authoritative.

  3. RTF routes gated: /rtfs only accessible when ALLOW_RTF=true.

  4. scripts/ excluded from tsconfig: Prevents TypeScript build errors from seed scripts.

  5. CSS zoom for font scaling: zoom property works in Chrome/Safari/Edge but may not work in older Firefox (< 126). A transform: scale() fallback is provided via @supports not (zoom: 1.1).

  6. Password protection requires ACCESS_KEY at runtime: The PasswordGate component checks the server at runtime via /api/verify-access. The env var must be available when the server is running, not just at build time.

Reference in New Issue View Git Blame Copy Permalink