diff --git a/PROJECT_CONTEXT.md b/PROJECT_CONTEXT.md index 41e5aa8f..84254f83 100644 --- a/PROJECT_CONTEXT.md +++ b/PROJECT_CONTEXT.md @@ -2,9 +2,9 @@ ## What this project is -A Next.js app that converts legacy RTF files containing Vedic scriptures and teachings by His Divine Grace A.C. Bhaktivedanta Swami Prabhupāda into structured, browsable content with proper Unicode Sanskrit. +A Next.js app presenting the books, letters, essays, lectures, and teachings of His Divine Grace A.C. Bhaktivedanta Swami Prabhupāda in a modern, accessible, and searchable format with proper Unicode Sanskrit (IAST + Devanagari). -The original RTF files use **custom font encodings** (Indevr, ScaGoudy, Inbenr/Inbeni, RM Devanagari, Balaram, etc.) where ASCII characters map to Devanagari/Bengali glyphs and IAST diacritical marks. The extraction pipeline decodes these into proper Unicode, generates Devanagari from IAST, and stores everything in MongoDB for serving via a Next.js frontend. +Content is sourced from the pre-1980s Vedabase, extracted and indexed into MongoDB, and served via a Next.js frontend. Extraction scripts and raw data live separately in the `extractor/` directory. --- @@ -24,186 +24,96 @@ The original RTF files use **custom font encodings** (Indevr, ScaGoudy, Inbenr/I ## Project Structure +The project is split into two directories: + +- **`knowledgebase/`** — Production Next.js app (~3.5MB excluding node_modules) +- **`extractor/`** — Extraction scripts, seed scripts, raw data, legacy fonts (~750MB) + ``` -spsays-training/ +knowledgebase/ ├── app/ -│ ├── page.tsx → Landing page with Logo, links to Books/Essays/Journals/Glossary/Other Documents/RTFs -│ ├── layout.tsx → ThemeProvider + TooltipProvider + PasswordGate wrapper +│ ├── page.tsx → Landing page (Books, Essays, Magazines, Letters, Transcriptions, etc.) +│ ├── layout.tsx → ThemeProvider + TooltipProvider + TextSelectionToolbar + TextHighlighter │ ├── globals.css → Vintage color palette + reading settings CSS -│ ├── books/ +│ ├── about/page.tsx → About page (mission, data source, roadmap) +│ ├── books/ → 37 book routes (verse-based + prose) │ │ ├── page.tsx → Books index (searchable grid, diacritic-aware search) -│ │ ├── books-grid.tsx → Client component for searchable book grid -│ │ ├── chapter-viewer.tsx → Shared client component for verse-based books (BG/SB/CC/SB1962) -│ │ ├── extracted-verse-count.tsx → (i) tooltip for verse counts -│ │ ├── bg/ → Bhagavad-gītā (18 chapters, 653 verses) -│ │ ├── sb/ → Śrīmad-Bhāgavatam (10 cantos, 8258 verses) -│ │ ├── cc/ → Caitanya-caritāmṛta (3 līlās, 11348 verses) -│ │ ├── sb1962/ → SB First Canto 1962 Edition (19 chapters, 762 verses) -│ │ ├── kb/ → Kṛṣṇa Book (89 chapters, 2 volumes) -│ │ ├── nod/ → The Nectar of Devotion (51 chapters, 4 parts) -│ │ ├── tlc/ → Teachings of Lord Caitanya (32 chapters, 2 parts) -│ │ ├── iso/ → Śrī Īśopaniṣad (18 mantras + invocation) -│ │ ├── noi/ → The Nectar of Instruction (11 texts) -│ │ ├── ej/ → Easy Journey to Other Planets (2 chapters) -│ │ ├── ek/ → Elevation to Kṛṣṇa Consciousness (6 chapters) -│ │ ├── mg/ → Kṛṣṇa Consciousness: The Matchless Gift (7 chapters) -│ │ ├── bbd/ → Beyond Birth and Death (5 chapters) -│ │ ├── owk/ → On the Way to Kṛṣṇa (5 chapters) -│ │ ├── rv/ → Rāja-vidyā: The King of Knowledge (8 chapters) -│ │ ├── poy/ → The Perfection of Yoga (11 chapters) -│ │ ├── pqpa/ → Perfect Questions Perfect Answers (9 chapters) -│ │ ├── ttp/ → Transcendental Teachings of Prahlāda Mahārāja (5 chapters) -│ │ ├── ochkm/ → On Chanting the Hare Kṛṣṇa Mantra (4 chapters) -│ │ ├── hkh/ → The Kṛṣṇa Consciousness Handbook (19 chapters) -│ │ ├── ssr/ → The Science of Self-Realization (8 chapters, 50 sub-chapters) -│ │ ├── ds/ → Dialectic Spiritualism (13 chapters, 36 sub-chapters) -│ │ └── va/ → Vaiśiṣṭyāṣṭakam (8 chapters) -│ ├── magazines/ -│ │ ├── page.tsx → Magazines index page (grouped by flag: "Pre 1978") -│ │ └── [slug]/ -│ │ ├── page.tsx → Magazine issue listing (grouped by volume) -│ │ └── [issue]/page.tsx → Individual issue reader with sidebar +│ │ ├── chapter-viewer.tsx → Shared client component for verse-based books +│ │ ├── bg/, sb/, cc/, sb1962/, iso/, noi/, bs/, mm/, nbs/ → Verse-based books +│ │ ├── kb/, nod/, tlc/, ej/, ek/, mg/, bbd/, owk/, rv/ → Prose books +│ │ ├── poy/, pqpa/, ttp/, ochkm/, hkh/, lcl/, lob/ → More prose books +│ │ ├── ssr/, ds/ → Hierarchical prose books (sub-chapters) +│ │ └── sva/, gg/, mog/, cbhc/, bk/, pk/, va/ → Other books +│ ├── magazines/ → Magazine browsing (BTG 1944-1960, BTG 1966-1973) │ ├── letters/ │ │ ├── page.tsx → Letters index with 3-way nav (year/recipient/location) -│ │ ├── letters-nav.tsx → Client component for tab navigation -│ │ ├── letters-browser.tsx → Client component with search + pagination -│ │ ├── [slug]/page.tsx → Individual letter reader with prev/next -│ │ ├── year/[year]/page.tsx → Letters filtered by year -│ │ ├── recipient/[name]/page.tsx → Letters filtered by recipient -│ │ └── location/[name]/page.tsx → Letters filtered by location -│ ├── essays/ -│ │ ├── page.tsx → Essays grid page (searchable) -│ │ ├── essays-grid.tsx → Client component for searchable essay grid -│ │ └── [slug]/page.tsx → Individual essay reader -│ ├── journals/ -│ │ ├── page.tsx → Journals index page (grid) -│ │ └── [slug]/ -│ │ ├── page.tsx → Journal chapter listing (hierarchical: months → days) -│ │ └── [chapter]/page.tsx → Individual journal entry reader -│ ├── other-documents/ -│ │ ├── page.tsx → Other documents grid page (searchable) -│ │ ├── other-documents-grid.tsx → Client component for searchable grid -│ │ └── [slug]/ -│ │ ├── page.tsx → Document chapter listing -│ │ └── [chapter]/page.tsx → Individual chapter reader -│ ├── glossary/ -│ │ ├── page.tsx → Server component wrapper -│ │ └── glossary-client.tsx → Searchable client component (2749 entries) -│ ├── rtfs/ -│ │ ├── page.tsx → RTF file listing (gated by ALLOW_RTF env) -│ │ └── [slug]/page.tsx → RTF iframe viewer +│ │ ├── filter-year-browser.tsx → Shared component: sidebar years + month-grouped content +│ │ ├── [slug]/page.tsx → Individual letter reader +│ │ ├── year/[year]/ +│ │ │ ├── page.tsx → Letters by year +│ │ │ └── year-month-browser.tsx → Sidebar months + letter list +│ │ ├── recipient/[name]/page.tsx → Letters by recipient (year sidebar + month groups) +│ │ └── location/[name]/page.tsx → Letters by location (year sidebar + month groups) +│ ├── essays/ → 23 essays across 6 groups +│ ├── journals/ → Jaladuta Diary, The Beginning +│ ├── other-documents/ → 12 miscellaneous documents +│ ├── glossary/ → 2,749 searchable Sanskrit terms +│ ├── disciples/ → 4,747 initiated disciples directory +│ │ ├── page.tsx → Disciples browser (letter nav, suffix filter, search, sort, pagination) +│ │ └── disciples-browser.tsx → Client component +│ ├── transcriptions/page.tsx → Coming Soon page (lectures & conversations) │ └── api/ -│ ├── books/ -│ │ ├── route.ts → GET all book metas -│ │ └── [book]/ -│ │ ├── route.ts → GET single book meta -│ │ └── verses/route.ts → GET verses (?chapter=&verse=&division=&indexOnly=&page=&limit=) -│ ├── glossary/route.ts → GET all glossary entries -│ └── verify-access/route.ts → GET (check if password enabled) + POST (validate key) +│ ├── og/route.tsx → Dynamic OG image generator (Noto Sans, Prabhupāda image) +│ ├── books/ → Book/verse API routes +│ └── glossary/route.ts → Glossary API ├── components/ │ ├── ui/ → shadcn components (button, card, badge, tooltip, input, drawer) -│ ├── page-header.tsx → Shared header with ReadingSettings -│ ├── reading-settings.tsx → Theme/font/size/justify accessibility settings panel +│ ├── selection-toolbar/ → Text selection toolbar sub-components +│ │ ├── shared.tsx → KautilyaIcon, types, hooks (useMediaQuery, useCopyFeedback) +│ │ ├── floating-toolbar.tsx → Desktop floating pill / mobile bottom bar +│ │ ├── share-modal.tsx → Share modal (desktop) / drawer (mobile) +│ │ ├── collections-drawer.tsx → Save to collection drawer +│ │ └── more-popover.tsx → More options popover (desktop) / drawer (mobile) +│ ├── text-selection-toolbar.tsx → Main orchestrator (event listeners, state management) +│ ├── text-highlighter.tsx → Processes incoming ?hl= URLs (scroll + highlight) +│ ├── page-header.tsx → Shared header with ReadingSettings (max-w-5xl) +│ ├── reading-settings.tsx → Theme/font/size/justify accessibility panel │ ├── mobile-nav.tsx → Left-slide drawer for mobile sidebar navigation -│ ├── logo.tsx → Theme-aware logo component (light/dark variants) -│ ├── password-gate.tsx → Client-side password protection (runtime check) +│ ├── site-footer.tsx → Global footer (About + Home links) +│ ├── logo.tsx → Theme-aware logo component │ └── theme-provider.tsx → next-themes wrapper ├── lib/ │ ├── db.ts → Prisma client singleton -│ ├── api.ts → Data access layer (getAllBooks, getBookMeta, getChapterVerses, etc.) +│ ├── api.ts → Core data access (books, verses, prose chapters, glossary) +│ ├── constants.ts → Shared constants (BOOK_NAMES, CC_LILAS) +│ ├── og.ts → OG image metadata builders (ogForVerse, ogForLetter, etc.) +│ ├── text-highlight.ts → Text fragment encode/decode, highlight matching +│ ├── source-reference.ts → Source reference from URL (for copy attribution) +│ ├── disciples.ts → Disciples data access (by letter, place, year, search) +│ ├── letters.ts → Letters data access (by year, recipient, location) │ ├── utils.ts → cn() helper -│ ├── nod.ts → NoD data access (Prisma) -│ ├── tlc.ts → TLC data access (Prisma) -│ ├── iso.ts → Īśopaniṣad data access (Prisma) -│ ├── noi.ts → NoI data access (Prisma) -│ ├── ej.ts → Easy Journey data access (Prisma) -│ ├── ek.ts → Elevation to KC data access (Prisma) -│ ├── mg.ts → Matchless Gift data access (Prisma) -│ ├── bbd.ts → Beyond Birth and Death data access (Prisma) -│ ├── owk.ts → On the Way to Kṛṣṇa data access (Prisma) -│ ├── rv.ts → Rāja-vidyā data access (Prisma) -│ ├── poy.ts → Perfection of Yoga data access (Prisma) -│ ├── pqpa.ts → Perfect Questions data access (Prisma) -│ ├── ttp.ts → Prahlāda Mahārāja data access (Prisma) -│ ├── ochkm.ts → On Chanting data access (Prisma) -│ ├── hkh.ts → KC Handbook data access (Prisma) -│ ├── ssr.ts → Science of Self-Realization data access (Prisma, hierarchical) -│ ├── ds.ts → Dialectic Spiritualism data access (Prisma, hierarchical) -│ ├── te.ts → Two Essays data access (Prisma, uses essays collection) -│ ├── va.ts → Vaiśiṣṭyāṣṭakam data access (Prisma) -│ ├── other-documents.ts → Other Documents data access (Prisma, dedicated collections) -│ ├── journals.ts → Journals data access (Prisma, dedicated collections) -│ ├── magazines.ts → Magazines data access (Prisma, dedicated collections) -│ ├── letters.ts → Letters data access (Prisma, queries by year/recipient/location) -│ └── jsons/books/ → Extracted JSON data (all books) -│ ├── bg/, sb/, cc/, sb-1962/, kb/ → Original verse/prose books -│ ├── nod/, tlc/, iso/, noi/ → Extracted from other.html -│ ├── ej/, ek/, mg/, bbd/, owk/ → Small prose books -│ ├── rv/, poy/, pqpa/, ttp/, ochkm/ → More prose books -│ ├── hkh/, ssr/, ds/ → Large structured books -│ ├── te/ → Two Essays -│ ├── va/ → Vaiśiṣṭyāṣṭakam -│ ├── glossary/index.json → 2,749 glossary entries -│ ├── essays/ → Essay JSON files (pre-1977, cc-essays, standalone, etc.) -│ ├── other-documents/ → Other document JSON files (12 documents) -│ ├── journals/ → Journal JSON files (2 journals, 300 chapters) -│ ├── magazines/ → Magazine JSON files (2 magazines, 59 issues) -│ │ ├── btg/ → Back to Godhead (1944–1960), 29 issues -│ │ └── btg-1978/ → Back to Godhead (1966–1973), 30 issues -│ └── letters/ → Letter JSON files (6,866 letters, grouped by year) -│ ├── index.json → Summary index (stats, top recipients/locations) -│ ├── 1947.json … 1977.json → 28 year-based letter files -│ └── undated.json → 58 undated letters +│ ├── [book].ts → Per-book data access layers (nod, tlc, iso, noi, ej, ek, etc.) +│ ├── other-documents.ts → Other Documents data access +│ ├── journals.ts → Journals data access +│ └── magazines.ts → Magazines data access ├── prisma/ -│ └── schema.prisma → Verse, BookMeta, ProseChapter, Essay, GlossaryEntry, OtherDocumentMeta, OtherDocumentChapter, JournalMeta, JournalChapter, MagazineMeta, MagazineIssue, Letter models -├── scripts/ -│ ├── bg_extract.py → BG extraction from bg.html -│ ├── sb_extract.py → SB extraction from sb.html -│ ├── cc_extract.py → CC extraction from cc.html -│ ├── sb1962_extract.py → 1962 SB extraction -│ ├── nod_extract.py → NoD extraction from other.html -│ ├── tlc_extract.py → TLC extraction from other.html -│ ├── iso_extract.py → Īśopaniṣad extraction from other.html -│ ├── noi_extract.py → NoI extraction from other.html -│ ├── ej_extract.py → Easy Journey extraction -│ ├── ek_extract.py → Elevation to KC extraction -│ ├── mg_extract.py → Matchless Gift extraction -│ ├── bbd_extract.py → Beyond Birth and Death extraction -│ ├── owk_extract.py → On the Way to Kṛṣṇa extraction -│ ├── rv_extract.py → Rāja-vidyā extraction -│ ├── poy_extract.py → Perfection of Yoga extraction -│ ├── pqpa_extract.py → Perfect Questions extraction -│ ├── ttp_extract.py → Prahlāda Mahārāja extraction -│ ├── ochkm_extract.py → On Chanting extraction -│ ├── hkh_extract.py → KC Handbook extraction -│ ├── ssr_extract.py → Science of Self-Realization extraction (hierarchical sub-chapters) -│ ├── ds_extract.py → Dialectic Spiritualism extraction (hierarchical sub-chapters) -│ ├── te_extract.py → Two Essays extraction -│ ├── glossary_extract.py → Glossary extraction -│ ├── seed-db.ts → Main MongoDB seeder (BG, SB, CC, SB1962, KB, glossary) -│ ├── seed-nod.ts … seed-ds.ts → Individual book seeders -│ ├── seed-te.ts → Two Essays seeder (uses essays collection) -│ ├── seed-va.ts → Vaiśiṣṭyāṣṭakam seeder -│ ├── seed-pre-1977-essays.ts → Pre-1977 essays seeder (13 essays) -│ ├── seed-other-documents.ts → Other documents seeder (dedicated collections) -│ ├── seed-journals.ts → Journals seeder (dedicated collections) -│ ├── jaladuta_diary_extract.py → Jaladuta Diary extraction (83 day entries) -│ ├── the_beginning_extract.py → The Beginning extraction (189 day entries + 20 excerpts) -│ ├── btg_extract.py → BTG (1944–1960) extraction from btg-1960.html -│ ├── btg_1978_extract.py → BTG (1966–1973) extraction from btg-1978.html -│ ├── seed-magazines.ts → Magazines seeder (dedicated collections) -│ ├── letters_extract.py → Letters extraction from letters.html (6,866 letters) -│ ├── seed-letters.ts → Letters seeder (per-document inserts) -│ ├── kb_extract_*.py → KB extraction scripts (intro, vol1, vol2) -│ ├── inject-fonts.py → Inject @font-face rules into converted HTML -│ └── fix-bengali-encoding.py → Fix Bengali font encoding in cc.html +│ └── schema.prisma → All models (Verse, BookMeta, ProseChapter, Essay, Letter, +│ GlossaryEntry, Disciple, Transcription, etc.) └── public/ - ├── rtfs/ → Original RTF files (~430MB, 17 files) - ├── html/ → HTML converted from RTF (textutil + @font-face injection) - ├── fonts/ → Custom TTF fonts (Balaram, ScaGoudy, Indevr, Inbenr, RM Devanagari, etc.) - ├── logo_dark.png → Logo for light theme (with "says.org") - ├── logo_light.png → Logo for dark theme (subtle script) - └── robots.txt → Disallow all crawlers + ├── fonts/Noto_Sans/static/ → NotoSans-Regular.ttf + NotoSans-Bold.ttf (OG images) + ├── images/ → OG card image (Prabhupāda) + ├── logo_dark.png → Logo for light theme + ├── logo_light.png → Logo for dark theme + └── robots.txt + +extractor/ → Extraction & seeding tools (not deployed) +├── scripts/ → Python extractors + TypeScript seed scripts +├── jsons/ → Extracted JSON data (books, letters, essays, etc.) +├── html/ → RTF→HTML converted files +├── rtfs/ → Original RTF source files +├── fonts/ → Legacy custom fonts (ScaGoudy, Indevr, Balaram, etc.) +├── app-rtfs/ → RTF viewer pages (archived) +└── .venv/ → Python virtual environment ``` --- @@ -743,12 +653,38 @@ SSR and DS have parent chapters with sub-chapters stored as individual `prose_ch - **Light mode ("Aged Parchment")**: Warm cream backgrounds, deep sepia text, saffron/ochre primary - **Dark mode ("Ancient Manuscript")**: Deep warm brown backgrounds, parchment text, golden saffron primary -### Password Protection -- Client-side gate component (`PasswordGate`) wrapping the entire app -- Runtime check via `GET /api/verify-access` (checks if `ACCESS_KEY` env var is set) -- Validation via `POST /api/verify-access` (compares submitted key) -- Stores valid key in `sessionStorage` -- Bypassed when `ACCESS_KEY` is not set +### Text Selection Toolbar +- Global floating toolbar appears on text selection across all reading pages (except RTFs) +- **Desktop**: floating pill above selection; **Mobile**: fixed bottom bar +- Actions: Ask Kautilya (Phase II), Share, Copy (with source reference), More +- **Share**: modal (desktop) / bottom drawer (mobile) — share to feed, copy highlight link +- **More**: popover (desktop) / bottom drawer (mobile) — bookmark, save to collection, copy link, mention +- **Copy** appends source reference: title, chapter/verse, highlight URL +- All toolbar surfaces use dark contrast treatment (bg-black, border-white/10) +- Kautilya lotus icon for the AI companion button + +### Text Highlight Links +- Selected text encoded as compact URL fragment: `?hl=prefix§suffix§length` +- Uses `§` delimiter (safe for Sanskrit, English, and all Vedic text) +- Short selections (≤8 words) use full text as prefix +- Incoming `?hl=` URLs: text is found in DOM, wrapped in ``, scrolled to, highlighted permanently +- `TextHighlighter` component runs on page load via `useSearchParams` +- Server-side `extractMatchedText()` searches paragraphs for OG image quote extraction + +### Dynamic OG Images +- `/api/og` route generates 1200×630 PNG images using `next/og` (Satori) +- Left panel: Prabhupāda image with quote mark and branding (from PNG) +- Right panel: title (bold), detail, quote text, "Read more at prabhupadasays.org" +- Font: Noto Sans (Regular + Bold TTF from `public/fonts/Noto_Sans/static/`) +- Full IAST diacritical support (ā, ī, ū, ṛ, ṣ, ṇ, ṭ, ḍ, ś, ñ, ṅ, ḥ, ṁ) +- Every content page has `generateMetadata` with contextual fallback descriptions +- When `?hl=` is present, the actual selected text is extracted from DB content for the OG quote + +### Site Footer +- Global footer on every page with © year, About link, Home link + +### Consistent Container Width +- All pages use `max-w-5xl` for both header and body containers --- @@ -756,8 +692,7 @@ SSR and DS have parent chapters with sub-chapters stored as individual `prose_ch ``` DATABASE_URL=mongodb://user:pass@host:port/spsays?authSource=admin -ALLOW_RTF=true # optional — enables RTF viewer routes -ACCESS_KEY="secret" # optional — enables password protection (quote if contains #) +NEXT_PUBLIC_SITE_URL=https://prabhupadasays.org # used for OG image URLs ``` --- @@ -765,31 +700,16 @@ ACCESS_KEY="secret" # optional — enables password protection (quote if co ## How to Run ```bash -# Development +# Development (from knowledgebase/) npm run dev # Prisma npx prisma generate # Regenerate Prisma client after schema changes -# Extraction scripts (Python 3 + beautifulsoup4 + indic-transliteration) -python3 scripts/bg_extract.py # Extract BG -python3 scripts/sb_extract.py # Extract SB -python3 scripts/cc_extract.py # Extract CC -python3 scripts/nod_extract.py # Extract NoD from other.html -python3 scripts/iso_extract.py # Extract Īśopaniṣad from other.html -python3 scripts/ssr_extract.py # Extract SSR from other.html -python3 scripts/ds_extract.py # Extract DS from other.html -# ... (similar for all other books) - -# Database seeding -npx tsx scripts/seed-db.ts # Seed BG, SB, CC, SB1962, KB, glossary -npx tsx scripts/seed-nod.ts # Seed NoD -npx tsx scripts/seed-iso.ts # Seed Īśopaniṣad -npx tsx scripts/seed-te.ts # Seed Two Essays -# ... (similar for all other books) - # Build npx next build + +# Extraction & seeding scripts are in extractor/ — see extractor/scripts/ ``` ---