kulnoor/drive
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
gosangam
2026-10-18 23:48:22 +05:30
commit 732a06f65d
38 changed files with 8943 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+510
View File
@@ -0,0 +1,510 @@
# Telegram Channel → Google Drive Organizer
[![Deploy to Heroku](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy?template=https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME)
> **Before the button works:** replace `https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME`
> in the line above with the URL of the **public** GitHub repository that contains this project
> (see [Publishing to GitHub](#publishing-to-github)). That URL is the only value to change.
> The button flow has not been tested against a live repository yet. Test it once after publishing.
A long-running worker that reads video posts from a Telegram channel, downloads each video, and
uploads it to Google Drive. The caption's **Title** becomes the file name and its **Topic** picks the
Drive folder. Videos are processed one at a time. All progress is stored in a database, so the bot
resumes after crashes, restarts and redeploys without uploading anything twice.
## Contents
1. [Overview](#1-overview) · 2. [Features](#2-features) · 3. [Caption format](#3-caption-format-and-metadata-extraction) ·
4. [Drive organization](#4-google-drive-topic-folder-organization) · 5. [Prerequisites](#5-prerequisites) ·
6. [One-click Heroku deployment](#6-one-click-heroku-deployment) · 7. [Telegram setup](#7-telegram-bot-setup) ·
8. [Historical access](#8-historical-channel-access-requirements) · 9. [Google Drive auth](#9-google-drive-authentication) ·
10. [Config Vars](#10-required-heroku-config-vars) · 11. [Database](#11-database-provisioning-and-configuration) ·
12. [First deployment](#12-first-deployment-and-startup) · 13. [Commands](#13-telegram-commands) · 14. [Logs](#14-checking-logs) ·
15. [Restart / redeploy](#15-restarting-and-redeploying) · 16. [Troubleshooting](#16-troubleshooting) ·
17. [Security](#17-security-and-credential-handling) · 18. [Limitations](#18-known-limitations-and-heroku-resource-constraints) ·
19. [Local development](#19-local-development)
---
## 1. Overview
```
Telegram channel ──(Telethon / MTProto)──► queue worker (one video at a time) ──► Google Drive
│ (resumable upload,
Admin chat ◄── bot commands & progress ─────────┤ verified by size + MD5)
▼
PostgreSQL (Heroku) / SQLite (local)
```
For each video: **discover → parse caption → resolve the topic folder → download to a temp disk →
upload (resumable) → verify size and MD5 in Drive → save the Drive file ID → mark COMPLETED → delete
the temp file → next video.**
## 2. Features
- Caption parsing for `Index` (one or many), `Title`, `Topic`, `Batch` and `Extracted By`.
- Topic folders are created once and reused. Optional `Batch/Topic` layout.
- Files are never overwritten. Name collisions get `(2)`, `(3)`, … or `[message id]`, or are skipped or failed, per your policy.
- No duplicate uploads, even if the process dies just after Drive accepted the file. Every upload carries a hidden `appProperties` tag (`channel:message`) that is checked before uploading.
- Resumable uploads that survive restarts. The upload session URI is stored in the database.
- Resumable downloads, continuing from the last 1 MiB boundary of a partial file.
- Explicit state machine: `PENDING → DOWNLOADING → DOWNLOADED → UPLOADING → COMPLETED`, plus `RETRY_WAIT`, `FAILED` and `SKIPPED`. Every transition is validated and logged in the `transitions` table.
- Bounded retries with exponential backoff, honouring Telegram flood waits and Drive rate limits. Permission errors are not retried.
- Disk-space checks before each download. A full disk, Drive quota or expired auth halts the queue cleanly instead of burning retries.
- Admin-only Telegram commands, plus one progress message that is edited in place rather than spammed.
- Graceful shutdown on `SIGTERM` (Heroku restarts) at a resumable checkpoint.
- A single-worker guard (PostgreSQL advisory lock) prevents two dynos from processing the same queue.
## 3. Caption format and metadata extraction
```
Index: 001
Index: 002
Title: Class 01 Aman sir.mp4
Topic: Misc• Grammar Practice
Batch: English ( SSC MAINS )
Extracted By: @PUTINxINDIA
```
| Field | Use |
|---|---|
| `Title` | Drive file name (`Class 01 Aman sir.mp4`) |
| `Topic` | Drive folder (`Misc• Grammar Practice`) |
| `Index` | All values kept in order (`["001", "002"]`); never used as the file name. The Telegram message ID is the unique identity. |
| `Batch` | Stored; optional `Batch/Topic` folders |
| `Extracted By` | Stored in the record |
Parsing rules:
- Fields can appear in any order.
- Labels are case-insensitive.
- Blank lines, emoji or bullet prefixes, extra spaces and full-width colons are allowed.
- Unicode text is preserved.
- Parsing is line-based, so one field never swallows the next.
- A flattened single-line caption is split at each known label.
Missing data never makes a video silently disappear:
| Case | Behaviour |
|---|---|
| No Topic | Folder `Uncategorized` (`FALLBACK_TOPIC_FOLDER`) |
| No Title | Telegram file name, else `telegram_<message id>.<ext>`. A warning is recorded. `MISSING_TITLE_POLICY=fail` marks it FAILED instead. |
| Title without extension | Extension taken from the real file's magic bytes, else from the MIME type |
| Title extension ≠ real container | Kept as written, with a warning. `EXTENSION_MISMATCH_POLICY=append` appends the real one. |
| `/` `\` or control characters | Replaced (`STRICT_FILENAMES=true` also replaces `<>:"\|?*`) |
## 4. Google Drive topic-folder organization
```
<GOOGLE_DRIVE_ROOT_FOLDER_ID>/
├── Misc• Grammar Practice/
│ ├── Class 01 Aman sir.mp4
│ └── Class 02 Aman sir.mp4
├── Error Detection/
│ └── Class 01 Aman sir.mp4
└── Uncategorized/
```
Folder lookup goes database cache → Drive search → create. If duplicate folders already exist, the
oldest is reused. If a folder-create response is lost, the bot searches again before retrying, so
it never creates a second copy. Set `BATCH_SUBFOLDERS_ENABLED=true` for `Root/Batch/Topic/`.
**Duplicate policy (`DUPLICATE_POLICY`)** applies when a *different* file already has the target
name in the folder:
| Value | Result |
|---|---|
| `suffix_number` (default) | `Class 01 Aman sir (2).mp4` |
| `suffix_message_id` | `Class 01 Aman sir [1234].mp4` |
| `skip` | Item marked SKIPPED with the reason |
| `fail` | Item marked FAILED |
An existing file is reused, not re-uploaded, only when its identity is verified. That means either
its hidden tag matches this Telegram message, or it has the same size and MD5
(`REUSE_IDENTICAL_FILES=true`). The same Telegram media re-posted under a new message ID is SKIPPED
by default (`DUPLICATE_MEDIA_POLICY`).
## 5. Prerequisites
- A Telegram account, plus API credentials from <https://my.telegram.org>.
- A Telegram bot from [@BotFather](https://t.me/BotFather).
- A Google account with a Google Cloud project (free) with the Drive API enabled.
- **A local computer with Python 3.12** for two one-time interactive logins. Telegram and Google logins need a human, so they can't run on Heroku.
- A Heroku account with billing enabled. The worker dyno and Heroku Postgres are paid resources; check current pricing on Heroku.
- Optional: the [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) and the [GitHub CLI](https://cli.github.com/).
## 6. One-click Heroku deployment
The button (top of this page) opens Heroku's deploy form. It:
- creates the app from this repository's `app.json`;
- builds it with the official Python buildpack;
- provisions **Heroku Postgres (`essential-0`)**, which sets `DATABASE_URL`;
- starts **one `basic` worker dyno** (`worker: python main.py run`); there is no web process.
**It is not fully one-click.** Two secrets must be created on your computer first, because they need
interactive logins:
1. `TELEGRAM_SESSION_STRING`, only for `TELEGRAM_MODE=user` (see [§7](#7-telegram-bot-setup)).
2. `GOOGLE_OAUTH_TOKEN_JSON` or `GOOGLE_SERVICE_ACCOUNT_JSON` (see [§9](#9-google-drive-authentication)).
Prepare those, click the button, fill the form, then press **Deploy app**. If add-on provisioning
fails (for example because billing isn't verified or the plan isn't available), create the app
anyway and follow [§11](#11-database-provisioning-and-configuration) manually. The worker refuses
to start on Heroku without a database rather than silently losing progress.
Manual alternative with the CLI (from the repository root):
```bash
heroku create YOUR_APP_NAME --stack heroku-24
heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME
heroku config:set TELEGRAM_API_ID=1234567 TELEGRAM_API_HASH=your_api_hash \
TELEGRAM_BOT_TOKEN=123456:your-bot-token ADMIN_USER_IDS=111111111 \
SOURCE_CHANNEL_ID=@your_channel --app YOUR_APP_NAME
heroku config:set TELEGRAM_SESSION_STRING="$(cat data/sessions/session_string.txt)" --app YOUR_APP_NAME
heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME
git push heroku main
heroku ps:scale worker=1 --app YOUR_APP_NAME
```
### Publishing to GitHub
This directory (the one containing `Procfile` and `app.json`) must be the **repository root**.
```bash
cd telegram_drive_bot
git init -b main
git add .
git status # confirm: no .env, credentials/, data/, *.session, downloads/
git commit -m "Telegram to Google Drive organizer"
gh repo create YOUR_REPOSITORY_NAME --public --source=. --push
# or create the repo on github.com, then:
# git remote add origin https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME.git && git push -u origin main
```
Then put that repository URL into the button link at the top of this README, commit, and push.
The Heroku button only works with **public** repositories.
## 7. Telegram bot setup
1. **API ID / hash:** log in at <https://my.telegram.org> → *API development tools* → create an app.
Copy `api_id` → `TELEGRAM_API_ID` and `api_hash` → `TELEGRAM_API_HASH`.
These are needed in both modes, because the bot talks MTProto through Telethon.
2. **Bot:** talk to @BotFather → `/newbot` → copy the token → `TELEGRAM_BOT_TOKEN`.
**Open your bot and press Start.** A bot can't message you until you do, so you won't get progress messages otherwise.
3. **Admin IDs:** message [@userinfobot](https://t.me/userinfobot) to get your numeric ID → `ADMIN_USER_IDS`
(comma-separate several). Only these users can run commands.
4. **Source channel (`SOURCE_CHANNEL_ID`):** any of these forms work:
- a public channel: `@channelname` or `https://t.me/channelname`;
- a private channel: the numeric ID `-100…`, or a post link `https://t.me/c/1234567890/55` (use *Copy Post Link* in Telegram);
- forwarding a post to @userinfobot also shows the channel ID.
5. **Choose a mode (`TELEGRAM_MODE`):**
- `user` (default): reads the channel as *your* account, which must be a member. Create the session string locally:
```bash
python main.py login --session-string
```
Telethon asks for your phone number, the login code and, only if enabled, your 2FA password.
None of these are stored. The resulting session string is written to
`data/sessions/session_string.txt` (mode 600). Copy it into `TELEGRAM_SESSION_STRING`, then delete the file.
- `bot`: add the bot to the channel as an **administrator**. No session string is needed.
## 8. Historical channel access requirements
| Capability | `user` mode | `bot` mode |
|---|---|---|
| New posts (live) | ✅ | ✅ (bot must be a channel admin) |
| Existing history | ✅ full history the account can see | ⚠️ best effort: bots can't list history, so message IDs are probed in batches until `BOT_SCAN_MAX_EMPTY` consecutive gaps (or up to `BOT_SCAN_END_ID`) |
| Large files | ✅ up to Telegram's file limit | ✅ uses MTProto, so the HTTP Bot API's 20 MB download limit does **not** apply |
| Private channel by numeric ID | ✅ | ⚠️ the bot learns the channel's access key from its first live post; the key is then stored in the database |
| Extra secret needed | `TELEGRAM_SESSION_STRING` | none |
`PROCESS_MODE` controls what is queued:
- `existing`: history only (from `START_MESSAGE_ID` / `START_DATE`).
- `new`: only posts after the first start.
- `both` (default).
The bot never deletes or edits Telegram messages and doesn't bypass channel restrictions. A video
in a channel with content protection (no forwarding/saving) may not be downloadable.
## 9. Google Drive authentication
Pick **one**.
### Option A — OAuth (personal Google Drive, recommended)
1. In the [Google Cloud Console](https://console.cloud.google.com/): create a project, then enable the **Google Drive API**.
2. Configure the **OAuth consent screen**: *External*, add yourself as a test user.
Then set the publishing status to **In production**. In *Testing* status, Google expires refresh
tokens after 7 days and the bot would lose Drive access weekly. With the default `drive.file`
scope no verification is required. With the full `drive` scope Google shows an "unverified app"
screen during your own login (*Advanced → continue*).
3. Under *Credentials*, create an **OAuth client ID** of type **Desktop app**. Download the JSON to
`credentials/client_secret.json` in this project.
4. Locally: `python main.py drive-auth`. This opens a browser; use `--no-browser` to get a URL to open yourself.
It writes `credentials/token.json` (mode 600).
5. Heroku: `heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME`.
Multi-line JSON works through the CLI. In the dashboard you can paste raw JSON, or a single-line
base64 version: `base64 < credentials/token.json | tr -d '\n'`.
**Scope and root folder.** `GOOGLE_DRIVE_SCOPE=drive.file` (least privilege) lets the bot see only
files and folders *it created*. So either:
- let the bot create the root folder: `python main.py drive-init --name "Telegram Videos"` prints the ID for `GOOGLE_DRIVE_ROOT_FOLDER_ID`; or
- use a folder you created yourself: set `GOOGLE_DRIVE_SCOPE=drive` **before** running `drive-auth`.
The folder ID is the part after `/folders/` in its URL. The bot checks at startup that it can see
the folder and stops with an explanation if it can't.
### Option B — Service account (Google Workspace Shared Drive)
A service account has **no storage quota in a personal My Drive**. Uploads into a folder that a
normal Gmail user merely shared with it fail with `storageQuotaExceeded`. Use it only with a
**Shared Drive**:
1. Create a service account, then create a JSON key for it.
2. In the Shared Drive, *Manage members* → add the service account's `client_email` as **Content manager**.
3. Set:
- `GOOGLE_AUTH_MODE=service_account`
- `GOOGLE_SERVICE_ACCOUNT_JSON="$(cat key.json)"`
- `DRIVE_SHARED_DRIVE_ID=<shared drive id>`
- `GOOGLE_DRIVE_ROOT_FOLDER_ID=<folder id inside it>`
- `GOOGLE_DRIVE_SCOPE=drive` (needed to use a folder the service account did not create).
Credentials from environment variables are parsed in memory. Nothing is written to the dyno's disk,
and they are validated by an authenticated Drive call at startup. Uploaded files are **private**
unless you set `DRIVE_SHARE_MODE=anyone_with_link`.
## 10. Required Heroku Config Vars
Set these in the dashboard (*Settings → Reveal Config Vars*) or with
`heroku config:set NAME=value --app YOUR_APP_NAME`.
| Variable | Required | Notes |
|---|---|---|
| `TELEGRAM_API_ID`, `TELEGRAM_API_HASH` | yes | my.telegram.org |
| `TELEGRAM_BOT_TOKEN` | yes | @BotFather; command interface (and channel access in bot mode) |
| `ADMIN_USER_IDS` | yes | numeric IDs, comma-separated |
| `SOURCE_CHANNEL_ID` | yes | `@name`, link, or `-100…` |
| `TELEGRAM_MODE` | yes (default `user`) | `user` or `bot` |
| `TELEGRAM_SESSION_STRING` | **when `TELEGRAM_MODE=user`** | from `main.py login --session-string` |
| `GOOGLE_AUTH_MODE` | yes (default `oauth`) | `oauth` or `service_account` |
| `GOOGLE_OAUTH_TOKEN_JSON` | **when `oauth`** | contents of `credentials/token.json` |
| `GOOGLE_SERVICE_ACCOUNT_JSON` | **when `service_account`** | key JSON |
| `GOOGLE_DRIVE_ROOT_FOLDER_ID` | recommended | empty = top level of My Drive |
| `GOOGLE_DRIVE_SCOPE` | no (`drive.file`) | must match the scope used in `drive-auth` |
| `DATABASE_URL` | **yes on Heroku** | set automatically by Heroku Postgres |
| `PROCESS_MODE` | no (`both`) | `existing` / `new` / `both` |
| `DUPLICATE_POLICY` | no (`suffix_number`) | see §4 |
| `BATCH_SUBFOLDERS_ENABLED` | no (`false`) | `Root/Batch/Topic` |
| `MAX_RETRIES` / `RETRY_DELAY` | no (`4` / `30`) | automatic retries / base delay in seconds |
| `PROGRESS_INTERVAL` | no (`10`) | seconds between progress-message edits |
| `DOWNLOAD_DIR` | no (`/tmp/telegram_drive_bot` in app.json) | temporary files |
| `UPLOAD_CHUNK_SIZE` | no (`8MB` in app.json) | multiple of 256KB; memory per chunk |
| `AUTO_START` | no (`false`) | start processing on every boot |
Step-by-step instructions for obtaining every value are in **[ENV_SETUP.md](ENV_SETUP.md)**. Every other supported variable is listed with an explanation in [`.env.example`](.env.example).
## 11. Database provisioning and configuration
Production state lives in **PostgreSQL**:
- video records with all caption metadata;
- states, attempts, errors, timestamps and Drive file IDs;
- upload session URIs;
- the folder map;
- discovery and completion cursors;
- duplicate-detection data;
- the transition history;
- the pause/run/stop choice.
- **Via the button:** `app.json` requests `heroku-postgresql:essential-0`, a paid plan.
- **Manually:** `heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME`
(or any plan, or any PostgreSQL reachable through `DATABASE_URL`).
- The schema is created and migrated automatically at startup. The version is tracked in the `kv` table.
- TLS (`sslmode=require`) is added automatically for remote hosts.
- **Without `DATABASE_URL` on Heroku the worker refuses to start.** SQLite on a dyno would be wiped
at every restart, and the bot would re-upload everything. To accept that risk explicitly, set
`ALLOW_EPHEMERAL_DATABASE=true`; a warning is logged at every start.
- Locally, without `DATABASE_URL`, SQLite at `data/bot.db` is used. This is for development only.
## 12. First deployment and startup
1. Deploy (button or CLI). Watch the logs: `heroku logs --tail --app YOUR_APP_NAME`.
You should see the database backend, `Google Drive credentials validated`, and `Source channel resolved`.
2. The bot messages each admin: *Bot online (PostgreSQL); send /sync then /run to start.*
3. Send `/sync` to discover channel videos, then `/run`. Watch `/progress`.
4. After `/run`, the choice is stored. After any dyno restart the bot syncs and resumes on its own.
`/pause` and `/stop` are also remembered.
## 13. Telegram commands
Send these to the bot in a **private chat**. Users not in `ADMIN_USER_IDS` are refused.
| Command | Effect |
|---|---|
| `/start`, `/help` | Instructions |
| `/status` | Queue state, the current video, live progress |
| `/progress` | Processed / pending / retrying / failed / skipped / remaining counts |
| `/sync` | Discover unseen messages and queue them (idempotent). `/sync all` rescans from `START_MESSAGE_ID`. |
| `/run` | Start processing |
| `/pause` | Stop the current transfer at the next chunk boundary (resumable) and wait |
| `/resume` | Continue |
| `/stop` | Checkpoint and end the worker until `/run` |
| `/retry` | Re-queue FAILED items with a fresh retry budget |
| `/failed` | Failed items with short error summaries |
| `/folders` | Folder count and uploads per topic |
| `/check ID` | Full record for one Telegram message ID |
Progress is a single message, edited every `PROGRESS_INTERVAL` seconds:
```
Processing video 12 of 150
Topic: Misc• Grammar Practice
Title: Class 01 Aman sir.mp4
Status: Uploading
Download: 100% (1.2 GB / 1.2 GB)
Upload: 45% (540.0 MB / 1.2 GB) · 18.3 MB/s · ETA 36s
```
## 14. Checking logs
```bash
heroku logs --tail --app YOUR_APP_NAME # live
heroku logs -n 1500 --app YOUR_APP_NAME # recent history
heroku ps --app YOUR_APP_NAME # dyno status
```
On Heroku, logs go to stdout only. Locally they also go to `logs/bot.log`, rotated at 10 MB.
Tokens, hashes, session strings, private keys and the database URL are redacted from log output.
## 15. Restarting and redeploying
```bash
heroku ps:restart worker --app YOUR_APP_NAME
git push heroku main # or push to GitHub with automatic deploys enabled
heroku ps:scale worker=1 --app YOUR_APP_NAME
```
On `SIGTERM`, Heroku allows 30 seconds before it kills the process. The bot:
1. stops at the next download or upload chunk boundary;
2. saves state;
3. disconnects.
On the next start it:
1. returns interrupted downloads to PENDING;
2. checks Drive for any upload that may have finished, using the hidden tag;
3. re-downloads files lost with the old dyno's disk;
4. **continues the saved upload session**, when the re-downloaded file is byte-identical (MD5);
5. resumes processing if it was running.
**Scaling:** run **exactly one** worker. A second worker is blocked by a PostgreSQL advisory lock.
It waits `INSTANCE_LOCK_WAIT` seconds (90 on Heroku, which covers deploy overlap), then exits with
code 3. Scaling above 1 therefore only produces crash-looping dynos, not parallel processing. The
pipeline is intentionally sequential.
## 16. Troubleshooting
| Symptom | Fix |
|---|---|
| `DATABASE_URL is not set` | Attach Heroku Postgres (§11) |
| `TELEGRAM_SESSION_STRING is required…` | Run `python main.py login --session-string` locally (§7), or use `TELEGRAM_MODE=bot` |
| `Telegram user session is not authorized` | The session was revoked (e.g. *Terminate all sessions*); create a new session string |
| `Cannot resolve SOURCE_CHANNEL_ID` | Use `@username` for public channels; make sure the account/bot is a member/admin; in bot mode post once in the channel |
| `DRIVE_ROOT_FOLDER_ID is not visible…` | `drive.file` scope can't see folders you created: use `drive-init`, or switch to `GOOGLE_DRIVE_SCOPE=drive` and redo `drive-auth` |
| `Google authorization expired or was revoked` | Redo `drive-auth` and update `GOOGLE_OAUTH_TOKEN_JSON`; publish the consent screen to *In production* (§9) |
| `invalid_scope` on refresh | The token was created with another scope; delete `credentials/token.json` and run `drive-auth` again |
| `storageQuotaExceeded` | Drive is full, or you used a service account without a Shared Drive |
| `Not enough free disk space` | A video is larger than the dyno's free disk; lower `MIN_FREE_DISK_SPACE` with care, or process that video locally |
| No progress messages | Press *Start* in your bot chat; check `ADMIN_USER_IDS` / `NOTIFY_CHAT_IDS` |
| `Another worker is already processing` | `heroku ps:scale worker=1`; locally, stop the other process |
| Item stuck in FAILED | `/check ID` shows the error; fix the cause, then `/retry` |
| Long pause after `FloodWait` | Telegram rate limit; the bot waits the time Telegram asks for |
## 17. Security and credential handling
- All secrets come from environment variables / Config Vars; nothing is hard-coded.
- `.gitignore` excludes `.env`, `credentials/`, `token.json`, `*.session`, `session_string.txt`, `data/` and `downloads/`.
- The session string and OAuth token are full-access credentials. If one leaks:
- Telegram: *Settings → Devices → terminate the session*;
- Google: <https://myaccount.google.com/permissions>.
- Uploaded files stay private by default. Drive scope defaults to least privilege (`drive.file`).
- Commands are admin-only and private-chat only. Replies never include credentials, local paths or stack traces.
- Telegram login codes and 2FA passwords are entered interactively, locally, and never stored.
## 18. Known limitations and Heroku resource constraints
- **Ephemeral disk:** each video is staged on the dyno's temporary filesystem. Heroku doesn't
guarantee a specific capacity, and the disk is wiped on every restart (at least daily).
- The bot checks free space before each download and halts with a message if a video won't fit.
- Multi-GB videos may not fit on a dyno; run the bot locally for those.
- **Memory:** one upload chunk (`UPLOAD_CHUNK_SIZE`) is held in RAM at a time. `8MB` is comfortable on 512 MB dynos.
- **Daily restarts:** in-flight transfers resume from their checkpoint. A partially downloaded file
on the old dyno is lost and is downloaded again, but the Drive upload session is reused.
- **Bot mode:** history is best effort (ID probing), and private channels need one live post before the numeric ID resolves.
- **Telegram:** flood limits apply. Downloads go through Telegram's servers at whatever speed they allow.
- **Drive:** resumable sessions expire after about a week. Older sessions are discarded and the
upload restarts after a Drive check for an already-finished copy.
- **Captions edited after discovery** are not re-read. Use `/check ID` to see what was stored.
- **Cost:** the worker dyno and Postgres add-on are paid Heroku resources.
### What has and has not been verified
- **Automated tests (171):** run against SQLite **and** a real local PostgreSQL 16, with mocked Telegram and Google Drive.
- **Not tested:** real Telegram logins and downloads, real Google Drive uploads, and an actual Heroku deployment. Follow the manual end-to-end procedure below before relying on it.
## 19. Local development
```bash
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
cp .env.example .env # fill in values
python main.py login # user mode: creates data/sessions/user.session
python main.py drive-auth # creates credentials/token.json
python main.py check-config
python main.py run # or ./start.sh
```
Other subcommands:
- `python main.py sync [--rescan]`
- `python main.py process` (headless: sync, process everything, exit)
- `python main.py status`
- `python main.py drive-init --name "Telegram Videos"`
### Tests
```bash
pytest
```
`pgserver` from `requirements-dev.txt` starts a throwaway PostgreSQL so every database test runs on
both backends. Without it, only SQLite is tested. The suite covers:
- caption parsing (the supplied example, multiple Index lines, missing fields, Unicode, flattened captions);
- naming and collisions;
- Drive folder reuse and dedupe;
- the resumable-upload protocol;
- state transitions and duplicate messages;
- download and upload failure recovery;
- restart during upload, and restart with a wiped disk;
- a successful upload followed by a database-write failure;
- sequential processing, pause/resume and graceful shutdown;
- admin-only commands;
- configuration and Heroku validation, env-var credentials, log redaction;
- `app.json` / Procfile consistency.
### Manual end-to-end test (real integrations)
1. Create a private test channel. Post four videos:
- two with `Topic: Test A`;
- one with `Topic: Test B`;
- one with no caption.
Give two of them the same `Title`.
2. Point the bot at a fresh Drive folder. Then `/sync`, `/run`, and watch `/progress`.
3. Check Drive:
- `Test A/` holds two files, one with a `(2)` suffix if the titles matched;
- `Test B/` holds one;
- `Uncategorized/` holds the caption-less video;
- file sizes match Telegram's.
4. Kill the process (`kill -9`, or `heroku ps:restart`) **during an upload** of a large video.
Restart it. Confirm the upload continues and Drive ends up with exactly one copy.
5. `/stop`, post a new video, `/run`: only the new video is processed.
6. Revoke Drive access temporarily and confirm the queue halts with a clear message. Restore access, then `/resume`.