619 lines
34 KiB
Markdown
619 lines
34 KiB
Markdown
# Telegram Channel → Google Drive Organizer
|
||
|
||
[](https://heroku.com/deploy?template=https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME)
|
||
|
||
> **Before the button works:** replace `https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME`
|
||
> in the line above with the URL of the **public** GitHub repository that contains this project
|
||
> (see [Publishing to GitHub](#publishing-to-github)). That URL is the only value to change.
|
||
> The button flow has not been tested against a live repository yet. Test it once after publishing.
|
||
|
||
A long-running worker that reads video posts from a Telegram channel, downloads each video, and
|
||
uploads it to Google Drive. You can process the whole channel (`/sync` + `/run`) or just a range of
|
||
posts with the interactive **`/upload`** command (send a start link and an end link, then `/confirm`). The caption's **Title** becomes the file name and its **Topic** picks the
|
||
Drive folder. Videos are processed one at a time. All progress is stored in a database, so the bot
|
||
resumes after crashes, restarts and redeploys without uploading anything twice.
|
||
|
||
## Contents
|
||
1. [Overview](#1-overview) · 2. [Features](#2-features) · 3. [Caption format](#3-caption-format-and-metadata-extraction) ·
|
||
4. [Drive organization](#4-google-drive-topic-folder-organization) · 5. [Prerequisites](#5-prerequisites) ·
|
||
6. [One-click Heroku deployment](#6-one-click-heroku-deployment) · 7. [Telegram setup](#7-telegram-bot-setup) ·
|
||
8. [Historical access](#8-historical-channel-access-requirements) · 9. [Google Drive auth](#9-google-drive-authentication) ·
|
||
10. [Config Vars](#10-required-heroku-config-vars) · 11. [Database](#11-database-provisioning-and-configuration) ·
|
||
12. [First deployment](#12-first-deployment-and-startup) · 13. [Commands](#13-telegram-commands) · 14. [Logs](#14-checking-logs) ·
|
||
15. [Restart / redeploy](#15-restarting-and-redeploying) · 16. [Troubleshooting](#16-troubleshooting) ·
|
||
17. [Security](#17-security-and-credential-handling) · 18. [Limitations](#18-known-limitations-and-heroku-resource-constraints) ·
|
||
19. [Local development](#19-local-development)
|
||
|
||
---
|
||
|
||
## 1. Overview
|
||
|
||
```
|
||
Telegram channel ──(Telethon / MTProto)──► queue worker (one video at a time) ──► Google Drive
|
||
│ (resumable upload,
|
||
Admin chat ◄── bot commands & progress ─────────┤ verified by size + MD5)
|
||
▼
|
||
PostgreSQL (Heroku) / SQLite (local)
|
||
```
|
||
|
||
For each video: **discover → parse caption → resolve the topic folder → download to a temp disk →
|
||
upload (resumable) → verify size and MD5 in Drive → save the Drive file ID → mark COMPLETED → delete
|
||
the temp file → next video.**
|
||
|
||
## 2. Features
|
||
|
||
- Caption parsing for `Index` (one or many), `Title`, `Topic`, `Batch` and `Extracted By`.
|
||
- Topic folders are created once and reused. Optional `Batch/Topic` layout.
|
||
- Files are never overwritten. Name collisions get `(2)`, `(3)`, … or `[message id]`, or are skipped or failed, per your policy.
|
||
- No duplicate uploads, even if the process dies just after Drive accepted the file. Every upload carries a hidden `appProperties` tag (`channel:message`) that is checked before uploading.
|
||
- Resumable uploads that survive restarts. The upload session URI is stored in the database.
|
||
- Resumable downloads, continuing from the last 1 MiB boundary of a partial file.
|
||
- Explicit state machine: `PENDING → DOWNLOADING → DOWNLOADED → UPLOADING → COMPLETED`, plus `RETRY_WAIT`, `FAILED` and `SKIPPED`. Every transition is validated and logged in the `transitions` table.
|
||
- Bounded retries with exponential backoff, honouring Telegram flood waits and Drive rate limits. Permission errors are not retried.
|
||
- Disk-space checks before each download. A full disk, Drive quota or expired auth halts the queue cleanly instead of burning retries.
|
||
- Admin-only Telegram commands, plus one progress message that is edited in place rather than spammed.
|
||
- **Interactive `/upload` tasks:** pick an inclusive message range by pasting two post links,
|
||
confirm, and watch a live task progress message. Tasks are stored in the database and survive restarts.
|
||
- **Automatic temp-file cleanup:** each video's local file is deleted right after its upload is
|
||
verified in Drive *and* recorded in the database. Abandoned temp files are swept at startup,
|
||
periodically, and whenever disk space runs low.
|
||
- Graceful shutdown on `SIGTERM` (Heroku restarts) at a resumable checkpoint.
|
||
- A single-worker guard (PostgreSQL advisory lock) prevents two dynos from processing the same queue.
|
||
|
||
## 3. Caption format and metadata extraction
|
||
|
||
```
|
||
Index: 001
|
||
|
||
Index: 002
|
||
|
||
Title: Class 01 Aman sir.mp4
|
||
|
||
Topic: Misc• Grammar Practice
|
||
|
||
Batch: English ( SSC MAINS )
|
||
|
||
Extracted By: @PUTINxINDIA
|
||
```
|
||
|
||
| Field | Use |
|
||
|---|---|
|
||
| `Title` | Drive file name (`Class 01 Aman sir.mp4`) |
|
||
| `Topic` | Drive folder (`Misc• Grammar Practice`) |
|
||
| `Index` | All values kept in order (`["001", "002"]`); never used as the file name. The Telegram message ID is the unique identity. |
|
||
| `Batch` | Stored; optional `Batch/Topic` folders |
|
||
| `Extracted By` | Stored in the record |
|
||
|
||
Parsing rules:
|
||
- Fields can appear in any order.
|
||
- Labels are case-insensitive.
|
||
- Blank lines, emoji or bullet prefixes, extra spaces and full-width colons are allowed.
|
||
- Unicode text is preserved.
|
||
- Parsing is line-based, so one field never swallows the next.
|
||
- A flattened single-line caption is split at each known label.
|
||
|
||
Missing data never makes a video silently disappear:
|
||
|
||
| Case | Behaviour |
|
||
|---|---|
|
||
| No Topic | Folder `Uncategorized` (`FALLBACK_TOPIC_FOLDER`) |
|
||
| No Title | Telegram file name, else `telegram_<message id>.<ext>`. A warning is recorded. `MISSING_TITLE_POLICY=fail` marks it FAILED instead. |
|
||
| Title without extension | Extension taken from the real file's magic bytes, else from the MIME type |
|
||
| Title extension ≠ real container | Kept as written, with a warning. `EXTENSION_MISMATCH_POLICY=append` appends the real one. |
|
||
| `/` `\` or control characters | Replaced (`STRICT_FILENAMES=true` also replaces `<>:"\|?*`) |
|
||
|
||
## 4. Google Drive topic-folder organization
|
||
|
||
```
|
||
<GOOGLE_DRIVE_ROOT_FOLDER_ID>/
|
||
├── Misc• Grammar Practice/
|
||
│ ├── Class 01 Aman sir.mp4
|
||
│ └── Class 02 Aman sir.mp4
|
||
├── Error Detection/
|
||
│ └── Class 01 Aman sir.mp4
|
||
└── Uncategorized/
|
||
```
|
||
|
||
Folder lookup goes database cache → Drive search → create. If duplicate folders already exist, the
|
||
oldest is reused. If a folder-create response is lost, the bot searches again before retrying, so
|
||
it never creates a second copy. Set `BATCH_SUBFOLDERS_ENABLED=true` for `Root/Batch/Topic/`.
|
||
|
||
**Duplicate policy (`DUPLICATE_POLICY`)** applies when a *different* file already has the target
|
||
name in the folder:
|
||
|
||
| Value | Result |
|
||
|---|---|
|
||
| `suffix_number` (default) | `Class 01 Aman sir (2).mp4` |
|
||
| `suffix_message_id` | `Class 01 Aman sir [1234].mp4` |
|
||
| `skip` | Item marked SKIPPED with the reason |
|
||
| `fail` | Item marked FAILED |
|
||
|
||
An existing file is reused, not re-uploaded, only when its identity is verified. That means either
|
||
its hidden tag matches this Telegram message, or it has the same size and MD5
|
||
(`REUSE_IDENTICAL_FILES=true`). The same Telegram media re-posted under a new message ID is SKIPPED
|
||
by default (`DUPLICATE_MEDIA_POLICY`).
|
||
|
||
## 5. Prerequisites
|
||
|
||
- A Telegram account, plus API credentials from <https://my.telegram.org>.
|
||
- A Telegram bot from [@BotFather](https://t.me/BotFather).
|
||
- A Google account with a Google Cloud project (free) with the Drive API enabled.
|
||
- **A local computer with Python 3.12** for two one-time interactive logins. Telegram and Google logins need a human, so they can't run on Heroku.
|
||
- A Heroku account with billing enabled. The worker dyno and Heroku Postgres are paid resources; check current pricing on Heroku.
|
||
- Optional: the [Heroku CLI](https://devcenter.heroku.com/articles/heroku-cli) and the [GitHub CLI](https://cli.github.com/).
|
||
|
||
## 6. One-click Heroku deployment
|
||
|
||
The button (top of this page) opens Heroku's deploy form. It:
|
||
- creates the app from this repository's `app.json`;
|
||
- builds it with the official Python buildpack;
|
||
- provisions **Heroku Postgres (`essential-0`)**, which sets `DATABASE_URL`;
|
||
- starts **one `basic` worker dyno** (`worker: python main.py run`); there is no web process.
|
||
|
||
**It is not fully one-click.** Two secrets must be created on your computer first, because they need
|
||
interactive logins:
|
||
|
||
1. `TELEGRAM_SESSION_STRING`, only for `TELEGRAM_MODE=user` (see [§7](#7-telegram-bot-setup)).
|
||
2. `GOOGLE_OAUTH_TOKEN_JSON` or `GOOGLE_SERVICE_ACCOUNT_JSON` (see [§9](#9-google-drive-authentication)).
|
||
|
||
Prepare those, click the button, fill the form, then press **Deploy app**. If add-on provisioning
|
||
fails (for example because billing isn't verified or the plan isn't available), create the app
|
||
anyway and follow [§11](#11-database-provisioning-and-configuration) manually. The worker refuses
|
||
to start on Heroku without a database rather than silently losing progress.
|
||
|
||
Manual alternative with the CLI (from the repository root):
|
||
|
||
```bash
|
||
heroku create YOUR_APP_NAME --stack heroku-24
|
||
heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME
|
||
heroku config:set TELEGRAM_API_ID=1234567 TELEGRAM_API_HASH=your_api_hash \
|
||
TELEGRAM_BOT_TOKEN=123456:your-bot-token ADMIN_USER_IDS=111111111 \
|
||
SOURCE_CHANNEL_ID=@your_channel --app YOUR_APP_NAME
|
||
heroku config:set TELEGRAM_SESSION_STRING="$(cat data/sessions/session_string.txt)" --app YOUR_APP_NAME
|
||
heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME
|
||
git push heroku main
|
||
heroku ps:scale worker=1 --app YOUR_APP_NAME
|
||
```
|
||
|
||
### Publishing to GitHub
|
||
|
||
This directory (the one containing `Procfile` and `app.json`) must be the **repository root**.
|
||
|
||
```bash
|
||
cd telegram_drive_bot
|
||
git init -b main
|
||
git add .
|
||
git status # confirm: no .env, credentials/, data/, *.session, downloads/
|
||
git commit -m "Telegram to Google Drive organizer"
|
||
gh repo create YOUR_REPOSITORY_NAME --public --source=. --push
|
||
# or create the repo on github.com, then:
|
||
# git remote add origin https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME.git && git push -u origin main
|
||
```
|
||
|
||
Then put that repository URL into the button link at the top of this README, commit, and push.
|
||
The Heroku button only works with **public** repositories.
|
||
|
||
## 7. Telegram bot setup
|
||
|
||
1. **API ID / hash:** log in at <https://my.telegram.org> → *API development tools* → create an app.
|
||
Copy `api_id` → `TELEGRAM_API_ID` and `api_hash` → `TELEGRAM_API_HASH`.
|
||
These are needed in both modes, because the bot talks MTProto through Telethon.
|
||
2. **Bot:** talk to @BotFather → `/newbot` → copy the token → `TELEGRAM_BOT_TOKEN`.
|
||
**Open your bot and press Start.** A bot can't message you until you do, so you won't get progress messages otherwise.
|
||
3. **Admin IDs:** message [@userinfobot](https://t.me/userinfobot) to get your numeric ID → `ADMIN_USER_IDS`
|
||
(comma-separate several). Only these users can run commands.
|
||
4. **Source channel (`SOURCE_CHANNEL_ID`):** any of these forms work:
|
||
- a public channel: `@channelname` or `https://t.me/channelname`;
|
||
- a private channel: the numeric ID `-100…`, or a post link `https://t.me/c/1234567890/55` (use *Copy Post Link* in Telegram);
|
||
- forwarding a post to @userinfobot also shows the channel ID.
|
||
5. **Choose a mode (`TELEGRAM_MODE`):**
|
||
- `user` (default): reads the channel as *your* account, which must be a member. Create the session string locally:
|
||
```bash
|
||
python main.py login --session-string
|
||
```
|
||
Telethon asks for your phone number, the login code and, only if enabled, your 2FA password.
|
||
None of these are stored. The resulting session string is written to
|
||
`data/sessions/session_string.txt` (mode 600). Copy it into `TELEGRAM_SESSION_STRING`, then delete the file.
|
||
- `bot`: add the bot to the channel as an **administrator**. No session string is needed.
|
||
|
||
## 8. Historical channel access requirements
|
||
|
||
| Capability | `user` mode | `bot` mode |
|
||
|---|---|---|
|
||
| New posts (live) | ✅ | ✅ (bot must be a channel admin) |
|
||
| Existing history | ✅ full history the account can see | ⚠️ best effort: bots can't list history, so message IDs are probed in batches until `BOT_SCAN_MAX_EMPTY` consecutive gaps (or up to `BOT_SCAN_END_ID`) |
|
||
| `/upload` message ranges | ✅ | ✅ messages are fetched by ID, which bots that are channel admins may do |
|
||
| Large files | ✅ up to Telegram's file limit | ✅ uses MTProto, so the HTTP Bot API's 20 MB download limit does **not** apply |
|
||
| Private channel by numeric ID | ✅ | ⚠️ the bot learns the channel's access key from its first live post; the key is then stored in the database |
|
||
| Extra secret needed | `TELEGRAM_SESSION_STRING` | none |
|
||
|
||
`PROCESS_MODE` controls what is queued:
|
||
- `existing`: history only (from `START_MESSAGE_ID` / `START_DATE`).
|
||
- `new`: only posts after the first start.
|
||
- `both` (default).
|
||
|
||
The bot never deletes or edits Telegram messages and doesn't bypass channel restrictions. A video
|
||
in a channel with content protection (no forwarding/saving) may not be downloadable.
|
||
|
||
## 9. Google Drive authentication
|
||
|
||
Pick **one**.
|
||
|
||
### Option A — OAuth (personal Google Drive, recommended)
|
||
|
||
1. In the [Google Cloud Console](https://console.cloud.google.com/): create a project, then enable the **Google Drive API**.
|
||
2. Configure the **OAuth consent screen**: *External*, add yourself as a test user.
|
||
Then set the publishing status to **In production**. In *Testing* status, Google expires refresh
|
||
tokens after 7 days and the bot would lose Drive access weekly. With the default `drive.file`
|
||
scope no verification is required. With the full `drive` scope Google shows an "unverified app"
|
||
screen during your own login (*Advanced → continue*).
|
||
3. Under *Credentials*, create an **OAuth client ID** of type **Desktop app**. Download the JSON to
|
||
`credentials/client_secret.json` in this project.
|
||
4. Locally: `python main.py drive-auth`. This opens a browser; use `--no-browser` to get a URL to open yourself.
|
||
It writes `credentials/token.json` (mode 600).
|
||
5. Heroku: `heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME`.
|
||
Multi-line JSON works through the CLI. In the dashboard you can paste raw JSON, or a single-line
|
||
base64 version: `base64 < credentials/token.json | tr -d '\n'`.
|
||
|
||
**Scope and root folder.** `GOOGLE_DRIVE_SCOPE=drive.file` (least privilege) lets the bot see only
|
||
files and folders *it created*. So either:
|
||
- let the bot create the root folder: `python main.py drive-init --name "Telegram Videos"` prints the ID for `GOOGLE_DRIVE_ROOT_FOLDER_ID`; or
|
||
- use a folder you created yourself: set `GOOGLE_DRIVE_SCOPE=drive` **before** running `drive-auth`.
|
||
|
||
The folder ID is the part after `/folders/` in its URL. The bot checks at startup that it can see
|
||
the folder and stops with an explanation if it can't.
|
||
|
||
### Option B — Service account (Google Workspace Shared Drive)
|
||
|
||
A service account has **no storage quota in a personal My Drive**. Uploads into a folder that a
|
||
normal Gmail user merely shared with it fail with `storageQuotaExceeded`. Use it only with a
|
||
**Shared Drive**:
|
||
|
||
1. Create a service account, then create a JSON key for it.
|
||
2. In the Shared Drive, *Manage members* → add the service account's `client_email` as **Content manager**.
|
||
3. Set:
|
||
- `GOOGLE_AUTH_MODE=service_account`
|
||
- `GOOGLE_SERVICE_ACCOUNT_JSON="$(cat key.json)"`
|
||
- `DRIVE_SHARED_DRIVE_ID=<shared drive id>`
|
||
- `GOOGLE_DRIVE_ROOT_FOLDER_ID=<folder id inside it>`
|
||
- `GOOGLE_DRIVE_SCOPE=drive` (needed to use a folder the service account did not create).
|
||
|
||
Credentials from environment variables are parsed in memory. Nothing is written to the dyno's disk,
|
||
and they are validated by an authenticated Drive call at startup. Uploaded files are **private**
|
||
unless you set `DRIVE_SHARE_MODE=anyone_with_link`.
|
||
|
||
## 10. Required Heroku Config Vars
|
||
|
||
Set these in the dashboard (*Settings → Reveal Config Vars*) or with
|
||
`heroku config:set NAME=value --app YOUR_APP_NAME`.
|
||
|
||
| Variable | Required | Notes |
|
||
|---|---|---|
|
||
| `TELEGRAM_API_ID`, `TELEGRAM_API_HASH` | yes | my.telegram.org |
|
||
| `TELEGRAM_BOT_TOKEN` | yes | @BotFather; command interface (and channel access in bot mode) |
|
||
| `ADMIN_USER_IDS` | yes | numeric IDs, comma-separated |
|
||
| `SOURCE_CHANNEL_ID` | yes | `@name`, link, or `-100…` |
|
||
| `TELEGRAM_MODE` | yes (default `user`) | `user` or `bot` |
|
||
| `TELEGRAM_SESSION_STRING` | **when `TELEGRAM_MODE=user`** | from `main.py login --session-string` |
|
||
| `GOOGLE_AUTH_MODE` | yes (default `oauth`) | `oauth` or `service_account` |
|
||
| `GOOGLE_OAUTH_TOKEN_JSON` | **when `oauth`** | contents of `credentials/token.json` |
|
||
| `GOOGLE_SERVICE_ACCOUNT_JSON` | **when `service_account`** | key JSON |
|
||
| `GOOGLE_DRIVE_ROOT_FOLDER_ID` | recommended | empty = top level of My Drive |
|
||
| `GOOGLE_DRIVE_SCOPE` | no (`drive.file`) | must match the scope used in `drive-auth` |
|
||
| `DATABASE_URL` | **yes on Heroku** | set automatically by Heroku Postgres |
|
||
| `PROCESS_MODE` | no (`both`) | `existing` / `new` / `both` |
|
||
| `DUPLICATE_POLICY` | no (`suffix_number`) | see §4 |
|
||
| `BATCH_SUBFOLDERS_ENABLED` | no (`false`) | `Root/Batch/Topic` |
|
||
| `MAX_RETRIES` / `RETRY_DELAY` | no (`4` / `30`) | automatic retries / base delay in seconds |
|
||
| `PROGRESS_INTERVAL` | no (`10`) | seconds between progress-message edits |
|
||
| `DOWNLOAD_DIR` | no (`/tmp/telegram_drive_bot` in app.json) | temporary files |
|
||
| `UPLOAD_CHUNK_SIZE` | no (`8MB` in app.json) | multiple of 256KB; memory per chunk |
|
||
| `AUTO_START` | no (`false`) | start processing on every boot |
|
||
| `UPLOAD_RESTRICT_TO_SOURCE_CHANNEL` | no (`true`) | `/upload` links must point to `SOURCE_CHANNEL_ID` |
|
||
| `UPLOAD_SETUP_TIMEOUT` | no (`30`) | minutes before an unfinished `/upload` setup expires |
|
||
| `UPLOAD_MAX_RANGE` | no (`5000`) | largest number of message IDs per task |
|
||
| `KEEP_FAILED_DOWNLOADS` | no (`false`) | keep local files of FAILED videos |
|
||
| `TEMP_CLEANUP_INTERVAL` | no (`3600`) | seconds between abandoned-temp-file sweeps |
|
||
|
||
Step-by-step instructions for obtaining every value are in **[ENV_SETUP.md](ENV_SETUP.md)**. Every other supported variable is listed with an explanation in [`.env.example`](.env.example).
|
||
|
||
## 11. Database provisioning and configuration
|
||
|
||
Production state lives in **PostgreSQL**:
|
||
- video records with all caption metadata;
|
||
- states, attempts, errors, timestamps and Drive file IDs;
|
||
- upload session URIs;
|
||
- the folder map;
|
||
- discovery and completion cursors;
|
||
- duplicate-detection data;
|
||
- the transition history;
|
||
- the pause/run/stop choice.
|
||
|
||
- **Via the button:** `app.json` requests `heroku-postgresql:essential-0`, a paid plan.
|
||
- **Manually:** `heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME`
|
||
(or any plan, or any PostgreSQL reachable through `DATABASE_URL`).
|
||
- The schema is created and migrated automatically at startup. The version is tracked in the `kv` table.
|
||
- TLS (`sslmode=require`) is added automatically for remote hosts.
|
||
- **Without `DATABASE_URL` on Heroku the worker refuses to start.** SQLite on a dyno would be wiped
|
||
at every restart, and the bot would re-upload everything. To accept that risk explicitly, set
|
||
`ALLOW_EPHEMERAL_DATABASE=true`; a warning is logged at every start.
|
||
- Locally, without `DATABASE_URL`, SQLite at `data/bot.db` is used. This is for development only.
|
||
|
||
## 12. First deployment and startup
|
||
|
||
1. Deploy (button or CLI). Watch the logs: `heroku logs --tail --app YOUR_APP_NAME`.
|
||
You should see the database backend, `Google Drive credentials validated`, and `Source channel resolved`.
|
||
2. The bot messages each admin: *Bot online (PostgreSQL); send /sync then /run to start.*
|
||
3. Either send `/upload` to process a message range (see [§13](#13-telegram-commands)), or send
|
||
`/sync` to discover all channel videos and then `/run`. Watch `/taskstatus` or `/progress`.
|
||
4. Running tasks and the `/run` choice are stored. After any dyno restart the bot resumes on its own.
|
||
`/pause` and `/stop` are also remembered.
|
||
|
||
## 13. Telegram commands
|
||
|
||
Send these to the bot in a **private chat**. Users not in `ADMIN_USER_IDS` are refused.
|
||
|
||
| Command | Effect |
|
||
|---|---|
|
||
| `/start`, `/help` | Instructions |
|
||
| `/status` | Queue state, the current video, live progress |
|
||
| `/progress` | Processed / pending / retrying / failed / skipped / remaining counts |
|
||
| `/sync` | Discover unseen messages and queue them (idempotent). `/sync all` rescans from `START_MESSAGE_ID`. |
|
||
| `/run` | Start processing |
|
||
| `/pause` | Stop the current transfer at the next chunk boundary (resumable) and wait |
|
||
| `/resume` | Continue |
|
||
| `/stop` | Checkpoint and end the worker until `/run` |
|
||
| `/retry` | Re-queue FAILED items with a fresh retry budget |
|
||
| `/failed` | Failed items with short error summaries |
|
||
| `/folders` | Folder count and uploads per topic |
|
||
| `/check ID` | Full record for one Telegram message ID |
|
||
| `/upload` | Start an interactive upload task for a message range |
|
||
| `/confirm` | Start the prepared task (only after both links were validated) |
|
||
| `/cancel` | Discard your setup, or cancel your active task at a safe checkpoint. `/cancel ID` targets a specific task. |
|
||
| `/taskstatus` | Your setup step, plus active tasks with range, counters and live progress. `/taskstatus ID` for one task. |
|
||
| `/uploadhistory` | The 10 most recent tasks and their results |
|
||
|
||
`/pause`, `/resume` and `/stop` also apply to upload tasks. `/stop` parks a running task as PAUSED;
|
||
`/resume` continues it.
|
||
|
||
### Processing a message range with `/upload`
|
||
|
||
1. Send `/upload`. The bot asks for the **start link**: in Telegram, right-click (or long-press) a
|
||
post → **Copy Post Link**, e.g. `https://t.me/c/3909337424/14`.
|
||
2. Send the link. The bot asks for the **end link**, e.g. `https://t.me/c/3909337424/20`.
|
||
3. The bot shows *Upload Task Ready* with both message IDs and the channel.
|
||
4. Send `/confirm` to start, or `/cancel` to discard. Nothing is downloaded before `/confirm`.
|
||
|
||
Link rules:
|
||
- Accepted formats: `t.me/c/<id>/<msg>` (private), `t.me/<username>/<msg>` (public), and
|
||
`t.me/s/<username>/<msg>`. `www.`, `telegram.me`, a trailing slash and `?single` are fine.
|
||
- Invite links and links without a message ID are rejected with an explanation.
|
||
- The `/c/` number is Telegram's internal channel ID. It is converted to the API ID
|
||
`-(10¹² + id)` and then **resolved through the bot's Telegram account**. A channel that account
|
||
can't access is refused; nothing bypasses membership or permissions.
|
||
- Both links must be from the same channel, the end ID must be ≥ the start ID, the range is
|
||
limited by `UPLOAD_MAX_RANGE`, and both messages must exist.
|
||
- By default the links must point to `SOURCE_CHANNEL_ID`. Set
|
||
`UPLOAD_RESTRICT_TO_SOURCE_CHANNEL=false` to allow any channel the account (or admin bot) can read.
|
||
- If the end link is invalid, the start link is kept; just send another end link.
|
||
|
||
How a task runs:
|
||
- **The range is inclusive** (14–20 means 14, 15, …, 20), processed in ascending order, one video at a time.
|
||
- **Text, photos, stickers and deleted messages** are counted as skipped, never as failures.
|
||
- **Videos already uploaded earlier** are counted as duplicates and are not uploaded again.
|
||
Repeating a finished task therefore uploads nothing new.
|
||
- **Caption metadata** (Title, Topic, …) is used exactly as for whole-channel processing.
|
||
- **Each local file is deleted** only after the Drive upload is verified *and* its COMPLETED state is committed.
|
||
- **Failed videos** are recorded, never marked completed, and can be re-queued with `/retry`.
|
||
Their local file is deleted (unless `KEEP_FAILED_DOWNLOADS=true`) because it can be downloaded again.
|
||
- **`/cancel` during processing** stops at the next safe checkpoint:
|
||
- finished uploads stay in Drive;
|
||
- the remaining videos are marked *cancelled* and their temp files are removed;
|
||
- a later task covering the same range picks the cancelled videos up again.
|
||
- **Setups are per administrator.** Two admins can prepare tasks at the same time without
|
||
affecting each other. A second `/upload` never overwrites your unfinished setup. A setup expires
|
||
after `UPLOAD_SETUP_TIMEOUT` minutes.
|
||
- **Confirmed tasks run one after another** in the order they were confirmed. Running and queued
|
||
tasks resume automatically after a restart or redeploy.
|
||
|
||
Task progress is one message, edited in place:
|
||
|
||
```
|
||
🚀 Upload Task Running
|
||
|
||
Task: #12
|
||
Range: 14–20
|
||
Current Message: 16
|
||
Current Video: Class 01 Aman sir.mp4
|
||
Topic: Misc• Grammar Practice
|
||
Status: Uploading to Google Drive
|
||
Upload: 45% (540.0 MB / 1.2 GB) · 18.3 MB/s · ETA 36s
|
||
|
||
Completed: 1
|
||
Skipped: 1
|
||
Failed: 0
|
||
```
|
||
|
||
When the task finishes, the bot sends a summary covering:
|
||
- the range;
|
||
- messages examined;
|
||
- eligible videos;
|
||
- uploads, duplicates, non-video messages and failures;
|
||
- temp files deleted;
|
||
- the Drive folders used.
|
||
|
||
For whole-channel processing, progress is a single message, edited every `PROGRESS_INTERVAL` seconds:
|
||
|
||
```
|
||
Processing video 12 of 150
|
||
Topic: Misc• Grammar Practice
|
||
Title: Class 01 Aman sir.mp4
|
||
Status: Uploading
|
||
Download: 100% (1.2 GB / 1.2 GB)
|
||
Upload: 45% (540.0 MB / 1.2 GB) · 18.3 MB/s · ETA 36s
|
||
```
|
||
|
||
## 14. Checking logs
|
||
|
||
```bash
|
||
heroku logs --tail --app YOUR_APP_NAME # live
|
||
heroku logs -n 1500 --app YOUR_APP_NAME # recent history
|
||
heroku ps --app YOUR_APP_NAME # dyno status
|
||
```
|
||
|
||
On Heroku, logs go to stdout only. Locally they also go to `logs/bot.log`, rotated at 10 MB.
|
||
Tokens, hashes, session strings, private keys and the database URL are redacted from log output.
|
||
|
||
## 15. Restarting and redeploying
|
||
|
||
```bash
|
||
heroku ps:restart worker --app YOUR_APP_NAME
|
||
git push heroku main # or push to GitHub with automatic deploys enabled
|
||
heroku ps:scale worker=1 --app YOUR_APP_NAME
|
||
```
|
||
|
||
On `SIGTERM`, Heroku allows 30 seconds before it kills the process. The bot:
|
||
1. stops at the next download or upload chunk boundary;
|
||
2. saves state;
|
||
3. disconnects.
|
||
|
||
On the next start it:
|
||
1. returns interrupted downloads to PENDING;
|
||
2. checks Drive for any upload that may have finished, using the hidden tag;
|
||
3. re-downloads files lost with the old dyno's disk;
|
||
4. **continues the saved upload session**, when the re-downloaded file is byte-identical (MD5);
|
||
5. resumes processing if it was running.
|
||
|
||
**Scaling:** run **exactly one** worker. A second worker is blocked by a PostgreSQL advisory lock.
|
||
It waits `INSTANCE_LOCK_WAIT` seconds (90 on Heroku, which covers deploy overlap), then exits with
|
||
code 3. Scaling above 1 therefore only produces crash-looping dynos, not parallel processing. The
|
||
pipeline is intentionally sequential.
|
||
|
||
## 16. Troubleshooting
|
||
|
||
| Symptom | Fix |
|
||
|---|---|
|
||
| `DATABASE_URL is not set` | Attach Heroku Postgres (§11) |
|
||
| `TELEGRAM_SESSION_STRING is required…` | Run `python main.py login --session-string` locally (§7), or use `TELEGRAM_MODE=bot` |
|
||
| `Telegram user session is not authorized` | The session was revoked (e.g. *Terminate all sessions*); create a new session string |
|
||
| `Cannot resolve SOURCE_CHANNEL_ID` | Use `@username` for public channels; make sure the account/bot is a member/admin; in bot mode post once in the channel |
|
||
| `DRIVE_ROOT_FOLDER_ID is not visible…` | `drive.file` scope can't see folders you created: use `drive-init`, or switch to `GOOGLE_DRIVE_SCOPE=drive` and redo `drive-auth` |
|
||
| `Google authorization expired or was revoked` | Redo `drive-auth` and update `GOOGLE_OAUTH_TOKEN_JSON`; publish the consent screen to *In production* (§9) |
|
||
| `invalid_scope` on refresh | The token was created with another scope; delete `credentials/token.json` and run `drive-auth` again |
|
||
| `storageQuotaExceeded` | Drive is full, or you used a service account without a Shared Drive |
|
||
| `Not enough free disk space` | A video is larger than the dyno's free disk; lower `MIN_FREE_DISK_SPACE` with care, or process that video locally |
|
||
| No progress messages | Press *Start* in your bot chat; check `ADMIN_USER_IDS` / `NOTIFY_CHAT_IDS` |
|
||
| `Another worker is already processing` | `heroku ps:scale worker=1`; locally, stop the other process |
|
||
| Item stuck in FAILED | `/check ID` shows the error; fix the cause, then `/retry` |
|
||
| Long pause after `FloodWait` | Telegram rate limit; the bot waits the time Telegram asks for |
|
||
|
||
## 17. Security and credential handling
|
||
|
||
- All secrets come from environment variables / Config Vars; nothing is hard-coded.
|
||
- `.gitignore` excludes `.env`, `credentials/`, `token.json`, `*.session`, `session_string.txt`, `data/` and `downloads/`.
|
||
- The session string and OAuth token are full-access credentials. If one leaks:
|
||
- Telegram: *Settings → Devices → terminate the session*;
|
||
- Google: <https://myaccount.google.com/permissions>.
|
||
- Uploaded files stay private by default. Drive scope defaults to least privilege (`drive.file`).
|
||
- Commands are admin-only and private-chat only. Replies never include credentials, local paths or stack traces.
|
||
- Telegram login codes and 2FA passwords are entered interactively, locally, and never stored.
|
||
|
||
## 18. Known limitations and Heroku resource constraints
|
||
|
||
- **Ephemeral disk:** each video is staged on the dyno's temporary filesystem. Heroku doesn't
|
||
guarantee a specific capacity, and the disk is wiped on every restart (at least daily).
|
||
- The bot checks free space before each download and halts with a message if a video won't fit.
|
||
- Multi-GB videos may not fit on a dyno; run the bot locally for those.
|
||
- **Temporary files:** downloads live only in `DOWNLOAD_DIR`. The bot refuses to use `/`, `/tmp`,
|
||
your home directory or the project directory itself. Cleanup only removes files named
|
||
`<channel>_<message>.part|.video` directly inside that directory, never follows symlinks, and
|
||
re-checks the database first. Files of unfinished items are kept for resuming.
|
||
- **Memory:** one upload chunk (`UPLOAD_CHUNK_SIZE`) is held in RAM at a time. `8MB` is comfortable on 512 MB dynos.
|
||
- **Daily restarts:** in-flight transfers resume from their checkpoint. A partially downloaded file
|
||
on the old dyno is lost and is downloaded again, but the Drive upload session is reused.
|
||
- **Bot mode:** history is best effort (ID probing), and private channels need one live post before the numeric ID resolves.
|
||
- **Telegram:** flood limits apply. Downloads go through Telegram's servers at whatever speed they allow.
|
||
- **Drive:** resumable sessions expire after about a week. Older sessions are discarded and the
|
||
upload restarts after a Drive check for an already-finished copy.
|
||
- **Captions edited after discovery** are not re-read. Use `/check ID` to see what was stored.
|
||
- **Cost:** the worker dyno and Postgres add-on are paid Heroku resources.
|
||
|
||
### What has and has not been verified
|
||
|
||
- **Automated tests (259):** run against SQLite **and** a real local PostgreSQL 16, with mocked Telegram and Google Drive.
|
||
- **Not tested:** real Telegram logins and downloads, real Google Drive uploads, and an actual Heroku deployment. Follow the manual end-to-end procedure below before relying on it.
|
||
|
||
## 19. Local development
|
||
|
||
```bash
|
||
python3.12 -m venv .venv && source .venv/bin/activate
|
||
pip install -r requirements-dev.txt
|
||
cp .env.example .env # fill in values
|
||
python main.py login # user mode: creates data/sessions/user.session
|
||
python main.py drive-auth # creates credentials/token.json
|
||
python main.py check-config
|
||
python main.py run # or ./start.sh
|
||
```
|
||
|
||
Other subcommands:
|
||
- `python main.py sync [--rescan]`
|
||
- `python main.py process` (headless: sync, process everything, exit)
|
||
- `python main.py status`
|
||
- `python main.py drive-init --name "Telegram Videos"`
|
||
|
||
### Tests
|
||
|
||
```bash
|
||
pytest
|
||
```
|
||
|
||
`pgserver` from `requirements-dev.txt` starts a throwaway PostgreSQL so every database test runs on
|
||
both backends. Without it, only SQLite is tested. The suite covers:
|
||
- caption parsing (the supplied example, multiple Index lines, missing fields, Unicode, flattened captions);
|
||
- naming and collisions;
|
||
- Drive folder reuse and dedupe;
|
||
- the resumable-upload protocol;
|
||
- state transitions and duplicate messages;
|
||
- download and upload failure recovery;
|
||
- restart during upload, and restart with a wiped disk;
|
||
- a successful upload followed by a database-write failure;
|
||
- sequential processing, pause/resume and graceful shutdown;
|
||
- admin-only commands;
|
||
- the `/upload` workflow:
|
||
- link parsing and validation;
|
||
- per-admin setups, persistence and expiry;
|
||
- inclusive ascending ranges, and skipping non-video messages;
|
||
- delete-only-after-verified-upload, and failed uploads never completed;
|
||
- repeat tasks without duplicates;
|
||
- cancellation;
|
||
- crash/restart recovery and pause/resume;
|
||
- temp-file cleanup safety;
|
||
- configuration and Heroku validation, env-var credentials, log redaction;
|
||
- `app.json` / Procfile consistency.
|
||
|
||
### Manual end-to-end test (real integrations)
|
||
|
||
1. Create a private test channel. Post four videos:
|
||
- two with `Topic: Test A`;
|
||
- one with `Topic: Test B`;
|
||
- one with no caption.
|
||
Give two of them the same `Title`.
|
||
2. Point the bot at a fresh Drive folder. Then `/sync`, `/run`, and watch `/progress`.
|
||
3. Check Drive:
|
||
- `Test A/` holds two files, one with a `(2)` suffix if the titles matched;
|
||
- `Test B/` holds one;
|
||
- `Uncategorized/` holds the caption-less video;
|
||
- file sizes match Telegram's.
|
||
4. Kill the process (`kill -9`, or `heroku ps:restart`) **during an upload** of a large video.
|
||
Restart it. Confirm the upload continues and Drive ends up with exactly one copy.
|
||
5. `/stop`, post a new video, `/run`: only the new video is processed.
|
||
6. Revoke Drive access temporarily and confirm the queue halts with a clear message. Restore access, then `/resume`.
|
||
7. `/upload`: send the link of the first test post, then the last one, then `/confirm`. Check that:
|
||
- only that range is processed;
|
||
- a text post inside the range is reported as a skipped non-video;
|
||
- the summary counts are correct.
|
||
Run the same range again and confirm it reports duplicates and uploads nothing.
|
||
8. Start another `/upload` task and send `/cancel` while a video is downloading. Check that:
|
||
- finished uploads remain in Drive;
|
||
- `downloads/` is empty.
|