Files

T

2026-10-18 23:48:22 +05:30

27 KiB

Raw Blame History

Telegram Channel → Google Drive Organizer

Before the button works: replace https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME in the line above with the URL of the public GitHub repository that contains this project (see Publishing to GitHub). That URL is the only value to change. The button flow has not been tested against a live repository yet. Test it once after publishing.

A long-running worker that reads video posts from a Telegram channel, downloads each video, and uploads it to Google Drive. The caption's Title becomes the file name and its Topic picks the Drive folder. Videos are processed one at a time. All progress is stored in a database, so the bot resumes after crashes, restarts and redeploys without uploading anything twice.

Overview · 2. Features · 3. Caption format ·
Drive organization · 5. Prerequisites ·
One-click Heroku deployment · 7. Telegram setup ·
Historical access · 9. Google Drive auth ·
Config Vars · 11. Database ·
First deployment · 13. Commands · 14. Logs ·
Restart / redeploy · 16. Troubleshooting ·
Security · 18. Limitations ·
Local development

1. Overview

Telegram channel ──(Telethon / MTProto)──► queue worker (one video at a time) ──► Google Drive
                                                │                                (resumable upload,
Admin chat ◄── bot commands & progress ─────────┤                                 verified by size + MD5)
                                                ▼
                                   PostgreSQL (Heroku) / SQLite (local)

For each video: discover → parse caption → resolve the topic folder → download to a temp disk → upload (resumable) → verify size and MD5 in Drive → save the Drive file ID → mark COMPLETED → delete the temp file → next video.

2. Features

Caption parsing for Index (one or many), Title, Topic, Batch and Extracted By.
Topic folders are created once and reused. Optional Batch/Topic layout.
Files are never overwritten. Name collisions get (2), (3), … or [message id], or are skipped or failed, per your policy.
No duplicate uploads, even if the process dies just after Drive accepted the file. Every upload carries a hidden appProperties tag (channel:message) that is checked before uploading.
Resumable uploads that survive restarts. The upload session URI is stored in the database.
Resumable downloads, continuing from the last 1 MiB boundary of a partial file.
Explicit state machine: PENDING → DOWNLOADING → DOWNLOADED → UPLOADING → COMPLETED, plus RETRY_WAIT, FAILED and SKIPPED. Every transition is validated and logged in the transitions table.
Bounded retries with exponential backoff, honouring Telegram flood waits and Drive rate limits. Permission errors are not retried.
Disk-space checks before each download. A full disk, Drive quota or expired auth halts the queue cleanly instead of burning retries.
Admin-only Telegram commands, plus one progress message that is edited in place rather than spammed.
Graceful shutdown on SIGTERM (Heroku restarts) at a resumable checkpoint.
A single-worker guard (PostgreSQL advisory lock) prevents two dynos from processing the same queue.

3. Caption format and metadata extraction

Index: 001

Index: 002

Title: Class 01 Aman sir.mp4

Topic: Misc• Grammar Practice

Batch: English ( SSC MAINS )

Extracted By: @PUTINxINDIA

Field	Use
`Title`	Drive file name (`Class 01 Aman sir.mp4`)
`Topic`	Drive folder (`Misc• Grammar Practice`)
`Index`	All values kept in order (`["001", "002"]`); never used as the file name. The Telegram message ID is the unique identity.
`Batch`	Stored; optional `Batch/Topic` folders
`Extracted By`	Stored in the record

Parsing rules:

Fields can appear in any order.
Labels are case-insensitive.
Blank lines, emoji or bullet prefixes, extra spaces and full-width colons are allowed.
Unicode text is preserved.
Parsing is line-based, so one field never swallows the next.
A flattened single-line caption is split at each known label.

Missing data never makes a video silently disappear:

Case	Behaviour
No Topic	Folder `Uncategorized` (`FALLBACK_TOPIC_FOLDER`)
No Title	Telegram file name, else `telegram_<message id>.<ext>`. A warning is recorded. `MISSING_TITLE_POLICY=fail` marks it FAILED instead.
Title without extension	Extension taken from the real file's magic bytes, else from the MIME type
Title extension ≠ real container	Kept as written, with a warning. `EXTENSION_MISMATCH_POLICY=append` appends the real one.
`/` `\` or control characters	Replaced (`STRICT_FILENAMES=true` also replaces `<>:"\|?*`)

4. Google Drive topic-folder organization

<GOOGLE_DRIVE_ROOT_FOLDER_ID>/
├── Misc• Grammar Practice/
│   ├── Class 01 Aman sir.mp4
│   └── Class 02 Aman sir.mp4
├── Error Detection/
│   └── Class 01 Aman sir.mp4
└── Uncategorized/

Folder lookup goes database cache → Drive search → create. If duplicate folders already exist, the oldest is reused. If a folder-create response is lost, the bot searches again before retrying, so it never creates a second copy. Set BATCH_SUBFOLDERS_ENABLED=true for Root/Batch/Topic/.

Duplicate policy (DUPLICATE_POLICY) applies when a different file already has the target name in the folder:

Value	Result
`suffix_number` (default)	`Class 01 Aman sir (2).mp4`
`suffix_message_id`	`Class 01 Aman sir [1234].mp4`
`skip`	Item marked SKIPPED with the reason
`fail`	Item marked FAILED

An existing file is reused, not re-uploaded, only when its identity is verified. That means either its hidden tag matches this Telegram message, or it has the same size and MD5 (REUSE_IDENTICAL_FILES=true). The same Telegram media re-posted under a new message ID is SKIPPED by default (DUPLICATE_MEDIA_POLICY).

5. Prerequisites

A Telegram account, plus API credentials from https://my.telegram.org.
A Telegram bot from @BotFather.
A Google account with a Google Cloud project (free) with the Drive API enabled.
A local computer with Python 3.12 for two one-time interactive logins. Telegram and Google logins need a human, so they can't run on Heroku.
A Heroku account with billing enabled. The worker dyno and Heroku Postgres are paid resources; check current pricing on Heroku.
Optional: the Heroku CLI and the GitHub CLI.

6. One-click Heroku deployment

The button (top of this page) opens Heroku's deploy form. It:

creates the app from this repository's app.json;
builds it with the official Python buildpack;
provisions Heroku Postgres (essential-0), which sets DATABASE_URL;
starts one basic worker dyno (worker: python main.py run); there is no web process.

It is not fully one-click. Two secrets must be created on your computer first, because they need interactive logins:

TELEGRAM_SESSION_STRING, only for TELEGRAM_MODE=user (see §7).
GOOGLE_OAUTH_TOKEN_JSON or GOOGLE_SERVICE_ACCOUNT_JSON (see §9).

Prepare those, click the button, fill the form, then press Deploy app. If add-on provisioning fails (for example because billing isn't verified or the plan isn't available), create the app anyway and follow §11 manually. The worker refuses to start on Heroku without a database rather than silently losing progress.

Manual alternative with the CLI (from the repository root):

heroku create YOUR_APP_NAME --stack heroku-24
heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME
heroku config:set TELEGRAM_API_ID=1234567 TELEGRAM_API_HASH=your_api_hash \
  TELEGRAM_BOT_TOKEN=123456:your-bot-token ADMIN_USER_IDS=111111111 \
  SOURCE_CHANNEL_ID=@your_channel --app YOUR_APP_NAME
heroku config:set TELEGRAM_SESSION_STRING="$(cat data/sessions/session_string.txt)" --app YOUR_APP_NAME
heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME
git push heroku main
heroku ps:scale worker=1 --app YOUR_APP_NAME

Publishing to GitHub

This directory (the one containing Procfile and app.json) must be the repository root.

cd telegram_drive_bot
git init -b main
git add .
git status            # confirm: no .env, credentials/, data/, *.session, downloads/
git commit -m "Telegram to Google Drive organizer"
gh repo create YOUR_REPOSITORY_NAME --public --source=. --push
# or create the repo on github.com, then:
# git remote add origin https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME.git && git push -u origin main

Then put that repository URL into the button link at the top of this README, commit, and push. The Heroku button only works with public repositories.

7. Telegram bot setup

API ID / hash: log in at https://my.telegram.org → API development tools → create an app. Copy api_id → TELEGRAM_API_ID and api_hash → TELEGRAM_API_HASH. These are needed in both modes, because the bot talks MTProto through Telethon.
Bot: talk to @BotFather → /newbot → copy the token → TELEGRAM_BOT_TOKEN. Open your bot and press Start. A bot can't message you until you do, so you won't get progress messages otherwise.
Admin IDs: message @userinfobot to get your numeric ID → ADMIN_USER_IDS (comma-separate several). Only these users can run commands.
Source channel (SOURCE_CHANNEL_ID): any of these forms work:
- a public channel: @channelname or https://t.me/channelname;
- a private channel: the numeric ID -100…, or a post link https://t.me/c/1234567890/55 (use Copy Post Link in Telegram);
- forwarding a post to @userinfobot also shows the channel ID.
Choose a mode (TELEGRAM_MODE):
- user (default): reads the channel as your account, which must be a member. Create the session string locally:
```
python main.py login --session-string
```
  Telethon asks for your phone number, the login code and, only if enabled, your 2FA password. None of these are stored. The resulting session string is written to data/sessions/session_string.txt (mode 600). Copy it into TELEGRAM_SESSION_STRING, then delete the file.
- bot: add the bot to the channel as an administrator. No session string is needed.

8. Historical channel access requirements

Capability	`user` mode	`bot` mode
New posts (live)	✅	✅ (bot must be a channel admin)
Existing history	✅ full history the account can see	⚠️ best effort: bots can't list history, so message IDs are probed in batches until `BOT_SCAN_MAX_EMPTY` consecutive gaps (or up to `BOT_SCAN_END_ID`)
Large files	✅ up to Telegram's file limit	✅ uses MTProto, so the HTTP Bot API's 20 MB download limit does not apply
Private channel by numeric ID	✅	⚠️ the bot learns the channel's access key from its first live post; the key is then stored in the database
Extra secret needed	`TELEGRAM_SESSION_STRING`	none

PROCESS_MODE controls what is queued:

existing: history only (from START_MESSAGE_ID / START_DATE).
new: only posts after the first start.
both (default).

The bot never deletes or edits Telegram messages and doesn't bypass channel restrictions. A video in a channel with content protection (no forwarding/saving) may not be downloadable.

9. Google Drive authentication

Pick one.

Option A — OAuth (personal Google Drive, recommended)

In the Google Cloud Console: create a project, then enable the Google Drive API.
Configure the OAuth consent screen: External, add yourself as a test user. Then set the publishing status to In production. In Testing status, Google expires refresh tokens after 7 days and the bot would lose Drive access weekly. With the default drive.file scope no verification is required. With the full drive scope Google shows an "unverified app" screen during your own login (Advanced → continue).
Under Credentials, create an OAuth client ID of type Desktop app. Download the JSON to credentials/client_secret.json in this project.
Locally: python main.py drive-auth. This opens a browser; use --no-browser to get a URL to open yourself. It writes credentials/token.json (mode 600).
Heroku: heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME. Multi-line JSON works through the CLI. In the dashboard you can paste raw JSON, or a single-line base64 version: base64 < credentials/token.json | tr -d '\n'.

Scope and root folder. GOOGLE_DRIVE_SCOPE=drive.file (least privilege) lets the bot see only files and folders it created. So either:

let the bot create the root folder: python main.py drive-init --name "Telegram Videos" prints the ID for GOOGLE_DRIVE_ROOT_FOLDER_ID; or
use a folder you created yourself: set GOOGLE_DRIVE_SCOPE=drive before running drive-auth.

The folder ID is the part after /folders/ in its URL. The bot checks at startup that it can see the folder and stops with an explanation if it can't.

Option B — Service account (Google Workspace Shared Drive)

A service account has no storage quota in a personal My Drive. Uploads into a folder that a normal Gmail user merely shared with it fail with storageQuotaExceeded. Use it only with a Shared Drive:

Create a service account, then create a JSON key for it.
In the Shared Drive, Manage members → add the service account's client_email as Content manager.
Set:
- GOOGLE_AUTH_MODE=service_account
- GOOGLE_SERVICE_ACCOUNT_JSON="$(cat key.json)"
- DRIVE_SHARED_DRIVE_ID=<shared drive id>
- GOOGLE_DRIVE_ROOT_FOLDER_ID=<folder id inside it>
- GOOGLE_DRIVE_SCOPE=drive (needed to use a folder the service account did not create).

Credentials from environment variables are parsed in memory. Nothing is written to the dyno's disk, and they are validated by an authenticated Drive call at startup. Uploaded files are private unless you set DRIVE_SHARE_MODE=anyone_with_link.

10. Required Heroku Config Vars

Set these in the dashboard (Settings → Reveal Config Vars) or with heroku config:set NAME=value --app YOUR_APP_NAME.

Variable	Required	Notes
`TELEGRAM_API_ID`, `TELEGRAM_API_HASH`	yes	my.telegram.org
`TELEGRAM_BOT_TOKEN`	yes	@BotFather; command interface (and channel access in bot mode)
`ADMIN_USER_IDS`	yes	numeric IDs, comma-separated
`SOURCE_CHANNEL_ID`	yes	`@name`, link, or `-100…`
`TELEGRAM_MODE`	yes (default `user`)	`user` or `bot`
`TELEGRAM_SESSION_STRING`	when `TELEGRAM_MODE=user`	from `main.py login --session-string`
`GOOGLE_AUTH_MODE`	yes (default `oauth`)	`oauth` or `service_account`
`GOOGLE_OAUTH_TOKEN_JSON`	when `oauth`	contents of `credentials/token.json`
`GOOGLE_SERVICE_ACCOUNT_JSON`	when `service_account`	key JSON
`GOOGLE_DRIVE_ROOT_FOLDER_ID`	recommended	empty = top level of My Drive
`GOOGLE_DRIVE_SCOPE`	no (`drive.file`)	must match the scope used in `drive-auth`
`DATABASE_URL`	yes on Heroku	set automatically by Heroku Postgres
`PROCESS_MODE`	no (`both`)	`existing` / `new` / `both`
`DUPLICATE_POLICY`	no (`suffix_number`)	see §4
`BATCH_SUBFOLDERS_ENABLED`	no (`false`)	`Root/Batch/Topic`
`MAX_RETRIES` / `RETRY_DELAY`	no (`4` / `30`)	automatic retries / base delay in seconds
`PROGRESS_INTERVAL`	no (`10`)	seconds between progress-message edits
`DOWNLOAD_DIR`	no (`/tmp/telegram_drive_bot` in app.json)	temporary files
`UPLOAD_CHUNK_SIZE`	no (`8MB` in app.json)	multiple of 256KB; memory per chunk
`AUTO_START`	no (`false`)	start processing on every boot

Step-by-step instructions for obtaining every value are in ENV_SETUP.md. Every other supported variable is listed with an explanation in .env.example.

11. Database provisioning and configuration

Production state lives in PostgreSQL:

video records with all caption metadata;
states, attempts, errors, timestamps and Drive file IDs;
upload session URIs;
the folder map;
discovery and completion cursors;
duplicate-detection data;
the transition history;
the pause/run/stop choice.
Via the button: app.json requests heroku-postgresql:essential-0, a paid plan.
Manually: heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME (or any plan, or any PostgreSQL reachable through DATABASE_URL).
The schema is created and migrated automatically at startup. The version is tracked in the kv table.
TLS (sslmode=require) is added automatically for remote hosts.
Without DATABASE_URL on Heroku the worker refuses to start. SQLite on a dyno would be wiped at every restart, and the bot would re-upload everything. To accept that risk explicitly, set ALLOW_EPHEMERAL_DATABASE=true; a warning is logged at every start.
Locally, without DATABASE_URL, SQLite at data/bot.db is used. This is for development only.

12. First deployment and startup

Deploy (button or CLI). Watch the logs: heroku logs --tail --app YOUR_APP_NAME. You should see the database backend, Google Drive credentials validated, and Source channel resolved.
The bot messages each admin: Bot online (PostgreSQL); send /sync then /run to start.
Send /sync to discover channel videos, then /run. Watch /progress.
After /run, the choice is stored. After any dyno restart the bot syncs and resumes on its own. /pause and /stop are also remembered.

13. Telegram commands

Send these to the bot in a private chat. Users not in ADMIN_USER_IDS are refused.

Command	Effect
`/start`, `/help`	Instructions
`/status`	Queue state, the current video, live progress
`/progress`	Processed / pending / retrying / failed / skipped / remaining counts
`/sync`	Discover unseen messages and queue them (idempotent). `/sync all` rescans from `START_MESSAGE_ID`.
`/run`	Start processing
`/pause`	Stop the current transfer at the next chunk boundary (resumable) and wait
`/resume`	Continue
`/stop`	Checkpoint and end the worker until `/run`
`/retry`	Re-queue FAILED items with a fresh retry budget
`/failed`	Failed items with short error summaries
`/folders`	Folder count and uploads per topic
`/check ID`	Full record for one Telegram message ID

Progress is a single message, edited every PROGRESS_INTERVAL seconds:

Processing video 12 of 150
Topic: Misc• Grammar Practice
Title: Class 01 Aman sir.mp4
Status: Uploading
Download: 100% (1.2 GB / 1.2 GB)
Upload: 45% (540.0 MB / 1.2 GB) · 18.3 MB/s · ETA 36s

14. Checking logs

heroku logs --tail --app YOUR_APP_NAME          # live
heroku logs -n 1500 --app YOUR_APP_NAME         # recent history
heroku ps --app YOUR_APP_NAME                   # dyno status

On Heroku, logs go to stdout only. Locally they also go to logs/bot.log, rotated at 10 MB. Tokens, hashes, session strings, private keys and the database URL are redacted from log output.

15. Restarting and redeploying

heroku ps:restart worker --app YOUR_APP_NAME
git push heroku main                 # or push to GitHub with automatic deploys enabled
heroku ps:scale worker=1 --app YOUR_APP_NAME

On SIGTERM, Heroku allows 30 seconds before it kills the process. The bot:

stops at the next download or upload chunk boundary;
saves state;
disconnects.

On the next start it:

returns interrupted downloads to PENDING;
checks Drive for any upload that may have finished, using the hidden tag;
re-downloads files lost with the old dyno's disk;
continues the saved upload session, when the re-downloaded file is byte-identical (MD5);
resumes processing if it was running.

Scaling: run exactly one worker. A second worker is blocked by a PostgreSQL advisory lock. It waits INSTANCE_LOCK_WAIT seconds (90 on Heroku, which covers deploy overlap), then exits with code 3. Scaling above 1 therefore only produces crash-looping dynos, not parallel processing. The pipeline is intentionally sequential.

16. Troubleshooting

Symptom	Fix
`DATABASE_URL is not set`	Attach Heroku Postgres (§11)
`TELEGRAM_SESSION_STRING is required…`	Run `python main.py login --session-string` locally (§7), or use `TELEGRAM_MODE=bot`
`Telegram user session is not authorized`	The session was revoked (e.g. Terminate all sessions); create a new session string
`Cannot resolve SOURCE_CHANNEL_ID`	Use `@username` for public channels; make sure the account/bot is a member/admin; in bot mode post once in the channel
`DRIVE_ROOT_FOLDER_ID is not visible…`	`drive.file` scope can't see folders you created: use `drive-init`, or switch to `GOOGLE_DRIVE_SCOPE=drive` and redo `drive-auth`
`Google authorization expired or was revoked`	Redo `drive-auth` and update `GOOGLE_OAUTH_TOKEN_JSON`; publish the consent screen to In production (§9)
`invalid_scope` on refresh	The token was created with another scope; delete `credentials/token.json` and run `drive-auth` again
`storageQuotaExceeded`	Drive is full, or you used a service account without a Shared Drive
`Not enough free disk space`	A video is larger than the dyno's free disk; lower `MIN_FREE_DISK_SPACE` with care, or process that video locally
No progress messages	Press Start in your bot chat; check `ADMIN_USER_IDS` / `NOTIFY_CHAT_IDS`
`Another worker is already processing`	`heroku ps:scale worker=1`; locally, stop the other process
Item stuck in FAILED	`/check ID` shows the error; fix the cause, then `/retry`
Long pause after `FloodWait`	Telegram rate limit; the bot waits the time Telegram asks for

17. Security and credential handling

All secrets come from environment variables / Config Vars; nothing is hard-coded.
.gitignore excludes .env, credentials/, token.json, *.session, session_string.txt, data/ and downloads/.
The session string and OAuth token are full-access credentials. If one leaks:
- Telegram: Settings → Devices → terminate the session;
- Google: https://myaccount.google.com/permissions.
Uploaded files stay private by default. Drive scope defaults to least privilege (drive.file).
Commands are admin-only and private-chat only. Replies never include credentials, local paths or stack traces.
Telegram login codes and 2FA passwords are entered interactively, locally, and never stored.

18. Known limitations and Heroku resource constraints

Ephemeral disk: each video is staged on the dyno's temporary filesystem. Heroku doesn't guarantee a specific capacity, and the disk is wiped on every restart (at least daily).
- The bot checks free space before each download and halts with a message if a video won't fit.
- Multi-GB videos may not fit on a dyno; run the bot locally for those.
Memory: one upload chunk (UPLOAD_CHUNK_SIZE) is held in RAM at a time. 8MB is comfortable on 512 MB dynos.
Daily restarts: in-flight transfers resume from their checkpoint. A partially downloaded file on the old dyno is lost and is downloaded again, but the Drive upload session is reused.
Bot mode: history is best effort (ID probing), and private channels need one live post before the numeric ID resolves.
Telegram: flood limits apply. Downloads go through Telegram's servers at whatever speed they allow.
Drive: resumable sessions expire after about a week. Older sessions are discarded and the upload restarts after a Drive check for an already-finished copy.
Captions edited after discovery are not re-read. Use /check ID to see what was stored.
Cost: the worker dyno and Postgres add-on are paid Heroku resources.

What has and has not been verified

Automated tests (171): run against SQLite and a real local PostgreSQL 16, with mocked Telegram and Google Drive.
Not tested: real Telegram logins and downloads, real Google Drive uploads, and an actual Heroku deployment. Follow the manual end-to-end procedure below before relying on it.

19. Local development

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
cp .env.example .env            # fill in values
python main.py login            # user mode: creates data/sessions/user.session
python main.py drive-auth       # creates credentials/token.json
python main.py check-config
python main.py run              # or ./start.sh

Other subcommands:

python main.py sync [--rescan]
python main.py process (headless: sync, process everything, exit)
python main.py status
python main.py drive-init --name "Telegram Videos"

Tests

pytest

pgserver from requirements-dev.txt starts a throwaway PostgreSQL so every database test runs on both backends. Without it, only SQLite is tested. The suite covers:

caption parsing (the supplied example, multiple Index lines, missing fields, Unicode, flattened captions);
naming and collisions;
Drive folder reuse and dedupe;
the resumable-upload protocol;
state transitions and duplicate messages;
download and upload failure recovery;
restart during upload, and restart with a wiped disk;
a successful upload followed by a database-write failure;
sequential processing, pause/resume and graceful shutdown;
admin-only commands;
configuration and Heroku validation, env-var credentials, log redaction;
app.json / Procfile consistency.

Manual end-to-end test (real integrations)

Create a private test channel. Post four videos:
- two with Topic: Test A;
- one with Topic: Test B;
- one with no caption. Give two of them the same Title.
Point the bot at a fresh Drive folder. Then /sync, /run, and watch /progress.
Check Drive:
- Test A/ holds two files, one with a (2) suffix if the titles matched;
- Test B/ holds one;
- Uncategorized/ holds the caption-less video;
- file sizes match Telegram's.
Kill the process (kill -9, or heroku ps:restart) during an upload of a large video. Restart it. Confirm the upload continues and Drive ends up with exactly one copy.
/stop, post a new video, /run: only the new video is processed.
Revoke Drive access temporarily and confirm the queue halts with a clear message. Restore access, then /resume.

27 KiB Raw Blame History