kulnoor/drive
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3ee8427dae5d04741cfebc1aad9e38d28b4c592d
drive/README.md
T
gosangam 3ee8427dae second commit
2026-10-18 23:52:30 +05:30

34 KiB
Raw Blame History

Telegram Channel → Google Drive Organizer

Deploy to Heroku

Before the button works: replace https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME in the line above with the URL of the public GitHub repository that contains this project (see Publishing to GitHub). That URL is the only value to change. The button flow has not been tested against a live repository yet. Test it once after publishing.

A long-running worker that reads video posts from a Telegram channel, downloads each video, and uploads it to Google Drive. You can process the whole channel (/sync + /run) or just a range of posts with the interactive /upload command (send a start link and an end link, then /confirm). The caption's Title becomes the file name and its Topic picks the Drive folder. Videos are processed one at a time. All progress is stored in a database, so the bot resumes after crashes, restarts and redeploys without uploading anything twice.

Contents

  1. Overview · 2. Features · 3. Caption format ·
  2. Drive organization · 5. Prerequisites ·
  3. One-click Heroku deployment · 7. Telegram setup ·
  4. Historical access · 9. Google Drive auth ·
  5. Config Vars · 11. Database ·
  6. First deployment · 13. Commands · 14. Logs ·
  7. Restart / redeploy · 16. Troubleshooting ·
  8. Security · 18. Limitations ·
  9. Local development

1. Overview

Telegram channel ──(Telethon / MTProto)──► queue worker (one video at a time) ──► Google Drive
                                                │                                (resumable upload,
Admin chat ◄── bot commands & progress ─────────┤                                 verified by size + MD5)
                                                ▼
                                   PostgreSQL (Heroku) / SQLite (local)

For each video: discover → parse caption → resolve the topic folder → download to a temp disk → upload (resumable) → verify size and MD5 in Drive → save the Drive file ID → mark COMPLETED → delete the temp file → next video.

2. Features

  • Caption parsing for Index (one or many), Title, Topic, Batch and Extracted By.
  • Topic folders are created once and reused. Optional Batch/Topic layout.
  • Files are never overwritten. Name collisions get (2), (3), … or [message id], or are skipped or failed, per your policy.
  • No duplicate uploads, even if the process dies just after Drive accepted the file. Every upload carries a hidden appProperties tag (channel:message) that is checked before uploading.
  • Resumable uploads that survive restarts. The upload session URI is stored in the database.
  • Resumable downloads, continuing from the last 1 MiB boundary of a partial file.
  • Explicit state machine: PENDING → DOWNLOADING → DOWNLOADED → UPLOADING → COMPLETED, plus RETRY_WAIT, FAILED and SKIPPED. Every transition is validated and logged in the transitions table.
  • Bounded retries with exponential backoff, honouring Telegram flood waits and Drive rate limits. Permission errors are not retried.
  • Disk-space checks before each download. A full disk, Drive quota or expired auth halts the queue cleanly instead of burning retries.
  • Admin-only Telegram commands, plus one progress message that is edited in place rather than spammed.
  • Interactive /upload tasks: pick an inclusive message range by pasting two post links, confirm, and watch a live task progress message. Tasks are stored in the database and survive restarts.
  • Automatic temp-file cleanup: each video's local file is deleted right after its upload is verified in Drive and recorded in the database. Abandoned temp files are swept at startup, periodically, and whenever disk space runs low.
  • Graceful shutdown on SIGTERM (Heroku restarts) at a resumable checkpoint.
  • A single-worker guard (PostgreSQL advisory lock) prevents two dynos from processing the same queue.

3. Caption format and metadata extraction

Index: 001

Index: 002

Title: Class 01 Aman sir.mp4

Topic: Misc• Grammar Practice

Batch: English ( SSC MAINS )

Extracted By: @PUTINxINDIA
Field Use
Title Drive file name (Class 01 Aman sir.mp4)
Topic Drive folder (Misc• Grammar Practice)
Index All values kept in order (["001", "002"]); never used as the file name. The Telegram message ID is the unique identity.
Batch Stored; optional Batch/Topic folders
Extracted By Stored in the record

Parsing rules:

  • Fields can appear in any order.
  • Labels are case-insensitive.
  • Blank lines, emoji or bullet prefixes, extra spaces and full-width colons are allowed.
  • Unicode text is preserved.
  • Parsing is line-based, so one field never swallows the next.
  • A flattened single-line caption is split at each known label.

Missing data never makes a video silently disappear:

Case Behaviour
No Topic Folder Uncategorized (FALLBACK_TOPIC_FOLDER)
No Title Telegram file name, else telegram_<message id>.<ext>. A warning is recorded. MISSING_TITLE_POLICY=fail marks it FAILED instead.
Title without extension Extension taken from the real file's magic bytes, else from the MIME type
Title extension ≠ real container Kept as written, with a warning. EXTENSION_MISMATCH_POLICY=append appends the real one.
/ \ or control characters Replaced (STRICT_FILENAMES=true also replaces <>:"|?*)

4. Google Drive topic-folder organization

<GOOGLE_DRIVE_ROOT_FOLDER_ID>/
├── Misc• Grammar Practice/
│   ├── Class 01 Aman sir.mp4
│   └── Class 02 Aman sir.mp4
├── Error Detection/
│   └── Class 01 Aman sir.mp4
└── Uncategorized/

Folder lookup goes database cache → Drive search → create. If duplicate folders already exist, the oldest is reused. If a folder-create response is lost, the bot searches again before retrying, so it never creates a second copy. Set BATCH_SUBFOLDERS_ENABLED=true for Root/Batch/Topic/.

Duplicate policy (DUPLICATE_POLICY) applies when a different file already has the target name in the folder:

Value Result
suffix_number (default) Class 01 Aman sir (2).mp4
suffix_message_id Class 01 Aman sir [1234].mp4
skip Item marked SKIPPED with the reason
fail Item marked FAILED

An existing file is reused, not re-uploaded, only when its identity is verified. That means either its hidden tag matches this Telegram message, or it has the same size and MD5 (REUSE_IDENTICAL_FILES=true). The same Telegram media re-posted under a new message ID is SKIPPED by default (DUPLICATE_MEDIA_POLICY).

5. Prerequisites

  • A Telegram account, plus API credentials from https://my.telegram.org.
  • A Telegram bot from @BotFather.
  • A Google account with a Google Cloud project (free) with the Drive API enabled.
  • A local computer with Python 3.12 for two one-time interactive logins. Telegram and Google logins need a human, so they can't run on Heroku.
  • A Heroku account with billing enabled. The worker dyno and Heroku Postgres are paid resources; check current pricing on Heroku.
  • Optional: the Heroku CLI and the GitHub CLI.

6. One-click Heroku deployment

The button (top of this page) opens Heroku's deploy form. It:

  • creates the app from this repository's app.json;
  • builds it with the official Python buildpack;
  • provisions Heroku Postgres (essential-0), which sets DATABASE_URL;
  • starts one basic worker dyno (worker: python main.py run); there is no web process.

It is not fully one-click. Two secrets must be created on your computer first, because they need interactive logins:

  1. TELEGRAM_SESSION_STRING, only for TELEGRAM_MODE=user (see §7).
  2. GOOGLE_OAUTH_TOKEN_JSON or GOOGLE_SERVICE_ACCOUNT_JSON (see §9).

Prepare those, click the button, fill the form, then press Deploy app. If add-on provisioning fails (for example because billing isn't verified or the plan isn't available), create the app anyway and follow §11 manually. The worker refuses to start on Heroku without a database rather than silently losing progress.

Manual alternative with the CLI (from the repository root):

heroku create YOUR_APP_NAME --stack heroku-24
heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME
heroku config:set TELEGRAM_API_ID=1234567 TELEGRAM_API_HASH=your_api_hash \
  TELEGRAM_BOT_TOKEN=123456:your-bot-token ADMIN_USER_IDS=111111111 \
  SOURCE_CHANNEL_ID=@your_channel --app YOUR_APP_NAME
heroku config:set TELEGRAM_SESSION_STRING="$(cat data/sessions/session_string.txt)" --app YOUR_APP_NAME
heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME
git push heroku main
heroku ps:scale worker=1 --app YOUR_APP_NAME

Publishing to GitHub

This directory (the one containing Procfile and app.json) must be the repository root.

cd telegram_drive_bot
git init -b main
git add .
git status            # confirm: no .env, credentials/, data/, *.session, downloads/
git commit -m "Telegram to Google Drive organizer"
gh repo create YOUR_REPOSITORY_NAME --public --source=. --push
# or create the repo on github.com, then:
# git remote add origin https://github.com/YOUR_GITHUB_USERNAME/YOUR_REPOSITORY_NAME.git && git push -u origin main

Then put that repository URL into the button link at the top of this README, commit, and push. The Heroku button only works with public repositories.

7. Telegram bot setup

  1. API ID / hash: log in at https://my.telegram.org → API development tools → create an app. Copy api_id → TELEGRAM_API_ID and api_hash → TELEGRAM_API_HASH. These are needed in both modes, because the bot talks MTProto through Telethon.
  2. Bot: talk to @BotFather → /newbot → copy the token → TELEGRAM_BOT_TOKEN. Open your bot and press Start. A bot can't message you until you do, so you won't get progress messages otherwise.
  3. Admin IDs: message @userinfobot to get your numeric ID → ADMIN_USER_IDS (comma-separate several). Only these users can run commands.
  4. Source channel (SOURCE_CHANNEL_ID): any of these forms work:
    • a public channel: @channelname or https://t.me/channelname;
    • a private channel: the numeric ID -100…, or a post link https://t.me/c/1234567890/55 (use Copy Post Link in Telegram);
    • forwarding a post to @userinfobot also shows the channel ID.
  5. Choose a mode (TELEGRAM_MODE):
    • user (default): reads the channel as your account, which must be a member. Create the session string locally: 
      python main.py login --session-string
      Telethon asks for your phone number, the login code and, only if enabled, your 2FA password. None of these are stored. The resulting session string is written to data/sessions/session_string.txt (mode 600). Copy it into TELEGRAM_SESSION_STRING, then delete the file.
    • bot: add the bot to the channel as an administrator. No session string is needed.

8. Historical channel access requirements

Capability user mode bot mode
New posts (live) ✅ ✅ (bot must be a channel admin)
Existing history ✅ full history the account can see ⚠️ best effort: bots can't list history, so message IDs are probed in batches until BOT_SCAN_MAX_EMPTY consecutive gaps (or up to BOT_SCAN_END_ID)
/upload message ranges ✅ ✅ messages are fetched by ID, which bots that are channel admins may do
Large files ✅ up to Telegram's file limit ✅ uses MTProto, so the HTTP Bot API's 20 MB download limit does not apply
Private channel by numeric ID ✅ ⚠️ the bot learns the channel's access key from its first live post; the key is then stored in the database
Extra secret needed TELEGRAM_SESSION_STRING none

PROCESS_MODE controls what is queued:

  • existing: history only (from START_MESSAGE_ID / START_DATE).
  • new: only posts after the first start.
  • both (default).

The bot never deletes or edits Telegram messages and doesn't bypass channel restrictions. A video in a channel with content protection (no forwarding/saving) may not be downloadable.

9. Google Drive authentication

Pick one.

Option A — OAuth (personal Google Drive, recommended)

  1. In the Google Cloud Console: create a project, then enable the Google Drive API.
  2. Configure the OAuth consent screen: External, add yourself as a test user. Then set the publishing status to In production. In Testing status, Google expires refresh tokens after 7 days and the bot would lose Drive access weekly. With the default drive.file scope no verification is required. With the full drive scope Google shows an "unverified app" screen during your own login (Advanced → continue).
  3. Under Credentials, create an OAuth client ID of type Desktop app. Download the JSON to credentials/client_secret.json in this project.
  4. Locally: python main.py drive-auth. This opens a browser; use --no-browser to get a URL to open yourself. It writes credentials/token.json (mode 600).
  5. Heroku: heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME. Multi-line JSON works through the CLI. In the dashboard you can paste raw JSON, or a single-line base64 version: base64 < credentials/token.json | tr -d '\n'.

Scope and root folder. GOOGLE_DRIVE_SCOPE=drive.file (least privilege) lets the bot see only files and folders it created. So either:

  • let the bot create the root folder: python main.py drive-init --name "Telegram Videos" prints the ID for GOOGLE_DRIVE_ROOT_FOLDER_ID; or
  • use a folder you created yourself: set GOOGLE_DRIVE_SCOPE=drive before running drive-auth.

The folder ID is the part after /folders/ in its URL. The bot checks at startup that it can see the folder and stops with an explanation if it can't.

Option B — Service account (Google Workspace Shared Drive)

A service account has no storage quota in a personal My Drive. Uploads into a folder that a normal Gmail user merely shared with it fail with storageQuotaExceeded. Use it only with a Shared Drive:

  1. Create a service account, then create a JSON key for it.
  2. In the Shared Drive, Manage members → add the service account's client_email as Content manager.
  3. Set:
    • GOOGLE_AUTH_MODE=service_account
    • GOOGLE_SERVICE_ACCOUNT_JSON="$(cat key.json)"
    • DRIVE_SHARED_DRIVE_ID=<shared drive id>
    • GOOGLE_DRIVE_ROOT_FOLDER_ID=<folder id inside it>
    • GOOGLE_DRIVE_SCOPE=drive (needed to use a folder the service account did not create).

Credentials from environment variables are parsed in memory. Nothing is written to the dyno's disk, and they are validated by an authenticated Drive call at startup. Uploaded files are private unless you set DRIVE_SHARE_MODE=anyone_with_link.

10. Required Heroku Config Vars

Set these in the dashboard (Settings → Reveal Config Vars) or with heroku config:set NAME=value --app YOUR_APP_NAME.

Variable Required Notes
TELEGRAM_API_ID, TELEGRAM_API_HASH yes my.telegram.org
TELEGRAM_BOT_TOKEN yes @BotFather; command interface (and channel access in bot mode)
ADMIN_USER_IDS yes numeric IDs, comma-separated
SOURCE_CHANNEL_ID yes @name, link, or -100…
TELEGRAM_MODE yes (default user) user or bot
TELEGRAM_SESSION_STRING when TELEGRAM_MODE=user from main.py login --session-string
GOOGLE_AUTH_MODE yes (default oauth) oauth or service_account
GOOGLE_OAUTH_TOKEN_JSON when oauth contents of credentials/token.json
GOOGLE_SERVICE_ACCOUNT_JSON when service_account key JSON
GOOGLE_DRIVE_ROOT_FOLDER_ID recommended empty = top level of My Drive
GOOGLE_DRIVE_SCOPE no (drive.file) must match the scope used in drive-auth
DATABASE_URL yes on Heroku set automatically by Heroku Postgres
PROCESS_MODE no (both) existing / new / both
DUPLICATE_POLICY no (suffix_number) see §4
BATCH_SUBFOLDERS_ENABLED no (false) Root/Batch/Topic
MAX_RETRIES / RETRY_DELAY no (4 / 30) automatic retries / base delay in seconds
PROGRESS_INTERVAL no (10) seconds between progress-message edits
DOWNLOAD_DIR no (/tmp/telegram_drive_bot in app.json) temporary files
UPLOAD_CHUNK_SIZE no (8MB in app.json) multiple of 256KB; memory per chunk
AUTO_START no (false) start processing on every boot
UPLOAD_RESTRICT_TO_SOURCE_CHANNEL no (true) /upload links must point to SOURCE_CHANNEL_ID
UPLOAD_SETUP_TIMEOUT no (30) minutes before an unfinished /upload setup expires
UPLOAD_MAX_RANGE no (5000) largest number of message IDs per task
KEEP_FAILED_DOWNLOADS no (false) keep local files of FAILED videos
TEMP_CLEANUP_INTERVAL no (3600) seconds between abandoned-temp-file sweeps

Step-by-step instructions for obtaining every value are in ENV_SETUP.md. Every other supported variable is listed with an explanation in .env.example.

11. Database provisioning and configuration

Production state lives in PostgreSQL:

  • video records with all caption metadata;

  • states, attempts, errors, timestamps and Drive file IDs;

  • upload session URIs;

  • the folder map;

  • discovery and completion cursors;

  • duplicate-detection data;

  • the transition history;

  • the pause/run/stop choice.

  • Via the button: app.json requests heroku-postgresql:essential-0, a paid plan.

  • Manually: heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME (or any plan, or any PostgreSQL reachable through DATABASE_URL).

  • The schema is created and migrated automatically at startup. The version is tracked in the kv table.

  • TLS (sslmode=require) is added automatically for remote hosts.

  • Without DATABASE_URL on Heroku the worker refuses to start. SQLite on a dyno would be wiped at every restart, and the bot would re-upload everything. To accept that risk explicitly, set ALLOW_EPHEMERAL_DATABASE=true; a warning is logged at every start.

  • Locally, without DATABASE_URL, SQLite at data/bot.db is used. This is for development only.

12. First deployment and startup

  1. Deploy (button or CLI). Watch the logs: heroku logs --tail --app YOUR_APP_NAME. You should see the database backend, Google Drive credentials validated, and Source channel resolved.
  2. The bot messages each admin: Bot online (PostgreSQL); send /sync then /run to start.
  3. Either send /upload to process a message range (see §13), or send /sync to discover all channel videos and then /run. Watch /taskstatus or /progress.
  4. Running tasks and the /run choice are stored. After any dyno restart the bot resumes on its own. /pause and /stop are also remembered.

13. Telegram commands

Send these to the bot in a private chat. Users not in ADMIN_USER_IDS are refused.

Command Effect
/start, /help Instructions
/status Queue state, the current video, live progress
/progress Processed / pending / retrying / failed / skipped / remaining counts
/sync Discover unseen messages and queue them (idempotent). /sync all rescans from START_MESSAGE_ID.
/run Start processing
/pause Stop the current transfer at the next chunk boundary (resumable) and wait
/resume Continue
/stop Checkpoint and end the worker until /run
/retry Re-queue FAILED items with a fresh retry budget
/failed Failed items with short error summaries
/folders Folder count and uploads per topic
/check ID Full record for one Telegram message ID
/upload Start an interactive upload task for a message range
/confirm Start the prepared task (only after both links were validated)
/cancel Discard your setup, or cancel your active task at a safe checkpoint. /cancel ID targets a specific task.
/taskstatus Your setup step, plus active tasks with range, counters and live progress. /taskstatus ID for one task.
/uploadhistory The 10 most recent tasks and their results

/pause, /resume and /stop also apply to upload tasks. /stop parks a running task as PAUSED; /resume continues it.

Processing a message range with /upload

  1. Send /upload. The bot asks for the start link: in Telegram, right-click (or long-press) a post → Copy Post Link, e.g. https://t.me/c/3909337424/14.
  2. Send the link. The bot asks for the end link, e.g. https://t.me/c/3909337424/20.
  3. The bot shows Upload Task Ready with both message IDs and the channel.
  4. Send /confirm to start, or /cancel to discard. Nothing is downloaded before /confirm.

Link rules:

  • Accepted formats: t.me/c/<id>/<msg> (private), t.me/<username>/<msg> (public), and t.me/s/<username>/<msg>. www., telegram.me, a trailing slash and ?single are fine.
  • Invite links and links without a message ID are rejected with an explanation.
  • The /c/ number is Telegram's internal channel ID. It is converted to the API ID -(10¹² + id) and then resolved through the bot's Telegram account. A channel that account can't access is refused; nothing bypasses membership or permissions.
  • Both links must be from the same channel, the end ID must be ≥ the start ID, the range is limited by UPLOAD_MAX_RANGE, and both messages must exist.
  • By default the links must point to SOURCE_CHANNEL_ID. Set UPLOAD_RESTRICT_TO_SOURCE_CHANNEL=false to allow any channel the account (or admin bot) can read.
  • If the end link is invalid, the start link is kept; just send another end link.

How a task runs:

  • The range is inclusive (14–20 means 14, 15, …, 20), processed in ascending order, one video at a time.
  • Text, photos, stickers and deleted messages are counted as skipped, never as failures.
  • Videos already uploaded earlier are counted as duplicates and are not uploaded again. Repeating a finished task therefore uploads nothing new.
  • Caption metadata (Title, Topic, …) is used exactly as for whole-channel processing.
  • Each local file is deleted only after the Drive upload is verified and its COMPLETED state is committed.
  • Failed videos are recorded, never marked completed, and can be re-queued with /retry. Their local file is deleted (unless KEEP_FAILED_DOWNLOADS=true) because it can be downloaded again.
  • /cancel during processing stops at the next safe checkpoint:
    • finished uploads stay in Drive;
    • the remaining videos are marked cancelled and their temp files are removed;
    • a later task covering the same range picks the cancelled videos up again.
  • Setups are per administrator. Two admins can prepare tasks at the same time without affecting each other. A second /upload never overwrites your unfinished setup. A setup expires after UPLOAD_SETUP_TIMEOUT minutes.
  • Confirmed tasks run one after another in the order they were confirmed. Running and queued tasks resume automatically after a restart or redeploy.

Task progress is one message, edited in place:

🚀 Upload Task Running

Task: #12
Range: 14–20
Current Message: 16
Current Video: Class 01 Aman sir.mp4
Topic: Misc• Grammar Practice
Status: Uploading to Google Drive
Upload: 45% (540.0 MB / 1.2 GB) · 18.3 MB/s · ETA 36s

Completed: 1
Skipped: 1
Failed: 0

When the task finishes, the bot sends a summary covering:

  • the range;
  • messages examined;
  • eligible videos;
  • uploads, duplicates, non-video messages and failures;
  • temp files deleted;
  • the Drive folders used.

For whole-channel processing, progress is a single message, edited every PROGRESS_INTERVAL seconds:

Processing video 12 of 150
Topic: Misc• Grammar Practice
Title: Class 01 Aman sir.mp4
Status: Uploading
Download: 100% (1.2 GB / 1.2 GB)
Upload: 45% (540.0 MB / 1.2 GB) · 18.3 MB/s · ETA 36s

14. Checking logs

heroku logs --tail --app YOUR_APP_NAME          # live
heroku logs -n 1500 --app YOUR_APP_NAME         # recent history
heroku ps --app YOUR_APP_NAME                   # dyno status

On Heroku, logs go to stdout only. Locally they also go to logs/bot.log, rotated at 10 MB. Tokens, hashes, session strings, private keys and the database URL are redacted from log output.

15. Restarting and redeploying

heroku ps:restart worker --app YOUR_APP_NAME
git push heroku main                 # or push to GitHub with automatic deploys enabled
heroku ps:scale worker=1 --app YOUR_APP_NAME

On SIGTERM, Heroku allows 30 seconds before it kills the process. The bot:

  1. stops at the next download or upload chunk boundary;
  2. saves state;
  3. disconnects.

On the next start it:

  1. returns interrupted downloads to PENDING;
  2. checks Drive for any upload that may have finished, using the hidden tag;
  3. re-downloads files lost with the old dyno's disk;
  4. continues the saved upload session, when the re-downloaded file is byte-identical (MD5);
  5. resumes processing if it was running.

Scaling: run exactly one worker. A second worker is blocked by a PostgreSQL advisory lock. It waits INSTANCE_LOCK_WAIT seconds (90 on Heroku, which covers deploy overlap), then exits with code 3. Scaling above 1 therefore only produces crash-looping dynos, not parallel processing. The pipeline is intentionally sequential.

16. Troubleshooting

Symptom Fix
DATABASE_URL is not set Attach Heroku Postgres (§11)
TELEGRAM_SESSION_STRING is required… Run python main.py login --session-string locally (§7), or use TELEGRAM_MODE=bot
Telegram user session is not authorized The session was revoked (e.g. Terminate all sessions); create a new session string
Cannot resolve SOURCE_CHANNEL_ID Use @username for public channels; make sure the account/bot is a member/admin; in bot mode post once in the channel
DRIVE_ROOT_FOLDER_ID is not visible… drive.file scope can't see folders you created: use drive-init, or switch to GOOGLE_DRIVE_SCOPE=drive and redo drive-auth
Google authorization expired or was revoked Redo drive-auth and update GOOGLE_OAUTH_TOKEN_JSON; publish the consent screen to In production (§9)
invalid_scope on refresh The token was created with another scope; delete credentials/token.json and run drive-auth again
storageQuotaExceeded Drive is full, or you used a service account without a Shared Drive
Not enough free disk space A video is larger than the dyno's free disk; lower MIN_FREE_DISK_SPACE with care, or process that video locally
No progress messages Press Start in your bot chat; check ADMIN_USER_IDS / NOTIFY_CHAT_IDS
Another worker is already processing heroku ps:scale worker=1; locally, stop the other process
Item stuck in FAILED /check ID shows the error; fix the cause, then /retry
Long pause after FloodWait Telegram rate limit; the bot waits the time Telegram asks for

17. Security and credential handling

  • All secrets come from environment variables / Config Vars; nothing is hard-coded.
  • .gitignore excludes .env, credentials/, token.json, *.session, session_string.txt, data/ and downloads/.
  • The session string and OAuth token are full-access credentials. If one leaks:
  • Uploaded files stay private by default. Drive scope defaults to least privilege (drive.file).
  • Commands are admin-only and private-chat only. Replies never include credentials, local paths or stack traces.
  • Telegram login codes and 2FA passwords are entered interactively, locally, and never stored.

18. Known limitations and Heroku resource constraints

  • Ephemeral disk: each video is staged on the dyno's temporary filesystem. Heroku doesn't guarantee a specific capacity, and the disk is wiped on every restart (at least daily).
    • The bot checks free space before each download and halts with a message if a video won't fit.
    • Multi-GB videos may not fit on a dyno; run the bot locally for those.
  • Temporary files: downloads live only in DOWNLOAD_DIR. The bot refuses to use /, /tmp, your home directory or the project directory itself. Cleanup only removes files named <channel>_<message>.part|.video directly inside that directory, never follows symlinks, and re-checks the database first. Files of unfinished items are kept for resuming.
  • Memory: one upload chunk (UPLOAD_CHUNK_SIZE) is held in RAM at a time. 8MB is comfortable on 512 MB dynos.
  • Daily restarts: in-flight transfers resume from their checkpoint. A partially downloaded file on the old dyno is lost and is downloaded again, but the Drive upload session is reused.
  • Bot mode: history is best effort (ID probing), and private channels need one live post before the numeric ID resolves.
  • Telegram: flood limits apply. Downloads go through Telegram's servers at whatever speed they allow.
  • Drive: resumable sessions expire after about a week. Older sessions are discarded and the upload restarts after a Drive check for an already-finished copy.
  • Captions edited after discovery are not re-read. Use /check ID to see what was stored.
  • Cost: the worker dyno and Postgres add-on are paid Heroku resources.

What has and has not been verified

  • Automated tests (259): run against SQLite and a real local PostgreSQL 16, with mocked Telegram and Google Drive.
  • Not tested: real Telegram logins and downloads, real Google Drive uploads, and an actual Heroku deployment. Follow the manual end-to-end procedure below before relying on it.

19. Local development

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt
cp .env.example .env            # fill in values
python main.py login            # user mode: creates data/sessions/user.session
python main.py drive-auth       # creates credentials/token.json
python main.py check-config
python main.py run              # or ./start.sh

Other subcommands:

  • python main.py sync [--rescan]
  • python main.py process (headless: sync, process everything, exit)
  • python main.py status
  • python main.py drive-init --name "Telegram Videos"

Tests

pytest

pgserver from requirements-dev.txt starts a throwaway PostgreSQL so every database test runs on both backends. Without it, only SQLite is tested. The suite covers:

  • caption parsing (the supplied example, multiple Index lines, missing fields, Unicode, flattened captions);
  • naming and collisions;
  • Drive folder reuse and dedupe;
  • the resumable-upload protocol;
  • state transitions and duplicate messages;
  • download and upload failure recovery;
  • restart during upload, and restart with a wiped disk;
  • a successful upload followed by a database-write failure;
  • sequential processing, pause/resume and graceful shutdown;
  • admin-only commands;
  • the /upload workflow:
    • link parsing and validation;
    • per-admin setups, persistence and expiry;
    • inclusive ascending ranges, and skipping non-video messages;
    • delete-only-after-verified-upload, and failed uploads never completed;
    • repeat tasks without duplicates;
    • cancellation;
    • crash/restart recovery and pause/resume;
  • temp-file cleanup safety;
  • configuration and Heroku validation, env-var credentials, log redaction;
  • app.json / Procfile consistency.

Manual end-to-end test (real integrations)

  1. Create a private test channel. Post four videos:
    • two with Topic: Test A;
    • one with Topic: Test B;
    • one with no caption. Give two of them the same Title.
  2. Point the bot at a fresh Drive folder. Then /sync, /run, and watch /progress.
  3. Check Drive:
    • Test A/ holds two files, one with a (2) suffix if the titles matched;
    • Test B/ holds one;
    • Uncategorized/ holds the caption-less video;
    • file sizes match Telegram's.
  4. Kill the process (kill -9, or heroku ps:restart) during an upload of a large video. Restart it. Confirm the upload continues and Drive ends up with exactly one copy.
  5. /stop, post a new video, /run: only the new video is processed.
  6. Revoke Drive access temporarily and confirm the queue halts with a clear message. Restore access, then /resume.
  7. /upload: send the link of the first test post, then the last one, then /confirm. Check that:
    • only that range is processed;
    • a text post inside the range is reported as a skipped non-video;
    • the summary counts are correct. Run the same range again and confirm it reports duplicates and uploads nothing.
  8. Start another /upload task and send /cancel while a video is downloading. Check that:
    • finished uploads remain in Drive;
    • downloads/ is empty.
Reference in New Issue View Git Blame Copy Permalink