first commit

2026-10-18 23:48:22 +05:30
commit 732a06f65d
38 changed files with 8943 additions and 0 deletions
@@ -0,0 +1,388 @@
+# How to Obtain Every Environment Variable
+
+This guide explains where each configuration value comes from and how to get it. Work through the
+sections in order: some steps (the Telegram session string and the Google token) need values from
+earlier steps.
+
+- **Local runs:** put the values in a `.env` file (`cp .env.example .env`).
+- **Heroku:** set them as Config Vars, either in the dashboard (*App → Settings → Reveal Config Vars*)
+  or with `heroku config:set NAME=value --app YOUR_APP_NAME`.
+
+> All values below are placeholders. Never commit real values or paste them into chats, issues or
+> screenshots. `.env`, `credentials/` and `data/` are already in `.gitignore`.
+
+---
+
+## Quick checklist
+
+| # | Variable | Required? | Where it comes from |
+|---|---|---|---|
+| 1 | `TELEGRAM_API_ID` | Always | my.telegram.org |
+| 2 | `TELEGRAM_API_HASH` | Always | my.telegram.org |
+| 3 | `TELEGRAM_BOT_TOKEN` | Always on Heroku (control commands); required in bot mode | @BotFather |
+| 4 | `ADMIN_USER_IDS` | When a bot token is set | @userinfobot |
+| 5 | `SOURCE_CHANNEL_ID` | Always | Channel username, link, or numeric ID |
+| 6 | `TELEGRAM_MODE` | Optional (default `user`) | Your choice: `user` or `bot` |
+| 7 | `TELEGRAM_SESSION_STRING` | `TELEGRAM_MODE=user` on Heroku | `python main.py login --session-string` |
+| 8 | `GOOGLE_AUTH_MODE` | Optional (default `oauth`) | Your choice: `oauth` or `service_account` |
+| 9 | `GOOGLE_OAUTH_TOKEN_JSON` | `oauth` on Heroku | `python main.py drive-auth` |
+| 10 | `GOOGLE_SERVICE_ACCOUNT_JSON` | `service_account` on Heroku | Google Cloud Console key |
+| 11 | `GOOGLE_DRIVE_ROOT_FOLDER_ID` | Recommended | Drive folder URL, or `python main.py drive-init` |
+| 12 | `GOOGLE_DRIVE_SCOPE` | Optional (default `drive.file`) | Your choice |
+| 13 | `DRIVE_SHARED_DRIVE_ID` | Shared Drive setups only | Shared Drive URL |
+| 14 | `DATABASE_URL` | Required on Heroku | Heroku Postgres (set automatically) |
+
+Everything else has a sensible default (see [Optional settings](#optional-settings)).
+
+---
+
+## Part 1 — Telegram
+
+### 1–2. `TELEGRAM_API_ID` and `TELEGRAM_API_HASH`
+
+These identify your Telegram API application. They are needed in **both** modes, because the bot talks
+to Telegram through Telethon (MTProto).
+
+1. Open <https://my.telegram.org> and log in with your phone number. Telegram sends a code to your
+   Telegram app, not by SMS.
+2. Click **API development tools**.
+3. If no app exists yet, fill in the form:
+   - *App title*: anything, for example `Drive Organizer`;
+   - *Short name*: 5–32 letters or digits, for example `driveorganizer`;
+   - *Platform*: `Desktop`;
+   - URL and description can stay empty.
+4. Click **Create application**.
+5. Copy the two values:
+   - **App api_id** (a number) → `TELEGRAM_API_ID`
+   - **App api_hash** (32 hex characters) → `TELEGRAM_API_HASH`
+
+```env
+TELEGRAM_API_ID=1234567
+TELEGRAM_API_HASH=0123456789abcdef0123456789abcdef
+```
+
+> The API hash is a secret. If my.telegram.org shows an "ERROR" when you create the app, try another
+> browser, disable ad blockers or VPNs, and use a short alphanumeric short name.
+
+### 3. `TELEGRAM_BOT_TOKEN`
+
+The bot provides the admin commands (`/run`, `/status`, …) and progress messages. In bot mode it also
+reads the channel.
+
+1. In Telegram, open [@BotFather](https://t.me/BotFather) and send `/newbot`.
+2. Enter a display name, then a username ending in `bot` (for example `my_drive_organizer_bot`).
+3. BotFather replies with a token like `123456789:AAH...`. Copy it → `TELEGRAM_BOT_TOKEN`.
+4. **Open your new bot and press Start.** Bots cannot message you until you do, so without this you
+   get no progress messages.
+
+```env
+TELEGRAM_BOT_TOKEN=123456789:replace-with-your-bot-token
+```
+
+If the token leaks, send `/revoke` to @BotFather to get a new one.
+
+### 4. `ADMIN_USER_IDS` (and optional `NOTIFY_CHAT_IDS`)
+
+Only these numeric Telegram user IDs may control the bot.
+
+1. Open [@userinfobot](https://t.me/userinfobot) and press Start.
+2. It replies with your **Id** (a number such as `111111111`). Copy it.
+3. For several admins, ask each person to do the same and separate the IDs with commas.
+
+```env
+ADMIN_USER_IDS=111111111,222222222
+```
+
+`NOTIFY_CHAT_IDS` is optional. It lists the chats that receive progress and error messages, and
+defaults to `ADMIN_USER_IDS`. Each person listed must have pressed Start in the bot.
+
+### 5. `SOURCE_CHANNEL_ID`
+
+Any of these formats is accepted:
+
+| Channel type | How to get it | Example value |
+|---|---|---|
+| Public channel | The `@username` shown in the channel info | `@my_channel` or `https://t.me/my_channel` |
+| Private channel: post link | Right-click (or long-press) any post → **Copy Post Link** | `https://t.me/c/1234567890/55` |
+| Private channel: numeric ID | Prefix the number from the post link with `-100` | `-1001234567890` |
+| Private channel: Telegram Web | Open the channel at <https://web.telegram.org/k/>; the URL ends with `#-100…` | `-1001234567890` |
+
+Another way: forward a channel post to [@JsonDumpBot](https://t.me/JsonDumpBot) and read
+`forward_from_chat.id`.
+
+```env
+SOURCE_CHANNEL_ID=-1001234567890
+```
+
+> Bot mode with a private channel: the bot must be a channel **administrator**. If the numeric ID
+> can't be resolved at the first start, post one message in the channel. The bot then learns the
+> channel and remembers it in the database.
+
+### 6. `TELEGRAM_MODE`
+
+This is a choice, not a credential.
+
+| Value | Use when | Needs |
+|---|---|---|
+| `user` (default) | You need the channel's **existing history**, or you are a member but not an admin | `TELEGRAM_SESSION_STRING` (Heroku) or a local login |
+| `bot` | You only need **new posts**, and you can make the bot a channel admin | Bot added as channel administrator |
+
+To add the bot as admin: open the channel → **Manage channel → Administrators → Add Admin** → search
+for your bot's username → save. No extra rights are needed beyond the default.
+
+### 7. `TELEGRAM_SESSION_STRING` (user mode only)
+
+This lets the bot act as your Telegram account without a session file. Heroku's disk is wiped on
+every restart, so a file wouldn't survive. You create it **once, on your own computer**.
+
+1. Do the [local setup](README.md#19-local-development): create the virtualenv and install
+   requirements.
+2. Create `.env` containing at least:
+   ```env
+   TELEGRAM_API_ID=1234567
+   TELEGRAM_API_HASH=0123456789abcdef0123456789abcdef
+   ```
+3. Run:
+   ```bash
+   python main.py login --session-string
+   ```
+4. Enter your phone number in international format (`+15551234567`), then the login code Telegram
+   sends to your app, then your 2FA password if you have one. **None of these are stored.**
+5. The string is written to `data/sessions/session_string.txt` with owner-only permissions.
+6. Put it on Heroku without printing it:
+   ```bash
+   heroku config:set TELEGRAM_SESSION_STRING="$(cat data/sessions/session_string.txt)" --app YOUR_APP_NAME
+   ```
+   For a local `.env`, paste the file's contents after `TELEGRAM_SESSION_STRING=`.
+7. Delete the file: `rm data/sessions/session_string.txt`.
+
+> The session string grants **full access to your Telegram account**. Treat it like a password. To
+> revoke it: Telegram → *Settings → Devices* → terminate the session. Then generate a new string.
+
+For local-only use you can skip the string and run `python main.py login`, which creates
+`data/sessions/user.session` instead.
+
+---
+
+## Part 2 — Google Drive
+
+Pick **one** authentication method:
+
+| Your Drive | Use | Variable to obtain |
+|---|---|---|
+| Personal Google account (`@gmail.com`) or your own Workspace Drive | **OAuth** (`GOOGLE_AUTH_MODE=oauth`) | `GOOGLE_OAUTH_TOKEN_JSON` |
+| Google Workspace **Shared Drive** | **Service account** (`GOOGLE_AUTH_MODE=service_account`) | `GOOGLE_SERVICE_ACCOUNT_JSON` |
+
+> A service account **cannot** upload into a personal My Drive: it has no storage quota there. Use
+> OAuth for personal Drives.
+
+### Common first step: Google Cloud project and Drive API
+
+1. Open <https://console.cloud.google.com/> and sign in with the Google account that owns the
+   destination Drive.
+2. Use the project selector at the top → **New Project** → give it a name → **Create**, then select
+   it.
+3. Go to **APIs & Services → Library**, search for **Google Drive API**, and click **Enable**.
+
+### 9. `GOOGLE_OAUTH_TOKEN_JSON` (OAuth)
+
+#### a) Configure the consent screen
+
+1. Go to **APIs & Services → OAuth consent screen**. In the newer console this is **Google Auth
+   Platform → Branding / Audience**.
+2. User type: **External** (or *Internal* if you use Workspace and only your organisation needs
+   access).
+3. Fill in the app name, your support email and developer contact email, then save.
+4. Under **Audience** (or *Test users*), add your own Google account.
+5. **Publish the app** (*Publishing status → In production*).
+   > If the app stays in *Testing*, Google expires its refresh tokens after **7 days** and the bot
+   > will lose Drive access every week. With the default `drive.file` scope no Google verification
+   > is needed. With the full `drive` scope you'll see an "unverified app" warning during your own
+   > login: click *Advanced → Go to (app name)*.
+
+#### b) Create the OAuth client
+
+1. Go to **APIs & Services → Credentials → Create Credentials → OAuth client ID**.
+2. Application type: **Desktop app**. Give it any name → **Create**.
+3. Click **Download JSON** and save it in the project as `credentials/client_secret.json`. Create the
+   `credentials/` folder if needed; it is gitignored.
+
+#### c) Choose the scope *before* authorizing
+
+| `GOOGLE_DRIVE_SCOPE` | Meaning | Choose it if |
+|---|---|---|
+| `drive.file` (default, least privilege) | The bot only sees files and folders **it created** | You let the bot create its own root folder (step 11, option B) |
+| `drive` | Full access to your Drive | You want uploads to go into a folder **you** created |
+
+Set it in `.env` (for example `GOOGLE_DRIVE_SCOPE=drive.file`). Use the **same value on Heroku**.
+
+#### d) Authorize and get the token
+
+```bash
+python main.py drive-auth            # opens a browser
+python main.py drive-auth --no-browser   # prints a URL to open yourself
+```
+
+Sign in, approve access, and the token is saved to `credentials/token.json`. It contains the refresh
+token, client ID and client secret. Then set it on Heroku:
+
+```bash
+heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME
+```
+
+If you prefer pasting into the Heroku dashboard, a single-line base64 version also works:
+
+```bash
+base64 < credentials/token.json | tr -d '\n'
+```
+
+```env
+GOOGLE_AUTH_MODE=oauth
+GOOGLE_OAUTH_TOKEN_JSON={"token": "...", "refresh_token": "...", "client_id": "...", "client_secret": "...", ...}
+```
+
+> If you change `GOOGLE_DRIVE_SCOPE` later, delete `credentials/token.json`, run `drive-auth` again,
+> and update the Config Var. To revoke access: <https://myaccount.google.com/permissions>.
+
+### 10. `GOOGLE_SERVICE_ACCOUNT_JSON` (Shared Drive)
+
+1. In the Cloud Console: **IAM & Admin → Service Accounts → Create service account**. Give it a name;
+   no project roles are needed. Click **Done**.
+2. Open the service account → **Keys → Add key → Create new key → JSON**. The file downloads once.
+   Store it safely.
+3. Copy the `client_email` value from the JSON (for example
+   `uploader@your-project.iam.gserviceaccount.com`).
+4. In Google Drive, open the **Shared Drive** → **Manage members** → add that email as
+   **Content manager**.
+5. Set the variables:
+   ```bash
+   heroku config:set GOOGLE_AUTH_MODE=service_account \
+     GOOGLE_DRIVE_SCOPE=drive \
+     GOOGLE_SERVICE_ACCOUNT_JSON="$(cat path/to/key.json)" --app YOUR_APP_NAME
+   ```
+   Use `GOOGLE_DRIVE_SCOPE=drive`, because the service account must reach a Shared Drive folder it
+   did not create.
+   Locally you can instead set `GOOGLE_SERVICE_ACCOUNT_FILE=path/to/key.json`.
+
+> Your organisation may block service-account key creation (policy
+> `iam.disableServiceAccountKeyCreation`). In that case use OAuth.
+
+### 11. `GOOGLE_DRIVE_ROOT_FOLDER_ID`
+
+This is the folder that will contain all topic folders. If you leave it empty, topic folders are
+created at the top level of My Drive.
+
+**Option A: a folder you created** (needs `GOOGLE_DRIVE_SCOPE=drive`, or a Shared Drive folder)
+1. Open the folder in Google Drive in a browser.
+2. The URL looks like `https://drive.google.com/drive/folders/1AbCdEfGhIjKlMnOpQrStUvWxYz`.
+3. The part after `/folders/` is the ID → `GOOGLE_DRIVE_ROOT_FOLDER_ID`.
+
+**Option B: let the bot create it** (works with the default `drive.file` scope)
+1. Run this after `drive-auth`:
+   ```bash
+   python main.py drive-init --name "Telegram Videos"
+   ```
+2. It prints `Set GOOGLE_DRIVE_ROOT_FOLDER_ID=...`. Copy that ID.
+
+```env
+GOOGLE_DRIVE_ROOT_FOLDER_ID=1AbCdEfGhIjKlMnOpQrStUvWxYz
+```
+
+At startup the bot checks that it can see this folder. If it can't, it stops and explains how to fix
+the problem.
+
+### 13. `DRIVE_SHARED_DRIVE_ID` (Shared Drive only)
+
+Open the Shared Drive itself (its top level). The URL is
+`https://drive.google.com/drive/folders/<ID>`, and that `<ID>` is the Shared Drive ID.
+
+```env
+DRIVE_SHARED_DRIVE_ID=0AbCdEfGhIjKlUk9PVA
+```
+
+### `DRIVE_SHARE_MODE`
+
+Leave it as `none` so uploads stay private. Set `anyone_with_link` only if every uploaded video
+should be viewable by anyone who has its link.
+
+---
+
+## Part 3 — Database
+
+### 14. `DATABASE_URL`
+
+- **Heroku Deploy button:** the Heroku Postgres add-on is provisioned and `DATABASE_URL` is set
+  automatically. Nothing to do.
+- **Manual Heroku setup:**
+  ```bash
+  heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME
+  heroku config:get DATABASE_URL --app YOUR_APP_NAME   # confirm it is set
+  ```
+  This is a paid plan; check Heroku's current pricing. Don't copy the URL anywhere else, because
+  Heroku may rotate it.
+- **Other PostgreSQL providers:** use their connection string
+  (`postgresql://USER:PASSWORD@HOST:5432/DBNAME`). TLS is added automatically for remote hosts.
+- **Local development:** leave it empty to use SQLite at `data/bot.db`.
+
+On Heroku the bot refuses to start without `DATABASE_URL`, unless you set
+`ALLOW_EPHEMERAL_DATABASE=true`. With that set, all progress is lost on every restart.
+
+---
+
+## Optional settings
+
+These are preferences, not credentials. Defaults are shown; full descriptions are in
+[`.env.example`](.env.example).
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `PROCESS_MODE` | `both` | `existing` (history), `new` (new posts), or `both` |
+| `START_MESSAGE_ID` | `1` | First message ID to scan |
+| `START_DATE` | empty | Only messages on or after this ISO date, for example `2024-01-31` |
+| `INCLUDE_DOCUMENT_VIDEOS` | `true` | Also process videos sent as files |
+| `FILTER_TOPICS` / `FILTER_BATCHES` | empty | Comma-separated allow-lists |
+| `BATCH_SUBFOLDERS_ENABLED` | `false` | `Root/Batch/Topic` folder layout |
+| `DUPLICATE_POLICY` | `suffix_number` | `suffix_number`, `suffix_message_id`, `skip`, `fail` |
+| `MAX_RETRIES` / `RETRY_DELAY` / `RETRY_MAX_DELAY` | `4` / `30` / `1800` | Retry behaviour (seconds) |
+| `PROGRESS_INTERVAL` | `10` | Seconds between progress-message edits |
+| `DOWNLOAD_DIR` | `downloads` (`/tmp/telegram_drive_bot` on Heroku) | Temporary video storage |
+| `UPLOAD_CHUNK_SIZE` | `16MB` (`8MB` on Heroku) | Upload chunk size (multiple of 256KB) |
+| `MIN_FREE_DISK_SPACE` | `1GB` (`512MB` on Heroku) | Free-disk reserve |
+| `AUTO_START` | `false` | Start processing at every boot |
+| `LOG_LEVEL` | `info` | `debug`, `info`, `warning`, `error` |
+| `BOT_SCAN_MAX_EMPTY` / `BOT_SCAN_END_ID` | `200` / empty | Bot-mode history probing limits |
+
+---
+
+## Verify your configuration
+
+Locally, with a filled `.env`:
+
+```bash
+python main.py check-config
+```
+
+This prints `Configuration OK.` or lists every problem with a fix. On Heroku, run
+`heroku logs --tail --app YOUR_APP_NAME` after deploying. A healthy start logs:
+- `Database: PostgreSQL`
+- `Google Drive credentials validated`
+- `Source channel resolved`
+
+The bot then sends you *Bot online* in Telegram.
+
+## Minimal example (Heroku, user mode, OAuth)
+
+```env
+TELEGRAM_MODE=user
+TELEGRAM_API_ID=1234567
+TELEGRAM_API_HASH=0123456789abcdef0123456789abcdef
+TELEGRAM_BOT_TOKEN=123456789:replace-with-your-bot-token
+TELEGRAM_SESSION_STRING=1BVtsOK...replace-with-your-session-string
+ADMIN_USER_IDS=111111111
+SOURCE_CHANNEL_ID=-1001234567890
+GOOGLE_AUTH_MODE=oauth
+GOOGLE_DRIVE_SCOPE=drive.file
+GOOGLE_OAUTH_TOKEN_JSON={"refresh_token": "...", "client_id": "...", "client_secret": "...", "token_uri": "https://oauth2.googleapis.com/token"}
+GOOGLE_DRIVE_ROOT_FOLDER_ID=1AbCdEfGhIjKlMnOpQrStUvWxYz
+# DATABASE_URL is set automatically by Heroku Postgres
+```