drive/ENV_SETUP.md

# How to Obtain Every Environment Variable

This guide explains where each configuration value comes from and how to get it. Work through the
sections in order: some steps (the Telegram session string and the Google token) need values from
earlier steps.

- **Local runs:** put the values in a `.env` file (`cp .env.example .env`).
- **Heroku:** set them as Config Vars, either in the dashboard (*App → Settings → Reveal Config Vars*)
  or with `heroku config:set NAME=value --app YOUR_APP_NAME`.

> All values below are placeholders. Never commit real values or paste them into chats, issues or
> screenshots. `.env`, `credentials/` and `data/` are already in `.gitignore`.

---

## Quick checklist

| # | Variable | Required? | Where it comes from |
|---|---|---|---|
| 1 | `TELEGRAM_API_ID` | Always | my.telegram.org |
| 2 | `TELEGRAM_API_HASH` | Always | my.telegram.org |
| 3 | `TELEGRAM_BOT_TOKEN` | Always on Heroku (control commands); required in bot mode | @BotFather |
| 4 | `ADMIN_USER_IDS` | When a bot token is set | @userinfobot |
| 5 | `SOURCE_CHANNEL_ID` | Always | Channel username, link, or numeric ID |
| 6 | `TELEGRAM_MODE` | Optional (default `user`) | Your choice: `user` or `bot` |
| 7 | `TELEGRAM_SESSION_STRING` | `TELEGRAM_MODE=user` on Heroku | `python main.py login --session-string` |
| 8 | `GOOGLE_AUTH_MODE` | Optional (default `oauth`) | Your choice: `oauth` or `service_account` |
| 9 | `GOOGLE_OAUTH_TOKEN_JSON` | `oauth` on Heroku | `python main.py drive-auth` |
| 10 | `GOOGLE_SERVICE_ACCOUNT_JSON` | `service_account` on Heroku | Google Cloud Console key |
| 11 | `GOOGLE_DRIVE_ROOT_FOLDER_ID` | Recommended | Drive folder URL, or `python main.py drive-init` |
| 12 | `GOOGLE_DRIVE_SCOPE` | Optional (default `drive.file`) | Your choice |
| 13 | `DRIVE_SHARED_DRIVE_ID` | Shared Drive setups only | Shared Drive URL |
| 14 | `DATABASE_URL` | Required on Heroku | Heroku Postgres (set automatically) |

Everything else has a sensible default (see [Optional settings](#optional-settings)).

---

## Part 1 — Telegram

### 1–2. `TELEGRAM_API_ID` and `TELEGRAM_API_HASH`

These identify your Telegram API application. They are needed in **both** modes, because the bot talks
to Telegram through Telethon (MTProto).

1. Open <https://my.telegram.org> and log in with your phone number. Telegram sends a code to your
   Telegram app, not by SMS.
2. Click **API development tools**.
3. If no app exists yet, fill in the form:
   - *App title*: anything, for example `Drive Organizer`;
   - *Short name*: 5–32 letters or digits, for example `driveorganizer`;
   - *Platform*: `Desktop`;
   - URL and description can stay empty.
4. Click **Create application**.
5. Copy the two values:
   - **App api_id** (a number) → `TELEGRAM_API_ID`
   - **App api_hash** (32 hex characters) → `TELEGRAM_API_HASH`

```env
TELEGRAM_API_ID=1234567
TELEGRAM_API_HASH=0123456789abcdef0123456789abcdef
```

> The API hash is a secret. If my.telegram.org shows an "ERROR" when you create the app, try another
> browser, disable ad blockers or VPNs, and use a short alphanumeric short name.

### 3. `TELEGRAM_BOT_TOKEN`

The bot provides the admin commands (`/run`, `/status`, …) and progress messages. In bot mode it also
reads the channel.

1. In Telegram, open [@BotFather](https://t.me/BotFather) and send `/newbot`.
2. Enter a display name, then a username ending in `bot` (for example `my_drive_organizer_bot`).
3. BotFather replies with a token like `123456789:AAH...`. Copy it → `TELEGRAM_BOT_TOKEN`.
4. **Open your new bot and press Start.** Bots cannot message you until you do, so without this you
   get no progress messages.

```env
TELEGRAM_BOT_TOKEN=123456789:replace-with-your-bot-token
```

If the token leaks, send `/revoke` to @BotFather to get a new one.

### 4. `ADMIN_USER_IDS` (and optional `NOTIFY_CHAT_IDS`)

Only these numeric Telegram user IDs may control the bot.

1. Open [@userinfobot](https://t.me/userinfobot) and press Start.
2. It replies with your **Id** (a number such as `111111111`). Copy it.
3. For several admins, ask each person to do the same and separate the IDs with commas.

```env
ADMIN_USER_IDS=111111111,222222222
```

`NOTIFY_CHAT_IDS` is optional. It lists the chats that receive progress and error messages, and
defaults to `ADMIN_USER_IDS`. Each person listed must have pressed Start in the bot.

### 5. `SOURCE_CHANNEL_ID`

Any of these formats is accepted:

| Channel type | How to get it | Example value |
|---|---|---|
| Public channel | The `@username` shown in the channel info | `@my_channel` or `https://t.me/my_channel` |
| Private channel: post link | Right-click (or long-press) any post → **Copy Post Link** | `https://t.me/c/1234567890/55` |
| Private channel: numeric ID | Prefix the number from the post link with `-100` | `-1001234567890` |
| Private channel: Telegram Web | Open the channel at <https://web.telegram.org/k/>; the URL ends with `#-100…` | `-1001234567890` |

Another way: forward a channel post to [@JsonDumpBot](https://t.me/JsonDumpBot) and read
`forward_from_chat.id`.

```env
SOURCE_CHANNEL_ID=-1001234567890
```

> The same post links are what you send to the bot's `/upload` command to pick a message range.

> Bot mode with a private channel: the bot must be a channel **administrator**. If the numeric ID
> can't be resolved at the first start, post one message in the channel. The bot then learns the
> channel and remembers it in the database.

### 6. `TELEGRAM_MODE`

This is a choice, not a credential.

| Value | Use when | Needs |
|---|---|---|
| `user` (default) | You need the channel's **existing history**, or you are a member but not an admin | `TELEGRAM_SESSION_STRING` (Heroku) or a local login |
| `bot` | You only need **new posts**, and you can make the bot a channel admin | Bot added as channel administrator |

To add the bot as admin: open the channel → **Manage channel → Administrators → Add Admin** → search
for your bot's username → save. No extra rights are needed beyond the default.

### 7. `TELEGRAM_SESSION_STRING` (user mode only)

This lets the bot act as your Telegram account without a session file. Heroku's disk is wiped on
every restart, so a file wouldn't survive. You create it **once, on your own computer**.

1. Do the [local setup](README.md#19-local-development): create the virtualenv and install
   requirements.
2. Create `.env` containing at least:
   ```env
   TELEGRAM_API_ID=1234567
   TELEGRAM_API_HASH=0123456789abcdef0123456789abcdef
   ```
3. Run:
   ```bash
   python main.py login --session-string
   ```
4. Enter your phone number in international format (`+15551234567`), then the login code Telegram
   sends to your app, then your 2FA password if you have one. **None of these are stored.**
5. The string is written to `data/sessions/session_string.txt` with owner-only permissions.
6. Put it on Heroku without printing it:
   ```bash
   heroku config:set TELEGRAM_SESSION_STRING="$(cat data/sessions/session_string.txt)" --app YOUR_APP_NAME
   ```
   For a local `.env`, paste the file's contents after `TELEGRAM_SESSION_STRING=`.
7. Delete the file: `rm data/sessions/session_string.txt`.

> The session string grants **full access to your Telegram account**. Treat it like a password. To
> revoke it: Telegram → *Settings → Devices* → terminate the session. Then generate a new string.

For local-only use you can skip the string and run `python main.py login`, which creates
`data/sessions/user.session` instead.

---

## Part 2 — Google Drive

Pick **one** authentication method:

| Your Drive | Use | Variable to obtain |
|---|---|---|
| Personal Google account (`@gmail.com`) or your own Workspace Drive | **OAuth** (`GOOGLE_AUTH_MODE=oauth`) | `GOOGLE_OAUTH_TOKEN_JSON` |
| Google Workspace **Shared Drive** | **Service account** (`GOOGLE_AUTH_MODE=service_account`) | `GOOGLE_SERVICE_ACCOUNT_JSON` |

> A service account **cannot** upload into a personal My Drive: it has no storage quota there. Use
> OAuth for personal Drives.

### Common first step: Google Cloud project and Drive API

1. Open <https://console.cloud.google.com/> and sign in with the Google account that owns the
   destination Drive.
2. Use the project selector at the top → **New Project** → give it a name → **Create**, then select
   it.
3. Go to **APIs & Services → Library**, search for **Google Drive API**, and click **Enable**.

### 9. `GOOGLE_OAUTH_TOKEN_JSON` (OAuth)

#### a) Configure the consent screen

1. Go to **APIs & Services → OAuth consent screen**. In the newer console this is **Google Auth
   Platform → Branding / Audience**.
2. User type: **External** (or *Internal* if you use Workspace and only your organisation needs
   access).
3. Fill in the app name, your support email and developer contact email, then save.
4. Under **Audience** (or *Test users*), add your own Google account.
5. **Publish the app** (*Publishing status → In production*).
   > If the app stays in *Testing*, Google expires its refresh tokens after **7 days** and the bot
   > will lose Drive access every week. With the default `drive.file` scope no Google verification
   > is needed. With the full `drive` scope you'll see an "unverified app" warning during your own
   > login: click *Advanced → Go to (app name)*.

#### b) Create the OAuth client

1. Go to **APIs & Services → Credentials → Create Credentials → OAuth client ID**.
2. Application type: **Desktop app**. Give it any name → **Create**.
3. Click **Download JSON** and save it in the project as `credentials/client_secret.json`. Create the
   `credentials/` folder if needed; it is gitignored.

#### c) Choose the scope *before* authorizing

| `GOOGLE_DRIVE_SCOPE` | Meaning | Choose it if |
|---|---|---|
| `drive.file` (default, least privilege) | The bot only sees files and folders **it created** | You let the bot create its own root folder (step 11, option B) |
| `drive` | Full access to your Drive | You want uploads to go into a folder **you** created |

Set it in `.env` (for example `GOOGLE_DRIVE_SCOPE=drive.file`). Use the **same value on Heroku**.

#### d) Authorize and get the token

```bash
python main.py drive-auth            # opens a browser
python main.py drive-auth --no-browser   # prints a URL to open yourself
```

Sign in, approve access, and the token is saved to `credentials/token.json`. It contains the refresh
token, client ID and client secret. Then set it on Heroku:

```bash
heroku config:set GOOGLE_OAUTH_TOKEN_JSON="$(cat credentials/token.json)" --app YOUR_APP_NAME
```

If you prefer pasting into the Heroku dashboard, a single-line base64 version also works:

```bash
base64 < credentials/token.json | tr -d '\n'
```

```env
GOOGLE_AUTH_MODE=oauth
GOOGLE_OAUTH_TOKEN_JSON={"token": "...", "refresh_token": "...", "client_id": "...", "client_secret": "...", ...}
```

> If you change `GOOGLE_DRIVE_SCOPE` later, delete `credentials/token.json`, run `drive-auth` again,
> and update the Config Var. To revoke access: <https://myaccount.google.com/permissions>.

### 10. `GOOGLE_SERVICE_ACCOUNT_JSON` (Shared Drive)

1. In the Cloud Console: **IAM & Admin → Service Accounts → Create service account**. Give it a name;
   no project roles are needed. Click **Done**.
2. Open the service account → **Keys → Add key → Create new key → JSON**. The file downloads once.
   Store it safely.
3. Copy the `client_email` value from the JSON (for example
   `uploader@your-project.iam.gserviceaccount.com`).
4. In Google Drive, open the **Shared Drive** → **Manage members** → add that email as
   **Content manager**.
5. Set the variables:
   ```bash
   heroku config:set GOOGLE_AUTH_MODE=service_account \
     GOOGLE_DRIVE_SCOPE=drive \
     GOOGLE_SERVICE_ACCOUNT_JSON="$(cat path/to/key.json)" --app YOUR_APP_NAME
   ```
   Use `GOOGLE_DRIVE_SCOPE=drive`, because the service account must reach a Shared Drive folder it
   did not create.
   Locally you can instead set `GOOGLE_SERVICE_ACCOUNT_FILE=path/to/key.json`.

> Your organisation may block service-account key creation (policy
> `iam.disableServiceAccountKeyCreation`). In that case use OAuth.

### 11. `GOOGLE_DRIVE_ROOT_FOLDER_ID`

This is the folder that will contain all topic folders. If you leave it empty, topic folders are
created at the top level of My Drive.

**Option A: a folder you created** (needs `GOOGLE_DRIVE_SCOPE=drive`, or a Shared Drive folder)
1. Open the folder in Google Drive in a browser.
2. The URL looks like `https://drive.google.com/drive/folders/1AbCdEfGhIjKlMnOpQrStUvWxYz`.
3. The part after `/folders/` is the ID → `GOOGLE_DRIVE_ROOT_FOLDER_ID`.

**Option B: let the bot create it** (works with the default `drive.file` scope)
1. Run this after `drive-auth`:
   ```bash
   python main.py drive-init --name "Telegram Videos"
   ```
2. It prints `Set GOOGLE_DRIVE_ROOT_FOLDER_ID=...`. Copy that ID.

```env
GOOGLE_DRIVE_ROOT_FOLDER_ID=1AbCdEfGhIjKlMnOpQrStUvWxYz
```

At startup the bot checks that it can see this folder. If it can't, it stops and explains how to fix
the problem.

### 13. `DRIVE_SHARED_DRIVE_ID` (Shared Drive only)

Open the Shared Drive itself (its top level). The URL is
`https://drive.google.com/drive/folders/<ID>`, and that `<ID>` is the Shared Drive ID.

```env
DRIVE_SHARED_DRIVE_ID=0AbCdEfGhIjKlUk9PVA
```

### `DRIVE_SHARE_MODE`

Leave it as `none` so uploads stay private. Set `anyone_with_link` only if every uploaded video
should be viewable by anyone who has its link.

---

## Part 3 — Database

### 14. `DATABASE_URL`

- **Heroku Deploy button:** the Heroku Postgres add-on is provisioned and `DATABASE_URL` is set
  automatically. Nothing to do.
- **Manual Heroku setup:**
  ```bash
  heroku addons:create heroku-postgresql:essential-0 --app YOUR_APP_NAME
  heroku config:get DATABASE_URL --app YOUR_APP_NAME   # confirm it is set
  ```
  This is a paid plan; check Heroku's current pricing. Don't copy the URL anywhere else, because
  Heroku may rotate it.
- **Other PostgreSQL providers:** use their connection string
  (`postgresql://USER:PASSWORD@HOST:5432/DBNAME`). TLS is added automatically for remote hosts.
- **Local development:** leave it empty to use SQLite at `data/bot.db`.

On Heroku the bot refuses to start without `DATABASE_URL`, unless you set
`ALLOW_EPHEMERAL_DATABASE=true`. With that set, all progress is lost on every restart.

---

## Optional settings

These are preferences, not credentials. Defaults are shown; full descriptions are in
[`.env.example`](.env.example).

| Variable | Default | Purpose |
|---|---|---|
| `PROCESS_MODE` | `both` | `existing` (history), `new` (new posts), or `both` |
| `START_MESSAGE_ID` | `1` | First message ID to scan |
| `START_DATE` | empty | Only messages on or after this ISO date, for example `2024-01-31` |
| `INCLUDE_DOCUMENT_VIDEOS` | `true` | Also process videos sent as files |
| `FILTER_TOPICS` / `FILTER_BATCHES` | empty | Comma-separated allow-lists |
| `BATCH_SUBFOLDERS_ENABLED` | `false` | `Root/Batch/Topic` folder layout |
| `DUPLICATE_POLICY` | `suffix_number` | `suffix_number`, `suffix_message_id`, `skip`, `fail` |
| `MAX_RETRIES` / `RETRY_DELAY` / `RETRY_MAX_DELAY` | `4` / `30` / `1800` | Retry behaviour (seconds) |
| `PROGRESS_INTERVAL` | `10` | Seconds between progress-message edits |
| `DOWNLOAD_DIR` | `downloads` (`/tmp/telegram_drive_bot` on Heroku) | Temporary video storage |
| `UPLOAD_CHUNK_SIZE` | `16MB` (`8MB` on Heroku) | Upload chunk size (multiple of 256KB) |
| `MIN_FREE_DISK_SPACE` | `1GB` (`512MB` on Heroku) | Free-disk reserve |
| `AUTO_START` | `false` | Start processing at every boot |
| `LOG_LEVEL` | `info` | `debug`, `info`, `warning`, `error` |
| `BOT_SCAN_MAX_EMPTY` / `BOT_SCAN_END_ID` | `200` / empty | Bot-mode history probing limits |
| `UPLOAD_RESTRICT_TO_SOURCE_CHANNEL` | `true` | `/upload` links must point to `SOURCE_CHANNEL_ID`; `false` allows any channel the account/bot can access |
| `UPLOAD_SETUP_TIMEOUT` | `30` | Minutes before an unfinished `/upload` setup expires |
| `UPLOAD_MAX_RANGE` | `5000` | Largest number of message IDs in one `/upload` task |
| `KEEP_FAILED_DOWNLOADS` | `false` | Keep the local file of a FAILED video instead of deleting it |
| `TEMP_CLEANUP_INTERVAL` | `3600` | Seconds between sweeps that delete abandoned temporary files |

---

## Verify your configuration

Locally, with a filled `.env`:

```bash
python main.py check-config
```

This prints `Configuration OK.` or lists every problem with a fix. On Heroku, run
`heroku logs --tail --app YOUR_APP_NAME` after deploying. A healthy start logs:
- `Database: PostgreSQL`
- `Google Drive credentials validated`
- `Source channel resolved`

The bot then sends you *Bot online* in Telegram.

## Minimal example (Heroku, user mode, OAuth)

```env
TELEGRAM_MODE=user
TELEGRAM_API_ID=1234567
TELEGRAM_API_HASH=0123456789abcdef0123456789abcdef
TELEGRAM_BOT_TOKEN=123456789:replace-with-your-bot-token
TELEGRAM_SESSION_STRING=1BVtsOK...replace-with-your-session-string
ADMIN_USER_IDS=111111111
SOURCE_CHANNEL_ID=-1001234567890
GOOGLE_AUTH_MODE=oauth
GOOGLE_DRIVE_SCOPE=drive.file
GOOGLE_OAUTH_TOKEN_JSON={"refresh_token": "...", "client_id": "...", "client_secret": "...", "token_uri": "https://oauth2.googleapis.com/token"}
GOOGLE_DRIVE_ROOT_FOLDER_ID=1AbCdEfGhIjKlMnOpQrStUvWxYz
# DATABASE_URL is set automatically by Heroku Postgres
```