diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c09fd9..c44b2f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,14 @@ Пользовательские описания релизов. Именно ЭТОТ файл (не заголовки коммитов) показывается в уведомлении об обновлении: раздел на версию, пункты — строки, начинающиеся с «- ». Заполняется ПЕРЕД тегированием релиза — scripts/release.sh откажется публиковать версию без её раздела здесь. Коммиты остаются техническими и пользователям не показываются. +## 1.3.0 + +- Большие файлы теперь пересылаются по ссылке. Из Telegram в MAX файл больше 20 МБ (столько бот может скачать из Telegram) загружается через страницу на вашем сервере и уходит в MAX обычным файлом. Из MAX в Telegram файл больше 50 МБ (столько бот может отправить) сохраняется на сервере, а в тему приходит кнопка «Скачать». Файлы больше 4 ГБ (лимит MAX) уходят собеседнику ссылкой +- Пульт → «📁 Файлы»: какие файлы лежат на сервере, новая ссылка или удаление. Файл хранится 3 дня от последней ссылки, потом удаляется сам. Перед каждой передачей мост проверяет свободное место и всегда оставляет 1,5 ГБ запаса для обновлений +- Для ссылок на сервере открываются порты 80 и 443 — только пока есть хотя бы одна действующая ссылка. Ссылок нет — служба файлов остановлена, порты закрыты. HTTPS настраивается сам, домен не нужен. Отключить совсем: FILES=off в .env +- Кнопки мини-приложений ботов MAX стали ссылками: нажатие открывает мини-приложение в MAX +- Если файл не прошёл по лимиту Telegram, мост объясняет, в чём дело, а не показывает «file is too big» + ## 1.2.1 — 2026-10-05 - Поиск по нику снова находит ботов MAX: ник можно писать с «@» или без, а можно вставить ссылку max.ru/ник. Найденный бот приходит карточкой с кнопкой «Начать чат», дальше с ним можно переписываться и нажимать его кнопки diff --git a/README.md b/README.md index c572bcd..a6802fb 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,8 @@ ## Возможности - Текстовые сообщения, фото, файлы, голосовые, видео и видео-кружки (обе стороны) +- Большие файлы — по ссылке: из Telegram в MAX больше 20 МБ, из MAX в Telegram больше 50 МБ, в MAX больше 4 ГБ (см. [«Большие файлы»](#большие-файлы)) +- Кнопки ботов MAX — в Telegram: ссылки, копирование, нажатия; мини-приложения открываются ссылкой в MAX - Стикеры — статичные и анимированные (обе стороны) - Геолокация и контакты (обе стороны) - Опросы: создание и голосование в обе стороны, актуальный счёт по команде `/poll` @@ -106,6 +108,31 @@ curl -fsSL https://raw.githubusercontent.com/Trollobot/Telemax/main/install.sh | После успешной авторизации сессия сохраняется зашифрованной в `./data` — повторной авторизации при перезапуске контейнера не требуется. Повторный вход тем же путём (`/login`) нужен только после `/kill`, смены номера или если MAX отклонил сессию (бот сам напишет об этом в группу). +## Большие файлы + +У ботов Telegram жёсткие лимиты: скачать из Telegram бот может файл не больше **20 МБ**, отправить в Telegram — не больше **50 МБ**. MAX принимает файлы до 4 ГБ. Всё, что не пролезает, мост пересылает по ссылке: + +- **Из Telegram в MAX, больше 20 МБ.** Бот отвечает в теме кнопкой «📤 Загрузить файл»: по ссылке открывается страница на вашем сервере, файл загружается из браузера, и мост отправляет его в MAX обычным файлом. Файл больше 4 ГБ (лимит MAX) уходит собеседнику ссылкой на скачивание — она ведёт на ваш сервер. +- **Из MAX в Telegram, больше 50 МБ.** Мост сохраняет файл на сервере и присылает в тему кнопку «⬇️ Скачать». +- **Пульт → «📁 Файлы».** Список файлов на сервере: новая ссылка (продлевает хранение) или удаление. Файл живёт 3 дня от последней выданной ссылки, потом удаляется сам. + +Как это устроено: + +- Ссылки обслуживает одна общая служба на сервер (`files-service/`: Caddy + маленькое приложение на Node) — для всех мостов сразу. Её запускает автообновление, и **только пока есть действующие ссылки**: тогда открыты порты 80 и 443. Ссылок нет — служба остановлена, порты закрыты. Поэтому первая ссылка может готовиться до пары минут. +- HTTPS — без домена: сертификат Let's Encrypt выпускается на IP-адрес сервера и продлевается сам. Если есть свой домен, укажите его: `FILES_DOMAIN=files.example.ru` в `.env` первого моста. +- Перед каждой передачей проверяется свободное место, и мост всегда оставляет запас 1,5 ГБ — столько нужно для обновления. +- По ссылке открывается страница с кнопкой, а не сам файл: предпросмотр ссылок в мессенджерах её не «съедает». Ссылки случайные, без списков файлов, с ограниченным сроком. +- Нужны свободные порты 80 и 443. Если их занимает другой веб-сервер, бот так и скажет, а файлы больше лимитов пересылаться не будут. + +Настройки в `.env` (необязательные): + +```bash +FILES=off # выключить пересылку больших файлов совсем +FILES_TTL_HOURS=72 # сколько живёт файл после выдачи ссылки +FILES_RESERVE_MB=1536 # сколько места всегда оставлять свободным +FILES_DOMAIN= # свой домен вместо IP-адреса +``` + ## Безопасность Бот отвечает только внутри той Telegram-группы, что указана в `TARGET_TELEGRAM_GROUP` — любое сообщение или нажатие кнопки из другого чата (личка боту, добавление в чужую группу и т.п.) игнорируется. @@ -114,7 +141,7 @@ curl -fsSL https://raw.githubusercontent.com/Trollobot/Telemax/main/install.sh | Учтите: любой участник Telegram-группы видит в ней **все** зеркалированные MAX-чаты (по теме на чат), а не какой-то один. Доступ к самой группе — чувствительная вещь, раздавайте его осознанно. -Мост не открывает ни одного входящего порта: MAX — исходящее TCP-соединение, Telegram — long-polling. Управление — только из Telegram-группы (админами) и по SSH через `./setup.sh`. Единственная открытая дверь на сервере — SSH; `install.sh` предлагает поставить Fail2ban, который банит подбор пароля (5 промахов за 10 минут → бан на 10 минут; IP вашей сессии установки — в белом списке). +Мост не открывает входящих портов: MAX — исходящее TCP-соединение, Telegram — long-polling. Единственное исключение — пересылка больших файлов (см. [«Большие файлы»](#большие-файлы)): пока на сервере есть хотя бы одна действующая ссылка на файл, открыты порты 80 и 443, а когда ссылок нет — служба останавливается и порты закрываются. Управление — только из Telegram-группы (админами) и по SSH через `./setup.sh`. Единственная открытая дверь на сервере — SSH; `install.sh` предлагает поставить Fail2ban, который банит подбор пароля (5 промахов за 10 минут → бан на 10 минут; IP вашей сессии установки — в белом списке). Сертификаты российского государственного УЦ (нужны только потому, что через них выпущен TLS-сертификат самого MAX) доверяются **только** соединениям с серверами MAX — запросы к Telegram, GitHub и всему остальному проверяются исключительно по стандартному набору корневых сертификатов Mozilla (подробнее — `certs/README.md`). diff --git a/files-service/reconcile.sh b/files-service/reconcile.sh new file mode 100644 index 0000000..d8103ae --- /dev/null +++ b/files-service/reconcile.sh @@ -0,0 +1,204 @@ +#!/usr/bin/env bash +# Starts or stops the shared Telemax files service (Caddy + files-service/server.mjs) — ONE per +# host for every bridge in /etc/telemax/instances. Called every minute by update-watcher.sh, i.e. +# by the root update dispatcher, so installs get it with a normal update (no setup.sh re-run). +# +# Ports 80/443 are open ONLY while some bridge has a live big-file link: each bridge keeps +# `data/files/active` while it has one (src/bridge/fileShare.ts). No marker anywhere → the service +# is stopped. The result goes back to every bridge as `data/files/service.json`. +# +# TLS without a domain: Let's Encrypt issues short-lived (~6-day) certificates for a bare IP +# (generally available since 2026-01; Caddy needs the `shortlived` profile and `default_sni`, +# because browsers send no SNI for an IP). FILES_DOMAIN in the first bridge's .env uses a domain +# instead. Caddy keeps its certificate in /var/lib/telemax-files/caddy-data between runs. +set -uo pipefail + +REG=/etc/telemax/instances +STATE_DIR=/var/lib/telemax-files +PROJECT=telemax-files +CADDY_IMAGE='caddy:2.11.7@sha256:f2a1290d0463aad60660d4ec134943f183ee2a5f6c3eb7bf32dd984f2f020772' +SELF="$(cd "$(dirname "$0")/.." && pwd)" + +[ -f "$REG" ] || exit 0 +[ "$(id -u)" = "0" ] || exit 0 +command -v docker >/dev/null 2>&1 || exit 0 + +# One owner: the first registered install that ships this script. Every bridge's watcher calls us +# each minute; without a single owner two installs on different versions would flip-flop the config. +LEADER="" +while IFS= read -r d; do + [ -n "$d" ] && [ -f "$d/files-service/reconcile.sh" ] && { LEADER="$d"; break; } +done < "$REG" +[ "$LEADER" = "$SELF" ] || exit 0 + +mkdir -p "$STATE_DIR" +log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*" >> "$STATE_DIR/reconcile.log"; tail -n 200 "$STATE_DIR/reconcile.log" > "$STATE_DIR/reconcile.log.tmp" 2>/dev/null && mv "$STATE_DIR/reconcile.log.tmp" "$STATE_DIR/reconcile.log"; } +env_value() { grep -E "^$2=" "$1/.env" 2>/dev/null | tail -1 | cut -d= -f2- | tr -d "\"' \r"; } + +INSTANCES=() +WANTED=0 +while IFS= read -r d; do + [ -n "$d" ] && [ -d "$d/data/files" ] || continue + case "$(env_value "$d" FILES | tr '[:upper:]' '[:lower:]')" in 0|false|off|no) continue ;; esac + INSTANCES+=("$d") + [ -f "$d/data/files/active" ] && WANTED=1 +done < "$REG" + +write_state() { # state url detail + local now detail + now=$(date +%s) + detail=$(printf '%s' "$3" | tr -d '"\\\n\r' | cut -c1-200) + for d in "${INSTANCES[@]}"; do + printf '{"state":"%s","url":"%s","at":%s,"detail":"%s"}\n' "$1" "$2" "$now" "$detail" > "$d/data/files/service.json.tmp" \ + && chmod 644 "$d/data/files/service.json.tmp" && mv -f "$d/data/files/service.json.tmp" "$d/data/files/service.json" + done +} + +running() { docker ps -q --filter "label=com.docker.compose.project=$PROJECT" | grep -q .; } +compose() { docker compose -p "$PROJECT" -f "$STATE_DIR/compose.yml" "$@"; } + +if [ "$WANTED" = 0 ]; then + if running; then + compose down >/dev/null 2>&1 && log "no live links — service stopped, ports 80/443 closed" + fi + rm -f "$STATE_DIR/starting-since" + write_state down "" "" + exit 0 +fi + +# --- where the links point --------------------------------------------------------------------- +is_public_ipv4() { + [[ "$1" =~ ^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$ ]] || return 1 + local a=${BASH_REMATCH[1]} b=${BASH_REMATCH[2]} + [ "$a" = 10 ] || [ "$a" = 127 ] || [ "$a" = 0 ] && return 1 + [ "$a" = 192 ] && [ "$b" = 168 ] && return 1 + [ "$a" = 172 ] && [ "$b" -ge 16 ] && [ "$b" -le 31 ] && return 1 + [ "$a" = 100 ] && [ "$b" -ge 64 ] && [ "$b" -le 127 ] && return 1 + [ "$a" = 169 ] && [ "$b" = 254 ] && return 1 + return 0 +} +DOMAIN=$(env_value "$LEADER" FILES_DOMAIN) +IS_IP=0 +if [ -n "$DOMAIN" ]; then + HOST="$DOMAIN" +else + IS_IP=1 + HOST="" + # Cached for a day: the lookup is the only outside request this script makes. + if [ -f "$STATE_DIR/public-ip" ] && [ -n "$(find "$STATE_DIR/public-ip" -mmin -1440 2>/dev/null)" ]; then + HOST=$(cat "$STATE_DIR/public-ip") + fi + if ! is_public_ipv4 "$HOST"; then + HOST=$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{for (i=1;i/dev/null || true) + is_public_ipv4 "$HOST" || HOST=$(curl -4 -fsS --max-time 6 https://ifconfig.me/ip 2>/dev/null || true) + is_public_ipv4 "$HOST" && printf '%s\n' "$HOST" > "$STATE_DIR/public-ip" + fi + if ! is_public_ipv4 "$HOST"; then + log "no public IPv4 found" + write_state no-ip "" "" + exit 0 + fi +fi + +# --- ports: someone else's web server must not be fought with --------------------------------- +if ! running && ss -ltnH '( sport = :80 or sport = :443 )' 2>/dev/null | grep -q .; then + log "ports 80/443 are taken by another service" + write_state ports-busy "" "" + exit 0 +fi + +# --- config ------------------------------------------------------------------------------------- +NODE_IMAGE=$(sed -n 's/^FROM \(node:[^ ]*\).*/\1/p' "$LEADER/Dockerfile" | head -1) +[ -n "$NODE_IMAGE" ] || NODE_IMAGE=node:22-slim +TZ_VALUE=$(env_value "$LEADER" TZ) + +{ + echo "{" + echo " admin off" + if [ "$IS_IP" = 1 ]; then + echo " default_sni $HOST" + echo " cert_issuer acme {" + echo " profile shortlived" + echo " }" + fi + echo "}" + echo "https://$HOST {" + echo " reverse_proxy app:8080" + echo "}" +} > "$STATE_DIR/Caddyfile.new" + +{ + echo "name: $PROJECT" + echo "services:" + echo " caddy:" + echo " image: $CADDY_IMAGE" + echo " restart: unless-stopped" + echo " ports: [\"80:80\", \"443:443\"]" + echo " volumes:" + echo " - $STATE_DIR/Caddyfile:/etc/caddy/Caddyfile:ro" + echo " - $STATE_DIR/caddy-data:/data" + echo " - $STATE_DIR/caddy-config:/config" + echo " depends_on: [app]" + echo " logging: {driver: json-file, options: {max-size: \"5m\", max-file: \"2\"}}" + echo " app:" + echo " image: $NODE_IMAGE" + echo " restart: unless-stopped" + echo " user: \"1000:1000\"" + echo " command: [\"node\", \"/svc/server.mjs\"]" + echo " environment:" + echo " TZ: \"${TZ_VALUE:-Europe/Moscow}\"" + echo " volumes:" + echo " - $LEADER/files-service:/svc:ro" + i=0 + for d in "${INSTANCES[@]}"; do + i=$((i + 1)) + echo " - $d/data/files:/srv/$i" + done + echo " logging: {driver: json-file, options: {max-size: \"5m\", max-file: \"2\"}}" +} > "$STATE_DIR/compose.yml.new" + +CADDY_CHANGED=0 +cmp -s "$STATE_DIR/Caddyfile.new" "$STATE_DIR/Caddyfile" 2>/dev/null || CADDY_CHANGED=1 +mv -f "$STATE_DIR/Caddyfile.new" "$STATE_DIR/Caddyfile" +mv -f "$STATE_DIR/compose.yml.new" "$STATE_DIR/compose.yml" + +WAS_RUNNING=0 +running && WAS_RUNNING=1 +if ! compose up -d --remove-orphans >"$STATE_DIR/compose-up.log" 2>&1; then + log "compose up failed: $(tail -n 3 "$STATE_DIR/compose-up.log" | tr '\n' ' ')" + write_state error "" "$(grep -iE 'error|bind|address already' "$STATE_DIR/compose-up.log" | tail -1)" + exit 0 +fi +[ "$WAS_RUNNING" = 0 ] && log "live links — service started for https://$HOST, ports 80/443 open" +[ "$WAS_RUNNING" = 1 ] && [ "$CADDY_CHANGED" = 1 ] && compose restart caddy >/dev/null 2>&1 +# server.mjs is mounted, not baked in: after an update brings a new one, restart the app to load it. +APP_SUM=$(sha256sum "$LEADER/files-service/server.mjs" 2>/dev/null | cut -c1-64) +if [ "$WAS_RUNNING" = 1 ] && [ -n "$APP_SUM" ] && [ "$APP_SUM" != "$(cat "$STATE_DIR/app.sum" 2>/dev/null)" ]; then + compose restart app >/dev/null 2>&1 && log "files service app restarted with the updated server.mjs" +fi +[ -n "$APP_SUM" ] && printf '%s\n' "$APP_SUM" > "$STATE_DIR/app.sum" + +# --- healthy? ----------------------------------------------------------------------------------- +# A fresh start needs a few seconds (and a certificate on the very first run): wait a little here +# rather than make the waiting bridge sit through another dispatcher minute. +healthy() { curl -fsS --max-time 8 "https://$HOST/health" >/dev/null 2>&1; } +if [ "$WAS_RUNNING" = 0 ]; then + for _ in 1 2 3 4 5 6 7 8; do healthy && break; sleep 4; done +fi +if healthy; then + rm -f "$STATE_DIR/starting-since" + write_state up "https://$HOST" "" + exit 0 +fi +[ -f "$STATE_DIR/starting-since" ] || date +%s > "$STATE_DIR/starting-since" +SINCE=$(cat "$STATE_DIR/starting-since" 2>/dev/null || date +%s) +if [ $(( $(date +%s) - SINCE )) -gt 240 ]; then + WHY=$(docker logs --tail 50 "$(docker ps -q --filter "label=com.docker.compose.project=$PROJECT" --filter "label=com.docker.compose.service=caddy" | head -1)" 2>&1 \ + | grep -oE '"(error|msg)":"[^"]*"' | grep -iE 'error|fail|challenge|refused|timeout' | tail -1) + log "not healthy after 4 min: $WHY" + write_state error "" "HTTPS не отвечает: ${WHY:-нет подробностей}" +else + write_state starting "" "" +fi +exit 0 diff --git a/files-service/server.mjs b/files-service/server.mjs new file mode 100644 index 0000000..1ad232c --- /dev/null +++ b/files-service/server.mjs @@ -0,0 +1,263 @@ +// Telemax files service — serves the bridges' big-file links (see src/bridge/fileShare.ts). +// +// Runs behind Caddy (TLS) in the shared `telemax-files` compose project that +// files-service/reconcile.sh starts on the host only while some bridge has a live link. Plain +// Node built-ins, no build step and no dependencies: the container is the stock node image with +// this directory mounted read-only. Each bridge's `data/files` directory is mounted at +// /srv/; a link token is looked up in every one of them. +// +// GET /health → ok +// GET /f/ → a page: «Скачать» for a download, a file picker for an upload +// GET /f//file → the file itself (Range supported — a broken download resumes) +// PUT /f//upload → the uploaded body, streamed to disk (X-File-Name: URI-encoded) +// +// A page is not the file: link previews (MAX and Telegram fetch every URL sent into a chat) only +// ever see the page, so they can't burn an upload link or start a multi-gigabyte download. +import http from 'node:http'; +import path from 'node:path'; +import { createReadStream, createWriteStream } from 'node:fs'; +import { mkdir, open, readFile, readdir, rename, rm, stat, statfs, writeFile } from 'node:fs/promises'; +import { pipeline } from 'node:stream/promises'; +import { Transform } from 'node:stream'; + +const ROOT = process.env.FILES_ROOT || '/srv'; +const PORT = Number(process.env.PORT || 8080); +const TZ = process.env.TZ || 'Europe/Moscow'; +const TOKEN_RE = /^[A-Za-z0-9_-]{20,64}$/; + +/** @returns {Promise<{root: string, token: string, body: any} | null>} */ +async function findToken(token) { + if (!TOKEN_RE.test(token)) return null; + let roots = []; + try { + roots = (await readdir(ROOT)).map((d) => path.join(ROOT, d)); + } catch { + return null; + } + for (const root of roots) { + try { + const body = JSON.parse(await readFile(path.join(root, 'tokens', `${token}.json`), 'utf8')); + if (typeof body.expiresAt !== 'number' || body.expiresAt <= Date.now()) return null; + return { root, token, body }; + } catch { + /* not this bridge */ + } + } + return null; +} + +/** Strict inside-the-root resolution: a token's path can't point anywhere else. */ +function inside(root, rel) { + const full = path.resolve(root, rel); + return full.startsWith(path.resolve(root) + path.sep) ? full : null; +} + +function safeName(raw) { + let name = String(raw ?? '').replace(/[\u0000-\u001f\u007f/\\:*?"<>|]/g, '_').trim().replace(/^\.+/, '').trim(); + if (!name) name = 'file'; + return name.length > 180 ? name.slice(0, 180) : name; +} + +function formatBytes(b) { + if (b >= 1024 ** 3) return `${(b / 1024 ** 3).toFixed(1)} ГБ`; + if (b >= 1024 ** 2) return `${Math.round(b / 1024 ** 2)} МБ`; + return `${Math.max(1, Math.round(b / 1024))} КБ`; +} + +function formatDate(ms) { + return new Intl.DateTimeFormat('ru-RU', { day: 'numeric', month: 'long', hour: '2-digit', minute: '2-digit', timeZone: TZ }).format(new Date(ms)); +} + +const esc = (s) => String(s).replace(/[&<>"']/g, (c) => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[c]); + +const SECURITY_HEADERS = { + 'X-Robots-Tag': 'noindex, nofollow', + 'Referrer-Policy': 'no-referrer', + 'X-Content-Type-Options': 'nosniff', + 'Cache-Control': 'no-store', +}; + +function page(res, status, title, body) { + res.writeHead(status, { + ...SECURITY_HEADERS, + 'Content-Type': 'text/html; charset=utf-8', + 'Content-Security-Policy': "default-src 'none'; style-src 'unsafe-inline'; script-src 'unsafe-inline'; connect-src 'self'; form-action 'none'; frame-ancestors 'none'", + }); + res.end(` +${esc(title)}
${body}
Telemax · ссылка действует ограниченное время, потом файл удаляется
`); +} + +function notFound(res) { + page(res, 404, 'Ссылка недействительна', '

Ссылка недействительна

Ссылка устарела, уже использована или набрана с ошибкой. Попросите прислать новую.

'); +} + +async function freeBytes(root) { + const s = await statfs(root); + return Number(s.bavail) * Number(s.bsize); +} + +async function showLink(res, t) { + const { body, token } = t; + if (body.kind === 'download') { + page( + res, + 200, + body.name, + `

${esc(body.name)}

${esc(formatBytes(body.size))} · ссылка действует до ${esc(formatDate(body.expiresAt))}

+Скачать`, + ); + return; + } + if (body.kind === 'upload') { + const done = await stat(path.join(t.root, 'incoming', token, 'done.json')).then(() => true, () => false); + if (done) { + page(res, 410, 'Файл уже получен', '

Файл уже получен

Мост отправит его в MAX. Эту страницу можно закрыть.

'); + return; + } + const maxBytes = Math.max(0, (await freeBytes(t.root)) - (body.reserveBytes ?? 0)); + page( + res, + 200, + 'Загрузка файла', + `

Загрузите файл

Ждём «${esc(body.name)}» (${esc(formatBytes(body.expectedSize ?? 0))}). Мост отправит его в MAX.
Можно выбрать и другой файл — до ${esc(formatBytes(maxBytes))}. Ссылка действует до ${esc(formatDate(body.expiresAt))}.

+ +
+`, + ); + return; + } + notFound(res); +} + +async function sendFile(req, res, t) { + const { root, body } = t; + if (body.kind !== 'download') return notFound(res); + const full = inside(root, body.path); + if (!full) return notFound(res); + let size; + try { + size = (await stat(full)).size; + } catch { + return notFound(res); + } + const name = safeName(body.name); + const headers = { + ...SECURITY_HEADERS, + 'Content-Type': 'application/octet-stream', + 'Content-Disposition': `attachment; filename="${name.replace(/[^\x20-\x7e]/g, '_')}"; filename*=UTF-8''${encodeURIComponent(name)}`, + 'Accept-Ranges': 'bytes', + }; + const m = /^bytes=(\d*)-(\d*)$/.exec(req.headers.range ?? ''); + if (m && (m[1] || m[2])) { + let start = m[1] ? Number(m[1]) : size - Number(m[2]); + let end = m[1] && m[2] ? Number(m[2]) : size - 1; + if (start < 0) start = 0; + if (end >= size) end = size - 1; + if (start > end || start >= size) { + res.writeHead(416, { ...headers, 'Content-Range': `bytes */${size}` }); + return res.end(); + } + res.writeHead(206, { ...headers, 'Content-Range': `bytes ${start}-${end}/${size}`, 'Content-Length': String(end - start + 1) }); + if (req.method === 'HEAD') return res.end(); + return pipeline(createReadStream(full, { start, end }), res).catch(() => {}); + } + res.writeHead(200, { ...headers, 'Content-Length': String(size) }); + if (req.method === 'HEAD') return res.end(); + return pipeline(createReadStream(full), res).catch(() => {}); +} + +function reply(res, status, text) { + res.writeHead(status, { ...SECURITY_HEADERS, 'Content-Type': 'text/plain; charset=utf-8' }); + res.end(text); +} + +async function receiveUpload(req, res, t) { + const { root, token, body } = t; + if (body.kind !== 'upload') return reply(res, 404, 'Ссылка недействительна'); + const length = Number(req.headers['content-length']); + if (!Number.isFinite(length) || length <= 0) return reply(res, 411, 'Не указан размер файла'); + const room = (await freeBytes(root)) - (body.reserveBytes ?? 0); + if (length > room) return reply(res, 507, `Не хватает места на сервере: свободно ${formatBytes(Math.max(0, room))}`); + + const dir = path.join(root, 'incoming', token); + await mkdir(dir, { recursive: true }); + if (await stat(path.join(dir, 'done.json')).then(() => true, () => false)) return reply(res, 410, 'Файл по этой ссылке уже получен'); + // One upload at a time per link: the lock file is created exclusively and removed on failure. + let lock; + try { + lock = await open(path.join(dir, '.lock'), 'wx'); + } catch { + return reply(res, 409, 'По этой ссылке уже идёт загрузка'); + } + let rawName = 'file'; + try { + rawName = decodeURIComponent(String(req.headers['x-file-name'] ?? 'file')); + } catch { + /* keep the default */ + } + const name = safeName(rawName); + const part = path.join(dir, `${name}.part`); + let received = 0; + const guard = new Transform({ + transform(chunk, _enc, cb) { + received += chunk.length; + cb(received > length ? new Error('body longer than Content-Length') : null, chunk); + }, + }); + try { + await pipeline(req, guard, createWriteStream(part)); + if (received !== length) throw new Error(`got ${received} of ${length} bytes`); + await rename(part, path.join(dir, name)); + await writeFile(path.join(dir, 'done.json'), JSON.stringify({ name, file: name, size: received, at: Date.now() })); + reply(res, 200, 'ok'); + } catch (err) { + await rm(part, { force: true }); + if (!res.headersSent) reply(res, 400, `Загрузка прервалась: ${err.message}`); + } finally { + await lock.close().catch(() => {}); + await rm(path.join(dir, '.lock'), { force: true }); + } +} + +const server = http.createServer(async (req, res) => { + try { + const url = new URL(req.url ?? '/', 'http://x'); + if (url.pathname === '/health') return reply(res, 200, 'ok'); + const m = /^\/f\/([A-Za-z0-9_-]+)(\/file|\/upload)?\/?$/.exec(url.pathname); + if (!m) return notFound(res); + const t = await findToken(m[1]); + if (!t) return m[2] === '/upload' ? reply(res, 404, 'Ссылка недействительна или устарела') : notFound(res); + if (!m[2] && (req.method === 'GET' || req.method === 'HEAD')) return await showLink(res, t); + if (m[2] === '/file' && (req.method === 'GET' || req.method === 'HEAD')) return await sendFile(req, res, t); + if (m[2] === '/upload' && (req.method === 'PUT' || req.method === 'POST')) return await receiveUpload(req, res, t); + reply(res, 405, 'Method not allowed'); + } catch (err) { + console.error(err); + if (!res.headersSent) reply(res, 500, 'Внутренняя ошибка'); + else res.destroy(); + } +}); +// Gigabyte uploads outlast Node's default 5-minute request timeout. +server.requestTimeout = 0; +server.listen(PORT, () => console.log(`telemax files service on :${PORT}, roots under ${ROOT}`)); +for (const sig of ['SIGTERM', 'SIGINT']) process.on(sig, () => server.close(() => process.exit(0))); diff --git a/package-lock.json b/package-lock.json index fa1c7b9..ff8de1b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "telemax", - "version": "1.2.1", + "version": "1.3.0-beta.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "telemax", - "version": "1.2.1", + "version": "1.3.0-beta.1", "license": "AGPL-3.0-only", "dependencies": { "dotenv": "^17.2.3", diff --git a/package.json b/package.json index 0f0aa29..03ed359 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "name": "telemax", "private": true, "license": "AGPL-3.0-only", - "version": "1.2.1", + "version": "1.3.0-beta.1", "type": "module", "scripts": { "dev": "tsx watch src/server/app.ts", diff --git a/src/bridge/attachments.ts b/src/bridge/attachments.ts index 63dee3b..64cce4a 100644 --- a/src/bridge/attachments.ts +++ b/src/bridge/attachments.ts @@ -189,6 +189,17 @@ async function withDownloadFallback( } } +/** + * The CDN URL of a MAX FILE attach (FILE_DOWNLOAD with the forward fallback) — for the big-file + * path, which streams it to disk itself. Null when MAX refuses for good; TransientDownloadError + * when the socket was down. + */ +export async function getMaxFileUrl(att: MaxAttachment, ctx: DownloadContext): Promise { + if (att._type !== 'FILE' || att.fileId == null) return null; + const fileId = att.fileId; + return withDownloadFallback(ctx, `FILE_DOWNLOAD fileId ${String(fileId)}`, (chatId, messageId) => ctx.max.getFileDownloadUrl(chatId, messageId, fileId)); +} + /** * Downloads a MAX attachment for re-upload to Telegram. Null means "no file to send" — the caller * posts describeAttachment()'s placeholder. A transient failure (see TransientDownloadError) is @@ -368,6 +379,21 @@ export function pressableButton(key: string): PressableButton | undefined { return pressable.get(key); } +/** + * The public deep link of a MAX mini-app button: `https://max.ru/?startapp=` (MAX + * docs, dev.max.ru/docs/webapps/introduction). Live shape 2026-10-06: {type:'OPEN_APP', text, + * webApp:'maxnotifications_bot', contactId, payload}. MAX drops a payload over 512 chars or with + * anything but [A-Za-z0-9_-], so such a payload is left out (the app opens on its start screen). + * The payload is per-account — fine, the button only ever reaches the account it was sent to. + */ +export function miniAppLink(b: Record | null | undefined): string | undefined { + const app = typeof b?.webApp === 'string' ? b.webApp : ''; + if (!/^[A-Za-z0-9_]{1,64}$/.test(app)) return undefined; + const payload = typeof b?.payload === 'string' ? b.payload : ''; + const start = /^[A-Za-z0-9_-]{1,512}$/.test(payload) ? `=${payload}` : ''; + return `https://max.ru/${app}?startapp${start}`; +} + /** callback_data of a MAX button with no Telegram counterpart (yet): pressing it only shows a hint. */ export const KEYBOARD_NA = 'tlmx_kb_na'; @@ -403,6 +429,9 @@ export function maxKeyboardToTelegram(att: unknown, chatId?: unknown): InlineKey } else if (chatId != null && b?.type === 'MESSAGE') { // Pressing it in MAX sends the button's text to the bot as the user's own message. buttons.push({ text, callback_data: registerButton({ chatId, callbackId: '', text, sendsText: true }) }); + } else if (b?.type === 'OPEN_APP' && miniAppLink(b)) { + // A mini-app can't run inside Telegram, but its public deep link opens it in MAX. + buttons.push({ text, url: miniAppLink(b)! }); } else if (b?.type === 'REQUEST_GEO_LOCATION') { buttons.push({ text, callback_data: KEYBOARD_GEO }); } else if (b?.type === 'REQUEST_CONTACT') { diff --git a/src/bridge/bigFiles.ts b/src/bridge/bigFiles.ts new file mode 100644 index 0000000..67c911e --- /dev/null +++ b/src/bridge/bigFiles.ts @@ -0,0 +1,198 @@ +/** + * The chat side of big files (storage and links: fileShare.ts). Every message here says WHY a file + * goes by link — to the user a link instead of a file otherwise looks like a bug. + */ +import { Markup, type Telegraf } from 'telegraf'; +import type { MaxClient } from '../max/client.js'; +import { maxFetchBig } from '../max/ca.js'; +import { createLogger } from '../logger.js'; +import { getMaxFileUrl, describeAttachment, type DownloadContext, type MaxAttachment } from './attachments.js'; +import { FileShare, MAX_FILE_LIMIT, nodeReadable, type UploadTicket } from './fileShare.js'; +import { formatBytes } from './status.js'; +import { uploadFileFromDiskToMax } from './upload.js'; +import { isTransientHttpStatus, isTransientNetworkError, TransientDownloadError } from './transient.js'; + +const logger = createLogger('big-files'); + +const WHY_TG_DOWNLOAD = 'Telegram не отдаёт ботам файлы больше 20 МБ'; +const WHY_TG_UPLOAD = 'Telegram не принимает от ботов файлы больше 50 МБ'; + +export function formatExpiry(ms: number): string { + return new Intl.DateTimeFormat('ru-RU', { day: 'numeric', month: 'long', hour: '2-digit', minute: '2-digit', timeZone: process.env.TZ || 'Europe/Moscow' }).format(new Date(ms)); +} + +type SentMessage = { message_id: number }; +type SendOpts = { message_thread_id: number; reply_parameters?: { message_id: number; allow_sending_without_reply: boolean } }; + +export interface BigFilesDeps { + bot: Telegraf; + max: MaxClient; + targetGroupId: string; + fileShare: FileShare; + /** Records a message the bridge itself sent to MAX (echo suppression + the TG↔MAX link). */ + onSentToMax: (ticket: UploadTicket, sent: { cid: number; messageId: unknown; time: unknown }) => void; +} + +export interface BigFiles { + /** Telegram → MAX: a file the bot can't download — answer with an upload link. */ + offerUpload(p: { maxChatId: string; topicId: number; telegramMessageId: number; caption?: string; name: string; size: number }): Promise; + /** MAX → Telegram: a FILE over the bot upload limit — save it and post a download link. */ + relayFromMax(att: MaxAttachment, ctx: DownloadContext, opts: SendOpts): Promise; +} + +export function createBigFiles({ bot, max, targetGroupId, fileShare, onSentToMax }: BigFilesDeps): BigFiles { + const send = (text: string, opts: SendOpts, markup?: ReturnType) => + bot.telegram.sendMessage(targetGroupId, text, { ...opts, ...(markup ? { reply_markup: markup.reply_markup } : {}) }); + const edit = async (messageId: number | undefined, topicId: number, text: string, markup?: ReturnType) => { + if (messageId != null) { + try { + await bot.telegram.editMessageText(targetGroupId, messageId, undefined, text, markup ? { reply_markup: markup.reply_markup } : {}); + return; + } catch (err) { + if (/not modified/i.test((err as Error).message ?? '')) return; + logger.error('Editing a big-file notice failed — posting a new one', err); + } + } + await send(text, { message_thread_id: topicId }, markup).catch((e) => logger.error('Posting a big-file notice failed', e)); + }; + const reserveText = formatBytes(fileShare.opts.reserveBytes); + + // --- Telegram → MAX ------------------------------------------------------------------------- + + async function offerUpload(p: Parameters[0]): Promise { + const label = `«${p.name}» (${formatBytes(p.size)})`; + const opts: SendOpts = { message_thread_id: p.topicId, reply_parameters: { message_id: p.telegramMessageId, allow_sending_without_reply: true } }; + if (!fileShare.enabled) { + await send(`⚠️ Файл ${label} не отправлен в MAX: ${WHY_TG_DOWNLOAD}. Отправьте его из самого MAX или поделите на части.`, opts); + return; + } + const room = await fileShare.roomFor(p.size); + if (!room.fits) { + await send( + `⚠️ Файл ${label} не отправлен в MAX: ${WHY_TG_DOWNLOAD}, а принять его по ссылке не выйдет — на сервере свободно ${formatBytes(room.freeBytes)}, нужно ${formatBytes(p.size)} и ещё ${reserveText} запаса для обновлений.`, + opts, + ); + return; + } + const prompt = await send(`⏳ Файл ${label} больше 20 МБ — ${WHY_TG_DOWNLOAD}. Готовлю ссылку, чтобы загрузить его в обход Telegram…`, opts); + const ticket = await fileShare.createUploadTicket({ + maxChatId: p.maxChatId, + topicId: p.topicId, + telegramMessageId: p.telegramMessageId, + caption: p.caption, + name: p.name, + expectedSize: p.size, + promptMessageId: prompt.message_id, + }); + logger.info(`TG -> MAX: ${p.name} is ${p.size} bytes, over the bot download limit — offering an upload link`); + const svc = await fileShare.ensureService(); + if (!svc.ok) { + await fileShare.dropTicket(ticket.token); + await edit(prompt.message_id, p.topicId, `⚠️ Файл ${label} не отправлен в MAX: ${WHY_TG_DOWNLOAD}, а ссылку для загрузки подготовить не удалось — ${svc.reason}.`); + return; + } + await edit( + prompt.message_id, + p.topicId, + `📤 Файл ${label} больше 20 МБ — ${WHY_TG_DOWNLOAD}.\nЗагрузите его по ссылке, и мост отправит его в MAX. Ссылка действует до ${formatExpiry(ticket.expiresAt)}.`, + Markup.inlineKeyboard([Markup.button.url('📤 Загрузить файл', `${svc.url}/f/${ticket.token}`)]), + ); + } + + fileShare.onUpload(async (ticket, filePath, size, name) => { + const label = `«${name}» (${formatBytes(size)})`; + if (size <= MAX_FILE_LIMIT) { + await edit(ticket.promptMessageId, ticket.topicId, `⏳ Файл ${label} получен — отправляю в MAX…`); + try { + const attach = await uploadFileFromDiskToMax(max, filePath, size, name); + const sent = await max.sendMessage(ticket.maxChatId, ticket.caption ?? null, [attach]); + onSentToMax(ticket, sent); + await fileShare.dropTicket(ticket.token); + await edit(ticket.promptMessageId, ticket.topicId, `✅ Файл ${label} отправлен в MAX.`); + logger.info(`TG -> MAX: uploaded ${name} (${size} bytes) sent to chat ${ticket.maxChatId}`); + } catch (err) { + logger.error(`Sending the uploaded ${name} to MAX failed — keeping it as a link`, err); + const file = await fileShare.adoptUpload(ticket, filePath, size, name); + await edit( + ticket.promptMessageId, + ticket.topicId, + `⚠️ Файл ${label} получен, но в MAX не ушёл: ${(err as Error).message}.\nОн сохранён до ${formatExpiry(file.expiresAt)} — ссылку на него можно взять в пульте → 📁 Файлы и отправить вручную.`, + ); + } + return; + } + // Over MAX's own limit: the recipient gets a download link instead of the file. + const file = await fileShare.adoptUpload(ticket, filePath, size, name); + const svc = await fileShare.ensureService(30_000); + if (!svc.ok) { + await edit(ticket.promptMessageId, ticket.topicId, `⚠️ Файл ${label} получен, но ссылку для MAX подготовить не удалось — ${svc.reason}. Файл сохранён до ${formatExpiry(file.expiresAt)}, ссылка — в пульте → 📁 Файлы.`); + return; + } + const url = `${svc.url}/f/${file.token}`; + const text = `📎 ${name} (${formatBytes(size)}) — файл больше 4 ГБ, MAX такие не принимает.\nСкачать до ${formatExpiry(file.expiresAt)}: ${url}${ticket.caption ? `\n\n${ticket.caption}` : ''}`; + try { + const sent = await max.sendMessage(ticket.maxChatId, text, []); + onSentToMax(ticket, sent); + await edit( + ticket.promptMessageId, + ticket.topicId, + `✅ Файл ${label} больше 4 ГБ — MAX такие не принимает, поэтому собеседнику ушла ссылка на скачивание (до ${formatExpiry(file.expiresAt)}). Ссылка ведёт на ваш сервер. Управлять файлом — в пульте → 📁 Файлы.`, + ); + } catch (err) { + logger.error(`Sending the link for ${name} to MAX failed`, err); + await edit(ticket.promptMessageId, ticket.topicId, `⚠️ Ссылку на файл ${label} отправить в MAX не удалось: ${(err as Error).message}. Файл сохранён — ссылка в пульте → 📁 Файлы.`); + } + }); + + // --- MAX → Telegram ------------------------------------------------------------------------- + + async function relayFromMax(att: MaxAttachment, ctx: DownloadContext, opts: SendOpts): Promise { + const name = att.name ?? 'file'; + const size = Number(att.size ?? 0); + const label = `«${name}» (${formatBytes(size)})`; + if (!fileShare.enabled) return send(`${describeAttachment(att)} ${formatBytes(size)} — ${WHY_TG_UPLOAD}. Откройте его в MAX.`, opts); + const room = await fileShare.roomFor(size); + if (!room.fits) { + return send( + `📎 Файл ${label} — ${WHY_TG_UPLOAD}, а сохранить его для скачивания не выйдет: на сервере свободно ${formatBytes(room.freeBytes)}, нужно ещё ${reserveText} запаса для обновлений. Откройте его в MAX.`, + opts, + ); + } + const url = await getMaxFileUrl(att, ctx); + if (!url) return send(describeAttachment(att), opts); + let file; + try { + const res = await maxFetchBig(url); + if (!res.ok || !res.body) { + if (isTransientHttpStatus(res.status)) throw new TransientDownloadError(`MAX CDN answered ${res.status}`); + throw new Error(`MAX отдал ${res.status}`); + } + file = await fileShare.storeStream(nodeReadable(res.body), name, size); + logger.info(`MAX -> TG: ${name} (${file.size} bytes) saved for a download link`); + } catch (err) { + const transient = err instanceof TransientDownloadError || isTransientNetworkError(err); + if (transient && ctx.throwOnTransient) throw err instanceof TransientDownloadError ? err : new TransientDownloadError(`big file download: ${(err as Error).message}`, err); + logger.error(`Saving the big MAX file ${name} failed`, err); + return send(`📎 Файл ${label} — ${WHY_TG_UPLOAD}, а сохранить его для скачивания не удалось (${(err as Error).message}). Откройте его в MAX.`, opts); + } + const msg = await send(`📎 Файл ${label} — ${WHY_TG_UPLOAD}. Готовлю ссылку для скачивания…`, opts); + // Not awaited: the service may take a minute or two to come up, and this runs inside the + // chat's queue — the next message must not wait for it. + void (async () => { + const svc = await fileShare.ensureService(); + if (!svc.ok) { + await edit(msg.message_id, opts.message_thread_id, `📎 Файл ${label} — ${WHY_TG_UPLOAD}. Ссылку подготовить не удалось: ${svc.reason}.\nФайл сохранён до ${formatExpiry(file.expiresAt)} — новую ссылку можно получить в пульте → 📁 Файлы.`); + return; + } + await edit( + msg.message_id, + opts.message_thread_id, + `📎 Файл ${label} — ${WHY_TG_UPLOAD}, поэтому он лежит на вашем сервере. Скачайте по ссылке до ${formatExpiry(file.expiresAt)} — потом файл удалится.`, + Markup.inlineKeyboard([Markup.button.url('⬇️ Скачать', `${svc.url}/f/${file.token}`)]), + ); + })().catch((err) => logger.error('Posting the download link failed', err)); + return msg; + } + + return { offerUpload, relayFromMax }; +} diff --git a/src/bridge/fileShare.ts b/src/bridge/fileShare.ts new file mode 100644 index 0000000..eed081a --- /dev/null +++ b/src/bridge/fileShare.ts @@ -0,0 +1,452 @@ +/** + * Big files through links — what the Bot API can't carry. A bot may DOWNLOAD from Telegram only + * ≤20 MB (getFile) and SEND only ≤50 MB, while MAX takes up to 4 GB. So: + * - Telegram → MAX over 20 MB: the topic gets a one-time UPLOAD link; the file arrives through a + * browser, then goes to MAX as a normal file (≤4 GB) or as a download link (bigger); + * - MAX → Telegram over 50 MB: the bridge saves the file and the topic gets a DOWNLOAD link. + * + * The links are served by ONE shared service per host (files-service/: Caddy + a small Node app), + * which the host's update dispatcher starts only while some bridge has a live link and stops when + * none is left — so ports 80/443 are open only then. The bridge (non-root, no Docker access) talks + * to it through files on the shared `.data/files` volume: + * active — exists while this bridge has a live link (the host reads it every minute); + * service.json — written by the host: {state, url, at, detail}; + * tokens/.json — what a link may do (download a stored file / receive one upload); + * items//… — stored files, deleted when their link expires; + * incoming//… — uploads written by the service (`done.json` once complete). + */ +import path from 'node:path'; +import { createWriteStream } from 'node:fs'; +import { mkdir, readFile, readdir, rename, rm, stat, statfs, unlink, writeFile } from 'node:fs/promises'; +import { randomBytes } from 'node:crypto'; +import { pipeline } from 'node:stream/promises'; +import { Readable, Transform } from 'node:stream'; +import { createLogger } from '../logger.js'; +import { parseEnvBool } from '../env.js'; + +const logger = createLogger('files'); + +/** Bot API getFile refuses bigger files with "400: Bad Request: file is too big". */ +export const TELEGRAM_BOT_DOWNLOAD_LIMIT = 20 * 1024 * 1024; +/** MAX's own limit for one file (the user's number, 2026-10-09); bigger goes to MAX as a link. */ +export const MAX_FILE_LIMIT = 4_000_000_000; + +export interface StoredFile { + id: string; + name: string; + size: number; + /** max2tg: saved from MAX for the topic; tg2max: uploaded for MAX (over MAX's limit). */ + direction: 'max2tg' | 'tg2max'; + token: string; + createdAt: number; + expiresAt: number; +} + +export interface UploadTicket { + token: string; + maxChatId: string; + topicId: number; + /** The Telegram message with the too-big file — the MAX copy gets linked to it. */ + telegramMessageId: number; + /** The bot's «загрузите по ссылке» message, edited as the upload progresses. */ + promptMessageId?: number; + caption?: string; + name: string; + expectedSize: number; + createdAt: number; + expiresAt: number; +} + +interface Index { + files: StoredFile[]; + tickets: UploadTicket[]; +} + +export type ServiceResult = { ok: true; url: string } | { ok: false; reason: string }; + +/** What the host writes into service.json. */ +interface ServiceState { + state?: string; + url?: string; + /** Unix seconds. */ + at?: number; + detail?: string; +} + +export interface FileShareOptions { + enabled: boolean; + ttlMs: number; + /** Kept free on top of any file: an update needs ~1.5 GB, a full disk would block it. */ + reserveBytes: number; +} + +export function fileShareOptionsFromEnv(env: NodeJS.ProcessEnv = process.env): FileShareOptions { + const ttlHours = Number(env.FILES_TTL_HOURS); + const reserveMb = Number(env.FILES_RESERVE_MB); + return { + enabled: parseEnvBool(env.FILES) !== false, + ttlMs: (Number.isFinite(ttlHours) && ttlHours > 0 ? ttlHours : 72) * 3600_000, + reserveBytes: (Number.isFinite(reserveMb) && reserveMb >= 0 ? reserveMb : 1536) * 1024 * 1024, + }; +} + +/** + * A file name safe for a path segment and a Content-Disposition: no separators, no control + * characters, no leading dots, at most 180 UTF-16 units (extension kept). Pure + exported for tests. + */ +export function safeFileName(raw: string | undefined): string { + let name = (raw ?? '').replace(/[\u0000-\u001f\u007f/\\:*?"<>|]/g, '_').trim(); + name = name.replace(/^\.+/, '').trim(); + if (!name) return 'file'; + if (name.length > 180) { + const dot = name.lastIndexOf('.'); + const ext = dot > 0 && name.length - dot <= 16 ? name.slice(dot) : ''; + name = name.slice(0, 180 - ext.length) + ext; + } + return name; +} + +const newToken = (): string => randomBytes(18).toString('base64url'); +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); + +/** How long service.json stays believable: the host rewrites it every minute while it matters. */ +const SERVICE_STATE_FRESH_MS = 150_000; +const SERVICE_WAIT_MS = 150_000; +const HEARTBEAT_STALE_MS = 5 * 60_000; + +export class FileShare { + readonly dir: string; + private readonly indexPath: string; + private index: Index = { files: [], tickets: [] }; + private chain: Promise = Promise.resolve(); + private uploadHandler: ((t: UploadTicket, filePath: string, size: number, name: string) => Promise) | null = null; + private busyTokens = new Set(); + + constructor( + dataDir: string, + readonly opts: FileShareOptions, + ) { + this.dir = path.join(dataDir, 'files'); + this.indexPath = path.join(this.dir, 'index.json'); + } + + get enabled(): boolean { + return this.opts.enabled; + } + + async init(): Promise { + await mkdir(path.join(this.dir, 'items'), { recursive: true }); + await mkdir(path.join(this.dir, 'tokens'), { recursive: true }); + await mkdir(path.join(this.dir, 'incoming'), { recursive: true }); + try { + const parsed = JSON.parse(await readFile(this.indexPath, 'utf8')) as Partial; + this.index = { files: Array.isArray(parsed.files) ? parsed.files : [], tickets: Array.isArray(parsed.tickets) ? parsed.tickets : [] }; + } catch { + this.index = { files: [], tickets: [] }; + } + await this.sweep(); + } + + /** Serialized: index writes and the token files they imply never interleave. */ + private locked(fn: () => Promise): Promise { + const run = this.chain.then(fn, fn); + this.chain = run.catch(() => {}); + return run; + } + + private async save(): Promise { + const tmp = `${this.indexPath}.tmp`; + await writeFile(tmp, JSON.stringify(this.index, null, 2), 'utf8'); + await rename(tmp, this.indexPath); + await this.syncActiveMarker(); + } + + /** `active` exists exactly while a link is alive — the host opens ports 80/443 only then. */ + private async syncActiveMarker(): Promise { + const marker = path.join(this.dir, 'active'); + if (this.index.files.length + this.index.tickets.length > 0) { + await writeFile(marker, new Date().toISOString(), 'utf8'); + } else { + await unlink(marker).catch(() => {}); + } + } + + private async writeToken(token: string, body: Record): Promise { + const file = path.join(this.dir, 'tokens', `${token}.json`); + await writeFile(`${file}.tmp`, JSON.stringify({ v: 1, ...body }), 'utf8'); + await rename(`${file}.tmp`, file); + } + + private async dropToken(token: string): Promise { + await unlink(path.join(this.dir, 'tokens', `${token}.json`)).catch(() => {}); + } + + list(): StoredFile[] { + return [...this.index.files].sort((a, b) => b.createdAt - a.createdAt); + } + + get(id: string): StoredFile | undefined { + return this.index.files.find((f) => f.id === id); + } + + pendingUploads(): number { + return this.index.tickets.length; + } + + /** Free bytes left for a file of `size` after the reserve; negative when it won't fit. */ + async roomFor(size: number): Promise<{ fits: boolean; freeBytes: number }> { + const s = await statfs(this.dir); + const freeBytes = Number(s.bavail) * Number(s.bsize); + return { fits: freeBytes - this.opts.reserveBytes >= size, freeBytes }; + } + + // --- Telegram → MAX --------------------------------------------------------------------- + + async createUploadTicket(t: Omit): Promise { + return this.locked(async () => { + const now = Date.now(); + const ticket: UploadTicket = { ...t, token: newToken(), createdAt: now, expiresAt: now + this.opts.ttlMs }; + await this.writeToken(ticket.token, { + kind: 'upload', + name: ticket.name, + expectedSize: ticket.expectedSize, + expiresAt: ticket.expiresAt, + reserveBytes: this.opts.reserveBytes, + }); + this.index.tickets.push(ticket); + await this.save(); + return ticket; + }); + } + + async setPromptMessage(token: string, messageId: number): Promise { + await this.locked(async () => { + const t = this.index.tickets.find((x) => x.token === token); + if (!t) return; + t.promptMessageId = messageId; + await this.save(); + }); + } + + async dropTicket(token: string): Promise { + await this.locked(async () => { + this.index.tickets = this.index.tickets.filter((t) => t.token !== token); + await this.dropToken(token); + await rm(path.join(this.dir, 'incoming', token), { recursive: true, force: true }); + await this.save(); + }); + } + + /** Moves a finished upload into storage (a file over MAX's limit lives on as a download link). */ + async adoptUpload(ticket: UploadTicket, filePath: string, size: number, name: string): Promise { + const file = await this.locked(async () => { + const id = newToken(); + await mkdir(path.join(this.dir, 'items', id), { recursive: true }); + const target = path.join(this.dir, 'items', id, safeFileName(name)); + await rename(filePath, target); + return this.addFile(id, safeFileName(name), size, 'tg2max'); + }); + await this.dropTicket(ticket.token); + return file; + } + + /** Called with each finished upload; the bridge sends it to MAX. */ + onUpload(handler: (t: UploadTicket, filePath: string, size: number, name: string) => Promise): void { + this.uploadHandler = handler; + } + + // --- MAX → Telegram --------------------------------------------------------------------- + + /** + * Streams `body` into storage and opens a download link for it. Throws when the stream fails or + * runs past `size` + the free space (the partial file is removed either way). + */ + async storeStream(body: Readable, name: string, size: number): Promise { + const id = newToken(); + const safe = safeFileName(name); + const dir = path.join(this.dir, 'items', id); + await mkdir(dir, { recursive: true }); + const part = path.join(dir, `${safe}.part`); + let written = 0; + const limit = Math.max(size, 0) + 1024 * 1024; // the declared size, a little slack + const guard = new Transform({ + transform(chunk: Buffer, _enc, cb) { + written += chunk.length; + if (written > limit) cb(new Error(`file grew past its declared ${size} bytes`)); + else cb(null, chunk); + }, + }); + try { + await pipeline(body, guard, createWriteStream(part)); + await rename(part, path.join(dir, safe)); + } catch (err) { + await rm(dir, { recursive: true, force: true }); + throw err; + } + return this.locked(() => this.addFile(id, safe, written, 'max2tg')); + } + + private async addFile(id: string, name: string, size: number, direction: StoredFile['direction']): Promise { + const now = Date.now(); + const file: StoredFile = { id, name, size, direction, token: newToken(), createdAt: now, expiresAt: now + this.opts.ttlMs }; + await this.writeToken(file.token, { kind: 'download', name, size, path: `items/${id}/${name}`, expiresAt: file.expiresAt }); + this.index.files.push(file); + await this.save(); + return file; + } + + // --- the panel ---------------------------------------------------------------------------- + + /** A fresh link (the old one stops working) and a fresh TTL. */ + async renew(id: string): Promise { + return this.locked(async () => { + const file = this.index.files.find((f) => f.id === id); + if (!file) return undefined; + await this.dropToken(file.token); + file.token = newToken(); + file.expiresAt = Date.now() + this.opts.ttlMs; + await this.writeToken(file.token, { kind: 'download', name: file.name, size: file.size, path: `items/${id}/${file.name}`, expiresAt: file.expiresAt }); + await this.save(); + return file; + }); + } + + async remove(id: string): Promise { + return this.locked(async () => { + const file = this.index.files.find((f) => f.id === id); + if (!file) return false; + this.index.files = this.index.files.filter((f) => f.id !== id); + await this.dropToken(file.token); + await rm(path.join(this.dir, 'items', id), { recursive: true, force: true }); + await this.save(); + return true; + }); + } + + // --- the host service ----------------------------------------------------------------------- + + /** + * Waits (≤2.5 min: the host checks once a minute, then Caddy may need a certificate) until the + * shared files service answers for this bridge. Call AFTER creating the link — its `active` + * marker is what makes the host start the service. + */ + async ensureService(waitMs = SERVICE_WAIT_MS): Promise { + if (!this.opts.enabled) return { ok: false, reason: 'пересылка больших файлов отключена в настройках (FILES=off)' }; + await this.syncActiveMarker(); + const deadline = Date.now() + waitMs; + let heartbeatChecked = false; + for (;;) { + const st = await this.readServiceState(); + const fresh = st?.at != null && Date.now() - st.at * 1000 < SERVICE_STATE_FRESH_MS; + if (fresh && st?.state === 'up' && st.url) return { ok: true, url: st.url.replace(/\/+$/, '') }; + if (fresh && st?.state && st.state !== 'up' && st.state !== 'starting' && st.state !== 'down') { + return { ok: false, reason: describeServiceProblem(st) }; + } + if (!heartbeatChecked) { + heartbeatChecked = true; + if (!(await this.watcherAlive())) { + return { + ok: false, + reason: 'службу файлов запускает автообновление, а оно на этом сервере не работает (setup.sh запускали не от root?)', + }; + } + } + if (Date.now() >= deadline) return { ok: false, reason: 'служба файлов не запустилась за 2,5 минуты — подробности в /var/lib/telemax-files на сервере' }; + await sleep(3000); + } + } + + private async readServiceState(): Promise { + try { + return JSON.parse(await readFile(path.join(this.dir, 'service.json'), 'utf8')) as ServiceState; + } catch { + return null; + } + } + + private async watcherAlive(): Promise { + try { + const s = await stat(path.join(this.dir, '..', 'watcher-heartbeat')); + return Date.now() - s.mtimeMs < HEARTBEAT_STALE_MS; + } catch { + return false; + } + } + + // --- housekeeping --------------------------------------------------------------------------- + + /** Drops expired links with their files, and anything on disk the index no longer knows. */ + async sweep(): Promise { + await this.locked(async () => { + const now = Date.now(); + const expiredFiles = this.index.files.filter((f) => f.expiresAt <= now); + const expiredTickets = this.index.tickets.filter((t) => t.expiresAt <= now && !this.busyTokens.has(t.token)); + this.index.files = this.index.files.filter((f) => f.expiresAt > now); + this.index.tickets = this.index.tickets.filter((t) => !expiredTickets.includes(t)); + for (const f of expiredFiles) logger.info(`File «${f.name}» expired — deleted`); + const liveTokens = new Set([...this.index.files.map((f) => f.token), ...this.index.tickets.map((t) => t.token)]); + const liveItems = new Set(this.index.files.map((f) => f.id)); + for (const name of await readdir(path.join(this.dir, 'tokens')).catch(() => [] as string[])) { + if (!liveTokens.has(name.replace(/\.json(\.tmp)?$/, ''))) await unlink(path.join(this.dir, 'tokens', name)).catch(() => {}); + } + for (const name of await readdir(path.join(this.dir, 'items')).catch(() => [] as string[])) { + if (!liveItems.has(name)) await rm(path.join(this.dir, 'items', name), { recursive: true, force: true }); + } + for (const name of await readdir(path.join(this.dir, 'incoming')).catch(() => [] as string[])) { + if (!liveTokens.has(name)) await rm(path.join(this.dir, 'incoming', name), { recursive: true, force: true }); + } + await this.save(); + }); + } + + /** Hands every finished upload to the handler, one at a time. */ + async pollUploads(): Promise { + if (!this.uploadHandler) return; + for (const ticket of [...this.index.tickets]) { + if (this.busyTokens.has(ticket.token)) continue; + const dir = path.join(this.dir, 'incoming', ticket.token); + let done: { name?: string; size?: number; file?: string } | null; + try { + done = JSON.parse(await readFile(path.join(dir, 'done.json'), 'utf8')) as typeof done; + } catch { + continue; + } + const name = safeFileName(done?.name ?? ticket.name); + const filePath = path.join(dir, done?.file ?? name); + this.busyTokens.add(ticket.token); + try { + const size = (await stat(filePath)).size; + logger.info(`Upload for «${name}» (${size} bytes) arrived — sending it on`); + await this.uploadHandler(ticket, filePath, size, name); + } catch (err) { + logger.error(`Processing the upload «${name}» failed`, err); + } finally { + this.busyTokens.delete(ticket.token); + } + } + } + + /** Upload polling every 5 s, sweeping every 10 min. Timers are unref'd. */ + startBackground(): void { + const poll = setInterval(() => void this.pollUploads().catch((err) => logger.error('pollUploads failed', err)), 5000); + poll.unref(); + const sweep = setInterval(() => void this.sweep().catch((err) => logger.error('sweep failed', err)), 10 * 60_000); + sweep.unref(); + } +} + +function describeServiceProblem(st: ServiceState): string { + switch (st.state) { + case 'ports-busy': + return 'порты 80/443 на сервере заняты другим сервисом'; + case 'no-ip': + return 'не удалось определить внешний IPv4-адрес сервера (можно указать свой домен: FILES_DOMAIN в .env)'; + default: + return `служба файлов не запустилась${st.detail ? `: ${st.detail}` : ''}`; + } +} + +/** Readable from a WHATWG stream (undici's Response.body). */ +export function nodeReadable(body: ReadableStream): Readable { + return Readable.fromWeb(body as Parameters[0]); +} diff --git a/src/bridge/panel.ts b/src/bridge/panel.ts index 8904d36..86cefc2 100644 --- a/src/bridge/panel.ts +++ b/src/bridge/panel.ts @@ -3,9 +3,11 @@ import { readFile, writeFile, mkdir } from 'node:fs/promises'; import { Markup, type Telegraf, type Context } from 'telegraf'; import type { MaxClient, MaxContactInfo } from '../max/client.js'; import { getAppVersion } from './version.js'; -import { maskPhone } from './status.js'; +import { formatBytes, maskPhone } from './status.js'; import { createLogger } from '../logger.js'; import { truncateUtf16 } from './text.js'; +import { formatExpiry } from './bigFiles.js'; +import type { FileShare, StoredFile } from './fileShare.js'; const logger = createLogger('panel'); @@ -38,6 +40,8 @@ export interface ControlPanelDeps { getStatus?: (pausedLabel: string | null) => Promise; /** The MAX pause (server/maxSession.ts owns it): a deliberate disconnect with no auto-reconnect until resumed or the timer fires. */ pause: PauseControl; + /** Big files kept for download links — the «📁 Файлы» section. */ + files?: FileShare; } export interface PauseControl { @@ -135,6 +139,56 @@ function rootView(phone: string, pausedUntil: number | null): View { markup: Markup.inlineKeyboard([ [Markup.button.callback('👤 Найти контакт', 'tlmx_panel:contacts'), Markup.button.callback('🚫 Чаты', 'tlmx_panel:chats')], [Markup.button.callback('🔐 Вход в MAX', 'tlmx_panel:web'), Markup.button.callback('⚙️ Система', 'tlmx_panel:system')], + [Markup.button.callback('📁 Файлы', 'tlmx_panel:files')], + ]).reply_markup, + }; +} + +/** 72 h → «3 дня», 12 h → «12 ч». */ +function ttlText(ms: number): string { + const hours = Math.round(ms / 3600_000); + if (hours % 24 !== 0) return `${hours} ч`; + const d = hours / 24; + const word = d % 10 === 1 && d % 100 !== 11 ? 'день' : [2, 3, 4].includes(d % 10) && ![12, 13, 14].includes(d % 100) ? 'дня' : 'дней'; + return `${d} ${word}`; +} + +function filesView(files: FileShare): View { + const back = [Markup.button.callback('◀️ Назад', 'tlmx_panel:root')]; + if (!files.enabled) { + return { text: '📁 Пересылка больших файлов отключена: FILES=off в .env.', markup: Markup.inlineKeyboard([back]).reply_markup }; + } + const list = files.list(); + const pending = files.pendingUploads(); + const pendingLine = pending ? `\n\n⏳ Ждут загрузки по ссылке: ${pending}.` : ''; + const ttl = ttlText(files.opts.ttlMs); + if (list.length === 0) { + return { + text: `📁 Файлов нет.\n\nСюда попадают большие файлы, которые мост пересылает по ссылке: из MAX — больше 50 МБ, в MAX — больше 4 ГБ. Файл живёт ${ttl} от последней выданной ссылки, потом удаляется.${pendingLine}`, + markup: Markup.inlineKeyboard([back]).reply_markup, + }; + } + const total = list.reduce((s, f) => s + f.size, 0); + const lines = list.map((f, i) => `${i + 1}. ${f.direction === 'max2tg' ? '⬇️' : '⬆️'} ${f.name} — ${formatBytes(f.size)}, до ${formatExpiry(f.expiresAt)}`); + return { + text: `📁 Файлы на сервере: ${list.length}, всего ${formatBytes(total)}.\n⬇️ — из MAX, ⬆️ — для MAX. Ссылка живёт ${ttl}, потом файл удаляется.\n\n${lines.join('\n')}${pendingLine}`, + markup: Markup.inlineKeyboard([ + ...list.slice(0, 20).map((f, i) => [Markup.button.callback(`${i + 1}. ${truncateUtf16(f.name, 48)}`, `tlmx_panel:file:${f.id}`)]), + back, + ]).reply_markup, + }; +} + +function fileCardView(f: StoredFile, files: FileShare, link?: string): View { + const where = f.direction === 'max2tg' ? 'пришёл из MAX' : 'загружен для MAX'; + const linkLine = link ? `\n\n🔗 ${link}\nСсылку можно переслать — по ней откроется страница со скачиванием.` : ''; + return { + text: `📄 ${f.name}\n${formatBytes(f.size)} · ${where}\nХранится до ${formatExpiry(f.expiresAt)}, потом удалится.${linkLine}`, + markup: Markup.inlineKeyboard([ + ...(link ? [[Markup.button.url('⬇️ Открыть ссылку', link)]] : []), + [Markup.button.callback(`🔗 Новая ссылка (+${ttlText(files.opts.ttlMs)})`, `tlmx_panel:frenew:${f.id}`)], + [Markup.button.callback('🗑 Удалить', `tlmx_panel:fdel:${f.id}`)], + [Markup.button.callback('◀️ К файлам', 'tlmx_panel:files')], ]).reply_markup, }; } @@ -197,7 +251,7 @@ function pauseView(): View { } export function wireControlPanel(deps: ControlPanelDeps): void { - const { bot, targetGroupId, max, getActivePhone, triggerFullResync, leaves, startDialog, resolveContactName, getStatus, pause } = deps; + const { bot, targetGroupId, max, getActivePhone, triggerFullResync, leaves, startDialog, resolveContactName, getStatus, pause, files } = deps; const root = (): View => rootView(getActivePhone(), pause.until()); const system = (): View => systemView(pause.until() != null); @@ -274,6 +328,44 @@ export function wireControlPanel(deps: ControlPanelDeps): void { await ctx.answerCbQuery(); await edit(ctx, pauseView()); }); + // --- Big files («📁 Файлы»): list, a fresh link (+TTL), delete ------------------------------ + bot.action('tlmx_panel:files', async (ctx) => { + await ctx.answerCbQuery(); + if (files) await edit(ctx, filesView(files)); + }); + bot.action(/^tlmx_panel:file:(.+)$/, async (ctx) => { + await ctx.answerCbQuery(); + const f = files?.get(ctx.match[1] ?? ''); + if (!files || !f) { + if (files) await edit(ctx, filesView(files)); + return; + } + await edit(ctx, fileCardView(f, files)); + }); + bot.action(/^tlmx_panel:frenew:(.+)$/, async (ctx) => { + const id = ctx.match[1] ?? ''; + // Answered first: the service may take a minute or two to start, far past the callback's life. + await ctx.answerCbQuery('Готовлю ссылку…').catch(() => {}); + const f = files ? await files.renew(id) : undefined; + if (!files || !f) { + if (files) await edit(ctx, filesView(files)); + return; + } + await ctx.editMessageText(`⏳ Готовлю ссылку на «${f.name}»… Если служба файлов не запущена, это займёт до пары минут.`).catch(() => {}); + const svc = await files.ensureService(); + if (!svc.ok) { + const card = fileCardView(f, files); + await ctx.editMessageText(`${card.text}\n\n⚠️ Ссылку подготовить не удалось: ${svc.reason}.`, { reply_markup: card.markup }).catch(() => {}); + return; + } + await edit(ctx, fileCardView(f, files, `${svc.url}/f/${f.token}`)); + }); + bot.action(/^tlmx_panel:fdel:(.+)$/, async (ctx) => { + const removed = files ? await files.remove(ctx.match[1] ?? '') : false; + await ctx.answerCbQuery(removed ? 'Файл удалён' : 'Файла уже нет').catch(() => {}); + if (files) await edit(ctx, filesView(files)); + }); + bot.action('tlmx_panel:dismiss', async (ctx) => { await ctx.answerCbQuery(); await ctx.deleteMessage().catch(() => {}); diff --git a/src/bridge/sync.ts b/src/bridge/sync.ts index 46ccfa4..4e4ef7b 100644 --- a/src/bridge/sync.ts +++ b/src/bridge/sync.ts @@ -12,6 +12,7 @@ import { downloadMaxAttachment, describeAttachment, telegramSendKind, + TELEGRAM_UPLOAD_LIMIT_BYTES, takeKeyboard, KEYBOARD_NA, KEYBOARD_PRESS, @@ -23,6 +24,8 @@ import { } from './attachments.js'; import { splitTelegramText, truncateCodePoints, truncateUtf16, MAX_TEXT_LIMIT, TELEGRAM_CAPTION_LIMIT, TELEGRAM_TEXT_LIMIT } from './text.js'; import { uploadTelegramAttachmentToMax } from './upload.js'; +import { FileShare, fileShareOptionsFromEnv, TELEGRAM_BOT_DOWNLOAD_LIMIT } from './fileShare.js'; +import { createBigFiles } from './bigFiles.js'; import { canRenderAnimatedStickers } from './lottie.js'; import { reportBridgeError } from './errorReporter.js'; import { wireControlPanel, type PauseControl } from './panel.js'; @@ -595,6 +598,9 @@ export function isRenderableAttach(att: MaxAttachment): boolean { type SentMessage = { message_id: number }; type AttachmentSendOpts = { message_thread_id: number; reply_parameters?: { message_id: number; allow_sending_without_reply: boolean } }; +/** Set by wireBridge: sends a MAX file over Telegram's bot upload limit as a download link. */ +let bigFileRelay: ((att: MaxAttachment, ctx: DownloadContext, opts: AttachmentSendOpts) => Promise) | null = null; + /** * Sends a MAX message's attachments into a topic, one Telegram message each, and returns EVERY * sent message_id in order — the caller links them all (buildLinkIds). Each Telegram call has its @@ -674,6 +680,10 @@ async function sendOneAttachment( ? withFloodRetry(() => bot.telegram.sendContact(groupId, `+${String(phone)}`, displayName, opts)) : withFloodRetry(() => bot.telegram.sendMessage(groupId, `👤 Контакт: ${displayName}`, opts)); } + // A file the bot can't upload to Telegram goes through a download link (bigFiles.ts). + if (bigFileRelay && att._type === 'FILE' && Number(att.size ?? 0) > TELEGRAM_UPLOAD_LIMIT_BYTES) { + return bigFileRelay(att, downloadCtx, opts); + } const downloaded = await downloadMaxAttachment(att, downloadCtx); // Bot API upload limits (50 MB, a photo 10 MB) — see telegramSendKind. const kind = downloaded ? telegramSendKind(downloaded.kind, downloaded.buffer.byteLength) : null; @@ -1548,6 +1558,25 @@ export function wireBridge({ chatSync.runInChat(chatId, () => chatSync.advanceCursor(chatId, serverTime, 'outgoing send')).catch(() => undefined); } + // Big files through links (fileShare.ts / bigFiles.ts): Telegram's bot limits are 20 MB down and + // 50 MB up, MAX takes 4 GB. + const fileShare = new FileShare(path.join(process.cwd(), '.data'), fileShareOptionsFromEnv()); + void fileShare + .init() + .then(() => fileShare.startBackground()) + .catch((err) => logger.error('Big-file storage failed to start', err)); + const bigFiles = createBigFiles({ + bot, + max, + targetGroupId, + fileShare, + onSentToMax: (ticket, sent) => { + rememberOutgoingSend(ticket.maxChatId, sent.cid, sent.time); + messageLinks.add({ maxChatId: ticket.maxChatId, maxMessageId: sent.messageId, telegramMessageId: ticket.telegramMessageId, telegramTopicId: ticket.topicId, outgoing: true }); + }, + }); + bigFileRelay = bigFiles.relayFromMax; + // `${chatId}:${messageId}` -> what we last relayed, so the sticky lastReactedMessageId/lastReaction // fields on CHAT_UPDATE (repeated across unrelated pushes) don't re-trigger the same Telegram call. // Doubles as the poll list for reaction removal; keeps the original messageId (BigInt) for MSG_GET_REACTIONS. @@ -2781,6 +2810,7 @@ export function wireBridge({ triggerFullResync, startDialog, pause, + files: fileShare, // Warm-cache name lookup (buildRoster already fetched the profiles); undefined on a miss lets // the panel fall back to CONTACT_INFO. resolveContactName: (uid) => { @@ -3514,6 +3544,22 @@ export function wireBridge({ return; } + // Over the Bot API download limit getFile answers «file is too big» — offer an upload link + // instead (bigFiles.ts). Fire-and-forget: the link can take a minute or two to get ready, and + // this handler must not hold up the next update meanwhile. + const fileFields = ctx.message as unknown as Record; + const tooBigKey = ['document', 'video', 'audio', 'animation', 'video_note', 'voice'].find( + (k) => (fileFields[k]?.file_size ?? 0) > TELEGRAM_BOT_DOWNLOAD_LIMIT, + ); + if (tooBigKey) { + const f = fileFields[tooBigKey]!; + const name = f.file_name ?? (tooBigKey === 'video_note' ? 'video_note.mp4' : tooBigKey === 'voice' ? 'voice.ogg' : 'file'); + void bigFiles + .offerUpload({ maxChatId: mapping.maxChatId, topicId, telegramMessageId: ctx.message.message_id, caption: caption || undefined, name, size: f.file_size! }) + .catch((err) => logger.error('Offering an upload link failed', err)); + return; + } + let attach: Record | null = null; const largestPhoto = photo?.[photo.length - 1]; if (largestPhoto) { diff --git a/src/bridge/upload.ts b/src/bridge/upload.ts index 010b2b0..c951ebf 100644 --- a/src/bridge/upload.ts +++ b/src/bridge/upload.ts @@ -6,6 +6,7 @@ */ import { gzipSync } from 'node:zlib'; import https from 'node:https'; +import { createReadStream } from 'node:fs'; import type { Agent } from 'node:http'; import type { Telegraf } from 'telegraf'; // undici's own FormData, not the global one: maxFetch runs on undici's fetch, and @@ -13,7 +14,7 @@ import type { Telegraf } from 'telegraf'; // undici-types copy behind the global — same class at runtime either way. import { FormData } from 'undici'; import type { MaxClient } from '../max/client.js'; -import { maxFetch } from '../max/ca.js'; +import { maxFetch, maxFetchBig } from '../max/ca.js'; import { getTelegramProxyAgent } from '../telegram/proxy.js'; import { renderTgsToWebm } from './lottie.js'; @@ -116,6 +117,34 @@ async function uploadFileToMax(max: MaxClient, buffer: Buffer, filename: string) return { _type: 'FILE', fileId: slot.fileId }; } +/** + * uploadFileToMax for a file on disk, streamed — a multi-gigabyte upload (fileShare.ts) must not + * pass through memory. Same slot, headers and Content-Range shape as the buffered version. + */ +export async function uploadFileFromDiskToMax(max: MaxClient, filePath: string, size: number, filename: string): Promise<{ _type: 'FILE'; fileId: unknown }> { + const slot = await max.requestFileUpload(); + const res = await maxFetchBig(slot.url, { + method: 'POST', + headers: { + 'Content-Type': 'application/octet-stream', + 'Content-Range': `bytes 0-${size - 1}/${size}`, + 'Content-Length': String(size), + 'Content-Disposition': contentDispositionFor(filename), + }, + body: createReadStream(filePath), + duplex: 'half', + }); + if (!res.ok) { + const bodyText = await res.text().catch(() => ''); + throw new Error(`File upload POST failed: ${res.status} ${res.statusText} — ${bodyText}`); + } + await res.body?.cancel().catch(() => {}); + // The buffered path's 3 s, plus time for MAX to digest a big file before MSG_SEND may use it + // (a guess: +1 s per 100 MB, capped at a minute). + await new Promise((resolve) => setTimeout(resolve, 3000 + Math.min(60_000, Math.floor(size / (100 * 1024 * 1024)) * 1000))); + return { _type: 'FILE', fileId: slot.fileId }; +} + /** * Voice notes go through the SAME upload opcode as video (0x52, `type: 2`), not * FILE_UPLOAD — confirmed live by the user 2026-08-13 after two earlier guesses diff --git a/src/max/ca.ts b/src/max/ca.ts index 12c1ac0..e880e81 100644 --- a/src/max/ca.ts +++ b/src/max/ca.ts @@ -58,3 +58,12 @@ const maxDispatcher = new Agent({ connect: { ca: MAX_TLS_CA } }); export function maxFetch(url: string, init: RequestInit = {}): Promise { return undiciFetch(url, { ...init, dispatcher: maxDispatcher }); } + +// Gigabyte transfers (fileShare.ts): the defaults (5 min to the response headers) are fine for a +// photo but not for MAX digesting a 4 GB upload; bodyTimeout stays an IDLE limit between chunks. +const maxBigDispatcher = new Agent({ connect: { ca: MAX_TLS_CA }, headersTimeout: 30 * 60_000, bodyTimeout: 5 * 60_000 }); + +/** maxFetch for streamed multi-gigabyte up/downloads — same trust set, longer timeouts. */ +export function maxFetchBig(url: string, init: RequestInit = {}): Promise { + return undiciFetch(url, { ...init, dispatcher: maxBigDispatcher }); +} diff --git a/tests/fileShare.test.ts b/tests/fileShare.test.ts new file mode 100644 index 0000000..5cf6998 --- /dev/null +++ b/tests/fileShare.test.ts @@ -0,0 +1,171 @@ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import os from 'node:os'; +import path from 'node:path'; +import { Readable } from 'node:stream'; +import { existsSync } from 'node:fs'; +import { mkdir, mkdtemp, readFile, readdir, rm, utimes, writeFile } from 'node:fs/promises'; +import { FileShare, fileShareOptionsFromEnv, safeFileName } from '../src/bridge/fileShare.js'; + +let dataDir: string; +const opts = { enabled: true, ttlMs: 72 * 3600_000, reserveBytes: 0 }; + +beforeEach(async () => { + dataDir = await mkdtemp(path.join(os.tmpdir(), 'tlmx-files-')); +}); +afterEach(async () => { + await rm(dataDir, { recursive: true, force: true }); +}); + +async function fresh(o = opts): Promise { + const share = new FileShare(dataDir, o); + await share.init(); + return share; +} +const token = (share: FileShare, t: string) => path.join(share.dir, 'tokens', `${t}.json`); + +describe('safeFileName', () => { + it('keeps a normal name and neutralizes separators, control chars and leading dots', () => { + expect(safeFileName('Документы.zip')).toBe('Документы.zip'); + expect(safeFileName('../../etc/passwd')).toBe('_.._etc_passwd'); + expect(safeFileName('a\u0000b\nc:d?.txt')).toBe('a_b_c_d_.txt'); + expect(safeFileName('...')).toBe('file'); + expect(safeFileName(undefined)).toBe('file'); + }); + + it('cuts a long name but keeps its extension', () => { + const n = safeFileName(`${'я'.repeat(300)}.tar.gz`); + expect(n.length).toBe(180); + expect(n.endsWith('.gz')).toBe(true); + }); +}); + +describe('fileShareOptionsFromEnv', () => { + it('is on by default with 3 days and a 1.5 GB reserve', () => { + expect(fileShareOptionsFromEnv({})).toEqual({ enabled: true, ttlMs: 72 * 3600_000, reserveBytes: 1536 * 1024 * 1024 }); + expect(fileShareOptionsFromEnv({ FILES: 'off', FILES_TTL_HOURS: '24', FILES_RESERVE_MB: '0' })).toEqual({ enabled: false, ttlMs: 24 * 3600_000, reserveBytes: 0 }); + }); +}); + +describe('FileShare', () => { + it('a stored file gets a download token, the active marker, and survives a restart', async () => { + const share = await fresh(); + const f = await share.storeStream(Readable.from([Buffer.from('hello')]), 'a.txt', 5); + expect(await readFile(path.join(share.dir, 'items', f.id, 'a.txt'), 'utf8')).toBe('hello'); + const t = JSON.parse(await readFile(token(share, f.token), 'utf8')); + expect(t).toMatchObject({ v: 1, kind: 'download', name: 'a.txt', size: 5, path: `items/${f.id}/a.txt` }); + expect(existsSync(path.join(share.dir, 'active'))).toBe(true); + const again = await fresh(); + expect(again.list().map((x) => x.id)).toEqual([f.id]); + }); + + it('renew issues a new token and a new TTL; remove deletes the file and closes the marker', async () => { + const share = await fresh(); + const f = await share.storeStream(Readable.from([Buffer.from('x')]), 'x.bin', 1); + const oldToken = f.token; + const renewed = await share.renew(f.id); + expect(renewed?.token).not.toBe(oldToken); + expect(existsSync(token(share, oldToken))).toBe(false); + expect(existsSync(token(share, renewed!.token))).toBe(true); + expect(await share.remove(f.id)).toBe(true); + expect(existsSync(path.join(share.dir, 'items', f.id))).toBe(false); + expect(existsSync(path.join(share.dir, 'active'))).toBe(false); + }); + + it('a stream that runs past its declared size is refused and leaves nothing behind', async () => { + const share = await fresh(); + const big = Buffer.alloc(3 * 1024 * 1024, 1); + await expect(share.storeStream(Readable.from([big]), 'lie.bin', 10)).rejects.toThrow(/declared/); + expect(await readdir(path.join(share.dir, 'items'))).toEqual([]); + }); + + it('sweep deletes expired files with their tokens', async () => { + const share = await fresh({ ...opts, ttlMs: 1 }); + const f = await share.storeStream(Readable.from([Buffer.from('x')]), 'old.bin', 1); + await new Promise((r) => setTimeout(r, 5)); + await share.sweep(); + expect(share.list()).toEqual([]); + expect(existsSync(token(share, f.token))).toBe(false); + expect(existsSync(path.join(share.dir, 'items', f.id))).toBe(false); + }); + + it('an upload ticket hands the finished upload to the handler once', async () => { + const share = await fresh(); + const ticket = await share.createUploadTicket({ maxChatId: '5', topicId: 7, telegramMessageId: 9, name: 'big.zip', expectedSize: 3 }); + expect(JSON.parse(await readFile(token(share, ticket.token), 'utf8'))).toMatchObject({ kind: 'upload', name: 'big.zip', expectedSize: 3 }); + const seen: Array<[string, number, string]> = []; + share.onUpload(async (t, filePath, size, name) => { + seen.push([t.token, size, name]); + expect(await readFile(filePath, 'utf8')).toBe('abc'); + await share.dropTicket(t.token); + }); + await share.pollUploads(); // nothing uploaded yet + const dir = path.join(share.dir, 'incoming', ticket.token); + await mkdir(dir, { recursive: true }); + await writeFile(path.join(dir, 'other.zip'), 'abc'); + await writeFile(path.join(dir, 'done.json'), JSON.stringify({ name: 'other.zip', file: 'other.zip', size: 3 })); + await share.pollUploads(); + await share.pollUploads(); + expect(seen).toEqual([[ticket.token, 3, 'other.zip']]); + expect(share.pendingUploads()).toBe(0); + expect(existsSync(dir)).toBe(false); + }); + + it('adoptUpload keeps a too-big upload as a stored file with a link', async () => { + const share = await fresh(); + const ticket = await share.createUploadTicket({ maxChatId: '5', topicId: 7, telegramMessageId: 9, name: 'huge.iso', expectedSize: 2 }); + const dir = path.join(share.dir, 'incoming', ticket.token); + await mkdir(dir, { recursive: true }); + await writeFile(path.join(dir, 'huge.iso'), 'zz'); + const f = await share.adoptUpload(ticket, path.join(dir, 'huge.iso'), 2, 'huge.iso'); + expect(f.direction).toBe('tg2max'); + expect(await readFile(path.join(share.dir, 'items', f.id, 'huge.iso'), 'utf8')).toBe('zz'); + expect(share.pendingUploads()).toBe(0); + }); + + it('roomFor counts the reserve', async () => { + const share = await fresh({ ...opts, reserveBytes: Number.MAX_SAFE_INTEGER }); + expect((await share.roomFor(1)).fits).toBe(false); + expect((await (await fresh()).roomFor(1)).fits).toBe(true); + }); +}); + +describe('FileShare.ensureService', () => { + const heartbeat = async (ageMs = 0) => { + const p = path.join(dataDir, 'watcher-heartbeat'); + await writeFile(p, 'x'); + const t = (Date.now() - ageMs) / 1000; + await utimes(p, t, t); + }; + const service = async (share: FileShare, state: string, url = '', ageSec = 0) => + writeFile(path.join(share.dir, 'service.json'), JSON.stringify({ state, url, at: Math.floor(Date.now() / 1000) - ageSec, detail: '' })); + + it('returns the url once the host reports the service up', async () => { + const share = await fresh(); + await heartbeat(); + await service(share, 'up', 'https://1.2.3.4/'); + expect(await share.ensureService(1000)).toEqual({ ok: true, url: 'https://1.2.3.4' }); + }); + + it('explains busy ports right away', async () => { + const share = await fresh(); + await heartbeat(); + await service(share, 'ports-busy'); + const r = await share.ensureService(1000); + expect(r.ok).toBe(false); + expect(!r.ok && r.reason).toMatch(/80\/443/); + }); + + it('says so when no host dispatcher serves this install', async () => { + const share = await fresh(); + const r = await share.ensureService(1000); + expect(!r.ok && r.reason).toMatch(/автообновление/); + }); + + it('ignores a stale "up" and is off when disabled', async () => { + const share = await fresh(); + await heartbeat(); + await service(share, 'up', 'https://1.2.3.4', 600); + expect((await share.ensureService(10)).ok).toBe(false); + expect((await (await fresh({ ...opts, enabled: false })).ensureService(10)).ok).toBe(false); + }); +}); diff --git a/tests/filesService.test.ts b/tests/filesService.test.ts new file mode 100644 index 0000000..b5aed27 --- /dev/null +++ b/tests/filesService.test.ts @@ -0,0 +1,87 @@ +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import os from 'node:os'; +import path from 'node:path'; +import { spawn, type ChildProcess } from 'node:child_process'; +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; + +// files-service/server.mjs as the host runs it: its own process, bridges' file dirs under a root. +let root: string; +let proc: ChildProcess; +let base: string; +const bridge = () => path.join(root, '1'); + +async function tokenFile(name: string, body: Record) { + await writeFile(path.join(bridge(), 'tokens', `${name}.json`), JSON.stringify({ v: 1, ...body })); +} + +beforeAll(async () => { + root = await mkdtemp(path.join(os.tmpdir(), 'tlmx-srv-')); + await mkdir(path.join(bridge(), 'tokens'), { recursive: true }); + await mkdir(path.join(bridge(), 'items', 'id1'), { recursive: true }); + await writeFile(path.join(bridge(), 'items', 'id1', 'отчёт.txt'), '0123456789'); + const later = Date.now() + 3600_000; + await tokenFile('dddddddddddddddddddddd', { kind: 'download', name: 'отчёт.txt', size: 10, path: 'items/id1/отчёт.txt', expiresAt: later }); + await tokenFile('eeeeeeeeeeeeeeeeeeeeee', { kind: 'download', name: 'x', size: 1, path: 'items/id1/отчёт.txt', expiresAt: Date.now() - 1 }); + await tokenFile('tttttttttttttttttttttt', { kind: 'download', name: 'x', size: 1, path: '../../../etc/hosts', expiresAt: later }); + await tokenFile('uuuuuuuuuuuuuuuuuuuuuu', { kind: 'upload', name: 'big.zip', expectedSize: 5, expiresAt: later, reserveBytes: 0 }); + await tokenFile('rrrrrrrrrrrrrrrrrrrrrr', { kind: 'upload', name: 'big.zip', expectedSize: 5, expiresAt: later, reserveBytes: Number.MAX_SAFE_INTEGER }); + const port = 18000 + Math.floor(Math.random() * 2000); + base = `http://127.0.0.1:${port}`; + proc = spawn(process.execPath, [path.join('files-service', 'server.mjs')], { env: { ...process.env, FILES_ROOT: root, PORT: String(port) }, stdio: ['ignore', 'pipe', 'pipe'] }); + await new Promise((resolve, reject) => { + const t = setTimeout(() => reject(new Error('server did not start')), 10_000); + proc.stdout!.on('data', (d) => String(d).includes('files service') && (clearTimeout(t), resolve())); + }); +}); + +afterAll(async () => { + proc?.kill(); + await rm(root, { recursive: true, force: true }); +}); + +describe('files service', () => { + it('answers health and hides everything without a token', async () => { + expect(await (await fetch(`${base}/health`)).text()).toBe('ok'); + expect((await fetch(`${base}/`)).status).toBe(404); + expect((await fetch(`${base}/f/short`)).status).toBe(404); + expect((await fetch(`${base}/f/eeeeeeeeeeeeeeeeeeeeee`)).status).toBe(404); // expired + }); + + it('shows a page first, then the file with its name and Range support', async () => { + const pageRes = await fetch(`${base}/f/dddddddddddddddddddddd`); + expect(pageRes.status).toBe(200); + expect(pageRes.headers.get('x-robots-tag')).toMatch(/noindex/); + expect(await pageRes.text()).toContain('/f/dddddddddddddddddddddd/file'); + const file = await fetch(`${base}/f/dddddddddddddddddddddd/file`); + expect(await file.text()).toBe('0123456789'); + expect(file.headers.get('content-disposition')).toContain(`filename*=UTF-8''${encodeURIComponent('отчёт.txt')}`); + const part = await fetch(`${base}/f/dddddddddddddddddddddd/file`, { headers: { Range: 'bytes=4-6' } }); + expect(part.status).toBe(206); + expect(await part.text()).toBe('456'); + expect(part.headers.get('content-range')).toBe('bytes 4-6/10'); + }); + + it('never serves a path outside the bridge directory', async () => { + expect((await fetch(`${base}/f/tttttttttttttttttttttt/file`)).status).toBe(404); + }); + + it('takes one upload per link and refuses a second', async () => { + expect(await (await fetch(`${base}/f/uuuuuuuuuuuuuuuuuuuuuu`)).text()).toContain('type="file"'); + const put = (body: string) => + fetch(`${base}/f/uuuuuuuuuuuuuuuuuuuuuu/upload`, { method: 'PUT', body, headers: { 'X-File-Name': encodeURIComponent('../архив.zip') } }); + const first = await put('hello'); + expect(first.status).toBe(200); + const dir = path.join(bridge(), 'incoming', 'uuuuuuuuuuuuuuuuuuuuuu'); + const done = JSON.parse(await readFile(path.join(dir, 'done.json'), 'utf8')); + expect(done).toMatchObject({ name: '_архив.zip', size: 5 }); + expect(await readFile(path.join(dir, done.file), 'utf8')).toBe('hello'); + expect((await put('again')).status).toBe(410); + expect(existsSync(path.join(dir, '.lock'))).toBe(false); + }); + + it('refuses an upload that would eat the reserve', async () => { + const r = await fetch(`${base}/f/rrrrrrrrrrrrrrrrrrrrrr/upload`, { method: 'PUT', body: 'hello' }); + expect(r.status).toBe(507); + }); +}); diff --git a/tests/keyboard.test.ts b/tests/keyboard.test.ts index 4ecf6e0..a1628ac 100644 --- a/tests/keyboard.test.ts +++ b/tests/keyboard.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from 'vitest'; import type { Telegraf } from 'telegraf'; -import { KEYBOARD_CONTACT, KEYBOARD_GEO, KEYBOARD_NA, KEYBOARD_PRESS, maxKeyboardToTelegram, pressableButton, takeKeyboard } from '../src/bridge/attachments.js'; +import { KEYBOARD_CONTACT, KEYBOARD_GEO, KEYBOARD_NA, KEYBOARD_PRESS, maxKeyboardToTelegram, miniAppLink, pressableButton, takeKeyboard } from '../src/bridge/attachments.js'; import { isMarkupRefusal, isRenderableAttach, sendAttachments, sendTextPieces } from '../src/bridge/sync.js'; const kb = (buttons: unknown) => ({ _type: 'INLINE_KEYBOARD', keyboard: { buttons }, callbackId: 'x' }); @@ -29,7 +29,7 @@ describe('maxKeyboardToTelegram', () => { [{ type: 'LINK', text: 'Сайт', url: 'tg://resolve?domain=x' }], ]); expect(maxKeyboardToTelegram(attach)).toEqual({ - inline_keyboard: [[na('Да'), na('Нет')], [na('Открыть')], [{ text: 'Контакт', callback_data: KEYBOARD_CONTACT }, { text: 'Гео', callback_data: KEYBOARD_GEO }, na('?'), na('без типа')], [{ text: 'Сайт', url: 'tg://resolve?domain=x' }]], + inline_keyboard: [[na('Да'), na('Нет')], [{ text: 'Открыть', url: 'https://max.ru/app?startapp' }], [{ text: 'Контакт', callback_data: KEYBOARD_CONTACT }, { text: 'Гео', callback_data: KEYBOARD_GEO }, na('?'), na('без типа')], [{ text: 'Сайт', url: 'tg://resolve?domain=x' }]], }); }); @@ -154,3 +154,18 @@ describe('CALLBACK buttons — presses relayed to MAX', () => { expect(pressableButton('no-such-key')).toBeUndefined(); }); }); + +describe('miniAppLink', () => { + it('builds the public deep link of the live «Настройки» sample (2026-10-06)', () => { + const b = { type: 'OPEN_APP', text: 'Настройки', webApp: 'maxnotifications_bot', contactId: 18948480, payload: 'v2bdo2qespQHrvVOL0TsKEKUXUk2RsTqvbcEFsHcdRp63ILky2my5' }; + expect(miniAppLink(b)).toBe('https://max.ru/maxnotifications_bot?startapp=v2bdo2qespQHrvVOL0TsKEKUXUk2RsTqvbcEFsHcdRp63ILky2my5'); + }); + + it('drops a payload MAX would strip anyway, and refuses a bad app name', () => { + expect(miniAppLink({ webApp: 'shop_bot', payload: 'a b' })).toBe('https://max.ru/shop_bot?startapp'); + expect(miniAppLink({ webApp: 'shop_bot', payload: 'x'.repeat(513) })).toBe('https://max.ru/shop_bot?startapp'); + expect(miniAppLink({ webApp: 'shop_bot' })).toBe('https://max.ru/shop_bot?startapp'); + expect(miniAppLink({ webApp: '../evil' })).toBeUndefined(); + expect(miniAppLink({ payload: 'x' })).toBeUndefined(); + }); +}); diff --git a/update-watcher.sh b/update-watcher.sh index 9971616..26ca9aa 100755 --- a/update-watcher.sh +++ b/update-watcher.sh @@ -16,6 +16,12 @@ cd "$(dirname "$0")" mkdir -p data date -u +%Y-%m-%dT%H:%M:%SZ > data/watcher-heartbeat 2>/dev/null || true +# The shared big-file service (files-service/): started only while some bridge has a live link, +# stopped when none is left. Never allowed to break or delay-fail the update check below. +if [ -f files-service/reconcile.sh ]; then + timeout 120 bash files-service/reconcile.sh >/dev/null 2>&1 || true +fi + MARKER="data/update-requested" [ -f "$MARKER" ] || exit 0