feat(monitor): wire isolated target recovery
Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
2d4e4878ae
commit
5f3a6953ee
16 files changed
+1922
-76
No files matched your search
+53
-1
@@ -107,7 +107,7 @@ and exact GGC filtering are unchanged.
|
|||||||
The isolated [Compose definition](deploy/managed-target/README.md) now supplies
|
The isolated [Compose definition](deploy/managed-target/README.md) now supplies
|
||||||
the marker, mount and ordered startup; it has not been built or deployed.
|
the marker, mount and ordered startup; it has not been built or deployed.
|
||||||
No Docker socket, remote/container operations, old target paths, Xray edits,
|
No Docker socket, remote/container operations, old target paths, Xray edits,
|
||||||
monitor loop or host firewall provisioning are included. Before deployment,
|
or host firewall provisioning are included. Before deployment,
|
||||||
provisioning must verify the pinned image honors `ZAPRET_CONFIG`, supports all
|
provisioning must verify the pinned image honors `ZAPRET_CONFIG`, supports all
|
||||||
listed variables, and starts ss-local while the baseline leaves nfqws2 disabled.
|
listed variables, and starts ss-local while the baseline leaves nfqws2 disabled.
|
||||||
Start the fresh target with that baseline, then verify active apply installs
|
Start the fresh target with that baseline, then verify active apply installs
|
||||||
@@ -117,6 +117,58 @@ uncertain, deployment must fail closed, not substitute `restart-daemons` or
|
|||||||
alter an old config. Persistent volume durability is a provisioning
|
alter an old config. Persistent volume durability is a provisioning
|
||||||
requirement; mountinfo alone cannot establish the storage lifecycle.
|
requirement; mountinfo alone cannot establish the storage lifecycle.
|
||||||
|
|
||||||
|
Internal `recoverManagedTarget` and `wireManagedRecovery` add injectable startup
|
||||||
|
bootstrap and saved-first recovery callbacks for the persisted monitor. Bootstrap
|
||||||
|
scans/checks/saves frontend and applies it alone with Player API post-check.
|
||||||
|
Recovery keeps an already healthy frontend, otherwise stops only the fixed
|
||||||
|
target zapret service, tries up to ten qualified saved IDs newest-first, and
|
||||||
|
scans only after complete valid non-pass results or health-rejected rollbacks.
|
||||||
|
Malformed reports/runner errors fail closed. GGC discovery uses the injected
|
||||||
|
managed-SOCKS path, revalidates against the fresh exact host, and saves new
|
||||||
|
immutable evidence before full-set apply. A frontend-only rollback boundary
|
||||||
|
survives GGC failure; unresolved GGC is reported separately, not playback health.
|
||||||
|
Cancelled/failed downtime restores the committed configuration and runs fixed
|
||||||
|
target `start` with an independent 30-second cleanup deadline. Failed cleanup or
|
||||||
|
rollback is fatal to the supervisor; unresolved journal state is never started.
|
||||||
|
The callback caller must hold the monitor singleton; the wiring repeats runtime
|
||||||
|
identity/SOCKS guards. Fake-only checks are added, but tests/build are deferred
|
||||||
|
to the orchestrator. `managed monitor` now binds those callbacks and only the
|
||||||
|
dedicated target entrypoint starts it; no live/runtime validation has occurred.
|
||||||
|
|
||||||
|
## Dedicated monitor integration (verification deferred)
|
||||||
|
|
||||||
|
`zapret-autotune managed monitor [--interval 5m] [--failure-threshold 3]
|
||||||
|
[--cooldown 30m]` uses the health environment defaults. Interval is positive and
|
||||||
|
at most 24h, threshold 1–1000, cooldown positive and at most 168h; explicit empty
|
||||||
|
or invalid settings fail before any lock. No executable/path/proxy flags exist.
|
||||||
|
The DB environment must be exactly `/var/lib/zapret-autotune/target/candidates.db`;
|
||||||
|
existing DB/sidecars must be private, owned regular files without aliases/hard links.
|
||||||
|
Runtime/SOCKS, directory, identity and DB path guards run before the singleton.
|
||||||
|
SQLite opens writable target-owned state; active IDs remain journal/DB/hash validated.
|
||||||
|
|
||||||
|
Probe reuses `health youtube`'s Player API contract, forced owned loopback SOCKS,
|
||||||
|
no direct fallback. Discovery uses that same SOCKS adapter. Scan/check execute
|
||||||
|
only `/usr/bin/blockcheckw`, daemon operations only fixed target-local
|
||||||
|
`/opt/zapret2/init.d/sysv/zapret2` start/stop/restart argv; no shell/eval command
|
||||||
|
strings. Context cancellation kills/waits process groups and bounded restoration
|
||||||
|
finishes before returning. Reports suppress untrusted diagnostics and credentials.
|
||||||
|
|
||||||
|
Startup reconciles transactions under the singleton and bootstraps only when no
|
||||||
|
qualified active frontend exists. Bootstrap/recovery checks saved candidates first;
|
||||||
|
GGCError reports unresolved and continues probing, not playback success. Other
|
||||||
|
DB/safety/journal/runner failures stop nonzero; joined cancellation plus a cleanup
|
||||||
|
failure is also fatal. A qualified frontend at startup is probed immediately,
|
||||||
|
not rescanned merely because the process restarted.
|
||||||
|
|
||||||
|
Entrypoint ordering is SS children → listener → managed init/status → zapret start
|
||||||
|
→ one installed `/usr/local/bin/zapret-autotune managed monitor` child. It supervises
|
||||||
|
all three children; monitor exit is fatal even if it exits zero unexpectedly.
|
||||||
|
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
|
||||||
|
SS child termination/wait. Existing targets and Xray remain unchanged. Deployment
|
||||||
|
must allow enough shutdown grace for independently bounded rollback/restoration
|
||||||
|
(potentially more than the currently defined 45s); this increment changes no Docker
|
||||||
|
configuration. Code integration is not image/routing/signal live verification.
|
||||||
|
|
||||||
Injected unit tests cover exact identity/path/config, mountpoint refusal,
|
Injected unit tests cover exact identity/path/config, mountpoint refusal,
|
||||||
old-container-like env, SOCKS port/process refusal, correct fresh-target success,
|
old-container-like env, SOCKS port/process refusal, correct fresh-target success,
|
||||||
non-overwrite, transaction rollback and redaction. Tests/build/network are
|
non-overwrite, transaction rollback and redaction. Tests/build/network are
|
||||||
|
|||||||
+11
-3
@@ -4,6 +4,14 @@
|
|||||||
|
|
||||||
## Готово
|
## Готово
|
||||||
|
|
||||||
|
- Добавлены `managed monitor` и supervision только dedicated target entrypoint: guarded target-owned DB/identity до singleton, policy 5m/3/30m с bounded CLI/env validation, owned-loopback Player API probe/discovery без direct fallback, fixed blockcheck/zapret argv с context cancellation/join. Startup bootstrap только без qualified active frontend, saved-first recovery; GGC unresolved продолжает мониторинг, DB/safety errors fatal. SIGTERM ждёт monitor restoration до zapret/SS stop. Добавлены local fake CLI/entrypoint tests. Tests/build/network не запускались; verification deferred orchestrator. Docker/remote/existing target/Xray не менялись. Нижние пункты описывают предшествующие increments, не отменяют эту интеграцию.
|
||||||
|
|
||||||
|
- Hardened managed recovery artifacts: общий `revalidateCandidate` удаляет старый combined JSON перед каждым check, remove errors/missing/malformed report fail closed; scan shortlist проверяется на regular file и mtime не раньше whole-second scan start (production использует unique temp path). Добавлены fake-only regression tests stale/missing shortlist, stale combined, remove failure и infrastructure classification/restoration. Tests/build/network не запускались; verification deferred orchestrator, CLI/entrypoint не добавлены.
|
||||||
|
|
||||||
|
- Добавлен injected `recoverManagedTarget` и callback wiring к persisted monitor: frontend bootstrap через scan/check/save/transaction, saved-first qualified IDs (до 10), strict valid non-pass fallback, target-local downtime и bounded restoration, frontend-only rollback boundary перед fresh exact-host GGC revalidation/scan. GGC failure отдельно unresolved, Player API не playback. Immutable candidates/committed IDs сохраняются; cleanup/rollback failure fatal. Fake-only tests добавлены; producer не запускал tests/build/network, verification deferred orchestrator. CLI/entrypoint/live runtime не подключены; старый target и Xray не менялись.
|
||||||
|
|
||||||
|
- Добавлены только reusable recovery building blocks: `listQualifiedCandidates` (known groups, newest-first, limit 1–10, shared qualification rule) и injected exact-endpoint revalidation сохранённых args. Только полный валидный two-TLS 3-pass nonpass даёт `errCandidateNonpass`; malformed/incomplete evidence, scope/args mismatch и infrastructure failures fail closed. Definitions/evidence не меняются, fresh GGC остаётся exact-host. Existing proposal recovery использует общий revalidator; managed recovery adapter, monitor CLI/entrypoint и remote calls не добавлены. Focused local-fixture tests добавлены; producer tests/build/network не запускал, verification отложена orchestrator.
|
||||||
|
|
||||||
- Group-wise managed activation: один или оба qualified frontend/exact-GGC candidate; full desired set (не patch), frontend-only bootstrap до Player API/GGC discovery. Journal v2 сохраняет committed/pending IDs вместе с hash; rollback/crash возвращает прежний список. `managed status` и JSON `managed active-ids` проверяют IDs/groups через DB и exact rendered hash; missing groups явно unresolved. Legacy v1 сохраняет deployment по exact hash без выдуманных IDs; unknown IDs fail closed для supervisor. Player API OK не означает GGC/video playback или full-profile health. Focused tests обновлены/добавлены; tests/build/network отложены orchestrator. Recovery callback, remote apply, текущий target и Xray не менялись.
|
- Group-wise managed activation: один или оба qualified frontend/exact-GGC candidate; full desired set (не patch), frontend-only bootstrap до Player API/GGC discovery. Journal v2 сохраняет committed/pending IDs вместе с hash; rollback/crash возвращает прежний список. `managed status` и JSON `managed active-ids` проверяют IDs/groups через DB и exact rendered hash; missing groups явно unresolved. Legacy v1 сохраняет deployment по exact hash без выдуманных IDs; unknown IDs fail closed для supervisor. Player API OK не означает GGC/video playback или full-profile health. Focused tests обновлены/добавлены; tests/build/network отложены orchestrator. Recovery callback, remote apply, текущий target и Xray не менялись.
|
||||||
|
|
||||||
- Добавлен internal `runManagedMonitor(ctx, deps)`: immediate/periodic probe, durable streak/reset, threshold/cooldown recovery через injected callbacks, cooldown от completion при любом результате, fatal DB/lock/finalization errors и cancellable wait. Shared runtime guards выполняются до lifetime nonblocking `flock` в target-owned directory. Одноразовая startup reconciliation под singleton использует schema-v5 PID/starttime API: exact lease снимается только при доказанном dead/PID-reused owner и запускает crash cooldown; live owner остаётся busy, legacy/unverifiable owner — fail closed. Fake-time unit-тесты scheduler core, reset/cooldown, reopen, concurrent start, cancellation и DB/finalization failures добавлены; tests/build/network не запускались в этом increment — verification отложена orchestrator. Actual recovery adapter/monitor command/entrypoint wiring и развёрнутый live target пока отсутствуют; deploy, контейнер и Xray не затронуты.
|
- Добавлен internal `runManagedMonitor(ctx, deps)`: immediate/periodic probe, durable streak/reset, threshold/cooldown recovery через injected callbacks, cooldown от completion при любом результате, fatal DB/lock/finalization errors и cancellable wait. Shared runtime guards выполняются до lifetime nonblocking `flock` в target-owned directory. Одноразовая startup reconciliation под singleton использует schema-v5 PID/starttime API: exact lease снимается только при доказанном dead/PID-reused owner и запускает crash cooldown; live owner остаётся busy, legacy/unverifiable owner — fail closed. Fake-time unit-тесты scheduler core, reset/cooldown, reopen, concurrent start, cancellation и DB/finalization failures добавлены; tests/build/network не запускались в этом increment — verification отложена orchestrator. Actual recovery adapter/monitor command/entrypoint wiring и развёрнутый live target пока отсутствуют; deploy, контейнер и Xray не затронуты.
|
||||||
@@ -19,7 +27,7 @@
|
|||||||
- Добавлена proposal-only команда `recover googlevideo --video-id ID --proxy socks5://HOST:PORT`: discovery одного свежего exact host → qualified schema-v3 сценарии newest-first → combined TLS1.2/1.3 3/3 точных сохранённых args. Первый успешный сценарий сохраняется новым immutable candidate/run и выдаётся как NOT ACTIVE без scan. Только отсутствие eligible candidates или валидные non-pass результаты разрешают обычный scan на уже найденном host; infrastructure/malformed-report ошибки fail closed. Apply/monitor и удалённый scan не затронуты.
|
- Добавлена proposal-only команда `recover googlevideo --video-id ID --proxy socks5://HOST:PORT`: discovery одного свежего exact host → qualified schema-v3 сценарии newest-first → combined TLS1.2/1.3 3/3 точных сохранённых args. Первый успешный сценарий сохраняется новым immutable candidate/run и выдаётся как NOT ACTIVE без scan. Только отсутствие eligible candidates или валидные non-pass результаты разрешают обычный scan на уже найденном host; infrastructure/malformed-report ошибки fail closed. Apply/monitor и удалённый scan не затронуты.
|
||||||
- Unit-тесты recovery порядка, fresh-host scope, short-circuit, fallback и fail-closed ошибок прошли; локальные `go test ./...`, `go build` и `git diff --check` успешны.
|
- Unit-тесты recovery порядка, fresh-host scope, short-circuit, fallback и fail-closed ошибок прошли; локальные `go test ./...`, `go build` и `git diff --check` успешны.
|
||||||
|
|
||||||
- CLI на Go: `tune youtube`, `tune googlevideo --video-id ID --proxy socks5://HOST:PORT`, `recover googlevideo --video-id ID --proxy socks5://HOST:PORT`, `discover youtube`, `health youtube`, `status`, `history`, `strategies youtube`, `render`, `managed init|status|apply`. `check youtube` пока намеренно отказывает; manual managed apply только внутри designated target, rollback автоматический; monitor scheduling core есть, но command/recovery wiring отсутствует.
|
- CLI на Go: `tune youtube`, `tune googlevideo --video-id ID --proxy socks5://HOST:PORT`, `recover googlevideo --video-id ID --proxy socks5://HOST:PORT`, `discover youtube`, `health youtube`, `status`, `history`, `strategies youtube`, `render`, `managed init|status|apply|monitor`. `check youtube` пока намеренно отказывает; managed activation/monitor только внутри designated target, rollback автоматический; command/entrypoint интегрированы, runtime verification отложена.
|
||||||
- Proposal-only GGC pipeline: Android 21.02.35 discovery через явный proxy → scan TLS1.2/1.3 только одного найденного hostname → data-transfer shortlist (`--take 10 --passes 3`) → проверка точной комбинации 3/3 на обоих TLS → один инертный профиль с exact-host фильтром. Ранжирование по blockcheckw, не Selectel. SQLite v3 атомарно сохраняет googlevideo candidate и оба endpoint×TLS evidence; отдельный additive journal сохраняет failed runs. `history` и `strategies youtube` показывают точный endpoint/group; `status` пока относится только к frontend journal.
|
- Proposal-only GGC pipeline: Android 21.02.35 discovery через явный proxy → scan TLS1.2/1.3 только одного найденного hostname → data-transfer shortlist (`--take 10 --passes 3`) → проверка точной комбинации 3/3 на обоих TLS → один инертный профиль с exact-host фильтром. Ранжирование по blockcheckw, не Selectel. SQLite v3 атомарно сохраняет googlevideo candidate и оба endpoint×TLS evidence; отдельный additive journal сохраняет failed runs. `history` и `strategies youtube` показывают точный endpoint/group; `status` пока относится только к frontend journal.
|
||||||
- Добавлены injectable unit-тесты GGC success/order/scope/filter/persistence, обязательных флагов, discovery failure, неполной/неуспешной combined проверки и fail-closed DB. После исправления exact-host фильтра `go test ./...`, `go build` и `git diff --check` прошли локально. Полный новый `tune googlevideo` ещё не прогонялся удалённо.
|
- Добавлены injectable unit-тесты GGC success/order/scope/filter/persistence, обязательных флагов, discovery failure, неполной/неуспешной combined проверки и fail-closed DB. После исправления exact-host фильтра `go test ./...`, `go build` и `git diff --check` прошли локально. Полный новый `tune googlevideo` ещё не прогонялся удалённо.
|
||||||
- `tune youtube` сканирует frontend `youtube.com`, проверяет shortlist через `blockcheckw` и выдаёт один TLS-профиль только после совместной проверки TLS 1.2 и 1.3 по 3/3. Это предложение конфигурации, оно ничего не применяет.
|
- `tune youtube` сканирует frontend `youtube.com`, проверяет shortlist через `blockcheckw` и выдаёт один TLS-профиль только после совместной проверки TLS 1.2 и 1.3 по 3/3. Это предложение конфигурации, оно ничего не применяет.
|
||||||
@@ -44,7 +52,7 @@
|
|||||||
## Не готово
|
## Не готово
|
||||||
|
|
||||||
- Проверка реального YouTube video-transfer/SABR. Прямой GET `serverAbrStreamingUrl` возвращает `sabr.malformed_config`; hostname discovery и `blockcheckw` data-transfer checks не равны проверке воспроизведения видео.
|
- Проверка реального YouTube video-transfer/SABR. Прямой GET `serverAbrStreamingUrl` возвращает `sabr.malformed_config`; hostname discovery и `blockcheckw` data-transfer checks не равны проверке воспроизведения видео.
|
||||||
- Применение подключено только к guarded manual CLI внутри designated target: transaction core, adapter и deployment definition есть, но нет proven image/runtime compatibility, выполненного container provisioning, actual recovery callback/monitor wiring и реально запущенного managed target. Injected supervisor scheduling core добавлен, но одноразовый Player API health и policy сами не запускают live automation. Цель: новый отдельный prod-like zapret2-ss контейнер только в тестовой среде; существующий контейнер не трогать. Стабильный настраиваемый loopback SOCKS для Xray; Xray config вне scope. Перед runtime пройти route/rollback gates.
|
- Guarded managed CLI/monitor интегрированы внутри designated target: transaction core, bootstrap/recovery callback и deployment definition есть, но нет proven image/runtime compatibility, выполненного container provisioning и реально запущенного managed target. Одноразовый Player API health сам не запускает live automation. Цель: новый отдельный prod-like zapret2-ss контейнер только в тестовой среде; существующий контейнер не трогать. Стабильный настраиваемый loopback SOCKS для Xray; Xray config вне scope. Перед runtime пройти route/rollback gates.
|
||||||
- Полный сервис из требований: YAML-конфигурация сервисов, endpoint-история/покрытие, daemon/периодический мониторинг, API/метрики и multi-service режим.
|
- Полный сервис из требований: YAML-конфигурация сервисов, endpoint-история/покрытие, daemon/периодический мониторинг, API/метрики и multi-service режим.
|
||||||
|
|
||||||
## Открытые ограничения и следующие шаги
|
## Открытые ограничения и следующие шаги
|
||||||
@@ -52,6 +60,6 @@
|
|||||||
1. Проверить новый `tune googlevideo` и `recover googlevideo` end-to-end на удалённом сервере; отдельно решить, как импортировать или переиспользовать уже полученные scan результаты без повторного перебора.
|
1. Проверить новый `tune googlevideo` и `recover googlevideo` end-to-end на удалённом сервере; отдельно решить, как импортировать или переиспользовать уже полученные scan результаты без повторного перебора.
|
||||||
2. Воспроизводимое описание отдельного prod-like target добавлено в `deploy/managed-target`; orchestrator должен проверить Compose syntax. Build/run отложены. До запуска проверить baseline-конфиг и NFQUEUE внутри container network namespace; текущие mounts, сеть, порты и конфиг не переиспользовать.
|
2. Воспроизводимое описание отдельного prod-like target добавлено в `deploy/managed-target`; orchestrator должен проверить Compose syntax. Build/run отложены. До запуска проверить baseline-конфиг и NFQUEUE внутри container network namespace; текущие mounts, сеть, порты и конфиг не переиспользовать.
|
||||||
3. Проверить готовый manual managed CLI на новом изолированном target; подтвердить shell config rendering, config binding, full firewall/daemon restart и pre/post health.
|
3. Проверить готовый manual managed CLI на новом изолированном target; подтвердить shell config rendering, config binding, full firewall/daemon restart и pre/post health.
|
||||||
4. Подключить готовый injected monitor supervisor к functional managed recovery (сохранённые сценарии перед новым scan); только после этого добавлять command/entrypoint wiring. Startup singleton reconciliation уже есть; live runtime ещё не проверен.
|
4. Проверить bootstrap/recovery callback и новую CLI/entrypoint интеграцию focused tests/build. Startup singleton reconciliation уже есть; live runtime ещё не проверен. До deployment проверить shutdown grace: bounded rollback/restoration может потребовать больше текущих 45s; Docker definition в этом increment не менялась.
|
||||||
5. До autonomous mode доказать safety gates из `Architecture.md`: изоляцию target, эквивалентность managed SOCKS пути, версионирование, atomic activation/rollback и crash recovery.
|
5. До autonomous mode доказать safety gates из `Architecture.md`: изоляцию target, эквивалентность managed SOCKS пути, версионирование, atomic activation/rollback и crash recovery.
|
||||||
6. Отдельно определить реальную video/SABR-проверку и traffic budget; API health не равен playback и не обнаруживает все GGC playback failures.
|
6. Отдельно определить реальную video/SABR-проверку и traffic budget; API health не равен playback и не обнаруживает все GGC playback failures.
|
||||||
@@ -21,7 +21,11 @@ binding. It uses full target-local firewall/daemon restart and Player API post-c
|
|||||||
atomic rollback on failure. See [MANAGED_TARGET.md](MANAGED_TARGET.md) for guards
|
atomic rollback on failure. See [MANAGED_TARGET.md](MANAGED_TARGET.md) for guards
|
||||||
and provisioning prerequisites. An isolated pinned deployment definition is in
|
and provisioning prerequisites. An isolated pinned deployment definition is in
|
||||||
[deploy/managed-target](deploy/managed-target/README.md); build/run are deferred.
|
[deploy/managed-target](deploy/managed-target/README.md); build/run are deferred.
|
||||||
No autonomous loop, remote activation or Xray changes are included. Image
|
An injected bootstrap/saved-first recovery adapter now connects the persisted
|
||||||
|
monitor foundation to blockcheck, immutable candidates and transactional apply.
|
||||||
|
`managed monitor` and the dedicated target entrypoint now wire these adapters;
|
||||||
|
this increment is not tested by the producer or live/runtime verified.
|
||||||
|
No remote activation or Xray changes are included. Image
|
||||||
compatibility and route/rollback gates are not yet remotely proven; adapter
|
compatibility and route/rollback gates are not yet remotely proven; adapter
|
||||||
tests/build are deferred to the orchestrator.
|
tests/build are deferred to the orchestrator.
|
||||||
|
|
||||||
@@ -35,6 +39,14 @@ and roll back together, including after a crash.
|
|||||||
|
|
||||||
## One-shot Player API health
|
## One-shot Player API health
|
||||||
|
|
||||||
|
Internal recovery building blocks enumerate qualified immutable candidates
|
||||||
|
newest-first with a 1–10 limit and recheck saved arguments on an exact endpoint
|
||||||
|
through an injected combined TLS checker. Only complete, valid two-TLS 3-pass
|
||||||
|
nonpassing evidence permits the nonpass result; malformed reports and runner,
|
||||||
|
filesystem or context failures remain infrastructure errors. These helpers do
|
||||||
|
not update stored evidence, apply configuration or wire managed recovery/monitor
|
||||||
|
commands. Their local fixture tests/build are deferred to the orchestrator.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
zapret-autotune health youtube
|
zapret-autotune health youtube
|
||||||
```
|
```
|
||||||
@@ -60,9 +72,9 @@ Configuration is parsed once per health invocation. The proxy must use `socks5`
|
|||||||
with a loopback IP or `localhost`, port 1–65535, and no credentials/path/query.
|
with a loopback IP or `localhost`, port 1–65535, and no credentials/path/query.
|
||||||
Video IDs must be 11 characters; durations and the failure threshold must be
|
Video IDs must be 11 characters; durations and the failure threshold must be
|
||||||
positive. Explicit empty values are invalid. Interval, consecutive-failure
|
positive. Explicit empty values are invalid. Interval, consecutive-failure
|
||||||
threshold and cooldown are validated future recovery policy settings only:
|
threshold and cooldown configure `managed monitor` (the one-shot command still
|
||||||
**An injected autonomous scheduling core exists, but no monitor command, actual
|
does not schedule anything). **CLI/entrypoint integration exists; tests/build
|
||||||
recovery callback or live managed target is wired yet**.
|
and live target validation remain deferred.**
|
||||||
|
|
||||||
The future managed target is a **brand-new prod-like zapret2-ss container in the
|
The future managed target is a **brand-new prod-like zapret2-ss container in the
|
||||||
test environment only**, not the existing container. Its stable configurable
|
test environment only**, not the existing container. Its stable configurable
|
||||||
@@ -105,9 +117,9 @@ their existing target constraint. Database failures emit no options.
|
|||||||
Signed URLs are not printed. Scanner diagnostics are retained in restricted
|
Signed URLs are not printed. Scanner diagnostics are retained in restricted
|
||||||
artifact files rather than relayed to the terminal. No configuration is applied,
|
artifact files rather than relayed to the terminal. No configuration is applied,
|
||||||
no profile is marked active, and no service is restarted. The staged pipeline is
|
no profile is marked active, and no service is restarted. The staged pipeline is
|
||||||
scan/propose → guarded manual managed-test activation → future monitoring;
|
scan/propose → guarded managed-test activation → dedicated target monitoring;
|
||||||
the deployment definition exists but is not deployed; the monitor scheduling core
|
the deployment definition exists but is not deployed. `managed monitor` wires
|
||||||
is internal and **not wired to a command or actual recovery**. Manual
|
the scheduler and recovery; live compatibility remains unverified. Manual
|
||||||
proposal-only recovery is available below.
|
proposal-only recovery is available below.
|
||||||
No-argument `tune youtube` remains frontend-only and unchanged.
|
No-argument `tune youtube` remains frontend-only and unchanged.
|
||||||
|
|
||||||
@@ -190,8 +202,8 @@ or duplicate completion is rejected without changing the lock, token or cooldown
|
|||||||
even after a newer acquisition starts. State survives reopening the database.
|
even after a newer acquisition starts. State survives reopening the database.
|
||||||
The internal `runManagedMonitor` supervisor probes immediately and periodically,
|
The internal `runManagedMonitor` supervisor probes immediately and periodically,
|
||||||
persists observations, and invokes injected recovery only at threshold outside
|
persists observations, and invokes injected recovery only at threshold outside
|
||||||
cooldown. Recovery errors are reported without stopping monitoring; DB, lock and
|
cooldown. Unresolved GGC is reported without stopping monitoring; DB, safety,
|
||||||
finalization errors are fatal. Cooldown starts when recovery returns, including
|
runner, lock and finalization errors are fatal. Cooldown starts when recovery returns, including
|
||||||
failure or cancellation. Callbacks must honor cancellation and join their children.
|
failure or cancellation. Callbacks must honor cancellation and join their children.
|
||||||
Existing target guards run before a lifetime nonblocking `flock` in the private
|
Existing target guards run before a lifetime nonblocking `flock` in the private
|
||||||
target state directory. With that exclusive singleton held, one-shot startup
|
target state directory. With that exclusive singleton held, one-shot startup
|
||||||
@@ -201,8 +213,14 @@ A live owner remains busy; legacy or unverifiable owners fail closed. Probe
|
|||||||
history is preserved. The lock file is never unlinked.
|
history is preserved. The lock file is never unlinked.
|
||||||
The scheduler core has fake-time tests for scheduling, reset, cooldown,
|
The scheduler core has fake-time tests for scheduling, reset, cooldown,
|
||||||
cancellation, parallel starts, fatal DB errors, reopen and safe startup
|
cancellation, parallel starts, fatal DB errors, reopen and safe startup
|
||||||
reconciliation. **There is no actual recovery adapter,
|
reconciliation. Internal injected managed recovery and monitor callbacks now
|
||||||
monitor CLI/entrypoint wiring or live target yet**; container/Xray are untouched.
|
support saved-first frontend checks, scan fallback only for valid non-pass
|
||||||
|
evidence, and fresh exact-host GGC recovery with a frontend-only rollback boundary.
|
||||||
|
Checks remove prior combined reports; missing/malformed reports fail closed.
|
||||||
|
Scan shortlists must be regular files no older than the scan start's whole second
|
||||||
|
(production scans use unique temporary paths). Fake-only orchestration tests are
|
||||||
|
added; verification is deferred. **Monitor CLI/entrypoint wiring exists but no
|
||||||
|
live target has been deployed**; existing container/Xray are untouched.
|
||||||
Manual managed activation remains separate. Verification of this increment is
|
Manual managed activation remains separate. Verification of this increment is
|
||||||
deferred to the orchestrator.
|
deferred to the orchestrator.
|
||||||
|
|
||||||
@@ -214,8 +232,8 @@ in the same transaction as the existing run. Qualification requires every declar
|
|||||||
endpoint to pass exactly 3/3 for both TLS1.2 and TLS1.3, with full success/coverage.
|
endpoint to pass exactly 3/3 for both TLS1.2 and TLS1.3, with full success/coverage.
|
||||||
Existing v2 history is unchanged and is not backfilled into independent candidates.
|
Existing v2 history is unchanged and is not backfilled into independent candidates.
|
||||||
Candidates have no DB active/installed state; managed activation uses its own
|
Candidates have no DB active/installed state; managed activation uses its own
|
||||||
durable config journal. Autonomous scheduling is internal only; actual recovery
|
durable config journal. Scheduling and recovery are wired only to the dedicated
|
||||||
and live monitoring remain unwired.
|
managed target; live monitoring has not been verified or deployed.
|
||||||
The explicit `tune googlevideo` path adds exact-cache proposals only.
|
The explicit `tune googlevideo` path adds exact-cache proposals only.
|
||||||
Proposed runs store a SHA-256 fingerprint of validated arguments with separator
|
Proposed runs store a SHA-256 fingerprint of validated arguments with separator
|
||||||
whitespace normalized and argument order preserved; repeated proposals retain
|
whitespace normalized and argument order preserved; repeated proposals retain
|
||||||
|
|||||||
@@ -121,17 +121,65 @@ type storedCandidate struct {
|
|||||||
Qualified bool
|
Qualified bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Qualification precedes LIMIT: at most ten immutable definitions per trial.
|
||||||
|
func qualifiedCandidateIDs(db *sql.DB, group string) ([]int64, error) {
|
||||||
|
candidates, err := listQualifiedCandidates(db, group, 10)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var ids []int64
|
||||||
|
for _, c := range candidates {
|
||||||
|
ids = append(ids, c.ID)
|
||||||
|
}
|
||||||
|
return ids, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
const candidateQualifiedSQL = `NOT EXISTS (SELECT 1 FROM json_each(c.endpoints) e CROSS JOIN
|
||||||
|
(SELECT 'tls12' AS protocol UNION ALL SELECT 'tls13') p
|
||||||
|
WHERE NOT EXISTS (SELECT 1 FROM candidate_evidence r WHERE r.candidate_id = c.id
|
||||||
|
AND r.endpoint = e.value AND r.protocol = p.protocol AND r.passes_ok = 3
|
||||||
|
AND r.passes_total = 3 AND r.success_rate = 1 AND r.coverage = 1))`
|
||||||
|
|
||||||
|
// Returns detached snapshots of immutable definitions, not active state.
|
||||||
|
func listQualifiedCandidates(db *sql.DB, group string, limit int) ([]storedCandidate, error) {
|
||||||
|
if group != "frontend" && group != "googlevideo" {
|
||||||
|
return nil, fmt.Errorf("unknown candidate group")
|
||||||
|
}
|
||||||
|
if limit < 1 || limit > 10 {
|
||||||
|
return nil, fmt.Errorf("candidate limit must be between 1 and 10")
|
||||||
|
}
|
||||||
|
rows, err := db.Query(`SELECT id, strategy_group, args, fingerprint, endpoints FROM strategy_candidates c
|
||||||
|
WHERE strategy_group = ? AND `+candidateQualifiedSQL+` ORDER BY id DESC LIMIT ?`, group, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var result []storedCandidate
|
||||||
|
for rows.Next() {
|
||||||
|
var c storedCandidate
|
||||||
|
var scope string
|
||||||
|
if err := rows.Scan(&c.ID, &c.Group, &c.Args, &c.Fingerprint, &scope); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal([]byte(scope), &c.Endpoints); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
c.Qualified = true
|
||||||
|
result = append(result, c)
|
||||||
|
}
|
||||||
|
if err := rows.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return result, rows.Close()
|
||||||
|
}
|
||||||
|
|
||||||
// One shared qualification rule for the complete endpoint × TLS matrix.
|
// One shared qualification rule for the complete endpoint × TLS matrix.
|
||||||
// Missing or failed evidence is never a qualification; this is not active state.
|
// Missing or failed evidence is never a qualification; this is not active state.
|
||||||
func readStrategyCandidate(db *sql.DB, id int64) (storedCandidate, error) {
|
func readStrategyCandidate(db *sql.DB, id int64) (storedCandidate, error) {
|
||||||
var c storedCandidate
|
var c storedCandidate
|
||||||
var scope string
|
var scope string
|
||||||
err := db.QueryRow(`SELECT id, strategy_group, args, fingerprint, endpoints,
|
err := db.QueryRow(`SELECT id, strategy_group, args, fingerprint, endpoints,
|
||||||
NOT EXISTS (SELECT 1 FROM json_each(c.endpoints) e CROSS JOIN
|
`+candidateQualifiedSQL+`
|
||||||
(SELECT 'tls12' AS protocol UNION ALL SELECT 'tls13') p
|
|
||||||
WHERE NOT EXISTS (SELECT 1 FROM candidate_evidence r WHERE r.candidate_id = c.id
|
|
||||||
AND r.endpoint = e.value AND r.protocol = p.protocol AND r.passes_ok = 3
|
|
||||||
AND r.passes_total = 3 AND r.success_rate = 1 AND r.coverage = 1))
|
|
||||||
FROM strategy_candidates c WHERE id = ?`, id).Scan(&c.ID, &c.Group, &c.Args, &c.Fingerprint, &scope, &c.Qualified)
|
FROM strategy_candidates c WHERE id = ?`, id).Scan(&c.ID, &c.Group, &c.Args, &c.Fingerprint, &scope, &c.Qualified)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return c, err
|
return c, err
|
||||||
|
|||||||
@@ -15,8 +15,8 @@ Commands:
|
|||||||
discover youtube --video-id ID --proxy socks5://HOST:PORT
|
discover youtube --video-id ID --proxy socks5://HOST:PORT
|
||||||
Discover cache hostname only (no media-transfer check)
|
Discover cache hostname only (no media-transfer check)
|
||||||
health youtube One-shot Player API health through configured loopback SOCKS
|
health youtube One-shot Player API health through configured loopback SOCKS
|
||||||
managed init|status|apply
|
managed init|status|apply|monitor
|
||||||
Manual activation inside the designated managed-test target only
|
Guarded activation/monitoring in the dedicated managed-test target only
|
||||||
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
|
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
|
||||||
tune youtube Test youtube.com frontend only; print options without applying
|
tune youtube Test youtube.com frontend only; print options without applying
|
||||||
tune googlevideo --video-id ID --proxy socks5://HOST:PORT
|
tune googlevideo --video-id ID --proxy socks5://HOST:PORT
|
||||||
|
|||||||
@@ -305,6 +305,9 @@ func managedRuntimeGuard(d managedDeps) (int, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
||||||
|
if len(args) > 0 && args[0] == "monitor" {
|
||||||
|
return managedMonitorCLI(args[1:], stdout, stderr, d)
|
||||||
|
}
|
||||||
command, ids, err := managedArguments(args)
|
command, ids, err := managedArguments(args)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintln(stderr, "error:", err)
|
fmt.Fprintln(stderr, "error:", err)
|
||||||
|
|||||||
@@ -0,0 +1,157 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"path/filepath"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Provisioning owns executable, database, proxy and target paths; no path flags.
|
||||||
|
func managedMonitorConfig(args []string, lookup func(string) (string, bool)) (healthConfig, error) {
|
||||||
|
cfg, err := parseHealthConfig(lookup)
|
||||||
|
if err != nil {
|
||||||
|
return cfg, err
|
||||||
|
}
|
||||||
|
f := flag.NewFlagSet("managed monitor", flag.ContinueOnError)
|
||||||
|
f.SetOutput(io.Discard)
|
||||||
|
f.DurationVar(&cfg.HealthInterval, "interval", cfg.HealthInterval, "")
|
||||||
|
f.IntVar(&cfg.FailureThreshold, "failure-threshold", cfg.FailureThreshold, "")
|
||||||
|
f.DurationVar(&cfg.RecoveryCooldown, "cooldown", cfg.RecoveryCooldown, "")
|
||||||
|
if f.Parse(args) != nil || f.NArg() != 0 || cfg.HealthInterval <= 0 || cfg.HealthInterval > 24*time.Hour || cfg.RecoveryCooldown <= 0 || cfg.RecoveryCooldown > 7*24*time.Hour || cfg.FailureThreshold < 1 || cfg.FailureThreshold > 1000 {
|
||||||
|
return healthConfig{}, fmt.Errorf("monitor requires --interval in (0,24h], --failure-threshold in [1,1000], --cooldown in (0,168h]; no path/command flags")
|
||||||
|
}
|
||||||
|
return cfg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func managedMonitorDatabase(d managedDeps) (string, error) {
|
||||||
|
path, _ := d.lookup("ZAPRET_AUTOTUNE_DB")
|
||||||
|
if path != managedTargetDirectory+"/candidates.db" {
|
||||||
|
return "", fmt.Errorf("monitor database must be target-owned candidates.db")
|
||||||
|
}
|
||||||
|
for _, suffix := range []string{"", "-wal", "-shm", "-journal"} {
|
||||||
|
p := filepath.Join(d.directory, "candidates.db"+suffix)
|
||||||
|
if _, err := os.Lstat(p); errors.Is(err, os.ErrNotExist) {
|
||||||
|
continue
|
||||||
|
} else if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if err := managedPrivateFile(p); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return filepath.Join(d.directory, "candidates.db"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func managedMonitorCLI(args []string, stdout, stderr io.Writer, target managedDeps) int {
|
||||||
|
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
defer cancel()
|
||||||
|
return managedMonitorCLIContext(ctx, args, stdout, stderr, target, runManagedMonitor)
|
||||||
|
}
|
||||||
|
|
||||||
|
func managedMonitorCLIContext(ctx context.Context, args []string, stdout, stderr io.Writer, target managedDeps, supervise func(context.Context, managedMonitorDeps) error) int {
|
||||||
|
cfg, err := managedMonitorConfig(args, target.lookup)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: invalid managed monitor configuration")
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
if code, err := managedRuntimeGuard(target); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: managed monitor target guard refused")
|
||||||
|
return code
|
||||||
|
}
|
||||||
|
m := &managedTransaction{Dir: target.directory, Name: "config"}
|
||||||
|
if err := managedDirectory(target.directory); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: unsafe managed monitor directory")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
if err := managedCheckIdentity(m); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: managed monitor identity refused")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
path, err := managedMonitorDatabase(target)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: unsafe managed monitor database")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
db, err := openStore(path)
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: managed monitor database unavailable")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
recovery, err := managedMonitorRecoveryAdapters(target, db, runBlockcheck, runCombinedBlockcheckForHost, func(ctx context.Context, path string, args ...string) error {
|
||||||
|
return runBlockcheckCommand(ctx, path, "", args, io.Discard, io.Discard)
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: managed monitor adapters refused")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
monitor := managedMonitorDeps{
|
||||||
|
target: target, db: db, serviceID: managedTargetID + "/youtube",
|
||||||
|
interval: cfg.HealthInterval, threshold: cfg.FailureThreshold, cooldown: cfg.RecoveryCooldown,
|
||||||
|
now: time.Now, wait: waitManagedMonitor,
|
||||||
|
reportRecoveryError: func(error) { fmt.Fprintln(stderr, "managed recovery failed (details suppressed)") },
|
||||||
|
}
|
||||||
|
if err := wireManagedRecovery(&monitor, recovery, func(error) {
|
||||||
|
fmt.Fprintln(stderr, "managed GGC unresolved; monitoring continues (not playback health)")
|
||||||
|
}); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "error: managed monitor wiring refused")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stdout, "managed monitor: Player API via owned loopback SOCKS; not playback health")
|
||||||
|
err = supervise(ctx, monitor)
|
||||||
|
if err != nil {
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
if ctx.Err() != nil && managedCancellationOnly(err, ctx.Err()) && !errors.As(err, &fatal) {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stderr, "error: managed monitor stopped; safety/database/runtime failure (details suppressed)")
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func managedMonitorRecoveryAdapters(target managedDeps, db *sql.DB, scanCommand cacheScan, checkCommand cacheCheck, command func(context.Context, string, ...string) error) (managedRecoveryDeps, error) {
|
||||||
|
if scanCommand == nil || checkCommand == nil || command == nil {
|
||||||
|
return managedRecoveryDeps{}, fmt.Errorf("complete managed process adapters required")
|
||||||
|
}
|
||||||
|
// The blockcheck process-group runner cancels and waits descendants.
|
||||||
|
scan := func(ctx context.Context, _ string, host string, _, _ io.Writer) (string, error) {
|
||||||
|
return scanCommand(ctx, "/usr/bin/blockcheckw", host, io.Discard, io.Discard)
|
||||||
|
}
|
||||||
|
check := func(ctx context.Context, _ string, path, args, host string, _, _ io.Writer) error {
|
||||||
|
return checkCommand(ctx, "/usr/bin/blockcheckw", path, args, host, io.Discard, io.Discard)
|
||||||
|
}
|
||||||
|
run := func(ctx context.Context, path string, args ...string) error {
|
||||||
|
if path != managedZapretScript || len(args) != 1 || (args[0] != "start" && args[0] != "stop" && args[0] != "restart") {
|
||||||
|
return fmt.Errorf("invalid target-local daemon operation")
|
||||||
|
}
|
||||||
|
bounded, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
return command(bounded, path, args...)
|
||||||
|
}
|
||||||
|
return managedRecoveryDependencies(target, db, scan, check, run)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Joined cancellation plus failed cleanup/DB/lock close is not a graceful exit.
|
||||||
|
func managedCancellationOnly(err, cancellation error) bool {
|
||||||
|
if err == cancellation {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if joined, ok := err.(interface{ Unwrap() []error }); ok {
|
||||||
|
for _, e := range joined.Unwrap() {
|
||||||
|
if !managedCancellationOnly(e, cancellation) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return len(joined.Unwrap()) != 0
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,289 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestManagedMonitorArguments(t *testing.T) {
|
||||||
|
lookup := func(string) (string, bool) { return "", false }
|
||||||
|
cfg, err := managedMonitorConfig(nil, lookup)
|
||||||
|
if err != nil || cfg.HealthInterval != 5*time.Minute || cfg.FailureThreshold != 3 || cfg.RecoveryCooldown != 30*time.Minute {
|
||||||
|
t.Fatal(cfg, err)
|
||||||
|
}
|
||||||
|
cfg, err = managedMonitorConfig([]string{"--interval", "2s", "--failure-threshold", "2", "--cooldown", "1m"}, lookup)
|
||||||
|
if err != nil || cfg.HealthInterval != 2*time.Second || cfg.FailureThreshold != 2 || cfg.RecoveryCooldown != time.Minute {
|
||||||
|
t.Fatal(cfg, err)
|
||||||
|
}
|
||||||
|
for _, args := range [][]string{{"--interval", "0"}, {"--interval", "25h"}, {"--cooldown", "169h"}, {"--cooldown", "-1s"}, {"--failure-threshold", "1001"}, {"--failure-threshold", "0"}, {"--command", "secret"}, {"--db", "/etc/db"}, {"extra"}} {
|
||||||
|
if _, err := managedMonitorConfig(args, lookup); err == nil {
|
||||||
|
t.Fatal("accepted", args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := managedMonitorConfig(nil, func(key string) (string, bool) { return "999h", key == "ZAPRET_AUTOTUNE_HEALTH_INTERVAL" }); err == nil {
|
||||||
|
t.Fatal("accepted oversized environment policy")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedMonitorCancellationSafety(t *testing.T) {
|
||||||
|
if !managedCancellationOnly(errors.Join(context.Canceled), context.Canceled) || managedCancellationOnly(errors.Join(context.Canceled, errors.New("DB failure")), context.Canceled) || managedCancellationOnly(&managedRecoveryFatal{Err: context.Canceled}, context.Canceled) {
|
||||||
|
t.Fatal("cancellation hid a fatal error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedMonitorProductionAdapterBinding(t *testing.T) {
|
||||||
|
recovery, _, _ := recoveryFixture(t, false)
|
||||||
|
target, _, _ := managedCLIFixture(t)
|
||||||
|
target.directory = recovery.transaction.Dir
|
||||||
|
target.player = func(ctx context.Context, args []string, out, _ io.Writer) error {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
t.Fatal(ctx.Err())
|
||||||
|
}
|
||||||
|
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5://127.0.0.1:11080|") {
|
||||||
|
t.Fatal(args)
|
||||||
|
}
|
||||||
|
_, err := io.WriteString(out, `{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"https://fresh.googlevideo.com/video?secret=never-print"}}`)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
scans, checks, commands := 0, 0, 0
|
||||||
|
adapted, err := managedMonitorRecoveryAdapters(target, recovery.db,
|
||||||
|
func(ctx context.Context, binary, host string, out, diagnostic io.Writer) (string, error) {
|
||||||
|
if binary != "/usr/bin/blockcheckw" {
|
||||||
|
t.Fatal(binary)
|
||||||
|
}
|
||||||
|
scans++
|
||||||
|
return recovery.scan(ctx, "blockcheckw", host, out, diagnostic)
|
||||||
|
},
|
||||||
|
func(ctx context.Context, binary, path, args, host string, out, diagnostic io.Writer) error {
|
||||||
|
if binary != "/usr/bin/blockcheckw" {
|
||||||
|
t.Fatal(binary)
|
||||||
|
}
|
||||||
|
checks++
|
||||||
|
return recovery.check(ctx, "blockcheckw", path, args, host, out, diagnostic)
|
||||||
|
},
|
||||||
|
func(ctx context.Context, path string, args ...string) error {
|
||||||
|
if _, ok := ctx.Deadline(); !ok {
|
||||||
|
t.Fatal("unbounded daemon operation")
|
||||||
|
}
|
||||||
|
commands++
|
||||||
|
return recovery.run(ctx, path, args...)
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
result, err := recoverManagedTarget(context.Background(), adapted)
|
||||||
|
if err != nil || result.GGCError != nil || result.FrontendID == 0 || result.GooglevideoID == 0 || scans != 2 || checks != 2 || commands == 0 {
|
||||||
|
t.Fatal(result, err, scans, checks, commands)
|
||||||
|
}
|
||||||
|
if err := adapted.run(context.Background(), "/bin/sh", "-c", "password"); err == nil {
|
||||||
|
t.Fatal("accepted shell command")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedMonitorBootstrapUnresolvedGGC(t *testing.T) {
|
||||||
|
recovery, _, _ := recoveryFixture(t, false)
|
||||||
|
monitor := monitorRuntimeFixture(t)
|
||||||
|
monitor.db, monitor.target.directory = recovery.db, recovery.transaction.Dir
|
||||||
|
identity := `{"Role":"managed-test","Directory":"` + recovery.transaction.Dir + `","ID":"` + strings.Repeat("01", 16) + `"}`
|
||||||
|
if err := recovery.transaction.replace(managedMarkerName, []byte(identity)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
recovery.discover = func(context.Context) (string, error) { return "", errors.New("secret") }
|
||||||
|
reports := 0
|
||||||
|
if err := wireManagedRecovery(&monitor, recovery, func(error) { reports++ }); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := monitor.bootstrap(context.Background()); err != nil || reports != 1 {
|
||||||
|
t.Fatal(err, reports)
|
||||||
|
}
|
||||||
|
if err := monitor.probe(context.Background()); err != nil {
|
||||||
|
t.Fatal("unresolved GGC stopped API probe", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedMonitorCLIGuardsBeforeLock(t *testing.T) {
|
||||||
|
for _, mode := range []string{"role", "identity", "db", "alias", "config"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, env, _ := managedCLIFixture(t)
|
||||||
|
initManagedCLI(t, d)
|
||||||
|
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
|
||||||
|
switch mode {
|
||||||
|
case "role":
|
||||||
|
env["ZAPRET_AUTOTUNE_TARGET_ROLE"] = "old-target"
|
||||||
|
case "identity":
|
||||||
|
os.Remove(filepath.Join(d.directory, managedMarkerName))
|
||||||
|
case "db":
|
||||||
|
env["ZAPRET_AUTOTUNE_DB"] = "/other/db"
|
||||||
|
case "alias":
|
||||||
|
if err := os.Symlink(filepath.Join(t.TempDir(), "db"), filepath.Join(d.directory, "candidates.db")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "config":
|
||||||
|
env["ZAPRET_AUTOTUNE_VIDEO_ID"] = "password-do-not-print"
|
||||||
|
}
|
||||||
|
var out bytes.Buffer
|
||||||
|
code := managedMonitorCLIContext(context.Background(), nil, &out, &out, d, func(context.Context, managedMonitorDeps) error { t.Fatal("unsafe supervisor start"); return nil })
|
||||||
|
if code == 0 || strings.Contains(out.String(), "password-do-not-print") {
|
||||||
|
t.Fatal(code, out.String())
|
||||||
|
}
|
||||||
|
if _, err := os.Lstat(filepath.Join(d.directory, managedMonitorLockName)); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatal("guard acquired lock", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedMonitorCLIBindingAndCancellation(t *testing.T) {
|
||||||
|
d, env, _ := managedCLIFixture(t)
|
||||||
|
initManagedCLI(t, d)
|
||||||
|
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
requests := 0
|
||||||
|
d.player = func(ctx context.Context, args []string, out, _ io.Writer) error {
|
||||||
|
requests++
|
||||||
|
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5://127.0.0.1:11080|") {
|
||||||
|
t.Fatal(args)
|
||||||
|
}
|
||||||
|
if requests == 1 {
|
||||||
|
_, err := io.WriteString(out, `{"playabilityStatus":{"status":"OK"}}`)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
<-ctx.Done()
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
var out bytes.Buffer
|
||||||
|
code := managedMonitorCLIContext(ctx, nil, &out, &out, d, func(ctx context.Context, m managedMonitorDeps) error {
|
||||||
|
if m.bootstrap == nil || m.recover == nil || m.interval != 5*time.Minute || m.threshold != 3 || m.cooldown != 30*time.Minute {
|
||||||
|
t.Fatal("missing wiring/defaults")
|
||||||
|
}
|
||||||
|
if err := m.probe(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = m.probe(ctx)
|
||||||
|
return ctx.Err()
|
||||||
|
})
|
||||||
|
if code != 0 || requests != 2 {
|
||||||
|
t.Fatal(code, requests, out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedMonitorBootstrapExistingFrontend(t *testing.T) {
|
||||||
|
recovery, _, events := recoveryFixture(t, true)
|
||||||
|
monitor := monitorRuntimeFixture(t)
|
||||||
|
monitor.db, monitor.target.directory = recovery.db, recovery.transaction.Dir
|
||||||
|
identity := `{"Role":"managed-test","Directory":"` + recovery.transaction.Dir + `","ID":"` + strings.Repeat("01", 16) + `"}`
|
||||||
|
if err := recovery.transaction.replace(managedMarkerName, []byte(identity)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := wireManagedRecovery(&monitor, recovery, func(error) { t.Fatal("unexpected GGC recovery") }); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := monitor.bootstrap(context.Background()); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(*events) != 0 {
|
||||||
|
t.Fatal("startup rescanned qualified active frontend", *events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Execute only rewritten target-local contracts and fake programs; no network,
|
||||||
|
// Docker, real credentials, zapret or host firewall. Exercise the actual traps.
|
||||||
|
func TestManagedEntrypointSupervision(t *testing.T) {
|
||||||
|
for _, mode := range []string{"term", "fatal"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
config := filepath.Join(root, "ss")
|
||||||
|
if err := os.Mkdir(config, 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
write := func(name, text string) string {
|
||||||
|
path := filepath.Join(root, name)
|
||||||
|
if err := os.WriteFile(path, []byte(text), 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
log := filepath.Join(root, "log")
|
||||||
|
write("secret", strings.Repeat("a", 32))
|
||||||
|
write("awk", "#!/bin/sh\ncase \"$*\" in */proc/net/tcp*) until grep -q ready-ss-server \"$LOG\" && grep -q ready-ss-local \"$LOG\"; do sleep 0.02; done;; esac\nexit 0\n")
|
||||||
|
child := "#!/bin/sh\ntrap 'echo stopped-$0 >> \"$LOG\"; exit 0' TERM INT\necho ready-$(basename \"$0\") >> \"$LOG\"\nwhile :; do sleep 0.1 & wait $!; done\n"
|
||||||
|
write("ss-server", child)
|
||||||
|
write("ss-local", child)
|
||||||
|
zapret := write("zapret", "#!/bin/sh\necho zapret-$1 >> \"$LOG\"\n")
|
||||||
|
monitor := write("autotune", "#!/bin/sh\nif [ \"$2\" != monitor ]; then echo managed-$2 >> \"$LOG\"; exit 0; fi\ntrap 'echo monitor-stopped >> \"$LOG\"; exit 0' TERM INT\necho managed-monitor >> \"$LOG\"\nif [ \"$MODE\" = fatal ]; then exit 7; fi\nwhile :; do sleep 0.1 & wait $!; done\n")
|
||||||
|
data, err := os.ReadFile("../../managed-target-entrypoint.sh")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
text := strings.NewReplacer("/run/secrets/ss_password", filepath.Join(root, "secret"), "/run/zapret-autotune-ss", config, "/usr/local/bin/zapret-autotune", monitor, managedZapretScript, zapret).Replace(string(data))
|
||||||
|
entry := write("entrypoint", text)
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
cmd := exec.CommandContext(ctx, "/bin/sh", entry)
|
||||||
|
cmd.Env = append(os.Environ(), "PATH="+root+":"+os.Getenv("PATH"), "LOG="+log, "MODE="+mode, "ZAPRET_CONFIG="+filepath.Join(root, "config"))
|
||||||
|
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||||
|
var output bytes.Buffer
|
||||||
|
cmd.Stdout, cmd.Stderr = &output, &output
|
||||||
|
if err := cmd.Start(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
done := make(chan error, 1)
|
||||||
|
exited := make(chan struct{})
|
||||||
|
go func() { done <- cmd.Wait(); close(exited) }()
|
||||||
|
t.Cleanup(func() { syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL); <-exited })
|
||||||
|
if mode == "term" {
|
||||||
|
for {
|
||||||
|
b, _ := os.ReadFile(log)
|
||||||
|
if strings.Contains(string(b), "managed-monitor") {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
t.Fatal("premature exit", err)
|
||||||
|
case <-ctx.Done():
|
||||||
|
t.Fatal(ctx.Err())
|
||||||
|
case <-time.After(20 * time.Millisecond):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
err = <-done
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
t.Fatal("entrypoint did not join children", ctx.Err())
|
||||||
|
}
|
||||||
|
if mode == "term" && err != nil {
|
||||||
|
t.Fatal(err, output.String())
|
||||||
|
}
|
||||||
|
if mode == "fatal" {
|
||||||
|
var exit *exec.ExitError
|
||||||
|
if !errors.As(err, &exit) || exit.ExitCode() != 7 {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
b, err := os.ReadFile(log)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
text = string(b)
|
||||||
|
if !strings.Contains(text, "managed-init\nzapret-start\nmanaged-monitor\n") || !strings.Contains(text, "zapret-stop") || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-server")) || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-local")) {
|
||||||
|
t.Fatal(text)
|
||||||
|
}
|
||||||
|
if mode == "term" && (!strings.Contains(text, "monitor-stopped") || strings.Index(text, "monitor-stopped") > strings.Index(text, "zapret-stop")) {
|
||||||
|
t.Fatal("stop before monitor joined", text)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,479 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const managedZapretScript = "/opt/zapret2/init.d/sysv/zapret2"
|
||||||
|
|
||||||
|
// Caller owns startup reconciliation and the monitor singleton. Discover and
|
||||||
|
// Health must bind to the guarded target SOCKS. No CLI parsing or remote work.
|
||||||
|
type managedRecoveryDeps struct {
|
||||||
|
db *sql.DB
|
||||||
|
transaction *managedTransaction
|
||||||
|
scan cacheScan
|
||||||
|
check cacheCheck
|
||||||
|
discover func(context.Context) (string, error)
|
||||||
|
health func(context.Context) (string, error)
|
||||||
|
run func(context.Context, string, ...string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Target-local adapter: reuses bounded Player requests without invoking a CLI.
|
||||||
|
// The injected command runner receives only fixed script/argv from orchestration.
|
||||||
|
func managedRecoveryDependencies(target managedDeps, db *sql.DB, scan cacheScan, check cacheCheck, run func(context.Context, string, ...string) error) (managedRecoveryDeps, error) {
|
||||||
|
d := managedRecoveryDeps{db: db, scan: scan, check: check, run: run}
|
||||||
|
if target.lookup == nil || target.player == nil || db == nil || scan == nil || check == nil || run == nil {
|
||||||
|
return d, fmt.Errorf("complete target-local recovery adapter dependencies required")
|
||||||
|
}
|
||||||
|
if _, err := managedRuntimeGuard(target); err != nil {
|
||||||
|
return d, err
|
||||||
|
}
|
||||||
|
cfg, err := parseHealthConfig(target.lookup)
|
||||||
|
if err != nil {
|
||||||
|
return d, err
|
||||||
|
}
|
||||||
|
d.transaction = &managedTransaction{Dir: target.directory, Name: "config"}
|
||||||
|
request := func(ctx context.Context) ([]byte, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if _, err := managedRuntimeGuard(target); err != nil {
|
||||||
|
return nil, &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
bounded, cancel := context.WithTimeout(ctx, 35*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
body, err := requestPlayer(cfg.VideoID, cfg.SOCKSProxy, func(_ context.Context, args []string, out, diagnostics io.Writer) error {
|
||||||
|
return target.player(bounded, args, out, diagnostics)
|
||||||
|
})
|
||||||
|
if bounded.Err() != nil {
|
||||||
|
return nil, bounded.Err()
|
||||||
|
}
|
||||||
|
return body, err
|
||||||
|
}
|
||||||
|
d.health = func(ctx context.Context) (string, error) {
|
||||||
|
body, err := request(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
var response struct {
|
||||||
|
Playability struct {
|
||||||
|
Status string `json:"status"`
|
||||||
|
} `json:"playabilityStatus"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(body, &response); err != nil {
|
||||||
|
return "", fmt.Errorf("malformed Player API health response")
|
||||||
|
}
|
||||||
|
if response.Playability.Status == "" {
|
||||||
|
return "", fmt.Errorf("missing Player API health status")
|
||||||
|
}
|
||||||
|
if response.Playability.Status != "OK" {
|
||||||
|
return response.Playability.Status, nil
|
||||||
|
}
|
||||||
|
if err := playerHealth(body); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return "OK", nil
|
||||||
|
}
|
||||||
|
d.discover = func(ctx context.Context) (string, error) {
|
||||||
|
body, err := request(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return playerHostname(body)
|
||||||
|
}
|
||||||
|
return d, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type managedRecoveryFatal struct{ Err error }
|
||||||
|
|
||||||
|
func (e *managedRecoveryFatal) Error() string {
|
||||||
|
return "managed recovery safety failure: " + e.Err.Error()
|
||||||
|
}
|
||||||
|
func (e *managedRecoveryFatal) Unwrap() error { return e.Err }
|
||||||
|
|
||||||
|
type managedRecoveryResult struct {
|
||||||
|
FrontendID, GooglevideoID int64
|
||||||
|
GGCError error // Unresolved; never a playback/Player API health claim.
|
||||||
|
}
|
||||||
|
|
||||||
|
func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result managedRecoveryResult, err error) {
|
||||||
|
if d.db == nil || d.transaction == nil || d.scan == nil || d.check == nil || d.discover == nil || d.health == nil || d.run == nil {
|
||||||
|
return result, fmt.Errorf("complete managed recovery dependencies required")
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
m := *d.transaction
|
||||||
|
// Rollback must restart even after job cancellation; independently bounded.
|
||||||
|
m.Reload = func() error {
|
||||||
|
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if err := d.run(cleanup, managedZapretScript, "restart"); err != nil {
|
||||||
|
return &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
m.Health = func() (string, error) { return d.health(ctx) }
|
||||||
|
ids, err := m.ActiveCandidates(d.db)
|
||||||
|
if err != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
for _, id := range ids {
|
||||||
|
c, e := readStrategyCandidate(d.db, id)
|
||||||
|
if e != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
if c.Group == "frontend" {
|
||||||
|
result.FrontendID = id
|
||||||
|
}
|
||||||
|
}
|
||||||
|
stopped := false
|
||||||
|
defer func() {
|
||||||
|
if stopped {
|
||||||
|
// Never start bytes left pending by a failed rollback. Reconcile restores
|
||||||
|
// the last committed set or fails closed without activating unknown bytes.
|
||||||
|
if e := m.Reconcile(d.db); e != nil {
|
||||||
|
err = &managedRecoveryFatal{Err: errors.Join(err, e)}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if e := d.run(cleanup, managedZapretScript, "start"); e != nil {
|
||||||
|
err = &managedRecoveryFatal{Err: errors.Join(err, e)}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
stop := func() error {
|
||||||
|
stopped = true // A partially failed stop also needs restoration.
|
||||||
|
return d.run(ctx, managedZapretScript, "stop")
|
||||||
|
}
|
||||||
|
apply := func(ids []int64) error {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
cleanupReload := m.Reload
|
||||||
|
first := true
|
||||||
|
m.Reload = func() error {
|
||||||
|
if !first {
|
||||||
|
return cleanupReload()
|
||||||
|
}
|
||||||
|
first = false
|
||||||
|
activation, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if err := d.run(activation, managedZapretScript, "restart"); err != nil {
|
||||||
|
return &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer func() { m.Reload = cleanupReload }()
|
||||||
|
e := m.Apply(d.db, ids)
|
||||||
|
if e == nil {
|
||||||
|
stopped = false
|
||||||
|
}
|
||||||
|
var rollback *managedRollbackError
|
||||||
|
if errors.As(e, &rollback) {
|
||||||
|
return &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
var rejection *managedHealthRejection
|
||||||
|
if e != nil && !errors.As(e, &rejection) && !errors.Is(e, context.Canceled) && !errors.Is(e, context.DeadlineExceeded) {
|
||||||
|
return &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
return e
|
||||||
|
}
|
||||||
|
healthy := false
|
||||||
|
if result.FrontendID != 0 {
|
||||||
|
status, e := d.health(ctx)
|
||||||
|
var rejection *managedHealthRejection
|
||||||
|
if e != nil && !errors.As(e, &rejection) {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
healthy = status == "OK"
|
||||||
|
if e != nil {
|
||||||
|
healthy = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !healthy {
|
||||||
|
if e := stop(); e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
// Both bootstrap and live recovery check saved definitions before scanning.
|
||||||
|
saved, e := qualifiedCandidateIDs(d.db, "frontend")
|
||||||
|
if e != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
for _, id := range saved {
|
||||||
|
c, e := readStrategyCandidate(d.db, id)
|
||||||
|
if e != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
if len(c.Endpoints) != 1 || c.Endpoints[0] != "youtube.com" {
|
||||||
|
return result, fmt.Errorf("saved frontend scope is not youtube.com")
|
||||||
|
}
|
||||||
|
_, e = managedRecheck(ctx, d.check, c.Args, "frontend", "youtube.com")
|
||||||
|
if errors.Is(e, errCandidateNonpass) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
e = apply([]int64{id})
|
||||||
|
if e == nil {
|
||||||
|
result.FrontendID, healthy = id, true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
var rejection *managedHealthRejection
|
||||||
|
if errors.As(e, &fatal) || !errors.As(e, &rejection) {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
// Health rollback restarted the prior config: stop before rechecking.
|
||||||
|
if e := stop(); e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !healthy {
|
||||||
|
id, e := managedScanCandidate(ctx, d, "frontend", "youtube.com")
|
||||||
|
if e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
if e := apply([]int64{id}); e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
result.FrontendID = id
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A frontend-only rollback boundary prevents cache failures undoing bootstrap.
|
||||||
|
current, e := m.ActiveCandidates(d.db)
|
||||||
|
if e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
if len(current) != 1 || current[0] != result.FrontendID {
|
||||||
|
if e := apply([]int64{result.FrontendID}); e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
host, e := d.discover(ctx)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return result, ctx.Err()
|
||||||
|
}
|
||||||
|
if e != nil || !discoveryHostname(host) || host != strings.ToLower(host) || !strings.HasSuffix(host, ".googlevideo.com") {
|
||||||
|
result.GGCError = fmt.Errorf("exact GGC discovery failed")
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
if e := stop(); e != nil {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
args, report, e := revalidateGooglevideo(ctx, d.db, host, io.Discard, d.check)
|
||||||
|
var cacheID int64
|
||||||
|
if e == nil && args != "" {
|
||||||
|
e = saveGooglevideoRun(d.db, time.Now(), host, args, report, nil)
|
||||||
|
if e != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
if e == nil {
|
||||||
|
cacheID, e = newestManagedCandidateForScope(d.db, "googlevideo", host, args)
|
||||||
|
if e != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if e == nil {
|
||||||
|
cacheID, e = managedScanCandidate(ctx, d, "googlevideo", host)
|
||||||
|
}
|
||||||
|
if e == nil {
|
||||||
|
e = apply([]int64{result.FrontendID, cacheID})
|
||||||
|
}
|
||||||
|
if e != nil {
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
if errors.As(e, &fatal) {
|
||||||
|
return result, e
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return result, ctx.Err()
|
||||||
|
}
|
||||||
|
result.GGCError = fmt.Errorf("GGC validation or activation unresolved")
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
result.GooglevideoID = cacheID
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newestManagedCandidate(db *sql.DB, group string) (int64, error) {
|
||||||
|
ids, err := qualifiedCandidateIDs(db, group)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return 0, fmt.Errorf("no qualified candidate")
|
||||||
|
}
|
||||||
|
return ids[0], nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newestManagedCandidateForScope(db *sql.DB, group, host, args string) (int64, error) {
|
||||||
|
id, err := newestManagedCandidate(db, group)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
c, err := readStrategyCandidate(db, id)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if c.Args != args || len(c.Endpoints) != 1 || c.Endpoints[0] != host {
|
||||||
|
return 0, fmt.Errorf("latest saved candidate scope changed")
|
||||||
|
}
|
||||||
|
return id, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func managedRecheck(ctx context.Context, check cacheCheck, args, group, host string) ([]byte, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
f, err := os.CreateTemp("", "blockcheckw-managed-*.txt")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
path := f.Name()
|
||||||
|
if err := f.Close(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return revalidateCandidate(ctx, storedCandidate{Group: group, Endpoints: []string{host}, Args: args, Qualified: true}, group, host, path, io.Discard, check)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reuses proposal selection and transactional persistence with caller context.
|
||||||
|
func managedScanCandidate(ctx context.Context, d managedRecoveryDeps, group, host string) (int64, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
started := time.Now()
|
||||||
|
path, err := d.scan(ctx, "blockcheckw", host, io.Discard, io.Discard)
|
||||||
|
var report []byte
|
||||||
|
var args, output string
|
||||||
|
if err == nil {
|
||||||
|
var info os.FileInfo
|
||||||
|
info, err = os.Stat(path + ".check.json")
|
||||||
|
// ponytail: whole-second mtime permits same-second artifacts from injected
|
||||||
|
// scans; production uses unique paths. Require that contract if precision grows.
|
||||||
|
if err == nil && (!info.Mode().IsRegular() || info.ModTime().Before(started.Truncate(time.Second))) {
|
||||||
|
err = fmt.Errorf("missing fresh scan shortlist")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
report, err = os.ReadFile(path + ".check.json")
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
args, err = selectBlockcheckArgsForTarget(bytes.NewReader(report), "", group, host)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
report, err = revalidateCandidate(ctx, storedCandidate{Group: group, Endpoints: []string{host}, Args: args, Qualified: true}, group, host, path, io.Discard, d.check)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
err = ctx.Err()
|
||||||
|
}
|
||||||
|
if group == "frontend" {
|
||||||
|
if err == nil {
|
||||||
|
output, err = combinedBlockcheckOutput(bytes.NewReader(report), args)
|
||||||
|
}
|
||||||
|
if e := saveRun(d.db, started, args, output, report, err); e != nil {
|
||||||
|
return 0, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
var safe error
|
||||||
|
if err != nil {
|
||||||
|
safe = fmt.Errorf("cache scan or verification failed")
|
||||||
|
}
|
||||||
|
if e := saveGooglevideoRun(d.db, started, host, args, report, safe); e != nil {
|
||||||
|
return 0, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
id, err := newestManagedCandidateForScope(d.db, group, host, args)
|
||||||
|
if err != nil {
|
||||||
|
return 0, &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
return id, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Installs callbacks. runManagedMonitor performs startup
|
||||||
|
// under its singleton. Target guards are repeated before every recovery/probe.
|
||||||
|
func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDeps, reportGGC func(error)) error {
|
||||||
|
if monitor == nil || monitor.db == nil || monitor.db != recovery.db || recovery.transaction == nil || reportGGC == nil || recovery.health == nil || recovery.run == nil || recovery.scan == nil || recovery.check == nil || recovery.discover == nil {
|
||||||
|
return fmt.Errorf("matching monitor/recovery database and hooks required")
|
||||||
|
}
|
||||||
|
if recovery.transaction.Dir != monitor.target.directory || recovery.transaction.Name != "config" {
|
||||||
|
return fmt.Errorf("recovery transaction must own the monitor target config")
|
||||||
|
}
|
||||||
|
guard := func() error {
|
||||||
|
if _, err := managedRuntimeGuard(monitor.target); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return managedCheckIdentity(recovery.transaction)
|
||||||
|
}
|
||||||
|
callback := func(ctx context.Context) error {
|
||||||
|
if err := guard(); err != nil {
|
||||||
|
return &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
result, err := recoverManagedTarget(ctx, recovery)
|
||||||
|
if result.GGCError != nil {
|
||||||
|
reportGGC(result.GGCError)
|
||||||
|
}
|
||||||
|
// The supervisor reports ordinary recovery failures and retries only
|
||||||
|
// after its durable cooldown. Safety failures (rollback/cleanup/drift)
|
||||||
|
// are already typed managedRecoveryFatal by the transaction boundary.
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
monitor.bootstrap = func(ctx context.Context) error {
|
||||||
|
if err := guard(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// Reconcile once at startup, never adopt a pending live deployment.
|
||||||
|
transaction := *recovery.transaction
|
||||||
|
transaction.Health = func() (string, error) { return recovery.health(ctx) }
|
||||||
|
transaction.Reload = func() error {
|
||||||
|
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
return recovery.run(cleanup, managedZapretScript, "restart")
|
||||||
|
}
|
||||||
|
if err := transaction.Reconcile(recovery.db); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
ids, err := transaction.ActiveCandidates(recovery.db)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, id := range ids {
|
||||||
|
candidate, err := readStrategyCandidate(recovery.db, id)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if candidate.Group == "frontend" && candidate.Qualified {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return callback(ctx)
|
||||||
|
}
|
||||||
|
monitor.recover = callback
|
||||||
|
monitor.probe = func(ctx context.Context) error {
|
||||||
|
if err := guard(); err != nil {
|
||||||
|
return &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
status, err := recovery.health(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if status != "OK" {
|
||||||
|
return &managedHealthRejection{Status: status}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,514 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func recoveryFixture(t *testing.T, active bool) (managedRecoveryDeps, int64, *[]string) {
|
||||||
|
t.Helper()
|
||||||
|
t.Setenv("TMPDIR", t.TempDir())
|
||||||
|
db := testStore(t)
|
||||||
|
m, _ := managedFixture(t)
|
||||||
|
if err := m.applyRendered([]byte(managedBaseline)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var front int64
|
||||||
|
if active {
|
||||||
|
front = managedCandidateFixture(t, db, "frontend", "youtube.com", "--lua-desync=fake", true)
|
||||||
|
if err := m.Apply(db, []int64{front}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
events := new([]string)
|
||||||
|
stopped := false
|
||||||
|
d := managedRecoveryDeps{
|
||||||
|
db: db, transaction: m,
|
||||||
|
health: func(context.Context) (string, error) { return "OK", nil },
|
||||||
|
discover: func(context.Context) (string, error) { return "fresh.googlevideo.com", nil },
|
||||||
|
run: func(ctx context.Context, path string, args ...string) error {
|
||||||
|
if path != managedZapretScript || len(args) != 1 {
|
||||||
|
t.Fatalf("unsafe command %s %v", path, args)
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
t.Fatal("cleanup received cancelled context", err)
|
||||||
|
}
|
||||||
|
*events = append(*events, args[0])
|
||||||
|
stopped = args[0] == "stop"
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
scan: func(ctx context.Context, binary, host string, _, _ io.Writer) (string, error) {
|
||||||
|
if !stopped || binary != "blockcheckw" {
|
||||||
|
t.Fatal("scan without dedicated downtime")
|
||||||
|
}
|
||||||
|
*events = append(*events, "scan:"+host)
|
||||||
|
path := filepath.Join(t.TempDir(), "scan")
|
||||||
|
return path, os.WriteFile(path+".check.json", []byte(combinedReportFixture("--lua-desync=fake")), 0600)
|
||||||
|
},
|
||||||
|
check: func(ctx context.Context, binary, path, args, host string, _, _ io.Writer) error {
|
||||||
|
if !stopped || binary != "blockcheckw" {
|
||||||
|
t.Fatal("check without dedicated downtime")
|
||||||
|
}
|
||||||
|
*events = append(*events, "check:"+host+":"+args)
|
||||||
|
return os.WriteFile(path+".combined.json", []byte(combinedReportFixture(args)), 0600)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return d, front, events
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryBootstrap(t *testing.T) {
|
||||||
|
d, _, events := recoveryFixture(t, false)
|
||||||
|
result, err := recoverManagedTarget(context.Background(), d)
|
||||||
|
if err != nil || result.GGCError != nil || result.FrontendID == 0 || result.GooglevideoID == 0 {
|
||||||
|
t.Fatalf("%+v %v", result, err)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, []int64{result.FrontendID, result.GooglevideoID})
|
||||||
|
config, err := d.transaction.read("config")
|
||||||
|
if err != nil || !strings.Contains(string(config), "--hostlist-domains=^fresh.googlevideo.com ") || strings.Contains(string(config), "--hostlist-domains=googlevideo.com") {
|
||||||
|
t.Fatalf("%s %v", config, err)
|
||||||
|
}
|
||||||
|
if (*events)[0] != "stop" || strings.Count(strings.Join(*events, ","), "restart") != 2 {
|
||||||
|
t.Fatal(*events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryBootstrapReusesSavedFrontend(t *testing.T) {
|
||||||
|
d, _, events := recoveryFixture(t, false)
|
||||||
|
savedID := managedCandidateFixture(t, d.db, "frontend", "youtube.com", "--lua-desync=saved", true)
|
||||||
|
result, err := recoverManagedTarget(context.Background(), d)
|
||||||
|
if err != nil || result.GGCError != nil || result.FrontendID != savedID {
|
||||||
|
t.Fatalf("result=%+v err=%v", result, err)
|
||||||
|
}
|
||||||
|
for _, event := range *events {
|
||||||
|
if event == "scan:youtube.com" {
|
||||||
|
t.Fatalf("frontend scan ran despite a passing saved candidate: %v", *events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !strings.Contains(strings.Join(*events, ","), "check:youtube.com:--lua-desync=saved") {
|
||||||
|
t.Fatalf("saved candidate was not revalidated: %v", *events)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, []int64{result.FrontendID, result.GooglevideoID})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoverySavedFirstAndFailClosed(t *testing.T) {
|
||||||
|
for _, mode := range []string{"pass", "nonpass", "malformed", "runtime", "checker-nonpass", "missing-report", "incomplete", "health-reject", "apply-infrastructure", "cancel", "cleanup-failure"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, old, events := recoveryFixture(t, true)
|
||||||
|
newID := managedCandidateFixture(t, d.db, "frontend", "youtube.com", "--lua-desync=multisplit", true)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
probes := 0
|
||||||
|
d.health = func(context.Context) (string, error) {
|
||||||
|
probes++
|
||||||
|
if mode == "apply-infrastructure" && probes == 2 {
|
||||||
|
return "", errors.New("probe infrastructure failed")
|
||||||
|
}
|
||||||
|
if probes == 1 || mode == "health-reject" && probes == 2 {
|
||||||
|
return "DEGRADED", nil
|
||||||
|
}
|
||||||
|
return "OK", nil
|
||||||
|
}
|
||||||
|
d.discover = func(context.Context) (string, error) { return "", errors.New("unresolved") }
|
||||||
|
checked := []string{}
|
||||||
|
baseCheck := d.check
|
||||||
|
d.check = func(ctx context.Context, binary, path, args, host string, out, stderr io.Writer) error {
|
||||||
|
checked = append(checked, args)
|
||||||
|
if mode == "checker-nonpass" {
|
||||||
|
return errCandidateNonpass
|
||||||
|
}
|
||||||
|
if mode == "runtime" || mode == "cleanup-failure" {
|
||||||
|
return errors.New("runner failed")
|
||||||
|
}
|
||||||
|
if mode == "cancel" {
|
||||||
|
cancel()
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if mode == "missing-report" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := baseCheck(ctx, binary, path, args, host, out, stderr); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
report := combinedReportFixture(args)
|
||||||
|
switch mode {
|
||||||
|
case "malformed":
|
||||||
|
report = `{}`
|
||||||
|
case "incomplete":
|
||||||
|
report = strings.ReplaceAll(report, "HTTPS/TLS1.3", "HTTPS/TLS1.2")
|
||||||
|
case "nonpass":
|
||||||
|
if !strings.Contains(path, "/scan") {
|
||||||
|
report = strings.Replace(report, `"passes_ok":3`, `"passes_ok":2`, 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return os.WriteFile(path+".combined.json", []byte(report), 0600)
|
||||||
|
}
|
||||||
|
if mode == "cleanup-failure" {
|
||||||
|
run := d.run
|
||||||
|
d.run = func(ctx context.Context, path string, args ...string) error {
|
||||||
|
if err := run(ctx, path, args...); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if args[0] == "start" {
|
||||||
|
return errors.New("cleanup failed")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result, err := recoverManagedTarget(ctx, d)
|
||||||
|
success := mode == "pass" || mode == "nonpass" || mode == "health-reject"
|
||||||
|
if (err == nil) != success {
|
||||||
|
t.Fatalf("result=%+v err=%v events=%v", result, err, *events)
|
||||||
|
}
|
||||||
|
if len(checked) == 0 || checked[0] != "--lua-desync=multisplit" {
|
||||||
|
t.Fatal(checked)
|
||||||
|
}
|
||||||
|
scanned := strings.Contains(strings.Join(*events, ","), "scan:")
|
||||||
|
if scanned != (mode == "nonpass") {
|
||||||
|
t.Fatal(*events)
|
||||||
|
}
|
||||||
|
want := old
|
||||||
|
if mode == "pass" {
|
||||||
|
want = newID
|
||||||
|
}
|
||||||
|
if mode == "nonpass" {
|
||||||
|
want = result.FrontendID
|
||||||
|
if want == old || want == newID {
|
||||||
|
t.Fatal("scan did not save fresh candidate")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, []int64{want})
|
||||||
|
if !success && (*events)[len(*events)-1] != "start" {
|
||||||
|
t.Fatal("no cleanup", *events)
|
||||||
|
}
|
||||||
|
if mode == "health-reject" && (len(checked) != 2 || strings.Count(strings.Join(*events, ","), "restart") != 3) {
|
||||||
|
t.Fatal("health rejection did not rollback and try older", *events, checked)
|
||||||
|
}
|
||||||
|
if mode == "cleanup-failure" {
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
if !errors.As(err, &fatal) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryScanArtifactsFailClosed(t *testing.T) {
|
||||||
|
for _, mode := range []string{"stale-combined", "remove-error", "missing-shortlist", "stale-shortlist", "malformed-combined", "cancel"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, _, events := recoveryFixture(t, false)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
path := filepath.Join(t.TempDir(), "scan")
|
||||||
|
d.scan = func(context.Context, string, string, io.Writer, io.Writer) (string, error) {
|
||||||
|
if mode != "missing-shortlist" {
|
||||||
|
if err := os.WriteFile(path+".check.json", []byte(combinedReportFixture("--lua-desync=fake")), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode == "stale-shortlist" {
|
||||||
|
old := time.Now().Add(-time.Hour)
|
||||||
|
if err := os.Chtimes(path+".check.json", old, old); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode == "remove-error" {
|
||||||
|
if err := os.Mkdir(path+".combined.json", 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(path+".combined.json", "keep"), nil, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
} else if err := os.WriteFile(path+".combined.json", []byte(combinedReportFixture("--lua-desync=fake")), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if mode == "cancel" {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return path, nil
|
||||||
|
}
|
||||||
|
checked := false
|
||||||
|
d.check = func(context.Context, string, string, string, string, io.Writer, io.Writer) error {
|
||||||
|
checked = true
|
||||||
|
if _, err := os.Stat(path + ".combined.json"); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatalf("old combined artifact survived: %v", err)
|
||||||
|
}
|
||||||
|
if mode == "malformed-combined" {
|
||||||
|
return os.WriteFile(path+".combined.json", []byte(`{}`), 0600)
|
||||||
|
}
|
||||||
|
return nil // No fresh report: old passing evidence must not qualify.
|
||||||
|
}
|
||||||
|
result, err := recoverManagedTarget(ctx, d)
|
||||||
|
if err == nil || result.FrontendID != 0 || result.GooglevideoID != 0 {
|
||||||
|
t.Fatalf("artifact qualified: %+v %v", result, err)
|
||||||
|
}
|
||||||
|
wantCheck := mode == "stale-combined" || mode == "malformed-combined"
|
||||||
|
if checked != wantCheck {
|
||||||
|
t.Fatalf("checked=%v want=%v", checked, wantCheck)
|
||||||
|
}
|
||||||
|
if mode == "cancel" && !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(*events, []string{"stop", "start"}) {
|
||||||
|
t.Fatalf("target not restored: %v", *events)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, nil)
|
||||||
|
assertManagedContent(t, d.transaction, managedBaseline)
|
||||||
|
ids, e := qualifiedCandidateIDs(d.db, "frontend")
|
||||||
|
if e != nil || len(ids) != 0 {
|
||||||
|
t.Fatalf("failed scan saved candidate: %v %v", ids, e)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryDiscoveryDropsOnlyOldGGC(t *testing.T) {
|
||||||
|
d, front, events := recoveryFixture(t, true)
|
||||||
|
cache := managedCandidateFixture(t, d.db, "googlevideo", "old.googlevideo.com", "--lua-desync=multisplit", true)
|
||||||
|
if err := d.transaction.Apply(d.db, []int64{front, cache}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
d.discover = func(context.Context) (string, error) { return "", errors.New("discovery failed") }
|
||||||
|
result, err := recoverManagedTarget(context.Background(), d)
|
||||||
|
if err != nil || result.GGCError == nil || result.FrontendID != front || result.GooglevideoID != 0 {
|
||||||
|
t.Fatalf("%+v %v", result, err)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
|
||||||
|
if !reflect.DeepEqual(*events, []string{"restart"}) {
|
||||||
|
t.Fatal(*events)
|
||||||
|
}
|
||||||
|
old, err := readStrategyCandidate(d.db, cache)
|
||||||
|
if err != nil || !old.Qualified || !reflect.DeepEqual(old.Endpoints, []string{"old.googlevideo.com"}) {
|
||||||
|
t.Fatal("immutable old GGC changed", old, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryRollbackFailureNeverTriesNext(t *testing.T) {
|
||||||
|
d, front, events := recoveryFixture(t, true)
|
||||||
|
managedCandidateFixture(t, d.db, "frontend", "youtube.com", "--lua-desync=multisplit", true)
|
||||||
|
d.health = func(context.Context) (string, error) { return "DEGRADED", nil }
|
||||||
|
run := d.run
|
||||||
|
restarts := 0
|
||||||
|
d.run = func(ctx context.Context, path string, args ...string) error {
|
||||||
|
if err := run(ctx, path, args...); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if args[0] == "restart" {
|
||||||
|
restarts++
|
||||||
|
if restarts == 2 {
|
||||||
|
return errors.New("rollback restart failed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
_, err := recoverManagedTarget(context.Background(), d)
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
if !errors.As(err, &fatal) {
|
||||||
|
t.Fatalf("rollback failure not fatal: %v", err)
|
||||||
|
}
|
||||||
|
joined := strings.Join(*events, ",")
|
||||||
|
if strings.Contains(joined, "scan:") || strings.Count(joined, "check:") != 1 {
|
||||||
|
t.Fatalf("rollback failure authorized another candidate: %v", *events)
|
||||||
|
}
|
||||||
|
if (*events)[len(*events)-1] != "start" {
|
||||||
|
t.Fatalf("reconciliation did not restore target: %v", *events)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryFreshGGCAndPartialFailure(t *testing.T) {
|
||||||
|
for _, mode := range []string{"pass", "check-error", "health-reject", "discovery-error", "wildcard", "cancel"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, front, events := recoveryFixture(t, true)
|
||||||
|
oldCache := managedCandidateFixture(t, d.db, "googlevideo", "old.googlevideo.com", "--lua-desync=multisplit", true)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
if mode == "discovery-error" {
|
||||||
|
d.discover = func(context.Context) (string, error) { return "", errors.New("secret URL") }
|
||||||
|
}
|
||||||
|
if mode == "wildcard" {
|
||||||
|
d.discover = func(context.Context) (string, error) { return "googlevideo.com", nil }
|
||||||
|
}
|
||||||
|
check := d.check
|
||||||
|
d.check = func(ctx context.Context, binary, path, args, host string, out, stderr io.Writer) error {
|
||||||
|
if host != "fresh.googlevideo.com" {
|
||||||
|
t.Fatal("old or wildcard scope", host)
|
||||||
|
}
|
||||||
|
if mode == "check-error" {
|
||||||
|
return errors.New("infrastructure")
|
||||||
|
}
|
||||||
|
if mode == "cancel" {
|
||||||
|
cancel()
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
return check(ctx, binary, path, args, host, out, stderr)
|
||||||
|
}
|
||||||
|
probes := 0
|
||||||
|
d.health = func(context.Context) (string, error) {
|
||||||
|
probes++
|
||||||
|
if mode == "health-reject" && probes > 1 {
|
||||||
|
return "DEGRADED", nil
|
||||||
|
}
|
||||||
|
return "OK", nil
|
||||||
|
}
|
||||||
|
result, err := recoverManagedTarget(ctx, d)
|
||||||
|
if mode == "cancel" {
|
||||||
|
if !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
} else if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if result.FrontendID != front {
|
||||||
|
t.Fatalf("%+v %v", result, err)
|
||||||
|
}
|
||||||
|
if strings.Contains(strings.Join(*events, ","), "scan:") {
|
||||||
|
t.Fatal("passing frontend/cache caused scan", *events)
|
||||||
|
}
|
||||||
|
want := []int64{front}
|
||||||
|
if mode == "pass" {
|
||||||
|
if result.GGCError != nil || result.GooglevideoID <= oldCache {
|
||||||
|
t.Fatalf("%+v", result)
|
||||||
|
}
|
||||||
|
want = append(want, result.GooglevideoID)
|
||||||
|
fresh, e := readStrategyCandidate(d.db, result.GooglevideoID)
|
||||||
|
old, oldErr := readStrategyCandidate(d.db, oldCache)
|
||||||
|
if e != nil || oldErr != nil || !reflect.DeepEqual(fresh.Endpoints, []string{"fresh.googlevideo.com"}) || !reflect.DeepEqual(old.Endpoints, []string{"old.googlevideo.com"}) {
|
||||||
|
t.Fatal(fresh, old, e, oldErr)
|
||||||
|
}
|
||||||
|
} else if mode != "cancel" && result.GGCError == nil {
|
||||||
|
t.Fatal("cache failure claimed success")
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedQualifiedCandidateBound(t *testing.T) {
|
||||||
|
db := testStore(t)
|
||||||
|
var last int64
|
||||||
|
for i := 0; i < 12; i++ {
|
||||||
|
last = managedCandidateFixture(t, db, "frontend", "youtube.com", "--lua-desync=fake", true)
|
||||||
|
}
|
||||||
|
managedCandidateFixture(t, db, "frontend", "youtube.com", "--lua-desync=fake", false)
|
||||||
|
ids, err := qualifiedCandidateIDs(db, "frontend")
|
||||||
|
if err != nil || len(ids) != 10 || ids[0] != last || ids[9] != last-9 {
|
||||||
|
t.Fatal(ids, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryMonitorCallbacks(t *testing.T) {
|
||||||
|
recovery, _, _ := recoveryFixture(t, false)
|
||||||
|
monitor := monitorRuntimeFixture(t)
|
||||||
|
monitor.db = recovery.db
|
||||||
|
monitor.target.directory = recovery.transaction.Dir
|
||||||
|
identity, err := json.Marshal(managedIdentity{Role: "managed-test", Directory: recovery.transaction.Dir, ID: strings.Repeat("01", 16)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := recovery.transaction.replace(managedMarkerName, identity); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var reports []error
|
||||||
|
if err := wireManagedRecovery(&monitor, recovery, func(err error) { reports = append(reports, err) }); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := monitor.bootstrap(context.Background()); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := monitor.probe(context.Background()); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(reports) != 0 {
|
||||||
|
t.Fatal(reports)
|
||||||
|
}
|
||||||
|
ids, err := recovery.transaction.ActiveCandidates(recovery.db)
|
||||||
|
if err != nil || len(ids) != 2 {
|
||||||
|
t.Fatal(ids, err)
|
||||||
|
}
|
||||||
|
// A drifted deployment is never adopted by startup/recovery callbacks.
|
||||||
|
if err := os.WriteFile(filepath.Join(recovery.transaction.Dir, "config"), []byte("external"), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := monitor.recover(context.Background()); !errors.Is(err, errManagedDrift) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryRuntimeFailures(t *testing.T) {
|
||||||
|
for _, mode := range []string{"stop", "restart", "health-infrastructure"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, front, events := recoveryFixture(t, true)
|
||||||
|
d.health = func(context.Context) (string, error) {
|
||||||
|
if mode == "health-infrastructure" {
|
||||||
|
return "", errors.New("request infrastructure failed")
|
||||||
|
}
|
||||||
|
return "DEGRADED", nil
|
||||||
|
}
|
||||||
|
run := d.run
|
||||||
|
d.run = func(ctx context.Context, path string, args ...string) error {
|
||||||
|
if err := run(ctx, path, args...); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if args[0] == mode {
|
||||||
|
return errors.New("runtime failed")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
_, err := recoverManagedTarget(context.Background(), d)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("runtime failure accepted")
|
||||||
|
}
|
||||||
|
if strings.Contains(strings.Join(*events, ","), "scan:") {
|
||||||
|
t.Fatal("infrastructure triggered scan", *events)
|
||||||
|
}
|
||||||
|
if mode == "restart" {
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
if !errors.As(err, &fatal) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Failed rollback restart keeps a pending journal; do not start it.
|
||||||
|
if ids, err := d.transaction.ActiveCandidates(d.db); err == nil || len(ids) != 0 {
|
||||||
|
t.Fatal(ids, err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
|
||||||
|
if mode == "stop" && (*events)[len(*events)-1] != "start" {
|
||||||
|
t.Fatal(*events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedRecoveryPlayerAdapter(t *testing.T) {
|
||||||
|
target, _, _ := managedCLIFixture(t)
|
||||||
|
recovery, _, _ := recoveryFixture(t, false)
|
||||||
|
d, err := managedRecoveryDependencies(target, recovery.db, recovery.scan, recovery.check, recovery.run)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
status, err := d.health(context.Background())
|
||||||
|
if status != "OK" || err != nil {
|
||||||
|
t.Fatal(status, err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
if _, err := d.health(ctx); !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := d.discover(context.Background()); err == nil {
|
||||||
|
t.Fatal("health-only response claimed GGC discovery")
|
||||||
|
}
|
||||||
|
if d.transaction.Dir != target.directory || d.transaction.Name != "config" {
|
||||||
|
t.Fatal("wrong target binding")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -18,6 +18,10 @@ import (
|
|||||||
|
|
||||||
var errManagedDrift = errors.New("managed config drift: refusing overwrite")
|
var errManagedDrift = errors.New("managed config drift: refusing overwrite")
|
||||||
|
|
||||||
|
type managedHealthRejection struct{ Status string }
|
||||||
|
|
||||||
|
func (e *managedHealthRejection) Error() string { return fmt.Sprintf("unhealthy status: %q", e.Status) }
|
||||||
|
|
||||||
type managedRollbackError struct{ Cause, Rollback error }
|
type managedRollbackError struct{ Cause, Rollback error }
|
||||||
|
|
||||||
func (e *managedRollbackError) Error() string {
|
func (e *managedRollbackError) Error() string {
|
||||||
@@ -410,7 +414,7 @@ func (m *managedTransaction) applyRenderedDB(b []byte, db *sql.DB, ids ...int64)
|
|||||||
var status string
|
var status string
|
||||||
status, err = m.Health()
|
status, err = m.Health()
|
||||||
if err == nil && status != "OK" {
|
if err == nil && status != "OK" {
|
||||||
err = fmt.Errorf("unhealthy status: %q", status)
|
err = &managedHealthRejection{Status: status}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err == nil {
|
if err == nil {
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
|
|
||||||
const managedMonitorLockName = ".autotune-monitor-lock"
|
const managedMonitorLockName = ".autotune-monitor-lock"
|
||||||
|
|
||||||
// No production adapter or entrypoint yet. Callbacks must honor ctx and join
|
// Callbacks must honor ctx and join
|
||||||
// their children before returning; the supervisor never detaches callbacks.
|
// their children before returning; the supervisor never detaches callbacks.
|
||||||
type managedMonitorDeps struct {
|
type managedMonitorDeps struct {
|
||||||
target managedDeps
|
target managedDeps
|
||||||
@@ -25,6 +25,7 @@ type managedMonitorDeps struct {
|
|||||||
cooldown time.Duration
|
cooldown time.Duration
|
||||||
probe func(context.Context) error
|
probe func(context.Context) error
|
||||||
recover func(context.Context) error
|
recover func(context.Context) error
|
||||||
|
bootstrap func(context.Context) error
|
||||||
now func() time.Time
|
now func() time.Time
|
||||||
wait func(context.Context, time.Duration) error
|
wait func(context.Context, time.Duration) error
|
||||||
reportRecoveryError func(error)
|
reportRecoveryError func(error)
|
||||||
@@ -120,11 +121,20 @@ func runManagedMonitor(ctx context.Context, d managedMonitorDeps) (err error) {
|
|||||||
if err := reconcileManagedMonitorRecovery(d.db, d.serviceID, singleton, d.cooldown, readMonitorProcessStarttime, d.now); err != nil {
|
if err := reconcileManagedMonitorRecovery(d.db, d.serviceID, singleton, d.cooldown, readMonitorProcessStarttime, d.now); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if d.bootstrap != nil {
|
||||||
|
if err := d.bootstrap(ctx); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
for {
|
for {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
probeErr := d.probe(ctx)
|
probeErr := d.probe(ctx)
|
||||||
|
var fatalProbe *managedRecoveryFatal
|
||||||
|
if errors.As(probeErr, &fatalProbe) {
|
||||||
|
return probeErr
|
||||||
|
}
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -144,6 +154,10 @@ func runManagedMonitor(ctx context.Context, d managedMonitorDeps) (err error) {
|
|||||||
}
|
}
|
||||||
if recoveryErr != nil {
|
if recoveryErr != nil {
|
||||||
d.reportRecoveryError(recoveryErr)
|
d.reportRecoveryError(recoveryErr)
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
if errors.As(recoveryErr, &fatal) {
|
||||||
|
return recoveryErr
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,32 +8,78 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Only complete, well-formed non-passing measurements permit scan fallback.
|
// Only complete, well-formed non-passing measurements permit scan fallback.
|
||||||
var errCandidateNonpass = errors.New("saved scenario did not pass both TLS protocols")
|
var errCandidateNonpass = errors.New("saved scenario did not pass both TLS protocols")
|
||||||
|
|
||||||
|
// Rechecks exact saved args using runCombinedBlockcheckForHost's cacheCheck
|
||||||
|
// contract. The caller supplies a fresh temporary artifact path; artifacts are
|
||||||
|
// retained, but no stored definition or evidence is updated. A fresh GGC host
|
||||||
|
// is allowed only as an exact endpoint, never as wildcard qualification.
|
||||||
|
func revalidateCandidate(ctx context.Context, c storedCandidate, group, host, path string, stderr io.Writer, check cacheCheck) ([]byte, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if (group != "frontend" && group != "googlevideo") || c.Group != group || !c.Qualified || path == "" || check == nil {
|
||||||
|
return nil, fmt.Errorf("invalid candidate revalidation request")
|
||||||
|
}
|
||||||
|
definition := candidate{Group: c.Group, Domains: append([]string(nil), c.Endpoints...), Args: c.Args}
|
||||||
|
if err := definition.validate(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !validDomain(host) || host != strings.ToLower(host) {
|
||||||
|
return nil, fmt.Errorf("invalid revalidation endpoint")
|
||||||
|
}
|
||||||
|
if group == "googlevideo" {
|
||||||
|
if !discoveryHostname(host) || !strings.HasSuffix(host, ".googlevideo.com") {
|
||||||
|
return nil, fmt.Errorf("invalid exact cache scope")
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
inScope := false
|
||||||
|
for _, endpoint := range c.Endpoints {
|
||||||
|
inScope = inScope || endpoint == host
|
||||||
|
}
|
||||||
|
if !inScope {
|
||||||
|
return nil, fmt.Errorf("frontend endpoint outside candidate scope")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stderr, "Rechecking saved candidate:", c.ID, "exact endpoint:", host, "Blockcheck artifacts:", path)
|
||||||
|
// A checker that writes nothing must not reuse an earlier measurement.
|
||||||
|
if err := os.Remove(path + ".combined.json"); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var raw bytes.Buffer
|
||||||
|
err := check(ctx, "blockcheckw", path, c.Args, host, &raw, &raw)
|
||||||
|
if e := os.WriteFile(path+".stdout", raw.Bytes(), 0600); e != nil {
|
||||||
|
return nil, e
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
// Only our report parser may issue the nonpass authorization.
|
||||||
|
if errors.Is(err, errCandidateNonpass) {
|
||||||
|
return nil, fmt.Errorf("candidate checker failed: %v", err)
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
report, err := os.ReadFile(path + ".combined.json")
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
_, err = selectBlockcheckEvidence(bytes.NewReader(report), c.Args, group, host, true)
|
||||||
|
if contextErr := ctx.Err(); contextErr != nil {
|
||||||
|
return nil, contextErr
|
||||||
|
}
|
||||||
|
return report, err
|
||||||
|
}
|
||||||
|
|
||||||
func revalidateGooglevideo(ctx context.Context, db *sql.DB, host string, stderr io.Writer, check cacheCheck) (string, []byte, error) {
|
func revalidateGooglevideo(ctx context.Context, db *sql.DB, host string, stderr io.Writer, check cacheCheck) (string, []byte, error) {
|
||||||
rows, err := db.Query(`SELECT id FROM strategy_candidates WHERE strategy_group = 'googlevideo' ORDER BY id DESC`)
|
ids, err := qualifiedCandidateIDs(db, "googlevideo")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, err
|
return "", nil, &managedRecoveryFatal{Err: err}
|
||||||
}
|
|
||||||
var ids []int64
|
|
||||||
for rows.Next() {
|
|
||||||
var id int64
|
|
||||||
if err := rows.Scan(&id); err != nil {
|
|
||||||
rows.Close()
|
|
||||||
return "", nil, err
|
|
||||||
}
|
|
||||||
ids = append(ids, id)
|
|
||||||
}
|
|
||||||
err = rows.Err()
|
|
||||||
closeErr := rows.Close() // Release the store's single connection before candidate reads.
|
|
||||||
if err == nil {
|
|
||||||
err = closeErr
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return "", nil, err
|
|
||||||
}
|
}
|
||||||
for _, id := range ids {
|
for _, id := range ids {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
@@ -41,7 +87,7 @@ func revalidateGooglevideo(ctx context.Context, db *sql.DB, host string, stderr
|
|||||||
}
|
}
|
||||||
c, err := readStrategyCandidate(db, id)
|
c, err := readStrategyCandidate(db, id)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, err
|
return "", nil, &managedRecoveryFatal{Err: err}
|
||||||
}
|
}
|
||||||
if !c.Qualified {
|
if !c.Qualified {
|
||||||
continue
|
continue
|
||||||
@@ -54,23 +100,7 @@ func revalidateGooglevideo(ctx context.Context, db *sql.DB, host string, stderr
|
|||||||
if err := artifact.Close(); err != nil {
|
if err := artifact.Close(); err != nil {
|
||||||
return "", nil, err
|
return "", nil, err
|
||||||
}
|
}
|
||||||
fmt.Fprintln(stderr, "Rechecking saved candidate:", id, "exact endpoint:", host, "Blockcheck artifacts:", path)
|
report, err := revalidateCandidate(ctx, c, "googlevideo", host, path, stderr, check)
|
||||||
var raw bytes.Buffer
|
|
||||||
err = check(ctx, "blockcheckw", path, c.Args, host, &raw, &raw)
|
|
||||||
if e := os.WriteFile(path+".stdout", raw.Bytes(), 0600); e != nil {
|
|
||||||
err = e
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return "", nil, err
|
|
||||||
}
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return "", nil, err
|
|
||||||
}
|
|
||||||
report, err := os.ReadFile(path + ".combined.json")
|
|
||||||
if err != nil {
|
|
||||||
return "", nil, err
|
|
||||||
}
|
|
||||||
_, err = selectBlockcheckEvidence(bytes.NewReader(report), c.Args, "googlevideo", host, true)
|
|
||||||
if errors.Is(err, errCandidateNonpass) {
|
if errors.Is(err, errCandidateNonpass) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,212 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func evidenceSnapshot(t *testing.T, db *sql.DB) string {
|
||||||
|
t.Helper()
|
||||||
|
var snapshot string
|
||||||
|
if err := db.QueryRow(`SELECT group_concat(row, ';') FROM (SELECT
|
||||||
|
quote(candidate_id)||','||quote(endpoint)||','||quote(protocol)||','||quote(passes_ok)||','||quote(passes_total)||','||quote(success_rate)||','||quote(coverage)||','||quote(median_latency_ms)||','||quote(median_speed_kbps)||','||quote(checked_at)||','||quote(raw_measurement) AS row
|
||||||
|
FROM candidate_evidence ORDER BY candidate_id, endpoint, protocol)`).Scan(&snapshot); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return snapshot
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestListQualifiedCandidates(t *testing.T) {
|
||||||
|
db := testStore(t)
|
||||||
|
args := "--lua-desync=multisplit"
|
||||||
|
for i := 0; i < 12; i++ {
|
||||||
|
saveProposal(t, db, args)
|
||||||
|
}
|
||||||
|
if err := saveGooglevideoRun(db, time.Now(), cacheTestHost, args, []byte(combinedReportFixture(args)), nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
tx, err := db.Begin()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := insertStrategyCandidate(tx, "frontend", []string{"youtube.com"}, args, nil); err != nil {
|
||||||
|
tx.Rollback()
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := tx.Commit(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
before := evidenceSnapshot(t, db)
|
||||||
|
for _, limit := range []int{1, 3, 10} {
|
||||||
|
candidates, err := listQualifiedCandidates(db, "frontend", limit)
|
||||||
|
if err != nil || len(candidates) != limit {
|
||||||
|
t.Fatalf("limit=%d candidates=%+v err=%v", limit, candidates, err)
|
||||||
|
}
|
||||||
|
for i, c := range candidates {
|
||||||
|
if c.ID != int64(12-i) || !c.Qualified || c.Group != "frontend" {
|
||||||
|
t.Fatalf("unexpected candidate: %+v", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Returned slices cannot mutate definitions or later snapshots.
|
||||||
|
candidates[0].Endpoints[0] = "changed.example"
|
||||||
|
stored, err := readStrategyCandidate(db, candidates[0].ID)
|
||||||
|
if err != nil || stored.Endpoints[0] != "youtube.com" {
|
||||||
|
t.Fatalf("stored snapshot changed: %+v %v", stored, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, limit := range []int{-1, 0, 11} {
|
||||||
|
if _, err := listQualifiedCandidates(db, "frontend", limit); err == nil {
|
||||||
|
t.Fatalf("accepted limit %d", limit)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := listQualifiedCandidates(db, "unknown", 1); err == nil {
|
||||||
|
t.Fatal("accepted unknown group")
|
||||||
|
}
|
||||||
|
ggc, err := listQualifiedCandidates(db, "googlevideo", 10)
|
||||||
|
if err != nil || len(ggc) != 1 || ggc[0].Group != "googlevideo" {
|
||||||
|
t.Fatalf("googlevideo=%+v err=%v", ggc, err)
|
||||||
|
}
|
||||||
|
if before != evidenceSnapshot(t, db) {
|
||||||
|
t.Fatal("listing mutated evidence")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRevalidateCandidate(t *testing.T) {
|
||||||
|
db := testStore(t)
|
||||||
|
args := "--lua-desync=multisplit"
|
||||||
|
saveProposal(t, db, args)
|
||||||
|
c, err := readStrategyCandidate(db, 1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
original := c
|
||||||
|
original.Endpoints = append([]string(nil), c.Endpoints...)
|
||||||
|
before := evidenceSnapshot(t, db)
|
||||||
|
passing := combinedReportFixture(args)
|
||||||
|
var parsed struct {
|
||||||
|
Strategies []json.RawMessage `json:"strategies"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal([]byte(passing), &parsed); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, mode := range []string{"pass", "one-nonpass", "both-nonpass", "nonpass-malformed", "malformed", "incomplete", "duplicate", "wrong-args", "wrong-host", "wrong-group", "missing-metric", "invalid-metric", "incomplete-passes", "trailing-json", "runner", "runner-sentinel", "missing-file", "stale-file", "filesystem", "cancelled"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "check")
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
report := passing
|
||||||
|
switch mode {
|
||||||
|
case "one-nonpass":
|
||||||
|
report = strings.Replace(report, `"passes_ok":3`, `"passes_ok":2`, 1)
|
||||||
|
case "both-nonpass":
|
||||||
|
report = strings.ReplaceAll(report, `"passes_ok":3`, `"passes_ok":0`)
|
||||||
|
case "nonpass-malformed":
|
||||||
|
report = strings.Replace(report, `"passes_ok":3`, `"passes_ok":2`, 1)
|
||||||
|
i := strings.LastIndex(report, `"coverage":1`)
|
||||||
|
report = report[:i] + strings.Replace(report[i:], `"coverage":1`, `"coverage":null`, 1)
|
||||||
|
case "malformed":
|
||||||
|
report = `{`
|
||||||
|
case "incomplete":
|
||||||
|
report = `{"strategies":[` + string(parsed.Strategies[0]) + `]}`
|
||||||
|
case "duplicate":
|
||||||
|
report = `{"strategies":[` + string(parsed.Strategies[0]) + `,` + string(parsed.Strategies[0]) + `,` + string(parsed.Strategies[1]) + `]}`
|
||||||
|
case "wrong-args":
|
||||||
|
report = combinedReportFixture("--lua-desync=fake")
|
||||||
|
case "wrong-host":
|
||||||
|
report = strings.ReplaceAll(report, `"protocol":`, `"domains":["other.example"],"protocol":`)
|
||||||
|
case "wrong-group":
|
||||||
|
report = strings.ReplaceAll(report, `"protocol":`, `"group":"googlevideo","protocol":`)
|
||||||
|
case "missing-metric":
|
||||||
|
report = strings.Replace(report, `"coverage":1`, `"coverage":null`, 1)
|
||||||
|
case "invalid-metric":
|
||||||
|
report = strings.Replace(report, `"median_speed_kbps":100`, `"median_speed_kbps":-1`, 1)
|
||||||
|
case "incomplete-passes":
|
||||||
|
report = strings.Replace(report, `"passes_total":3`, `"passes_total":4`, 1)
|
||||||
|
case "trailing-json":
|
||||||
|
report += `{}`
|
||||||
|
case "filesystem":
|
||||||
|
path = filepath.Join(path, "missing", "check")
|
||||||
|
case "stale-file":
|
||||||
|
if err := os.WriteFile(path+".combined.json", []byte(passing), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
check := func(_ context.Context, binary, gotPath, gotArgs, host string, _, _ io.Writer) error {
|
||||||
|
if binary != "blockcheckw" || gotPath != path || gotArgs != args || host != "youtube.com" {
|
||||||
|
t.Fatalf("wrong combined check: %s %s %s %s", binary, gotPath, gotArgs, host)
|
||||||
|
}
|
||||||
|
switch mode {
|
||||||
|
case "runner":
|
||||||
|
return fmt.Errorf("runner failed")
|
||||||
|
case "runner-sentinel":
|
||||||
|
return errCandidateNonpass
|
||||||
|
case "missing-file", "stale-file", "filesystem":
|
||||||
|
return nil
|
||||||
|
case "cancelled":
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return os.WriteFile(path+".combined.json", []byte(report), 0600)
|
||||||
|
}
|
||||||
|
got, err := revalidateCandidate(ctx, c, "frontend", "youtube.com", path, io.Discard, check)
|
||||||
|
if mode == "pass" {
|
||||||
|
if err != nil || string(got) != passing {
|
||||||
|
t.Fatalf("passing report=%s err=%v", got, err)
|
||||||
|
}
|
||||||
|
} else if mode == "one-nonpass" || mode == "both-nonpass" {
|
||||||
|
if !errors.Is(err, errCandidateNonpass) {
|
||||||
|
t.Fatalf("expected nonpass, got %v", err)
|
||||||
|
}
|
||||||
|
} else if err == nil || errors.Is(err, errCandidateNonpass) {
|
||||||
|
t.Fatalf("expected infrastructure error, got %v", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(c, original) || before != evidenceSnapshot(t, db) {
|
||||||
|
t.Fatal("revalidation mutated candidate or evidence")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRevalidateCandidateScope(t *testing.T) {
|
||||||
|
db := testStore(t)
|
||||||
|
args := "--lua-desync=multisplit"
|
||||||
|
if err := saveGooglevideoRun(db, time.Now(), "old.googlevideo.com", args, []byte(combinedReportFixture(args)), nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
c, err := readStrategyCandidate(db, 1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, target := range []struct{ group, host string }{
|
||||||
|
{"googlevideo", cacheTestHost}, {"googlevideo", "googlevideo.com"},
|
||||||
|
{"googlevideo", "^" + cacheTestHost}, {"googlevideo", "*.googlevideo.com"},
|
||||||
|
{"googlevideo", "youtube.com"}, {"frontend", "youtube.com"}, {"unknown", cacheTestHost},
|
||||||
|
} {
|
||||||
|
calls := 0
|
||||||
|
check := func(_ context.Context, _, path, gotArgs, host string, _, _ io.Writer) error {
|
||||||
|
calls++
|
||||||
|
if gotArgs != args || host != cacheTestHost {
|
||||||
|
t.Fatal("widened scope or changed args")
|
||||||
|
}
|
||||||
|
return os.WriteFile(path+".combined.json", []byte(combinedReportFixture(args)), 0600)
|
||||||
|
}
|
||||||
|
_, err := revalidateCandidate(context.Background(), c, target.group, target.host, filepath.Join(t.TempDir(), "check"), io.Discard, check)
|
||||||
|
valid := target.group == "googlevideo" && target.host == cacheTestHost
|
||||||
|
if valid && (err != nil || calls != 1) || !valid && (err == nil || calls != 0 || errors.Is(err, errCandidateNonpass)) {
|
||||||
|
t.Fatalf("target=%+v calls=%d err=%v", target, calls, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
stored, err := readStrategyCandidate(db, 1)
|
||||||
|
if err != nil || !reflect.DeepEqual(c, stored) || c.Endpoints[0] != "old.googlevideo.com" {
|
||||||
|
t.Fatalf("changed stored scope: %+v %v", stored, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
# Isolated managed-test definition — deployment deferred
|
# Isolated managed-test definition — deployment deferred
|
||||||
|
|
||||||
This definition is **not autonomous acceptance-ready**: no `managed monitor`
|
This definition is **not autonomous acceptance-ready**: `managed monitor` and
|
||||||
or integrated recovery loop exists. Do not build/pull/start it as part of this
|
entrypoint recovery integration exist in code but are not live verified. Do not build/pull/start it as part of this
|
||||||
change. Existing containers, production and Xray configuration are out of scope.
|
change. Existing containers, production and Xray configuration are out of scope.
|
||||||
|
|
||||||
For a later approved deployment, from this directory:
|
For a later approved deployment, from this directory:
|
||||||
@@ -61,10 +61,11 @@ or host filesystem restrictions.
|
|||||||
Go builder matches `Dockerfile`; runtime digest is pinned and image-owned
|
Go builder matches `Dockerfile`; runtime digest is pinned and image-owned
|
||||||
Lua/fakes retained. Startup: ss-server/ss-local → internal listener 1080
|
Lua/fakes retained. Startup: ss-server/ss-local → internal listener 1080
|
||||||
(30-second bound) → guarded managed init on fresh state (status on recreation)
|
(30-second bound) → guarded managed init on fresh state (status on recreation)
|
||||||
→ zapret2 start using target config. Drift/pending state refuses startup;
|
→ zapret2 start using target config → one installed managed monitor child. Drift/pending state refuses startup;
|
||||||
no automatic adoption/reconciliation. SIGTERM/SIGINT stop only target-local
|
no automatic adoption of unknown state. SIGTERM/SIGINT cancels and waits the monitor
|
||||||
zapret firewall and the entrypoint's own Shadowsocks children. Process
|
before stopping target-local zapret and the entrypoint's Shadowsocks children.
|
||||||
supervision is **not** health/recovery; no restart policy hides fail-closed exits.
|
Monitor fatal exits stop the entrypoint; no restart policy hides fail-closed exits.
|
||||||
|
Monitor probes Player API via owned loopback SOCKS; unresolved GGC is not playback health.
|
||||||
|
|
||||||
Inside target, `zapret-autotune health youtube` uses
|
Inside target, `zapret-autotune health youtube` uses
|
||||||
`socks5://127.0.0.1:1080`, never the host port. It is one-shot Player API health,
|
`socks5://127.0.0.1:1080`, never the host port. It is one-shot Player API health,
|
||||||
@@ -79,5 +80,7 @@ the `tpws` user drop). Prove target-local routing,
|
|||||||
NFQUEUE activation/rollback, signal cleanup and persistent recreation on the
|
NFQUEUE activation/rollback, signal cleanup and persistent recreation on the
|
||||||
new target only. Compose syntax, build and runtime integration are unverified
|
new target only. Compose syntax, build and runtime integration are unverified
|
||||||
by this producer. Do not widen capabilities or reuse existing config/network
|
by this producer. Do not widen capabilities or reuse existing config/network
|
||||||
as a workaround. Autonomous acceptance also needs monitor/recovery integration
|
as a workaround. Autonomous acceptance also needs tests/build and live verification
|
||||||
and crash reconciliation.
|
of monitor/recovery and crash reconciliation, including shutdown grace sufficient
|
||||||
|
for bounded restoration (which may exceed the defined 45s). Docker configuration
|
||||||
|
has not been changed by this code integration increment.
|
||||||
@@ -27,9 +27,15 @@ unset ss_password
|
|||||||
|
|
||||||
server_pid=
|
server_pid=
|
||||||
local_pid=
|
local_pid=
|
||||||
|
monitor_pid=
|
||||||
zapret_started=0
|
zapret_started=0
|
||||||
cleanup() {
|
cleanup() {
|
||||||
trap - EXIT INT TERM
|
trap - EXIT INT TERM
|
||||||
|
# Let the monitor cancel/join scans and restore committed bytes BEFORE stop.
|
||||||
|
if [ -n "$monitor_pid" ]; then
|
||||||
|
kill -TERM "$monitor_pid" 2>/dev/null || true
|
||||||
|
wait "$monitor_pid" 2>/dev/null || true
|
||||||
|
fi
|
||||||
# Dedicated PID/network namespace: no host/other-target stop.
|
# Dedicated PID/network namespace: no host/other-target stop.
|
||||||
if [ "$zapret_started" -eq 1 ]; then
|
if [ "$zapret_started" -eq 1 ]; then
|
||||||
/opt/zapret2/init.d/sysv/zapret2 stop || true
|
/opt/zapret2/init.d/sysv/zapret2 stop || true
|
||||||
@@ -59,17 +65,26 @@ until awk '$2 == "00000000:0438" && $4 == "0A" { found=1 } END { exit !found }'
|
|||||||
sleep 1
|
sleep 1
|
||||||
done
|
done
|
||||||
if [ ! -e "$ZAPRET_CONFIG" ]; then
|
if [ ! -e "$ZAPRET_CONFIG" ]; then
|
||||||
zapret-autotune managed init
|
/usr/local/bin/zapret-autotune managed init
|
||||||
else
|
else
|
||||||
# Never reinitialize/adopt persisted state; drift/pending state fails closed.
|
# Never reinitialize/adopt persisted state; drift/pending state fails closed.
|
||||||
zapret-autotune managed status
|
/usr/local/bin/zapret-autotune managed status
|
||||||
fi
|
fi
|
||||||
zapret_started=1
|
zapret_started=1
|
||||||
/opt/zapret2/init.d/sysv/zapret2 start
|
/opt/zapret2/init.d/sysv/zapret2 start
|
||||||
|
|
||||||
# Process supervision only; no managed monitor/recovery command exists yet.
|
/usr/local/bin/zapret-autotune managed monitor &
|
||||||
while kill -0 "$server_pid" && kill -0 "$local_pid"; do
|
monitor_pid=$!
|
||||||
|
while kill -0 "$server_pid" && kill -0 "$local_pid" && kill -0 "$monitor_pid"; do
|
||||||
sleep 1
|
sleep 1
|
||||||
done
|
done
|
||||||
echo 'error: target Shadowsocks process exited' >&2
|
if ! kill -0 "$monitor_pid" 2>/dev/null; then
|
||||||
|
code=0
|
||||||
|
wait "$monitor_pid" || code=$?
|
||||||
|
monitor_pid=
|
||||||
|
echo 'error: target managed monitor exited' >&2
|
||||||
|
[ "$code" -ne 0 ] || code=1
|
||||||
|
exit "$code"
|
||||||
|
fi
|
||||||
|
echo 'error: target child process exited' >&2
|
||||||
exit 1
|
exit 1
|
||||||
Reference in new issue
Block a user