feat(runtime): bound recovery and add safe release supervision

Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
ZeusinaandGPT-6 Bot committed 2026-10-10 22:51:58 +03:00
1 parent 780ca71e17
commit b1e458a0a5
16 files changed
+941 -66

No files matched your search

+156 -4
View File
@@ -135,7 +135,7 @@ identity/SOCKS guards. Fake-only checks are added, but tests/build are deferred
to the orchestrator. `managed monitor` now binds those callbacks and only the
dedicated target entrypoint starts it; no live/runtime validation has occurred.
## Dedicated monitor integration (verification deferred)
## Dedicated monitor integration (live acceptance pending)
`zapret-autotune managed monitor [--interval 5m] [--failure-threshold 3]
[--cooldown 30m]` uses the health environment defaults. Interval is positive and
@@ -165,9 +165,9 @@ Entrypoint ordering is SS children → listener → managed init/status → zapr
all three children; monitor exit is fatal even if it exits zero unexpectedly.
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
SS child termination/wait. Existing targets and Xray remain unchanged. Deployment
must allow enough shutdown grace for independently bounded rollback/restoration
(potentially more than the currently defined 45s); this increment changes no Docker
configuration. Code integration is not image/routing/signal live verification.
allows 120s Compose shutdown grace and 180s systemd stop timeout for bounded
monitor cancellation/restoration and child joins. Code integration is not
image/routing/signal live verification.
Injected unit tests cover exact identity/path/config, mountpoint refusal,
old-container-like env, SOCKS port/process refusal, correct fresh-target success,
@@ -178,3 +178,155 @@ deferred to the orchestrator; not run by this increment:
go test ./cmd/zapret-autotune -run TestManaged
go build ./cmd/zapret-autotune
```
## Production preparation
This is a standalone **new isolated target** candidate, not a production-approved
deployment and not an integration into existing zapret/Xray. Keep the exact
`managed-test` role, ID, marker, mount, config, DB and owned-listener guards above.
Do not change the role to `production`, adopt old state, share an old network
namespace, or relax a refusal. Provisioning and optional service installation
are described in [deploy/managed-target/README.md](deploy/managed-target/README.md).
### Build once, promote the same image
Use `Dockerfile.target` for the supervised target; root `Dockerfile` is the
proposal CLI image, not the managed service. Both pin Go 1.25.3 and the runtime
by digest, use `GOTOOLCHAIN=local`, `CGO_ENABLED=0`, `go.sum`, and `-trimpath`.
Pins constrain build inputs; they do not establish live acceptance or guarantee
bit-for-bit builds on different platforms. Record the source commit, platform,
builder/runtime digests and resulting image digest for every release.
Example (replace registry and release; never reuse a release tag):
```sh
IMAGE=registry.example/team/zapret-autotune-managed-target:2026-10-10-r1
docker build --pull -f Dockerfile.target -t "$IMAGE" .
docker push "$IMAGE"
docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE"
```
Promote the printed `registry/repository@sha256:...` reference unchanged from
validation to deployment; do not rebuild or use `latest`. Authenticate separately,
never put registry credentials in these commands. The supplied Compose file
selects `${ZAPRET_AUTOTUNE_IMAGE:-zapret-autotune-managed-target:local}`. Set
`ZAPRET_AUTOTUNE_IMAGE=registry.example/team/zapret-autotune-managed-target@sha256:<digest>`
in the same private deployment `.env` as `SS_PASSWORD` (mode `0600`), using the
actual printed digest. Tags and the local default are build/test candidates only;
an immutable managed-target digest is required for production promotion. From
the repository root, validate and pull without printing the rendered secrets:
```sh
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml config --quiet
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml pull target
```
The supplied systemd unit uses this same `.env` and starts with `--no-build`;
no image-selection override or unit edit is needed. Keep the selected digest and
private configuration together for upgrades and rollback. Never run a build
against the production digest selector; build/tag separately as shown above.
### Configuration and secrets
Create fresh empty `target-state` owned by root/runtime UID 0, mode `0700`, and
the private deployment `.env` mode `0600`; rootless/remapped Docker needs the
mapped owner. Bootstrap is for empty provisioned state only; recreation retains
identity, journal, candidate DB, sidecars and lock file. Never automatically wipe
state to make startup succeed. The entrypoint creates Shadowsocks JSON in private
tmpfs (`0700` directory, `0600` files), unsets the password before children, and
does not put it in argv. Docker administrators/root can still read container env
and runtime config. Do not print `.env`, rendered Compose config, diagnostics or
signed URLs; use `docker compose ... config --quiet` for validation.
| Existing setting | Default / accepted value for managed target |
| --- | --- |
| `ZAPRET_AUTOTUNE_IMAGE` | Local build/test default `zapret-autotune-managed-target:local`; production promotion requires a managed-target `repository@sha256:<digest>` |
| `SS_PASSWORD` | Required independent hex-only secret, at least 32 characters; deployment README generates 64 |
| `ZAPRET_AUTOTUNE_HOST_SOCKS_PORT` | `11080`; choose an available TCP/UDP port 1–65535, published only on `127.0.0.1` |
| `SOCKS_PORT` | Supplied deployment fixes internal listener to `1080`; not a host-port override |
| `ZAPRET_AUTOTUNE_SOCKS_PROXY` | Fixed `socks5://127.0.0.1:1080` inside this target, no credentials/direct fallback |
| `ZAPRET_AUTOTUNE_VIDEO_ID` | `dQw4w9WgXcQ`; exactly 11 ASCII letters, digits, `_` or `-` |
| `ZAPRET_AUTOTUNE_HEALTH_INTERVAL` | `5m`; duration in `(0,24h]` for monitor |
| `ZAPRET_AUTOTUNE_FAILURE_THRESHOLD` | `3`; integer 1–1000 for monitor |
| `ZAPRET_AUTOTUNE_RECOVERY_COOLDOWN` | `30m`; duration in `(0,168h]` for monitor |
| `ZAPRET_AUTOTUNE_DB` | Fixed `/var/lib/zapret-autotune/target/candidates.db` |
The monitor's three policy flags override corresponding environment defaults;
explicit empty/invalid environment settings are rejected. The entrypoint accepts
no arguments. The supplied Compose environment does not forward the video/policy
settings from `.env`: add the desired existing variables to `target.environment`
in your deployment copy/override if needed. Keep all fixed identity/path settings
unchanged. No public SS listener is published; Xray configuration is external and
operator-owned, optionally using the host's loopback SOCKS port. Do not modify an
existing Xray deployment as part of bootstrap.
### Supervision and failure handling
Compose restart is disabled. Optional native systemd runs **foreground** Compose
with `--no-build --abort-on-container-exit --exit-code-from target`; do not also
run detached Compose for that project. The unit requests restart only for exit
75 via `Restart=no` plus `RestartForceExitStatus=75`, bounded by
`StartLimitBurst=3` in six hours (starts, including the initial
start, not three additional retries). Compose stop grace is 120s, systemd 180s.
| Target exit | Operator meaning |
| --- | --- |
| `64` | Invalid configuration; fix settings before starting |
| `78` | Safety/DB/journal/rollback/cleanup failure; investigate, no automatic restart |
| `75` | Technical failure eligible for bounded restart only after safe cleanup |
| `0` | Graceful signal shutdown; no automatic restart |
Bootstrap/recovery has a fixed **90-minute whole-job deadline**, plus an
independent **30-second restoration reserve**. This is a catastrophic technical
ceiling, not a normal candidate-check timeout and not an operator tunable. Valid
nonpassing candidates/unresolved GGC are normal recovery outcomes governed by
the persisted threshold/cooldown, not a reason to restart the container. Failed
restoration or unsafe state vetoes retry: timeout exit 75 requires the job's own
deadline and only timeout/cancellation-related errors, with no joined cleanup,
DB or safety failure. Unexpected monitor/SS failures are separately technical
only if shutdown cleanup succeeds. Shutdown joins the monitor for up to 60s,
target-local zapret stop gets 30s then forced kill after five more, and SS children
get 5s each; failure to join/restore is fail-closed. The reserve is not extra
scanning time. The 90-minute budget is shared by the whole bootstrap/recovery
job, not renewed per candidate or phase.
Monitor health means Player API `OK` through owned SOCKS. GGC evidence covers
only the **exact discovered hostname** and tested TLS protocols, not wildcard
googlevideo coverage, video transfer, SABR or playback. Unresolved GGC must not
be reported as complete playback health.
### Upgrade, rollback and operator recovery
1. Record the current image digest and private deployment configuration. Stop
the service (or foreground Compose), wait for complete cleanup, and verify
the target is stopped. If cleanup fails, investigate before upgrading.
2. With the target stopped, make a root-only backup of the **entire** state
directory (including DB sidecars, journal, config, identity and lock file),
preserving ownership/modes. Store `.env` and image/config records privately
with it; do not print archive contents, diagnostics or signed URLs. A live
copy of only `candidates.db` is not a consistent rollback backup.
3. Select/pull the new immutable digest and start without building. Inspect
sanitized status/logs, verify identity/hash and expected active groups, owned
SOCKS, target-local queues, health scope and persistent recreation. Keep the
old image and backup until acceptance; DB migrations may prevent image-only
downgrade.
4. To roll back, stop and verify cleanup again. Preserve failed state separately
for investigation. Restore the matched stopped-state backup **as a whole**
into the same dedicated mount with private ownership/modes, select its old
digest/config, then start. Do not merge journals/DB generations or re-init.
A restored legacy/live/unverifiable recovery owner can fail closed; never
clear its lease or unlink the lock merely to force startup.
On drift, pending transaction, rollback failure, or exhausted systemd start
limit, leave state intact and service stopped. Diagnose using sanitized status
and private artifacts; repair the actual provisioning/configuration issue or
restore a known consistent backup. Only then manually start (and reset the
systemd failed/start-limit state if needed). There is no automatic wipe or
automatic adoption path.
Rollout acceptance remains pending real isolated-target restart/deadline tests:
empty bootstrap, saved-first recovery, exact-host GGC rotation, image binding,
NFQUEUE activation/rollback, SIGTERM during downtime, deadline restoration,
retry limits/exit propagation through foreground Compose/systemd, persistent
recreation and backup restore. Perform these on the new target only; local unit
or static checks cannot establish production readiness.