feat(runtime): bound recovery and add safe release supervision
Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
780ca71e17
commit
b1e458a0a5
16 files changed
+941
-66
No files matched your search
+156
-4
@@ -135,7 +135,7 @@ identity/SOCKS guards. Fake-only checks are added, but tests/build are deferred
|
||||
to the orchestrator. `managed monitor` now binds those callbacks and only the
|
||||
dedicated target entrypoint starts it; no live/runtime validation has occurred.
|
||||
|
||||
## Dedicated monitor integration (verification deferred)
|
||||
## Dedicated monitor integration (live acceptance pending)
|
||||
|
||||
`zapret-autotune managed monitor [--interval 5m] [--failure-threshold 3]
|
||||
[--cooldown 30m]` uses the health environment defaults. Interval is positive and
|
||||
@@ -165,9 +165,9 @@ Entrypoint ordering is SS children → listener → managed init/status → zapr
|
||||
all three children; monitor exit is fatal even if it exits zero unexpectedly.
|
||||
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
|
||||
SS child termination/wait. Existing targets and Xray remain unchanged. Deployment
|
||||
must allow enough shutdown grace for independently bounded rollback/restoration
|
||||
(potentially more than the currently defined 45s); this increment changes no Docker
|
||||
configuration. Code integration is not image/routing/signal live verification.
|
||||
allows 120s Compose shutdown grace and 180s systemd stop timeout for bounded
|
||||
monitor cancellation/restoration and child joins. Code integration is not
|
||||
image/routing/signal live verification.
|
||||
|
||||
Injected unit tests cover exact identity/path/config, mountpoint refusal,
|
||||
old-container-like env, SOCKS port/process refusal, correct fresh-target success,
|
||||
@@ -178,3 +178,155 @@ deferred to the orchestrator; not run by this increment:
|
||||
go test ./cmd/zapret-autotune -run TestManaged
|
||||
go build ./cmd/zapret-autotune
|
||||
```
|
||||
|
||||
## Production preparation
|
||||
|
||||
This is a standalone **new isolated target** candidate, not a production-approved
|
||||
deployment and not an integration into existing zapret/Xray. Keep the exact
|
||||
`managed-test` role, ID, marker, mount, config, DB and owned-listener guards above.
|
||||
Do not change the role to `production`, adopt old state, share an old network
|
||||
namespace, or relax a refusal. Provisioning and optional service installation
|
||||
are described in [deploy/managed-target/README.md](deploy/managed-target/README.md).
|
||||
|
||||
### Build once, promote the same image
|
||||
|
||||
Use `Dockerfile.target` for the supervised target; root `Dockerfile` is the
|
||||
proposal CLI image, not the managed service. Both pin Go 1.25.3 and the runtime
|
||||
by digest, use `GOTOOLCHAIN=local`, `CGO_ENABLED=0`, `go.sum`, and `-trimpath`.
|
||||
Pins constrain build inputs; they do not establish live acceptance or guarantee
|
||||
bit-for-bit builds on different platforms. Record the source commit, platform,
|
||||
builder/runtime digests and resulting image digest for every release.
|
||||
|
||||
Example (replace registry and release; never reuse a release tag):
|
||||
|
||||
```sh
|
||||
IMAGE=registry.example/team/zapret-autotune-managed-target:2026-10-10-r1
|
||||
docker build --pull -f Dockerfile.target -t "$IMAGE" .
|
||||
docker push "$IMAGE"
|
||||
docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE"
|
||||
```
|
||||
|
||||
Promote the printed `registry/repository@sha256:...` reference unchanged from
|
||||
validation to deployment; do not rebuild or use `latest`. Authenticate separately,
|
||||
never put registry credentials in these commands. The supplied Compose file
|
||||
selects `${ZAPRET_AUTOTUNE_IMAGE:-zapret-autotune-managed-target:local}`. Set
|
||||
`ZAPRET_AUTOTUNE_IMAGE=registry.example/team/zapret-autotune-managed-target@sha256:<digest>`
|
||||
in the same private deployment `.env` as `SS_PASSWORD` (mode `0600`), using the
|
||||
actual printed digest. Tags and the local default are build/test candidates only;
|
||||
an immutable managed-target digest is required for production promotion. From
|
||||
the repository root, validate and pull without printing the rendered secrets:
|
||||
|
||||
```sh
|
||||
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml config --quiet
|
||||
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml pull target
|
||||
```
|
||||
|
||||
The supplied systemd unit uses this same `.env` and starts with `--no-build`;
|
||||
no image-selection override or unit edit is needed. Keep the selected digest and
|
||||
private configuration together for upgrades and rollback. Never run a build
|
||||
against the production digest selector; build/tag separately as shown above.
|
||||
|
||||
### Configuration and secrets
|
||||
|
||||
Create fresh empty `target-state` owned by root/runtime UID 0, mode `0700`, and
|
||||
the private deployment `.env` mode `0600`; rootless/remapped Docker needs the
|
||||
mapped owner. Bootstrap is for empty provisioned state only; recreation retains
|
||||
identity, journal, candidate DB, sidecars and lock file. Never automatically wipe
|
||||
state to make startup succeed. The entrypoint creates Shadowsocks JSON in private
|
||||
tmpfs (`0700` directory, `0600` files), unsets the password before children, and
|
||||
does not put it in argv. Docker administrators/root can still read container env
|
||||
and runtime config. Do not print `.env`, rendered Compose config, diagnostics or
|
||||
signed URLs; use `docker compose ... config --quiet` for validation.
|
||||
|
||||
| Existing setting | Default / accepted value for managed target |
|
||||
| --- | --- |
|
||||
| `ZAPRET_AUTOTUNE_IMAGE` | Local build/test default `zapret-autotune-managed-target:local`; production promotion requires a managed-target `repository@sha256:<digest>` |
|
||||
| `SS_PASSWORD` | Required independent hex-only secret, at least 32 characters; deployment README generates 64 |
|
||||
| `ZAPRET_AUTOTUNE_HOST_SOCKS_PORT` | `11080`; choose an available TCP/UDP port 1–65535, published only on `127.0.0.1` |
|
||||
| `SOCKS_PORT` | Supplied deployment fixes internal listener to `1080`; not a host-port override |
|
||||
| `ZAPRET_AUTOTUNE_SOCKS_PROXY` | Fixed `socks5://127.0.0.1:1080` inside this target, no credentials/direct fallback |
|
||||
| `ZAPRET_AUTOTUNE_VIDEO_ID` | `dQw4w9WgXcQ`; exactly 11 ASCII letters, digits, `_` or `-` |
|
||||
| `ZAPRET_AUTOTUNE_HEALTH_INTERVAL` | `5m`; duration in `(0,24h]` for monitor |
|
||||
| `ZAPRET_AUTOTUNE_FAILURE_THRESHOLD` | `3`; integer 1–1000 for monitor |
|
||||
| `ZAPRET_AUTOTUNE_RECOVERY_COOLDOWN` | `30m`; duration in `(0,168h]` for monitor |
|
||||
| `ZAPRET_AUTOTUNE_DB` | Fixed `/var/lib/zapret-autotune/target/candidates.db` |
|
||||
|
||||
The monitor's three policy flags override corresponding environment defaults;
|
||||
explicit empty/invalid environment settings are rejected. The entrypoint accepts
|
||||
no arguments. The supplied Compose environment does not forward the video/policy
|
||||
settings from `.env`: add the desired existing variables to `target.environment`
|
||||
in your deployment copy/override if needed. Keep all fixed identity/path settings
|
||||
unchanged. No public SS listener is published; Xray configuration is external and
|
||||
operator-owned, optionally using the host's loopback SOCKS port. Do not modify an
|
||||
existing Xray deployment as part of bootstrap.
|
||||
|
||||
### Supervision and failure handling
|
||||
|
||||
Compose restart is disabled. Optional native systemd runs **foreground** Compose
|
||||
with `--no-build --abort-on-container-exit --exit-code-from target`; do not also
|
||||
run detached Compose for that project. The unit requests restart only for exit
|
||||
75 via `Restart=no` plus `RestartForceExitStatus=75`, bounded by
|
||||
`StartLimitBurst=3` in six hours (starts, including the initial
|
||||
start, not three additional retries). Compose stop grace is 120s, systemd 180s.
|
||||
|
||||
| Target exit | Operator meaning |
|
||||
| --- | --- |
|
||||
| `64` | Invalid configuration; fix settings before starting |
|
||||
| `78` | Safety/DB/journal/rollback/cleanup failure; investigate, no automatic restart |
|
||||
| `75` | Technical failure eligible for bounded restart only after safe cleanup |
|
||||
| `0` | Graceful signal shutdown; no automatic restart |
|
||||
|
||||
Bootstrap/recovery has a fixed **90-minute whole-job deadline**, plus an
|
||||
independent **30-second restoration reserve**. This is a catastrophic technical
|
||||
ceiling, not a normal candidate-check timeout and not an operator tunable. Valid
|
||||
nonpassing candidates/unresolved GGC are normal recovery outcomes governed by
|
||||
the persisted threshold/cooldown, not a reason to restart the container. Failed
|
||||
restoration or unsafe state vetoes retry: timeout exit 75 requires the job's own
|
||||
deadline and only timeout/cancellation-related errors, with no joined cleanup,
|
||||
DB or safety failure. Unexpected monitor/SS failures are separately technical
|
||||
only if shutdown cleanup succeeds. Shutdown joins the monitor for up to 60s,
|
||||
target-local zapret stop gets 30s then forced kill after five more, and SS children
|
||||
get 5s each; failure to join/restore is fail-closed. The reserve is not extra
|
||||
scanning time. The 90-minute budget is shared by the whole bootstrap/recovery
|
||||
job, not renewed per candidate or phase.
|
||||
|
||||
Monitor health means Player API `OK` through owned SOCKS. GGC evidence covers
|
||||
only the **exact discovered hostname** and tested TLS protocols, not wildcard
|
||||
googlevideo coverage, video transfer, SABR or playback. Unresolved GGC must not
|
||||
be reported as complete playback health.
|
||||
|
||||
### Upgrade, rollback and operator recovery
|
||||
|
||||
1. Record the current image digest and private deployment configuration. Stop
|
||||
the service (or foreground Compose), wait for complete cleanup, and verify
|
||||
the target is stopped. If cleanup fails, investigate before upgrading.
|
||||
2. With the target stopped, make a root-only backup of the **entire** state
|
||||
directory (including DB sidecars, journal, config, identity and lock file),
|
||||
preserving ownership/modes. Store `.env` and image/config records privately
|
||||
with it; do not print archive contents, diagnostics or signed URLs. A live
|
||||
copy of only `candidates.db` is not a consistent rollback backup.
|
||||
3. Select/pull the new immutable digest and start without building. Inspect
|
||||
sanitized status/logs, verify identity/hash and expected active groups, owned
|
||||
SOCKS, target-local queues, health scope and persistent recreation. Keep the
|
||||
old image and backup until acceptance; DB migrations may prevent image-only
|
||||
downgrade.
|
||||
4. To roll back, stop and verify cleanup again. Preserve failed state separately
|
||||
for investigation. Restore the matched stopped-state backup **as a whole**
|
||||
into the same dedicated mount with private ownership/modes, select its old
|
||||
digest/config, then start. Do not merge journals/DB generations or re-init.
|
||||
A restored legacy/live/unverifiable recovery owner can fail closed; never
|
||||
clear its lease or unlink the lock merely to force startup.
|
||||
|
||||
On drift, pending transaction, rollback failure, or exhausted systemd start
|
||||
limit, leave state intact and service stopped. Diagnose using sanitized status
|
||||
and private artifacts; repair the actual provisioning/configuration issue or
|
||||
restore a known consistent backup. Only then manually start (and reset the
|
||||
systemd failed/start-limit state if needed). There is no automatic wipe or
|
||||
automatic adoption path.
|
||||
|
||||
Rollout acceptance remains pending real isolated-target restart/deadline tests:
|
||||
empty bootstrap, saved-first recovery, exact-host GGC rotation, image binding,
|
||||
NFQUEUE activation/rollback, SIGTERM during downtime, deadline restoration,
|
||||
retry limits/exit propagation through foreground Compose/systemd, persistent
|
||||
recreation and backup restore. Perform these on the new target only; local unit
|
||||
or static checks cannot establish production readiness.
|
||||
Reference in new issue
Block a user