fix(runtime): keep supervision and absence retries in container

Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
ZeusinaandGPT-6 Bot committed 2026-10-11 09:01:52 +03:00
1 parent 226d93e2aa
commit 3f9fd2f36f
21 files changed
+1056 -137

No files matched your search

+47 -23
View File
@@ -162,10 +162,10 @@ not rescanned merely because the process restarted.
Entrypoint ordering is SS children → listener → fresh managed init if needed → managed reconcile → zapret start
→ one installed `/usr/local/bin/zapret-autotune managed monitor` child. It supervises
all three children; monitor exit is fatal even if it exits zero unexpectedly.
all three children; unexpected monitor exit is not a successful shutdown.
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
SS child termination/wait. Existing targets and Xray remain unchanged. Deployment
allows 120s Compose shutdown grace and 180s systemd stop timeout for bounded
allows 240s Compose shutdown grace for bounded
monitor cancellation/restoration and child joins. Code integration is not
image/routing/signal live verification.
@@ -185,7 +185,7 @@ This is a standalone **new isolated target** candidate, not a production-approve
deployment and not an integration into existing zapret/Xray. Keep the exact
`managed-test` role, ID, marker, mount, config, DB and owned-listener guards above.
Do not change the role to `production`, adopt old state, share an old network
namespace, or relax a refusal. Provisioning and optional service installation
namespace, or relax a refusal. Container provisioning instructions
are described in [deploy/managed-target/README.md](deploy/managed-target/README.md).
### Build once, promote the same image
@@ -227,8 +227,8 @@ docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/co
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml pull target
```
The supplied systemd unit uses this same `.env` and starts with `--no-build`;
no image-selection override or unit edit is needed. Keep the selected digest and
Start the selected container image with Compose `up --no-build`;
no host service installation is required. Keep the selected digest and
private configuration together for upgrades and rollback. Never run a build
against the production digest selector; build/tag separately as shown above.
@@ -268,31 +268,55 @@ existing Xray deployment as part of bootstrap.
### Supervision and failure handling
Compose restart is disabled. Optional native systemd runs **foreground** Compose
with `--no-build --abort-on-container-exit --exit-code-from target`; do not also
run detached Compose for that project. The unit requests restart only for exit
75 via `Restart=no` plus `RestartForceExitStatus=75`, bounded by
`StartLimitBurst=3` in six hours (starts, including the initial
start, not three additional retries). Compose stop grace is 120s, systemd 180s.
The container entrypoint is PID 1 and owns all technical child restart control.
After a retryable monitor/SS failure it joins children, reconciles trusted state
and stops target-local zapret before starting a new runtime cycle in the **same
container**. Retries use cancellable backoff (15s, 30s, capped at 60s) and at most
three runtime starts per rolling six hours, including the initial start. Exhaustion
stops fail-closed with 78; configuration, safety, DB, journal, rollback or cleanup
errors never authorize retry. No new dependency, host service installation or
external process manager is required. SIGTERM/SIGINT cancels backoff or the running
monitor, waits for bounded restoration and joins children before exit.
Compose restart remains disabled: a blanket container restart could loop over a
safety refusal or reset the in-container retry budget. Compose stop grace is 240s:
the sequential cleanup budget is 140s (45s monitor join, 45s reconciliation plus
5s kill grace, 30s zapret stop plus 5s kill grace, and two 5s SS joins), with room
for a deferred 45s startup command plus kill grace and scheduling overhead.
The join watchdog runs in a fresh shell and is cancelled when its child finishes;
graceful shutdown does not wait out the watchdog's full deadline.
Container-platform lifecycle, creation/recreation and host-boot policy belong to
the deployment operator, not an external manager supplied by this project.
| Target exit | Operator meaning |
| --- | --- |
| `64` | Invalid configuration; fix settings before starting |
| `78` | Safety/DB/journal/rollback/cleanup failure; investigate, no automatic restart |
| `75` | Technical failure eligible for bounded restart only after safe cleanup |
| `75` | Internal technical-cycle result; PID 1 retries only after safe cleanup, within its budget |
| `0` | Graceful signal shutdown; no automatic restart |
Bootstrap/recovery has a fixed **90-minute whole-job deadline**, plus an
independent **30-second restoration reserve**. This is a catastrophic technical
ceiling, not a normal candidate-check timeout and not an operator tunable. Valid
nonpassing candidates/unresolved GGC are normal recovery outcomes governed by
the persisted threshold/cooldown, not a reason to restart the container. Failed
nonpassing candidates/unresolved GGC are normal recovery outcomes, not a reason
to restart the container. Genuine **no qualified strategy** from validated scan/check
evidence persists a recovery wait of at least **two hours** in SQLite, including
empty bootstrap and frontend/GGC exhaustion. The same container stays alive and
continues light periodic Player API probes; it does not scan or apply during that
wait. Startup respects the durable expiry, and recovery resumes after expiry under
the normal failure-threshold policy. In both fresh bootstrap and periodic recovery,
the delay is the greater of two hours and the configured recovery cooldown;
a longer configured cooldown is not shortened.
Other ordinary recovery outcomes retain the configured cooldown (default **30m**).
Malformed/incomplete reports, runner errors and unsafe state are not strategy
absence and must not enter this wait. The two-hour minimum absence policy is not a
new operator setting, technical retry backoff, or the **90-minute watchdog**. Failed
restoration or unsafe state vetoes retry: timeout exit 75 requires the job's own
deadline and only timeout/cancellation-related errors, with no joined cleanup,
DB or safety failure. Unexpected monitor/SS failures are separately technical
only if shutdown cleanup succeeds. Shutdown joins the monitor for up to 60s,
target-local zapret stop gets 30s then forced kill after five more, and SS children
get 5s each; failure to join/restore is fail-closed. The reserve is not extra
only if shutdown cleanup succeeds. Shutdown uses bounded monitor join,
guarded reconciliation, target-local zapret stop and SS child joins;
failure to join/restore is fail-closed. The reserve is not extra
scanning time. The 90-minute budget is shared by the whole bootstrap/recovery
job, not renewed per candidate or phase.
@@ -304,7 +328,7 @@ be reported as complete playback health.
### Upgrade, rollback and operator recovery
1. Record the current image digest and private deployment configuration. Stop
the service (or foreground Compose), wait for complete cleanup, and verify
the target with Compose, wait for complete cleanup, and verify
the target is stopped. If cleanup fails, investigate before upgrading.
2. With the target stopped, make a root-only backup of the **entire** state
directory (including DB sidecars, journal, config, identity and lock file),
@@ -329,16 +353,16 @@ is rolled back to qualified committed bytes before any reload/start; fresh
baseline and known committed state are idempotent. Unknown/legacy state, drift,
missing qualification or failed rollback blocks startup with a non-retry exit.
On drift, unreconcilable pending transaction, rollback failure, or exhausted systemd start
limit, leave state intact and service stopped. Diagnose using sanitized status
On drift, unreconcilable pending transaction, rollback failure, or exhausted PID 1 retry
budget, leave state intact and the container stopped. Diagnose using sanitized status
and private artifacts; repair the actual provisioning/configuration issue or
restore a known consistent backup. Only then manually start (and reset the
systemd failed/start-limit state if needed). There is no automatic wipe or
restore a known consistent backup. Only then manually start the target with Compose.
There is no automatic wipe or
automatic adoption path.
Rollout acceptance remains pending real isolated-target restart/deadline tests:
empty bootstrap, saved-first recovery, exact-host GGC rotation, image binding,
NFQUEUE activation/rollback, SIGTERM during downtime, deadline restoration,
retry limits/exit propagation through foreground Compose/systemd, persistent
in-container retry limits/backoff, durable two-hour absence wait and resumption, persistent
recreation and backup restore. Perform these on the new target only; local unit
or static checks cannot establish production readiness.