fix(runtime): keep supervision and absence retries in container
Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
226d93e2aa
commit
3f9fd2f36f
21 files changed
+1056
-137
No files matched your search
+47
-23
@@ -162,10 +162,10 @@ not rescanned merely because the process restarted.
|
||||
|
||||
Entrypoint ordering is SS children → listener → fresh managed init if needed → managed reconcile → zapret start
|
||||
→ one installed `/usr/local/bin/zapret-autotune managed monitor` child. It supervises
|
||||
all three children; monitor exit is fatal even if it exits zero unexpectedly.
|
||||
all three children; unexpected monitor exit is not a successful shutdown.
|
||||
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
|
||||
SS child termination/wait. Existing targets and Xray remain unchanged. Deployment
|
||||
allows 120s Compose shutdown grace and 180s systemd stop timeout for bounded
|
||||
allows 240s Compose shutdown grace for bounded
|
||||
monitor cancellation/restoration and child joins. Code integration is not
|
||||
image/routing/signal live verification.
|
||||
|
||||
@@ -185,7 +185,7 @@ This is a standalone **new isolated target** candidate, not a production-approve
|
||||
deployment and not an integration into existing zapret/Xray. Keep the exact
|
||||
`managed-test` role, ID, marker, mount, config, DB and owned-listener guards above.
|
||||
Do not change the role to `production`, adopt old state, share an old network
|
||||
namespace, or relax a refusal. Provisioning and optional service installation
|
||||
namespace, or relax a refusal. Container provisioning instructions
|
||||
are described in [deploy/managed-target/README.md](deploy/managed-target/README.md).
|
||||
|
||||
### Build once, promote the same image
|
||||
@@ -227,8 +227,8 @@ docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/co
|
||||
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml pull target
|
||||
```
|
||||
|
||||
The supplied systemd unit uses this same `.env` and starts with `--no-build`;
|
||||
no image-selection override or unit edit is needed. Keep the selected digest and
|
||||
Start the selected container image with Compose `up --no-build`;
|
||||
no host service installation is required. Keep the selected digest and
|
||||
private configuration together for upgrades and rollback. Never run a build
|
||||
against the production digest selector; build/tag separately as shown above.
|
||||
|
||||
@@ -268,31 +268,55 @@ existing Xray deployment as part of bootstrap.
|
||||
|
||||
### Supervision and failure handling
|
||||
|
||||
Compose restart is disabled. Optional native systemd runs **foreground** Compose
|
||||
with `--no-build --abort-on-container-exit --exit-code-from target`; do not also
|
||||
run detached Compose for that project. The unit requests restart only for exit
|
||||
75 via `Restart=no` plus `RestartForceExitStatus=75`, bounded by
|
||||
`StartLimitBurst=3` in six hours (starts, including the initial
|
||||
start, not three additional retries). Compose stop grace is 120s, systemd 180s.
|
||||
The container entrypoint is PID 1 and owns all technical child restart control.
|
||||
After a retryable monitor/SS failure it joins children, reconciles trusted state
|
||||
and stops target-local zapret before starting a new runtime cycle in the **same
|
||||
container**. Retries use cancellable backoff (15s, 30s, capped at 60s) and at most
|
||||
three runtime starts per rolling six hours, including the initial start. Exhaustion
|
||||
stops fail-closed with 78; configuration, safety, DB, journal, rollback or cleanup
|
||||
errors never authorize retry. No new dependency, host service installation or
|
||||
external process manager is required. SIGTERM/SIGINT cancels backoff or the running
|
||||
monitor, waits for bounded restoration and joins children before exit.
|
||||
|
||||
Compose restart remains disabled: a blanket container restart could loop over a
|
||||
safety refusal or reset the in-container retry budget. Compose stop grace is 240s:
|
||||
the sequential cleanup budget is 140s (45s monitor join, 45s reconciliation plus
|
||||
5s kill grace, 30s zapret stop plus 5s kill grace, and two 5s SS joins), with room
|
||||
for a deferred 45s startup command plus kill grace and scheduling overhead.
|
||||
The join watchdog runs in a fresh shell and is cancelled when its child finishes;
|
||||
graceful shutdown does not wait out the watchdog's full deadline.
|
||||
Container-platform lifecycle, creation/recreation and host-boot policy belong to
|
||||
the deployment operator, not an external manager supplied by this project.
|
||||
|
||||
| Target exit | Operator meaning |
|
||||
| --- | --- |
|
||||
| `64` | Invalid configuration; fix settings before starting |
|
||||
| `78` | Safety/DB/journal/rollback/cleanup failure; investigate, no automatic restart |
|
||||
| `75` | Technical failure eligible for bounded restart only after safe cleanup |
|
||||
| `75` | Internal technical-cycle result; PID 1 retries only after safe cleanup, within its budget |
|
||||
| `0` | Graceful signal shutdown; no automatic restart |
|
||||
|
||||
Bootstrap/recovery has a fixed **90-minute whole-job deadline**, plus an
|
||||
independent **30-second restoration reserve**. This is a catastrophic technical
|
||||
ceiling, not a normal candidate-check timeout and not an operator tunable. Valid
|
||||
nonpassing candidates/unresolved GGC are normal recovery outcomes governed by
|
||||
the persisted threshold/cooldown, not a reason to restart the container. Failed
|
||||
nonpassing candidates/unresolved GGC are normal recovery outcomes, not a reason
|
||||
to restart the container. Genuine **no qualified strategy** from validated scan/check
|
||||
evidence persists a recovery wait of at least **two hours** in SQLite, including
|
||||
empty bootstrap and frontend/GGC exhaustion. The same container stays alive and
|
||||
continues light periodic Player API probes; it does not scan or apply during that
|
||||
wait. Startup respects the durable expiry, and recovery resumes after expiry under
|
||||
the normal failure-threshold policy. In both fresh bootstrap and periodic recovery,
|
||||
the delay is the greater of two hours and the configured recovery cooldown;
|
||||
a longer configured cooldown is not shortened.
|
||||
Other ordinary recovery outcomes retain the configured cooldown (default **30m**).
|
||||
Malformed/incomplete reports, runner errors and unsafe state are not strategy
|
||||
absence and must not enter this wait. The two-hour minimum absence policy is not a
|
||||
new operator setting, technical retry backoff, or the **90-minute watchdog**. Failed
|
||||
restoration or unsafe state vetoes retry: timeout exit 75 requires the job's own
|
||||
deadline and only timeout/cancellation-related errors, with no joined cleanup,
|
||||
DB or safety failure. Unexpected monitor/SS failures are separately technical
|
||||
only if shutdown cleanup succeeds. Shutdown joins the monitor for up to 60s,
|
||||
target-local zapret stop gets 30s then forced kill after five more, and SS children
|
||||
get 5s each; failure to join/restore is fail-closed. The reserve is not extra
|
||||
only if shutdown cleanup succeeds. Shutdown uses bounded monitor join,
|
||||
guarded reconciliation, target-local zapret stop and SS child joins;
|
||||
failure to join/restore is fail-closed. The reserve is not extra
|
||||
scanning time. The 90-minute budget is shared by the whole bootstrap/recovery
|
||||
job, not renewed per candidate or phase.
|
||||
|
||||
@@ -304,7 +328,7 @@ be reported as complete playback health.
|
||||
### Upgrade, rollback and operator recovery
|
||||
|
||||
1. Record the current image digest and private deployment configuration. Stop
|
||||
the service (or foreground Compose), wait for complete cleanup, and verify
|
||||
the target with Compose, wait for complete cleanup, and verify
|
||||
the target is stopped. If cleanup fails, investigate before upgrading.
|
||||
2. With the target stopped, make a root-only backup of the **entire** state
|
||||
directory (including DB sidecars, journal, config, identity and lock file),
|
||||
@@ -329,16 +353,16 @@ is rolled back to qualified committed bytes before any reload/start; fresh
|
||||
baseline and known committed state are idempotent. Unknown/legacy state, drift,
|
||||
missing qualification or failed rollback blocks startup with a non-retry exit.
|
||||
|
||||
On drift, unreconcilable pending transaction, rollback failure, or exhausted systemd start
|
||||
limit, leave state intact and service stopped. Diagnose using sanitized status
|
||||
On drift, unreconcilable pending transaction, rollback failure, or exhausted PID 1 retry
|
||||
budget, leave state intact and the container stopped. Diagnose using sanitized status
|
||||
and private artifacts; repair the actual provisioning/configuration issue or
|
||||
restore a known consistent backup. Only then manually start (and reset the
|
||||
systemd failed/start-limit state if needed). There is no automatic wipe or
|
||||
restore a known consistent backup. Only then manually start the target with Compose.
|
||||
There is no automatic wipe or
|
||||
automatic adoption path.
|
||||
|
||||
Rollout acceptance remains pending real isolated-target restart/deadline tests:
|
||||
empty bootstrap, saved-first recovery, exact-host GGC rotation, image binding,
|
||||
NFQUEUE activation/rollback, SIGTERM during downtime, deadline restoration,
|
||||
retry limits/exit propagation through foreground Compose/systemd, persistent
|
||||
in-container retry limits/backoff, durable two-hour absence wait and resumption, persistent
|
||||
recreation and backup restore. Perform these on the new target only; local unit
|
||||
or static checks cannot establish production readiness.
|
||||
Reference in new issue
Block a user