Files
zapret-autotune/MANAGED_TARGET.md
T
ZeusinaandGPT-6 Bot 371382c2c4 feat(managed): add explicit experimental Googlevideo suffix policy
Keep operator-owned native circular extrapolation distinct from cold/adaptive qualification and playback readiness. Preserve qualified frontend/assets, use guarded journal transactions and rollback, and apply private startup requests before the owned monitor begins. Retain legacy rendering and qualification gates.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 07:04:12 +03:00

27 KiB
Raw Blame History

Fresh managed-test target contract (not deployed)

managed init|reconcile|status|active-ids|apply is exclusively for a new dedicated container which owns its entire config. The existing/dev/production container must refuse these commands. A role env or CLI-created identity is not enough. Provisioning must supply all of the following before any managed command can run:

Setting Exact required value
ZAPRET_AUTOTUNE_TARGET_ROLE managed-test
ZAPRET_AUTOTUNE_TARGET_ID zapret-autotune-managed-v1
ZAPRET_AUTOTUNE_TARGET_DIR /var/lib/zapret-autotune/target
ZAPRET_CONFIG /var/lib/zapret-autotune/target/config
SOCKS_PORT Base image's actual ss-local service port (canonical decimal, 1–65535)
ZAPRET_AUTOTUNE_SOCKS_PROXY socks5://127.0.0.1:$SOCKS_PORT (expanded value)

The fixed directory must already be a private runtime-user-owned dedicated persistent volume mountpoint, not just a directory under another mount. The production verifier checks the exact path in /proc/self/mountinfo. Provisioning must also supply /etc/zapret-autotune/target-id containing exactly zapret-autotune-managed-v1 followed by a newline. Init never writes that marker. Base runtime executables must exist: /opt/zapret2/init.d/sysv/zapret2, /opt/zapret2/nfq2/nfqws2, and /usr/bin/blockcheckw. There are no arbitrary directory flags or env overrides of these fixed values. These are operator safety guards, not protection against a privileged user deliberately forging a provisioned runtime.

The managed config is a standalone full config only in this fresh volume. Never replace, adopt, or symlink over /opt/zapret2/config or any existing service config. The new deployment entrypoint must ensure the base daemon actually honors the exact ZAPRET_CONFIG path; compatibility remains unproven. Existing services would require a separately designed managed include fragment, not this full-config adapter. No such integration is implemented.

# Only after fresh-target provisioning establishes the entire contract above:
zapret-autotune managed init
zapret-autotune managed status
zapret-autotune managed apply --frontend-candidate 1 # bootstrap before discovery
zapret-autotune managed active-ids # JSON for supervisor; no health probe
zapret-autotune managed apply --frontend-candidate 1 --googlevideo-candidate 2

Init refuses nonempty directories except an exact private disabled baseline; it never overwrites/adopts another config. It creates identity, journal and backup; repeated initialization, symlinks, hardlinks, public/unowned files and unsafe ancestors are refused. Status reports generation/hash/transaction state, validated active IDs/groups and unresolved groups, not config, secrets or live health. Pending or unverifiable state reports unknown groups and no active IDs; drift fails closed. active-ids fails closed on pending or unverifiable state. Init/status do not reload or probe, but both verify runtime and listener ownership.

All managed commands verify that the expected IPv4 listener is owned by a running ss-local executable in the target PID/network namespace: /proc/net/tcp socket inodes must match that process's file descriptors. A process alone, a wrong port, or another loopback proxy is insufficient. Health repeats this validation before its Player API request. The host's published SOCKS port may map to this listener, but is never used inside the target. The standalone health youtube command retains its separate generic loopback contract; managed activation is stricter. Ambient proxies/direct fallback are disabled. Player API OK is not playback.

Apply accepts one or both frontend/exact-GGC SQLite candidate IDs, requiring distinct positive IDs, at most one per group, complete endpoint × TLS evidence and inert shell-safe arguments. The arguments are the full desired active set, not patches: omitting a group removes it. Frontend-only bootstrap supports Player API discovery before GGC scanning; GGC-only is also permitted. No missing profile is invented, and missing groups remain unresolved. Player API OK never asserts GGC/video playback or full-profile health. A supervisor can read managed active-ids and explicitly retain the other group while replacing a failed one. It opens the candidate DB read-only via ZAPRET_AUTOTUNE_DB; candidates are not active state. Bindings/inputs are checked before pending transaction reconciliation. Production reload is exclusively the fixed argv /opt/zapret2/init.d/sysv/zapret2 restart, no shell, with a 30-second deadline. Apply and rollback both use this full firewall/daemon restart; daemon-only restart is not a supported fallback. This changes only the new target container's dedicated network namespace, never the host or old container's firewall. Provisioning must not use host networking or share an existing container's network namespace.

The baseline explicitly sets INIT_APPLY_FW=1, NFQWS2_ENABLE=0, empty NFQWS2_OPT, TCP ports 80,443, UDP ports 443, and TCP/UDP in/out packet limits of 15. QNUM is deliberately inherited from the base runtime default. Active configs keep those fixed firewall values and set NFQWS2_ENABLE=1 with the qualified managed strategy arguments. Restoring the baseline and running full restart removes its target-local queues and stops nfqws2; restoring an earlier active config reinstates its firewall/daemon state. The config replacement is atomic; the init script's stop/start sequence is not a zero-interruption or kernel-atomic firewall transaction.

Journal v2 persists complete committed and pending candidate ID lists alongside their hashes. Commit promotes the pending list; rollback/crash reconciliation restores the prior list and hash. Reporting validates qualification/groups and exact rendered hash against the read-only DB. Status reports invalid refs as unresolved; active-ids and activation fail closed on invalid refs. Legacy v1 journals keep the exact deployed hash and unknown IDs: status marks both groups unresolved, and active-ids refuses to infer a list. A guarded full-set apply upgrades to v2 only after the exact old hash matches; rollback preserves the unknown legacy deployment. Re-init cannot adopt an existing deployment.

The transaction engine retains same-directory staging/fsync/rename, durable journal/backup, advisory locking, external drift rejection, rollback and fail-closed crash reconciliation. Failed rollback reload remains pending. Directories must not be writable by untrusted processes; hostile concurrent filesystem mutation is outside the transaction boundary. Shell config safety and exact GGC filtering are unchanged.

The isolated Compose definition now supplies the marker, mount and ordered startup; it has not been built or deployed. No Docker socket, remote/container operations, old target paths, Xray edits, or host firewall provisioning are included. Before deployment, provisioning must verify the pinned image honors ZAPRET_CONFIG, supports all listed variables, and starts ss-local while the baseline leaves nfqws2 disabled. Start the fresh target with that baseline, then verify active apply installs target-local nft NFQUEUE rules and starts nfqws2, and rollback restores both. These image/startup/routing gates remain unverified: if any is unsupported or uncertain, deployment must fail closed, not substitute restart-daemons or alter an old config. Persistent volume durability is a provisioning requirement; mountinfo alone cannot establish the storage lifecycle.

Internal recoverManagedTarget and wireManagedRecovery add injectable startup bootstrap and saved-first recovery callbacks for the persisted monitor. Bootstrap scans/checks/saves frontend and applies it alone with Player API post-check. Recovery keeps an already healthy frontend, otherwise stops only the fixed target zapret service, tries up to ten qualified saved IDs newest-first, and scans only after complete valid non-pass results or health-rejected rollbacks. Malformed reports/runner errors fail closed. GGC discovery uses the injected managed-SOCKS path, revalidates against the fresh exact host, and saves new immutable evidence before full-set apply. A frontend-only rollback boundary survives GGC failure; unresolved GGC is reported separately, not playback health. Cancelled/failed downtime restores the committed configuration and runs fixed target start with an independent 30-second cleanup deadline. Failed cleanup or rollback is fatal to the supervisor; unresolved journal state is never started. The callback caller must hold the monitor singleton; the wiring repeats runtime identity/SOCKS guards. Fake-only checks are added, but tests/build are deferred to the orchestrator. managed monitor now binds those callbacks and only the dedicated target entrypoint starts it; no live/runtime validation has occurred.

Dedicated monitor integration (isolated clean-run acceptance passed)

zapret-autotune managed monitor [--interval 5m] [--failure-threshold 3] [--cooldown 30m] uses the health environment defaults. Interval is positive and at most 24h, threshold 1–1000, cooldown positive and at most 168h; explicit empty or invalid settings fail before any lock. No executable/path/proxy flags exist. The DB environment must be exactly /var/lib/zapret-autotune/target/candidates.db; existing DB/sidecars must be private, owned regular files without aliases/hard links. Runtime/SOCKS, directory, identity and DB path guards run before the singleton. SQLite opens writable target-owned state; active IDs remain journal/DB/hash validated.

Probe reuses health youtube's Player API contract, forced owned loopback SOCKS, no direct fallback. Discovery uses that same SOCKS adapter. Scan/check execute only /usr/bin/blockcheckw, daemon operations only fixed target-local /opt/zapret2/init.d/sysv/zapret2 start/stop/restart argv; no shell/eval command strings. Context cancellation kills/waits process groups and bounded restoration finishes before returning. Reports suppress untrusted diagnostics and credentials.

Startup reconciles transactions under the singleton and bootstraps only when no qualified active frontend exists. Bootstrap/recovery checks saved candidates first; GGCError reports unresolved and continues probing, not playback success. Other DB/safety/journal/runner failures stop nonzero; joined cancellation plus a cleanup failure is also fatal. A qualified frontend at startup is probed immediately, not rescanned merely because the process restarted.

Entrypoint ordering is SS children → listener → fresh managed init if needed → managed reconcile → zapret start → one installed /usr/local/bin/zapret-autotune managed monitor child. It supervises all three children; unexpected monitor exit is not a successful shutdown. SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and SS child termination/wait. Existing targets and Xray remain unchanged. Deployment allows 240s Compose shutdown grace for bounded monitor cancellation/restoration and child joins. Clean RC d8316e6 has passed isolated image/routing, bootstrap, periodic probe, monitor-crash cleanup, graceful stop and persistent recreation gates; see the recorded source/image and PASS report. This does not establish playback or blanket production approval.

Injected unit tests cover exact identity/path/config, mountpoint refusal, old-container-like env, SOCKS port/process refusal, correct fresh-target success, non-overwrite, transaction rollback and redaction. Verification results are recorded in PROGRESS.md. Focused local checks can be rerun with:

go test ./cmd/zapret-autotune -run TestManaged
go build ./cmd/zapret-autotune

Production preparation

This is a standalone new isolated target candidate, not a production-approved deployment and not an integration into existing zapret/Xray. Keep the exact managed-test role, ID, marker, mount, config, DB and owned-listener guards above. Do not change the role to production, adopt old state, share an old network namespace, or relax a refusal. Container provisioning instructions are described in deploy/managed-target/README.md.

Build once, promote the same image

Use Dockerfile.target for the supervised target; root Dockerfile is the proposal CLI image, not the managed service. Both pin Go 1.25.3 and the runtime by digest, use GOTOOLCHAIN=local, CGO_ENABLED=0, go.sum, and -trimpath. Pins constrain build inputs; they do not establish live acceptance or guarantee bit-for-bit builds on different platforms. Record the source commit, platform, builder/runtime digests and resulting image digest for every release.

Both Dockerfiles require BuildKit for the compilation-cache mount at /root/.cache/go-build (enable with DOCKER_BUILDKIT=1 if not already default). This cache only speeds rebuilds; it is not copied into the final runtime image and is not a secret store. No deployment credentials are needed or passed to the build; keep secrets out of build arguments and cache contents.

Example (replace registry and release; never reuse a release tag):

IMAGE=registry.example/team/zapret-autotune-managed-target:2026-10-10-r1
docker build --pull -f Dockerfile.target -t "$IMAGE" .
docker push "$IMAGE"
docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE"

Promote the printed registry/repository@sha256:... reference unchanged from validation to deployment; do not rebuild or use latest. Authenticate separately, never put registry credentials in these commands. The supplied Compose file selects ${ZAPRET_AUTOTUNE_IMAGE:-zapret-autotune-managed-target:local}. Set ZAPRET_AUTOTUNE_IMAGE=registry.example/team/zapret-autotune-managed-target@sha256:<digest> in the same private deployment .env as SS_PASSWORD (mode 0600), using the actual printed digest. Tags and the local default are build/test candidates only; an immutable managed-target digest is required for production promotion. From the repository root, validate and pull without printing the rendered secrets:

docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml config --quiet
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml pull target

Start the selected container image with Compose up --no-build; no host service installation is required. Keep the selected digest and private configuration together for upgrades and rollback. Never run a build against the production digest selector; build/tag separately as shown above.

Configuration and secrets

Create fresh empty target-state owned by root/runtime UID 0, mode 0700, and the private deployment .env mode 0600; rootless/remapped Docker needs the mapped owner. Bootstrap is for empty provisioned state only; recreation retains identity, journal, candidate DB, sidecars and lock file. Never automatically wipe state to make startup succeed. The entrypoint creates Shadowsocks JSON in private tmpfs (0700 directory, 0600 files), unsets the password before children, and does not put it in argv. Docker administrators/root can still read container env and runtime config. Do not print .env, rendered Compose config, diagnostics or signed URLs; use docker compose ... config --quiet for validation.

Existing setting Default / accepted value for managed target
ZAPRET_AUTOTUNE_IMAGE Local build/test default zapret-autotune-managed-target:local; production promotion requires a managed-target repository@sha256:<digest>
SS_PASSWORD Required independent hex-only secret, at least 32 characters; deployment README generates 64
SS_DNS_SERVER Empty (unchanged resolver); optional literal IPv4/IPv6 for target-only native ss-server DNS, validated before launch
ZAPRET_AUTOTUNE_YOUTUBE_SERVICE 0 (legacy) or opt-in 1; explicitly forwarded by Compose
ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE cold default or adaptive (requires service 1); explicitly forwarded
ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS JSON array of 1..5 unique VideoIDs; defaults to only the primary ID; explicitly forwarded
ZAPRET_AUTOTUNE_HOST_SOCKS_PORT 11080; choose an available TCP/UDP port 1–65535, published only on 127.0.0.1
SOCKS_PORT Supplied deployment fixes internal listener to 1080; not a host-port override
ZAPRET_AUTOTUNE_SOCKS_PROXY Fixed socks5://127.0.0.1:1080 inside this target, no credentials/direct fallback
ZAPRET_AUTOTUNE_VIDEO_ID dQw4w9WgXcQ; exactly 11 ASCII letters, digits, _ or -
ZAPRET_AUTOTUNE_HEALTH_INTERVAL 5m; duration in (0,24h] for monitor
ZAPRET_AUTOTUNE_FAILURE_THRESHOLD 3; integer 1–1000 for monitor
ZAPRET_AUTOTUNE_RECOVERY_COOLDOWN 30m; duration in (0,168h] for monitor
ZAPRET_AUTOTUNE_DB Fixed /var/lib/zapret-autotune/target/candidates.db

The monitor's three policy flags override corresponding environment defaults; explicit empty/invalid environment settings are rejected. The entrypoint accepts no arguments. Compose forwards the primary/sample VideoIDs and service/mode settings. It does not forward the interval/threshold/cooldown policy settings from .env: add those existing variables to target.environment in your deployment copy/override if needed. Keep all fixed identity/path settings unchanged. No public SS listener is published; Xray configuration is external and operator-owned, optionally using the host's loopback SOCKS port. Do not modify an existing Xray deployment as part of bootstrap.

Current service/adaptive hardware acceptance is pending, independent of the recorded legacy RC PASS. Adaptive means post-warmup, not cold: first connections may fail and native learning repeats after restart. At least three real unique signed ordinary media host samples are required; multiple VideoIDs may yield one CDN host and leave DEGRADED with no authorized suffix. Sampled thumbnail/avatar routes do not prove availability of the whole thumbnail pool. The reported 8.8.8.8 DNS diagnostic corrected a tampered avatar, not all thumbnail routes; native DNS correction is not full asset availability proof. Target DNS uses validated native -d argv (nonsecret), with installed-option support checked at image build; passwords remain private JSON only. No DoH bridge, permanent Google IP pins, global host DNS edits or external startup manager are introduced.

Supervision and failure handling

The container entrypoint is PID 1 and owns all technical child restart control. After a retryable monitor/SS failure it joins children, reconciles trusted state and stops target-local zapret before starting a new runtime cycle in the same container. Retries use cancellable backoff (15s, 30s, capped at 60s) and at most three runtime starts per rolling six hours, including the initial start. Exhaustion stops fail-closed with 78; configuration, safety, DB, journal, rollback or cleanup errors never authorize retry. No new dependency, host service installation or external process manager is required. SIGTERM/SIGINT cancels backoff or the running monitor, waits for bounded restoration and joins children before exit.

Compose restart remains disabled: a blanket container restart could loop over a safety refusal or reset the in-container retry budget. Compose stop grace is 240s: the sequential cleanup budget is 140s (45s monitor join, 45s reconciliation plus 5s kill grace, 30s zapret stop plus 5s kill grace, and two 5s SS joins), with room for a deferred 45s startup command plus kill grace and scheduling overhead. The join watchdog runs in a fresh shell and is cancelled when its child finishes; graceful shutdown does not wait out the watchdog's full deadline. Container-platform lifecycle, creation/recreation and host-boot policy belong to the deployment operator, not an external manager supplied by this project.

Target exit Operator meaning
64 Invalid configuration; fix settings before starting
78 Safety/DB/journal/rollback/cleanup failure; investigate, no automatic restart
75 Internal technical-cycle result; PID 1 retries only after safe cleanup, within its budget
0 Graceful signal shutdown; no automatic restart

Bootstrap/recovery has a fixed 90-minute whole-job deadline, plus an independent 30-second restoration reserve. This is a catastrophic technical ceiling, not a normal candidate-check timeout and not an operator tunable. Valid nonpassing candidates/unresolved GGC are normal recovery outcomes, not a reason to restart the container. Genuine no qualified strategy from validated scan/check evidence persists a recovery wait of at least two hours in SQLite, including empty bootstrap and frontend/GGC exhaustion. The same container stays alive and continues light periodic Player API probes; it does not scan or apply during that wait. Startup respects the durable expiry, and recovery resumes after expiry under the normal failure-threshold policy. In both fresh bootstrap and periodic recovery, the delay is the greater of two hours and the configured recovery cooldown; a longer configured cooldown is not shortened. Other ordinary recovery outcomes retain the configured cooldown (default 30m). Malformed/incomplete reports, runner errors and unsafe state are not strategy absence and must not enter this wait. The two-hour minimum absence policy is not a new operator setting, technical retry backoff, or the 90-minute watchdog. Failed restoration or unsafe state vetoes retry: timeout exit 75 requires the job's own deadline and only timeout/cancellation-related errors, with no joined cleanup, DB or safety failure. Unexpected monitor/SS failures are separately technical only if shutdown cleanup succeeds. Shutdown uses bounded monitor join, guarded reconciliation, target-local zapret stop and SS child joins; failure to join/restore is fail-closed. The reserve is not extra scanning time. The 90-minute budget is shared by the whole bootstrap/recovery job, not renewed per candidate or phase.

Monitor health means Player API OK through owned SOCKS. GGC evidence covers only the exact discovered hostname and tested TLS protocols, not wildcard googlevideo coverage, video transfer, SABR or playback. Unresolved GGC must not be reported as complete playback health.

Upgrade, rollback and operator recovery

  1. Record the current image digest and private deployment configuration. Stop the target with Compose, wait for complete cleanup, and verify the target is stopped. If cleanup fails, investigate before upgrading.
  2. With the target stopped, make a root-only backup of the entire state directory (including DB sidecars, journal, config, identity and lock file), preserving ownership/modes. Store .env and image/config records privately with it; do not print archive contents, diagnostics or signed URLs. A live copy of only candidates.db is not a consistent rollback backup.
  3. Select/pull the new immutable digest and start without building. Inspect sanitized status/logs, verify identity/hash and expected active groups, owned SOCKS, target-local queues, health scope and persistent recreation. Keep the old image and backup until acceptance; DB migrations may prevent image-only downgrade.
  4. To roll back, stop and verify cleanup again. Preserve failed state separately for investigation. Restore the matched stopped-state backup as a whole into the same dedicated mount with private ownership/modes, select its old digest/config, then start. Do not merge journals/DB generations or re-init. A restored legacy/live/unverifiable recovery owner can fail closed; never clear its lease or unlink the lock merely to force startup.

On startup, managed reconcile accepts no flags and uses only the fixed private /var/lib/zapret-autotune/target/candidates.db. Known interrupted prepared state is rolled back to qualified committed bytes before any reload/start; fresh baseline and known committed state are idempotent. Unknown/legacy state, drift, missing qualification or failed rollback blocks startup with a non-retry exit.

On drift, unreconcilable pending transaction, rollback failure, or exhausted PID 1 retry budget, leave state intact and the container stopped. Diagnose using sanitized status and private artifacts; repair the actual provisioning/configuration issue or restore a known consistent backup. Only then manually start the target with Compose. There is no automatic wipe or automatic adoption path.

The recorded clean RC PASS covers empty-state bootstrap through qualified frontend and exact-host GGC apply, image/config binding, ongoing probes, in-container monitor crash restart, graceful stop and persistent recreation. Earlier recovery, rollback, rotation, retry and absence fixtures are distinguished in PROGRESS.md; do not attribute all of them to the clean run. No media playback, real 90-minute watchdog expiry or real two-hour absence expiry wait is claimed. Production rollout and backup-restore acceptance remain operator gates on a new isolated target; local unit/static checks and isolated PASS do not establish blanket production readiness.

Explicit experimental Googlevideo suffix

Normal --googlevideo-scope suffix still requires representative qualification, media proof, and both qualified image profiles. An operator may explicitly own the audited native circular A/E composite without claiming those results:

zapret-autotune managed apply --googlevideo-experiment native-circular --googlevideo-scope experimental_googlevideo_suffix

This command requires the monitor singleton to be free, a matching native Lua hash, a fresh Player-discovered exact GGC identity, and existing qualified frontend/assets. It preserves their candidate IDs and uses the normal guarded transaction/reload/Player-health/rollback hooks. The new candidate definition has no cold/adaptive evidence and is not a qualified candidate. Both pending and committed journal policy record experimental_googlevideo_suffix; legacy rendered bytes and qualification gates are unchanged. Unknown policies fail closed.

For PID-1-controlled application, while this dedicated target is stopped and after taking a complete private state backup, create a private empty file named .autotune-experimental-suffix-request in its owned state directory. Startup reconciles/starts the known deployment, performs the command before launching the monitor, and consumes the request only on success. Failure exits closed; retain artifacts and use the matched whole-state/image rollback procedure above.

Resource probes still run. This policy remains DEGRADED/GGC-unresolved even when a bounded media check succeeds: neither playback READY nor family qualification is inferred. Automatic frontend/image repairs preserve the operator suffix; automatic GGC recovery does not replace or qualify it.