Files
zapret-autotune/RESOURCE_PLAN.md
T
2026-10-14 22:02:00 +03:00

19 KiB
Raw Blame History

Resource compatibility planner and isolated qualifier

plan remains proposal-only and never changes runtime or config. Explicit generic startup apply is implemented and independently reviewed locally; remote acceptance is pending. This is not automatic recovery or production READY. Explicit isolated qualification temporarily installs an owned NFQ worker and queue rules, but never changes legacy SQL groups, renderer or apply journals. Monitor HEALTHY, old SQL measurements and public check JSON are not proof. Without a completed owned qualification batch, compatibility is UNKNOWN and hosts are unassigned. No PASS/evidence import is exposed.

Commands require root and the existing managed-test target settings:

zapret-autotune strategy-import --strategies /private/strategies.json
zapret-autotune plan --config /private/resources.json
zapret-autotune qualify --config /private/resources.json --strategies /private/strategies.json --isolated

strategy-import requires an owned, regular, single-link 0600 file. Only definitions are accepted; duplicate keys, unknown fields and evidence fields are rejected. Example:

{
  "version": 1,
  "strategies": [
    {
      "id": "split-one",
      "args": "--payload=tls_client_hello --lua-desync=multisplit:pos=1",
      "selectors": {"transport":"tcp","ports":"443","l7":"tls","tls":"both"}
    }
  ]
}

The registry is one immutable batch (1–32 unique IDs). This conservative slice accepts only TCP/443/TLS selectors and tls12, tls13, or both policy. Argument whitespace is canonicalized; option/Lua stage order is preserved. Existing payload/range validation is reused. Lua stages are restricted to fake, multisplit, multidisorder, fakeddisorder, split and disorder; parameters to pos, repeats, tcp_ts, tcp_flags_set, seqovl and blob=fake_default_tls. File/inline Lua, queue/user/fwmark/filter options and arbitrary stages are rejected. Acceptance of a definition is not runtime verification.

The private target directory holds .resource-plan-key (32 random bytes), .resource-plan-strategies.json and, when an owned producer exists, .resource-plan-matrix.json. JSON envelopes have payload and an HMAC-SHA256 mac. Each file is write-once, owned, 0600, no-follow, single-link and bounded to 64 KiB. Held directory/file identities and exact snapshots are checked; foreign entries, replacement inodes and modified bytes are never clobbered or silently adopted. Interrupted writes remain blocked for explicit operator repair. There is no automatic rotation/update/repair in this slice. The private key and same-UID/root boundary must be protected: HMAC is not protection from an actor who can read the key and replace the whole target state between processes.

Matrix v1 binds the registry digest and at most one completed run per candidate. Each run binds ID, definition digest (canonical args and selectors), keyed config fingerprint, start/completion milliseconds, resource/TLS outcomes and keyed resource fingerprints. Its attestation records collector/runtime, definition digest, composite executable/Lua input hash, installed rules hash, actual worker-argv hash, queue and worker PID/start identity, plus restored/joined completion. No URLs, headers, resolved IPs, bodies or raw errors are persisted. A nonserialized producer seal protects the completed in-memory snapshot against mutation before publication. Only the owned producer and test-internal fixtures create seals; JSON DTOs cannot authorize publication. Every resource/protocol cell records exactly three actual attempts; any nonpass in those attempts makes the cell negative (never an average or partial PASS).

plan loads only authenticated private state and the existing resource-config snapshot (including resolved environment secrets, DNS/proxy and private headers in keyed fingerprints). Every resource on an exact host, across all services and input group labels, must have matching complete results for every required TLS protocol. Complete failed checks are negative; missing, expired (24 hours), future, mismatched or partial checks are unknown. Changed config invalidates old run evidence. Malformed/unauthenticated state yields blocked, unassigned output and exit 78; valid missing evidence yields unknown output and exit 0.

Output contains proposal_only, evidence (authentication status, not health), per-host validated/negative/unknown candidate alternatives, assignments and unassigned. Compatible equal args/selectors merge coverage across input labels. Greedy greatest-uncovered coverage, then lexical candidate ID, produces deterministic disjoint exact-host assignments. It does not claim global optimality or suffix coverage. Explicit legacy YouTube suffix policy is unchanged.

INTRUSIVE / ISOLATED startup only

Set both ZAPRET_AUTOTUNE_QUALIFY_CONFIG and ZAPRET_AUTOTUNE_STRATEGY_CONFIG to container paths on a dedicated target. Do not send working user traffic to this target's published SOCKS port. This is not noninterrupting background search. Provision the private resource and 0600 strategy files separately; compose forwards paths but does not mount files for you. Start with the managed disabled baseline, not an existing active profile. Qualification v1 accepts lowercase DNS hostnames, not literal IP targets; the ordinary resource-check command's broader literal-IP policy is unchanged.

PID 1 starts its owned SS+DoH services, initializes/reconciles known state, then launches one finite qualify ... --isolated session instead of starting the live zapret daemon or monitor. Qualifier checks root/target namespace, PID 1's actual entrypoint/environment, direct-child executable/arguments/session ACK, owned SS and DNS readiness anchors. A flag or docker-exec environment alone does not authorize qualification. Missing either startup path fails closed. Without --isolated, qualify remains UNKNOWN/exit 78 without runtime changes.

The shared monitor singleton excludes legacy and generic controllers. Any existing NFQ worker/queue, active profile, unknown config/journal or existing immutable matrix is refused. Known baseline config/journal/backup/identity inodes and bytes remain held unchanged throughout the job. This first producer does not pause/stop/restart existing active profiles; job-bound maintenance rollback is deferred rather than pretending a path-only daemon stop is safe.

Each candidate uses the existing worker-before-nft/queue-ready lifecycle on queue 200, fixed Lua imports and fwmark 0x10000000, plus native --filter-l7=tls. The actual child executable, argv, parent, start identity, queue socket ownership, executable/Lua hashes, native --version output 1.0.2 and installed nft rule bytes are attested before capture and checked again at each collector guard. DoH's existing 0x50000000 socket mark bypass remains. TLS policy is collector/eligibility policy, not a nonexistent native TLS-version filter. All configured protocols are measured; incompatible selector policies remain UNKNOWN in the proposal.

The existing GET/HEAD generic collector preserves TLS origin verification, configured body/status/content-type expectations, private stdin and IP pinning. Completed expectation failures and typed peer response/TLS/transport negatives do not stop the remaining repeats/cells. Guard/source/unknown infrastructure or batch cancellation veto the entire batch; no partial positive matrix is saved. Config (including resolved environment/header secrets) and strategy-source snapshots are checked after DNS/before transfer and before sealing/publication. Workers are joined and temporary nft queues removed before verifying unchanged disabled baseline/runtime and sealing. The batch has a fixed 15-minute deadline with separate bounded cleanup; PID 1 keeps SS+DoH alive through cleanup. The bounded 64 KiB matrix size (including its authenticated envelope and worst-case field widths) is checked before any worker/nft operation; excessive catalogue/resource combinations return UNKNOWN/exit 64 even within individual count caps (32 strategies, 64 resources).

Successful batches, including complete negatives, return the assignment proposal. Incomplete/unsafe jobs return UNKNOWN/blocked exit 78 without proof. PID 1 then stays safe-idle with SS+DoH for inspection, for either success or blocked completion: no qualification retry loop, duplicate controller or automatic generic monitor start. Retests require future explicit immutable snapshot rotation, not overwriting prior evidence.

Completed controller cleanup is acknowledged separately from qualification in an exclusive, 0600 HMAC-authenticated qualification-completed.json on the current boot's SS tmpfs. It binds the launched PID, exit outcome (0 or 78) and verified cleanup; it is not strategy proof. PID 1 checks that ACK against its private session launch and revalidates the disabled baseline, singleton, SS/DoH ownership and absence of NFQ workers/temporary nft queues before shutting down dependencies. Thus TERM after a clean blocked qualifier has exited but before its parent reaps it can exit cleanly. Worker/nft/join/restoration failure, missing/foreign/mutated ACK or unverified early failure still veto clean shutdown with exit 78. Ordinary legacy/generic monitor exit 78 remains a safety veto.

No unsupported --filter-mark or blockcheck embedded/stream/progress/deadline flags are used. Background noninterrupting search, active-profile maintenance and immutable snapshot rotation for repeated runs remain deferred.

Explicit generic transactional startup apply (local implementation)

On the target with its already completed private qualification batch, unset both qualification startup paths and set:

ZAPRET_AUTOTUNE_RESOURCE_CONFIG=/private/resources.json
ZAPRET_AUTOTUNE_RESOURCE_APPLY=1
ZAPRET_AUTOTUNE_STOP_GRACE_PERIOD=90s

Compose forwards the apply flag and configurable stop grace (legacy default remains 240s). Generic PID 1 cleanup has one shared 80-second budget. No external manager, new proof importer, SQL frontend surrogate or parallel journal exists. PID 1 starts owned SS/DoH, reconciles known bytes, launches the acknowledged managed resource-apply --resource-config FILE child before the checks-only monitor, and starts the native daemon only with verified committed config. This is one finite apply invocation per startup cycle, not exactly once per container: PID 1 can retry cycles under its existing restart budget. Identical committed requests resume by authenticated refs without reapplying the transaction. The command refuses standalone/docker-exec callers: actual PID 1 environment, entrypoint, namespace, direct-child executable/argv and held launch ACK are required, plus the existing controller singleton and no pre-existing NFQ worker.

First apply supports disabled baseline → generic only. Active legacy profiles and different generic plan rotation are refused; no maintenance interruption is implemented. Every configured resource must require both TLS protocols and every assigned exact host must have complete, candidate-bound passing proof for all its resources across all input labels. New apply requires completion within 24 hours, never in the future. Negative/unknown/unassigned hosts cannot authorize apply. Registry definitions may still describe single-TLS proposals, but native apply rejects them instead of widening unsupported TLS-version selectors.

Canonical ordered args/selectors define profiles separated by --new; each contains --filter-tcp=443 --filter-l7=tls --hostlist-domains=^a,^b. Exact-host semantics use the pinned native source https://raw.githubusercontent.com/bol-van/zapret2/4df94497869e18ed4fdf1272141427b2fbfc6604/nfq2/hostlist.c: inline AppendHostlistItem uses addpool, leading ^ sets HOSTLIST_POOL_FLAG_STRICT_MATCH, and SearchHostList rejects suffix fallback when that flag is set and !bHostFull. Existing YouTube suffix policy is unchanged. The packaged executable/Lua hash must match private attestation; held native, Lua, source, marker and proof identities/bytes are guarded through the transaction. No active queue version probe or mark/filter change is made (DoH500/probe100/live400).

The existing journal gains v4 independently tagged committed/pending refs (baseline, legacy, generic). Generic refs exclude legacy IDs/scopes and bind exact authenticated registry/matrix envelope hashes, source fingerprint, internally derived canonical assignment digest and complete config hash. Unknown versions/tags, mixed refs and unresolved historical reconstruction block without adopting or erasing foreign state; v1–v3 legacy behavior remains. Staging/backup/prepared/replace/fixed restart/health/commit reuse the same engine. All required resources/TLS are checked through the existing synchronous owned SS/DoH collector (all-answer SSRF validation, CONNECT pin, SNI/cert validation, private stdin). No Player API health or public check JSON authorizes commit.

Committed identical requests are idempotent: historical proof expiry alone does not trigger reload, reapply or rejection on restart. Authenticated integrity, source and native drift still block. Prepared recovery never promotes pending bytes, including after successful health: it restores prior bytes and reloads. Generic rollback additionally checks all resources before finalizing the journal. Strict first-slice ceiling: a DPI-broken disabled baseline can legitimately fail these checks; restored bytes then remain prepared/BLOCKED until operator repair. Physical restoration is not falsely reported as healthy rollback. Legacy rollback has no new health requirement. Cancellation joins synchronous checks before rollback; PID 1 bounds the owned session and retains prepared state on unsafe/unfinished cleanup. ActiveCandidates refuses generic deployments; managed/resource status validates generic refs without inventing legacy groups.

Local root fixture tests cover the production snapshot factory and shared engine; independent local review passed. Actual native/network/PID 1 apply acceptance is not run. The next gate is isolated bounded remote acceptance, not live rollout.

Bounded remote acceptance — source 5cdc0cd

The isolated producer/registry/matrix/planner slice is bounded-qualified. Independent privileged evidence audits passed the observed positive, fault and active-state refusal gates. Full remote concurrency acceptance is PARTIAL; this is not generic apply, active-profile maintenance or service READY.

Root-private proof references (retained reports, not logs copied into this doc):

  • /home/agent/zapret2-autotune-matrix-5cdc0cd/execution-free-private/execution-free-report.md
  • /home/agent/zapret2-autotune-matrix-5cdc0cd/fault-final-private/qualification-fault-report.md
  • /home/agent/zapret2-autotune-matrix-5cdc0cd/refusal-gap-private/remaining-refusal-reports.md

Observed positive and fault gates

  • The execution-free private batch made 60 actual transfers: two candidates, five resources, both TLS protocols, three attempts per cell. Its 20 aggregated cells contain 16 PASS and four expectation_failed: mandatory resource d's HTTP 500 fails for both candidates/protocols, leaving d unassigned. Exact hosts a/b/c receive the greedy multidisorder proposal with authentic multisplit alternatives. Input groups remain metadata, not assignment boundaries; there is no generic suffix coverage or automatic apply.
  • Independent verification authenticated the exact stored payload HMAC, not a reserialized JSON equivalent. Distinct native arguments/worker-argv hashes, executable/Lua/rules evidence, queue 200 and NFQ 1.0.2 attestations bind the two joined/restored runs. Disabled baseline bytes stayed unchanged; sampled workers, queue and temporary NFT rules were gone. Official PID 1 shutdown exited 0 in 1.31s; foreign inspect hashes stayed unchanged.
  • Mid-batch cancellation and effective semantic config drift each stopped after two requests, published no matrix and preserved the native baseline. Official PID 1 exits were 0 in 2.46s and 0.44s respectively; the deliberately changed config source was retained, not silently rolled back.
  • Unsafe Lua/queue/fwmark/user/file definitions and evidence/schema imports were rejected without transfers. Oversized capacity was rejected before worker launch; unauthorized isolated startup was refused. Corrupt MAC blocked planning while preserving payload bytes; changed effective config produced unknown/unassigned output without changing the original authenticated matrix. Immutable catalogue replacement and completed-matrix retest were refused without transfers or overwriting original state.
  • A genuine archived v3 committed journal with two candidates passed official reconcile/ActiveCandidates validation before qualification refused active profile state. Profile/config, journal, last-good and target identity remained byte-identical; zero transfers/workers and no matrix. Actual PID 1 exit 78 was an expected early preflight refusal, not an exit-0 cleanup ACK. This did not activate or maintain a running active-profile NFQ daemon; that remains unsupported. Whole-DB byte preservation is not claimed.

Explicit limits and testing lesson

  • Remote qualifier singleton conflict is NOT RUN. The earlier planted-lock fixture failed initialization before that branch and does not count. The shared-lock unit second-acquire EWOULDBLOCK check passed, but is not remote qualifier concurrency acceptance. No deterministic official lifecycle seam exists for this bounded test; no test framework or racy injection was added.
  • Natural 24-hour remote expiry is NOT RUN. The existing unit check rejects a structurally valid 25-hour-old proof; this is not observed remote expiry.
  • Privacy sentinel absence was checked in tested logs/state and sampled fault/ refusal argv. Positive active argv and exhaustive DNS packet-level privacy were not proved. Configured/observed encrypted DNS is not exhaustive proof against transient fallback.
  • Compatibility on this baseline is not causal DPI-desync benefit. Historical worker-scan EACCES origin remains unknown; nonrecurrence in the execution-free batch does not identify its cause. Earlier harness failures remain recorded in the referenced reports, not erased or relabeled as product acceptance.
  • Stage all inputs before target startup. From startup through qualification completion, prohibit target exec, cp and nsenter (even passive exec starts a process). Observe host /proc metadata and Docker logs/inspect only; inspect runtime and run plan after finalization/cleanup. The explicit official stop used for cancellation is the documented exception, not an observer exec.
  • Owned test containers/network were removed and private evidence retained. Foreign SS1080 stayed running and legacy11082 stayed stopped with exit 78. No live upgrade or push occurred; unresolved legacy exit 78, YouTube/image readiness, transactional generic apply/profile-journal migration, immutable rotation and noninterrupting search remain outside this acceptance.