Latch explicit resource apply once per PID 1 lifetime. Fail closed before launch and permanently pause after authoritative cleanup on failed or incomplete apply; retain reconciliation on successful monitor and service restarts. Cover deterministic 180-second retry windows, launch/ACK/child/session/service failures, cancellation, successful restarts and fresh-PID1 flags. Document RAM-only reset limits and repair-only startup; no remote health acceptance is claimed. Co-authored-by: GPT-6 <gpt-6@openai.com>
324 lines
20 KiB
Markdown
324 lines
20 KiB
Markdown
# Resource compatibility planner and isolated qualifier
|
||
|
||
`plan` remains **proposal-only** and never changes runtime or config.
|
||
Explicit generic startup apply is implemented and independently reviewed locally;
|
||
remote acceptance is pending. This is not automatic recovery or production READY.
|
||
The per-PID-1 one-shot latch correction passed independent local review only.
|
||
Explicit isolated qualification temporarily installs an owned NFQ worker and
|
||
queue rules, but never changes legacy SQL groups, renderer or apply journals.
|
||
Monitor HEALTHY, old SQL measurements and public check JSON are not proof.
|
||
Without a completed owned qualification batch, compatibility is UNKNOWN and
|
||
hosts are unassigned. No PASS/evidence import is exposed.
|
||
|
||
Commands require root and the existing `managed-test` target settings:
|
||
|
||
```
|
||
zapret-autotune strategy-import --strategies /private/strategies.json
|
||
zapret-autotune plan --config /private/resources.json
|
||
zapret-autotune qualify --config /private/resources.json --strategies /private/strategies.json --isolated
|
||
```
|
||
|
||
`strategy-import` requires an owned, regular, single-link 0600 file. Only
|
||
definitions are accepted; duplicate keys, unknown fields and evidence fields
|
||
are rejected. Example:
|
||
|
||
```json
|
||
{
|
||
"version": 1,
|
||
"strategies": [
|
||
{
|
||
"id": "split-one",
|
||
"args": "--payload=tls_client_hello --lua-desync=multisplit:pos=1",
|
||
"selectors": {"transport":"tcp","ports":"443","l7":"tls","tls":"both"}
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
The registry is one immutable batch (1–32 unique IDs). This conservative slice
|
||
accepts only TCP/443/TLS selectors and `tls12`, `tls13`, or `both` policy.
|
||
Argument whitespace is canonicalized; option/Lua stage order is preserved.
|
||
Existing payload/range validation is reused. Lua stages are restricted to
|
||
fake, multisplit, multidisorder, fakeddisorder, split and disorder; parameters
|
||
to pos, repeats, tcp_ts, tcp_flags_set, seqovl and blob=fake_default_tls.
|
||
File/inline Lua, queue/user/fwmark/filter options and arbitrary stages are
|
||
rejected. Acceptance of a definition is **not** runtime verification.
|
||
|
||
The private target directory holds `.resource-plan-key` (32 random bytes),
|
||
`.resource-plan-strategies.json` and, when an owned producer exists,
|
||
`.resource-plan-matrix.json`. JSON envelopes have `payload` and an HMAC-SHA256
|
||
`mac`. Each file is write-once, owned, 0600, no-follow, single-link and bounded
|
||
to 64 KiB. Held directory/file identities and exact snapshots are checked;
|
||
foreign entries, replacement inodes and modified bytes are never clobbered or
|
||
silently adopted. Interrupted writes remain blocked for explicit operator
|
||
repair. There is no automatic rotation/update/repair in this slice. The private
|
||
key and same-UID/root boundary must be protected: HMAC is not protection from an
|
||
actor who can read the key and replace the whole target state between processes.
|
||
|
||
Matrix v1 binds the registry digest and at most one completed run per candidate.
|
||
Each run binds ID, definition digest (canonical args **and selectors**), keyed
|
||
config fingerprint, start/completion milliseconds, resource/TLS outcomes and
|
||
keyed resource fingerprints. Its attestation records collector/runtime,
|
||
definition digest, composite executable/Lua input hash, installed rules hash,
|
||
actual worker-argv hash, queue and worker PID/start identity, plus
|
||
restored/joined completion. No URLs, headers, resolved IPs, bodies or raw errors
|
||
are persisted. A nonserialized producer seal protects the completed in-memory
|
||
snapshot against mutation before publication. Only the owned producer and
|
||
test-internal fixtures create seals; JSON DTOs cannot authorize publication.
|
||
Every resource/protocol cell records exactly three actual attempts; any nonpass
|
||
in those attempts makes the cell negative (never an average or partial PASS).
|
||
|
||
`plan` loads only authenticated private state and the existing resource-config
|
||
snapshot (including resolved environment secrets, DNS/proxy and private headers
|
||
in keyed fingerprints). Every resource on an exact host, across all services
|
||
and input group labels, must have matching complete results for every required
|
||
TLS protocol. Complete failed checks are negative; missing, expired (24 hours),
|
||
future, mismatched or partial checks are unknown. Changed config invalidates old
|
||
run evidence. Malformed/unauthenticated state yields blocked, unassigned output
|
||
and exit 78; valid missing evidence yields unknown output and exit 0.
|
||
|
||
Output contains `proposal_only`, `evidence` (authentication status, **not**
|
||
health), per-host `validated`/`negative`/`unknown` candidate alternatives,
|
||
`assignments` and `unassigned`. Compatible equal args/selectors merge coverage
|
||
across input labels. Greedy greatest-uncovered coverage, then lexical candidate
|
||
ID, produces deterministic disjoint exact-host assignments. It does not claim
|
||
global optimality or suffix coverage. Explicit legacy YouTube suffix policy is
|
||
unchanged.
|
||
|
||
## INTRUSIVE / ISOLATED startup only
|
||
|
||
Set **both** `ZAPRET_AUTOTUNE_QUALIFY_CONFIG` and
|
||
`ZAPRET_AUTOTUNE_STRATEGY_CONFIG` to container paths on a dedicated target.
|
||
Do not send working user traffic to this target's published SOCKS port. This is
|
||
not noninterrupting background search. Provision the private resource and 0600
|
||
strategy files separately; compose forwards paths but does not mount files for
|
||
you. Start with the managed disabled baseline, not an existing active profile.
|
||
Qualification v1 accepts lowercase DNS hostnames, not literal IP targets; the
|
||
ordinary resource-check command's broader literal-IP policy is unchanged.
|
||
|
||
PID 1 starts its owned SS+DoH services, initializes/reconciles known state, then
|
||
launches one finite `qualify ... --isolated` session **instead of** starting the
|
||
live zapret daemon or monitor. Qualifier checks root/target namespace, PID 1's
|
||
actual entrypoint/environment, direct-child executable/arguments/session ACK,
|
||
owned SS and DNS readiness anchors. A flag or docker-exec environment alone
|
||
does not authorize qualification. Missing either startup path fails closed.
|
||
Without `--isolated`, `qualify` remains UNKNOWN/exit 78 without runtime changes.
|
||
|
||
The shared monitor singleton excludes legacy and generic controllers. Any
|
||
existing NFQ worker/queue, active profile, unknown config/journal or existing
|
||
immutable matrix is refused. Known baseline config/journal/backup/identity
|
||
inodes and bytes remain held unchanged throughout the job. This first producer
|
||
does **not** pause/stop/restart existing active profiles; job-bound maintenance
|
||
rollback is deferred rather than pretending a path-only daemon stop is safe.
|
||
|
||
Each candidate uses the existing worker-before-nft/queue-ready lifecycle on
|
||
queue 200, fixed Lua imports and fwmark 0x10000000, plus native
|
||
`--filter-l7=tls`. The actual child executable, argv, parent, start identity,
|
||
queue socket ownership, executable/Lua hashes, native `--version` output 1.0.2
|
||
and installed nft rule bytes are attested before capture and checked again at
|
||
each collector guard. DoH's existing 0x50000000 socket mark bypass remains.
|
||
TLS policy is collector/eligibility policy, not a nonexistent native TLS-version
|
||
filter. All configured protocols are measured; incompatible selector policies
|
||
remain UNKNOWN in the proposal.
|
||
|
||
The existing GET/HEAD generic collector preserves TLS origin verification,
|
||
configured body/status/content-type expectations, private stdin and IP pinning.
|
||
Completed expectation failures and typed peer response/TLS/transport negatives
|
||
do not stop the remaining repeats/cells. Guard/source/unknown infrastructure or
|
||
batch cancellation veto the entire batch; no partial positive matrix is saved.
|
||
Config (including resolved environment/header secrets) and strategy-source
|
||
snapshots are checked after DNS/before transfer and before sealing/publication.
|
||
Workers are joined and temporary nft queues removed before verifying unchanged
|
||
disabled baseline/runtime and sealing. The batch has a fixed 15-minute deadline
|
||
with separate bounded cleanup; PID 1 keeps SS+DoH alive through cleanup.
|
||
The bounded 64 KiB matrix size (including its authenticated envelope and
|
||
worst-case field widths) is checked before any worker/nft operation; excessive
|
||
catalogue/resource combinations return UNKNOWN/exit 64 even within individual
|
||
count caps (32 strategies, 64 resources).
|
||
|
||
Successful batches, including complete negatives, return the assignment proposal.
|
||
Incomplete/unsafe jobs return UNKNOWN/blocked exit 78 without proof. PID 1 then
|
||
stays safe-idle with SS+DoH for inspection, for either success or blocked
|
||
completion: no qualification retry loop, duplicate controller or automatic
|
||
generic monitor start. Retests require future explicit immutable snapshot
|
||
rotation, not overwriting prior evidence.
|
||
|
||
Completed controller cleanup is acknowledged separately from qualification in
|
||
an exclusive, 0600 HMAC-authenticated `qualification-completed.json` on the
|
||
current boot's SS tmpfs. It binds the launched PID, exit outcome (0 or 78) and
|
||
verified cleanup; it is **not** strategy proof. PID 1 checks that ACK against its
|
||
private session launch and revalidates the disabled baseline, singleton,
|
||
SS/DoH ownership and absence of NFQ workers/temporary nft queues before shutting
|
||
down dependencies. Thus TERM after a clean blocked qualifier has exited but
|
||
before its parent reaps it can exit cleanly. Worker/nft/join/restoration failure,
|
||
missing/foreign/mutated ACK or unverified early failure still veto clean shutdown
|
||
with exit 78. Ordinary legacy/generic monitor exit 78 remains a safety veto.
|
||
|
||
No unsupported `--filter-mark` or blockcheck embedded/stream/progress/deadline
|
||
flags are used. Background noninterrupting search, active-profile maintenance
|
||
and immutable snapshot rotation for repeated runs remain deferred.
|
||
|
||
## Explicit generic transactional startup apply (local implementation)
|
||
|
||
On the target with its already completed private qualification batch, unset both
|
||
qualification startup paths and set:
|
||
|
||
```
|
||
ZAPRET_AUTOTUNE_RESOURCE_CONFIG=/private/resources.json
|
||
ZAPRET_AUTOTUNE_RESOURCE_APPLY=1
|
||
ZAPRET_AUTOTUNE_STOP_GRACE_PERIOD=90s
|
||
```
|
||
|
||
Compose forwards the apply flag and configurable stop grace (legacy default
|
||
remains 240s). Generic PID 1 cleanup has one shared 80-second budget. No external
|
||
manager, new proof importer, SQL frontend surrogate or parallel journal exists.
|
||
PID 1 starts owned SS/DoH, reconciles known bytes, launches the acknowledged
|
||
`managed resource-apply --resource-config FILE` child before the checks-only
|
||
monitor, and starts the native daemon only with verified committed config.
|
||
This is at most one apply launch per PID 1 lifetime. A RAM latch is marked failed
|
||
before launch and succeeds only after child wait=0 and owned-session cleanup.
|
||
Launch/ACK failure, nonzero/signal exit, SS/DoH loss, or incomplete cleanup leaves
|
||
it failed: after authoritative cleanup PID 1 remains cancellably BLOCKED, without
|
||
restarting SS/DoH, native zapret, apply or monitoring, even across former 60s retry
|
||
windows. Cleanup still stops SS/DoH (DNS last); BLOCKED does not keep them alive.
|
||
Successful apply is skipped on later monitor/SS restarts, but every runtime cycle
|
||
still reconciles committed refs before native start. Qualification and legacy
|
||
retry behavior are unchanged.
|
||
|
||
**Ceiling:** this is not a durable manual-intent latch. A fresh PID 1 (including
|
||
an external Docker restart policy) resets authorization and, with flag=1, permits
|
||
one new attempt. Operators must disable unattended external restart/recreation
|
||
for this workflow. Set `ZAPRET_AUTOTUNE_RESOURCE_APPLY=0` for explicit repair-only
|
||
startup: prepared state is reconciled, never promoted or reapplied. Identical
|
||
committed requests still resume by authenticated refs without a new transaction.
|
||
The command refuses standalone/docker-exec callers: actual PID 1 environment,
|
||
entrypoint, namespace, direct-child executable/argv and held launch ACK are
|
||
required, plus the existing controller singleton and no pre-existing NFQ worker.
|
||
|
||
First apply supports **disabled baseline → generic only**. Active legacy profiles
|
||
and different generic plan rotation are refused; no maintenance interruption is
|
||
implemented. Every configured resource must require `both` TLS protocols and
|
||
every assigned exact host must have complete, candidate-bound passing proof for
|
||
all its resources across all input labels. New apply requires completion within
|
||
24 hours, never in the future. Negative/unknown/unassigned hosts cannot authorize
|
||
apply. Registry definitions may still describe single-TLS proposals, but native
|
||
apply rejects them instead of widening unsupported TLS-version selectors.
|
||
|
||
Canonical ordered args/selectors define profiles separated by `--new`; each
|
||
contains `--filter-tcp=443 --filter-l7=tls --hostlist-domains=^a,^b`. Exact-host
|
||
semantics use the pinned native source
|
||
`https://raw.githubusercontent.com/bol-van/zapret2/4df94497869e18ed4fdf1272141427b2fbfc6604/nfq2/hostlist.c`:
|
||
inline `AppendHostlistItem` uses `addpool`, leading `^` sets
|
||
`HOSTLIST_POOL_FLAG_STRICT_MATCH`, and `SearchHostList` rejects suffix fallback
|
||
when that flag is set and `!bHostFull`. Existing YouTube suffix policy is unchanged.
|
||
The packaged executable/Lua hash must match private attestation; held native,
|
||
Lua, source, marker and proof identities/bytes are guarded through the transaction.
|
||
No active queue version probe or mark/filter change is made (DoH500/probe100/live400).
|
||
|
||
The **existing** journal gains v4 independently tagged committed/pending refs
|
||
(`baseline`, `legacy`, `generic`). Generic refs exclude legacy IDs/scopes and bind
|
||
exact authenticated registry/matrix envelope hashes, source fingerprint,
|
||
internally derived canonical assignment digest and complete config hash.
|
||
Unknown versions/tags, mixed refs and unresolved historical reconstruction block
|
||
without adopting or erasing foreign state; v1–v3 legacy behavior remains.
|
||
Staging/backup/prepared/replace/fixed restart/health/commit reuse the same engine.
|
||
All required resources/TLS are checked through the existing synchronous owned
|
||
SS/DoH collector (all-answer SSRF validation, CONNECT pin, SNI/cert validation,
|
||
private stdin). No Player API health or public check JSON authorizes commit.
|
||
|
||
Committed identical requests are idempotent: historical proof expiry alone does
|
||
not trigger reload, reapply or rejection on restart. Authenticated integrity,
|
||
source and native drift still block. Prepared recovery **never** promotes pending
|
||
bytes, including after successful health: it restores prior bytes and reloads.
|
||
Generic rollback additionally checks all resources before finalizing the journal.
|
||
**Strict first-slice ceiling:** a DPI-broken disabled baseline can legitimately
|
||
fail these checks; restored bytes then remain `prepared`/BLOCKED until operator
|
||
repair. Physical restoration is not falsely reported as healthy rollback.
|
||
Legacy rollback has no new health requirement. Cancellation joins synchronous
|
||
checks before rollback; PID 1 bounds the owned session and retains prepared state
|
||
on unsafe/unfinished cleanup. `ActiveCandidates` refuses generic deployments;
|
||
managed/resource status validates generic refs without inventing legacy groups.
|
||
|
||
Local root fixture tests cover the production snapshot factory and shared engine;
|
||
independent local review passed. Actual native/network/PID 1 apply acceptance is
|
||
**not run**. The next gate is isolated bounded remote acceptance, not live rollout.
|
||
The per-PID-1 latch correction passed independent local review and deterministic
|
||
180-second simulated retry-window fixtures; no remote PASS is claimed for it.
|
||
Roadmap: authorized isolated acceptance; no remote deployment is authorized by
|
||
local fixtures.
|
||
|
||
## Bounded remote acceptance — source `5cdc0cd`
|
||
|
||
The isolated producer/registry/matrix/planner slice is **bounded-qualified**.
|
||
Independent privileged evidence audits passed the observed positive, fault and
|
||
active-state refusal gates. **Full remote concurrency acceptance is PARTIAL**;
|
||
this is not generic apply, active-profile maintenance or service READY.
|
||
|
||
Root-private proof references (retained reports, not logs copied into this doc):
|
||
|
||
- `/home/agent/zapret2-autotune-matrix-5cdc0cd/execution-free-private/execution-free-report.md`
|
||
- `/home/agent/zapret2-autotune-matrix-5cdc0cd/fault-final-private/qualification-fault-report.md`
|
||
- `/home/agent/zapret2-autotune-matrix-5cdc0cd/refusal-gap-private/remaining-refusal-reports.md`
|
||
|
||
### Observed positive and fault gates
|
||
|
||
- The execution-free private batch made 60 actual transfers: two candidates,
|
||
five resources, both TLS protocols, three attempts per cell. Its 20 aggregated
|
||
cells contain 16 PASS and four `expectation_failed`: mandatory resource d's
|
||
HTTP 500 fails for both candidates/protocols, leaving d unassigned. Exact
|
||
hosts a/b/c receive the greedy multidisorder proposal with authentic multisplit
|
||
alternatives. Input groups remain metadata, not assignment boundaries;
|
||
there is no generic suffix coverage or automatic apply.
|
||
- Independent verification authenticated the exact stored payload HMAC, not a
|
||
reserialized JSON equivalent. Distinct native arguments/worker-argv hashes,
|
||
executable/Lua/rules evidence, queue 200 and NFQ 1.0.2 attestations bind the
|
||
two joined/restored runs. Disabled baseline bytes stayed unchanged; sampled
|
||
workers, queue and temporary NFT rules were gone. Official PID 1 shutdown
|
||
exited 0 in 1.31s; foreign inspect hashes stayed unchanged.
|
||
- Mid-batch cancellation and effective semantic config drift each stopped
|
||
after two requests, published no matrix and preserved the native baseline.
|
||
Official PID 1 exits were 0 in 2.46s and 0.44s respectively; the deliberately
|
||
changed config source was retained, not silently rolled back.
|
||
- Unsafe Lua/queue/fwmark/user/file definitions and evidence/schema imports
|
||
were rejected without transfers. Oversized capacity was rejected before
|
||
worker launch; unauthorized isolated startup was refused. Corrupt MAC blocked
|
||
planning while preserving payload bytes; changed effective config produced
|
||
unknown/unassigned output without changing the original authenticated matrix.
|
||
Immutable catalogue replacement and completed-matrix retest were refused
|
||
without transfers or overwriting original state.
|
||
- A genuine archived v3 committed journal with two candidates passed official
|
||
`reconcile`/`ActiveCandidates` validation before qualification refused active
|
||
profile state. Profile/config, journal, last-good and target identity remained
|
||
byte-identical; zero transfers/workers and no matrix. Actual PID 1 exit 78 was
|
||
an expected early preflight refusal, **not** an exit-0 cleanup ACK. This did
|
||
not activate or maintain a running active-profile NFQ daemon; that remains
|
||
unsupported. Whole-DB byte preservation is not claimed.
|
||
|
||
### Explicit limits and testing lesson
|
||
|
||
- Remote qualifier singleton conflict is **NOT RUN**. The earlier planted-lock
|
||
fixture failed initialization before that branch and does not count. The
|
||
shared-lock unit second-acquire `EWOULDBLOCK` check passed, but is not remote
|
||
qualifier concurrency acceptance. No deterministic official lifecycle seam
|
||
exists for this bounded test; no test framework or racy injection was added.
|
||
- Natural 24-hour remote expiry is **NOT RUN**. The existing unit check rejects
|
||
a structurally valid 25-hour-old proof; this is not observed remote expiry.
|
||
- Privacy sentinel absence was checked in tested logs/state and sampled fault/
|
||
refusal argv. Positive active argv and exhaustive DNS packet-level privacy
|
||
were not proved. Configured/observed encrypted DNS is not exhaustive proof
|
||
against transient fallback.
|
||
- Compatibility on this baseline is not causal DPI-desync benefit. Historical
|
||
worker-scan EACCES origin remains unknown; nonrecurrence in the execution-free
|
||
batch does not identify its cause. Earlier harness failures remain recorded
|
||
in the referenced reports, not erased or relabeled as product acceptance.
|
||
- Stage all inputs before target startup. From startup through qualification
|
||
completion, prohibit target `exec`, `cp` and `nsenter` (even passive exec starts
|
||
a process). Observe host `/proc` metadata and Docker logs/inspect only; inspect
|
||
runtime and run `plan` after finalization/cleanup. The explicit official stop
|
||
used for cancellation is the documented exception, not an observer exec.
|
||
- Owned test containers/network were removed and private evidence retained.
|
||
Foreign SS1080 stayed running and legacy11082 stayed stopped with exit 78.
|
||
No live upgrade or push occurred; unresolved legacy exit 78, YouTube/image
|
||
readiness, transactional generic apply/profile-journal migration, immutable
|
||
rotation and noninterrupting search remain outside this acceptance.
|