Compare commits

..
50 Commits
Author SHA1 Message Date
ZeusinaandGPT-6 Bot 88564b340a fix: include adapter packages in container builds
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 18:25:03 +03:00
ZeusinaandGPT-6 Bot e270810f33 refactor: execute adapter plans through shared core
Execute the private YouTube qualification plan through core-owned scheduling while retaining worker cleanup, proof sealing and legacy policy boundaries. Preserve unsafe collector failures alongside concurrent cancellation or deadlines.

Send bounded Player JSON through stdin instead of argv and update all shared runner callers. Generic controller cutover and remote runtime acceptance remain pending.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 17:25:12 +03:00
ZeusinaandGPT-6 Bot f4853173b3 feat: extract versioned YouTube service adapter
Wire pure Player parsing and strict qualification planning through a compiled-in kind/version registry. Fail closed before discovery requests when the required module is absent or mismatched.

Require sealed module-bound v2 evidence for new writes while preserving fresh trusted v1 apply eligibility and historical rollback. Keep legacy SOCKS5h routing, separate suffix/asset policy and public configuration unchanged; generic controller cutover and field acceptance remain pending.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 16:23:54 +03:00
ZeusinaandGPT-6 Bot d324b4137b docs: record scoped managed v2 verification
Record bounded typed qualification, apply, monitor fault/repair and dated independent cleanup; retain body/live-FD and untested v1/YouTube acceptance limits. Keep adapter registry and version-bound module migration explicitly future work.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 15:33:26 +03:00
ZeusinaandGPT-6 Bot cd05e75189 feat: add typed generic image and MP4 checks
Preserve strict v1 semantics and fingerprints; bind v2 typed expectations to a new proof namespace. Reuse bounded shared collection across checks, monitoring, qualification and apply health. Document prefix-only MP4 and deferred public GIF/plugin/runtime boundaries.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 15:07:54 +03:00
ZeusinaandGPT-6 Bot 9ab9fd73a9 docs: record shared-core parity and rotation fault checks
Record scoped real-curl parity separately from older generic rotation fault evidence and pending managed runtime acceptance. Preserve strict YouTube qualification, refusal attribution limits and the next typed generic image/media capability.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 12:17:00 +03:00
ZeusinaandGPT-6 Bot 45d28e3733 refactor: share strict resource validation through generic core
Use private typed generic, image and MP4 response contracts in the existing bounded HTTPS execution pipeline. Preserve generic mismatch measurements, strict legacy DTOs and producer behavior; reject overflowing Content-Range integers before normalization.

Add injected-wire adapter parity and direct-core regressions. Local targeted EUID-0/race, full Go and config fixtures passed; native acceptance remains pending. The unrelated long Python supervision timeout remains unresolved. No public schema, suffix-learning or controller migration.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 12:04:28 +03:00
ZeusinaandGPT-6 Bot 2149a4abc8 docs: record bounded verified-profile rotation observations
Separate positive A/B/replay/A field observations from missing full native and rotation fault acceptance. Retain evidence provenance, cleanup limits and the next old-policy rollback/prepared-repair gate.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 11:09:49 +03:00
ZeusinaandGPT-6 Bot 8d43dd9363 feat: rotate generic profiles across verified alternatives
Select bounded canonical strategy allowlists at explicit PID1 startup within the same immutable authenticated registry, matrix and source. Reconstruct stored policies independently for historical replay, reconciliation and rollback; preserve freshness for new policies even when rendered bytes match. Retain legacy refusal and one-shot startup guards.

Local fixtures and independent local review passed; live native rotation acceptance remains pending. No uninterrupted rotation, new qualification or proof-store overwrite.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 10:18:14 +03:00
ZeusinaandGPT-6 Bot 5d0be0288f docs: record guarded generic apply fault acceptance
Record independently verified bounded replay, rollback, prepared repair and cancellation gates; retain conservative TERM78, per-PID1 authorization and coverage limits.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 08:21:57 +03:00
ZeusinaandGPT-6 Bot 643d0c6220 fix: prevent automatic retries of failed startup apply
Latch explicit resource apply once per PID 1 lifetime. Fail closed before launch and permanently pause after authoritative cleanup on failed or incomplete apply; retain reconciliation on successful monitor and service restarts.

Cover deterministic 180-second retry windows, launch/ACK/child/session/service failures, cancellation, successful restarts and fresh-PID1 flags. Document RAM-only reset limits and repair-only startup; no remote health acceptance is claimed.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-15 08:05:27 +03:00
ZeusinaandGPT-6 Bot 36aace02da fix: publish controller startup acknowledgements atomically
Publish a complete PID acknowledgement by private temporary file and same-directory rename. Preserve unknown acknowledgement artifacts and remove only retained owned stale state. Keep malformed acknowledgement failures closed.

Cover deterministic publication barriers, shared apply/monitor/qualify launch, foreign artifacts, owned replacement, and private-directory checks. This does not establish hostile root or same-UID writer no-clobber guarantees; the private mode-700 directory is the trust boundary.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 23:00:06 +03:00
ZeusinaandGPT-6 Bot ade3b89137 chore: add private-safe controller startup diagnostics
Add fixed startup branch labels and numeric apply exit status without relaxing startup guards or exposing private values. Cover shared launch/ACK and wrapper branches plus failed apply cleanup. Diagnostics only; the ACK publication race remains unchanged.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 22:37:48 +03:00
ZeusinaandGPT-6 Bot de453ed2c8 feat: add guarded generic startup apply
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 22:02:00 +03:00
ZeusinaandGPT-6 3743cdbd3e docs: record bounded qualification and planner acceptance
Co-authored-by: GPT-6 <noreply@openai.com>
2026-10-14 19:23:40 +03:00
ZeusinaandGPT-6 Bot 5cdc0cd767 fix: diagnose qualification worker scan failures
Report fixed procfs scan reasons, typed errno and numeric worker identities without exposing paths or errors. Preserve scan metadata through attestation and restore normalization while retaining fail-closed decisions and cleanup classification.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 15:40:48 +03:00
ZeusinaandGPT-6 Bot 83921350fd fix: diagnose opaque qualification predicates
Add fixed attestation, factory guard and restoration reasons without changing fail-closed predicates or cleanup order. Retain nested reasons under joined ownership vetoes and distinguish prior cleanup veto from a new restoration failure.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 15:13:32 +03:00
ZeusinaandGPT-6 Bot 63cdc7d490 fix: add privacy-safe qualification failure diagnostics
Report fixed operation and collector reason enums with measurement ordinals, keeping peer-negative classification and cleanup/ACK vetoes unchanged. Preserve original errors without emitting their values.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 14:53:08 +03:00
ZeusinaandGPT-6 Bot 610916ce66 fix: reject concealed qualification alias traversal
Validate the raw pinned package alias before path normalization so symlink/.. traversal cannot attest one executable and launch another. Preserve package ownership, held-file, hash, inode and Lua checks.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 14:19:33 +03:00
ZeusinaandGPT-6 Bot a1a76bff76 fix: attest trusted packaged qualification engine alias
Resolve only the fixed nfq2 package alias into trusted binaries parents, hold the no-follow engine descriptor and verify inode/hash against the running worker before nft activation. Keep Lua/import guards and completed-proof publication unchanged.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 13:57:58 +03:00
ZeusinaandGPT-6 Bot 573da3f797 feat: add candidate-bound domain qualification and planning
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 12:21:52 +03:00
ZeusinaandGPT-6 Bot 2242643000 docs: record generic monitor fault acceptance
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 01:06:19 +03:00
ZeusinaandGPT-6 Bot 5c85f3f0b9 fix: validate live resource monitor ownership in status
Bind authenticated snapshots to controller PID/starttime and validate process identity and held inode anchors before reporting current check health. Preserve identical signed foreign replacements and fail closed when live ownership is unavailable.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-14 00:28:24 +03:00
ZeusinaandGPT-6 Bot 4efc066625 feat: add guarded periodic resource monitoring
Add checks-only scheduling, read-only status, private authenticated persistence and guarded PID1 lifecycle. Keep strategy selection/recovery and remote fault acceptance out of scope.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 23:33:31 +03:00
ZeusinaandGPT-6 cfce8170d7 docs: record declarative resource-check acceptance
Co-authored-by: GPT-6 <noreply@openai.com>
2026-10-13 19:47:25 +03:00
ZeusinaandGPT-6 Bot 4d7c4b33c3 feat: add declarative HTTPS resource checks
Add strict check-only resource configuration, owned DNS resolution and pinned HTTPS probes with bounded private output. Document the schema limits and pending remote acceptance; leave generic controller and strategy migration out of scope.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 18:38:46 +03:00
ZeusinaandGPT-6 Bot 0623fa3323 docs: record DoH outage and live validation
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 16:10:37 +03:00
ZeusinaandGPT-6 Bot 544dda7c85 fix: accept explicit socks5h for one-shot health
Preserve configured proxy DNS semantics while reusing strict URL and loopback validation. Cover proxy forwarding, legacy socks5, and unsafe endpoints with offline tests; document the managed DoH boundary.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 15:47:20 +03:00
ZeusinaandGPT-6 Bot 721cad42ce fix: preserve DNS TCP replies after upstream timeout
Separate bounded TCP read, upstream and write phases while retaining the total connection lifetime. Add net.Pipe regressions for upstream timeout, partial reads, cancellation and lifetime limits.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 15:02:41 +03:00
ZeusinaandGPT-6 Bot 367d956ae3 feat: add configurable PID1-owned DoH forwarding
Default managed SOCKS hostname DNS to verified HTTPS with literal-IP bootstrap and no plaintext fallback. Supervise the loopback forwarder with private anchored readiness and preserve foreign markers. Include configuration, deployment, documentation and offline checks.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 14:32:58 +03:00
ZeusinaandGPT-6 Bot ea886e5bc8 docs: record autonomous three-CDN qualification acceptance
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 11:32:15 +03:00
ZeusinaandGPT-6 Bot 3fc8563995 feat: discover and qualify Googlevideo CDNs autonomously
Retain strict three-host native qualification, discover fresh representative inventory automatically, and recheck deployed media before promotion. Document offline-only verification and outstanding field gates.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-13 10:12:10 +03:00
ZeusinaandGPT-6 Bot 9984f21466 docs: record final managed RC field acceptance and remaining gates
Record supplied fd25cb8 managed-image watch, representative post-warmup media transfers and degraded recovery continuation without claiming avatar or durable family qualification. Require explicit Dockerfile.target and positive entrypoint/marker checks before image promotion; preserve the user README badge.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 23:34:38 +03:00
ZeusinaandGPT-6 Bot fd25cb8939 fix: bind experimental rechecks to recovery transaction hooks
An isolated SHA-verified replay reproduces exit 78 at managed_experimental_recovery.go:148 with the nested required-hooks guard at managed_transaction.go:69. The production recheck closure retained the adapter path-only transaction instead of the job-bound transaction. Pass the current transaction explicitly without weakening ownership, Lua, runtime or reconciliation guards.

Regression starts recovery with path-only hooks and checks bound ownership for cold/adaptive rechecks. Full native Go tests, affected race tests, go vet and diff checks passed. Corrected image replay and production restoration remain unverified.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 23:02:19 +03:00
ZeusinaandGPT-6 Bot 1afee7d3bd fix: resolve managed probes through owned SS DNS
Canonicalize validated managed proxy schemes to socks5h for Player, metadata, resource and warmup probes; retain listener ownership, TLS/privacy checks and legacy CLI/pinned checker semantics. Stop canceled inventory requests before Player and metadata fallback, with explicit cancellation and runner-only DNS regressions.

Verification: full native Go tests, affected race tests, go vet and static/config/supervision fixtures passed. An extra whole-package race attempt exceeded its 180s deadline in unrelated TestTuneCLI; affected race coverage completed separately.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 22:28:13 +03:00
ZeusinaandGPT-6 Bot e9a28c8cd3 fix: keep rejected image payloads recoverable across verification
Type decoder payload rejections once and share the image nonpass classifier across live health and owned verification. Retain strict qualification, malformed metadata, source-kind separation and mixed safety vetoes; exercise seed fallback, joined workers, fresh evidence and durable absence cooldown.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 22:09:11 +03:00
ZeusinaandGPT-6 Bot 2804d97063 fix: classify resource peer TLS rejections as recoverable damage
Keep verified curl peer TLS negatives unqualified without making them process-wide safety failures. Preserve local proxy/configuration, guard, cleanup and mixed cancellation vetoes; test owned verification, rollback and durable absence cooldown.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 21:55:10 +03:00
ZeusinaandGPT-6 Bot 50a78608ce fix: discover avatar metadata from trusted watch pages
Fall back only on missing homepage metadata, preserve private bounded inventory and fail-closed error classification. Record supplied field results without claiming image or family qualification.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 21:21:19 +03:00
ZeusinaandGPT-6 Bot a60f62de22 feat: recover experimental Googlevideo media autonomously
Preserve operator-owned profiles while rechecking fresh media and bounded native relearning in the deployed daemon. Keep mixed safety failures fatal and expose active coverage separately from qualification. Document source-only acceptance and retain existing recovery budgets and privacy boundaries.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 20:19:16 +03:00
Zeusina 1a35cb3019 docs: add warning vibecoded badge to readme 2026-10-12 11:51:50 +03:00
ZeusinaandGPT-6 Bot ac4d2b73ab docs: record browser-verified Googlevideo suffix RC
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 11:41:34 +03:00
ZeusinaandGPT-6 Bot eade051c2b fix(managed): guard experimental database writes before opening store
Reuse managedMonitorDatabase to validate the canonical target-owned database and private files/sidecars before openStore can create, chmod or migrate anything. Keep ordinary history read-only and experimental authorization unchanged. Cover unsafe paths, links, modes, owners and sidecars plus valid existing stores.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 07:57:16 +03:00
ZeusinaandGPT-6 Bot 53e59688df fix(managed): open writable store for experimental candidate insertion
Use a dedicated writable database dependency only for guarded experimental insertion. Ordinary managed history remains read-only; discovery and authorization gates are unchanged. Reproduced SQLite readonly failure in an isolated copy with private startup logs retained and the restored live target untouched.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 07:43:28 +03:00
ZeusinaandGPT-6 Bot 117441c04f fix(managed): enforce experimental CLI policy under transaction lock
Bind ordinary candidate-ID CLI activation to a transaction-local prohibition on experimental policy. Resolve and reject both requested and inherited experimental scope while holding the journal lock, leaving guarded operator activation and trusted monitor/reconciliation behavior unchanged. Cover deterministic operator-policy interleaving without timing-dependent concurrency.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 07:23:22 +03:00
ZeusinaandGPT-6 Bot f88c8e9c45 fix(managed): guard experimental authorization and startup requests
Reject explicit candidate-ID and inherited-policy CLI bypasses of the guarded native-circular operator path. Validate empty private runtime-owned request inodes without following links before state mutation, atomically claim and sync them before apply, retain failed claims, and never delete replacement requests. Keep internal journal reconciliation and legacy qualification unchanged.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 07:14:20 +03:00
ZeusinaandGPT-6 Bot 371382c2c4 feat(managed): add explicit experimental Googlevideo suffix policy
Keep operator-owned native circular extrapolation distinct from cold/adaptive qualification and playback readiness. Preserve qualified frontend/assets, use guarded journal transactions and rollback, and apply private startup requests before the owned monitor begins. Retain legacy rendering and qualification gates.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-12 07:04:12 +03:00
ZeusinaandGPT-6 Bot 0398e6fe28 fix(managed): guard dropped-UID queue ownership inspection
Require effective SYS_PTRACE only for native owned workers and distinguish denied FD inspection from foreign queue ownership. Keep inode proof authoritative and constrain deployment to its private PID namespace.

Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-11 21:48:06 +03:00
ZeusinaandGPT-6 Bot 18038e4112 fix: propagate transaction hooks into nested managed recovery
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-11 20:36:30 +03:00
ZeusinaandGPT-6 Bot b4e1e1e9ca feat: add autonomous YouTube resource recovery and adaptive GGC
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-11 20:15:16 +03:00
ZeusinaandGPT-6 Bot 8826c591dc docs: record clean RC bootstrap and lifecycle acceptance
Co-authored-by: GPT-6 <gpt-6@openai.com>
2026-10-11 10:59:25 +03:00
125 changed files with 25027 additions and 326 deletions

No files matched your search

+2
View File
@@ -3,6 +3,8 @@
!go.sum
!cmd/
!cmd/**
!internal/
!internal/**
!Dockerfile
!autotune-entrypoint.sh
!Dockerfile.target
+1
View File
@@ -4,6 +4,7 @@ ENV CGO_ENABLED=0 GOTOOLCHAIN=local
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/ ./cmd/
COPY internal/ ./internal/
RUN --mount=type=cache,target=/root/.cache/go-build go build -trimpath -o /out/zapret-autotune ./cmd/zapret-autotune
FROM vernette/ss-zapret2@sha256:ba158180f27cd71a291ee521c291490374fd3fcf2f81eb1434ddfb19d35de45b
+5
View File
@@ -4,6 +4,7 @@ ENV CGO_ENABLED=0 GOTOOLCHAIN=local
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/ ./cmd/
COPY internal/ ./internal/
RUN --mount=type=cache,target=/root/.cache/go-build go build -trimpath -o /out/zapret-autotune ./cmd/zapret-autotune
FROM vernette/ss-zapret2@sha256:ba158180f27cd71a291ee521c291490374fd3fcf2f81eb1434ddfb19d35de45b
@@ -17,6 +18,10 @@ RUN if [ ! -e /opt/zapret2/lua ] && [ ! -L /opt/zapret2/lua ]; then \
fi \
&& test -d /opt/zapret2/lua \
&& command -v setsid \
&& command -v stat \
&& test "$(/usr/local/bin/zapret-autotune dns-forwarder mode)" = doh \
&& test "$(/usr/local/bin/zapret-autotune dns-forwarder address)" = 127.0.0.1:5353 \
&& (ss-server -h 2>&1 | grep -q -- '-d <addr>') \
&& mkdir -p /etc/zapret-autotune /var/lib/zapret-autotune/target \
&& printf 'zapret-autotune-managed-v1\n' > /etc/zapret-autotune/target-id \
&& chmod 700 /var/lib/zapret-autotune /var/lib/zapret-autotune/target
+34
View File
@@ -0,0 +1,34 @@
# Bounded representative Googlevideo discovery
The YouTube default inventory is the configured primary plus `aqz-KE-bpKQ` and
`jNQXAC9IVRw`, deduplicated. Explicit configured inventories override it. The
inventory (one to five IDs) is queried freshly
in round-robin order for up to three rounds, bounded by three minutes. No new
manual CDN addresses, host rewriting or IP pinning is added.
With only one configured video, diversity depends on that video's real Player
backend rotation; if it does not rotate, qualification remains unresolved.
Discovery deduplicates actual ordinary signed MP4 media hosts, refetches expired
inventory and selects three hosts passing the existing root/prefix diversity
gate. URLs need twenty minutes remaining for the three sequential owned native
warmups and verified TLS transfers. URLs stay transient; unresolved diagnostics
contain only attempt/host and rejection counts. Discovery is not media proof.
Production adaptive recovery uses this collector. Existing experimental suffixes
retain their one-host operational recheck; promotion requires fresh three-host
inventory and the unchanged version-1 owned native proof factory. Insufficient
inventory leaves the committed experimental profile intact and retries after
the configured cooldown, not the exhausted-strategy two-hour pause. Native GGC
scope requires qualified frontend and any supplied asset candidates remain
qualified and nonoverlapping; missing avatar inventory is not its prerequisite.
Apply checks frontend and all three actual media resources in the newly deployed
daemon, including bounded learning and three verified transfers per TLS version.
Whole-service monitoring still reports avatar degradation independently.
Single-CDN version-2 qualification has been withdrawn. Maximum strategy catalogs,
new video discovery sources and controller stay-alive changes are separate work.
The three-CDN requirement was reaffirmed on 2026-10-13. The common suffix remains
authorized extrapolative coverage; only exact measured endpoints have transfer
evidence. This candidate is offline unit/fixture-verified only: owned factory,
autonomous discovery/promotion and deployed learning still need a scoped actual
three-CDN field test. No remote network checks or deployment accompany this update.
+318 -30
View File
@@ -1,4 +1,154 @@
# Fresh managed-test target contract (not deployed)
# Fresh managed-test target contract (isolated acceptance, not production approval)
## Declarative HTTPS resource checks v1 (check-only)
Inside the provisioned owned managed target, with the tracked example available:
```sh
zapret-autotune check --config resource-check.example.json
```
This is not a standalone host checker for arbitrary external proxies: owned-target,
SS and configured wire-DNS readiness/ownership guards must pass before probes.
Version1 supports `kind: http`, GET/HEAD on HTTPS **DNS-hostname** URLs,
TLS1.2/1.3/both, timeout and expected HTTP statuses, MIME content type and
minimum/maximum bytes. Outcomes identify stable configured service/resource IDs
and report HTTP status, bytes and latency. URL/header secrets come from env via
private stdin, not argv; keep the config and env private when sensitive.
All resolved IPs are policy-checked before numeric CONNECT pinning; TLS verifies
the original hostname/SNI. Redirects and system-DNS fallback are forbidden.
Private destinations require explicit opt-in **and** CIDR restriction. IPv6
literal URLs and address failover are not supported in v1. Image/media decoding,
Range and suffix-scope qualification are not generic HTTP-check features.
Schema/loader and executable checks have unit and isolated remote acceptance for
source `4d7c4b33c36186d43ea89fd24354b1834b21692c`, image
`sha256:be5a1c45c6d40b390a29bfc9a5ab3975c3a8e0c015b98a524696312807dad9a4`.
The orchestrator supplied verified private evidence, not reread by this docs-only
update; see [PROGRESS.md](PROGRESS.md) for gates and cleanup. The positive private
TLS fixture trusted a **staging-only test CA**, never bypassed TLS or changed
production CAs. The existing DNS backend and earlier DoH outage evidence are reused,
not a new DoH implementation or new outage test here.
**Acceptance scope:** staging YouTube monitoring was SIGSTOP-isolated from checks,
then SIGCONT-resumed before graceful stop (exit0, owned cleanup, foreign baseline
unchanged). A prior managed monitor exit78 remains a runtime caveat, not proof of
whole-controller resilience. LIVE11082 was already stopped, not upgraded/restarted;
old SS1080/Xray were untouched. Historical LIVE observations below are not current
production availability. No network operations or deployment accompany this update.
Check-only neither selects/applies strategies nor writes strategy DB entries or
declares full-service READY. Periodic checks/state/status/reload/independent retry
and generic PID1 BLOCKED foundations now have narrow remote fault acceptance;
unlike the earlier check-only run, this monitor ran without SIGSTOP.
Use `managed monitor --resource-config FILE`, or PID1 environment
`ZAPRET_AUTOTUNE_RESOURCE_CONFIG`; `managed resource-status` reads that same env
source. See [RESOURCE_MONITOR.md](RESOURCE_MONITOR.md) for ownership and repair.
HEALTHY is CHECKS_ONLY, not automatic strategy assignment. Generic strategy DB,
domain qualification matrix, runtime coverage/profile-group rendering and apply,
plugins/autotuning, maximum strategy collection and Prometheus remain TODO.
Three-CDN discovery/qualification is a YouTube
adapter contract, not a universal arbitrary-service requirement.
## Managed DoH live acceptance (2026-10-13): DNS verified, NOT READY
Supplied verified reports bind source `544dda7c85965fc3200de3af17fca9ef9999b2ce`
(DoH `367d956`, TCP deadline `721cad4`, public-health `socks5h` fix `544dda7`)
to image `sha256:b3085b7ceca4a629c28bd58f23bf97f02a34c2a38df08b57756e40594ba2d427`,
container `4ad54175a54c`, owned SOCKS `127.0.0.1:11082`.
The 303-second LIVE snapshot retained root/DNS22/SS59,60/monitor253 PIDs,
children PPID1, owned readiness ACK and parent FD8/9 anchors. UDP/TCP DNS
IDs/questions matched, rcode0, answers2/8; SOCKS `0500`, WWW HTTP200/SSL0
and configured `health youtube` exit0 passed. Wire evidence: 41 marked direct
DoH connections, zero external port53. Scope `suffix`, generation9, IDs `[1,9]`
and hash preservation versus staging are measured snapshots, not universal
state; autonomous updates have occurred since the earlier generation8 snapshot.
In isolated staging, blocking both bootstrap IPs for 61 seconds returned
UDP/TCP SERVFAIL/rcode2 at about 5000ms without changing root, DNS, SS or monitor
PIDs or sending external port53 traffic. DNS recovered immediately after removal;
WWW HTTP200 passed after 46 seconds without restart. Graceful stop: 1 second,
exit0, no survivors. Staging nft fault rules were removed; this is a tested DNS
dependency failure, not all-outage or nonconfiguration-guard acceptance.
No new runtime firewall policy beyond normal generated native rules is supplied.
Existing shared GGC qualification is separate evidence, not an 18-transfer rerun
under DoH. Fresh TLS1.3 media: HTTP206/65536 bytes/MP4 `ftyp`/SSL0; TLS1.2
TIMEOUT is not qualified. Avatar/thumbnail qualification remains independently
outstanding; no whole-service READY claim. Historical media results below do
not close these gates on this image. Generic service integration and maximum
strategy collection remain future work, without a new policy framework.
Private supplied reports (not reread here):
`/home/agent/zapret2-autotune-rc-doh-721cad4/live-upgrade-private/`, then latest
`/home/agent/zapret2-autotune-rc-doh-544dda7/live-upgrade-private/`.
No deployment/probes accompany this docs update; later docs are not tested source.
State/credentials/private env were preserved, staging containers removed, old
user SS1080 and user-stopped proxy unchanged. See the deployment guide for the
four DNS settings, LIVE env location, privacy boundary and recreation lifecycle.
## Autonomous three-CDN acceptance (2026-10-13): media qualified, NOT READY
Supplied field results bind source `3fc8563995e890310580ba8f78850b92fea42136`
to managed image `sha256:a902e4efd949a7ffce4414fd99ab4a815a643b40cf9b9e30e2bd2db7b452817a`,
container `4c6b34…`, Compose project `rc0398e6f…`, SOCKS `127.0.0.1:11082`.
Native post-cooldown recovery at 08:07–08:09 automatically discovered three actual
ordinary signed-media hosts without manual domains and promoted Adaptive candidate10
through fresh version-1 `owned-nfq-media-range-v1` proof and deployed checks.
Generation9→10, IDs `[1,3,9]`→`[1,9,10]`, scope
`experimental_googlevideo_suffix`→`suffix`; frontend1, thumbnail9 and common
`googlevideo.com` hostlist were preserved (config hash prefix `dc997ac91…`).
Measured hosts were `rr16---sn-n8v7znsz.googlevideo.com`,
`rr18---sn-n8v7zns6.googlevideo.com`, and `rr5---sn-n8v7kn76.googlevideo.com`:
each passed post-warmup 3/3 on TLS1.2 and TLS1.3, all18 HTTP206/65536-byte
MP4 `ftyp` transfers with Range `0..65535` and valid SSL verification.
Separate warm attempts/failures were 2/0, 3/1, 3/1 respectively, not cold evidence.
Fresh owned proof checked at `2026-10-13T08:08:28.752Z` retains actual endpoints,
source IDs and native-owner Lua binding; three roots/prefixes were measured.
Native request statistics were not persisted; manual request count is unknown.
At 08:21:48 the reported container and monitor were alive, but whole-service
health remained false / DEGRADED. Probe failures405 at 08:20:37 do not independently
establish an avatar-specific cause; yesterday's avatar timeouts are separate.
The orchestrator's fresh owned-SOCKS greeting was `0500`; native `socks5h` uses
SS DNS `8.8.8.8`. The passive SSH watch vanished after about ten minutes;
no normally completed 18-minute watch is claimed. No cooldown was reset.
Private supplied proof: `/home/agent/zapret2-autotune-rc-3fc8563/field-private/proof/final.safe.json`
(not independently reread by this docs update). No new deployment/probes here.
See [PROGRESS.md](PROGRESS.md) for the measured snapshot and remaining gates.
The user's three-CDN requirement is retained; the proposed single-CDN relaxation
was withdrawn. Service-mode discovery needs no manual domains: missing/empty
`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` uses primary `ZAPRET_AUTOTUNE_VIDEO_ID`
(default `dQw4w9WgXcQ`) plus `aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated.
A nonempty JSON array of 1..5 unique VideoIDs overrides this inventory. VideoIDs
are discovery inputs, not proof of distinct CDNs.
Fresh round-robin Player requests are bounded to three rounds (at most 15 requests)
and three minutes. Qualification selects three actual signed MP4 media hosts,
with at least two roots and two prefixes, and twenty-minute remaining URL TTL.
Only the existing version-1 owned native proof factory authorizes qualification:
post-warmup actual media 3/3 on both TLS protocols for each measured endpoint.
The common `googlevideo.com` suffix is an authorized extrapolative scope, not
measured evidence for every Googlevideo host or cold/uniform behavior.
Experimental promotion uses the same proof and fresh deployed three-host media
checks before committing apply. Insufficient inventory preserves the current
experiment and uses configured retry cooldown; genuine exhausted strategy search
still waits at least two hours. Qualified frontend is required; avatar success
does not block this media stage, but whole-service health remains DEGRADED until
actual avatar checks pass. Legacy service-off/exact-host behavior is unchanged.
The representative three-CDN Adaptive promotion is field-qualified above;
whole-service/avatar acceptance remains open. The 2026-10-12 measured results
below are historical, not the new factory evidence. New images still require
their own source/image binding and guarded apply/rollback acceptance. Maximum
strategy collection/search diversity and PID1 global-failure stay-alive remain unfinished future
increments. Declarative check-only configuration is accepted above; generic
controller integration remains future work. No new generic policy framework is
supplied. See [GOOGLEVIDEO_DISCOVERY.md](GOOGLEVIDEO_DISCOVERY.md).
`managed init|reconcile|status|active-ids|apply` is exclusively for a **new dedicated container**
which owns its entire config. The existing/dev/production container must refuse
@@ -12,7 +162,7 @@ must supply all of the following before any managed command can run:
| `ZAPRET_AUTOTUNE_TARGET_DIR` | `/var/lib/zapret-autotune/target` |
| `ZAPRET_CONFIG` | `/var/lib/zapret-autotune/target/config` |
| `SOCKS_PORT` | Base image's actual ss-local service port (canonical decimal, 1–65535) |
| `ZAPRET_AUTOTUNE_SOCKS_PROXY` | `socks5://127.0.0.1:$SOCKS_PORT` (expanded value) |
| `ZAPRET_AUTOTUNE_SOCKS_PROXY` | `socks5h://127.0.0.1:$SOCKS_PORT` (expanded value; legacy `socks5` validated and canonicalized internally) |
The fixed directory must already be a private runtime-user-owned **dedicated
persistent volume mountpoint**, not just a directory under another mount.
@@ -27,8 +177,9 @@ deliberately forging a provisioned runtime.
The managed `config` is a standalone **full config only in this fresh volume**.
Never replace, adopt, or symlink over `/opt/zapret2/config` or any existing
service config. The new deployment entrypoint must ensure the base daemon
actually honors the exact `ZAPRET_CONFIG` path; compatibility remains unproven.
service config. The deployment entrypoint must ensure the base daemon actually
honors the exact `ZAPRET_CONFIG` path; isolated RC acceptance does not waive
that check for a new image.
Existing services would require a separately designed managed **include
fragment**, not this full-config adapter. No such integration is implemented.
@@ -104,15 +255,15 @@ Directories must not be writable by untrusted processes; hostile concurrent
filesystem mutation is outside the transaction boundary. Shell config safety
and exact GGC filtering are unchanged.
The isolated [Compose definition](deploy/managed-target/README.md) now supplies
the marker, mount and ordered startup; it has not been built or deployed.
No Docker socket, remote/container operations, old target paths, Xray edits,
or host firewall provisioning are included. Before deployment,
The isolated [Compose definition](deploy/managed-target/README.md) supplies
the marker, mount and ordered startup; isolated deployments are recorded below.
No Docker socket, old target paths, Xray edits or host firewall provisioning
are included. Before each new deployment,
provisioning must verify the pinned image honors `ZAPRET_CONFIG`, supports all
listed variables, and starts ss-local while the baseline leaves nfqws2 disabled.
Start the fresh target with that baseline, then verify active apply installs
target-local nft NFQUEUE rules and starts nfqws2, and rollback restores both.
These image/startup/routing gates remain unverified: if any is unsupported or
If any image/startup/routing requirement is unsupported or
uncertain, deployment must fail closed, not substitute `restart-daemons` or
alter an old config. Persistent volume durability is a provisioning
requirement; mountinfo alone cannot establish the storage lifecycle.
@@ -131,11 +282,11 @@ Cancelled/failed downtime restores the committed configuration and runs fixed
target `start` with an independent 30-second cleanup deadline. Failed cleanup or
rollback is fatal to the supervisor; unresolved journal state is never started.
The callback caller must hold the monitor singleton; the wiring repeats runtime
identity/SOCKS guards. Fake-only checks are added, but tests/build are deferred
to the orchestrator. `managed monitor` now binds those callbacks and only the
dedicated target entrypoint starts it; no live/runtime validation has occurred.
identity/SOCKS guards. `managed monitor` binds those callbacks and only the
dedicated target entrypoint starts it; local and isolated runtime validation
results, with their limits, are recorded in [PROGRESS.md](PROGRESS.md).
## Dedicated monitor integration (live acceptance pending)
## Dedicated monitor integration (isolated clean-run acceptance passed)
`zapret-autotune managed monitor [--interval 5m] [--failure-threshold 3]
[--cooldown 30m]` uses the health environment defaults. Interval is positive and
@@ -160,19 +311,22 @@ DB/safety/journal/runner failures stop nonzero; joined cancellation plus a clean
failure is also fatal. A qualified frontend at startup is probed immediately,
not rescanned merely because the process restarted.
Entrypoint ordering is SS children → listener → fresh managed init if needed → managed reconcile → zapret start
Entrypoint ordering is owned DNS readiness (default DoH) → SS children → listener → fresh managed init if needed → managed reconcile → zapret start
→ one installed `/usr/local/bin/zapret-autotune managed monitor` child. It supervises
all three children; unexpected monitor exit is not a successful shutdown.
the DNS, SS and monitor children; unexpected monitor exit is not a successful shutdown.
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
SS child termination/wait. Existing targets and Xray remain unchanged. Deployment
allows 240s Compose shutdown grace for bounded
monitor cancellation/restoration and child joins. Code integration is not
image/routing/signal live verification.
monitor cancellation/restoration and child joins. Clean RC `d8316e6` has passed
isolated image/routing, bootstrap, periodic probe, monitor-crash cleanup, graceful
stop and persistent recreation gates; see the
[recorded source/image and PASS report](deploy/managed-target/README.md#recorded-clean-run-acceptance-2026-10-11-pass).
This does not establish playback or blanket production approval.
Injected unit tests cover exact identity/path/config, mountpoint refusal,
old-container-like env, SOCKS port/process refusal, correct fresh-target success,
non-overwrite, transaction rollback and redaction. Tests/build/network are
deferred to the orchestrator; not run by this increment:
non-overwrite, transaction rollback and redaction. Verification results are
recorded in [PROGRESS.md](PROGRESS.md). Focused local checks can be rerun with:
```sh
go test ./cmd/zapret-autotune -run TestManaged
@@ -208,6 +362,8 @@ Example (replace registry and release; never reuse a release tag):
```sh
IMAGE=registry.example/team/zapret-autotune-managed-target:2026-10-10-r1
docker build --pull -f Dockerfile.target -t "$IMAGE" .
test "$(docker image inspect --format '{{json .Config.Entrypoint}}' "$IMAGE")" = '["/bin/sh","/opt/managed-target-entrypoint.sh"]'
docker run --rm --network none --entrypoint /bin/sh "$IMAGE" -ec 'test "$(cat /etc/zapret-autotune/target-id)" = zapret-autotune-managed-v1; test -f /opt/managed-target-entrypoint.sh'
docker push "$IMAGE"
docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE"
```
@@ -227,6 +383,11 @@ docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/co
docker compose --env-file deploy/managed-target/.env -f deploy/managed-target/compose.yaml pull target
```
Do not omit `-f Dockerfile.target`: root `Dockerfile` builds the proposal/prototype,
not the PID-1-managed target. Assert both entrypoint and provisioned marker on the
exact selected image before promotion; the assertions above are image-contract
checks, not live resource qualification. A managed-looking tag is insufficient.
Start the selected container image with Compose `up --no-build`;
no host service installation is required. Keep the selected digest and
private configuration together for upgrades and rollback. Never run a build
@@ -248,24 +409,54 @@ signed URLs; use `docker compose ... config --quiet` for validation.
| --- | --- |
| `ZAPRET_AUTOTUNE_IMAGE` | Local build/test default `zapret-autotune-managed-target:local`; production promotion requires a managed-target `repository@sha256:<digest>` |
| `SS_PASSWORD` | Required independent hex-only secret, at least 32 characters; deployment README generates 64 |
| `ZAPRET_AUTOTUNE_DNS_MODE` | `doh` default; explicit `native` opts into the old resolver, never automatic fallback |
| `ZAPRET_AUTOTUNE_DOH_URL` | `https://dns.google/dns-query`; HTTPS/443 path, no userinfo/query/fragment; path may be secret |
| `ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS` | Compose default empty: parser selects `8.8.8.8,8.8.4.4` only for exact default Google URL; custom provider requires explicit 1–4 comma-separated literal IPs without spaces |
| `ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC` | `5`; integer 1–30 |
| `SS_DNS_SERVER` | Explicit native mode only: empty retains base resolver or literal IPv4/IPv6; ignored in DoH mode |
| `ZAPRET_AUTOTUNE_YOUTUBE_SERVICE` | `0` (legacy) or opt-in `1`; explicitly forwarded by Compose |
| `ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE` | `cold` default or `adaptive` (requires service `1`); explicitly forwarded |
| `ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` | JSON array of 1..5 unique VideoIDs; defaults to only the primary ID; explicitly forwarded |
| `ZAPRET_AUTOTUNE_HOST_SOCKS_PORT` | `11080`; choose an available TCP/UDP port 1–65535, published only on `127.0.0.1` |
| `SOCKS_PORT` | Supplied deployment fixes internal listener to `1080`; not a host-port override |
| `ZAPRET_AUTOTUNE_SOCKS_PROXY` | Fixed `socks5://127.0.0.1:1080` inside this target, no credentials/direct fallback |
| `ZAPRET_AUTOTUNE_SOCKS_PROXY` | Fixed `socks5h://127.0.0.1:1080` inside this target, no credentials/direct fallback |
| `ZAPRET_AUTOTUNE_VIDEO_ID` | `dQw4w9WgXcQ`; exactly 11 ASCII letters, digits, `_` or `-` |
| `ZAPRET_AUTOTUNE_HEALTH_INTERVAL` | `5m`; duration in `(0,24h]` for monitor |
| `ZAPRET_AUTOTUNE_FAILURE_THRESHOLD` | `3`; integer 1–1000 for monitor |
| `ZAPRET_AUTOTUNE_RECOVERY_COOLDOWN` | `30m`; duration in `(0,168h]` for monitor |
| `ZAPRET_AUTOTUNE_DB` | Fixed `/var/lib/zapret-autotune/target/candidates.db` |
Managed Player, metadata, image, media and warmup requests send hostnames through
the owned SS listener (`socks5h`), so they respect target-native DNS instead of
bypassing it with curl's local resolution. Older private environment overrides
using `socks5` are canonicalized internally after validation; arbitrary proxies
remain rejected by the listener/port/process ownership guard. Legacy public CLI
discovery and root checker `--resolve` pinning are unchanged. This corrects a DNS
path bypass, not a proven explanation of earlier peer certificate failures.
The monitor's three policy flags override corresponding environment defaults;
explicit empty/invalid environment settings are rejected. The entrypoint accepts
no arguments. The supplied Compose environment does not forward the video/policy
settings from `.env`: add the desired existing variables to `target.environment`
no arguments. Compose forwards the primary/sample VideoIDs and service/mode settings.
It does not forward the interval/threshold/cooldown policy
settings from `.env`: add those existing variables to `target.environment`
in your deployment copy/override if needed. Keep all fixed identity/path settings
unchanged. No public SS listener is published; Xray configuration is external and
operator-owned, optionally using the host's loopback SOCKS port. Do not modify an
existing Xray deployment as part of bootstrap.
Current service/adaptive hardware acceptance is **pending**, independent of the
recorded legacy RC PASS. Adaptive means post-warmup, not cold: first connections
may fail and native learning repeats after restart. At least three real unique
signed ordinary media host samples are required; multiple VideoIDs may yield one
CDN host and leave `DEGRADED` with no authorized suffix. Sampled thumbnail/avatar
routes do not prove availability of the whole thumbnail pool. The reported
`8.8.8.8` DNS diagnostic corrected a tampered avatar, not all thumbnail routes;
native DNS correction is not full asset availability proof. Default DoH uses
SS `-d 127.0.0.1:5353`; explicit native mode uses validated literal `-d` argv,
with installed-option support checked at image build. Passwords remain private
JSON only. No permanent service IP pins, global host DNS edits or external
startup manager are introduced.
### Supervision and failure handling
The container entrypoint is PID 1 and owns all technical child restart control.
@@ -278,10 +469,30 @@ errors never authorize retry. No new dependency, host service installation or
external process manager is required. SIGTERM/SIGINT cancels backoff or the running
monitor, waits for bounded restoration and joins children before exit.
In default DoH mode PID 1 first starts the stdlib UDP/TCP loopback DNS forwarder
inside the existing Go binary (no dependency or new container),
waits at most five seconds for its owned private PID marker, and only then starts
ss-server with `-d 127.0.0.1:5353`. No host DNS port or DNS/iptables changes are
made. Hostname SOCKS traffic and managed `socks5h` checks use the same encrypted
HTTPS resolver; clients must enable remote SOCKS DNS, not resolve locally or
send IP literals. Bootstrap uses configured literal IPs with verified TLS and
mark `0x50000000` direct sockets excluding probe/live NFQUEUE, with no system DNS, proxy
or plaintext fallback. The DoH resolver sees queries directly, not SS-tunneled;
the plaintext helper exchange stays on loopback5353. Endpoint failures produce
SERVFAIL without child exit or plaintext fallback. A forwarder crash, unlike an
upstream outage, uses the same bounded technical retry budget. The DNS child
stays alive through monitor restoration, zapret stop and SS joins; it is then
terminated/joined within five seconds and only its validated readiness inode is
removed using PID1-held directory/file anchors. Foreign/stale/replaced readiness
is preserved and vetoes startup/retry; SIGKILL does not lose owned cleanup proof. Explicit
native mode has no DNS child. See the deployment README for provider settings
and scope limits. Supplied staged-outage and LIVE DNS acceptance are recorded
above; they do not establish all-outage resilience or whole-service readiness.
Compose restart remains disabled: a blanket container restart could loop over a
safety refusal or reset the in-container retry budget. Compose stop grace is 240s:
the sequential cleanup budget is 140s (45s monitor join, 45s reconciliation plus
5s kill grace, 30s zapret stop plus 5s kill grace, and two 5s SS joins), with room
the sequential cleanup budget is 145s (45s monitor join, 45s reconciliation plus
5s kill grace, 30s zapret stop plus 5s kill grace, two 5s SS joins and a 5s DNS join), with room
for a deferred 45s startup command plus kill grace and scheduling overhead.
The join watchdog runs in a fresh shell and is cancelled when its child finishes;
graceful shutdown does not wait out the watchdog's full deadline.
@@ -360,9 +571,86 @@ restore a known consistent backup. Only then manually start the target with Comp
There is no automatic wipe or
automatic adoption path.
Rollout acceptance remains pending real isolated-target restart/deadline tests:
empty bootstrap, saved-first recovery, exact-host GGC rotation, image binding,
NFQUEUE activation/rollback, SIGTERM during downtime, deadline restoration,
in-container retry limits/backoff, durable two-hour absence wait and resumption, persistent
recreation and backup restore. Perform these on the new target only; local unit
or static checks cannot establish production readiness.
The recorded clean RC PASS covers empty-state bootstrap through qualified frontend
and exact-host GGC apply, image/config binding, ongoing probes, in-container monitor
crash restart, graceful stop and persistent recreation. Earlier recovery, rollback,
rotation, retry and absence fixtures are distinguished in [PROGRESS.md](PROGRESS.md);
do not attribute all of them to the clean run. No media playback, real 90-minute
watchdog expiry or real two-hour absence expiry wait is claimed. Production rollout
and backup-restore acceptance remain operator gates on a new isolated target;
local unit/static checks and isolated PASS do not establish blanket production readiness.
### Explicit experimental Googlevideo suffix
Normal `--googlevideo-scope suffix` still requires representative qualification,
media proof, and both qualified image profiles. An operator may explicitly own
the audited native circular A/E composite without claiming those results:
```sh
zapret-autotune managed apply --googlevideo-experiment native-circular --googlevideo-scope experimental_googlevideo_suffix
```
This command requires the monitor singleton to be free, a matching native Lua
hash, a fresh Player-discovered exact GGC identity, and existing qualified
frontend/assets. It preserves their candidate IDs and uses the normal guarded
transaction/reload/Player-health/rollback hooks. The new candidate definition
has no cold/adaptive evidence and is not a qualified candidate. Both pending
and committed journal policy record `experimental_googlevideo_suffix`; legacy
rendered bytes and qualification gates are unchanged. Unknown policies fail closed.
For PID-1-controlled application, while this dedicated target is stopped and
after taking a complete private state backup, create a private empty file named
`.autotune-experimental-suffix-request` in its owned state directory. It must be
an empty regular 0600 file owned by the runtime UID, with no symlink/hardlink.
Startup validates it before init/reconcile, then atomically claims the validated
inode before guarded apply (before launching the monitor). Only a successful
apply removes that claim; a replacement request is never deleted. Failure exits
closed and retains the private `.autotune-experimental-request-claim-*` artifact;
do not blindly retry it. Use the matched whole-state/image rollback above.
Candidate-ID CLI apply cannot select/inherit the experimental policy; explicit
native-circular authorization is required. Internal journal recovery is unchanged.
Resource probes still run. In service mode (legacy or adaptive), healthy actual
image/media probes short-circuit recovery, but operational health does not mean
family qualification or playback READY. Experimental media recovery preserves
the operator suffix and all committed profile IDs: a validated transfer failure
permits one guarded restart of that same deployed daemon, then at most 10 native
relearning attempts within 120s and a final fresh signed media 3/3 for each of
TLS1.2 and TLS1.3. Learning stays in the persistent daemon's RAM; no temporary
NFQ worker or replacement/qualified candidate is created. Expiring media is
rediscovered, never rewritten or persisted. Missing inventory remains degraded,
not strategy absence; mixed, guard, infrastructure and cancellation errors do
not authorize relearning. Unrelated missing image inventory may coexist with
usable media, but successful media recovery does not repair or qualify it.
Avatar inventory first parses fresh homepage `ytInitialData`. Only absent metadata
or no allowed avatar permits one fallback to the configured video's watch page,
using a validated video ID, the same trusted HTTPS WWW origin and SOCKS proxy,
bounded time/body size, no redirects, and discarded diagnostics. Malformed JSON,
transfer errors and safety/cancellation failures do not trigger fallback. Both
pages and discovered resource URLs remain ephemeral; discovering an allowed URL
is not image-transfer qualification or a guarantee of full-family coverage.
`managed status` reports active-profile `covered=` separately from qualified
`groups` and `unresolved_groups`; `managed active-ids` adds `covered_groups` and
`googlevideo_scope` to its JSON. An experimental Googlevideo profile can be
covered yet qualification-unresolved; one image profile does not establish full
asset-inventory success. Status is read-only and reports live health NOT CHECKED.
Historical `a60f62d` restart/recreation and status checks passed, but its upgrade
hit `BrokenPipe` and rolled back; that run did not accept media recovery. Later
supplied final field results for source `fd25cb8`, managed image
`sha256:73e955e7fdc6efaef88c62ebe4f7afeab71c29d29ba3c88b2a2b7547bb647da5`
on SOCKS11082 completed a 15-minute watch with restart count0 and a live monitor.
Validated rejection/no-qualified-strategy recovery continued with naturally
advanced cooldown and unchanged experimental profiles; it did not repair avatars.
Three fresh media hosts across two roots passed post-warmup 3/3 on each TLS
protocol (18 actual HTTP206/65536-byte MP4 transfers). This is field evidence,
not persisted family qualification, a cold/all-host claim or candidate promotion.
Watch fallback discovered allowed avatar metadata, but both-TLS curl28 transfers
had zero samples: avatar evidence/profile qualification is still absent and
overall health is **DEGRADED / NOT READY**. Automatic durable multi-host family
qualification remains unimplemented. See [PROGRESS.md](PROGRESS.md) for exact
hosts, separate warmup results, timestamps, private proof and remaining gates.
No new network checks accompany this documentation. Generic production
media behavior and signed-URL/SS secret privacy remain unchanged, as do the
persisted minimum two-hour strategy-absence pause and 90-minute whole-job budget.
+132 -1
View File
@@ -1,9 +1,140 @@
# Прогресс реализации
Срез на **2026-10-11**. Этот файл описывает фактическое состояние кода, а не весь объём требований из `Requirements.md` и `Architecture.md`.
Срез на **2026-10-14**. Этот файл описывает фактическое состояние кода, а не весь объём требований из `Requirements.md` и `Architecture.md`.
## Текущий статус
### Generic periodic monitor fault acceptance: narrow PASS, not autotuning/READY
- Supplied orchestrator-verified privileged read of `/home/agent/zapret2-autotune-monitor-5c85f3f/field-private/report.safe.json` covers `4efc066` plus **`5c85f3f0b9ef04e600f67593d02cae526fac901a`**. Not independently reread here; native full/race checks and independent security review were reported GO. This docs-only update performs no probes/deployment; later docs commits are not tested source.
- Actual periodic monitoring ran **without SIGSTOP**: latest window 21:50:22–21:54:30 retained PID132 and three rows across two services (alpha stable/flaky, beta slow). Healthy rows progressed while flaky was DEGRADED, then repair made all healthy; earlier `4efc066` ran stably for 305s with independent failure/retry and valid reload/removal.
- Snapshot/key inode replacement, wrong UID/mode, symlink, corrupt/missing files and readiness tampering yielded **BLOCKED, stale=true**; private DNS/SNI/server counters were unchanged during faults. Invalid source/wrong ownership made zero DNS requests. Foreign bytes/unknown files were preserved; original-inode/operator repair resumed the same PID. Root wrong-owner field test passed (local unit case skipped).
- Public state was authenticated with a private 32-byte root-owned mode0600 HMAC key; secret-marker/address scans were clean. Idle 30s left content/inode/mtime unchanged. Cancelling an actual active curl left the unfinished row's LastCompleted/Failures/NextDue untouched; completed rows remained durable.
- Graceful stop: **1.45s**, exit0, all owned PIDs joined and network namespace gone; restart PID121 accepted HMAC/prior rows, force-recreate PID126 retained the state-directory inode and accepted HMAC. Additional stops: 0.59s/0.57s. Offline native worst-case cleanup verifies bounded final reaps within 90s, not a measured remote 80s worst-case wait.
- Strategy config/journal hashes were unchanged. Old SS1080, stopped proxy and unrelated stopped RC11082/exit78 were unchanged; legacy YouTube exit78 is **not fixed**. HEALTHY means **CHECKS_ONLY**, not generic strategy recovery, whole-production READY or full P0 acceptance.
- Periodic controller/state/status/reload/independent retry foundations are DONE. Next P0: actual per-domain qualification matrix, then runtime coverage/profile groups, renderer transactions and saving multiple qualified candidates; generic strategy DB/migration/apply remain TODO. Prometheus P1 docs were researched, but no metrics code exists; future labels must stay bounded to configured IDs/enums.
### Declarative resource-check acceptance (2026-10-13): check-only PASS, not controller acceptance
- Supplied orchestrator-verified remote evidence binds source **`4d7c4b33c36186d43ea89fd24354b1834b21692c`** to image **`sha256:be5a1c45c6d40b390a29bfc9a5ab3975c3a8e0c015b98a524696312807dad9a4`**. Private report: `/home/agent/zapret2-autotune-resources-4d7c4b3/field-private/final.safe.json`; privileged-read by the orchestrator, **not independently reread here**. Reported `checker_gates_passed=true`; later documentation commits are not the tested source. No new network checks or deployment accompany this update.
- JSON v1 schema/loader and executable HTTPS GET/HEAD checks are unit-verified and now remotely accepted: public GET/HEAD on TLS1.2/1.3; private opt-in plus CIDR with a staging-only trusted test CA, original TLS/SNI hostname and numeric CONNECT pin. No `--insecure`, production trust-store pollution or DoH implementation change. Checks reuse the owned SS/configured wire-DNS backend; the earlier DoH outage acceptance recorded below remains separate evidence.
- Default/private-policy, mixed DNS answers, metadata/link-local/ULA/loopback denials made **zero destination connections**; rebinding used the first pinned address and denied the later answer. Ownership-negative cases made zero DNS/resource probes. Cancellation exited1 with joined workers in **0.38s**; the secret sentinel was absent from argv, output and persisted artifacts.
- **Check-only isolation:** staging YouTube monitor was SIGSTOP-isolated during checks, then SIGCONT-resumed **before** graceful stop; reported `monitor_resumed=true`, Docker exit0. Cleanup left zero owned host PIDs, no owned namespace or temporary containers/networks, and an unchanged foreign baseline. An earlier existing managed monitor **exit78** contributed before isolation: this does **not** prove whole-controller resilience, generic recovery or production availability. LIVE11082 was already **stopped** before the task and was **not upgraded or restarted**; old SS1080/Xray were untouched. Earlier LIVE snapshots below are historical, not current uptime claims.
- At this earlier check-only acceptance, periodic integration was pending; the narrow controller/BLOCKED foundations are now accepted above. Remaining P0: config/strategy DB integration, matrix and plugins/autotuning; maximum strategy collection and Prometheus P1 remain TODO. Three-CDN qualification belongs to the YouTube adapter, not every arbitrary service. See [MANAGED_TARGET.md](MANAGED_TARGET.md#declarative-https-resource-checks-v1-check-only) and the tracked [resource-check.example.json](resource-check.example.json).
### Managed DoH acceptance (2026-10-13): live DNS path and staged outage verified, NOT READY
- Managed target defaults to the stdlib Go UDP/TCP loopback DoH forwarder; PID 1 validates owned readiness before ss-server `-d 127.0.0.1:5353`. SOCKS remote-hostname traffic and managed `socks5h` checks share this path. Provider URL/bootstrap IPs/timeout are private-env configurable; explicit native mode alone preserves legacy DNS, never automatic fallback.
- DNS stays available during monitor restoration, zapret stop and SS joins, then is terminated/joined with a five-second watchdog and readiness cleanup. DNS child death uses existing bounded retries; upstream failures return SERVFAIL without stopping the forwarder. No host DNS port, global DNS changes, new dependency or capability.
- Security follow-up: PID 1 retains directory/readiness inode descriptors for validated owned cleanup after graceful exit or SIGKILL; foreign/stale/replaced markers are preserved and fail closed. Compose passes empty bootstrap by default, allowing Google defaults only for the exact Google URL; custom URLs require explicit IPs.
- Earlier offline shell lifecycle/static/config fixtures (real readiness CLI with inherited anchors), sh/bash syntax checks, full Go and targeted readiness/entrypoint race checks passed as previously reported. Coverage includes foreign regular/symlink preservation, SIGKILL owned cleanup/retry, custom-provider Compose validation, signal/cooldown cancellation and DNS-last cleanup. This docs-only update adds no source/runtime changes or probes and does not rerun the Go suite.
- Supplied verified reports bind DoH implementation `367d956`, TCP reply deadline fix `721cad4`, and explicit public-health `socks5h` fix **`544dda7c85965fc3200de3af17fca9ef9999b2ce`** to LIVE image **`sha256:b3085b7ceca4a629c28bd58f23bf97f02a34c2a38df08b57756e40594ba2d427`**, container **`4ad54175a54c`**, owned SOCKS **`127.0.0.1:11082`**. Private reports: `/home/agent/zapret2-autotune-rc-doh-721cad4/live-upgrade-private/`, then latest `/home/agent/zapret2-autotune-rc-doh-544dda7/live-upgrade-private/`; not independently reread here. Later documentation commits are not the tested source.
- Real isolated staging blocked **both** bootstrap IPs for **61 seconds**: UDP and TCP replies returned SERVFAIL (rcode2) at about **5000ms**; DNS, monitor, SS and root PIDs stayed unchanged, with zero external port53 traffic. After removal, DNS recovered immediately and WWW HTTP200 was verified after **46 seconds**, without process restart. Graceful stop took **1 second**, exit0, no survivors. This proves the tested DNS dependency outage, not every conceivable outage or nonconfiguration guard behavior. Staging-only nft fault rules were removed; no new runtime firewall policy was added beyond normal generated native rules.
- LIVE **303-second** measured snapshot: same root and children DNS22, SS59/60, monitor253, all children PPID1; owned readiness ACK and PID1-held FD8/9 anchors were positive. UDP/TCP DNS IDs and questions matched, rcode0, answer counts2/8. SOCKS greeting `0500`, WWW HTTP200/SSLVerifyResult0, and configured `health youtube` exit0 passed. Wire evidence counted **41 marked DoH connections**, zero external port53; direct HTTPS sockets use mark **`0x50000000`** to bypass live/probe NFQUEUE. Scope `suffix`, generation9, IDs `[1,9]`, config hash preserved versus staging are measured snapshot values, not universal constants: autonomous state has advanced since an earlier generation8 observation.
- Existing normal shared GGC qualification remains separate proof; no new 18-transfer qualification under DoH is claimed. Fresh media TLS1.3 returned HTTP206/65536 bytes/MP4 `ftyp`/SSLVerifyResult0; TLS1.2 timed out and is **not qualified**. Avatar and thumbnail qualification remain independent outstanding gates; no full-service READY claim. Maximum strategy collection and generic service integration remain future work, not a new framework.
- DNS configuration is the private mode0600 LIVE `/home/agent/zapret-autotune-cleanops-3fc8563-1791882112/.env`; the four public DNS keys and defaults are documented in the deployment guide. URL/bootstrap come from env, never argv; changing DNS requires official Compose recreation under the controlled PID1 lifecycle. Credentials/private env and state were preserved. Staging cleanup left no new running test containers; the old user SS1080 and user-stopped proxy were unchanged.
- Default `https://dns.google/dns-query` selects literal `8.8.8.8,8.8.4.4` bootstrap only when missing/empty; custom endpoints require explicit IPs. TLS is verified, with no system-DNS, proxy or plaintext fallback; native mode is manual and explicit only. The direct DoH resolver sees queries: they are **not SS-tunneled**. Plaintext UDP/TCP exists only at internal loopback5353. IP-transmitting SOCKS clients using local DNS are outside this guarantee; use remote DNS/`socks5h`. The existing Go binary supplies the stdlib forwarder, no dependency or extra container, bounded to 4096-byte queries, 65535-byte replies, 8192-byte HTTP headers and 16 workers, with no forwarder cache.
### Autonomous three-CDN field acceptance (2026-10-13): media qualified, NOT READY
- Supplied field results for source `3fc8563995e890310580ba8f78850b92fea42136`, managed `Dockerfile.target` image `sha256:a902e4efd949a7ffce4414fd99ab4a815a643b40cf9b9e30e2bd2db7b452817a`, container `4c6b34…`, Compose project `rc0398e6f…`, isolated SOCKS `127.0.0.1:11082`. This documentation update performs no deployment or network probes. Private sanitized artifact: `/home/agent/zapret2-autotune-rc-3fc8563/field-private/proof/final.safe.json`; its contents are reported by the orchestrator, not independently reread here. Later docs commits are not the tested image source.
- From the existing 08:01:50 cooldown state, native recovery ran naturally at 08:07–08:09 without resetting cooldown. Missing/blank `ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` selected primary `dQw4w9WgXcQ`, `aqz-KE-bpKQ`, and `jNQXAC9IVRw`. The program discovered three actual ordinary signed-media hosts without manual domains; VideoIDs are inputs, not CDN evidence.
| Actual discovered host | TLS1.2 steady media | TLS1.3 steady media | Separate warm attempts / failures |
| --- | --- | --- | --- |
| `rr16---sn-n8v7znsz.googlevideo.com` | 3/3 | 3/3 | 2 / 0 |
| `rr18---sn-n8v7zns6.googlevideo.com` | 3/3 | 3/3 | 3 / 1 |
| `rr5---sn-n8v7kn76.googlevideo.com` | 3/3 | 3/3 | 3 / 1 |
All **18 post-warmup transfers** returned HTTP206, valid Range `0..65535`, 65536 bytes, MP4 `ftyp`, and valid SSL verification. Warm failures are separate, not cold qualification. Fresh version-1 `owned-nfq-media-range-v1` attestation in Adaptive mode retained actual host/endpoints, source IDs and native-owner Lua binding; proof checked at `2026-10-13T08:08:28.752Z`, with three roots and three prefixes. Native request statistics were not persisted: manual request count is unknown, not zero or an invented total.
- The operator automatically **PROMOTED** one newly qualified Adaptive candidate10: generation9→10, active IDs `[1,3,9]`→`[1,9,10]`, scope `experimental_googlevideo_suffix`→`suffix`, current config hash prefix `dc997ac91…`. Common hostlist `googlevideo.com`, frontend1 and thumbnail9 were retained. Qualification and fresh deployed checks used the transaction-bound apply safety path (asset/frontend hooks have source test coverage). This qualifies the measured representative three-CDN composite profile, not every Googlevideo host, cold behavior, or a maximum strategy catalogue. The shared profile remains two native strategies with per-host learning.
- Final reported snapshot at **08:21:48**: container running and monitor alive; whole-service health still false / **DEGRADED, NOT READY**. Probe failures reached405 at 08:20:37; their avatar-specific cause was not independently measured in this phase. The prior day's avatar timeouts are historical, not today's diagnosis. The orchestrator's fresh owned-SOCKS check returned `0500`; role-owned `socks5h` uses native SS DNS `8.8.8.8`. The passive SSH watch session vanished after about ten minutes: no normally completed 18-minute watch or interruption of the controller is claimed.
- Remaining: actual avatar/service health qualification, maximum strategy collection and broader search diversity, and controller security/global-failure stay-alive policy (current safety guards remain fatal). Generic periodic per-service failure isolation/controller integration remain future requirements; declarative check-only configuration is accepted above, not a new plugin framework. No further build, probes or cooldown changes accompany these docs.
### Autonomous three-CDN source behavior (2026-10-13)
- User reaffirmed **three actual CDN checks**, discovered by the program without
manual domains; the earlier single-CDN/version-2 proposal is withdrawn. Missing
or empty service-mode VideoID inventory selects primary (default `dQw4w9WgXcQ`)
plus `aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated. A nonempty 1..5 unique-ID
override is respected; three videos do not prove three CDN hosts.
- Fresh discovery runs up to three round-robin rounds, at most 15 requests in
three minutes; it deduplicates actual signed ordinary MP4 hosts, requires
twenty-minute remaining URL TTL, and selects three hosts across at least two
roots and two prefixes. The unchanged version-1 owned native proof factory
requires actual post-warmup media 3/3 on both TLS protocols at each endpoint.
Common Googlevideo suffix authorization is extrapolative, not all-host evidence.
- Existing experiments can now seek promotion through that shared qualification
path, then frontend and fresh deployed three-host media checks under the current
transaction/ownership guards. Insufficient inventory leaves the experiment
committed and retries at configured cooldown; genuine strategy exhaustion keeps
its minimum two-hour pause. Missing avatar does not block the media stage;
whole-service monitoring remains DEGRADED until actual avatar health passes.
Legacy service-off/exact-host behavior and cancellation safety remain unchanged.
- The source was unit/fixture-tested and subsequently field-accepted for the
representative media promotion recorded above; whole-service/avatar acceptance
remains open. Earlier statements below that automatic qualification is
unimplemented describe historical RCs, not this source; historical measurements
are not the new owned factory evidence.
- Maximum strategy collection and PID1 global-failure stay-alive are still future
increments; broad maximum-profile live staging is deferred. Declarative checks
are accepted above; generic service controller integration remains future work. Details:
[GOOGLEVIDEO_DISCOVERY.md](GOOGLEVIDEO_DISCOVERY.md).
### Final managed RC field acceptance (2026-10-12): DEGRADED, not READY
- Supplied final field results, not network checks rerun by this docs update: source `fd25cb89398300bb9849db4b5511af75bf86eccf`, correct managed image `sha256:73e955e7fdc6efaef88c62ebe4f7afeab71c29d29ba3c88b2a2b7547bb647da5`, active container `a4e845a6d7…`, official managed PID 1, isolated SOCKS `127.0.0.1:11082`. The earlier `37823…` prototype image is **not active** and is not this acceptance image. Private sanitized proof: `/home/agent/zapret2-autotune-rc-fd25cb8/proof/final.safe.json`. The later docs commit is not the tested image source.
- Watch started `2026-10-12T20:15:07.686Z`; check at `2026-10-12T20:30:54.826Z` completed over 15 minutes with DockerRestartCount0 and monitor alive. A fresh orchestrator check after the report returned SOCKS greeting `0500` and WWW HTTP200 with SSLVerifyResult0. Managed `socks5h` probes respect native SS DNS `-d 8.8.8.8`; no local-resolution bypass or certificate bypass is claimed.
- At this observation, experimental journal generation9, IDs [1,3,9], hash prefix `0e34…` and covered scope were retained unchanged. Saved candidate3 remains experimental, without qualification/evidence. These are snapshot values, not universal IDs/generation. Runtime recovery classification accepted validated peer-TLS/payload rejections as nonpass, with writable-store/ownership guards and the current job-bound transaction hooks intact. Genuine no-qualified-strategy recovery advanced its cooldown **naturally** while the monitor stayed alive; no cooldown reset or fabricated ERROR/source result was used. This validates degraded recovery continuation, not successful avatar repair or family promotion.
- Fresh actual Range media results, **post-warmup**, not cold qualification:
| Sampled hostname | TLS1.2 | TLS1.3 |
| --- | --- | --- |
| `rr4---sn-axq7sn7z.googlevideo.com` | 3/3 | 3/3 |
| `rr5---sn-n8v7kn76.googlevideo.com` | 3/3 | 3/3 |
| `rr18---sn-axq7sn7z.googlevideo.com` | 3/3 | 3/3 |
All **18** transfers verified HTTP206, 65536 bytes, MP4 `ftyp` and valid Range `0..65535`. Separately, initial TLS1.2 warmup had three timeouts; initial TLS1.3 warmup had three passes. Three hosts across two roots provide representative field transfer evidence only; no cold/all-host or durable family qualification was created. The older rr5 `sn-ixh` counterexample was **not tested** here and is not resolved by the new rr5 `sn-n8v7kn76` sample.
- Thumbnail `i.ytimg.com`: JPEG, 2888 bytes, SSLVerifyResult0, decoded 3/3 on each TLS protocol. Fresh watch metadata discovered an allowed exact `yt3.ggpht.com` avatar URL, but actual avatar transfers returned curl28 on both TLS protocols, zero samples. No saved avatar evidence or qualified avatar profile exists. Overall live health remains **DEGRADED / NOT READY**; URL discovery and successful media do not close the avatar gate.
- Other production/test targets on 1080/11080/11081 retained their IDs/start times; Xray was untouched. No further remote operations accompany this docs update. Build explicitly with `-f Dockerfile.target` and assert the managed entrypoint plus target marker **before promotion**: a tag alone previously mistook root `Dockerfile`'s prototype for the managed image.
Remaining work: actual avatar transfer/strategy qualification, then durable representative multi-host family qualification and mode acceptance. Automatic family qualification is not implemented; no experimental candidate was promoted. Restart/recreation observations from older RCs remain separate from this final 15-minute watch. The historical open-media/fallback/deployment labels below are superseded only by the specific final results above, not by blanket production approval.
### Earlier peer TLS rejection classification fix (2026-10-12)
- Follow-up independent production repro exposed a separate decoder rejection bug: a bounded/origin-verified truncated image degraded live health but failed owned verification as infrastructure before remaining seeds. Image decoding now emits typed payload rejections (invalid magic/dimensions, truncated PNG/JPEG/GIF, body budget and content-type mismatch); both live health and candidate verification share that class, without accepting any payload or fabricating evidence. Missing/corrupt normalized evidence, malformed transfer metrics/Player JSON, arbitrary same-text errors, explicit guards and mixed cancellation remain fatal; image rejection cannot become media absence. Permanent production monitor tests cover joined workers, seed exhaustion/two-hour pause and a later seed's fresh both-TLS 3/3 evidence with preserved frontend/GGC profiles. The independent overlay repro, affected Go tests and race tests, vet, CLI build and diff checks passed locally; no remote deployment was performed.
- Read-only field inspection: RC source `50a78608ce40dfc26289b5a63d01eeb3fc569ace`, image `sha256:1a97f34cfbe3bd606d5fb13599767f8a51273b374fa441f5e67802c74d814701`, container `d361b5d581bf710af28f14d3998d6a9a699661b827248c12c9352182df94f98d` is now exited78 (finished `2026-10-12T18:44:23.941802233Z`); host port11082 has no TCP listener. The earlier `proof-v2/final.safe.json` observation at 18:44:10 reported it running, so it is not current uptime evidence. Private artifacts: `/home/agent/zapret2-autotune-rc-50a7860/proof-v2/`.
- That safe collector evidence contains a fresh allowed avatar URL, avatar TLS1.2 curl60 and TLS1.3 curl35, both zero samples/FAIL with generic fatal wrappers; thumbnail and sampled media passed. Terminal safe logs subsequently show avatar degradation, **thumbnail** degradation, recovery failure, and the generic safety/database/runtime terminal branch. They do not identify the exact terminal underlying error: do not claim avatar TLS alone is proven to have caused exit78 or that a real local guard failed.
- Source fix distinguishes validated resource-curl peer TLS rejection (35/51/60) from process-wide operational safety. It remains rejected/unqualified, with certificate verification, origin checks and SSLVerifyResult=0 qualification unchanged. Probe/recovery can remain DEGRADED; genuine exhausted asset verification enters the existing minimum two-hour pause with workers joined and committed profiles preserved. Local proxy connection7, client-certificate configuration58, unknown/mixed cancellation and explicit guard/cleanup failures remain infrastructure/safety failures. No certificate bypass, host pinning, DNS change, family promotion or READY claim.
- The corrected source has not been deployed or image-qualified. Existing remote containers, profiles and other working ports were not restarted or modified by this fix; a tested new image and renewed actual-resource checks are the next dispatch.
- Local affected-area verification: managed/resource/experimental/asset/Googlevideo/monitor tests passed normally and under `-race`; local CLI build and `git diff --check` passed. Fixture checks cover production avatar35/60 degradation, owned verification exhaustion and post-apply rejection/rollback, joined workers, durable two-hour cooldown, experimental same-profile media35/60 recovery, and retained local/mixed safety failures. These are not live image acceptance.
### Googlevideo suffix RC — browser-verified experiment (2026-10-12)
- Initial source RC `eade051c2bc335b71b1616d697326ca4c8019d70`, immutable image `sha256:fdb60593e29d7c337a413a47aaddfbcaaeabdeb877042c58f506d4abdc98a13b`, isolated host SOCKS `127.0.0.1:11082`. Historical initial journal: generation3, active candidates [1,2,3], scope `experimental_googlevideo_suffix`; `www.youtube.com` and `i.ytimg.com` profiles preserved. Older targets were unchanged. These are recorded field observations, not current candidate IDs or a new probe/deployment by this source update; its SHA is not the tested image source.
- Fresh signed media on `rr16---sn-n8v7znsz.googlevideo.com` returned HTTP206, 65536 bytes, MP4 `ftyp`, TLS1.2. On **2026-10-12 the user confirmed browser videos work** with this live RC and accepted one shared profile for Googlevideo, including initial learning. Browser images also appeared, but that does not qualify every image route.
- One suffix profile is a **composite native circular A/E profile with two strategies and per-host learning**, not one uniform static strategy. Explicit operator experimental policy is durable across restart with hash/rollback guards; it is neither family-qualified nor `READY`, and no all-host cold 3/3 result is claimed.
- Source now wires autonomous experimental media recovery in either configured service mode. Healthy actual-resource probes short-circuit recovery; a validated media transfer failure permits one guarded restart of the same committed profiles and bounded native relearning in the persistent deployed daemon (10 attempts / 120s), followed by fresh signed media 3/3 on both TLS protocols. Learning stays in daemon RAM, not a temporary verification worker; expired inventory is refreshed, never rewritten or persisted. Missing inventory is degraded, not strategy absence; guard/infrastructure/cancellation and mixed failures remain fail-closed. No experimental candidate is replaced or promoted to qualified.
- Status distinguishes active-profile coverage (`covered=` / JSON `covered_groups`, `googlevideo_scope`) from qualified `groups` and `unresolved_groups`. Experimental Googlevideo remains unresolved for qualification even when operational media is healthy; a thumbnail profile does not mean the full asset inventory is qualified. Status itself performs no live health check. Generic production media handling, signed-URL/SS privacy, persisted minimum two-hour strategy-absence pause and the separate 90-minute whole-job budget are unchanged.
- **Reported field follow-up for `a60f62d`:** privileged checks passed restart/recreation and covered-versus-qualified status checks. The live upgrade encountered a runner `BrokenPipe` and automatically rolled back to the same last-good image; this is not a successful autonomous media-recovery acceptance. Fresh Range media passed once on TLS1.2, then failed on repeat; no TLS1.3 media PASS is claimed. The underlying field failure remains unknown. These are supplied field results, not network checks rerun by this source update.
- Avatar discovery now conditionally falls back from a homepage with absent metadata/no allowed avatar to one fresh watch page for the validated configured video ID, on the same trusted HTTPS WWW origin and proxy. Malformed JSON, request failures, cancellation and safety failures do not authorize fallback. Reported watch metadata contained an allowed avatar URL where the homepage did not; the updated fallback has not been field-deployed or image-qualified. URL discovery alone does not qualify the avatar route/profile or close a family gate.
- Local source verification for the watch fallback: fresh `go test ./... -count=1 -timeout=180s`, focused avatar inventory/parser/classifier, experimental production-probe and private-inventory regression tests under `-race`, and `git diff --check` passed. These are unit/fixture checks only, not new field evidence.
Historical acceptance gates at that RC (updated by final field results above):
1. Representative fresh signed media evidence: at least three unique hosts across at least two roots/prefixes, post-warmup 3/3 for both TLS protocols plus actual media transfer. Do not impose uniform static/cold behavior on the accepted learning experiment or label it normally qualified without evidence.
2. Qualify the avatar route/profile: only the thumbnail image profile is currently active; browser-visible images do not close this gate.
3. Restart/recreation checks are reported PASS for `a60f62d`; renewed learning with repeatable actual media success remains an open gate.
4. Field-accept the new autonomous same-profile experimental recovery and resource-error classifier without replacing or promoting operator-approved scope. Unit coverage exists; live upgrade/rollback observations do not prove successful recovery. Automatic family qualification is not implemented.
5. Covered-versus-qualified field checks are reported PASS for `a60f62d`; validate the new conditional watch fallback and qualify actual avatar image transfers separately. No family-qualified/`READY` or blanket production claim yet.
### Earlier isolated-target acceptance (2026-10-11)
- **2026-10-11 clean RC acceptance — PASS:** source RC `d8316e695bc36e1243c111e19bb1e527355907b6`, tag `zapret-autotune-managed-target:rc-d8316e6`, immutable image `sha256:5636aa9031fd0f6a0513f2e22cc0c39b1c806e9714b7f9ac040c8713d6a26c6c`, built from the source git archive with pinned inputs. These source/image identifiers predate this documentation update; a later docs commit is not the tested image source. Private source root: `/home/agent/zapret2-autotune-rc-d8316e6`. Additional clean target root: `/home/agent/zapret2-managed-clean-d8316e6-45bb14ba`, Compose project `zapret-autotune-clean-d8316e6-45bb14ba`, SOCKS TCP/UDP `127.0.0.1:11081`; left running for inspection. The existing target on `11080` was not replaced.
- **Clean bootstrap/evidence:** state was empty before launch (root-owned directory mode `0700`, independent 64-hex secret, `.env` mode `0600`); real `blockcheckw` binary, no mocks or copied config/candidates. Fresh init was observed, but fast bootstrap prevented capturing an exact generation-0 status snapshot. Generation2 committed frontend ID1 and GGC ID2 for exact `rr17---sn-n8v7kn7d.googlevideo.com`; both candidates have valid raw TLS1.2/TLS1.3 evidence, 3/3 each. Actual root config uses the verified layout-safe fake + multidisorder frontend seed. Periodic `lastProbeAt` advanced with failure counter0 and idle recovery.
- **Clean lifecycle gates:** monitor SIGKILL changed host PID522765→525271 inside the same container/PID1, DockerRestartCount0; IDs[1,2] and config hash `57e639c98420be73a8ec4754b20ba975d81aced44215ed547b1ac029c89c14de` unchanged. Graceful stop exited0 in 0.61s, with no worker leaks. Forced recreation (`5af686e2ad94…`, host PID525625) preserved generation2/IDs/hash. Initial post-recreation failure counters1 then2 are historical observations with unproven cause: a fresh-discovery GGC mismatch sample does not prove the actual probe error cause. The next default probe at 07:54 reset the counter to0 with no active recovery. Player API OK and host SOCKS curl to `www.youtube.com` HTTP200; no media playback/SABR claim.
- **Scope/limits:** both older targets' images, container IDs, PIDs, start times and ports stayed unchanged (existing managed PID518298, old SS PID266922); no host-manager or Xray changes. Private final PASS report: `/home/agent/zapret2-managed-clean-d8316e6-45bb14ba/final-cleanrun-report.json`, raw artifacts in `final-evidence/`. This is isolated clean-run acceptance, not blanket production approval, a playback test, or a real 90-minute watchdog wait. The full 90-minute wall-clock expiry was not waited out; deadline behavior remains covered by shortened/injected tests. The working-target entry below describes the unchanged existing target, not the additional clean RC.
- **2026-10-11 container-owned control, committed/deployed:** `5258ae2`, `6467108`, `fe23eec` replace external supervision with PID1-owned runtime-cycle retries (three starts per rolling six hours, cancellable 15s/30s backoff capped at 60s). Guard/config/DB/rollback/cleanup failures remain fail-closed. Reconciliation precedes activation; fresh-shell watchdogs avoid EXIT/TERM trap deadlock; private monitor sessions allow cleanup of orphaned scan workers before restart. SS uses private JSON, never exported password/argv. Graceful SIGTERM joins/restores before stopping owned daemons; Compose has no restart policy and 240s grace, monitor join 45s. Host/container-platform lifecycle belongs to the operator, not a generated host manager. Historical transient acceptance units were unloaded and the tracked systemd unit removed.
- **2026-10-11 strategy absence:** genuine validated no-qualified-strategy results persist `max(2h, configured cooldown)` in SQLite, preserve longer existing deadlines and other leases, and survive monitor/container startup. Lightweight probes continue; heavy saved-first/scan/apply waits until expiry. Other ordinary outcomes retain 30m cooldown. Malformed reports and explicit nfqws2 infrastructure failures are vetoed before both positive and negative selection; they are never disguised as absence. Qualification remains both TLS protocols 3/3. The separate 90-minute whole-job watchdog plus bounded restoration remains abnormal, not normal absence.
- **Verification:** full Go suite, targeted race tests and shell/static/Python fixtures pass. Negative-only checker fault injection demonstrated a persisted 7200-second pause, unchanged checker count 7→7 through internal monitor restart, same container/PID1, DockerRestartCount0, zero candidates and graceful exit0. Active-check crash fixture demonstrated old separate-PGID workers gone before replacement activation, unowned session peer preserved, direct unchanged PID1/container and exit0. Private reports: `/home/agent/container-absence-acceptance-roqcpxq1/actualreport.json`, `/home/agent/active-scan-crash-5z2a6xb_/actual-report.json`. These validate control flow, not new network qualification; real two-hour wall-clock expiry was not waited out (expiry covered by injected-clock tests). Numeric PID signalling retains the usual read/signal PID-reuse race; strict reuse-proof signalling would require pidfds.
+83 -11
View File
@@ -1,16 +1,87 @@
# zapret2-autotune
![Warning Vibecoded](https://img.shields.io/badge/warning-vibecoded-orange?style=for-the-badge)
Go CLI for scanning and checking YouTube TLS bypass strategies with `blockcheckw`.
## Production preparation (acceptance pending)
The managed target now defaults to encrypted DoH for both SOCKS hostname traffic
and managed checks, with no plaintext fallback. Configure provider URL/bootstrap
IPs in the private deployment `.env`; native DNS requires an explicit opt-out.
Clients must enable SOCKS5 remote DNS (e.g. browser remote DNS on the chosen
loopback SOCKS port). See [DNS configuration and limits](deploy/managed-target/README.md).
This new integration is not covered by the earlier clean-run acceptance.
## Configured HTTPS checks (checks only)
Inside the owned managed target, with its configured DNS/SOCKS environment, run:
```sh
zapret-autotune check --config /path/to/resource-check.example.json
```
See [the public example](resource-check.example.json). This does **not** select
strategies, migrate candidates/transactions, implement metrics, or declare READY;
`check youtube` and the YouTube qualification contract remain unchanged.
Version 1 supports `kind: "http"`, GET/HEAD, exact HTTPS DNS-hostname origins
(port 443 only), `tls12`, `tls13`, or `both`. All fields shown in the example are
required except `content_type` and `min_bytes` (default zero); IDs/group are lowercase bounded identifiers.
There are at most 16 services and 64 resources, 100–30000 ms per TLS probe,
and 1–5242880 maximum decoded bytes. HEAD requires `min_bytes: 0`; its advertised
Content-Length is not downloaded or used as a body limit. No redirects are accepted.
Expected statuses are explicit, non-redirect 200–599 codes; MIME is an optional
exact media type (parameters are ignored in the response).
This slice checks HTTP status, MIME and byte bounds only: no image/media decode,
Range probes, suffix scopes or per-resource DNS settings.
Replace `url` with `url_env` for a private URL; optional `headers_env` maps header
names to environment variable names (e.g. `{"Authorization":"CHECK_TOKEN"}`).
Secrets travel only over private child stdin, never argv/results/database.
Transport/framing headers (including Range and Accept-Encoding) cannot be
overridden. Range probes need a future explicit probe field, not a secret header.
Config files must be regular, non-symlink files no larger than 1 MiB; permissions
are left untouched. Unknown fields, duplicates, missing/invalid settings and
environment references reject the entire configuration before any probes.
Checks issue explicit framed A/AAAA DNS queries to the configured local DoH
forwarder: no `/etc/hosts`, search domains or system DNS fallback. CNAME chains
are bounded and validated; unrelated Additional glue is never a destination.
The root-owned forwarder must match PID 1's private readiness anchors, executable,
PID/network namespaces and both UDP/TCP listeners; the owned Shadowsocks chain
must actually route its resolver to that same listener before any probes.
Explicit native mode requires an IP-valued `SS_DNS_SERVER` and is **plaintext DNS,
not encrypted DoH**. Every answer must pass
the destination policy, then the chosen IP is pinned for SOCKS CONNECT while the
original hostname remains the TLS/SNI/certificate authority. Addresses are resolved
fresh per probe, not stored. Private targets require `allow_private: true` **and**
explicit `allowed_networks` CIDRs. CIDRs always restrict; metadata, link-local,
CGNAT, loopback, benchmark and other reserved ranges remain denied even with that opt-in.
Output is one JSON line per resource/TLS probe: IDs, bounded outcome, TLS,
decoded bytes, milliseconds and HTTP status only. Exit 0 means all checks passed;
1 means a resource failed/cancellation/output failure; 2 means invalid config or
unavailable ownership. Resource failures continue to other resources; ownership
failure stops all probes. These checks have unit/loopback-fixture coverage only,
not new remote acceptance or generic automated recovery/controller support.
Generic `managed resource-status` validates the authenticated snapshot against
the live controller's PID/starttime, executable, command, UID, namespaces and
held directory/key/state/singleton inodes. Cached check results alone cannot
report current health: missing ownership or an identical signed replacement
returns `BLOCKED` with `stale: true` without modifying the foreign file.
`CHECKS_ONLY`/`HEALTHY` still do not mean candidate or production readiness.
## Production preparation (isolated clean-run acceptance passed)
The standalone deployment candidate is a **new isolated target**, never an
upgrade/adoption of an existing zapret or Xray service. The legacy `managed-test`
role and fixed identity guards remain intentional, including for prod-like runs.
See [release, configuration, upgrade and rollback guidance](MANAGED_TARGET.md#production-preparation)
and the [container deployment instructions](deploy/managed-target/README.md).
Pinned builds and supervision are preparation, not production acceptance: real
restart/deadline tests and end-to-end target validation remain rollout gates.
The clean RC `d8316e6` passed isolated bootstrap, qualification, ongoing probes,
monitor-crash cleanup, graceful stop and persistent recreation on loopback SOCKS
`11081`; the existing target on `11080` remains untouched. See the deployment
guide for the exact source/image and private PASS report. This is not blanket
production approval or playback verification; no real 90-minute expiry wait was
performed. Each production rollout still needs its own validation.
Select an immutable managed-target digest with `ZAPRET_AUTOTUNE_IMAGE` in the
private deployment `.env`; tags/local defaults are only build/test candidates.
Bootstrap/recovery shares a 90-minute whole-job deadline plus a separate 30-second
@@ -30,7 +101,7 @@ recovery preserve a configured recovery cooldown longer than two hours.
This is not a watchdog timeout:
ordinary recovery cooldown stays 30m, and the abnormal whole-job ceiling stays 90m.
## Manual managed-test activation (not deployed)
## Manual managed-test activation
`managed init|reconcile|status|active-ids|apply` requires exact role `managed-test`, target ID
`zapret-autotune-managed-v1`, a provisioned runtime marker and zapret executables,
@@ -48,14 +119,14 @@ group, fixed target init executable and explicit runtime-config
binding. It uses full target-local firewall/daemon restart and Player API post-check, with
atomic rollback on failure. See [MANAGED_TARGET.md](MANAGED_TARGET.md) for guards
and provisioning prerequisites. An isolated pinned deployment definition is in
[deploy/managed-target](deploy/managed-target/README.md); acceptance remains pending.
[deploy/managed-target](deploy/managed-target/README.md), including recorded clean-run PASS.
An injected bootstrap/saved-first recovery adapter now connects the persisted
monitor foundation to blockcheck, immutable candidates and transactional apply.
`managed monitor` and the dedicated target entrypoint now wire these adapters;
this increment is not tested by the producer or live/runtime verified.
No remote activation or Xray changes are included. Image
compatibility and route/rollback gates are not yet remotely proven; adapter
tests/build are deferred to the orchestrator.
the isolated target has been remotely activated and runtime-verified as recorded
in [PROGRESS.md](PROGRESS.md). Existing targets and Xray remain unchanged.
Clean-run PASS does not imply every recovery/fault gate was exercised in that run;
the progress log distinguishes earlier tests and remaining production limits.
The apply flags are the entire desired group set, not a merge: a single-group
apply removes the other group. Frontend-only supports bootstrap discovery;
@@ -96,8 +167,9 @@ not used and there is no direct fallback. The request times out after 30 seconds
| `ZAPRET_AUTOTUNE_FAILURE_THRESHOLD` | `3` |
| `ZAPRET_AUTOTUNE_RECOVERY_COOLDOWN` | `30m` |
Configuration is parsed once per health invocation. The proxy must use `socks5`
with a loopback IP or `localhost`, port 1–65535, and no credentials/path/query.
Configuration is parsed once per health invocation. The proxy must use `socks5` or `socks5h`
with a loopback IP or `localhost`, port 1–65535, and no credentials/path/query/fragment.
Explicit `socks5h` uses proxy-side hostname resolution (encrypted DNS only with managed DoH configured); explicit `socks5` retains local DNS semantics.
Video IDs must be 11 characters; durations and the failure threshold must be
positive. Explicit empty values are invalid. Interval, consecutive-failure
threshold and cooldown configure `managed monitor` (the one-shot command still
+166
View File
@@ -0,0 +1,166 @@
# Periodic configured resource checks
`managed monitor --resource-config FILE` runs resource-config v1 or v2 periodically.
PID 1 selects the same mode when `ZAPRET_AUTOTUNE_RESOURCE_CONFIG` is nonempty;
unset preserves the existing YouTube monitor. The modes share one singleton
lock. Supply an absolute container-visible config path (for example mount a
read-only operator config directory using a Compose override). Configuration
must be a regular root/runtime-owned, non-writable-by-others file without
symlink/hardlink aliases. Environment references are resolved on every reload.
Each resource accepts `interval_ms` (default 300000) and `cooldown_ms` (default
30000), both integers in [100, 86400000]. Checks are sequential round robin
with independent due times; `tls: both` requires both passes. Ordinary DNS,
TLS, timeout or expectation failures mark only that resource DEGRADED and
retry after its cooldown. There is **no strategy selection, recovery or apply**,
and no no-strategy two-hour pause. HEALTHY means configured checks passed,
not playback health, generic recovery or whole-production READY: HEALTHY is
**CHECKS_ONLY**, never READY. The scheduling, status, persistence and lifecycle
base passed native full/race checks, independent security review and isolated
remote fault acceptance for `4efc066` plus ownership-status fix `5c85f3f`.
See [PROGRESS.md](PROGRESS.md) for supplied evidence and its narrow scope.
Both versions are bounded to 16 services / 64 resources with stable configured IDs;
valid configuration/state does not imply an automatic strategy assignment.
V1 remains HTTP GET/HEAD only, with explicit `kind: http`. See
`resource-check-v2.example.json` for v2: omitted kind defaults to `http`; all
expectations explicitly declare statuses/content_type/min_bytes/max_bytes.
`image` is GET-only, [200], PNG/JPEG with full bounded decode (5 MiB maximum,
16 MiPixels); public GIF is deferred pending an aggregate frame/pixel budget,
without changing legacy GIF adapters. `mp4_range` is GET-only, [206], audio/mp4
or video/mp4, 16–65536 bytes, checked Content-Range and ftyp. Range stays fixed
at 0-65535; a smaller max_bytes bounds the actual response, not the requested end.
Operator Range/Accept-Encoding headers remain forbidden in both versions.
Timeouts (100–30000 ms) and declared byte limits apply to every kind. Typed
wire-valid semantic failures degrade checks; malformed/unsafe execution cannot
qualify a candidate. V2 version/kind/expectations invalidate old HMAC rows/proofs;
historic v1 fingerprints remain unchanged. MP4 prefix health is not playback or
YouTube proof: three discovered CDNs, >=1024-byte producer samples, warmup,
suffix authorization and adaptive policy still belong to the legacy adapter.
The private compiled-in YouTube module and core `executeAdapterPlan[T]` bridge
now drive the existing owned qualifier's ordered warmup and steady collection.
They do not add public module configuration or change this monitor's schemas,
controller, suffix authorization or legacy SOCKS5h routing boundary. Generic
controller cutover still needs dynamic-inventory fingerprints and warmup-bearing
generic proof binding; images/avatars and full YouTube READY remain unresolved.
Local source/closure review and injected-wire checks are
recorded alongside scoped managed v2 field observations below; full acceptance
is not implied.
### Scoped managed v2 monitor observations (2026-10-15)
Independent closeout at **2026-10-15T12:28:52Z** records **SCOPED_PASS** for source
`cd05e75189f90c32e1346536c50f0973e7105cb3`, audit runtime unchanged. See
[RESOURCE_PLAN.md](RESOURCE_PLAN.md#managed-v2-scoped_pass-observations-2026-10-15)
for the root-private report/audit pointers, qualification/apply counts, dated
cleanup and full evidence limits. Official owned PID1 SS/DoH/public v2 paths
qualified PNG/JPEG and video/audio MP4 prefixes on both TLS versions (16 M3
cells / 48 TLS transfers); eight apply and eight initial monitor checks passed.
Five HMAC-authenticated snapshots, including initial state, show real host-fixture
PNG truncation and MP4 total overflow degrading only the affected row to
`DEGRADED / expectation_failed`, followed by all-HEALTHY repair. The SAME
controller PID 379/start 62777447 and journal/config/proof/key/input hashes were
retained, without restart or requalification. HEALTHY is still CHECKS_ONLY.
Both official stops exited 0; dated audit found all five owned containers,
owned network and 174 recorded process identities/namespaces gone, and three
foreign raw inspect snapshots unchanged. Historical Xray equality is only a
harness assertion without retained initial raw process inventory.
Typed semantics are supported by real production validation, authenticated
outcomes and fixture metadata, **not independent re-decoding of retained bodies**:
bodies were deliberately not retained for privacy. Live executable/held-FD
snapshots were also not retained; PID/start/parent/netns/native argv checks and
reviewed ACK enforcement are not a durable live-FD certificate. The initial
DoH-forwarder/worker observer misclassification was corrected in the harness,
retaining logs and the same qualified proof, not fixing a product failure.
No new public-v1 managed field run, negative M3/cancel/freshness/old-matrix remote
campaign, actual YouTube discovery/three-CDN/suffix/adaptive/warmup/avatar/playback
was run. Local v1 parity and unit rejection coverage do not fill those gaps.
The static test image did not replace the user's working SS service. Missing
body/live-FD evidence and untested cases remain limits before full acceptance
and YouTube adapter cutover; there is no full safety, migration or READY claim.
Invalid resource JSON initially or after reload is BLOCKED, with no DNS or
transfers until repaired. Bad static target role, proxy or DNS settings still
fail startup validation; resource JSON is deliberately not part of
`target-config-check`. Ownership and fresh source snapshots are checked before
DNS and again before transfer; existing DNS/IP policy, pinned SOCKS transfers,
TLS verification and private stdin are unchanged. No cached addresses bypass
DNS policy. Existing zapret reconciliation remains mandatory and generic
startup never consumes experimental apply requests.
State is independent of recovery SQL leases, fixed in the private target
directory as `.resource-monitor-state.json` and `.resource-monitor-key`.
Both are private 0600 files. Fingerprints HMAC the resolved configuration
(including secrets); state is also authenticated and never contains URLs,
headers, addresses or raw errors. Missing state initializes UNKNOWN. A missing
key with existing state, corrupt or unsafe file is preserved and BLOCKED,
never silently replaced. Restore the original key and a valid version-1 signed
state from a trusted backup to repair it. The controller holds directory, key
and state inode anchors: an identical signed replacement is still BLOCKED and
preserved. Repair on the original held inode is reloaded only after version/MAC
validation; replacing an inode requires deliberate operator repair/restart.
A persistence failure pauses probes;
deliberate repair requires a different valid authenticated snapshot, not just
retrying a failed write. Never chmod/adopt unknown files to bypass validation.
This is not isolation from a hostile same-UID actor: that actor can read the
HMAC key and race filesystem publication; root/container administrators can
also read private data. HMAC fingerprints protect stored secret values from
unkeyed guessing, not from an actor with access to the key.
`managed resource-status` is read-only and reports BLOCKED or STALE explicitly
(a check overdue by >65s, unknown state age >65s, or changed configuration).
Idle reloads do not write a disk heartbeat: only configured state changes,
BLOCKED transitions and completed checks are flushed. Cancelled checks join
their children but do not change prior health, counts or completion/due times.
It reads the config path from
`ZAPRET_AUTOTUNE_RESOURCE_CONFIG`; use that environment setting also for manual
monitor/status invocations. Status rows report only configured-check health.
TERM cancels/joins transfers before finalizing a completed snapshot and exits
0 on safe cleanup. PID 1 retains DNS through monitor/SS cleanup. In generic
mode transient child restart-budget exhaustion pauses until the rolling window
opens, rather than terminating the container. Unknown/unsafe cleanup stays
BLOCKED pending operator repair/restart; no unsafe daemon sourcing is allowed.
Generic cleanup shares an 80-second budget, including watchdog setup and
command kill escalation, with bounded final reaping; it is designed to fit
within a 90-second Docker stop limit (the supplied Compose grants 240 seconds).
Bounded final reaps/worst-case cleanup were verified offline; measured remote
normal stops were 1.45s, 0.59s and 0.57s, not an 80-second runtime worst-case test.
These changes do not claim to fix legacy YouTube exit 78.
### Internal service-adapter boundary (local extraction, field acceptance pending)
Legacy YouTube execution now explicitly selects compiled-in `youtube/1` via
`cmd/zapret-autotune/service_adapters.go`. `internal/adapter/youtube` parses
bounded Player media JSON and supplies signed-MP4 freshness/diversity semantics
and the owned qualification plan. Unknown/absent kind or version fails closed,
without a generic HTTP fallback. The shared `internal/adapter` DTOs are internal
only: signed URLs/inventory and plan caps have `json:"-"` fields.
The existing controller still owns discovery requests/retries, transport,
origin/SNI/certificate/TLS checks, worker guards/cleanup and sealing after join.
Legacy probes retain SOCKS5h and the owned encrypted-resolver guard; they do not
run the generic all-A/AAAA approval/connect-to IP pinning path. Full request-policy
capabilities (including future metadata POST) and parity are prerequisites before
generic cutover, not guarantees supplied by this minimal extraction.
Public generic checks and their HMAC/state formats have not migrated to this
module interface. Suffix/experimental authorization is separate from measured
endpoints, and avatar/thumbnail health remains independent. A module plan is
neither proof nor permission to claim READY.
New adaptive proof v2 seals adapter kind/version/policy digest along with the
measured facts. Unknown identities, mismatched policy, missing attestation and
unsealed DTOs cannot publish adaptive evidence. Legacy v1 is explicitly the
immutable historical YouTube policy for journal reconstruction, not a request
to reinterpret old evidence using the latest module. Expired historical proofs
can reconstruct their owned config. Fresh trusted stored v1 retains its original
apply eligibility; expiry still blocks fresh apply. Only new producer writes
require sealed module-bound v2; public/unsealed DTOs cannot publish evidence.
Discovery/asset/health entrypoints preflight the required kind/version/policy
before requests. Missing/mismatched modules are fatal configuration failures,
not retryable inventory absence or a genuine-no-strategy cooldown.
No public module field, dynamic plugin ABI, controller cutover, same-UID sandbox,
remote YouTube acceptance, or fix for legacy exit 78 is claimed here.
+849
View File
@@ -0,0 +1,849 @@
# Resource compatibility planner and isolated qualifier
## First collector migration (shared-core parity verified; managed v2 scoped observations)
Private typed response contracts now select generic GET/HEAD, full decoded image,
or bounded MP4 range validation inside the existing shared HTTPS transfer core.
The redundant legacy forwarding wrapper and media boolean are removed; image and
media adapters retain their measurements and GooglevideoPass DTOs. Generic v1
status/MIME/byte expectations use the same validation entry, with a distinct
completed-mismatch result retaining the same measured facts. No public Range config, schema,
proof importer, registry or dependency was added.
Injected-wire tests exercise the real generic producer and legacy callers, both
TLS versions, normalized byte/hash/timing facts, private stdin/pinning, origin,
redirect, framing, bounds and strict range/ftyp/image decoder rejection. Direct
core tests ensure strict validation is not an unused legacy-only helper.
MP4 requests use Range `0-65535` (a 64 KiB upper bound, not an exact response
size requirement). The core accepts validated 16–65536-byte ranges; the existing
producer additionally requires at least 1024 measured bytes. Content-Range end
and total must parse without overflow, with total greater than end; rejected or
unknown totals cannot produce a media PASS DTO.
This is collector migration only: public schemas and the legacy YouTube producer
remain unchanged, including three discovered CDNs, adaptive A/E, warmup then 3/3
samples on each TLS version, explicit googlevideo.com suffix authorization, fresh
discovery, and independent qualification of image/avatar assets. The legacy path
is retained. Generic GET success is never full YouTube qualification.
Recorded local verification: archived `2149a4a` versus current source passed 86
parity subcases; 14 new integration cases, 11 focused EUID-0 serial/race tests and
full Go tests passed (full Go about 55s). Python static/config/startup checks passed;
the long supervision check timed out at 18s and again at 120s and is unresolved.
Do not call all tests green or infer a regression from that timeout: entrypoint
code was unchanged and later full Go entry tests passed. These are prior results,
not reruns in this docs-only update.
### Scoped remote shared-core parity (2026-10-15)
Privileged-read evidence: `/home/agent/core-runtime-parity-45d28e3-20261015/report.md`.
Exact-source baseline `2149a4a` and current
`45d28e37330069ddf0efa3120fec8a558d37ba7f` each passed 48 real curl/HTTPS/SOCKS5h
fixture subtests (24 cases × TLS1.2/1.3). Of the normalized rows, 46 matched and
two intentionally differed: overflowing Content-Range total, formerly saturated
by ParseInt, now rejects without a PASS DTO. Known negative classifications are
preserved; invalid/unknown/overflow totals are not proof.
Coverage includes GET/HEAD expectations and retained mismatch facts, bounds,
redirect/framing/trust rejection, real PNG/JPEG/GIF decode and rejection, strict
206 range/ftyp checks, and stdin/argv/result privacy sentinels. Real 65536-byte
and 1024-byte media responses passed the same Range request. Each version made
60 SOCKS5h connections/SNI handshakes and 58 authenticated HTTP requests, including
18 actual Range requests; two untrusted-certificate attempts stopped before HTTP.
Original hostname/SNI was retained; curl verified a fresh private fixture CA,
without TLS bypass. Timing/speed were positive but excluded from exact normalized
parity hashes; this is not a performance comparison or fatal-reason-tree parity.
The proxy maps only allowlisted fixture names to a loopback TLS listener: it is
not an SS proxy or DoH resolver. **NOT RUN:** normal public CLI under owned PID1
SS/DoH, managed ownership/guard checks and probe orchestration, NFQ/native behavior,
real YouTube discovery/three-CDN qualification, or candidate/apply/monitor flow.
No DoH validation or production qualification follows from helper tests. No
candidate/PASS/DB/JSON proof imports occurred. Helpers exited and joined listeners;
no owned containers/networks were created. Root-private artifacts, keys, manifests
and hashes remain retained. Selected foreign identity/running/start facts matched
before/after, not arbitrary foreign application-data hashes.
### Public typed checks (v2; scoped managed field verification)
`resource-check-v2.example.json` adds closed `kind` values `http`, `image` and
`mp4_range`; omitted v2 kind canonicalizes to `http`. V1 still requires explicit
`http` and retains its serialization/fingerprints. V2 requires explicit statuses,
content_type, min_bytes and max_bytes. HTTP retains GET/HEAD semantics; typed
checks require GET. Image requires exactly [200] and image/png or image/jpeg,
full decoding, positive dimensions at most 16 MiPixels and a configured body
ceiling at most 5 MiB. Public GIF/omitted image MIME are deliberately deferred
until a decoder with an aggregate frame/pixel budget exists; legacy GIF remains.
MP4 requires exactly [206], video/mp4 or audio/mp4, and 16 <= min <= max <= 65536.
The core always requests bytes 0-65535, even with a smaller max_bytes; actual
returned bytes must fit that smaller bound and the validated Content-Range/ftyp.
This proves a bounded MP4 prefix, **not playback or YouTube qualification**.
All kinds use configured 100–30000 ms timeouts and byte ceilings, private stdin,
strict framing/TLS/origin and all-answer DNS policy through the same public check,
periodic monitor, isolated M=3 qualifier and apply-health path. Wire-valid typed
semantic failures are complete expectation negatives; unsafe output, malformed
metrics/framing, cancellation and ownership failures cannot become benign proof.
V2 source and row HMACs bind version, normalized kind and expectations, so old
plain-HTTP evidence/state cannot promote typed checks. V1 historical journal
reconstruction is unchanged. No public proof import or module framework exists.
Local verification for this v2 slice: full Go tests (`-p 1 -parallel 1`), affected
resource race tests, vet, CGO-disabled build, four managed-target Python fixtures
and shell syntax passed. Current and archived `45d28e3` each passed the existing
86 legacy adapter/generic-mismatch parity subtests. Privileged tests ran as EUID 0
inside the current user namespace (UID 0 maps to host UID 100000), not host-initial
root. These are local injected-wire/fixture results, not remote network acceptance.
Next let the YouTube adapter supply qualified exact endpoints/dynamic inventory into
the generic registry, preserving fresh discovery and the three-CDN contract.
Keep measured endpoint proof separate from explicit suffix/adaptive-learning
authorization: both capabilities are prerequisites to complete managed migration.
No plugin framework, legacy deletion or full controller/journal/profile migration
is implied. Full runtime acceptance remains incomplete; avatars and legacy exit78 remain
unresolved. No production READY claim.
### Managed v2 SCOPED_PASS observations (2026-10-15)
Source `cd05e75189f90c32e1346536c50f0973e7105cb3`. Privileged reads of
`/home/agent/managed-v2-cd05e75-20261015/report.md` and
`field-private/independent-audit.safe.json` record independent closeout at
**2026-10-15T12:28:52Z**, verdict **SCOPED_PASS**, with
`runtime_state_changed_by_audit: false`. These are bounded positive observations,
not unequivocal whole-criterion acceptance, full safety or READY. The static test
binary/image is not a deployment to the user's working SS service. The audit
matched 137 archived product files and the pinned native NFQ engine (1.0.2).
- Official public v2 CLI under owned PID1 SS/DoH exercised PNG/JPEG full decode
and video/mp4 + audio/mp4 bounded prefixes, each over TLS1.2/1.3 with verified
fixture CA/SNI. Two candidates × four resources × both TLS versions produced
16 M=3 cells / 48 real qualification TLS transfers. Queue-200 worker argv and
PID/start matched; exact-payload registry/matrix HMACs, v2 source and semantic
row fingerprints were independently recomputed, not imported PASS evidence.
- Official apply reused that proof without requalification: queue 300, canonical
`multidisorder`, three exact fixture hosts, journal v4 / generation 1 / committed
generic. Eight actual typed apply checks and eight initial monitor checks passed.
Source fingerprint:
`cc5a2622133daed58df0b2c22254f6112a3d9da7fbd9391ce7d1c6f467761886`;
committed config SHA-256:
`27f9a8ffd56a4c19d4bd522dd9f5a5e8b49b18b0ae48d41431ef9b22975ea772`.
- Five authenticated monitor snapshots (including initial state) showed truncated
PNG and overflowing video MP4 Content-Range each degrading **only** its affected
row to `DEGRADED / expectation_failed`, then repair restoring all four HEALTHY.
Host fixture changes produced real responses; controller PID 379/start 62777447,
journal/config/proofs/key/input hashes stayed the same, without requalification.
Apply/monitor wire evidence totals 48 transfers (33 healthy-mode, 7 PNG-fault,
8 MP4-fault); fixture timing/size/status is not durable product DTO telemetry.
- Dated closeout independently rechecked both official target stops exited 0;
all five recorded owned containers, owned network and 174 recorded PID/start
identities/namespaces were absent. Three foreign raw Docker inspect snapshots
remained byte-identical: `f6c9d42858e1` running, `4ad54175a54c` and `8cc36c813d6f`
stopped. Historical Xray inventory equality is a harness assertion only: its
initial raw process inventory was not retained, so it is not independently proved.
**Evidence limits:** production strict validation, authenticated outcomes and actual
fixture metadata establish this scoped typed-response observation. Bodies were
intentionally not retained to avoid data exposure and were **not independently
re-decoded**. Actual PID/start/parent/netns/native argv were independently checked;
live executable/held-FD snapshots were **not retained**. ACK/snapshot guards are
reviewed code/log enforcement, not a durable live-FD certificate. These missing
body/live-FD certificates prevent an unqualified full acceptance claim.
The initial observer misclassified the intentionally live DoH forwarder as an
unjoined worker. Original logs/harness remain; the corrected continuation used
the SAME proof, with no new qualification. This is a harness assertion correction,
not an observed product bug.
**NOT RUN in this field campaign:** public v1 managed GET/HEAD (86 current/old
local parity cases passed, not a remote v1 run); negative M3 qualification,
cancellation, freshness/24h expiry or old-matrix rejection; actual YouTube
discovery/three-CDN, suffix authorization, adaptive learning, warmup, avatar or
playback. New kind/version cannot promote old proofs by local unit coverage,
not a fresh remote rejection campaign. MP4 proves a prefix, not playback or an
exact 64-KiB body; public GIF remains deferred and legacy GIF retained. Prior full
Go/race/Python results are recorded above, not rerun for this docs-only change.
**Next minimum slice (design, NOT implemented):** a real YouTube/core adapter
boundary wrapping existing discovery and strict qualification calls, not a full
framework. Core owns networking, limits, proof, persistence and apply; YouTube
owns fresh URLs, three-CDN warmup/sampling, per-host adaptive learning and explicit
suffix policy. Plan a minimal compiled-in Go registry keyed by kind + version,
fail closed for unknown contracts; bind module/policy version into source/proof
and preserve authenticated historical restore under its original contract. No
dynamic plugin/RPC API exists. Do not generalize public POST/discovery until an
actual adapter needs that request contract. Retain legacy functionality until
parity and the remaining acceptance gaps are closed before full adapter cutover.
`plan` remains **proposal-only** and never changes runtime or config.
Explicit generic startup apply/replay and the per-PID-1 failure latch have passed
bounded remote acceptance with independent privileged verification (see below).
This is not automatic recovery, full concurrency acceptance or production READY.
Explicit isolated qualification temporarily installs an owned NFQ worker and
queue rules, but never changes legacy SQL groups, renderer or apply journals.
Monitor HEALTHY, old SQL measurements and public check JSON are not proof.
Without a completed owned qualification batch, compatibility is UNKNOWN and
hosts are unassigned. No PASS/evidence import is exposed.
Commands require root and the existing `managed-test` target settings:
```
zapret-autotune strategy-import --strategies /private/strategies.json
zapret-autotune plan --config /private/resources.json
zapret-autotune qualify --config /private/resources.json --strategies /private/strategies.json --isolated
```
`strategy-import` requires an owned, regular, single-link 0600 file. Only
definitions are accepted; duplicate keys, unknown fields and evidence fields
are rejected. Example:
```json
{
"version": 1,
"strategies": [
{
"id": "split-one",
"args": "--payload=tls_client_hello --lua-desync=multisplit:pos=1",
"selectors": {"transport":"tcp","ports":"443","l7":"tls","tls":"both"}
}
]
}
```
The registry is one immutable batch (1–32 unique IDs). This conservative slice
accepts only TCP/443/TLS selectors and `tls12`, `tls13`, or `both` policy.
Argument whitespace is canonicalized; option/Lua stage order is preserved.
Existing payload/range validation is reused. Lua stages are restricted to
fake, multisplit, multidisorder, fakeddisorder, split and disorder; parameters
to pos, repeats, tcp_ts, tcp_flags_set, seqovl and blob=fake_default_tls.
File/inline Lua, queue/user/fwmark/filter options and arbitrary stages are
rejected. Acceptance of a definition is **not** runtime verification.
The private target directory holds `.resource-plan-key` (32 random bytes),
`.resource-plan-strategies.json` and, when an owned producer exists,
`.resource-plan-matrix.json`. JSON envelopes have `payload` and an HMAC-SHA256
`mac`. Each file is write-once, owned, 0600, no-follow, single-link and bounded
to 64 KiB. Held directory/file identities and exact snapshots are checked;
foreign entries, replacement inodes and modified bytes are never clobbered or
silently adopted. Interrupted writes remain blocked for explicit operator
repair. There is no automatic rotation/update/repair in this slice. The private
key and same-UID/root boundary must be protected: HMAC is not protection from an
actor who can read the key and replace the whole target state between processes.
Matrix v1 binds the registry digest and at most one completed run per candidate.
Each run binds ID, definition digest (canonical args **and selectors**), keyed
config fingerprint, start/completion milliseconds, resource/TLS outcomes and
keyed resource fingerprints. Its attestation records collector/runtime,
definition digest, composite executable/Lua input hash, installed rules hash,
actual worker-argv hash, queue and worker PID/start identity, plus
restored/joined completion. No URLs, headers, resolved IPs, bodies or raw errors
are persisted. A nonserialized producer seal protects the completed in-memory
snapshot against mutation before publication. Only the owned producer and
test-internal fixtures create seals; JSON DTOs cannot authorize publication.
Every resource/protocol cell records exactly three actual attempts; any nonpass
in those attempts makes the cell negative (never an average or partial PASS).
`plan` loads only authenticated private state and the existing resource-config
snapshot (including resolved environment secrets, DNS/proxy and private headers
in keyed fingerprints). Every resource on an exact host, across all services
and input group labels, must have matching complete results for every required
TLS protocol. Complete failed checks are negative; missing, expired (24 hours),
future, mismatched or partial checks are unknown. Changed config invalidates old
run evidence. Malformed/unauthenticated state yields blocked, unassigned output
and exit 78; valid missing evidence yields unknown output and exit 0.
Output contains `proposal_only`, `evidence` (authentication status, **not**
health), per-host `validated`/`negative`/`unknown` candidate alternatives,
`assignments` and `unassigned`. Compatible equal args/selectors merge coverage
across input labels. Greedy greatest-uncovered coverage, then lexical candidate
ID, produces deterministic disjoint exact-host assignments. It does not claim
global optimality or suffix coverage. Explicit legacy YouTube suffix policy is
unchanged.
## INTRUSIVE / ISOLATED startup only
Set **both** `ZAPRET_AUTOTUNE_QUALIFY_CONFIG` and
`ZAPRET_AUTOTUNE_STRATEGY_CONFIG` to container paths on a dedicated target.
Do not send working user traffic to this target's published SOCKS port. This is
not noninterrupting background search. Provision the private resource and 0600
strategy files separately; compose forwards paths but does not mount files for
you. Start with the managed disabled baseline, not an existing active profile.
Qualification v1 accepts lowercase DNS hostnames, not literal IP targets; the
ordinary resource-check command's broader literal-IP policy is unchanged.
PID 1 starts its owned SS+DoH services, initializes/reconciles known state, then
launches one finite `qualify ... --isolated` session **instead of** starting the
live zapret daemon or monitor. Qualifier checks root/target namespace, PID 1's
actual entrypoint/environment, direct-child executable/arguments/session ACK,
owned SS and DNS readiness anchors. A flag or docker-exec environment alone
does not authorize qualification. Missing either startup path fails closed.
Without `--isolated`, `qualify` remains UNKNOWN/exit 78 without runtime changes.
The shared monitor singleton excludes legacy and generic controllers. Any
existing NFQ worker/queue, active profile, unknown config/journal or existing
immutable matrix is refused. Known baseline config/journal/backup/identity
inodes and bytes remain held unchanged throughout the job. This first producer
does **not** pause/stop/restart existing active profiles; job-bound maintenance
rollback is deferred rather than pretending a path-only daemon stop is safe.
Each candidate uses the existing worker-before-nft/queue-ready lifecycle on
queue 200, fixed Lua imports and fwmark 0x10000000, plus native
`--filter-l7=tls`. The actual child executable, argv, parent, start identity,
queue socket ownership, executable/Lua hashes, native `--version` output 1.0.2
and installed nft rule bytes are attested before capture and checked again at
each collector guard. DoH's existing 0x50000000 socket mark bypass remains.
TLS policy is collector/eligibility policy, not a nonexistent native TLS-version
filter. All configured protocols are measured; incompatible selector policies
remain UNKNOWN in the proposal.
The existing GET/HEAD generic collector preserves TLS origin verification,
configured body/status/content-type expectations, private stdin and IP pinning.
Completed expectation failures and typed peer response/TLS/transport negatives
do not stop the remaining repeats/cells. Guard/source/unknown infrastructure or
batch cancellation veto the entire batch; no partial positive matrix is saved.
Config (including resolved environment/header secrets) and strategy-source
snapshots are checked after DNS/before transfer and before sealing/publication.
Workers are joined and temporary nft queues removed before verifying unchanged
disabled baseline/runtime and sealing. The batch has a fixed 15-minute deadline
with separate bounded cleanup; PID 1 keeps SS+DoH alive through cleanup.
The bounded 64 KiB matrix size (including its authenticated envelope and
worst-case field widths) is checked before any worker/nft operation; excessive
catalogue/resource combinations return UNKNOWN/exit 64 even within individual
count caps (32 strategies, 64 resources).
Successful batches, including complete negatives, return the assignment proposal.
Incomplete/unsafe jobs return UNKNOWN/blocked exit 78 without proof. PID 1 then
stays safe-idle with SS+DoH for inspection, for either success or blocked
completion: no qualification retry loop, duplicate controller or automatic
generic monitor start. Retests require future explicit immutable snapshot
rotation, not overwriting prior evidence.
Completed controller cleanup is acknowledged separately from qualification in
an exclusive, 0600 HMAC-authenticated `qualification-completed.json` on the
current boot's SS tmpfs. It binds the launched PID, exit outcome (0 or 78) and
verified cleanup; it is **not** strategy proof. PID 1 checks that ACK against its
private session launch and revalidates the disabled baseline, singleton,
SS/DoH ownership and absence of NFQ workers/temporary nft queues before shutting
down dependencies. Thus TERM after a clean blocked qualifier has exited but
before its parent reaps it can exit cleanly. Worker/nft/join/restoration failure,
missing/foreign/mutated ACK or unverified early failure still veto clean shutdown
with exit 78. Ordinary legacy/generic monitor exit 78 remains a safety veto.
No unsupported `--filter-mark` or blockcheck embedded/stream/progress/deadline
flags are used. Background noninterrupting search, active-profile maintenance
and immutable snapshot rotation for repeated runs remain deferred.
## Explicit generic transactional startup apply (local implementation)
On the target with its already completed private qualification batch, unset both
qualification startup paths and set:
```
ZAPRET_AUTOTUNE_RESOURCE_CONFIG=/private/resources.json
ZAPRET_AUTOTUNE_RESOURCE_APPLY=1
ZAPRET_AUTOTUNE_STOP_GRACE_PERIOD=90s
```
Compose forwards the apply flag and configurable stop grace (legacy default
remains 240s). Generic PID 1 cleanup has one shared 80-second budget. No external
manager, new proof importer, SQL frontend surrogate or parallel journal exists.
PID 1 starts owned SS/DoH, reconciles known bytes, launches the acknowledged
`managed resource-apply --resource-config FILE` child before the checks-only
monitor, and starts the native daemon only with verified committed config.
This is at most one apply launch per PID 1 lifetime. A RAM latch is marked failed
before launch and succeeds only after child wait=0 and owned-session cleanup.
Launch/ACK failure, nonzero/signal exit, SS/DoH loss, or incomplete cleanup leaves
it failed: after authoritative cleanup PID 1 remains cancellably BLOCKED, without
restarting SS/DoH, native zapret, apply or monitoring, even across former 60s retry
windows. Cleanup still stops SS/DoH (DNS last); BLOCKED does not keep them alive.
Successful apply is skipped on later monitor/SS restarts, but every runtime cycle
still reconciles committed refs before native start. Qualification and legacy
retry behavior are unchanged.
**Ceiling:** this is not a durable manual-intent latch. A fresh PID 1 (including
an external Docker restart policy) resets authorization and, with flag=1, permits
one new attempt. Operators must disable unattended external restart/recreation
for this workflow. Set `ZAPRET_AUTOTUNE_RESOURCE_APPLY=0` for explicit repair-only
startup: prepared state is reconciled, never promoted or reapplied. Identical
committed requests still resume by authenticated refs without a new transaction.
The command refuses standalone/docker-exec callers: actual PID 1 environment,
entrypoint, namespace, direct-child executable/argv and held launch ACK are
required, plus the existing controller singleton and no pre-existing NFQ worker.
First apply supports **disabled baseline → generic**. Explicit PID 1 startup
selection also permits generic → generic rotation among already verified
alternatives in the **same immutable authenticated registry/matrix/source**.
Active legacy profiles, changed source/catalogue/proofs and uninterrupted live
rotation remain unsupported. Every configured resource must require `both` TLS protocols and
every assigned exact host must have complete, candidate-bound passing proof for
all its resources across all input labels. New apply requires completion within
24 hours, never in the future. Negative/unknown/unassigned hosts cannot authorize
apply. Registry definitions may still describe single-TLS proposals, but native
apply rejects them instead of widening unsupported TLS-version selectors.
Canonical ordered args/selectors define profiles separated by `--new`; each
contains `--filter-tcp=443 --filter-l7=tls --hostlist-domains=^a,^b`. Exact-host
semantics use the pinned native source
`https://raw.githubusercontent.com/bol-van/zapret2/4df94497869e18ed4fdf1272141427b2fbfc6604/nfq2/hostlist.c`:
inline `AppendHostlistItem` uses `addpool`, leading `^` sets
`HOSTLIST_POOL_FLAG_STRICT_MATCH`, and `SearchHostList` rejects suffix fallback
when that flag is set and `!bHostFull`. Existing YouTube suffix policy is unchanged.
The packaged executable/Lua hash must match private attestation; held native,
Lua, source, marker and proof identities/bytes are guarded through the transaction.
No active queue version probe or mark/filter change is made (DoH500/probe100/live400).
The **existing** journal gains v4 independently tagged committed/pending refs
(`baseline`, `legacy`, `generic`). Generic refs exclude legacy IDs/scopes and bind
exact authenticated registry/matrix envelope hashes, source fingerprint,
internally derived canonical assignment digest and complete config hash.
Optional scalar `strategy_ids` retains the canonical selection policy on each
generic ref; missing/empty means the original default greedy policy. No policy
is permitted on baseline/legacy refs. This remains v4: older strict-decoding
engines reject the new field, so downgrade is blocked, not silently reinterpreted.
Unknown versions/tags, mixed refs and unresolved historical reconstruction block
without adopting or erasing foreign state; v1–v3 legacy behavior remains.
Staging/backup/prepared/replace/fixed restart/health/commit reuse the same engine.
All required resources/TLS are checked through the existing synchronous owned
SS/DoH collector (all-answer SSRF validation, CONNECT pin, SNI/cert validation,
private stdin). No Player API health or public check JSON authorizes commit.
Committed identical requests are idempotent: historical proof expiry alone does
not trigger reload, reapply or rejection on restart. Authenticated integrity,
source and native drift still block. Prepared recovery **never** promotes pending
bytes, including after successful health: it restores prior bytes and reloads.
Generic rollback additionally checks all resources before finalizing the journal.
**Strict first-slice ceiling:** a DPI-broken disabled baseline can legitimately
fail these checks; restored bytes then remain `prepared`/BLOCKED until operator
repair. Physical restoration is not falsely reported as healthy rollback.
Legacy rollback has no new health requirement. Cancellation joins synchronous
checks before rollback; PID 1 bounds the owned session and retains prepared state
on unsafe/unfinished cleanup. `ActiveCandidates` refuses generic deployments;
managed/resource status validates generic refs without inventing legacy groups.
### Explicit startup selection (implemented; bounded positive field observations)
Set `ZAPRET_AUTOTUNE_RESOURCE_STRATEGY_IDS=s1,s2` with
`ZAPRET_AUTOTUNE_RESOURCE_CONFIG` and `ZAPRET_AUTOTUNE_RESOURCE_APPLY=1` on a
fresh container PID 1. Empty retains default greedy selection. The bounded list
accepts at most 32 unique registry IDs, canonicalized in lexical order; malformed,
duplicate, unknown or incomplete-coverage requests cannot authorize a transaction.
Eligibility is filtered **before** equal-args/selectors coverage merging; excluded
candidates cannot lend proof to a selected profile. Compatibility evidence and
all alternatives remain visible; the policy is not proof or a new qualification.
No registry/matrix files are written by rotation. Exact hosts, both TLS versions,
all required cells/M=3, native/source integrity and the 24-hour freshness gate
remain required for every new selection. Identical committed selection replays
without reload even after expiry. Distinct redundant allowlists are distinct
policies and may create a transaction even if rendered bytes coincide.
PID 1 forwards the setting in environment only; the apply child authenticates it
against PID 1's original environment, not CLI arguments. Syntax is validated
even with APPLY=0; nonempty policy requires generic resource config. With APPLY=0
selection is ignored for reconciliation/monitoring (registry membership is not
resolved): committed and pending refs reconstruct independently from their own
stored policies and historical completion horizon, never the requested setting.
Prepared recovery restores the old generic bytes **and policy**, never promotes
the pending side; failed rollback health retains prepared state for repair.
Startup still allows just one attempt per PID 1; external restart policies reset
that authorization. There is no monitor autotuning, new qualification, proof
import, snapshot overwrite or uninterrupted maintenance in this slice.
Local root fixture tests cover the production snapshot factory and shared engine;
independent local review and deterministic 180-second retry-window fixtures passed.
The earlier bounded baseline gates below cover guarded apply/replay, strict rollback,
prepared repair and the per-PID-1 failure latch; they do not establish rotation fault
acceptance or authorize live rollout.
### Verified-profile rotation observations — source `8d43dd9`
**Positive field observations only; partial native acceptance, not full rotation
safety acceptance or READY.** Saved-artifact privileged audits observed default A
generation 1 → selected multisplit B generation 2 → identical B replay generation 2
→ A return generation 3. No repeat positive sequence is required for the next gate.
- A config SHA-256 was `27f9a8ffd56a4c19d4bd522dd9f5a5e8b49b18b0ae48d41431ef9b22975ea772`;
B was `51faacf53e76fa0ce95fead1f02da3846c1706ecb9edc8ddaba095c089ea2c82`.
Replay retained transaction `09dfd03e9741abdf8e6b9f3bd44b2118`; A return used
`49a29d560f086a11f9de770913777e07` and restored the exact original A bytes.
Historical prepared snapshots held actual old-A backup before B and actual old-B
backup before A. Sampled old-committed → prepared → new-committed transitions
demonstrate these observations, not crash safety or failed-rotation rollback.
- Fourteen saved registry/matrix snapshots passed exact-payload HMAC and byte
invariance checks; source fingerprint was unchanged. The two original runs,
16 M=3 cells and 48 prior both-TLS qualification transfers were reused, about
12 hours old at startup, **not requalified**. Native A/B argument hashes differed;
A return matched original A, with native queue 300 socket-inode/PID linkage
observed. Early A-return process inventory is missing; full phase-path native
argv acceptance is not established.
- Six authenticated monitor snapshots each showed four HEALTHY resources and zero
failures. Saved fixture records showed 64 successful GETs, TLS 32/32, matching
SNI/Host and expected-header presence; A return contributed 32, TLS 16/16.
This proves fixture compatibility, **not causal DPI benefit** or continuously live
change. Original A-return official stop exited 0; fixture stop exited 2, not an
observed product failure in this run.
- Evidence base: `/home/agent/rotation-8d43dd9/field-private/` (root-private).
`report.safe.json` covers only generations 1/2/2, not final completion. Generation
3 is supported separately by `a-return-timeline.private.json`, retained state and
`final-passive-evidence/{before,stopped}/` snapshots. The initial harness ended
with KeyboardInterrupt; two continuation-construction bugs and observer filename
alias corrections are disclosed in `harness-path-alias.safe.json`. Corrections
affected host observer artifacts only, not proof inputs.
- Final saved cleanup verification is
`final-passive-evidence/final-cleanup-verification.safe.json`, timestamp
`1792051407108`: both exact owned stage containers and the owned network absent,
eight recorded PID paths absent, both known network namespaces without PIDs,
and three foreign full-inspect hashes unchanged. Owned containers/network were
explicitly removed with orchestrator approval; private state/evidence/keys were
retained. Foreign SS remained running and the two older containers unchanged.
Xray was not mutated; no fresh Xray inventory or exhaustive process audit is claimed.
- The first independent forensic audit passed the bounded 14-HMAC positive evidence,
but the automated grader rejected missing independently refreshed dated evidence
and exact native argv phase paths. Do not upgrade this to full native/safety
acceptance. Unknown-ID refusal and old-profile fault/repair observations were
subsequently recorded below, with narrower attribution than a branch-specific
refusal proof. Crash safety remains untested. Natural 24-hour expiry remains
unit-only, without a 24-hour wait.
- Prior local verification recorded 17 EUID-0 fixtures in one race/parallel pass
with 57 subrecords; namespace UID 0 mapped to host UID 100000, not host-initial
root. Full producer Go/race and four Python checks passed serially after parallel
resource-contention failures; this does not establish a contention-free baseline.
These are prior checks, not tests rerun for this documentation-only update.
### Subsequent old-generic rotation faults (bounded saved-artifact acceptance)
Actual remote source was `8d43dd9`, **not** shared-core `45d28e3`; these separate
results do not establish end-to-end acceptance of a combined release. Privileged
reads verified `/home/agent/rotation-fault-8d43dd9/field-private/ACCEPTANCE.safe.md`
and `standaloneprivilegedaudit.safe.json`, dated `1792054235151`
(2026-10-15 08:50:35.151 UTC), verdict
`PASS_BOUNDED_SAVED_ARTIFACTS_AND_CURRENT_CLEANUP`.
- All 13 snapshots passed exact-serialized-payload HMAC/immutable proof checks.
Old A is generic with empty/default policy, not baseline; B selects multisplit.
Historical old/new refs reconstruct independently.
- One-shot: eight B apply checks (one HTTP500), eight successful A rollback checks,
A committed/pending clear, no automatic reapply through 80s; official stop0.
- Persistent: 24 checks across apply/rollback/cleanup reconciliation, three
HTTP500; A bytes physically restored with PREPARED old-A/pending-B retained,
no reapply through 80s; stop78 retained the unsafe-cleanup latch. Healthy fixture
plus only APPLY=1→0 (requested B still set) gave eight reconcile + eight monitor
successes, old A committed/pending clear and four HEALTHY resources; stop0.
- Total: 56 authenticated requests, TLS28/28, four controlled HTTP500. At the
dated audit all 150 recorded owned PID/start identities were gone, recorded netns
had no processes, five owned containers and network were removed, and three
foreign raw-inspect hashes were unchanged. Root-private evidence was retained;
sampled identities are not an exhaustive all-helper inventory.
- Valid-syntax `unknown-safe-id` reached the actual CLI apply entry, which exited
78 and latched BLOCKED, with zero requests. Before/stopped config, journal,
backup and proof bytes were unchanged; no staging was sampled. This is an
**observed refusal with an unknown-ID request**, not uniquely attributed to the
unknown-ID branch: the generic diagnostic cannot exclude another trust error.
Source validation order supports an inference, not a specific logged reason
or complete pre-mutation proof. Official stop0 is separate from CLI exit78.
Startup remains one-shot per PID1, without automatic apply retry. These fixture
faults do not establish crash/cancellation safety, continuously live rotation,
causal DPI benefit or READY. Next migration work is the minimal typed generic
image/media capability above; owned managed v2 scoped observations are recorded
above, with full runtime acceptance still incomplete.
New-batch/versioned stores, uninterrupted rotation, strategy breadth and Prometheus
are separate future work, not prerequisites to documenting this collector slice.
## Bounded generic apply/replay and failure-latch acceptance
**DONE within the stated bounds**, not full generic autotuning or service READY.
Fault acceptance used source `643d0c6`; the earlier healthy committed replay used
`36aace0` after the atomic launch-ACK correction. Independent privileged audit
returned `PASS_BOUNDED_GENERIC_APPLY_FAULTS` for the observed A/B/C safety gates.
Root-private evidence references (no private payloads reproduced here):
- `/home/agent/generic-latch-643d0c6/field-private/field-report.md`
- `/home/agent/generic-latch-643d0c6/field-private/independent-audit.safe.json`
- `/home/agent/ack-recreate-36aace0/field-private/field-report.md`
### Observed acceptance
- Real apply health reused authentic, still-fresh qualification: 16 M=3 cells,
48 prior TLS transfers, two strategies and four resources. No new qualification,
timestamp rewrite, fake ACK or imported proof was used. Pinned native 1.0.2/Lua
stayed unchanged. The earlier healthy start/recreate each made eight TLS checks
with four authenticated HEALTHY resources; v4 committed generation 1, refs,
config and proof bytes remained unchanged. Both official stops exited 0.
- **A — one HTTP 500:** eight apply plus eight rollback requests, one 500 total.
Rollback committed prior baseline generation 1 and cleared pending. Request
count, journal and config stayed frozen at 45.096/90.084/135.124 seconds;
one apply only, no automatic retry. PID 1 stayed alive/BLOCKED with SS/DNS and
other owned runtime stopped. Official stop exited 0.
- **B — persistent HTTP 500:** 24 requests and three 500s: eight apply, eight
rollback and eight cleanup-reconcile checks, not a second apply. Prior baseline
bytes were physically restored, but failed health retained PREPARED committed
baseline/pending generic refs. State stayed unchanged for 80.097 seconds with
no reapply; official stop exited 78. After repairing only fixture behavior and
changing startup APPLY=1 to APPLY=0, the same state/source/environment completed
eight reconcile and eight monitor checks. Baseline became committed, pending
cleared, and generic was never promoted; official stop exited 0.
- **C — TERM during a real slow request:** one canceled request plus eight passing
rollback checks. The canceled curl was joined before the first rollback request;
baseline generation 1 committed and pending cleared, with no false generic
commit. Actual PID 1 exit was **78**, with `monitor cleanup status 78`, not 0.
Apply health cancellation is collapsed into `errPlan`/CLI 78; outside the
qualification-specific completion path, the supervisor accepts only 0/143 and
retains a sticky failure even after verified rollback. This is conservative
**outcome conflation**, not an observed rollback-health failure or cleanup
defect. Do not blanket-allow 78 or manufacture a completion ACK. A normal-0
path would require authenticated completion plus fresh guards; that optional
future refinement is not a blocker for this bounded P0 safety slice.
### Bounds and remaining work
- A/B/repair/C totalled 65 actual HTTPS requests. All 171 observed owned PID/start
identities disappeared, owned netns were empty, and owned targets/network were
removed. Foreign SS/container full-inspect hashes stayed unchanged. Passive
100ms inventories do not cover every shorter-lived helper; no exhaustive argv
or packet-level privacy audit is claimed. Same-root HMAC-key mutation is outside
the trust boundary.
- The latch is one RAM authorization per PID 1, **not durable manual intent**.
Fresh startup/external restart policy reauthorizes ENV=1; keep unattended
external restart/recreation disabled. ENV=0 repairs/reconciles only. Historical
committed replay/repair allows proof expiry; a new apply still requires fresh,
complete source-bound qualification within 24 hours. Natural remote 24-hour
expiry remains unit-covered only; remote qualifier singleton conflict is still
untested.
- Use an execution-free remote harness: stage inputs before startup; no target
`exec`, `cp` or `nsenter` during active guards. Observe host metadata/private
files and Docker logs/inspect; official stop is the cancellation operation.
Earlier scan EACCES nonrecurrence does not establish its cause.
- No production READY, causal DPI benefit, active-profile maintenance, plan or
snapshot rotation, automatic recovery or monitor autotuning is accepted here.
Next roadmap work is the missing rotation fault/repair gate above, then dynamic
profile/plan rotation and compatible selectors,
generic resource tuning and bounded strategy-DB collections; bounded-ID
Prometheus remains a separate planned function. No new framework, probes or
native/live upgrade was introduced. Legacy YouTube exit 78 and unfinished
images/avatars remain NOT READY.
## Bounded remote acceptance — source `5cdc0cd`
The isolated producer/registry/matrix/planner slice is **bounded-qualified**.
Independent privileged evidence audits passed the observed positive, fault and
active-state refusal gates. **Full remote concurrency acceptance is PARTIAL**;
this is not generic apply, active-profile maintenance or service READY.
Root-private proof references (retained reports, not logs copied into this doc):
- `/home/agent/zapret2-autotune-matrix-5cdc0cd/execution-free-private/execution-free-report.md`
- `/home/agent/zapret2-autotune-matrix-5cdc0cd/fault-final-private/qualification-fault-report.md`
- `/home/agent/zapret2-autotune-matrix-5cdc0cd/refusal-gap-private/remaining-refusal-reports.md`
### Observed positive and fault gates
- The execution-free private batch made 60 actual transfers: two candidates,
five resources, both TLS protocols, three attempts per cell. Its 20 aggregated
cells contain 16 PASS and four `expectation_failed`: mandatory resource d's
HTTP 500 fails for both candidates/protocols, leaving d unassigned. Exact
hosts a/b/c receive the greedy multidisorder proposal with authentic multisplit
alternatives. Input groups remain metadata, not assignment boundaries;
there is no generic suffix coverage or automatic apply.
- Independent verification authenticated the exact stored payload HMAC, not a
reserialized JSON equivalent. Distinct native arguments/worker-argv hashes,
executable/Lua/rules evidence, queue 200 and NFQ 1.0.2 attestations bind the
two joined/restored runs. Disabled baseline bytes stayed unchanged; sampled
workers, queue and temporary NFT rules were gone. Official PID 1 shutdown
exited 0 in 1.31s; foreign inspect hashes stayed unchanged.
- Mid-batch cancellation and effective semantic config drift each stopped
after two requests, published no matrix and preserved the native baseline.
Official PID 1 exits were 0 in 2.46s and 0.44s respectively; the deliberately
changed config source was retained, not silently rolled back.
- Unsafe Lua/queue/fwmark/user/file definitions and evidence/schema imports
were rejected without transfers. Oversized capacity was rejected before
worker launch; unauthorized isolated startup was refused. Corrupt MAC blocked
planning while preserving payload bytes; changed effective config produced
unknown/unassigned output without changing the original authenticated matrix.
Immutable catalogue replacement and completed-matrix retest were refused
without transfers or overwriting original state.
- A genuine archived v3 committed journal with two candidates passed official
`reconcile`/`ActiveCandidates` validation before qualification refused active
profile state. Profile/config, journal, last-good and target identity remained
byte-identical; zero transfers/workers and no matrix. Actual PID 1 exit 78 was
an expected early preflight refusal, **not** an exit-0 cleanup ACK. This did
not activate or maintain a running active-profile NFQ daemon; that remains
unsupported. Whole-DB byte preservation is not claimed.
### Explicit limits and testing lesson
- Remote qualifier singleton conflict is **NOT RUN**. The earlier planted-lock
fixture failed initialization before that branch and does not count. The
shared-lock unit second-acquire `EWOULDBLOCK` check passed, but is not remote
qualifier concurrency acceptance. No deterministic official lifecycle seam
exists for this bounded test; no test framework or racy injection was added.
- Natural 24-hour remote expiry is **NOT RUN**. The existing unit check rejects
a structurally valid 25-hour-old proof; this is not observed remote expiry.
- Privacy sentinel absence was checked in tested logs/state and sampled fault/
refusal argv. Positive active argv and exhaustive DNS packet-level privacy
were not proved. Configured/observed encrypted DNS is not exhaustive proof
against transient fallback.
- Compatibility on this baseline is not causal DPI-desync benefit. Historical
worker-scan EACCES origin remains unknown; nonrecurrence in the execution-free
batch does not identify its cause. Earlier harness failures remain recorded
in the referenced reports, not erased or relabeled as product acceptance.
- Stage all inputs before target startup. From startup through qualification
completion, prohibit target `exec`, `cp` and `nsenter` (even passive exec starts
a process). Observe host `/proc` metadata and Docker logs/inspect only; inspect
runtime and run `plan` after finalization/cleanup. The explicit official stop
used for cancellation is the documented exception, not an observer exec.
- Owned test containers/network were removed and private evidence retained.
Foreign SS1080 stayed running and legacy11082 stayed stopped with exit 78.
No live upgrade or push occurred; unresolved legacy exit 78, YouTube/image
readiness, transactional generic apply/profile-journal migration, immutable
rotation and noninterrupting search remain outside this producer acceptance;
the later bounded apply/replay gates are recorded separately above.
## Internal YouTube adapter slice (local; field acceptance pending)
`internal/adapter` is a small closed compiled-in contract (`Binding`, ephemeral
`Resource`/`Plan`, `Module`); `internal/adapter/youtube` implements `youtube/1`.
The explicit kind/version registry in `service_adapters.go` has no init hooks,
dynamic loading, public configuration, HTTP fallback or module privileges.
Player media parsing (`playerMediaResource`) and owned native qualification
(`verifyGooglevideo`) actually call it. Discovery retry/rotation/cancellation and
representative selection remain in the legacy controller; signed MP4 freshness
and the host/TLS qualification plan are pure service semantics in the module.
The next private execution bridge is now wired: `verifyGooglevideo` runs the
actual owned-worker collector through `executeAdapterPlan[T]`. Core independently
pins the known module binding and exact three-host/TLS1.2+1.3, 10-attempt/120-second
warmup, three steady samples/protocol and 1024-byte policy before worker startup
and probes. Invalid, duplicate, overflowing or weakened plans fail closed. The
executor preserves host order, alternating warmups, discarded learning successes,
strict measured steady facts and negative/fatal classification. It has no worker,
goroutine, persistence or seal capability; existing core range/ftyp/TLS validation
and post-worker-join proof validation/sealing remain the authority.
This is a **private execution bridge**, not public adapter configuration or a new
generic service/matrix proof. Dynamic-inventory fingerprints and warmup-bearing
generic proof binding are still required before generic-controller cutover.
The inherited legacy SOCKS5h boundary below is unchanged, not an all-DNS-answer
pinning guarantee. Three-CDN qualification does not itself grant suffix scope;
images/avatars and runtime acceptance remain separate and not READY.
Player JSON now travels through bounded private stdin (`curl --json @-`), not
process argv; the shared runner passes no stdin for metadata GETs. Video IDs are
validated before execution, POST bytes and headers remain unchanged, and opaque
public errors retain typed cancellation/unsafe-output causes. This is a private
Player transport change, not generic public POST/discovery configuration.
Bridge-local checks: an unchanged fixture run against archived `f485317` and the
wired producer has normalized proof SHA256
`fb380324e320a78f1ceb011c59b82b9de4ff4f31f5f032f469a181292987e3b2`.
Only warmup duration, attestation time and private seals are excluded; exact
probe order and cleanup/join are checked. Baseline and focused serial/race logs:
`/tmp/opencode/adapter-executor-{baseline,focused}.log`. These are injected local
checks, not network/native acceptance or a full-suite rerun.
Cancellation review follow-up: both warmup and steady probe boundaries now use
`managedRecoveryContextError`, preserving concurrent unsafe I/O/output overflow
alongside cancellation/deadline causes. Production collector regressions check
fatal/non-restartable classification, full cleanup/join and no attestation/seal;
plain cancellation remains timeout-safe. A pre-fix Go-overlay negative control
fails all eight mixed-failure cases; the fixed producer retains the hash above.
Actual `sudo` EUID-0 serial full Go checks passed (1806 test/subtest pass events),
as did 227 affected serial/race events, separate module race tests, vet and a
CGO-disabled build. Logs: `/tmp/opencode/adapter-executor-root-full.json`,
`adapter-executor-context-root-race.json`, `adapter-executor-context-focused.log`
and `adapter-executor-context-negative-control.log` in that same directory.
The reviewer-reported earlier `SavedFirstAndFailClosed/nonpass` freshness failure
did not recur in ten focused repeats or this full run; no freshness guards or
fixtures were changed, and its earlier cause is not established as a baseline
flake. Python checks were not rerun in this follow-up. These are local tests,
not remote runtime acceptance, a commit/deployment or a READY claim.
Core origin/TLS validation, shared image/media transfer code, bounded worker
lifecycle and post-cleanup/join sealing remain authoritative. Legacy media and
metadata routing retains SOCKS5h with the owned encrypted-resolver guard and
exact origin/SNI/certificate/redirect checks. It does **not** use public generic
`runResourceProtocol`'s all-A/AAAA approval and connect-to IP pinning. That inherited
ceiling is unchanged, not a pinning guarantee: full request-policy capabilities
(including future metadata POST) and parity are prerequisites to generic cutover.
No new DNS/IP routing policy is introduced by this extraction. Both TLS
versions must learn separately, then each has three measured MP4 samples per
representative host, minimum 1024 bytes (not an exact 64 KiB requirement).
Measured endpoints do not authorize a suffix: existing exact/suffix/experimental
policy and independent thumbnail/avatar qualification are unchanged. The module
cannot manufacture evidence, request wildcard apply or claim YouTube READY.
New adaptive proof **v2** includes adapter kind/version/immutable policy SHA256
inside the producer seal. Public JSON cannot recreate that seal. Missing or
mismatched binding/version/digest fails closed; v1 with v2 fields is rejected.
Historical proof **v1** explicitly uses the pinned legacy YouTube v1 contract,
including historical freshness-independent journal reconstruction. Its config
bytes/hash and journal refs remain stable. Fresh stored trusted v1 proofs retain
their original apply/qualification eligibility, including unchanged attestation
freshness and media gates. Expired v1 can reconstruct but cannot authorize fresh
apply. Only **new production/persistence** requires sealed, module-bound v2;
neither sealed-new-v1 nor public unsealed v1/v2 DTOs can publish new evidence.
Cold v1 evidence, SQL schema 7, native arguments,
public GET/HEAD v1/v2, state keys and HMAC formats are unchanged.
This slice is not full generic-controller migration, native-engine runtime
acceptance, a security sandbox (including against same-UID actors), or a fix for
unresolved avatars/legacy exit 78. No legacy path is deleted. Local tests and
archived baseline parity are recorded separately; no remote validation or commit
is part of this extraction review.
Discovery preflight resolves the required module identity and policy digest
before any Player/WWW/avatar/front/media requests, including injected callbacks,
legacy proposal discovery and managed discovery hooks. Missing/mismatched modules
produce a fixed private sentinel wrapped as a fatal configuration failure, not
retryable unavailable inventory, a transfer-negative or genuine no-strategy pause.
The compiled production registry remains fixed; synchronized test replacement
does not add public runtime module configuration or loading.
### Local verification for this extraction (2026-10-15)
- Fresh serial `go test -json -p 1 ./... -parallel 1 -count=1 -timeout 600s`:
399 top-level / 1737 including subtest PASS events, zero failures.
- Affected serial race run (`-race -p 1 -parallel 1 -count=1`) covers adapters,
Player/discovery, owned qualification/proof gates, historical journal parity,
independent assets and generic resource core: 61 top-level / 307 including
subtest PASS events, zero failures. Full regex and logs are retained at
`/tmp/opencode/youtube-adapter-fixes/{full,race}.json`.
- Archived `d324b4137b2e95f1c5f4d6c8240470e3f8749f0c` and current builds each
passed 275 affected test/subtest events. The portable legacy parser/journal
corpus ran **unmodified** on both: file SHA256
`7b8eea9c9bbcdd8f0fadde3f78596d41e9d032a0a4f7b2ca90d1223d106584d5`.
Both reconstructed historical config SHA256
`172c7377388334e4baa2dc1009270478dc271a7b2f00d28cfada631f44a35364`,
retaining journal bytes/refs after reconcile and ActiveCandidates validation.
An additional unchanged portable v1 corpus passed 2 top-level / 8 including
subtest events on **each** build: fresh stored v1 shortlist/apply eligibility,
expired fresh-apply refusal, and staged/backup/prepared/replaced/reloaded/healthy
crash rollback with expired historical proofs. Backup/config bytes and hashes
are stable; pre-prepare journal bytes are unchanged and post-prepare generation/
reload/ref transitions follow the original contract. Corpus SHA256:
`6df53b12af5d80d164ea3201a614a5fd5723561dc7b2d6bc2d2aa37f845f053d`.
New negative tests separately reject even internally sealed NEW v1 writes and
public unsealed DTOs; missing/changed module identity/policy makes zero Player,
WWW/avatar/media/front/worker calls and does not become a two-hour absence.
- All four `python3 deploy/managed-target/check_{static,config,startup_diagnostics,supervision}.py`
checks passed serially; supervision was bounded with `timeout 600s`, no retry.
`go vet ./...`, `CGO_ENABLED=0 go build`, `sh -n` on all three entrypoint/
supervision scripts, and `git diff --check` passed. Build SHA256:
`8327faf86098145bb41306b3de895cdd622aeccdcda90f8ec73e19a759c8f98a`.
- Ownership-sensitive final runs used `sudo -n` EUID 0 **inside the existing
user namespace**, mapping UID 0 to parent/host UID 100000, not initial-host
root. The initial extra single-UID `unshare -Ur` run failed ownership fixtures
(`chown: invalid argument` / `invalid private qualification data`); a stale
test assertion also expected newly produced proof v1 and was updated to v2.
Those prior extraction failures are retained in
`/tmp/opencode/youtube-adapter-review/go-full.json`, not reported as passes. Vet's
initial unkeyed shared-DTO literals were fixed before final successful checks.
Reproduction commands (run serially; root is only for offline ownership fixtures):
```sh
pattern='Test(YoutubeLegacy|ManagedGooglevideo|ManagedAdaptive|ExperimentalPromotion|Googlevideo|Resource(Core|Discovery|Contract)|Player|Discover|ManagedAsset|Asset)'
sudo -n env GOPROXY=off GOTOOLCHAIN=local GOMODCACHE="$(go env GOMODCACHE)" "$(command -v go)" test -json -p 1 ./... -parallel 1 -count=1 -timeout 600s
sudo -n env GOPROXY=off GOTOOLCHAIN=local GOMODCACHE="$(go env GOMODCACHE)" "$(command -v go)" test -race -json -p 1 ./internal/adapter/... ./cmd/zapret-autotune -parallel 1 -run "${pattern}|Test(ServiceAdapter|LegacyYoutubeV1|QualificationPlan|SignedSample|DiscoverResponse)" -count=1 -timeout 600s
# Baseline/current parity: the same $pattern, without -race, on both trees.
# Baseline tree: git archive d324b4137b2e95f1c5f4d6c8240470e3f8749f0c;
# add identical youtube_legacy_parity_test.go and youtube_legacy_apply_parity_test.go.
# The additional portable v1 corpus runs separately with -run '^TestLegacyYoutubeV1'.
```
+125
View File
@@ -0,0 +1,125 @@
package main
import (
"context"
"errors"
"time"
"zapret-autotune/internal/adapter"
"zapret-autotune/internal/adapter/youtube"
)
var errAdapterPlan = errors.New("invalid private adapter execution plan")
// This is core policy, not module-supplied caps. Adding a binding requires an
// audited execution policy here; even a correctly identified module is untrusted.
func validateAdapterPlan(binding adapter.Binding, plan adapter.Plan, resources map[string]mediaResource) error {
if binding != (youtube.Module{}).Binding() || len(plan.Hosts) != 3 ||
plan.Protocols != [2]string{"tls12", "tls13"} || plan.Samples != 3 ||
plan.MinBytes != 1024 || plan.WarmupLimit != 10 || plan.WarmupTimeout != 120*time.Second {
return errAdapterPlan
}
if (youtube.Module{}).ValidateHosts(plan.Hosts) != nil {
return errAdapterPlan
}
for _, host := range plan.Hosts {
resource, ok := resources[host]
if !ok || !validDomain(host) || resource.host != host {
return errAdapterPlan
}
if _, err := privateResourceURL(resource.url, host); err != nil {
return errAdapterPlan
}
if !youtube.SignedSampleFresh(youtube.Resource{URL: resource.url, Host: host, Kind: resource.kind}, time.Now(), 120*time.Second) {
return errAdapterPlan
}
}
return nil
}
// Typed measurements stay private and ephemeral. Only core callbacks collect
// and classify them; modules cannot supply callbacks, attestations or seals.
type adapterMeasurement[T any] struct {
value T
bytes int64
}
type adapterHostMeasurements[T any] struct {
host string
warmupAttempts, warmupFailures int
warmupLimit int
warmupMS float64
samples []T
}
// executeAdapterPlan owns scheduling only. Its caller owns the worker, transport
// validation and post-join proof validation/sealing. No goroutines or persistence.
func executeAdapterPlan[T any](ctx context.Context, binding adapter.Binding, plan adapter.Plan, resources map[string]mediaResource,
probe func(context.Context, mediaResource, string) (adapterMeasurement[T], error), nonpass func(error) bool,
) ([]adapterHostMeasurements[T], error) {
if err := validateAdapterPlan(binding, plan, resources); err != nil {
return nil, err
}
if err := ctx.Err(); err != nil {
return nil, err
}
hosts := make([]adapterHostMeasurements[T], len(plan.Hosts))
for i, host := range plan.Hosts {
hosts[i].host = host
}
for i := range hosts {
h := &hosts[i]
h.warmupLimit = plan.WarmupLimit
start := time.Now()
warmCtx, cancel := context.WithTimeout(ctx, plan.WarmupTimeout)
learned := map[string]bool{}
for h.warmupAttempts < h.warmupLimit && len(learned) < len(plan.Protocols) && warmCtx.Err() == nil {
protocol := plan.Protocols[h.warmupAttempts%len(plan.Protocols)]
h.warmupAttempts++
measurement, err := probe(warmCtx, resources[h.host], protocol)
err = managedRecoveryContextError(warmCtx, err)
if err != nil {
h.warmupFailures++
if !nonpass(err) && (warmCtx.Err() == nil || !managedRecoveryTimeoutSafe(err)) {
cancel()
return hosts, err
}
} else if measurement.bytes < plan.MinBytes {
h.warmupFailures++
} else {
learned[protocol] = true
}
}
h.warmupMS = float64(time.Since(start).Microseconds()) / 1000
cancel()
if err := ctx.Err(); err != nil {
return hosts, err
}
if len(learned) != len(plan.Protocols) || h.warmupMS > float64(plan.WarmupTimeout/time.Millisecond) {
return hosts, errCandidateNonpass
}
for _, protocol := range plan.Protocols {
for n := 0; n < plan.Samples; n++ {
if err := ctx.Err(); err != nil {
return hosts, err
}
measurement, err := probe(ctx, resources[h.host], protocol)
err = managedRecoveryContextError(ctx, err)
if ctx.Err() != nil {
return hosts, err
}
if err != nil {
if nonpass(err) {
return hosts, errCandidateNonpass
}
return hosts, err
}
if measurement.bytes < plan.MinBytes {
return hosts, errCandidateNonpass
}
h.samples = append(h.samples, measurement.value)
}
}
}
return hosts, nil
}
@@ -0,0 +1,148 @@
package main
import (
"context"
"crypto/sha256"
"encoding/json"
"errors"
"fmt"
"io"
"reflect"
"strings"
"testing"
"time"
)
// Also runnable unchanged against archived f485317. Only wall-clock duration,
// attestation time and process-private seals are excluded from the golden hash.
func TestAdapterExecutorLegacyParity(t *testing.T) {
r, c, resources, events := googlevideoRuntimeFixture(t)
runner := r.curl
var order []string
r.curl = func(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
protocol := "tls12"
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
protocol = "tls13"
}
order = append(order, protocol)
if len(order) <= 2 {
return context.DeadlineExceeded
}
return runner(ctx, args, stdin, out, diagnostic)
}
p, err := r.verifyGooglevideo(context.Background(), c, resources)
if err != nil || p.producerSeal == nil {
t.Fatal(err, p)
}
if !strings.HasSuffix(strings.Join(*events, ","), "delete,join") {
t.Fatal(*events)
}
expected := []string{"tls12", "tls13"}
for range c.Endpoints {
expected = append(expected, "tls12", "tls13", "tls12", "tls12", "tls12", "tls13", "tls13", "tls13")
}
if !reflect.DeepEqual(order, expected) {
t.Fatal(order, expected)
}
for i := range p.Hosts {
p.Hosts[i].WarmupMS = 0
}
p.Attestation.CheckedAt = time.Unix(1700000000, 0).UTC()
raw, err := json.Marshal(p)
if err != nil {
t.Fatal(err)
}
hash := fmt.Sprintf("%x", sha256.Sum256(raw))
t.Log("normalized proof sha256:", hash)
const baseline = "fb380324e320a78f1ceb011c59b82b9de4ff4f31f5f032f469a181292987e3b2"
if hash != baseline {
t.Fatalf("proof parity changed: %s\n%s", hash, raw)
}
}
func TestAdapterExecutorProductionCancellationJoinsWithoutSeal(t *testing.T) {
for _, at := range []int{1, 3} {
t.Run(fmt.Sprint(at), func(t *testing.T) {
r, c, resources, events := googlevideoRuntimeFixture(t)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
runner, calls := r.curl, 0
r.curl = func(probeCtx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
calls++
if calls == at {
cancel()
return ctx.Err()
}
return runner(probeCtx, args, stdin, out, diagnostic)
}
p, err := r.verifyGooglevideo(ctx, c, resources)
if !errors.Is(err, context.Canceled) || calls != at || p.Attestation != nil || p.producerSeal != nil ||
!strings.HasSuffix(strings.Join(*events, ","), "delete,join") {
t.Fatal(p, err, calls, *events)
}
if !managedRecoveryTimeoutSafe(err) {
t.Fatal("plain cancellation became unsafe", err)
}
})
}
}
func TestAdapterExecutorProductionCancellationPreservesUnsafeTransfer(t *testing.T) {
unsafeIO := errors.New("unsafe collector IO")
for _, at := range []int{1, 3} { // first warmup or first steady after both TLS versions learn
for _, deadline := range []bool{false, true} {
for _, overflow := range []bool{false, true} {
t.Run(fmt.Sprintf("probe%d/deadline%t/overflow%t", at, deadline, overflow), func(t *testing.T) {
r, c, resources, events := googlevideoRuntimeFixture(t)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
wantContext := context.Canceled
if deadline {
var stop context.CancelFunc
ctx, stop = context.WithTimeout(ctx, time.Second)
defer stop()
wantContext = context.DeadlineExceeded
}
runner, calls := r.curl, 0
r.curl = func(probeCtx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
calls++
if calls != at {
return runner(probeCtx, args, stdin, out, diagnostic)
}
if deadline {
<-ctx.Done()
} else {
cancel()
}
if overflow {
// Exercise the real bounded collector writer, not a forged fatal DTO.
_, writeErr := out.Write(make([]byte, 256<<10))
return errors.Join(writeErr, ctx.Err())
}
return errors.Join(unsafeIO, ctx.Err())
}
p, err := r.verifyGooglevideo(ctx, c, resources)
var fatal *managedRecoveryFatal
if !errors.Is(err, wantContext) || !errors.As(err, &fatal) || managedRecoveryTimeoutSafe(err) || managedResourceTransferNonpass(err, false, false) {
t.Fatalf("lost mixed safety/context error: %v", err)
}
if overflow {
var collected *resourceError
if !errors.As(err, &collected) || collected.reason != reasonOutputBound {
t.Fatal("lost collector overflow", err)
}
} else if !errors.Is(err, unsafeIO) {
t.Fatal("lost unsafe IO cause", err)
}
if calls != at || p.Attestation != nil || p.producerSeal != nil ||
!strings.HasSuffix(strings.Join(*events, ","), "delete,join") {
t.Fatal(p, err, calls, *events)
}
if at == 3 && (p.Hosts[0].WarmupAttempts != 2 || len(p.Hosts[0].Passes) != 0) {
t.Fatal("learning leaked into steady proof", p.Hosts)
}
})
}
}
}
}
@@ -0,0 +1,248 @@
package main
import (
"context"
"errors"
"math"
"reflect"
"testing"
"time"
"zapret-autotune/internal/adapter"
"zapret-autotune/internal/adapter/youtube"
)
func adapterExecutorFixture(t *testing.T) (adapter.Binding, adapter.Plan, map[string]mediaResource) {
t.Helper()
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
resources := map[string]mediaResource{}
var samples []youtube.Resource
for _, host := range hosts {
url := "https://" + host + "/videoplayback?expire=4102444800&sig=private"
resources[host] = mediaResource{host: host, url: url, kind: "video/mp4"}
samples = append(samples, youtube.Resource{Host: host, URL: url, Kind: "video/mp4"})
}
m := youtube.Module{}
p, err := m.PlanQualification(hosts, samples, time.Now())
if err != nil {
t.Fatal(err)
}
return m.Binding(), p, resources
}
func TestAdapterExecutorRejectsUntrustedPlanBeforeProbe(t *testing.T) {
mutations := map[string]func(*adapter.Binding, *adapter.Plan, map[string]mediaResource){
"binding-kind": func(b *adapter.Binding, _ *adapter.Plan, _ map[string]mediaResource) { b.Kind = "other" },
"binding-version": func(b *adapter.Binding, _ *adapter.Plan, _ map[string]mediaResource) { b.Version++ },
"binding-policy": func(b *adapter.Binding, _ *adapter.Plan, _ map[string]mediaResource) { b.PolicySHA256 = "" },
"missing-host": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.Hosts = p.Hosts[:2] },
"extra-host": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) {
p.Hosts = append(p.Hosts, "rr9---sn-delta.googlevideo.com")
},
"duplicate-host": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.Hosts[1] = p.Hosts[0] },
"host-url": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) {
p.Hosts[0] = "https://example.com"
},
"host-case": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) {
p.Hosts[0] = "RR1---sn-alpha.googlevideo.com"
},
"host-suffix": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) {
p.Hosts[0] = "rr1---sn-alpha.example.com"
},
"missing-resource": func(_ *adapter.Binding, p *adapter.Plan, r map[string]mediaResource) { delete(r, p.Hosts[0]) },
"resource-host": func(_ *adapter.Binding, p *adapter.Plan, r map[string]mediaResource) {
v := r[p.Hosts[0]]
v.host = p.Hosts[1]
r[p.Hosts[0]] = v
},
"resource-url": func(_ *adapter.Binding, p *adapter.Plan, r map[string]mediaResource) {
v := r[p.Hosts[0]]
v.url = "https://evil.example/"
r[p.Hosts[0]] = v
},
"missing-protocol": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.Protocols[1] = "" },
"duplicate-protocol": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.Protocols[1] = p.Protocols[0] },
"unknown-protocol": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.Protocols[1] = "http" },
"protocol-order": func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) {
p.Protocols = [2]string{"tls13", "tls12"}
},
}
for _, n := range []int{math.MinInt, -1, 0, 1, 4, math.MaxInt} {
value := n
mutations["samples-"+time.Duration(n).String()] = func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.Samples = value }
}
for _, n := range []int{-1, 0, 1, 11, math.MaxInt} {
value := n
mutations["warmup-"+time.Duration(n).String()] = func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.WarmupLimit = value }
}
for _, n := range []int64{math.MinInt64, -1, 0, 1, 1025, math.MaxInt64} {
value := n
mutations["bytes-"+time.Duration(n).String()] = func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.MinBytes = value }
}
for _, n := range []time.Duration{-1, 0, time.Second, 121 * time.Second, time.Duration(math.MaxInt64)} {
value := n
mutations["timeout-"+n.String()] = func(_ *adapter.Binding, p *adapter.Plan, _ map[string]mediaResource) { p.WarmupTimeout = value }
}
for name, mutate := range mutations {
t.Run(name, func(t *testing.T) {
b, p, resources := adapterExecutorFixture(t)
mutate(&b, &p, resources)
_, err := executeAdapterPlan(context.Background(), b, p, resources,
func(context.Context, mediaResource, string) (adapterMeasurement[int], error) {
t.Fatal("invalid plan probed")
return adapterMeasurement[int]{}, nil
},
func(error) bool { t.Fatal("invalid plan classified"); return false })
if !errors.Is(err, errAdapterPlan) {
t.Fatal(err)
}
})
}
}
func TestAdapterExecutorOrderedWarmupAndTypedSamples(t *testing.T) {
b, p, resources := adapterExecutorFixture(t)
var order []string
results, err := executeAdapterPlan(context.Background(), b, p, resources,
func(ctx context.Context, r mediaResource, protocol string) (adapterMeasurement[int], error) {
if _, ok := ctx.Deadline(); len(order)%8 < 2 && !ok {
t.Fatal("warmup lacks deadline")
}
order = append(order, r.host+"/"+protocol)
return adapterMeasurement[int]{value: len(order), bytes: 1024}, nil
}, func(error) bool { return false })
if err != nil {
t.Fatal(err)
}
var expected []string
for i, host := range p.Hosts {
for _, protocol := range []string{"tls12", "tls13", "tls12", "tls12", "tls12", "tls13", "tls13", "tls13"} {
expected = append(expected, host+"/"+protocol)
}
h := results[i]
base := i * 8
if h.host != host || h.warmupAttempts != 2 || h.warmupFailures != 0 || h.warmupLimit != 10 || !reflect.DeepEqual(h.samples, []int{base + 3, base + 4, base + 5, base + 6, base + 7, base + 8}) {
t.Fatal(h)
}
}
if !reflect.DeepEqual(order, expected) {
t.Fatal(order, expected)
}
}
func TestAdapterExecutorFailureAndCancellation(t *testing.T) {
fatal := errors.New("infrastructure failure")
negative := errors.New("known negative")
for _, mode := range []string{"warm-negative", "warm-small", "warm-fatal", "warm-mixed", "steady-negative", "steady-small", "steady-fatal", "cancel-before", "cancel-warm", "deadline-warm", "cancel-steady"} {
t.Run(mode, func(t *testing.T) {
b, p, resources := adapterExecutorFixture(t)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
want := errCandidateNonpass
wantCalls := 10
if mode == "cancel-before" {
cancel()
want = context.Canceled
wantCalls = 0
}
if mode == "warm-fatal" {
want = fatal
wantCalls = 1
}
if mode == "warm-mixed" {
want = fatal
wantCalls = 1
}
if mode == "steady-negative" || mode == "steady-small" {
wantCalls = 3
}
if mode == "steady-fatal" {
want = fatal
wantCalls = 3
}
if mode == "cancel-warm" {
want = context.Canceled
wantCalls = 1
}
if mode == "cancel-steady" {
want = context.Canceled
wantCalls = 3
}
if mode == "deadline-warm" {
var stop context.CancelFunc
ctx, stop = context.WithTimeout(ctx, time.Second)
defer stop()
want = context.DeadlineExceeded
wantCalls = 1
}
calls := 0
hosts, err := executeAdapterPlan(ctx, b, p, resources,
func(probeCtx context.Context, _ mediaResource, _ string) (adapterMeasurement[int], error) {
calls++
m := adapterMeasurement[int]{value: calls, bytes: 1024}
switch mode {
case "warm-negative":
return m, negative
case "warm-small":
m.bytes = 1023
case "warm-fatal":
return m, fatal
case "warm-mixed":
return m, errors.Join(negative, fatal)
case "steady-negative":
if calls == 3 {
return m, negative
}
case "steady-small":
if calls == 3 {
m.bytes = 0
}
case "steady-fatal":
if calls == 3 {
return m, fatal
}
case "cancel-warm":
cancel()
return m, ctx.Err()
case "deadline-warm":
<-probeCtx.Done()
return m, probeCtx.Err()
case "cancel-steady":
if calls == 3 {
cancel()
return m, ctx.Err()
}
}
return m, nil
}, func(err error) bool { return err == negative })
if !errors.Is(err, want) || calls != wantCalls {
t.Fatal(err, calls, want, wantCalls)
}
for _, host := range hosts {
if len(host.samples) != 0 {
t.Fatal("phantom steady sample", host)
}
}
if calls == 10 && (hosts[0].warmupAttempts != 10 || hosts[0].warmupFailures != 10) {
t.Fatal(hosts)
}
})
}
}
type weakenedExecutorAdapter struct{ youtube.Module }
func (m weakenedExecutorAdapter) PlanQualification(hosts []string, resources []adapter.Resource, now time.Time) (adapter.Plan, error) {
p, err := m.Module.PlanQualification(hosts, resources, now)
p.Samples = 1
return p, err
}
func TestAdapterExecutorProductionPreflightRejectsWeakenedModule(t *testing.T) {
replaceServiceAdapters(t, serviceAdapterRegistry{{youtube.Kind, youtube.Version}: weakenedExecutorAdapter{}})
r, c, resources, events := googlevideoRuntimeFixture(t)
p, err := r.verifyGooglevideo(context.Background(), c, resources)
if !errors.Is(err, errAdapterPlan) || len(*events) != 0 || p.Attestation != nil || p.producerSeal != nil {
t.Fatal("module weakened production execution policy", p, err, *events)
}
}
+247
View File
@@ -0,0 +1,247 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"database/sql"
"encoding/json"
"fmt"
"image"
"image/gif"
_ "image/jpeg"
_ "image/png"
"io"
"math"
"sort"
"strings"
"time"
)
// Rebuild only the two bounded candidate tables, inside migrate's transaction.
// Refuse unexpected objects rather than silently discarding custom constraints.
func migrateAssetCandidates(ctx context.Context, conn *sql.Conn) error {
var actual []string
rows, err := conn.QueryContext(ctx, `SELECT sql FROM sqlite_master WHERE tbl_name IN ('strategy_candidates', 'candidate_evidence') AND sql IS NOT NULL ORDER BY sql`)
if err != nil {
return err
}
for rows.Next() {
var s string
if err := rows.Scan(&s); err != nil {
rows.Close()
return err
}
actual = append(actual, strings.TrimSpace(s))
}
err = rows.Err()
rows.Close()
if err != nil {
return err
}
var expected []string
for _, s := range strings.Split(strings.TrimSpace(candidateSchema), ";\n") {
if !strings.HasPrefix(s, "PRAGMA") {
expected = append(expected, strings.TrimSpace(s))
}
}
sort.Strings(expected)
if strings.Join(actual, "\n") != strings.Join(expected, "\n") {
return fmt.Errorf("unsupported candidate schema; asset migration refused")
}
if _, err := conn.ExecContext(ctx, `CREATE TEMP TABLE asset_candidate_backup AS SELECT * FROM strategy_candidates;
CREATE TEMP TABLE asset_evidence_backup AS SELECT * FROM candidate_evidence;
DROP TABLE candidate_evidence;
DROP TABLE strategy_candidates;`); err != nil {
return err
}
schema := strings.Replace(candidateSchema, "('frontend', 'googlevideo')", "('frontend', 'googlevideo', 'assets')", 1)
if _, err := conn.ExecContext(ctx, schema); err != nil {
return err
}
if _, err := conn.ExecContext(ctx, `CREATE TRIGGER asset_candidate_scope BEFORE INSERT ON strategy_candidates
WHEN NEW.strategy_group = 'assets' AND (json_array_length(NEW.endpoints) > 2
OR EXISTS (SELECT 1 FROM json_each(NEW.endpoints) WHERE type != 'text' OR value NOT IN ('i.ytimg.com', 'yt3.ggpht.com'))
OR (SELECT count(DISTINCT value) FROM json_each(NEW.endpoints)) != json_array_length(NEW.endpoints))
BEGIN SELECT RAISE(ABORT, 'invalid asset service inventory'); END;`); err != nil {
return err
}
_, err = conn.ExecContext(ctx, `INSERT INTO strategy_candidates SELECT * FROM asset_candidate_backup;
INSERT INTO candidate_evidence SELECT * FROM asset_evidence_backup;
DROP TABLE asset_candidate_backup;
DROP TABLE asset_evidence_backup;
PRAGMA user_version = 6;`)
return err
}
// No URLs: the caller supplies actual response bodies, never a blockcheck report.
// Three transfers per exact TLS version are required; no probing is wired here.
type assetTransfer struct {
EffectiveHost string `json:"effective_host"`
SSLVerifyResult *int `json:"ssl_verify_result"`
Redirects int `json:"redirects"`
ContentType string `json:"content_type"`
BodyHash string `json:"body_sha256"`
Format string `json:"format"`
Size int `json:"size"`
TLSVersion uint16 `json:"tls_version"`
Status int `json:"status"`
Body []byte `json:"-"`
LatencyMS float64 `json:"latency_ms"`
SpeedKBPS float64 `json:"speed_kbps"`
}
type assetMeasurement struct {
Source string `json:"source"`
Endpoint string `json:"endpoint"`
Protocol string `json:"protocol"`
Args string `json:"args"`
Transfers []assetTransfer `json:"transfers"`
}
func (t *assetTransfer) UnmarshalJSON(raw []byte) error {
type fields assetTransfer
var keys map[string]json.RawMessage
if err := json.Unmarshal(raw, &keys); err != nil {
return err
}
for _, key := range []string{"effective_host", "ssl_verify_result", "redirects", "content_type", "body_sha256", "format", "size", "tls_version", "status", "latency_ms", "speed_kbps"} {
if value, ok := keys[key]; !ok || bytes.Equal(bytes.TrimSpace(value), []byte("null")) {
return fmt.Errorf("missing image transfer field")
}
}
var value fields
d := json.NewDecoder(bytes.NewReader(raw))
d.DisallowUnknownFields()
if err := d.Decode(&value); err != nil {
return err
}
*t = assetTransfer(value)
return nil
}
func saveAssetEvidence(tx *sql.Tx, id int64, m assetMeasurement, checked time.Time) error {
if m.Source != "image-transfer" || !assetEndpoint(m.Endpoint) || (m.Protocol != "tls12" && m.Protocol != "tls13") || checked.IsZero() || len(m.Transfers) != 3 {
return fmt.Errorf("invalid image-transfer evidence")
}
var group, args string
if err := tx.QueryRow(`SELECT strategy_group, args FROM strategy_candidates WHERE id = ?`, id).Scan(&group, &args); err != nil {
return err
}
if group != "assets" || m.Args != args {
return fmt.Errorf("asset evidence definition mismatch")
}
latency, speed, err := validateAssetTransfers(m)
if err != nil {
return err
}
for i := range m.Transfers {
if err := normalizeAssetBody(&m.Transfers[i]); err != nil {
return err
}
}
raw, err := json.Marshal(m)
if err != nil {
return err
}
_, err = tx.Exec(`INSERT INTO candidate_evidence (candidate_id, endpoint, protocol, passes_ok, passes_total, success_rate, coverage, median_latency_ms, median_speed_kbps, checked_at, raw_measurement) VALUES (?, ?, ?, 3, 3, 1, 1, ?, ?, ?, ?)`, id, m.Endpoint, m.Protocol, latency, speed, checked.UTC().Format(time.RFC3339Nano), string(raw))
return err
}
func validateAssetTransfers(m assetMeasurement) (float64, float64, error) {
if m.Source != "image-transfer" || !assetEndpoint(m.Endpoint) || (m.Protocol != "tls12" && m.Protocol != "tls13") || len(m.Transfers) != 3 {
return 0, 0, fmt.Errorf("invalid image-transfer evidence")
}
tlsVersion := uint16(0x0303)
if m.Protocol == "tls13" {
tlsVersion = 0x0304
}
var latency, speed []float64
for _, transfer := range m.Transfers {
if transfer.TLSVersion != tlsVersion || transfer.Status != 200 || transfer.EffectiveHost != m.Endpoint || transfer.SSLVerifyResult == nil || *transfer.SSLVerifyResult != 0 || transfer.Redirects != 0 {
return 0, 0, fmt.Errorf("image transfer must match TLS and HTTP 200 with a bounded image body")
}
if err := normalizeAssetBody(&transfer); err != nil {
return 0, 0, err
}
for _, n := range []float64{transfer.LatencyMS, transfer.SpeedKBPS} {
if n < 0 || math.IsNaN(n) || math.IsInf(n, 0) {
return 0, 0, fmt.Errorf("invalid image transfer metric")
}
}
latency = append(latency, transfer.LatencyMS)
speed = append(speed, transfer.SpeedKBPS)
}
sort.Float64s(latency)
sort.Float64s(speed)
return latency[1], speed[1], nil
}
// Only supplied body bytes can produce this typed rejection. Missing/corrupt
// normalized evidence remains a local format error, never candidate absence.
type imageBodyRejection string
func (e imageBodyRejection) Error() string { return string(e) }
// Bodies are transient. Stored evidence carries only verified normalized facts.
func normalizeAssetBody(t *assetTransfer) error {
if len(t.Body) != 0 {
if len(t.Body) > 5<<20 {
return imageBodyRejection("image body exceeds limit")
}
cfg, format, err := image.DecodeConfig(bytes.NewReader(t.Body))
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 || cfg.Width > (16<<20)/cfg.Height {
return imageBodyRejection("invalid image dimensions")
}
if format == "gif" {
if _, err := gif.DecodeAll(bytes.NewReader(t.Body)); err != nil {
return imageBodyRejection("invalid GIF body")
}
} else if _, _, err := image.Decode(bytes.NewReader(t.Body)); err != nil {
return imageBodyRejection("invalid image body")
}
t.Format, t.Size, t.BodyHash = format, len(t.Body), fmt.Sprintf("%x", sha256.Sum256(t.Body))
}
want := map[string]string{"png": "image/png", "jpeg": "image/jpeg", "gif": "image/gif"}[t.Format]
if want == "" || t.ContentType != want || t.Size <= 0 || t.Size > 5<<20 || len(t.BodyHash) != 64 || strings.Trim(t.BodyHash, "0123456789abcdef") != "" {
if len(t.Body) != 0 {
return imageBodyRejection("invalid image content type")
}
return fmt.Errorf("invalid normalized image evidence")
}
return nil
}
// Revalidate asset bodies on reads too: corrupted raw evidence cannot qualify.
func validateStoredAssetEvidence(db *sql.DB, c storedCandidate) (bool, error) {
rows, err := db.Query(`SELECT endpoint, protocol, median_latency_ms, median_speed_kbps, checked_at, raw_measurement FROM candidate_evidence WHERE candidate_id = ?`, c.ID)
if err != nil {
return false, err
}
defer rows.Close()
for rows.Next() {
var endpoint, protocol, checkedAt, raw string
var latency, speed float64
if err := rows.Scan(&endpoint, &protocol, &latency, &speed, &checkedAt, &raw); err != nil {
return false, err
}
checked, err := time.Parse(time.RFC3339Nano, checkedAt)
if err != nil || checked.IsZero() {
return false, nil
}
var m assetMeasurement
d := json.NewDecoder(strings.NewReader(raw))
d.DisallowUnknownFields()
if err := d.Decode(&m); err != nil {
return false, nil
}
if err := d.Decode(new(interface{})); err != io.EOF {
return false, nil
}
l, s, err := validateAssetTransfers(m)
if err != nil || m.Endpoint != endpoint || m.Protocol != protocol || m.Args != c.Args || l != latency || s != speed {
return false, nil
}
}
return true, rows.Err()
}
+317
View File
@@ -0,0 +1,317 @@
package main
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"image"
"image/png"
"io"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
// Recreate an actual pre-assets schema, not just a misleading version number.
func restoreV5CandidateSchema(t *testing.T, db *sql.DB) {
t.Helper()
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
_, err = tx.Exec(`CREATE TEMP TABLE old_candidates AS SELECT id, source_run_id, strategy_group, endpoints, args, fingerprint FROM strategy_candidates;
CREATE TEMP TABLE old_evidence AS SELECT * FROM candidate_evidence;
DROP TABLE googlevideo_scope_evidence;
DROP TABLE candidate_evidence; DROP TABLE strategy_candidates;` + candidateSchema + `
INSERT INTO strategy_candidates SELECT * FROM old_candidates;
INSERT INTO candidate_evidence SELECT * FROM old_evidence;
DROP TABLE old_candidates; DROP TABLE old_evidence;
PRAGMA user_version = 5;`)
if err != nil {
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
}
// Synthetic unit fixture only: no claim that any endpoint was reachable.
func unitImageMeasurement(t *testing.T, host, protocol, args string) assetMeasurement {
t.Helper()
var body bytes.Buffer
if err := png.Encode(&body, image.NewRGBA(image.Rect(0, 0, 1, 1))); err != nil {
t.Fatal(err)
}
v := uint16(0x0303)
if protocol == "tls13" {
v = 0x0304
}
m := assetMeasurement{Source: "image-transfer", Endpoint: host, Protocol: protocol, Args: args}
for i := 0; i < 3; i++ {
zero := 0
m.Transfers = append(m.Transfers, assetTransfer{EffectiveHost: host, SSLVerifyResult: &zero, ContentType: "image/png", TLSVersion: v, Status: 200, Body: body.Bytes(), LatencyMS: 10, SpeedKBPS: 100})
}
return m
}
func unitAssetCandidate(t *testing.T, db *sql.DB, host, args string, qualify bool) int64 {
t.Helper()
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
id, err := insertStrategyCandidate(tx, "assets", []string{host}, args, nil)
if err != nil {
t.Fatal(err)
}
if qualify {
for _, protocol := range []string{"tls12", "tls13"} {
if err := saveAssetEvidence(tx, id, unitImageMeasurement(t, host, protocol, args), time.Now()); err != nil {
t.Fatal(err)
}
}
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
return id
}
func TestAssetMigrationLegacyBytesAndRollback(t *testing.T) {
db := testStore(t)
saveProposal(t, db, "--lua-desync=fake")
restoreV5CandidateSchema(t, db)
before, err := readStrategyCandidate(db, 1)
if err != nil {
t.Fatal(err)
}
config, err := renderManagedConfig(db, []int64{1})
if err != nil {
t.Fatal(err)
}
var evidence string
if err := db.QueryRow(`SELECT group_concat(raw_measurement) FROM candidate_evidence`).Scan(&evidence); err != nil {
t.Fatal(err)
}
if err := migrate(db); err != nil {
t.Fatal(err)
}
after, err := readStrategyCandidate(db, 1)
if err != nil || !reflect.DeepEqual(before, after) {
t.Fatalf("changed candidate: %+v %v", after, err)
}
rendered, err := renderManagedConfig(db, []int64{1})
if err != nil || !bytes.Equal(config, rendered) || managedHash(config) != managedHash(rendered) {
t.Fatalf("legacy bytes changed: %v", err)
}
var afterEvidence string
if err := db.QueryRow(`SELECT group_concat(raw_measurement) FROM candidate_evidence`).Scan(&afterEvidence); err != nil || evidence != afterEvidence {
t.Fatalf("evidence changed: %v", err)
}
if _, err := db.Exec(`UPDATE strategy_candidates SET args = '--lua-desync=multisplit'`); err == nil {
t.Fatal("immutable trigger lost")
}
restoreV5CandidateSchema(t, db)
// A foreign reference forces failure after the migration starts rebuilding.
if _, err := db.Exec(`CREATE TABLE candidate_external_ref (id INTEGER REFERENCES strategy_candidates(id)); INSERT INTO candidate_external_ref VALUES (1);`); err != nil {
t.Fatal(err)
}
if err := migrate(db); err == nil {
t.Fatal("expected atomic migration failure")
}
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 5 {
t.Fatalf("version=%d %v", version, err)
}
rendered, err = renderManagedConfig(db, []int64{1})
if err != nil || !bytes.Equal(config, rendered) {
t.Fatalf("rollback lost legacy bytes: %v", err)
}
if _, err := db.Exec(`DROP TABLE candidate_external_ref; CREATE TRIGGER custom_candidate BEFORE INSERT ON strategy_candidates BEGIN SELECT 1; END;`); err != nil {
t.Fatal(err)
}
if err := migrate(db); err == nil {
t.Fatal("unknown schema silently adopted")
}
}
func TestAssetEvidenceAndDisjointRender(t *testing.T) {
db := testStore(t)
a := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
b := unitAssetCandidate(t, db, "yt3.ggpht.com", "--lua-desync=multisplit", true)
config, err := renderManagedConfig(db, []int64{a, b})
if err != nil || !strings.Contains(string(config), "--hostlist-domains=^i.ytimg.com --lua-desync=fake --new") || !strings.Contains(string(config), "--hostlist-domains=^yt3.ggpht.com --lua-desync=multisplit") {
t.Fatalf("render: %s %v", config, err)
}
ids, err := qualifiedCandidateIDs(db, "assets")
if err != nil || !reflect.DeepEqual(ids, []int64{b, a}) {
t.Fatalf("IDs %v %v", ids, err)
}
if _, err := db.Exec(`UPDATE candidate_evidence SET checked_at = 'fixture' WHERE candidate_id = ? AND protocol = 'tls12'`, a); err != nil {
t.Fatal(err)
}
if _, err := renderManagedConfig(db, []int64{a}); err == nil {
t.Fatal("invalid checked timestamp qualified")
}
if _, err := db.Exec(`UPDATE candidate_evidence SET checked_at = ? WHERE candidate_id = ? AND protocol = 'tls12'`, time.Now().UTC().Format(time.RFC3339Nano), a); err != nil {
t.Fatal(err)
}
overlap := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=multisplit", true)
if _, err := renderManagedConfig(db, []int64{a, overlap}); err == nil {
t.Fatal("accepted overlap")
}
missing := unitAssetCandidate(t, db, "yt3.ggpht.com", "--lua-desync=fake", false)
if _, err := renderManagedConfig(db, []int64{a, missing}); err == nil {
t.Fatal("accepted missing evidence")
}
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
for _, host := range []string{"ytimg.com", "other.ytimg.com", "*.ytimg.com", "youtube.com"} {
if _, err := insertStrategyCandidate(tx, "assets", []string{host}, "--lua-desync=fake", nil); err == nil {
t.Fatalf("accepted %s", host)
}
}
if err := saveCandidateEvidence(tx, missing, "yt3.ggpht.com", "tls12", []byte(`{}`), time.Now()); err == nil {
t.Fatal("accepted blockcheck for assets")
}
for _, damage := range []string{"status", "magic", "tls", "count", "source", "scope"} {
m := unitImageMeasurement(t, "yt3.ggpht.com", "tls12", "--lua-desync=fake")
switch damage {
case "status":
m.Transfers[0].Status = 400
case "magic":
m.Transfers[0].Body = []byte("not an image")
case "tls":
m.Transfers[0].TLSVersion = 0x0304
case "count":
m.Transfers = m.Transfers[:2]
case "source":
m.Source = "blockcheck"
case "scope":
m.Endpoint = "i.ytimg.com"
}
if err := saveAssetEvidence(tx, missing, m, time.Now()); err == nil {
t.Fatalf("accepted %s", damage)
}
}
tx.Rollback()
m := unitImageMeasurement(t, "yt3.ggpht.com", "tls13", "--lua-desync=multisplit")
m.Transfers[0].Status = 400
raw, _ := json.Marshal(m)
if _, err := db.Exec(`UPDATE candidate_evidence SET raw_measurement = ? WHERE candidate_id = ? AND protocol = 'tls13'`, string(raw), b); err != nil {
t.Fatal(err)
}
if _, err := renderManagedConfig(db, []int64{b}); err == nil {
t.Fatal("corrupt HTTP400 evidence qualified")
}
if _, err := db.Exec("PRAGMA user_version = 8"); err != nil {
t.Fatal(err)
}
if _, err := renderManagedConfig(db, []int64{a}); err == nil {
t.Fatal("unknown schema rendered")
}
}
func TestAssetCLIApplyPartialFailureAndStatus(t *testing.T) {
for _, fail := range []bool{false, true} {
t.Run(fmt.Sprint(fail), func(t *testing.T) {
d, env, _ := managedCLIFixture(t)
env["ZAPRET_AUTOTUNE_VIDEO_ID"] = "dQw4w9WgXcQ"
path := filepath.Join(t.TempDir(), "assets.db")
db, err := openStore(path)
if err != nil {
t.Fatal(err)
}
saveProposal(t, db, "--lua-desync=fake")
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
ggc, err := insertStrategyCandidate(tx, "googlevideo", []string{"rr1.unit.googlevideo.com"}, "--lua-desync=fake", nil)
if err != nil {
tx.Rollback()
t.Fatal(err)
}
var report struct {
Strategies []json.RawMessage `json:"strategies"`
}
if err := json.Unmarshal([]byte(combinedReportFixture("--lua-desync=fake")), &report); err != nil {
tx.Rollback()
t.Fatal(err)
}
for i, protocol := range []string{"tls12", "tls13"} {
if err := saveCandidateEvidence(tx, ggc, "rr1.unit.googlevideo.com", protocol, report.Strategies[i], time.Now()); err != nil {
tx.Rollback()
t.Fatal(err)
}
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
a := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
b := unitAssetCandidate(t, db, "yt3.ggpht.com", "--lua-desync=multisplit", true)
db.Close()
d.openDB = func() (*sql.DB, error) { return openHistoryStore(path) }
initManagedCLI(t, d)
if code, out, diagnostic := callManaged(t, d, "status"); code != 0 || strings.Contains(out, "assets") {
t.Fatalf("legacy status: %s %s", out, diagnostic)
}
if code, out, diagnostic := callManaged(t, d, "status", "--assets"); code != 0 || !strings.Contains(out, "unresolved_groups=frontend,googlevideo,assets") {
t.Fatalf("requested status: %s %s", out, diagnostic)
}
if fail {
d.player = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
return errors.New("unit health failure")
}
}
code, _, diagnostic := callManaged(t, d, "apply", "--asset-candidate", fmt.Sprintf("%d,%d", a, b))
if (code != 0) != fail {
t.Fatalf("apply=%d %s", code, diagnostic)
}
config, err := os.ReadFile(filepath.Join(d.directory, "config"))
if err != nil {
t.Fatal(err)
}
if fail && string(config) != managedBaseline {
t.Fatal("failure did not restore baseline")
}
if !fail {
if code, out, diagnostic := callManaged(t, d, "status"); code != 0 || !strings.Contains(out, "groups=assets") || strings.Contains(out, "unresolved_groups=frontend,googlevideo,assets") {
t.Fatalf("asset status: %s %s", out, diagnostic)
}
if code, _, diagnostic := callManaged(t, d, "apply", "--asset-candidate", fmt.Sprint(a)); code != 0 {
t.Fatalf("partial apply: %s", diagnostic)
}
if code, out, diagnostic := callManaged(t, d, "status"); code != 0 || !strings.Contains(out, "unresolved_groups=frontend,googlevideo,assets") {
t.Fatalf("partial status: %s %s", out, diagnostic)
}
if code, _, diagnostic := callManaged(t, d, "apply", "--frontend-candidate", "1", "--googlevideo-candidate", fmt.Sprint(ggc), "--asset-candidate", fmt.Sprint(a), "--asset-candidate", fmt.Sprint(b)); code != 0 {
t.Fatalf("four candidate apply: %s", diagnostic)
}
if code, out, diagnostic := callManaged(t, d, "active-ids"); code != 0 || !strings.Contains(out, fmt.Sprintf(`"active_candidate_ids":[1,%d,%d,%d]`, ggc, a, b)) || !strings.Contains(out, `"unresolved_groups":null`) {
t.Fatalf("four IDs: %s %s", out, diagnostic)
}
}
})
}
for _, args := range [][]string{{"apply", "--asset-candidate", "1,1"}, {"apply", "--asset-candidate", "1,"}, {"apply", "--asset-candidate", "1,2,3"}, {"apply", "--frontend-candidate", "1,2"}, {"apply", "--frontend-candidate", "1", "--asset-candidate", "1"}} {
if _, _, err := managedArguments(args); err == nil {
t.Fatalf("accepted %v", args)
}
}
_, ids, err := managedArguments([]string{"apply", "--asset-candidate", "3", "--googlevideo-candidate", "2", "--asset-candidate", "4", "--frontend-candidate", "1"})
if err != nil || !reflect.DeepEqual(ids, []int64{1, 2, 3, 4}) {
t.Fatalf("IDs %v %v", ids, err)
}
}
+368 -7
View File
@@ -5,8 +5,12 @@ import (
"database/sql"
"encoding/json"
"fmt"
"io"
"math"
"strings"
"time"
"zapret-autotune/internal/adapter/youtube"
)
// Additive only: old proposals are not independently qualified candidates.
@@ -47,7 +51,17 @@ CREATE TRIGGER evidence_scope_update BEFORE UPDATE ON candidate_evidence
BEGIN SELECT RAISE(ABORT, 'evidence endpoint outside candidate scope'); END;
PRAGMA user_version = 3;`
func insertStrategyCandidate(tx *sql.Tx, group string, endpoints []string, args string, sourceRun interface{}) (int64, error) {
func insertStrategyCandidate(tx *sql.Tx, group string, endpoints []string, args string, sourceRun interface{}, modes ...QualificationMode) (int64, error) {
mode := QualificationCold
if len(modes) > 1 {
return 0, fmt.Errorf("one qualification mode required")
}
if len(modes) == 1 {
mode = modes[0]
}
if mode != QualificationCold && (mode != QualificationAdaptive || group != "googlevideo") {
return 0, fmt.Errorf("invalid qualification mode/group")
}
c := candidate{Group: group, Domains: append([]string(nil), endpoints...), Args: args}
if err := c.validateMeasurements(); err != nil {
return 0, err
@@ -65,7 +79,13 @@ func insertStrategyCandidate(tx *sql.Tx, group string, endpoints []string, args
if err != nil {
return 0, err
}
result, err := tx.Exec(`INSERT INTO strategy_candidates (source_run_id, strategy_group, endpoints, args, fingerprint) VALUES (?, ?, ?, ?, ?)`, sourceRun, group, string(scope), args, fingerprint)
query := `INSERT INTO strategy_candidates (source_run_id, strategy_group, endpoints, args, fingerprint) VALUES (?, ?, ?, ?, ?)`
values := []interface{}{sourceRun, group, string(scope), args, fingerprint}
if mode == QualificationAdaptive {
query = `INSERT INTO strategy_candidates (source_run_id, strategy_group, endpoints, args, fingerprint, qualification_mode) VALUES (?, ?, ?, ?, ?, ?)`
values = append(values, mode)
}
result, err := tx.Exec(query, values...)
if err != nil {
return 0, err
}
@@ -74,6 +94,19 @@ func insertStrategyCandidate(tx *sql.Tx, group string, endpoints []string, args
// Evidence is tied to the immutable definition, never inferred from scan output.
func saveCandidateEvidence(tx *sql.Tx, id int64, endpoint, protocol string, raw []byte, checked time.Time) error {
var version int
if err := tx.QueryRow("PRAGMA user_version").Scan(&version); err != nil {
return err
}
if version >= 7 {
var mode QualificationMode
if err := tx.QueryRow("SELECT qualification_mode FROM strategy_candidates WHERE id = ?", id).Scan(&mode); err != nil {
return err
}
if mode != QualificationCold {
return fmt.Errorf("adaptive evidence must not use cold metrics")
}
}
if !validDomain(endpoint) || (protocol != "tls12" && protocol != "tls13") || checked.IsZero() {
return fmt.Errorf("invalid evidence endpoint, protocol or checked timestamp")
}
@@ -82,6 +115,9 @@ func saveCandidateEvidence(tx *sql.Tx, id int64, endpoint, protocol string, raw
if err := tx.QueryRow(`SELECT strategy_group, args FROM strategy_candidates WHERE id = ?`, id).Scan(&group, &args); err != nil {
return err
}
if group == "assets" {
return fmt.Errorf("assets require image-transfer evidence")
}
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
return err
@@ -119,6 +155,7 @@ type storedCandidate struct {
Group, Args, Fingerprint string
Endpoints []string
Qualified bool
Mode QualificationMode
}
// Qualification precedes LIMIT: at most ten immutable definitions per trial.
@@ -138,18 +175,41 @@ const candidateQualifiedSQL = `NOT EXISTS (SELECT 1 FROM json_each(c.endpoints)
(SELECT 'tls12' AS protocol UNION ALL SELECT 'tls13') p
WHERE NOT EXISTS (SELECT 1 FROM candidate_evidence r WHERE r.candidate_id = c.id
AND r.endpoint = e.value AND r.protocol = p.protocol AND r.passes_ok = 3
AND r.passes_total = 3 AND r.success_rate = 1 AND r.coverage = 1))`
AND r.passes_total = 3 AND r.success_rate = 1 AND r.coverage = 1
AND (c.strategy_group != 'assets' OR (json_extract(r.raw_measurement, '$.source') = 'image-transfer'
AND json_extract(r.raw_measurement, '$.endpoint') = e.value
AND json_extract(r.raw_measurement, '$.protocol') = p.protocol
AND json_extract(r.raw_measurement, '$.args') = c.args
AND json_array_length(r.raw_measurement, '$.transfers') = 3))))`
// Returns detached snapshots of immutable definitions, not active state.
func listQualifiedCandidates(db *sql.DB, group string, limit int) ([]storedCandidate, error) {
if group != "frontend" && group != "googlevideo" {
if group != "frontend" && group != "googlevideo" && group != "assets" {
return nil, fmt.Errorf("unknown candidate group")
}
if limit < 1 || limit > 10 {
return nil, fmt.Errorf("candidate limit must be between 1 and 10")
}
rows, err := db.Query(`SELECT id, strategy_group, args, fingerprint, endpoints FROM strategy_candidates c
WHERE strategy_group = ? AND `+candidateQualifiedSQL+` ORDER BY id DESC LIMIT ?`, group, limit)
query := `SELECT id, strategy_group, args, fingerprint, endpoints FROM strategy_candidates c
WHERE strategy_group = ? AND ` + candidateQualifiedSQL + ` ORDER BY id DESC`
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil {
return nil, err
}
if version > 7 {
return nil, fmt.Errorf("unsupported qualification schema")
}
if version == 7 {
query = `SELECT id, strategy_group, args, fingerprint, endpoints FROM strategy_candidates c
WHERE strategy_group = ? AND ((qualification_mode = 'cold' AND ` + candidateQualifiedSQL + `) OR qualification_mode = 'adaptive') ORDER BY id DESC`
}
queryArgs := []interface{}{group}
// Asset bodies must validate before LIMIT, not just their SQL summaries.
if group == "frontend" {
query += " LIMIT ?"
queryArgs = append(queryArgs, limit)
}
rows, err := db.Query(query, queryArgs...)
if err != nil {
return nil, err
}
@@ -165,17 +225,51 @@ func listQualifiedCandidates(db *sql.DB, group string, limit int) ([]storedCandi
return nil, err
}
c.Qualified = true
c.Mode = QualificationCold
result = append(result, c)
}
if err := rows.Err(); err != nil {
return nil, err
}
return result, rows.Close()
if err := rows.Close(); err != nil {
return nil, err
}
if group == "assets" || group == "googlevideo" {
var validated []storedCandidate
for _, c := range result {
ok := true
var err error
if group == "assets" {
ok, err = validateStoredAssetEvidence(db, c)
} else {
c.Mode, err = readQualificationMode(db, c.ID)
ok = c.Mode == QualificationCold
if err == nil && c.Mode == QualificationAdaptive {
ok, err = validateStoredAdaptiveGooglevideoEvidence(db, c)
}
}
if err != nil {
return nil, err
}
if ok {
validated = append(validated, c)
if len(validated) == limit {
break
}
}
}
return validated, nil
}
return result, nil
}
// One shared qualification rule for the complete endpoint × TLS matrix.
// Missing or failed evidence is never a qualification; this is not active state.
func readStrategyCandidate(db *sql.DB, id int64) (storedCandidate, error) {
return readStrategyCandidatePolicy(db, id, true)
}
func readStrategyCandidatePolicy(db *sql.DB, id int64, fresh bool) (storedCandidate, error) {
var c storedCandidate
var scope string
err := db.QueryRow(`SELECT id, strategy_group, args, fingerprint, endpoints,
@@ -185,5 +279,272 @@ func readStrategyCandidate(db *sql.DB, id int64) (storedCandidate, error) {
return c, err
}
err = json.Unmarshal([]byte(scope), &c.Endpoints)
if err == nil {
c.Mode, err = readQualificationMode(db, id)
if c.Mode != QualificationCold {
c.Qualified = false
if err == nil && c.Mode == QualificationAdaptive {
c.Qualified, err = validateAdaptiveGooglevideoEvidence(db, c, fresh)
}
}
}
if err == nil && c.Qualified && c.Group == "assets" {
c.Qualified, err = validateStoredAssetEvidence(db, c)
}
return c, err
}
type QualificationMode string
const QualificationCold QualificationMode = "cold"
const QualificationAdaptive QualificationMode = "adaptive"
const NativeCircularPostWarmup = "native-circular-post-warmup"
// Additive migration: no candidate/evidence/asset/owner table rebuilds.
const googlevideoQualificationSchema = `
ALTER TABLE strategy_candidates ADD COLUMN qualification_mode TEXT NOT NULL DEFAULT 'cold'
CHECK (qualification_mode = 'cold' OR (qualification_mode = 'adaptive' AND strategy_group = 'googlevideo'));
CREATE TABLE googlevideo_scope_evidence (
candidate_id INTEGER PRIMARY KEY REFERENCES strategy_candidates(id),
proof TEXT NOT NULL CHECK (json_valid(proof))
);
PRAGMA user_version = 7;`
func readQualificationMode(db *sql.DB, id int64) (QualificationMode, error) {
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil {
return "", err
}
if version < 7 {
return QualificationCold, nil
}
if version != 7 {
return "", fmt.Errorf("unsupported qualification schema")
}
var mode QualificationMode
err := db.QueryRow("SELECT qualification_mode FROM strategy_candidates WHERE id = ?", id).Scan(&mode)
if err == nil && mode != QualificationCold && mode != QualificationAdaptive {
err = fmt.Errorf("unknown qualification mode")
}
return mode, err
}
// Collector DTO: normalized facts only. No URL, query, body, or headers can be
// persisted. HTTP root/404 connectivity is deliberately not media evidence.
type GooglevideoPass struct {
Protocol string `json:"protocol"`
VerifiedTLS bool `json:"verified_tls"`
Status int `json:"status"`
RangeStart int64 `json:"range_start"`
RangeEnd int64 `json:"range_end"`
ResourceBytes int64 `json:"resource_bytes"`
Bytes int64 `json:"bytes"`
FTYP bool `json:"ftyp"`
LatencyMS float64 `json:"latency_ms"`
}
type GooglevideoHostProof struct {
Host string `json:"host"`
WarmupAttempts int `json:"warmup_attempts"`
WarmupFailures int `json:"warmup_failures"`
WarmupLimit int `json:"warmup_limit"`
WarmupMS float64 `json:"warmup_ms"`
Passes []GooglevideoPass `json:"steady_passes"`
}
type GooglevideoQualificationProof struct {
Adapter *youtube.Binding `json:"adapter,omitempty"`
Version int `json:"version"`
Mode QualificationMode `json:"mode"`
SourceKind string `json:"source_kind"`
Args string `json:"args"`
Hosts []GooglevideoHostProof `json:"hosts"`
Attestation *GooglevideoRuntimeAttestation `json:"attestation,omitempty"`
producerSeal *googlevideoProducerSeal
}
func (p GooglevideoQualificationProof) validate(c storedCandidate) error {
if err := p.validateStored(c); err != nil {
return err
}
if p.Mode == QualificationAdaptive {
return p.Attestation.validate(c)
}
return nil
}
func (p GooglevideoQualificationProof) validateStored(c storedCandidate) error {
// Missing binding is explicitly the immutable legacy YouTube v1 policy,
// never the current registry default. Retain historical reconstruction.
if p.Version == 1 {
if p.Adapter != nil {
return fmt.Errorf("legacy proof cannot carry adapter binding")
}
} else {
if p.Mode != QualificationAdaptive || p.Adapter == nil {
return fmt.Errorf("version-bound adaptive proof required")
}
module, err := lookupServiceAdapter(p.Adapter.Kind, p.Adapter.Version)
if err != nil {
return err
}
if *p.Adapter != module.Binding() {
return fmt.Errorf("adapter policy mismatch")
}
}
if (p.Version != 1 && p.Version != 2) || p.Mode != c.Mode || p.Args != c.Args || len(p.Hosts) != len(c.Endpoints) || len(p.Hosts) < 3 {
return fmt.Errorf("invalid representative proof definition")
}
if (p.Mode == QualificationCold && p.SourceKind != "media-range") || (p.Mode == QualificationAdaptive && p.SourceKind != NativeCircularPostWarmup) || (p.Mode != QualificationCold && p.Mode != QualificationAdaptive) {
return fmt.Errorf("invalid proof source/mode")
}
if err := (&candidate{Args: p.Args}).validateArgs(); err != nil {
return err
}
if err := validateLegacyYoutubeV1Hosts(c.Endpoints); err != nil {
return err
}
if p.Mode == QualificationCold && p.Attestation != nil {
return fmt.Errorf("cold evidence cannot contain native attestation")
}
wanted, seen, roots, prefixes := map[string]bool{}, map[string]bool{}, map[string]bool{}, map[string]bool{}
for _, host := range c.Endpoints {
wanted[host] = true
}
for _, h := range p.Hosts {
prefix, root, ok := strings.Cut(strings.TrimSuffix(h.Host, ".googlevideo.com"), "---sn-")
if !ok || !strings.HasSuffix(h.Host, ".googlevideo.com") || !validDomain(h.Host) || h.Host != strings.ToLower(h.Host) || !wanted[h.Host] || seen[h.Host] || root == "" || len(prefix) < 3 || !strings.HasPrefix(prefix, "rr") || strings.Trim(prefix[2:], "0123456789") != "" {
return fmt.Errorf("invalid exact GGC sample")
}
seen[h.Host], roots[strings.Split(root, "-")[0]], prefixes[prefix] = true, true, true
if math.IsNaN(h.WarmupMS) || math.IsInf(h.WarmupMS, 0) || h.WarmupMS < 0 || h.WarmupMS > 120000 || h.WarmupAttempts < 0 || h.WarmupFailures < 0 || h.WarmupFailures > h.WarmupAttempts || h.WarmupLimit < 0 || h.WarmupLimit > 10 || h.WarmupAttempts > h.WarmupLimit {
return fmt.Errorf("unbounded warmup")
}
if p.Mode == QualificationCold && (h.WarmupAttempts != 0 || h.WarmupFailures != 0 || h.WarmupLimit != 0 || h.WarmupMS != 0) {
return fmt.Errorf("cold evidence cannot hide warmup")
}
if p.Mode == QualificationAdaptive && (h.WarmupLimit == 0 || h.WarmupAttempts < 2 || h.WarmupFailures > h.WarmupAttempts-2 || h.WarmupMS <= 0) {
return fmt.Errorf("adaptive requires bounded warmup evidence")
}
counts := map[string]int{}
for _, pass := range h.Passes {
if (pass.Protocol != "tls12" && pass.Protocol != "tls13") || !pass.VerifiedTLS || pass.Status != 206 || pass.RangeStart != 0 || pass.RangeEnd < 1023 || pass.ResourceBytes <= pass.RangeEnd || pass.Bytes != pass.RangeEnd+1 || !pass.FTYP || math.IsNaN(pass.LatencyMS) || math.IsInf(pass.LatencyMS, 0) || pass.LatencyMS <= 0 {
return fmt.Errorf("verified ranged media transfer required")
}
counts[pass.Protocol]++
}
if counts["tls12"] != 3 || counts["tls13"] != 3 {
return fmt.Errorf("three actual steady transfers per TLS protocol required")
}
}
if len(roots) < 2 || len(prefixes) < 2 {
return fmt.Errorf("diverse GGC root groups and host prefixes required")
}
if p.Mode == QualificationAdaptive {
return p.Attestation.validateStored(c)
}
return nil
}
func validateGooglevideoSuffixEvidence(db *sql.DB, c storedCandidate) error {
return validateGooglevideoSuffixEvidencePolicy(db, c, true)
}
func validateGooglevideoSuffixEvidencePolicy(db *sql.DB, c storedCandidate, fresh bool) error {
if c.Group != "googlevideo" || !c.Qualified {
return fmt.Errorf("qualified googlevideo candidate required")
}
proof, err := readGooglevideoProof(db, c.ID)
if err != nil {
return err
}
if fresh {
return proof.validate(c)
}
return proof.validateStored(c)
}
func readGooglevideoProof(db *sql.DB, id int64) (GooglevideoQualificationProof, error) {
var proof GooglevideoQualificationProof
var raw string
if err := db.QueryRow("SELECT proof FROM googlevideo_scope_evidence WHERE candidate_id = ?", id).Scan(&raw); err != nil {
return proof, err
}
return decodeGooglevideoProof(raw)
}
func decodeGooglevideoProof(raw string) (GooglevideoQualificationProof, error) {
var proof GooglevideoQualificationProof
d := json.NewDecoder(strings.NewReader(raw))
d.DisallowUnknownFields()
if err := d.Decode(&proof); err != nil {
return proof, err
}
if err := d.Decode(new(interface{})); err != io.EOF {
return proof, fmt.Errorf("unexpected proof data")
}
return proof, nil
}
func validateStoredAdaptiveGooglevideoEvidence(db *sql.DB, c storedCandidate) (bool, error) {
return validateAdaptiveGooglevideoEvidence(db, c, true)
}
func validateAdaptiveGooglevideoEvidence(db *sql.DB, c storedCandidate, fresh bool) (bool, error) {
if c.Group != "googlevideo" || c.Mode != QualificationAdaptive {
return false, nil
}
var raw string
if err := db.QueryRow("SELECT proof FROM googlevideo_scope_evidence WHERE candidate_id = ?", c.ID).Scan(&raw); err != nil {
if err == sql.ErrNoRows {
return false, nil
}
return false, err
}
// Malformed/stale proof is unqualified, not a reason to accept cold metrics.
proof, err := decodeGooglevideoProof(raw)
if err != nil {
return false, nil
}
if !fresh {
return proof.validateStored(c) == nil, nil
}
return proof.validate(c) == nil, nil
}
// One transactional insert, no cold candidate_evidence summaries. A private
// producer seal is required as well as valid DTO facts and stored definition.
func saveAdaptiveGooglevideoEvidence(tx *sql.Tx, id int64, proof GooglevideoQualificationProof) error {
// Existing trusted v1 rows retain their original read/apply contract. Only
// this new-write boundary requires the version-bound producer protocol.
if proof.Version != 2 || proof.producerSeal == nil || *proof.producerSeal != googlevideoProofSeal(proof) {
return fmt.Errorf("completed owned collector attestation required")
}
var c storedCandidate
var scope string
if err := tx.QueryRow("SELECT strategy_group, args, fingerprint, endpoints, qualification_mode FROM strategy_candidates WHERE id = ?", id).Scan(&c.Group, &c.Args, &c.Fingerprint, &scope, &c.Mode); err != nil {
return err
}
if err := json.Unmarshal([]byte(scope), &c.Endpoints); err != nil {
return err
}
if c.Group != "googlevideo" || c.Mode != QualificationAdaptive {
return fmt.Errorf("adaptive definition required")
}
if err := proof.validate(c); err != nil {
return err
}
var coldRows int
if err := tx.QueryRow("SELECT count(*) FROM candidate_evidence WHERE candidate_id = ?", id).Scan(&coldRows); err != nil {
return err
}
if coldRows != 0 {
return fmt.Errorf("adaptive evidence cannot contain cold rows")
}
raw, err := json.Marshal(proof)
if err != nil {
return err
}
_, err = tx.Exec("INSERT INTO googlevideo_scope_evidence (candidate_id, proof) VALUES (?, ?)", id, string(raw))
return err
}
+2 -2
View File
@@ -58,7 +58,7 @@ func populatedV2Store(t *testing.T) *sql.DB {
if err := migrate(db); err != nil {
t.Fatal(err)
}
if _, err := db.Exec(`DROP TABLE monitor_state; DROP TABLE candidate_evidence; DROP TABLE strategy_candidates; PRAGMA user_version = 2;`); err != nil {
if _, err := db.Exec(`DROP TABLE monitor_state; DROP TABLE googlevideo_scope_evidence; DROP TABLE candidate_evidence; DROP TABLE strategy_candidates; PRAGMA user_version = 2;`); err != nil {
t.Fatal(err)
}
return db
@@ -99,7 +99,7 @@ func TestPopulatedV2CandidateMigration(t *testing.T) {
}
}
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 5 {
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 7 {
t.Fatalf("version=%d err=%v", version, err)
}
if before != legacySnapshot(t, db) {
+91 -4
View File
@@ -1,6 +1,7 @@
package main
import (
"encoding/json"
"fmt"
"net"
"net/url"
@@ -28,12 +29,18 @@ func parseHealthConfig(lookup func(string) (string, bool)) (healthConfig, error)
SOCKSProxy: value("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5://127.0.0.1:11080"),
VideoID: value("ZAPRET_AUTOTUNE_VIDEO_ID", "dQw4w9WgXcQ"),
}
if _, err := managedServiceEnabled(lookup); err != nil {
return healthConfig{}, err
}
if !validLoopbackSOCKS(cfg.SOCKSProxy) {
return healthConfig{}, fmt.Errorf("ZAPRET_AUTOTUNE_SOCKS_PROXY requires socks5://LOOPBACK:PORT without credentials")
return healthConfig{}, fmt.Errorf("ZAPRET_AUTOTUNE_SOCKS_PROXY requires socks5:// or socks5h://LOOPBACK:PORT without credentials")
}
if !videoIDPattern.MatchString(cfg.VideoID) {
return healthConfig{}, fmt.Errorf("ZAPRET_AUTOTUNE_VIDEO_ID requires an 11-character video ID")
}
if _, _, err := managedGooglevideoConfig(lookup, cfg.VideoID); err != nil {
return healthConfig{}, err
}
for _, setting := range []struct {
key string
fallback string
@@ -56,14 +63,94 @@ func parseHealthConfig(lookup func(string) (string, bool)) (healthConfig, error)
return cfg, nil
}
func managedGooglevideoConfig(lookup func(string) (string, bool), primary string) (QualificationMode, []string, error) {
mode := QualificationCold
if raw, ok := lookup("ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE"); ok {
mode = QualificationMode(raw)
}
if mode != QualificationCold && mode != QualificationAdaptive {
return "", nil, fmt.Errorf("ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE requires cold or adaptive")
}
service, _ := managedServiceEnabled(lookup)
if mode == QualificationAdaptive && !service {
return "", nil, fmt.Errorf("adaptive googlevideo requires ZAPRET_AUTOTUNE_YOUTUBE_SERVICE=1")
}
ids := []string{primary}
// YouTube-specific public discovery samples, not CDN addresses. Explicit
// inventories below override these defaults; health/discovery primary stays put.
if service {
for _, id := range []string{"aqz-KE-bpKQ", "jNQXAC9IVRw"} {
if id != primary {
ids = append(ids, id)
}
}
}
if raw, ok := lookup("ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS"); ok && raw != "" {
if json.Unmarshal([]byte(raw), &ids) != nil || len(ids) == 0 || len(ids) > 5 {
return "", nil, fmt.Errorf("GOOGLEVIDEO_VIDEO_IDS requires a JSON array of 1..5 video IDs")
}
seen := map[string]bool{}
for _, id := range ids {
if !videoIDPattern.MatchString(id) || seen[id] {
return "", nil, fmt.Errorf("GOOGLEVIDEO_VIDEO_IDS requires unique 11-character video IDs")
}
seen[id] = true
}
}
return mode, ids, nil
}
func managedServiceEnabled(lookup func(string) (string, bool)) (bool, error) {
value, exists := lookup("ZAPRET_AUTOTUNE_YOUTUBE_SERVICE")
if !exists {
return false, nil
}
if value != "0" && value != "1" {
return false, fmt.Errorf("ZAPRET_AUTOTUNE_YOUTUBE_SERVICE requires 0 or 1")
}
return value == "1", nil
}
func validLoopbackSOCKS(raw string) bool {
if !validDiscoveryProxy(raw) {
u, err := url.Parse(raw)
if err != nil || (u.Scheme != "socks5" && u.Scheme != "socks5h") || strings.IndexFunc(raw, func(r rune) bool { return r < 32 || r == 127 }) >= 0 {
return false
}
u, err := url.Parse(raw)
if err != nil {
if !validDiscoveryProxy(strings.Replace(raw, "socks5h://", "socks5://", 1)) {
return false
}
host := u.Hostname()
return strings.EqualFold(host, "localhost") || net.ParseIP(host).IsLoopback()
}
// Managed probes send hostnames through the owned SS listener, not local DNS.
// Public health preserves the explicit scheme; public discovery keeps socks5.
func managedProbeProxy(raw string) (string, error) {
u, err := url.Parse(raw)
if err != nil || (u.Scheme != "socks5" && u.Scheme != "socks5h") || strings.IndexFunc(raw, func(r rune) bool { return r < 32 || r == 127 }) >= 0 {
return "", fmt.Errorf("invalid managed SOCKS proxy")
}
if !validLoopbackSOCKS(strings.Replace(raw, "socks5h://", "socks5://", 1)) {
return "", fmt.Errorf("invalid managed SOCKS proxy")
}
u.Scheme = "socks5h"
return u.String(), nil
}
func parseManagedHealthConfig(lookup func(string) (string, bool)) (healthConfig, error) {
// Reuse all legacy setting validation without changing its public contract.
cfg, err := parseHealthConfig(func(key string) (string, bool) {
raw, ok := lookup(key)
if key == "ZAPRET_AUTOTUNE_SOCKS_PROXY" && ok {
if proxy, e := managedProbeProxy(raw); e == nil {
return strings.Replace(proxy, "socks5h://", "socks5://", 1), true
}
}
return raw, ok
})
if err != nil {
return cfg, err
}
cfg.SOCKSProxy, err = managedProbeProxy(cfg.SOCKSProxy)
return cfg, err
}
+11 -3
View File
@@ -24,7 +24,7 @@ const playerResponseLimit = 1 << 20
var videoIDPattern = regexp.MustCompile(`^[A-Za-z0-9_-]{11}$`)
var discoveryDNSLabel = regexp.MustCompile(`^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$`)
type playerRunner func(context.Context, []string, io.Writer, io.Writer) error
type playerRunner func(context.Context, []string, io.Reader, io.Writer, io.Writer) error
// Keep request diagnostics private while retaining typed causes for supervision.
type playerRequestError struct{ cause error }
@@ -32,8 +32,9 @@ type playerRequestError struct{ cause error }
func (e *playerRequestError) Error() string { return "player request failed or exceeded output limit" }
func (e *playerRequestError) Unwrap() error { return e.cause }
func runPlayerCurl(ctx context.Context, args []string, stdout, stderr io.Writer) error {
func runPlayerCurl(ctx context.Context, args []string, stdin io.Reader, stdout, stderr io.Writer) error {
cmd := exec.CommandContext(ctx, "curl", args...)
cmd.Stdin = stdin
cmd.Stdout, cmd.Stderr = stdout, stderr
return cmd.Run()
}
@@ -148,6 +149,12 @@ func discoverYoutube(args []string, stdout, stderr io.Writer, runner playerRunne
}
func requestPlayer(videoID, proxy string, runner playerRunner) ([]byte, error) {
if _, err := requiredYoutubeAdapter(); err != nil {
return nil, err
}
if !videoIDPattern.MatchString(videoID) {
return nil, errors.New("invalid player video ID")
}
payload, _ := json.Marshal(map[string]interface{}{
"videoId": videoID,
"context": map[string]interface{}{"client": map[string]interface{}{
@@ -161,8 +168,9 @@ func requestPlayer(videoID, proxy string, runner playerRunner) ([]byte, error) {
output := &playerBoundedWriter{dst: &body, limit: playerResponseLimit}
diagnostics := &playerBoundedWriter{dst: io.Discard, limit: 64 << 10}
// Disable curlrc and NO_PROXY bypasses: the explicit proxy must be used.
// Keep the bounded JSON body off argv; curl retains its JSON POST headers.
err := runner(ctx, []string{"--disable", "--silent", "--show-error", "--fail", "--max-time", "30",
"--noproxy", "", "--proxy", proxy, "--user-agent", playerUserAgent, "--json", string(payload), playerURL}, output, diagnostics)
"--noproxy", "", "--proxy", proxy, "--user-agent", playerUserAgent, "--json", "@-", playerURL}, bytes.NewReader(payload), output, diagnostics)
if err != nil || output.exceeded || diagnostics.exceeded {
if output.exceeded || diagnostics.exceeded {
// An output failure remains unsafe even if the child was also canceled.
+6 -6
View File
@@ -20,13 +20,13 @@ func discoveryArgs() []string {
func TestDiscoverYoutubeSuccess(t *testing.T) {
var stdout, stderr bytes.Buffer
called := false
runner := func(ctx context.Context, args []string, out, diagnostics io.Writer) error {
runner := func(ctx context.Context, args []string, stdin io.Reader, out, diagnostics io.Writer) error {
called = true
if _, ok := ctx.Deadline(); !ok {
t.Error("request must have a deadline")
}
prefix := []string{"--disable", "--silent", "--show-error", "--fail", "--max-time", "30", "--noproxy", "", "--proxy", "socks5://localhost:1080", "--user-agent", playerUserAgent, "--json"}
if len(args) != len(prefix)+2 || !reflect.DeepEqual(args[:len(prefix)], prefix) || args[len(args)-1] != playerURL {
if len(args) != len(prefix)+2 || !reflect.DeepEqual(args[:len(prefix)], prefix) || args[len(prefix)] != "@-" || args[len(args)-1] != playerURL {
t.Fatalf("unexpected curl arguments: %q", args)
}
var payload struct {
@@ -42,7 +42,7 @@ func TestDiscoverYoutubeSuccess(t *testing.T) {
} `json:"client"`
} `json:"context"`
}
if err := json.Unmarshal([]byte(args[len(prefix)]), &payload); err != nil {
if err := json.NewDecoder(stdin).Decode(&payload); err != nil {
t.Fatal(err)
}
client := payload.Context.Client
@@ -79,7 +79,7 @@ func TestDiscoverYoutubeInvalidArgs(t *testing.T) {
}
for _, args := range tests {
var stdout, stderr bytes.Buffer
runner := func(context.Context, []string, io.Writer, io.Writer) error {
runner := func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("invalid arguments must not invoke curl")
return nil
}
@@ -114,7 +114,7 @@ func TestPlayerHostnameFailsClosed(t *testing.T) {
}
for _, fixture := range fixtures {
var stdout, stderr bytes.Buffer
runner := func(_ context.Context, _ []string, out, diagnostics io.Writer) error {
runner := func(_ context.Context, _ []string, _ io.Reader, out, diagnostics io.Writer) error {
io.WriteString(diagnostics, "SECRET")
_, err := io.WriteString(out, fixture)
return err
@@ -136,7 +136,7 @@ func TestDiscoverYoutubeRequestFailures(t *testing.T) {
for _, mode := range []string{"runner", "stdout limit", "stderr limit", "write result"} {
t.Run(mode, func(t *testing.T) {
var stdout, stderr bytes.Buffer
runner := func(_ context.Context, _ []string, out, diagnostics io.Writer) error {
runner := func(_ context.Context, _ []string, _ io.Reader, out, diagnostics io.Writer) error {
switch mode {
case "runner":
io.WriteString(out, discoveryFixture)
+676
View File
@@ -0,0 +1,676 @@
package main
import (
"bytes"
"context"
"crypto/tls"
"encoding/binary"
"errors"
"fmt"
"io"
"mime"
"net"
"net/http"
"os"
"os/signal"
"path/filepath"
"strconv"
"sync"
"sync/atomic"
"syscall"
"time"
)
const dnsSocketMark = 0x50000000 // Exclude both deployed (0x40000000) and probe (0x10000000) queues.
const dnsMaxQuery = 4096
const dnsConcurrency = 16
const dnsMaxUDPPayload = 65507 // Fixed IPv4 listener: maximum UDP payload, excluding IP/UDP headers.
var errDNSForwarder = errors.New("DNS forwarder unavailable")
func markDNSSocket(_, _ string, raw syscall.RawConn) error {
return setDNSSocketMark(raw, syscall.SetsockoptInt)
}
func setDNSSocketMark(raw syscall.RawConn, set func(int, int, int, int) error) error {
var markErr error
err := raw.Control(func(fd uintptr) { markErr = set(int(fd), syscall.SOL_SOCKET, syscall.SO_MARK, dnsSocketMark) })
if err != nil {
return err
}
return markErr
}
type dnsDialFunc func(context.Context, string, string) (net.Conn, error)
func dnsBootstrapDial(cfg managedDNSConfig, dial dnsDialFunc) dnsDialFunc {
var next atomic.Uint64
return func(ctx context.Context, network, address string) (net.Conn, error) {
if network != "tcp" || address != net.JoinHostPort(cfg.Host, "443") || len(cfg.BootstrapIPs) == 0 {
return nil, errDNSForwarder
}
ctx, cancelAll := context.WithTimeout(ctx, cfg.Timeout)
defer cancelAll()
start := int((next.Add(1) - 1) % uint64(len(cfg.BootstrapIPs)))
for i := range cfg.BootstrapIPs {
if ctx.Err() != nil {
break
}
// Give every bootstrap a bounded share of the remaining deadline.
remaining := cfg.Timeout
if deadline, ok := ctx.Deadline(); ok {
remaining = time.Until(deadline)
}
attempt, cancel := context.WithTimeout(ctx, remaining/time.Duration(len(cfg.BootstrapIPs)-i))
conn, err := dial(attempt, "tcp", net.JoinHostPort(cfg.BootstrapIPs[(start+i)%len(cfg.BootstrapIPs)].String(), "443"))
cancel()
if err == nil {
return conn, nil
}
}
return nil, errDNSForwarder
}
}
func newDNSHTTPClient(cfg managedDNSConfig, control func(string, string, syscall.RawConn) error) (*http.Client, *http.Transport) {
dialer := &net.Dialer{Timeout: cfg.Timeout, Control: control}
transport := &http.Transport{
Proxy: nil, DialContext: dnsBootstrapDial(cfg, dialer.DialContext),
TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12, ServerName: cfg.Host},
TLSHandshakeTimeout: cfg.Timeout, ResponseHeaderTimeout: cfg.Timeout,
DisableCompression: true, MaxConnsPerHost: dnsConcurrency,
MaxIdleConns: dnsConcurrency, MaxIdleConnsPerHost: dnsConcurrency,
IdleConnTimeout: 30 * time.Second, MaxResponseHeaderBytes: 8192,
}
return &http.Client{Transport: transport, Timeout: cfg.Timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errDNSForwarder }}, transport
}
// DNS names may use compression. Bound both pointer traversal and expanded name size.
func dnsNameEnd(wire []byte, start int) (int, bool) {
end, _, ok := dnsName(wire, start)
return end, ok
}
func dnsName(wire []byte, start int) (int, []byte, bool) {
pos, end, expanded := start, -1, 0
var name []byte
for steps := 0; steps < 256; steps++ {
if pos < 12 || pos >= len(wire) {
return 0, nil, false
}
n := int(wire[pos])
pos++
if n == 0 {
if end < 0 {
end = pos
}
return end, append(name, 0), true
}
if n&0xc0 == 0xc0 {
if pos >= len(wire) {
return 0, nil, false
}
target := (n&0x3f)<<8 | int(wire[pos])
if target < 12 || target >= len(wire) {
return 0, nil, false
}
if end < 0 {
end = pos + 1
}
pos = target
continue
}
if n > 63 || pos+n >= len(wire) {
return 0, nil, false
}
expanded += n + 1
if expanded > 254 {
return 0, nil, false
}
name = append(name, byte(n))
name = append(name, wire[pos:pos+n]...)
pos += n
}
return 0, nil, false
}
func dnsQuestionEnd(wire []byte) (int, bool) {
end, _, ok := dnsQuestion(wire)
return end, ok
}
func dnsQuestion(wire []byte) (int, []byte, bool) {
if len(wire) < 12 || binary.BigEndian.Uint16(wire[4:6]) != 1 {
return 0, nil, false
}
end, name, ok := dnsName(wire, 12)
if !ok || end+4 > len(wire) {
return 0, nil, false
}
if binary.BigEndian.Uint16(wire[end:end+2]) == 0 || binary.BigEndian.Uint16(wire[end+2:end+4]) == 0 {
return 0, nil, false
}
return end + 4, append(name, wire[end:end+4]...), true
}
func validDNSQuery(wire []byte) bool {
if len(wire) < 12 || len(wire) > dnsMaxQuery || wire[2]&0xfe != 0 || wire[3]&0xcf != 0 || binary.BigEndian.Uint16(wire[6:8]) != 0 || binary.BigEndian.Uint16(wire[8:10]) != 0 {
return false
}
end, ok := dnsQuestionEnd(wire)
if !ok {
return false
}
_, ok = dnsQueryUDPLimit(wire, end)
return ok
}
// Query additional records have the same framing as answers. Only OPT affects
// downstream UDP sizing; its root owner and option framing must be valid.
func dnsQueryUDPLimit(wire []byte, end int) (int, bool) {
limit, seenOPT := 512, false
for count := int(binary.BigEndian.Uint16(wire[10:12])); count > 0; count-- {
header, name, ok := dnsName(wire, end)
if !ok || header+10 > len(wire) {
return 0, false
}
end = header + 10 + int(binary.BigEndian.Uint16(wire[header+8:header+10]))
if end > len(wire) {
return 0, false
}
if binary.BigEndian.Uint16(wire[header:header+2]) != 41 {
continue
}
if seenOPT || len(name) != 1 || wire[header+4] != 0 || wire[header+5] != 0 || binary.BigEndian.Uint16(wire[header+6:header+8])&0x7fff != 0 {
return 0, false
}
seenOPT = true
limit = int(binary.BigEndian.Uint16(wire[header+2 : header+4]))
if limit < 512 {
limit = 512
}
if limit > dnsMaxUDPPayload {
limit = dnsMaxUDPPayload
}
for option := header + 10; option < end; {
if option+4 > end {
return 0, false
}
option += 4 + int(binary.BigEndian.Uint16(wire[option+2:option+4]))
if option > end {
return 0, false
}
}
}
return limit, end == len(wire)
}
// Validate only record boundaries (including EDNS/DNSSEC), never decode or rewrite RDATA.
func validDNSRecordFraming(wire []byte, end, count int) bool {
for ; count > 0; count-- {
var ok bool
end, ok = dnsNameEnd(wire, end)
if !ok || end+10 > len(wire) {
return false
}
end += 10 + int(binary.BigEndian.Uint16(wire[end+8:end+10]))
if end > len(wire) {
return false
}
}
return end == len(wire)
}
func validDNSResponse(query, response []byte) bool {
if len(query) < 12 || len(response) < 12 || len(response) > 65535 || response[2]&0x80 == 0 || response[2]&0x78 != query[2]&0x78 || !bytes.Equal(query[:2], response[:2]) || response[3]&0x40 != 0 {
return false
}
_, question, qok := dnsQuestion(query)
re, responseQuestion, rok := dnsQuestion(response)
if !qok || !rok || !bytes.Equal(question, responseQuestion) {
return false
}
count := int(binary.BigEndian.Uint16(response[6:8])) + int(binary.BigEndian.Uint16(response[8:10])) + int(binary.BigEndian.Uint16(response[10:12]))
return validDNSRecordFraming(response, re, count)
}
func dnsServfail(query []byte) []byte {
if !validDNSQuery(query) {
return nil
}
reply := dnsQuestionReply(query, query)
reply[2] = 0x80 | query[2]&0x79 // QR, opcode and RD, not AA/TC.
reply[3] = 0x02 // SERVFAIL, no fabricated recursion/DNSSEC assertions.
return reply
}
// Re-encode the question without pointers into records omitted from this reply.
func dnsQuestionReply(query, header []byte) []byte {
_, question, ok := dnsQuestion(query)
if !ok || len(header) < 12 {
return nil
}
reply := append([]byte(nil), header[:12]...)
clear(reply[6:12])
return append(reply, question...)
}
func dnsUDPReply(query, response []byte) []byte {
if response == nil {
return nil
}
end, ok := dnsQuestionEnd(query)
if !ok {
return nil
}
limit, ok := dnsQueryUDPLimit(query, end)
if !ok {
return nil
}
if len(response) <= limit {
return response
}
reply := dnsQuestionReply(query, response)
if reply != nil {
reply[2] |= 0x82 // QR and TC; preserve upstream opcode, RD, AA and other flags.
reply[3] &^= 0x40 // Reserved Z bit; no records/assertions are invented.
}
return reply
}
func writeDNSDatagram(udp net.PacketConn, addr net.Addr, query, response []byte) {
if reply := dnsUDPReply(query, response); reply != nil {
if _, err := udp.WriteTo(reply, addr); err != nil {
return
} // Drop only this datagram; never retry plaintext or stop the server.
}
}
type dnsForwarder struct {
cfg managedDNSConfig
client *http.Client
slots chan struct{}
}
func (f *dnsForwarder) exchange(ctx context.Context, query []byte) []byte {
if !validDNSQuery(query) {
return nil
}
ctx, cancel := context.WithTimeout(ctx, f.cfg.Timeout)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodPost, f.cfg.URL, bytes.NewReader(query))
if err != nil {
return dnsServfail(query)
}
req.Header.Set("Content-Type", "application/dns-message")
req.Header.Set("Accept", "application/dns-message")
resp, err := f.client.Do(req)
if err != nil {
return dnsServfail(query)
}
defer resp.Body.Close()
typ, _, err := mime.ParseMediaType(resp.Header.Get("Content-Type"))
if err != nil || typ != "application/dns-message" || resp.StatusCode != http.StatusOK || resp.Header.Get("Content-Encoding") != "" {
return dnsServfail(query)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, 65536))
if err != nil || resp.ContentLength >= 0 && resp.ContentLength != int64(len(body)) || !validDNSResponse(query, body) {
return dnsServfail(query)
}
return body
}
// One connection owns one request slot; frames are sequential, capped in number
// and total lifetime, so slow/idle clients cannot create unbounded workers.
func (f *dnsForwarder) tcpWorker(ctx context.Context, conn net.Conn) {
defer conn.Close()
stop := context.AfterFunc(ctx, func() { conn.Close() })
defer stop()
lifetime := time.Now().Add(16 * f.cfg.Timeout)
for frame := 0; frame < 16; frame++ {
deadline := time.Now().Add(f.cfg.Timeout)
if lifetime.Before(deadline) {
deadline = lifetime
}
if err := conn.SetReadDeadline(deadline); err != nil {
return
}
var size [2]byte
if _, err := io.ReadFull(conn, size[:]); err != nil {
return
}
n := int(binary.BigEndian.Uint16(size[:]))
if n < 12 || n > dnsMaxQuery {
return
}
query := make([]byte, n)
if _, err := io.ReadFull(conn, query); err != nil {
return
}
if !time.Now().Before(lifetime) {
return
}
requestCtx, cancel := context.WithDeadline(ctx, lifetime)
reply := f.exchange(requestCtx, query)
cancel()
if reply == nil || !time.Now().Before(lifetime) {
return
}
// An upstream timeout must still leave a bounded window to send SERVFAIL.
deadline = time.Now().Add(f.cfg.Timeout)
if lifetime.Before(deadline) {
deadline = lifetime
}
if err := conn.SetWriteDeadline(deadline); err != nil {
return
}
binary.BigEndian.PutUint16(size[:], uint16(len(reply)))
if err := writeDNSFrame(conn, size[:], reply); err != nil {
return
}
}
}
func writeDNSFrame(w io.Writer, size, reply []byte) error {
for _, part := range [][]byte{size, reply} {
for len(part) > 0 {
n, err := w.Write(part)
if err != nil {
return err
}
if n == 0 {
return io.ErrShortWrite
}
part = part[n:]
}
}
return nil
}
func (f *dnsForwarder) run(ctx context.Context, udp net.PacketConn, tcp net.Listener, ready func() (func(), error)) error {
ctx, cancel := context.WithCancel(ctx)
defer cancel()
stop := context.AfterFunc(ctx, func() { udp.Close(); tcp.Close() })
defer stop()
defer udp.Close()
defer tcp.Close()
cleanup, err := ready()
if err != nil {
return errDNSForwarder
}
defer cleanup()
var workers sync.WaitGroup
var loops sync.WaitGroup
var failed atomic.Bool
loops.Add(2)
go func() {
defer loops.Done()
buffer := make([]byte, dnsMaxQuery+1)
for {
n, addr, err := udp.ReadFrom(buffer)
if err != nil {
if ctx.Err() == nil {
failed.Store(true)
cancel()
}
return
}
if !validDNSQuery(buffer[:n]) {
continue
}
select {
case f.slots <- struct{}{}:
query := append([]byte(nil), buffer[:n]...)
workers.Add(1)
go func() {
defer workers.Done()
defer func() { <-f.slots }()
if reply := f.exchange(ctx, query); reply != nil {
writeDNSDatagram(udp, addr, query, reply)
}
}()
default:
writeDNSDatagram(udp, addr, buffer[:n], dnsServfail(buffer[:n]))
}
}
}()
go func() {
defer loops.Done()
for {
conn, err := tcp.Accept()
if err != nil {
if ctx.Err() == nil {
failed.Store(true)
cancel()
}
return
}
select {
case f.slots <- struct{}{}:
workers.Add(1)
go func() { defer workers.Done(); defer func() { <-f.slots }(); f.tcpWorker(ctx, conn) }()
default:
conn.Close()
}
}
}()
loops.Wait()
workers.Wait()
if failed.Load() {
return errDNSForwarder
}
return nil
}
func createDNSReady(path string) (func(), error) {
dir := filepath.Dir(path)
if err := managedDirectory(dir); err != nil {
return nil, errDNSForwarder
}
info, err := os.Lstat(dir)
if err != nil || info.Mode().Perm() != 0700 || info.Mode()&(os.ModeSetuid|os.ModeSetgid|os.ModeSticky) != 0 {
return nil, errDNSForwarder
}
fd, err := syscall.Open(dir, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, errDNSForwarder
}
parent := os.NewFile(uintptr(fd), dir)
opened, err := parent.Stat()
if err != nil || !os.SameFile(info, opened) {
parent.Close()
return nil, errDNSForwarder
}
heldDir := fmt.Sprintf("/proc/self/fd/%d", fd)
file, err := os.CreateTemp(heldDir, ".dns-ready-")
if err != nil {
parent.Close()
return nil, errDNSForwarder
}
owned, statErr := file.Stat()
if statErr == nil && owned.Mode().Perm() != 0600 {
statErr = errDNSForwarder
}
var once sync.Once
cleanup := func() {
once.Do(func() {
// Inspect and unlink relative to the held private parent, not a replacement directory.
for _, name := range []string{filepath.Base(path), filepath.Base(file.Name())} {
current, e := os.Lstat(filepath.Join(heldDir, name))
if e == nil && owned != nil && os.SameFile(owned, current) {
syscall.Unlinkat(fd, name)
}
}
parent.Close()
// Hold the inode until all identity checks/unlinks finish, preventing reuse.
file.Close()
})
}
if statErr == nil {
err = publishDNSReady(file, filepath.Join(heldDir, filepath.Base(path)))
}
if statErr != nil || err != nil {
cleanup()
return nil, errDNSForwarder
}
if err := syscall.Unlinkat(fd, filepath.Base(file.Name())); err != nil {
cleanup()
return nil, errDNSForwarder
}
return cleanup, nil
}
func publishDNSReady(file *os.File, destination string) error {
if _, err := fmt.Fprintf(file, "%d\n", os.Getpid()); err != nil {
return err
}
if err := file.Sync(); err != nil {
return err
}
return os.Link(file.Name(), destination) // Atomic complete-PID publication, EXCL even for a symlink destination.
}
func serveDNSForwarder(ctx context.Context, cfg managedDNSConfig) error {
udp, err := net.ListenPacket("udp4", dnsListenAddress)
if err != nil {
return errDNSForwarder
}
tcp, err := net.Listen("tcp4", dnsListenAddress)
if err != nil {
udp.Close()
return errDNSForwarder
}
client, transport := newDNSHTTPClient(cfg, markDNSSocket)
defer transport.CloseIdleConnections()
f := &dnsForwarder{cfg: cfg, client: client, slots: make(chan struct{}, dnsConcurrency)}
return f.run(ctx, udp, tcp, func() (func(), error) { return createDNSReady(dnsReadyPath) })
}
func dnsForwarderCLI(args []string, stdout, stderr io.Writer) int {
if len(args) > 0 && (args[0] == "readiness-check" || args[0] == "readiness-cleanup") {
return dnsReadinessCLI(args, stderr)
}
if len(args) != 1 || (args[0] != "mode" && args[0] != "address" && args[0] != "serve") {
fmt.Fprintln(stderr, "configerror: invalid DNS forwarder command")
return 64
}
cfg, err := parseManagedDNSConfig(os.LookupEnv)
if err != nil || args[0] == "serve" && cfg.Mode != "doh" {
fmt.Fprintln(stderr, "configerror: invalid managed DNS configuration")
return 64
}
switch args[0] {
case "mode":
_, err = fmt.Fprintln(stdout, cfg.Mode)
case "address":
_, err = fmt.Fprintln(stdout, cfg.serverAddress())
case "serve":
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()
err = serveDNSForwarder(ctx, cfg)
}
if err != nil {
fmt.Fprintln(stderr, "error: DNS forwarder unavailable")
return 75
}
return 0
}
// PID 1 holds FD 8 (private directory) and FD 9 (readiness inode) across the
// child's lifetime, including SIGKILL. Helpers borrow duplicates, never offsets.
func duplicateDNSAnchor(fd int) (*os.File, error) {
// Keep duplicates above both reserved descriptors; a missing anchor must
// never be filled by duplicating the other one.
duplicate, _, err := syscall.Syscall(syscall.SYS_FCNTL, uintptr(fd), syscall.F_DUPFD_CLOEXEC, 10)
if err != 0 {
return nil, errDNSForwarder
}
return os.NewFile(duplicate, "DNS readiness anchor"), nil
}
func dnsReadinessCLI(args []string, stderr io.Writer) int {
if len(args) != 2 {
fmt.Fprintln(stderr, "configerror: invalid DNS readiness command")
return 64
}
pid, err := strconv.ParseInt(args[1], 10, 32)
if err != nil || pid <= 1 || strconv.FormatInt(pid, 10) != args[1] {
fmt.Fprintln(stderr, "configerror: invalid DNS readiness command")
return 64
}
cfg, err := parseManagedDNSConfig(os.LookupEnv)
if err != nil || cfg.Mode != "doh" {
fmt.Fprintln(stderr, "configerror: invalid managed DNS configuration")
return 64
}
anchor, _ := duplicateDNSAnchor(9)
if anchor != nil {
defer anchor.Close()
}
parent, err := duplicateDNSAnchor(8)
if err == nil {
defer parent.Close()
err = dnsReadinessAction(parent, anchor, int(pid), args[0] == "readiness-cleanup")
}
if err != nil {
fmt.Fprintln(stderr, "error: DNS readiness ownership unavailable")
return 78
}
return 0
}
func dnsOwnedAnchor(info os.FileInfo, directory bool) bool {
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok || stat.Uid != uint32(os.Geteuid()) || info.Mode()&(os.ModeSetuid|os.ModeSetgid|os.ModeSticky) != 0 {
return false
}
if directory {
return info.IsDir() && info.Mode().Perm() == 0700
}
return info.Mode().IsRegular() && info.Mode().Perm() == 0600
}
func dnsReadinessAction(parent, anchor *os.File, expectedPID int, cleanup bool) error {
if parent == nil || expectedPID <= 1 || int64(expectedPID) > 2147483647 {
return errDNSForwarder
}
info, err := parent.Stat()
if err != nil || !dnsOwnedAnchor(info, true) {
return errDNSForwarder
}
name := filepath.Base(dnsReadyPath)
fd, err := syscall.Openat(int(parent.Fd()), name, syscall.O_RDONLY|syscall.O_NONBLOCK|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
if err != nil {
if cleanup && errors.Is(err, syscall.ENOENT) {
return nil
}
return errDNSForwarder
}
current := os.NewFile(uintptr(fd), "DNS readiness entry")
defer current.Close()
if anchor == nil {
return errDNSForwarder
}
owned, err := anchor.Stat()
if err != nil || !dnsOwnedAnchor(owned, false) {
return errDNSForwarder
}
entry, err := current.Stat()
if err != nil || !dnsOwnedAnchor(entry, false) || !os.SameFile(owned, entry) {
return errDNSForwarder
}
expected := strconv.Itoa(expectedPID) + "\n"
if owned.Size() != int64(len(expected)) {
return errDNSForwarder
}
buffer := make([]byte, len(expected))
if _, err := anchor.ReadAt(buffer, 0); err != nil || string(buffer) != expected {
return errDNSForwarder
}
if cleanup {
if err := syscall.Unlinkat(int(parent.Fd()), name); err != nil && !errors.Is(err, syscall.ENOENT) {
return errDNSForwarder
}
}
return nil
}
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"errors"
"net"
"net/url"
"strconv"
"strings"
"time"
)
const defaultDoHURL = "https://dns.google/dns-query"
const dnsListenAddress = "127.0.0.1:5353"
const dnsReadyPath = "/run/zapret-autotune-ss/dns-ready"
var errDNSConfig = errors.New("invalid managed DNS configuration")
type managedDNSConfig struct {
Mode string
URL string
Host string
BootstrapIPs []net.IP
Timeout time.Duration
NativeServer string
}
func parseManagedDNSConfig(lookup func(string) (string, bool)) (managedDNSConfig, error) {
cfg := managedDNSConfig{Mode: "doh", URL: defaultDoHURL, Timeout: 5 * time.Second}
if v, ok := lookup("ZAPRET_AUTOTUNE_DNS_MODE"); ok {
cfg.Mode = v
}
if cfg.Mode == "native" {
cfg.NativeServer, _ = lookup("SS_DNS_SERVER")
if cfg.NativeServer != "" && net.ParseIP(cfg.NativeServer) == nil {
return cfg, errDNSConfig
}
return cfg, nil
}
if cfg.Mode != "doh" {
return cfg, errDNSConfig
}
if v, ok := lookup("ZAPRET_AUTOTUNE_DOH_URL"); ok {
cfg.URL = v
}
u, err := url.Parse(cfg.URL)
if err != nil || u.Scheme != "https" || u.Opaque != "" || u.User != nil || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || strings.Contains(cfg.URL, "#") || u.Host == "" || u.Path == "" || !strings.HasPrefix(u.Path, "/") || (u.Port() != "" && u.Port() != "443") {
return cfg, errDNSConfig
}
cfg.Host = u.Hostname()
// Do not accept opaque/control-bearing authorities, zone IDs or ambiguous ports.
if cfg.Host == "" || strings.ContainsAny(cfg.URL, "\x00\r\n\t ") || strings.Contains(cfg.Host, "%") {
return cfg, errDNSConfig
}
if net.ParseIP(cfg.Host) == nil {
for _, label := range strings.Split(cfg.Host, ".") {
if len(label) == 0 || len(label) > 63 || label[0] == '-' || label[len(label)-1] == '-' {
return cfg, errDNSConfig
}
for _, c := range label {
if !(c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '-') {
return cfg, errDNSConfig
}
}
}
if len(cfg.Host) > 253 {
return cfg, errDNSConfig
}
}
if u.Host != net.JoinHostPort(cfg.Host, "443") && u.Host != cfg.Host && u.Host != "["+cfg.Host+"]" {
return cfg, errDNSConfig
}
bootstrap, explicit := lookup("ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS")
if !explicit || bootstrap == "" {
if cfg.URL != defaultDoHURL {
return cfg, errDNSConfig
}
bootstrap = "8.8.8.8,8.8.4.4"
}
parts := strings.Split(bootstrap, ",")
if len(parts) > 4 {
return cfg, errDNSConfig
}
for _, part := range parts {
ip := net.ParseIP(part)
if ip == nil {
return cfg, errDNSConfig
}
cfg.BootstrapIPs = append(cfg.BootstrapIPs, ip)
}
if v, ok := lookup("ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC"); ok {
seconds, err := strconv.Atoi(v)
if err != nil || seconds < 1 || seconds > 30 || strconv.Itoa(seconds) != v {
return cfg, errDNSConfig
}
cfg.Timeout = time.Duration(seconds) * time.Second
}
return cfg, nil
}
func (c managedDNSConfig) serverAddress() string {
if c.Mode == "doh" {
return dnsListenAddress
}
return c.NativeServer
}
@@ -0,0 +1,91 @@
package main
import (
"bytes"
"strings"
"testing"
"time"
)
func dnsConfigForTest(t *testing.T, env map[string]string) managedDNSConfig {
t.Helper()
cfg, err := parseManagedDNSConfig(func(k string) (string, bool) { v, ok := env[k]; return v, ok })
if err != nil {
t.Fatal(err)
}
return cfg
}
func TestDNSConfigDefaultsAndNative(t *testing.T) {
cfg := dnsConfigForTest(t, map[string]string{"SS_DNS_SERVER": "ignored\nsecret"})
if cfg.Mode != "doh" || cfg.URL != defaultDoHURL || cfg.Host != "dns.google" || cfg.Timeout != 5*time.Second || cfg.serverAddress() != dnsListenAddress || len(cfg.BootstrapIPs) != 2 {
t.Fatal(cfg)
}
emptyBootstrap := dnsConfigForTest(t, map[string]string{"ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS": ""})
if len(emptyBootstrap.BootstrapIPs) != 2 || emptyBootstrap.BootstrapIPs[0].String() != "8.8.8.8" || emptyBootstrap.BootstrapIPs[1].String() != "8.8.4.4" {
t.Fatal("empty default bootstrap did not use default IPs")
}
for _, ip := range []string{"", "8.8.8.8", "::1"} {
cfg = dnsConfigForTest(t, map[string]string{"ZAPRET_AUTOTUNE_DNS_MODE": "native", "SS_DNS_SERVER": ip, "ZAPRET_AUTOTUNE_DOH_URL": "ignored"})
if cfg.serverAddress() != ip {
t.Fatal(cfg)
}
}
cfg = dnsConfigForTest(t, map[string]string{"ZAPRET_AUTOTUNE_DOH_URL": "https://resolver.example:443/dns-query", "ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS": "1.1.1.1,2001:db8::1", "ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC": "30"})
if cfg.Host != "resolver.example" || cfg.Timeout != 30*time.Second {
t.Fatal(cfg)
}
}
func TestDNSConfigRejectsOpaqueInputs(t *testing.T) {
cases := []map[string]string{
{"ZAPRET_AUTOTUNE_DNS_MODE": ""}, {"ZAPRET_AUTOTUNE_DNS_MODE": "auto"},
{"ZAPRET_AUTOTUNE_DNS_MODE": "native", "SS_DNS_SERVER": "dns.google"},
{"ZAPRET_AUTOTUNE_DNS_MODE": "native", "SS_DNS_SERVER": "8.8.8.8:53"},
{"ZAPRET_AUTOTUNE_DOH_URL": "https://other.example/dns-query"},
{"ZAPRET_AUTOTUNE_DOH_URL": "https://other.example/dns-query", "ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS": ""},
{"ZAPRET_AUTOTUNE_DOH_URL": "https://dns.google/custom-path", "ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS": ""},
}
for _, raw := range []string{"", "http://dns.google/dns-query", "https://token@dns.google/dns-query", "https://dns.google/dns-query?token=secret", "https://dns.google/dns-query#", "https://dns.google:5353/dns-query", "https://dns.google:/dns-query", "https://dns.google", "https://dns.google/\nsecret", "https://dns_google/dns-query"} {
cases = append(cases, map[string]string{"ZAPRET_AUTOTUNE_DOH_URL": raw, "ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS": "8.8.8.8"})
}
for _, raw := range []string{"dns.google", "1.1.1.1,", " 1.1.1.1", "1.1.1.1:443", "[::1]", "fe80::1%eth0", "1.1.1.1,1.1.1.1,1.1.1.1,1.1.1.1,1.1.1.1"} {
cases = append(cases, map[string]string{"ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS": raw})
}
for _, raw := range []string{"", "0", "31", "-1", "05", "1\n"} {
cases = append(cases, map[string]string{"ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC": raw})
}
for i, env := range cases {
_, err := parseManagedDNSConfig(func(k string) (string, bool) { v, ok := env[k]; return v, ok })
if err != errDNSConfig {
t.Errorf("case %d: %v", i, err)
}
}
}
func TestDNSForwarderCLI(t *testing.T) {
t.Setenv("ZAPRET_AUTOTUNE_DNS_MODE", "doh")
t.Setenv("ZAPRET_AUTOTUNE_DOH_URL", defaultDoHURL)
t.Setenv("ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS", "8.8.8.8")
t.Setenv("ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC", "5")
t.Setenv("SS_DNS_SERVER", "secret-invalid-legacy")
for cmd, want := range map[string]string{"mode": "doh\n", "address": dnsListenAddress + "\n"} {
var out, diagnostic bytes.Buffer
if code := run([]string{"dns-forwarder", cmd}, &out, &diagnostic); code != 0 || out.String() != want || diagnostic.Len() != 0 {
t.Fatal(code, out.String(), diagnostic.String())
}
}
t.Setenv("ZAPRET_AUTOTUNE_DNS_MODE", "native")
t.Setenv("SS_DNS_SERVER", "")
var out, diagnostic bytes.Buffer
if code := run([]string{"dns-forwarder", "address"}, &out, &diagnostic); code != 0 || out.String() != "\n" {
t.Fatal(code, out.String())
}
for _, args := range [][]string{{"serve"}, {"wrong"}, {"mode", "extra"}} {
out.Reset()
diagnostic.Reset()
if code := dnsForwarderCLI(args, &out, &diagnostic); code != 64 || out.Len() != 0 || !strings.HasPrefix(diagnostic.String(), "configerror:") {
t.Fatal(code, out.String(), diagnostic.String())
}
}
}
@@ -0,0 +1,295 @@
package main
import (
"bufio"
"bytes"
"context"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
)
func dnsReadyAnchorsForTest(t *testing.T) (string, *os.File, *os.File, func()) {
t.Helper()
dir := t.TempDir()
if err := os.Chmod(dir, 0700); err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "dns-ready")
cleanup, err := createDNSReady(path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(cleanup)
parent, err := os.Open(dir)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { parent.Close() })
anchor, err := os.Open(path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { anchor.Close() })
return path, parent, anchor, cleanup
}
func TestDNSReadinessAnchorActions(t *testing.T) {
path, parent, anchor, childCleanup := dnsReadyAnchorsForTest(t)
pid := os.Getpid()
if _, err := anchor.Seek(1, io.SeekStart); err != nil {
t.Fatal(err)
}
if err := dnsReadinessAction(parent, anchor, pid, false); err != nil {
t.Fatal(err)
}
if offset, _ := anchor.Seek(0, io.SeekCurrent); offset != 1 {
t.Fatal("shared anchor offset changed", offset)
}
if err := dnsReadinessAction(parent, anchor, pid+1, false); err == nil {
t.Fatal("wrong PID accepted")
}
if err := dnsReadinessAction(parent, nil, pid, false); err == nil {
t.Fatal("missing file anchor accepted")
}
if err := dnsReadinessAction(parent, nil, pid, true); err == nil {
t.Fatal("unanchored entry removed")
}
if _, err := os.Stat(path); err != nil {
t.Fatal("unanchored entry not preserved", err)
}
if err := dnsReadinessAction(parent, anchor, pid, true); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Fatal("owned entry not removed", err)
}
if err := dnsReadinessAction(parent, nil, pid, true); err != nil {
t.Fatal("missing entry cleanup not idempotent", err)
}
if err := dnsReadinessAction(parent, anchor, pid, false); err == nil {
t.Fatal("unlinked entry reported ready")
}
childCleanup() // Graceful child cleanup after parent cleanup also remains safe.
}
func TestDNSReadinessForeignReplacement(t *testing.T) {
for _, kind := range []string{"same-PID-copy", "symlink", "directory", "FIFO"} {
t.Run(kind, func(t *testing.T) {
path, parent, anchor, childCleanup := dnsReadyAnchorsForTest(t)
os.Remove(path)
switch kind {
case "same-PID-copy":
if err := os.WriteFile(path, []byte(fmt.Sprintf("%d\n", os.Getpid())), 0600); err != nil {
t.Fatal(err)
}
case "symlink":
if err := os.Symlink("missing", path); err != nil {
t.Fatal(err)
}
case "directory":
if err := os.Mkdir(path, 0700); err != nil {
t.Fatal(err)
}
case "FIFO":
if err := syscall.Mkfifo(path, 0600); err != nil {
t.Fatal(err)
}
}
foreign, err := os.Lstat(path)
if err != nil {
t.Fatal(err)
}
for _, cleanup := range []bool{false, true} {
if err := dnsReadinessAction(parent, anchor, os.Getpid(), cleanup); err == nil {
t.Fatal("foreign entry accepted", cleanup)
}
}
childCleanup()
current, err := os.Lstat(path)
if err != nil || !os.SameFile(foreign, current) {
t.Fatal("foreign entry deleted/replaced", err)
}
})
}
}
func TestDNSReadinessRejectsUnsafeAnchors(t *testing.T) {
path, parent, anchor, _ := dnsReadyAnchorsForTest(t)
pid := os.Getpid()
if err := dnsReadinessAction(nil, anchor, pid, true); err == nil {
t.Fatal("missing directory anchor accepted")
}
if err := dnsReadinessAction(anchor, anchor, pid, true); err == nil {
t.Fatal("regular file used as directory anchor")
}
for _, body := range []string{"", strconv.Itoa(pid), strconv.Itoa(pid) + "\nextra", "secret-content\n", strings.Repeat("x", 1024)} {
if err := os.WriteFile(path, []byte(body), 0600); err != nil {
t.Fatal(err)
}
if err := dnsReadinessAction(parent, anchor, pid, true); err == nil {
t.Fatal("malformed PID body accepted")
}
}
os.WriteFile(path, []byte(fmt.Sprintf("%d\n", pid)), 0600)
os.Chmod(path, 0644)
if err := dnsReadinessAction(parent, anchor, pid, true); err == nil {
t.Fatal("public file accepted")
}
os.Chmod(path, 0600)
os.Chmod(filepath.Dir(path), 0755)
if err := dnsReadinessAction(parent, anchor, pid, true); err == nil {
t.Fatal("public directory accepted")
}
os.Chmod(filepath.Dir(path), 0700)
}
// Subprocess helper exercises the actual inherited FD 8/9 CLI contract.
func TestDNSReadinessCLIHelper(t *testing.T) {
if os.Getenv("DNS_TEST_READINESS_CLI") != "1" {
return
}
for i, arg := range os.Args {
if arg == "--" {
os.Exit(run(os.Args[i+1:], os.Stdout, os.Stderr))
}
}
os.Exit(99)
}
func dnsReadinessSubprocess(t *testing.T, parent, anchor *os.File, command, pid string) (int, string) {
t.Helper()
binary, err := os.Executable()
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, binary, "-test.run=^TestDNSReadinessCLIHelper$", "--", "dns-forwarder", command, pid)
cmd.Env = append(os.Environ(), "DNS_TEST_READINESS_CLI=1", "ZAPRET_AUTOTUNE_DNS_MODE=doh", "ZAPRET_AUTOTUNE_DOH_URL="+defaultDoHURL, "ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS=", "ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC=5")
padding, err := os.Open(os.DevNull)
if err != nil {
t.Fatal(err)
}
defer padding.Close()
cmd.ExtraFiles = []*os.File{padding, padding, padding, padding, padding, parent}
if anchor != nil {
cmd.ExtraFiles = append(cmd.ExtraFiles, anchor)
}
output, err := cmd.CombinedOutput()
if ctx.Err() != nil {
t.Fatal("readiness helper timed out")
}
if err == nil {
return 0, string(output)
}
if exit, ok := err.(*exec.ExitError); ok {
return exit.ExitCode(), string(output)
}
t.Fatal(err)
return -1, ""
}
func TestDNSReadinessInheritedCLI(t *testing.T) {
path, parent, anchor, _ := dnsReadyAnchorsForTest(t)
pid := strconv.Itoa(os.Getpid())
for _, bad := range []string{"0", "1", "-2", "+2", "02", "2\n", "2147483648", "secret-token"} {
code, out := dnsReadinessSubprocess(t, parent, anchor, "readiness-check", bad)
if code != 64 || out != "configerror: invalid DNS readiness command\n" {
t.Fatal(code, out)
}
}
if code, out := dnsReadinessSubprocess(t, parent, anchor, "readiness-check", pid); code != 0 || out != "" {
t.Fatal(code, out)
}
if code, out := dnsReadinessSubprocess(t, nil, anchor, "readiness-cleanup", pid); code != 78 || out != "error: DNS readiness ownership unavailable\n" {
t.Fatal("missing directory anchor accepted", code, out)
}
if offset, _ := anchor.Seek(0, io.SeekCurrent); offset != 0 {
t.Fatal("CLI changed parent's offset", offset)
}
if code, out := dnsReadinessSubprocess(t, parent, nil, "readiness-cleanup", pid); code != 78 || out != "error: DNS readiness ownership unavailable\n" {
t.Fatal(code, out)
}
if code, out := dnsReadinessSubprocess(t, parent, anchor, "readiness-cleanup", pid); code != 0 || out != "" {
t.Fatal(code, out)
}
if _, err := os.Stat(path); !os.IsNotExist(err) {
t.Fatal(err)
}
if code, out := dnsReadinessSubprocess(t, parent, nil, "readiness-cleanup", pid); code != 0 || out != "" {
t.Fatal(code, out)
}
}
func TestDNSReadinessPublisherHelper(t *testing.T) {
dir := os.Getenv("DNS_TEST_READINESS_PUBLISH_DIR")
if dir == "" {
return
}
cleanup, err := createDNSReady(filepath.Join(dir, "dns-ready"))
if err != nil {
os.Exit(99)
}
defer cleanup()
fmt.Fprintln(os.Stdout, "published")
time.Sleep(time.Hour)
}
func TestDNSReadinessAnchorsSurviveSIGKILL(t *testing.T) {
dir := t.TempDir()
os.Chmod(dir, 0700)
binary, err := os.Executable()
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, binary, "-test.run=^TestDNSReadinessPublisherHelper$")
cmd.Env = append(os.Environ(), "DNS_TEST_READINESS_PUBLISH_DIR="+dir)
stdout, err := cmd.StdoutPipe()
if err != nil {
t.Fatal(err)
}
var diagnostic bytes.Buffer
cmd.Stderr = &diagnostic
if err := cmd.Start(); err != nil {
t.Fatal(err)
}
defer cmd.Process.Kill()
if line, err := bufio.NewReader(stdout).ReadString('\n'); err != nil || line != "published\n" {
t.Fatal("publisher failed", line, err)
}
parent, err := os.Open(dir)
if err != nil {
t.Fatal(err)
}
defer parent.Close()
path := filepath.Join(dir, "dns-ready")
anchor, err := os.Open(path)
if err != nil {
t.Fatal(err)
}
defer anchor.Close()
pid := cmd.Process.Pid
if err := dnsReadinessAction(parent, anchor, pid, false); err != nil {
t.Fatal(err)
}
if err := cmd.Process.Kill(); err != nil {
t.Fatal(err)
}
cmd.Wait()
if err := dnsReadinessAction(parent, anchor, pid, true); err != nil {
t.Fatal("anchored SIGKILL cleanup failed", err)
}
if _, err := os.Lstat(path); !os.IsNotExist(err) {
t.Fatal("SIGKILL left owned readiness entry", err)
}
}
@@ -0,0 +1,142 @@
package main
import (
"bytes"
"context"
"encoding/binary"
"io"
"net"
"net/http"
"sync/atomic"
"testing"
"time"
)
func TestDNSTCPUpstreamTimeoutStillWritesServfail(t *testing.T) {
for _, slowRead := range []bool{false, true} {
name := "upstream-timeout"
if slowRead {
name = "partial-frame-then-upstream-timeout"
}
t.Run(name, func(t *testing.T) {
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
<-r.Context().Done()
return nil, r.Context().Err()
})
f.cfg.Timeout = 80 * time.Millisecond
server, client := net.Pipe()
defer client.Close()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan struct{})
go func() { defer close(done); f.tcpWorker(ctx, server) }()
client.SetDeadline(time.Now().Add(time.Second))
query := testDNSQuery()
var size [2]byte
binary.BigEndian.PutUint16(size[:], uint16(len(query)))
if _, err := client.Write(size[:]); err != nil {
t.Fatal(err)
}
if slowRead {
if _, err := client.Write(query[:1]); err != nil {
t.Fatal(err)
}
time.Sleep(20 * time.Millisecond)
query = query[1:]
}
if _, err := client.Write(query); err != nil {
t.Fatal(err)
}
if _, err := io.ReadFull(client, size[:]); err != nil {
t.Fatal("upstream timeout closed TCP instead of writing SERVFAIL", err)
}
reply := make([]byte, int(binary.BigEndian.Uint16(size[:])))
if _, err := io.ReadFull(client, reply); err != nil || !bytes.Equal(reply, dnsServfail(testDNSQuery())) {
t.Fatal("invalid framed SERVFAIL", reply, err)
}
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("worker did not join")
}
})
}
}
func TestDNSTCPCancellationDuringUpstream(t *testing.T) {
started := make(chan struct{})
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
close(started)
<-r.Context().Done()
return nil, r.Context().Err()
})
server, client := net.Pipe()
defer client.Close()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan struct{})
go func() { defer close(done); f.tcpWorker(ctx, server) }()
client.SetDeadline(time.Now().Add(time.Second))
query := testDNSQuery()
var size [2]byte
binary.BigEndian.PutUint16(size[:], uint16(len(query)))
if err := writeDNSFrame(client, size[:], query); err != nil {
t.Fatal(err)
}
<-started
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("upstream cancellation did not join TCP worker")
}
if _, err := client.Read(size[:]); err == nil {
t.Fatal("cancelled TCP connection remained open")
}
}
func TestDNSTCPTimeoutPhasesDoNotExtendLifetime(t *testing.T) {
var requests atomic.Int32
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
requests.Add(1)
<-r.Context().Done()
return nil, r.Context().Err()
})
f.cfg.Timeout = 20 * time.Millisecond
server, client := net.Pipe()
defer client.Close()
done := make(chan struct{})
start := time.Now()
go func() { defer close(done); f.tcpWorker(context.Background(), server) }()
client.SetDeadline(start.Add(2 * time.Second))
query := testDNSQuery()
var size [2]byte
for frame := 0; frame < 17; frame++ {
binary.BigEndian.PutUint16(size[:], uint16(len(query)))
if err := writeDNSFrame(client, size[:], query); err != nil {
break
}
if _, err := io.ReadFull(client, size[:]); err != nil {
break
}
reply := make([]byte, int(binary.BigEndian.Uint16(size[:])))
if _, err := io.ReadFull(client, reply); err != nil {
break
}
if !bytes.Equal(reply, dnsServfail(query)) {
t.Fatal("invalid SERVFAIL")
}
}
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("total lifetime not enforced")
}
if elapsed := time.Since(start); elapsed > 16*f.cfg.Timeout+200*time.Millisecond {
t.Fatal("phase deadlines extended total lifetime", elapsed)
}
if n := requests.Load(); n < 2 || n > 16 {
t.Fatal("unexpected request count", n)
}
}
+664
View File
@@ -0,0 +1,664 @@
package main
import (
"bufio"
"bytes"
"context"
"crypto/rand"
"crypto/rsa"
"crypto/tls"
"crypto/x509"
"crypto/x509/pkix"
"encoding/binary"
"errors"
"fmt"
"io"
"math/big"
"net"
"net/http"
"os"
"path/filepath"
"strconv"
"sync"
"sync/atomic"
"syscall"
"testing"
"time"
)
type dnsRoundTripFunc func(*http.Request) (*http.Response, error)
func (f dnsRoundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
func testDNSQuery() []byte {
return []byte{0x12, 0x34, 1, 0x20, 0, 1, 0, 0, 0, 0, 0, 0, 3, 'w', 'w', 'w', 7, 'e', 'x', 'a', 'm', 'p', 'l', 'e', 0, 0, 1, 0, 1}
}
func testDNSResponse(query []byte) []byte {
response := append([]byte(nil), query...)
response[2] |= 0x80
response[3] |= 0x80
return response
}
func testDNSForwarder(t *testing.T, roundTrip dnsRoundTripFunc) *dnsForwarder {
t.Helper()
return &dnsForwarder{cfg: dnsConfigForTest(t, nil), client: &http.Client{Transport: roundTrip}, slots: make(chan struct{}, dnsConcurrency)}
}
func dnsHTTPResponse(body []byte) *http.Response {
return &http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"application/dns-message"}}, Body: io.NopCloser(bytes.NewReader(body)), ContentLength: int64(len(body))}
}
func TestDNSWireValidation(t *testing.T) {
query := testDNSQuery()
if !validDNSQuery(query) || !validDNSResponse(query, testDNSResponse(query)) {
t.Fatal("valid wire rejected")
}
edns := append([]byte(nil), query...)
edns[11] = 1
edns = append(edns, 0, 0, 41, 0x10, 0, 0, 0, 0x80, 0, 0, 0)
if !validDNSQuery(edns) {
t.Fatal("EDNS/DNSSEC query rejected")
}
failure := dnsServfail(edns)
if len(failure) != len(query) || !bytes.Equal(failure[:2], query[:2]) || failure[2] != 0x81 || failure[3] != 2 || !bytes.Equal(failure[12:], query[12:]) || !bytes.Equal(failure[6:12], make([]byte, 6)) {
t.Fatal(failure)
}
for _, mutate := range []func([]byte) []byte{
func(q []byte) []byte { return q[:11] },
func(q []byte) []byte { q[2] |= 0x80; return q },
func(q []byte) []byte { q[2] |= 0x08; return q },
func(q []byte) []byte { q[2] |= 0x04; return q },
func(q []byte) []byte { q[3] |= 0x40; return q },
func(q []byte) []byte { q[5] = 2; return q },
func(q []byte) []byte { q[7] = 1; return q },
func(q []byte) []byte { q[11] = 255; return q },
func(q []byte) []byte { q[12] = 64; return q },
func(q []byte) []byte { q[12] = 0xc0; q[13] = 12; return q },
func(q []byte) []byte { q[12] = 0xc0; q[13] = 255; return q },
func(q []byte) []byte { return q[:len(q)-1] },
func(q []byte) []byte { return append(q, 0) },
func(q []byte) []byte { return append(q, make([]byte, 4096)...) },
} {
bad := mutate(append([]byte(nil), query...))
if validDNSQuery(bad) || dnsServfail(bad) != nil {
t.Fatal("malformed query accepted")
}
}
// A backwards compression reference is legal; a self-reference is not.
if end, ok := dnsNameEnd(append(query, 0xc0, 12), len(query)); !ok || end != len(query)+2 {
t.Fatal(end, ok)
}
}
func TestDNSExchangePassThroughAndRejects(t *testing.T) {
query := testDNSQuery()
response := testDNSResponse(query)
// Opaque record bytes (including DNSSEC and TTL) must remain untouched.
response[7] = 1
response = append(response, 0xc0, 12, 0, 1, 0, 1, 0, 0, 0, 37, 0, 4, 1, 2, 3, 4)
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
body, _ := io.ReadAll(r.Body)
if r.Method != "POST" || r.URL.String() != defaultDoHURL || r.Header.Get("Content-Type") != "application/dns-message" || r.Header.Get("Accept") != "application/dns-message" || !bytes.Equal(body, query) {
t.Error("incorrect DoH request")
}
return dnsHTTPResponse(response), nil
})
if actual := f.exchange(context.Background(), query); !bytes.Equal(actual, response) {
t.Fatal(actual)
}
for name, change := range map[string]func(*http.Response){
"status": func(r *http.Response) { r.StatusCode = 503 },
"type": func(r *http.Response) { r.Header.Set("Content-Type", "text/plain") },
"encoding": func(r *http.Response) { r.Header.Set("Content-Encoding", "gzip") },
"length": func(r *http.Response) { r.ContentLength++ },
"overflow": func(r *http.Response) {
r.Body = io.NopCloser(bytes.NewReader(make([]byte, 65536)))
r.ContentLength = 65536
},
"short": func(r *http.Response) { r.Body = io.NopCloser(bytes.NewReader(response[:11])); r.ContentLength = 11 },
"id": func(r *http.Response) {
b := append([]byte(nil), response...)
b[0]++
r.Body = io.NopCloser(bytes.NewReader(b))
},
"qr": func(r *http.Response) {
b := append([]byte(nil), response...)
b[2] &= 0x7f
r.Body = io.NopCloser(bytes.NewReader(b))
},
"opcode": func(r *http.Response) {
b := append([]byte(nil), response...)
b[2] |= 0x08
r.Body = io.NopCloser(bytes.NewReader(b))
},
"question": func(r *http.Response) {
b := append([]byte(nil), response...)
b[13] = 'x'
r.Body = io.NopCloser(bytes.NewReader(b))
},
"qdcount": func(r *http.Response) {
b := append([]byte(nil), response...)
b[5] = 0
r.Body = io.NopCloser(bytes.NewReader(b))
},
"record-count": func(r *http.Response) {
b := append([]byte(nil), response...)
b[7] = 2
r.Body = io.NopCloser(bytes.NewReader(b))
},
"record-length": func(r *http.Response) {
b := append([]byte(nil), response...)
b[len(query)+11] = 255
r.Body = io.NopCloser(bytes.NewReader(b))
},
"truncated": func(r *http.Response) { r.Body = io.NopCloser(dnsBrokenReader{}) },
} {
t.Run(name, func(t *testing.T) {
f := testDNSForwarder(t, func(*http.Request) (*http.Response, error) { r := dnsHTTPResponse(response); change(r); return r, nil })
if actual := f.exchange(context.Background(), query); !bytes.Equal(actual, dnsServfail(query)) {
t.Fatal(actual)
}
})
}
f = testDNSForwarder(t, func(*http.Request) (*http.Response, error) { return nil, errors.New("secret upstream") })
if !bytes.Equal(f.exchange(context.Background(), query), dnsServfail(query)) {
t.Fatal("outage not SERVFAIL")
}
}
type dnsBrokenReader struct{}
func (dnsBrokenReader) Read([]byte) (int, error) { return 0, io.ErrUnexpectedEOF }
func TestDNSBootstrapAndHTTPPolicy(t *testing.T) {
cfg := dnsConfigForTest(t, nil)
var addresses []string
dial := dnsBootstrapDial(cfg, func(ctx context.Context, network, address string) (net.Conn, error) {
if network != "tcp" {
t.Fatal(network)
}
if _, ok := ctx.Deadline(); !ok {
t.Fatal("no bounded deadline")
}
addresses = append(addresses, address)
return nil, errDNSForwarder
})
for i := 0; i < 2; i++ {
dial(context.Background(), "tcp", "dns.google:443")
}
if len(addresses) != 4 || addresses[0] != "8.8.8.8:443" || addresses[2] != "8.8.4.4:443" {
t.Fatal(addresses)
}
for _, addr := range []string{"other:443", "dns.google:80", "8.8.8.8:443"} {
dial(context.Background(), "tcp", addr)
}
if len(addresses) != 4 {
t.Fatal("unexpected endpoint dialed")
}
cfg.BootstrapIPs = []net.IP{net.ParseIP("2001:db8::1")}
dnsBootstrapDial(cfg, func(_ context.Context, _, addr string) (net.Conn, error) {
if addr != "[2001:db8::1]:443" {
t.Fatal(addr)
}
return nil, errDNSForwarder
})(context.Background(), "tcp", "dns.google:443")
client, transport := newDNSHTTPClient(cfg, markDNSSocket)
defer transport.CloseIdleConnections()
if transport.Proxy != nil || transport.TLSClientConfig.InsecureSkipVerify || transport.TLSClientConfig.ServerName != "dns.google" || transport.TLSClientConfig.MinVersion != tls.VersionTLS12 || !transport.DisableCompression || transport.MaxConnsPerHost != 16 {
t.Fatal("unsafe transport policy")
}
if client.CheckRedirect(&http.Request{}, nil) == nil {
t.Fatal("redirect allowed")
}
}
type dnsFakeRaw struct{ err error }
func (r dnsFakeRaw) Control(f func(uintptr)) error {
if r.err == nil {
f(42)
}
return r.err
}
func (dnsFakeRaw) Read(func(uintptr) bool) error { return nil }
func (dnsFakeRaw) Write(func(uintptr) bool) error { return nil }
func TestDNSSocketMarkStrict(t *testing.T) {
called := false
if err := setDNSSocketMark(dnsFakeRaw{}, func(fd, level, opt, mark int) error {
called = true
if fd != 42 || level != syscall.SOL_SOCKET || opt != syscall.SO_MARK || mark != 0x50000000 || mark&0x40000000 == 0 || mark&0x10000000 == 0 {
t.Fatal(fd, level, opt, mark)
}
return nil
}); err != nil || !called {
t.Fatal(err, called)
}
failure := errors.New("mark denied")
if err := setDNSSocketMark(dnsFakeRaw{}, func(int, int, int, int) error { return failure }); err != failure {
t.Fatal(err)
}
if err := setDNSSocketMark(dnsFakeRaw{err: failure}, func(int, int, int, int) error { t.Fatal("called on raw failure"); return nil }); err != failure {
t.Fatal(err)
}
}
func TestDNSTCPFramesAndCancellation(t *testing.T) {
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
q, _ := io.ReadAll(r.Body)
return dnsHTTPResponse(testDNSResponse(q)), nil
})
server, client := net.Pipe()
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() { defer close(done); f.tcpWorker(ctx, server) }()
defer client.Close()
client.SetDeadline(time.Now().Add(3 * time.Second))
q := testDNSQuery()
for i := 0; i < 2; i++ {
q[1] = byte(i)
var size [2]byte
binary.BigEndian.PutUint16(size[:], uint16(len(q)))
if err := writeDNSFrame(client, size[:], q); err != nil {
t.Fatal(err)
}
if _, err := io.ReadFull(client, size[:]); err != nil {
t.Fatal(err)
}
b := make([]byte, int(binary.BigEndian.Uint16(size[:])))
if _, err := io.ReadFull(client, b); err != nil || !bytes.Equal(b, testDNSResponse(q)) {
t.Fatal(err, b)
}
}
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("idle TCP worker did not join")
}
}
func TestDNSTCPLimits(t *testing.T) {
for _, n := range []int{11, 4097, 65535} {
t.Run(fmt.Sprint(n), func(t *testing.T) {
f := testDNSForwarder(t, func(*http.Request) (*http.Response, error) {
t.Error("invalid frame forwarded")
return nil, errDNSForwarder
})
server, client := net.Pipe()
done := make(chan struct{})
go func() { defer close(done); f.tcpWorker(context.Background(), server) }()
client.SetDeadline(time.Now().Add(time.Second))
var size [2]byte
binary.BigEndian.PutUint16(size[:], uint16(n))
client.Write(size[:])
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("invalid frame not closed")
}
client.Close()
})
}
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
q, _ := io.ReadAll(r.Body)
return dnsHTTPResponse(testDNSResponse(q)), nil
})
server, client := net.Pipe()
defer client.Close()
client.SetDeadline(time.Now().Add(3 * time.Second))
done := make(chan struct{})
go func() { defer close(done); f.tcpWorker(context.Background(), server) }()
q := testDNSQuery()
var size [2]byte
for i := 0; i < 16; i++ {
binary.BigEndian.PutUint16(size[:], uint16(len(q)))
if err := writeDNSFrame(client, size[:], q); err != nil {
t.Fatal(err)
}
if _, err := io.ReadFull(client, size[:]); err != nil {
t.Fatal(err)
}
if _, err := io.CopyN(io.Discard, client, int64(binary.BigEndian.Uint16(size[:]))); err != nil {
t.Fatal(err)
}
}
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("frame lifetime cap not enforced")
}
}
func TestDNSRedirectNeverFollowed(t *testing.T) {
cfg := dnsConfigForTest(t, nil)
client, transport := newDNSHTTPClient(cfg, markDNSSocket)
defer transport.CloseIdleConnections()
var calls int
client.Transport = dnsRoundTripFunc(func(*http.Request) (*http.Response, error) {
calls++
return &http.Response{StatusCode: 302, Header: http.Header{"Location": {"https://secret.example/dns-query"}}, Body: io.NopCloser(bytes.NewReader(nil))}, nil
})
f := &dnsForwarder{cfg: cfg, client: client}
if !bytes.Equal(f.exchange(context.Background(), testDNSQuery()), dnsServfail(testDNSQuery())) || calls != 1 {
t.Fatal("redirect followed", calls)
}
}
func TestDNSReadinessFailureClosesListeners(t *testing.T) {
f := testDNSForwarder(t, func(*http.Request) (*http.Response, error) {
t.Error("request before ready")
return nil, errDNSForwarder
})
p := &dnsFakePacketConn{closed: make(chan struct{})}
l := &dnsIdleListener{closed: make(chan struct{})}
if err := f.run(context.Background(), p, l, func() (func(), error) { return nil, errDNSForwarder }); err == nil {
t.Fatal("readiness error ignored")
}
select {
case <-p.closed:
default:
t.Fatal("UDP not closed")
}
select {
case <-l.closed:
default:
t.Fatal("TCP not closed")
}
}
func TestDNSReadyOwnership(t *testing.T) {
dir := t.TempDir()
os.Chmod(dir, 0700)
path := filepath.Join(dir, "dns-ready")
cleanup, err := createDNSReady(path)
if err != nil {
t.Fatal(err)
}
info, _ := os.Stat(path)
body, _ := os.ReadFile(path)
if info.Mode().Perm() != 0600 || string(body) != fmt.Sprintf("%d\n", os.Getpid()) {
t.Fatal(info, string(body))
}
if _, err := createDNSReady(path); err == nil {
t.Fatal("existing readiness overwritten")
}
if entries, err := os.ReadDir(dir); err != nil || len(entries) != 1 || entries[0].Name() != "dns-ready" {
t.Fatal("readiness publication/failure left a temporary link", entries, err)
}
// The forwarder itself must hold both descriptors, not the test, so an
// unlinked readiness inode cannot be recycled for a foreign replacement.
fileFD := dnsHeldFD(t, info)
parentInfo, _ := os.Stat(dir)
parentFD := dnsHeldFD(t, parentInfo)
os.Remove(path)
var held syscall.Stat_t
if err := syscall.Fstat(fileFD, &held); err != nil || held.Nlink != 0 {
t.Fatal("readiness inode not held after unlink", err, held.Nlink)
}
os.WriteFile(path, []byte("replacement"), 0600)
cleanup()
cleanup() // Idempotent even after its descriptors have been closed.
for _, fd := range []int{fileFD, parentFD} {
if err := syscall.Fstat(fd, &held); err != syscall.EBADF {
t.Fatal("cleanup leaked descriptor", fd, err)
}
}
if b, err := os.ReadFile(path); err != nil || string(b) != "replacement" {
t.Fatal("foreign readiness removed", err)
}
os.Remove(path)
cleanup, err = createDNSReady(path)
if err != nil {
t.Fatal(err)
}
cleanup()
if _, err := os.Lstat(path); !os.IsNotExist(err) {
t.Fatal(err)
}
os.Symlink("missing", path)
if _, err := createDNSReady(path); err == nil {
t.Fatal("symlink accepted")
}
os.Remove(path)
os.Chmod(dir, 0755)
if _, err := createDNSReady(path); err == nil {
t.Fatal("public parent accepted")
}
}
func dnsHeldFD(t *testing.T, info os.FileInfo) int {
t.Helper()
entries, err := os.ReadDir("/proc/self/fd")
if err != nil {
t.Fatal(err)
}
want := info.Sys().(*syscall.Stat_t)
for _, entry := range entries {
fd, err := strconv.Atoi(entry.Name())
if err != nil {
continue
}
var stat syscall.Stat_t
if syscall.Fstat(fd, &stat) == nil && stat.Ino == want.Ino && stat.Dev == want.Dev {
return fd
}
}
t.Fatal("owned inode has no held descriptor")
return -1
}
func TestDNSReadyAtomicPublication(t *testing.T) {
dir := t.TempDir()
file, err := os.CreateTemp(dir, ".dns-ready-")
if err != nil {
t.Fatal(err)
}
defer file.Close()
path := filepath.Join(dir, "dns-ready")
if _, err := os.Lstat(path); !os.IsNotExist(err) {
t.Fatal("readiness published before PID", err)
}
if err := publishDNSReady(file, path); err != nil {
t.Fatal(err)
}
if body, err := os.ReadFile(path); err != nil || string(body) != fmt.Sprintf("%d\n", os.Getpid()) {
t.Fatal("partial PID published", string(body), err)
}
other, err := os.CreateTemp(dir, "foreign-")
if err != nil {
t.Fatal(err)
}
defer other.Close()
if err := publishDNSReady(other, path); !os.IsExist(err) {
t.Fatal("publication overwrote existing inode", err)
}
if body, _ := os.ReadFile(path); string(body) != fmt.Sprintf("%d\n", os.Getpid()) {
t.Fatal("original PID changed")
}
}
// Verify actual certificate validation over net.Pipe, never a TCP network listener.
func TestDNSVerifiedTLSOverPipe(t *testing.T) {
key, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
t.Fatal(err)
}
template := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "dns.google"}, DNSNames: []string{"dns.google"}, NotBefore: time.Now().Add(-time.Hour), NotAfter: time.Now().Add(time.Hour), KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, IsCA: true, BasicConstraintsValid: true}
der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key)
if err != nil {
t.Fatal(err)
}
cert, err := x509.ParseCertificate(der)
if err != nil {
t.Fatal(err)
}
for _, name := range []string{"trusted", "untrusted", "wrong-name"} {
t.Run(name, func(t *testing.T) {
cfg := dnsConfigForTest(t, nil)
client, transport := newDNSHTTPClient(cfg, markDNSSocket)
defer transport.CloseIdleConnections()
if name != "untrusted" {
roots := x509.NewCertPool()
roots.AddCert(cert)
transport.TLSClientConfig.RootCAs = roots
}
if name == "wrong-name" {
transport.TLSClientConfig.ServerName = "wrong.example"
}
var servers sync.WaitGroup
transport.DialContext = dnsBootstrapDial(cfg, func(ctx context.Context, _, addr string) (net.Conn, error) {
if addr != "8.8.8.8:443" && addr != "8.8.4.4:443" {
t.Error(addr)
}
a, b := net.Pipe()
servers.Add(1)
go func() {
defer servers.Done()
defer b.Close()
b.SetDeadline(time.Now().Add(3 * time.Second))
tlsConn := tls.Server(b, &tls.Config{Certificates: []tls.Certificate{{Certificate: [][]byte{der}, PrivateKey: key}}, MinVersion: tls.VersionTLS12})
if err := tlsConn.Handshake(); err != nil {
return
}
if tlsConn.ConnectionState().ServerName != "dns.google" {
t.Error("SNI changed")
}
request, err := http.ReadRequest(bufio.NewReader(tlsConn))
if err != nil {
return
}
q, _ := io.ReadAll(request.Body)
request.Body.Close()
response := dnsHTTPResponse(testDNSResponse(q))
response.Close = true
response.Write(tlsConn)
}()
return a, nil
})
f := &dnsForwarder{cfg: cfg, client: client}
actual := f.exchange(context.Background(), testDNSQuery())
want := dnsServfail(testDNSQuery())
if name == "trusted" {
want = testDNSResponse(testDNSQuery())
}
if !bytes.Equal(actual, want) {
t.Fatal(actual)
}
transport.CloseIdleConnections()
servers.Wait()
})
}
}
type dnsTestPacket struct {
data []byte
addr net.Addr
}
type dnsFakePacketConn struct {
incoming chan dnsTestPacket
outgoing chan dnsTestPacket
closed chan struct{}
once sync.Once
}
func (p *dnsFakePacketConn) ReadFrom(b []byte) (int, net.Addr, error) {
select {
case q := <-p.incoming:
return copy(b, q.data), q.addr, nil
case <-p.closed:
return 0, nil, net.ErrClosed
}
}
func (p *dnsFakePacketConn) WriteTo(b []byte, a net.Addr) (int, error) {
select {
case p.outgoing <- dnsTestPacket{append([]byte(nil), b...), a}:
return len(b), nil
case <-p.closed:
return 0, net.ErrClosed
}
}
func (p *dnsFakePacketConn) Close() error { p.once.Do(func() { close(p.closed) }); return nil }
func (p *dnsFakePacketConn) LocalAddr() net.Addr { return &net.UDPAddr{} }
func (p *dnsFakePacketConn) SetDeadline(time.Time) error { return nil }
func (p *dnsFakePacketConn) SetReadDeadline(time.Time) error { return nil }
func (p *dnsFakePacketConn) SetWriteDeadline(time.Time) error { return nil }
type dnsIdleListener struct {
closed chan struct{}
once sync.Once
}
func (l *dnsIdleListener) Accept() (net.Conn, error) { <-l.closed; return nil, net.ErrClosed }
func (l *dnsIdleListener) Close() error { l.once.Do(func() { close(l.closed) }); return nil }
func (l *dnsIdleListener) Addr() net.Addr { return &net.TCPAddr{} }
func TestDNSUDPBoundedCopiesAndJoins(t *testing.T) {
var active, maximum atomic.Int32
started := make(chan struct{}, dnsConcurrency)
f := testDNSForwarder(t, func(r *http.Request) (*http.Response, error) {
n := active.Add(1)
defer active.Add(-1)
for old := maximum.Load(); n > old; old = maximum.Load() {
if maximum.CompareAndSwap(old, n) {
break
}
}
q, _ := io.ReadAll(r.Body)
started <- struct{}{}
<-r.Context().Done()
if q[1] >= dnsConcurrency {
t.Error("UDP buffer was reused")
}
return nil, r.Context().Err()
})
p := &dnsFakePacketConn{incoming: make(chan dnsTestPacket, 32), outgoing: make(chan dnsTestPacket, 32), closed: make(chan struct{})}
l := &dnsIdleListener{closed: make(chan struct{})}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
ready := make(chan struct{})
cleaned := make(chan struct{})
done := make(chan error, 1)
go func() {
done <- f.run(ctx, p, l, func() (func(), error) { close(ready); return func() { close(cleaned) }, nil })
}()
<-ready
for i := 0; i < dnsConcurrency; i++ {
q := testDNSQuery()
q[1] = byte(i)
p.incoming <- dnsTestPacket{q, &net.UDPAddr{}}
}
for i := 0; i < dnsConcurrency; i++ {
select {
case <-started:
case <-time.After(time.Second):
t.Fatal("worker not started")
}
}
q := testDNSQuery()
q[1] = 99
p.incoming <- dnsTestPacket{q, &net.UDPAddr{}}
select {
case response := <-p.outgoing:
if !bytes.Equal(response.data, dnsServfail(q)) {
t.Fatal(response)
}
case <-time.After(time.Second):
t.Fatal("saturation not SERVFAIL")
}
cancel()
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(time.Second):
t.Fatal("workers not joined")
}
<-cleaned
if maximum.Load() != dnsConcurrency || active.Load() != 0 || len(f.slots) != 0 {
t.Fatal(maximum.Load(), active.Load(), len(f.slots))
}
}
@@ -0,0 +1,169 @@
package main
import (
"bytes"
"context"
"encoding/binary"
"errors"
"io"
"net"
"net/http"
"testing"
"time"
)
func dnsTestEDNSQuery(payload uint16) []byte {
q := testDNSQuery()
q[11] = 1
return append(q, 0, 0, 41, byte(payload>>8), byte(payload), 0, 0, 0x80, 0, 0, 0)
}
func dnsTestSizedResponse(query []byte, size int) []byte {
response := dnsQuestionReply(query, query)
response[2] = 0x81
response[3] = 0xa0 // RA and AD survive truncation.
response[7] = 1
dataSize := size - len(response) - 12
response = append(response, 0xc0, 12, 0, 16, 0, 1, 0, 0, 0, 37, byte(dataSize>>8), byte(dataSize))
return append(response, make([]byte, dataSize)...)
}
func TestDNSUDPPayloadLimits(t *testing.T) {
for _, tc := range []struct {
name string
query []byte
size, limit int
truncated bool
}{
{"no-EDNS-full-TCP", testDNSQuery(), 65535, 512, true},
{"no-EDNS-at-limit", testDNSQuery(), 512, 512, false},
{"EDNS-at-limit", dnsTestEDNSQuery(1232), 1232, 1232, false},
{"EDNS-over-limit", dnsTestEDNSQuery(1232), 1233, 1232, true},
{"EDNS-small-clamp", dnsTestEDNSQuery(1), 513, 512, true},
{"EDNS-max-at-IPv4-limit", dnsTestEDNSQuery(65535), 65507, 65507, false},
{"EDNS-max-over-IPv4-limit", dnsTestEDNSQuery(65535), 65535, 65507, true},
} {
t.Run(tc.name, func(t *testing.T) {
response := dnsTestSizedResponse(tc.query, tc.size)
if !validDNSQuery(tc.query) || !validDNSResponse(tc.query, response) {
t.Fatal("invalid fixture")
}
actual := dnsUDPReply(tc.query, response)
if len(actual) > tc.limit || (actual[2]&2 != 0) != tc.truncated {
t.Fatal("incorrect UDP ceiling/TC", len(actual), actual[:12])
}
if !tc.truncated {
if !bytes.Equal(actual, response) {
t.Fatal("in-range answer changed")
}
return
}
if !validDNSResponse(tc.query, actual) || !bytes.Equal(actual[:2], response[:2]) || actual[2] != response[2]|2 || actual[3] != response[3] || !bytes.Equal(actual[6:12], make([]byte, 6)) {
t.Fatal("invalid truncated reply", actual)
}
})
}
}
func TestDNSOversizedUDPThenFullTCP(t *testing.T) {
q := testDNSQuery()
response := dnsTestSizedResponse(q, 65535)
f := testDNSForwarder(t, func(*http.Request) (*http.Response, error) { return dnsHTTPResponse(response), nil })
p := &dnsFakePacketConn{outgoing: make(chan dnsTestPacket, 1), closed: make(chan struct{})}
writeDNSDatagram(p, &net.UDPAddr{}, q, f.exchange(context.Background(), q))
if packet := <-p.outgoing; len(packet.data) > 512 || packet.data[2]&2 == 0 {
t.Fatal("oversized UDP did not force TCP")
}
server, client := net.Pipe()
defer client.Close()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan struct{})
go func() { defer close(done); f.tcpWorker(ctx, server) }()
client.SetDeadline(time.Now().Add(3 * time.Second))
var size [2]byte
binary.BigEndian.PutUint16(size[:], uint16(len(q)))
if err := writeDNSFrame(client, size[:], q); err != nil {
t.Fatal(err)
}
if _, err := io.ReadFull(client, size[:]); err != nil || binary.BigEndian.Uint16(size[:]) != 65535 {
t.Fatal("incorrect full TCP frame", size, err)
}
actual := make([]byte, 65535)
if _, err := io.ReadFull(client, actual); err != nil || !bytes.Equal(actual, response) {
t.Fatal("full TCP reply changed", err)
}
cancel()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("TCP worker did not join")
}
}
func TestDNSCompressedQuestionReplies(t *testing.T) {
plain := testDNSQuery()
// Question points forward into an additional RR's owner; copying just the
// original question would leave a dangling pointer after dropping that RR.
q := append([]byte(nil), plain[:12]...)
q[11] = 1
q = append(q, 0xc0, 18, 0, 1, 0, 1)
q = append(q, plain[12:len(plain)-4]...)
q = append(q, 0, 1, 0, 1, 0, 0, 0, 37, 0, 4, 1, 2, 3, 4)
if !validDNSQuery(q) {
t.Fatal("compressed query rejected")
}
for _, reply := range [][]byte{dnsServfail(q), dnsUDPReply(q, dnsTestSizedResponse(q, 65535))} {
if !validDNSResponse(q, reply) || !bytes.Equal(reply[12:], plain[12:]) || reply[12]&0xc0 != 0 || !bytes.Equal(reply[6:12], make([]byte, 6)) {
t.Fatal("dangling/compressed synthesized question", reply)
}
}
response := dnsTestSizedResponse(q, 65535)
f := testDNSForwarder(t, func(*http.Request) (*http.Response, error) { return dnsHTTPResponse(response), nil })
if !bytes.Equal(f.exchange(context.Background(), q), response) {
t.Fatal("equivalent uncompressed upstream question rejected")
}
q[18] = 0xc0
q[19] = 12 // Question -> owner -> question compression loop.
if validDNSQuery(q) || dnsServfail(q) != nil {
t.Fatal("compression loop accepted")
}
}
func TestDNSMalformedEDNSQueries(t *testing.T) {
q := dnsTestEDNSQuery(1232)
duplicate := append(append([]byte(nil), q...), q[len(testDNSQuery()):]...)
duplicate[11] = 2
badOwner := append([]byte(nil), q...)
start := len(testDNSQuery())
badOwner = append(badOwner[:start], append([]byte{1, 'x'}, badOwner[start:]...)...)
badOptions := append([]byte(nil), q...)
badOptions[len(badOptions)-1] = 3
badOptions = append(badOptions, 0, 1, 0)
for i, bad := range [][]byte{duplicate, badOwner, badOptions, q[:len(q)-1]} {
if validDNSQuery(bad) {
t.Fatal("malformed EDNS accepted", i)
}
}
}
type dnsFailingPacketConn struct {
*dnsFakePacketConn
writes int
}
func (p *dnsFailingPacketConn) WriteTo([]byte, net.Addr) (int, error) {
p.writes++
return 0, errors.New("datagram dropped")
}
func TestDNSUDPWriteErrorDropsOnlyDatagram(t *testing.T) {
p := &dnsFailingPacketConn{}
q := testDNSQuery()
for i := 0; i < 2; i++ {
writeDNSDatagram(p, &net.UDPAddr{}, q, dnsTestSizedResponse(q, 65535))
}
if p.writes != 2 {
t.Fatal("write error stopped subsequent datagrams", p.writes)
}
}
+1 -1
View File
@@ -117,7 +117,7 @@ func TestPopulatedV1FingerprintMigration(t *testing.T) {
}
assertRowCounts(t, db, 3, 4)
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 5 {
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 7 {
t.Fatalf("version=%d err=%v", version, err)
}
for id, input := range map[int]string{7: "--lua-desync=fake", 9: args} {
+4
View File
@@ -44,6 +44,10 @@ func googlevideoProposal(flags []string, stdout, stderr io.Writer, discover play
fmt.Fprintln(stderr, "error: tune googlevideo", err)
return 2
}
if _, err := requiredYoutubeAdapter(); err != nil {
fmt.Fprintln(stderr, "error: tune googlevideo", err)
return 1
}
started := time.Now()
db, err := openStore(databasePath())
if err != nil {
@@ -0,0 +1,312 @@
package main
import (
"bytes"
"database/sql"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
)
func scopeFixture(t *testing.T, db *sql.DB) (int64, GooglevideoQualificationProof) {
t.Helper()
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
id, err := insertStrategyCandidate(tx, "googlevideo", hosts, "--lua-desync=fake", nil)
if err != nil {
t.Fatal(err)
}
p := GooglevideoQualificationProof{Version: 1, Mode: QualificationCold, SourceKind: "media-range", Args: "--lua-desync=fake"}
for _, host := range hosts {
h := GooglevideoHostProof{Host: host}
for _, protocol := range []string{"tls12", "tls13"} {
if _, err := tx.Exec(`INSERT INTO candidate_evidence VALUES (?, ?, ?, 3, 3, 1, 1, 1, 1, 'fixture', '{}')`, id, host, protocol); err != nil {
t.Fatal(err)
}
for i := 0; i < 3; i++ {
h.Passes = append(h.Passes, GooglevideoPass{Protocol: protocol, VerifiedTLS: true, Status: 206, RangeEnd: 1023, ResourceBytes: 4096, Bytes: 1024, FTYP: true, LatencyMS: 10})
}
}
p.Hosts = append(p.Hosts, h)
}
raw, err := json.Marshal(p)
if err != nil {
t.Fatal(err)
}
if _, err = tx.Exec("INSERT INTO googlevideo_scope_evidence VALUES (?, ?)", id, string(raw)); err != nil {
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
return id, p
}
func TestGooglevideoScopeTransactional(t *testing.T) {
db := testStore(t)
id, _ := scopeFixture(t, db)
assets := []int64{unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true), unitAssetCandidate(t, db, "yt3.ggpht.com", "--lua-desync=fake", true)}
ids := append([]int64{id}, assets...)
exact, err := renderManagedConfig(db, ids)
if err != nil {
t.Fatal(err)
}
explicit, err := renderManagedConfigScope(db, ids, GooglevideoScopeExact)
if err != nil || !bytes.Equal(exact, explicit) {
t.Fatalf("legacy bytes changed: %v", err)
}
suffix, err := renderManagedConfigScope(db, ids, GooglevideoScopeSuffix)
if err != nil || !strings.Contains(string(suffix), "--hostlist-domains=googlevideo.com ") {
t.Fatalf("suffix=%s %v", suffix, err)
}
if _, err := renderManagedConfigScope(db, []int64{id}, GooglevideoScopeSuffix); err == nil {
t.Fatal("missing assets accepted")
}
m, _ := managedFixture(t)
if err := m.Apply(db, ids, GooglevideoScopeSuffix); err != nil {
t.Fatal(err)
}
j, err := m.load()
if err != nil || j.Version != 3 || j.Scope != GooglevideoScopeSuffix || !reflect.DeepEqual(j.Candidates, ids) {
t.Fatalf("journal=%+v %v", j, err)
}
if err := m.Apply(db, ids); err != nil {
t.Fatal(err)
} // omitted policy preserves suffix
assertManagedContent(t, m, string(suffix))
m.Health = func() (string, error) { return "FAIL", nil }
if err := m.Apply(db, []int64{id}, GooglevideoScopeExact); err == nil {
t.Fatal("health rejection accepted")
}
j, err = m.load()
if err != nil || j.Scope != GooglevideoScopeSuffix || !reflect.DeepEqual(j.Candidates, ids) || j.PendingScope != "" {
t.Fatalf("rollback=%+v %v", j, err)
}
assertManagedContent(t, m, string(suffix))
m.Health = func() (string, error) { return "OK", nil }
m.fault = func(s string) error {
if s == "replaced" {
return errors.New("death")
}
return nil
}
if err := m.Apply(db, []int64{id}, GooglevideoScopeExact); err == nil {
t.Fatal("fault absent")
}
j, err = m.load()
if err != nil || j.Scope != GooglevideoScopeSuffix || j.PendingScope != GooglevideoScopeExact {
t.Fatalf("pending=%+v %v", j, err)
}
m.fault = nil
if err := m.Reconcile(db); err != nil {
t.Fatal(err)
}
j, err = m.load()
if err != nil || j.Scope != GooglevideoScopeSuffix || !reflect.DeepEqual(j.Candidates, ids) {
t.Fatalf("reconcile=%+v %v", j, err)
}
assertManagedContent(t, m, string(suffix))
j.Scope = "future"
if err := m.journal(j); err != nil {
t.Fatal(err)
}
if _, err := m.load(); err == nil {
t.Fatal("unknown scope accepted")
}
}
func TestGooglevideoProofGate(t *testing.T) {
db := testStore(t)
id, base := scopeFixture(t, db)
c, err := readStrategyCandidate(db, id)
if err != nil {
t.Fatal(err)
}
if err := base.validate(c); err != nil {
t.Fatal(err)
}
for _, name := range []string{"samples", "warmup", "404", "tls", "range", "ftyp", "steady", "diversity", "args", "version"} {
t.Run(name, func(t *testing.T) {
raw, _ := json.Marshal(base)
var p GooglevideoQualificationProof
if err := json.Unmarshal(raw, &p); err != nil {
t.Fatal(err)
}
switch name {
case "samples":
p.Hosts = p.Hosts[:2]
case "warmup":
p.Mode = QualificationAdaptive
p.SourceKind = NativeCircularPostWarmup
c.Mode = QualificationAdaptive
p.Hosts[0].WarmupAttempts = 11
p.Hosts[0].WarmupLimit = 10
case "404":
p.Hosts[0].Passes[0].Status = 404
case "tls":
p.Hosts[0].Passes[0].VerifiedTLS = false
case "range":
p.Hosts[0].Passes[0].ResourceBytes = 0
case "ftyp":
p.Hosts[0].Passes[0].FTYP = false
case "steady":
p.Hosts[0].Passes = p.Hosts[0].Passes[:5]
case "diversity":
for i := range p.Hosts {
p.Hosts[i].Host = "rr1---sn-alpha.googlevideo.com"
}
case "args":
p.Args = "--lua-desync=multisplit"
case "version":
p.Version = 2
}
if err := p.validate(c); err == nil {
t.Fatal("invalid proof accepted")
}
c.Mode = QualificationCold
})
}
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
adaptive, err := insertStrategyCandidate(tx, "googlevideo", c.Endpoints, c.Args, nil, QualificationAdaptive)
if err != nil {
t.Fatal(err)
}
for _, host := range c.Endpoints {
for _, protocol := range []string{"tls12", "tls13"} {
if _, err := tx.Exec(`INSERT INTO candidate_evidence VALUES (?, ?, ?, 3, 3, 1, 1, 1, 1, 'fixture', '{}')`, adaptive, host, protocol); err != nil {
t.Fatal(err)
}
}
}
if err := saveAdaptiveGooglevideoEvidence(tx, adaptive, base); err == nil {
t.Fatal("unwired collector accepted")
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
a, err := readStrategyCandidate(db, adaptive)
if err != nil || a.Qualified || a.Mode != QualificationAdaptive {
t.Fatalf("cold metrics qualified adaptive: %+v %v", a, err)
}
qualified, err := listQualifiedCandidates(db, "googlevideo", 1)
if err != nil || len(qualified) != 1 || qualified[0].ID != id {
t.Fatalf("limit filtered incorrectly: %+v %v", qualified, err)
}
if _, err := renderManagedConfig(db, []int64{adaptive}); err == nil {
t.Fatal("adaptive applied without collector attestation")
}
}
func TestGooglevideoV6MigrationPreservesBytes(t *testing.T) {
db := testStore(t)
id := managedCandidateFixture(t, db, "googlevideo", "rr5---sn-alpha.googlevideo.com", "--lua-desync=fake", true)
before, err := renderManagedConfig(db, []int64{id})
if err != nil {
t.Fatal(err)
}
if _, err := db.Exec("DROP TABLE googlevideo_scope_evidence; ALTER TABLE strategy_candidates DROP COLUMN qualification_mode; PRAGMA user_version=6;"); err != nil {
t.Fatal(err)
}
legacy, err := renderManagedConfig(db, []int64{id})
if err != nil || !bytes.Equal(before, legacy) {
t.Fatalf("v6=%s %v", legacy, err)
}
if err := migrate(db); err != nil {
t.Fatal(err)
}
after, err := renderManagedConfig(db, []int64{id})
if err != nil || !bytes.Equal(before, after) || managedHash(before) != managedHash(after) {
t.Fatalf("migration changed legacy: %v", err)
}
if _, err := db.Exec("UPDATE strategy_candidates SET qualification_mode='adaptive' WHERE id=?", id); err == nil {
t.Fatal("immutable definition trigger lost")
}
}
func TestGooglevideoScopeCLIArguments(t *testing.T) {
for _, scope := range []string{"exact", "suffix"} {
if _, ids, err := managedArguments([]string{"apply", "--googlevideo-candidate", "1", "--googlevideo-scope", scope}); err != nil || len(ids) != 1 {
t.Fatalf("%s %v", scope, err)
}
}
for _, args := range [][]string{{"apply", "--googlevideo-scope", "suffix"}, {"apply", "--googlevideo-candidate", "1", "--googlevideo-scope", "future"}, {"apply", "--googlevideo-candidate", "1", "--googlevideo-scope", "suffix", "--googlevideo-scope", "exact"}} {
if _, _, err := managedArguments(args); err == nil {
t.Fatalf("accepted %v", args)
}
}
}
func TestGooglevideoScopeRejectsNonMediaAndSecrets(t *testing.T) {
db := testStore(t)
id, proof := scopeFixture(t, db)
ids := []int64{id, unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true), unitAssetCandidate(t, db, "yt3.ggpht.com", "--lua-desync=fake", true)}
proof.Hosts[0].Passes[0].Status = 404
raw, err := json.Marshal(proof)
if err != nil {
t.Fatal(err)
}
if _, err := db.Exec("UPDATE googlevideo_scope_evidence SET proof=? WHERE candidate_id=?", string(raw), id); err != nil {
t.Fatal(err)
}
if _, err := renderManagedConfigScope(db, ids, GooglevideoScopeSuffix); err == nil {
t.Fatal("root 404 passed media gate")
}
proof.Hosts[0].Passes[0].Status = 206
raw, err = json.Marshal(proof)
if err != nil {
t.Fatal(err)
}
var fields map[string]json.RawMessage
if err := json.Unmarshal(raw, &fields); err != nil {
t.Fatal(err)
}
fields["signed_url"] = json.RawMessage(`"https://secret.invalid/videoplayback?token=private"`)
raw, err = json.Marshal(fields)
if err != nil {
t.Fatal(err)
}
if _, err := db.Exec("UPDATE googlevideo_scope_evidence SET proof=? WHERE candidate_id=?", string(raw), id); err != nil {
t.Fatal(err)
}
if _, err := renderManagedConfigScope(db, ids, GooglevideoScopeSuffix); err == nil {
t.Fatal("secret-bearing evidence accepted")
}
if _, err := renderManagedConfig(db, ids); err != nil {
t.Fatalf("legacy exact evidence changed: %v", err)
}
}
func TestGooglevideoV6MigrationAtomicFailure(t *testing.T) {
db := testStore(t)
id := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
before, err := renderManagedConfig(db, []int64{id})
if err != nil {
t.Fatal(err)
}
if _, err := db.Exec("DROP TABLE googlevideo_scope_evidence; ALTER TABLE strategy_candidates DROP COLUMN qualification_mode; CREATE TABLE googlevideo_scope_evidence (conflict INTEGER); PRAGMA user_version=6;"); err != nil {
t.Fatal(err)
}
if err := migrate(db); err == nil {
t.Fatal("conflicting migration accepted")
}
var version, columns int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 6 {
t.Fatalf("version=%d %v", version, err)
}
if err := db.QueryRow("SELECT count(*) FROM pragma_table_info('strategy_candidates') WHERE name='qualification_mode'").Scan(&columns); err != nil || columns != 0 {
t.Fatalf("partial column migration: %d %v", columns, err)
}
after, err := renderManagedConfig(db, []int64{id})
if err != nil || !bytes.Equal(before, after) {
t.Fatalf("assets changed on failed migration: %v", err)
}
}
+9 -5
View File
@@ -17,16 +17,20 @@ var cacheTestFlags = []string{"--video-id", "dQw4w9WgXcQ", "--proxy", "socks5://
func cachePlayer(t *testing.T, order *[]string) playerRunner {
t.Helper()
return func(_ context.Context, args []string, out, diagnostics io.Writer) error {
return func(_ context.Context, args []string, stdin io.Reader, out, diagnostics io.Writer) error {
*order = append(*order, "discover")
invocation := strings.Join(args, " ")
for _, want := range []string{"--proxy socks5://localhost:1080", "21.02.35", "dQw4w9WgXcQ", playerURL} {
for _, want := range []string{"--proxy socks5://localhost:1080", "21.02.35", "--json @-", playerURL} {
if !strings.Contains(invocation, want) {
t.Fatalf("discovery missing %q: %s", want, invocation)
}
}
payload, err := io.ReadAll(stdin)
if err != nil || !strings.Contains(string(payload), `"videoId":"dQw4w9WgXcQ"`) || strings.Contains(invocation, "dQw4w9WgXcQ") {
t.Fatal("player payload must be supplied only on stdin")
}
fmt.Fprint(diagnostics, "https://secret.invalid/signed?token=SECRET")
_, err := fmt.Fprintf(out, `{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"https://%s/videoplayback?token=SECRET"}}`, cacheTestHost)
_, err = fmt.Fprintf(out, `{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"https://%s/videoplayback?token=SECRET"}}`, cacheTestHost)
return err
}
}
@@ -154,7 +158,7 @@ fi
if err := db.QueryRow(`PRAGMA user_version`).Scan(&version); err != nil {
t.Fatal(err)
}
if evidence != 2 || legacy != 0 || version != 5 {
if evidence != 2 || legacy != 0 || version != 7 {
t.Fatalf("evidence=%d legacy=%d version=%d", evidence, legacy, version)
}
for _, command := range [][]string{{"history"}, {"strategies", "youtube"}} {
@@ -209,7 +213,7 @@ func TestTuneGooglevideoFailures(t *testing.T) {
var order []string
player := cachePlayer(t, &order)
if mode == "discovery" {
player = func(context.Context, []string, io.Writer, io.Writer) error {
player = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
order = append(order, "discover")
return fmt.Errorf("https://signed.invalid/?SECRET")
}
+53 -40
View File
@@ -21,8 +21,8 @@ func configLookup(values map[string]string) func(string) (string, bool) {
func TestHealthConfigDefaultsAndOverrides(t *testing.T) {
cfg, err := parseHealthConfig(configLookup(nil))
want := healthConfig{"socks5://127.0.0.1:11080", "dQw4w9WgXcQ", 5 * time.Minute, 3, 30 * time.Minute}
if err != nil || cfg != want {
want := healthConfig{SOCKSProxy: "socks5://127.0.0.1:11080", VideoID: "dQw4w9WgXcQ", HealthInterval: 5 * time.Minute, FailureThreshold: 3, RecoveryCooldown: 30 * time.Minute}
if err != nil || !reflect.DeepEqual(cfg, want) {
t.Fatalf("defaults = %+v, %v; want %+v", cfg, err, want)
}
values := map[string]string{
@@ -33,12 +33,12 @@ func TestHealthConfigDefaultsAndOverrides(t *testing.T) {
"ZAPRET_AUTOTUNE_RECOVERY_COOLDOWN": "1h",
}
cfg, err = parseHealthConfig(configLookup(values))
want = healthConfig{"socks5://[::1]:12080", "abcdefghijk", 90 * time.Second, 5, time.Hour}
if err != nil || cfg != want {
want = healthConfig{SOCKSProxy: "socks5://[::1]:12080", VideoID: "abcdefghijk", HealthInterval: 90 * time.Second, FailureThreshold: 5, RecoveryCooldown: time.Hour}
if err != nil || !reflect.DeepEqual(cfg, want) {
t.Fatalf("overrides = %+v, %v; want %+v", cfg, err, want)
}
for _, proxy := range []string{"socks5://localhost:1", "socks5://LOCALHOST:65535", "socks5://127.0.0.2:11080", "socks5://[::1]:11080"} {
if _, err := parseHealthConfig(configLookup(map[string]string{"ZAPRET_AUTOTUNE_SOCKS_PROXY": proxy})); err != nil {
for _, proxy := range []string{"socks5://localhost:1", "socks5://LOCALHOST:65535", "socks5://127.0.0.2:11080", "socks5://[::1]:11080", "socks5h://localhost:1", "socks5h://LOCALHOST:65535", "socks5h://127.0.0.2:11080", "socks5h://[::1]:11080"} {
if cfg, err := parseHealthConfig(configLookup(map[string]string{"ZAPRET_AUTOTUNE_SOCKS_PROXY": proxy})); err != nil || cfg.SOCKSProxy != proxy {
t.Errorf("valid loopback rejected: %s: %v", proxy, err)
}
}
@@ -47,12 +47,14 @@ func TestHealthConfigDefaultsAndOverrides(t *testing.T) {
func TestHealthConfigRejectsInvalid(t *testing.T) {
invalid := map[string][]string{
"ZAPRET_AUTOTUNE_SOCKS_PROXY": {
"", "http://127.0.0.1:11080", "socks5h://127.0.0.1:11080",
"", "http://127.0.0.1:11080", "https://127.0.0.1:11080", "socks4://127.0.0.1:11080",
"socks5://192.168.1.1:11080", "socks5://0.0.0.0:11080", "socks5://[::]:11080",
"socks5://proxy.example:11080", "socks5://localhost.evil:11080",
"socks5://user:SECRET@localhost:11080", "socks5://localhost", "socks5://localhost:0",
"socks5://localhost:65536", "socks5://localhost:bad", "socks5://localhost:11080/path",
"socks5://localhost:11080?SECRET", "socks5://localhost:11080#SECRET", " socks5://localhost:11080",
"socks5://@localhost:11080", "socks5://SECRET@localhost:11080", "socks5://localhost:11080?", "socks5://localhost:11080#",
"socks5://localhost:11080\x00", "socks5://localhost:11080\x1f", "socks5://localhost:11080\x7f",
},
"ZAPRET_AUTOTUNE_VIDEO_ID": {"", "short", "dQw4w9WgXc!"},
"ZAPRET_AUTOTUNE_HEALTH_INTERVAL": {"", "0", "-5m", "5", "SECRET", "999999999999999999999h"},
@@ -60,6 +62,13 @@ func TestHealthConfigRejectsInvalid(t *testing.T) {
"ZAPRET_AUTOTUNE_FAILURE_THRESHOLD": {"", "0", "-3", "1.5", "SECRET", "999999999999999999999"},
}
for key, values := range invalid {
if key == "ZAPRET_AUTOTUNE_SOCKS_PROXY" {
for _, value := range values {
if strings.HasPrefix(value, "socks5://") {
values = append(values, strings.Replace(value, "socks5://", "socks5h://", 1))
}
}
}
for _, value := range values {
if _, err := parseHealthConfig(configLookup(map[string]string{key: value})); err == nil || !strings.Contains(err.Error(), key) || strings.Contains(err.Error(), "SECRET") {
t.Errorf("%s=%q: expected sanitized config error, got %v", key, value, err)
@@ -105,37 +114,41 @@ func setHealthEnvironment(t *testing.T) {
}
func TestRunHealthYoutube(t *testing.T) {
setHealthEnvironment(t)
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5://[::1]:12080")
t.Setenv("ZAPRET_AUTOTUNE_VIDEO_ID", "abcdefghijk")
for _, key := range []string{"ALL_PROXY", "HTTPS_PROXY", "https_proxy", "all_proxy"} {
t.Setenv(key, "http://untrusted.example:8080")
}
t.Setenv("NO_PROXY", "*")
var stdout, stderr bytes.Buffer
calls := 0
runner := func(ctx context.Context, args []string, out, diagnostics io.Writer) error {
calls++
deadline, ok := ctx.Deadline()
if !ok || time.Until(deadline) <= 0 || time.Until(deadline) > 35*time.Second {
t.Error("request must have bounded deadline")
}
prefix := []string{"--disable", "--silent", "--show-error", "--fail", "--max-time", "30", "--noproxy", "", "--proxy", "socks5://[::1]:12080", "--user-agent", playerUserAgent, "--json"}
if len(args) != len(prefix)+2 || !reflect.DeepEqual(args[:len(prefix)], prefix) || args[len(args)-1] != playerURL {
t.Fatalf("unexpected proxy/API arguments: %q", args)
}
var payload struct {
VideoID string `json:"videoId"`
}
if err := json.Unmarshal([]byte(args[len(prefix)]), &payload); err != nil || payload.VideoID != "abcdefghijk" {
t.Fatalf("unexpected video payload: %+v %v", payload, err)
}
io.WriteString(diagnostics, "SECRET")
_, err := io.WriteString(out, `{"playabilityStatus":{"status":"OK"}}`)
return err
}
if code := runWithPlayer([]string{"health", "youtube"}, &stdout, &stderr, runner); code != 0 || calls != 1 || stderr.Len() != 0 || !strings.Contains(stdout.String(), "Player API: OK (not a playback check)") {
t.Fatalf("code=%d calls=%d stdout=%q stderr=%q", code, calls, &stdout, &stderr)
for _, proxy := range []string{"socks5://[::1]:12080", "socks5h://[::1]:12080", "socks5h://127.0.0.1:11082"} {
t.Run(proxy, func(t *testing.T) {
setHealthEnvironment(t)
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", proxy)
t.Setenv("ZAPRET_AUTOTUNE_VIDEO_ID", "abcdefghijk")
for _, key := range []string{"ALL_PROXY", "HTTPS_PROXY", "https_proxy", "all_proxy"} {
t.Setenv(key, "http://untrusted.example:8080")
}
t.Setenv("NO_PROXY", "*")
var stdout, stderr bytes.Buffer
calls := 0
runner := func(ctx context.Context, args []string, stdin io.Reader, out, diagnostics io.Writer) error {
calls++
deadline, ok := ctx.Deadline()
if !ok || time.Until(deadline) <= 0 || time.Until(deadline) > 35*time.Second {
t.Error("request must have bounded deadline")
}
prefix := []string{"--disable", "--silent", "--show-error", "--fail", "--max-time", "30", "--noproxy", "", "--proxy", proxy, "--user-agent", playerUserAgent, "--json"}
if len(args) != len(prefix)+2 || !reflect.DeepEqual(args[:len(prefix)], prefix) || args[len(prefix)] != "@-" || args[len(args)-1] != playerURL {
t.Fatalf("unexpected proxy/API arguments: %q", args)
}
var payload struct {
VideoID string `json:"videoId"`
}
if err := json.NewDecoder(stdin).Decode(&payload); err != nil || payload.VideoID != "abcdefghijk" {
t.Fatalf("unexpected video payload: %+v %v", payload, err)
}
io.WriteString(diagnostics, "SECRET")
_, err := io.WriteString(out, `{"playabilityStatus":{"status":"OK"}}`)
return err
}
if code := runWithPlayer([]string{"health", "youtube"}, &stdout, &stderr, runner); code != 0 || calls != 1 || stderr.Len() != 0 || !strings.Contains(stdout.String(), "Player API: OK (not a playback check)") {
t.Fatalf("code=%d calls=%d stdout=%q stderr=%q", code, calls, &stdout, &stderr)
}
})
}
}
@@ -145,7 +158,7 @@ func TestHealthYoutubeFailures(t *testing.T) {
t.Run(mode, func(t *testing.T) {
var stdout, stderr bytes.Buffer
calls := 0
runner := func(_ context.Context, _ []string, out, diagnostics io.Writer) error {
runner := func(_ context.Context, _ []string, _ io.Reader, out, diagnostics io.Writer) error {
calls++
io.WriteString(diagnostics, "SECRET")
switch mode {
@@ -179,7 +192,7 @@ func TestHealthYoutubeFailures(t *testing.T) {
func TestHealthDispatchAndConfigFailWithoutRequest(t *testing.T) {
setHealthEnvironment(t)
runner := func(context.Context, []string, io.Writer, io.Writer) error {
runner := func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("invalid command/config must not request or fall back")
return nil
}
@@ -0,0 +1,200 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"image"
"image/jpeg"
"io"
"strings"
"testing"
"time"
)
func rejectedImageRunner(mode string) subprocessRunner {
return func(_ context.Context, args []string, in io.Reader, out, diag io.Writer) error {
config, err := io.ReadAll(in)
if err != nil {
return err
}
raw := strings.TrimSuffix(strings.TrimPrefix(string(config), "url = \""), "\"\n")
body, contentType := resourceFixture(), "image/png"
switch mode {
case "jpeg":
var b bytes.Buffer
if err := jpeg.Encode(&b, image.NewRGBA(image.Rect(0, 0, 8, 8)), nil); err != nil {
return err
}
body, contentType = b.Bytes(), "image/jpeg"
if start := bytes.Index(body, []byte{0xff, 0xda}); start >= 0 {
body = body[:start+12]
}
case "png":
body = body[:len(body)/2]
case "magic":
body = []byte("not an image")
case "gif":
body, contentType = []byte("GIF89a\x01\x00\x01\x00\x00\x00\x00"), "image/gif"
case "dimensions":
body, contentType = []byte("GIF89a\xff\xff\xff\xff\x00\x00\x00"), "image/gif"
case "type":
contentType = "image/jpeg"
case "unknown":
return errors.New("invalid image body")
}
fmt.Fprintf(out, "HTTP/1.1 200 OK\r\nContent-Type: %s\r\nContent-Length: %d\r\n\r\n", contentType, len(body))
if _, err := out.Write(body); err != nil {
return err
}
metrics, _ := json.Marshal(map[string]interface{}{"url_effective": raw, "ssl_verify_result": 0, "num_redirects": 0, "http_code": 200, "time_total": 0.1, "speed_download": 10000, "size_download": len(body)})
if mode == "footer" {
metrics = []byte("{malformed")
}
fmt.Fprint(out, resourceTrailer, string(metrics))
protocol := "TLSv1.2"
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
protocol = "TLSv1.3"
}
fmt.Fprintln(diag, "SSL connection using "+protocol)
return nil
}
}
func TestImageBodyRejectionCollectorBoundaries(t *testing.T) {
for _, mode := range []string{"jpeg", "png", "magic", "gif", "dimensions", "type", "unknown", "footer"} {
t.Run(mode, func(t *testing.T) {
m, err := collectAssetMeasurement(context.Background(), "socks5://127.0.0.1:1080", "https://i.ytimg.com/image", "i.ytimg.com", "tls12", "", rejectedImageRunner(mode))
if err == nil || len(m.Transfers) != 0 {
t.Fatal("invalid image qualified", m, err)
}
ordinary := mode != "unknown" && mode != "footer"
var fatal *managedRecoveryFatal
if errors.As(err, &fatal) == ordinary || managedAssetTransferNonpass(err) != ordinary || managedAssetHealthNonpass(err) != ordinary {
t.Fatal("wrong rejection class", err)
}
if managedGooglevideoTransferNonpass(err) {
t.Fatal("image rejection became media absence")
}
for _, veto := range []error{context.Canceled, errors.New("unknown infrastructure"), &managedRecoveryFatal{Err: imageBodyRejection("invalid image body")}} {
mixed := resourceFailure(context.Background(), errors.Join(err, veto))
if !errors.As(mixed, &fatal) || managedAssetTransferNonpass(mixed) || managedAssetHealthNonpass(mixed) {
t.Fatal("mixed safety veto lost", mixed)
}
}
})
}
_, _, metadataErr := playerMediaResource([]byte("{malformed"))
if metadataErr == nil {
t.Fatal("malformed Player JSON accepted")
}
for _, err := range []error{metadataErr, errors.New("invalid normalized image evidence")} {
failure := resourceFailure(context.Background(), err)
var fatal *managedRecoveryFatal
if !errors.As(failure, &fatal) || managedAssetTransferNonpass(failure) || managedGooglevideoTransferNonpass(failure) {
t.Fatal("local metadata damage became absence", failure)
}
}
oversized := assetTransfer{Body: make([]byte, (5<<20)+1)}
if err := normalizeAssetBody(&oversized); err == nil || !managedAssetTransferNonpass(resourceFailure(context.Background(), err)) {
t.Fatal("image decoder budget rejection lost", err)
}
}
func TestManagedTruncatedImageMonitorRecovery(t *testing.T) {
for _, recoverable := range []bool{false, true} {
t.Run(fmt.Sprint(recoverable), func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
target, _, _ := managedCLIFixture(t)
initManagedCLI(t, target)
d.transaction.Dir = target.directory
r, events, _ := assetRuntimeFixture(t)
good, start := r.curl, r.start
attempts := 0
r.start = func(args []string) (managedAssetWorker, error) { attempts++; return start(args) }
r.curl = func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
if recoverable && attempts >= 2 {
return good(ctx, args, in, out, diag)
}
return rejectedImageRunner("jpeg")(ctx, args, in, out, diag)
}
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
ids := []int64{front, ggc}
if err := d.transaction.Apply(d.db, ids); err != nil {
t.Fatal(err)
}
d.service = true
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image"}, proxy: r.proxy, runner: r.curl}, nil
}
d.verifyAssets = r.verify
d.health = managedServiceHealth(r.proxy, d.frontHealth, d.resources, r.curl)
monitor := monitorRuntimeFixture(t)
monitor.db, monitor.target, monitor.threshold = d.db, target, 1
if err := wireManagedRecovery(&monitor, d, func(error) {}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
monitor.reportRecoveryError = func(err error) {
if recoverable || !errors.Is(err, errNoQualifiedStrategy) {
t.Fatal("wrong recovery result", err)
}
}
waits := 0
monitor.wait = func(ctx context.Context, _ time.Duration) error {
state, err := readMonitorState(d.db, monitor.serviceID)
if err != nil || state.RecoveryRunning {
t.Fatal("recovery ownership retained", state, err)
}
if !recoverable && state.CooldownUntil.Before(monitor.now().Add(2*time.Hour)) {
t.Fatal("missing two-hour pause", state)
}
waits++
if waits == 2 {
cancel()
return ctx.Err()
}
return nil
}
if err := runManagedMonitor(ctx, monitor); !errors.Is(err, context.Canceled) {
t.Fatal("invalid payload killed monitor", err, *events)
}
want := len(managedAssetSeeds)
if recoverable {
want = 2
}
joins := 0
for _, event := range *events {
if event == "join" {
joins++
}
}
if attempts != want || joins != want {
t.Fatal("seed fallback/worker join failed", attempts, joins, *events)
}
active, err := d.transaction.ActiveCandidates(d.db)
if err != nil {
t.Fatal(err)
}
if recoverable {
if len(active) != 3 {
t.Fatal("fresh image not applied", active)
}
fresh, err := readStrategyCandidate(d.db, active[2])
if err != nil || !fresh.Qualified || fresh.Args != managedAssetSeeds[1] {
t.Fatal("invalid fresh evidence", fresh, err)
}
ids = append(ids, fresh.ID)
var rows int
if err := d.db.QueryRow("SELECT count(*) FROM candidate_evidence WHERE candidate_id=? AND passes_ok=3 AND passes_total=3", fresh.ID).Scan(&rows); err != nil || rows != 2 {
t.Fatal("missing both TLS 3/3 rows", rows, err)
}
}
assertActiveCandidates(t, d.transaction, d.db, ids)
})
}
}
+58
View File
@@ -9,6 +9,16 @@ import (
const usage = `Usage: zapret-autotune <command>
Commands:
strategy-import --strategies FILE
Import immutable definitions only into the private target registry
plan --config FILE
Authenticated domain compatibility proposal (never apply)
qualify --config FILE --strategies FILE --isolated
Intrusive finite batch; only PID 1 isolated startup on disabled baseline
dns-forwarder mode|address|serve
Managed DNS configuration or loopback DoH forwarder
dns-forwarder readiness-check|readiness-cleanup <expected-dns-pid>
Validate or clean readiness using inherited PID 1 anchors
status Show latest recorded tune run/proposal (not live service health)
history [--limit N] List recorded tune attempts (default 20; proposals not active)
strategies youtube List recorded proposals by group (not active)
@@ -17,7 +27,13 @@ Commands:
health youtube One-shot Player API health through configured loopback SOCKS
managed init|reconcile|status|apply|monitor
Guarded activation/monitoring in the dedicated managed-test target only
managed monitor --resource-config FILE
Periodic configured checks only (no recovery/apply)
managed resource-status
Read-only configured-check snapshot (explicit stale/blocked)
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
check --config FILE
Check configured HTTPS resources (checks only; no strategy selection)
tune youtube Test youtube.com frontend only; print options without applying
tune googlevideo --video-id ID --proxy socks5://HOST:PORT
Discover and verify one exact cache host; proposal only
@@ -35,6 +51,48 @@ func run(args []string, stdout, stderr io.Writer) int {
}
func runWithPlayer(args []string, stdout, stderr io.Writer, player playerRunner) int {
if len(args) > 0 && args[0] == "qualification-completion-check" {
return qualificationCompletionCLI(args[1:], managedDependencies(player))
}
if len(args) > 0 && (args[0] == "plan" || args[0] == "strategy-import" || args[0] == "qualify") {
return resourcePlanCLI(args, stdout, stderr, managedDependencies(player))
}
if len(args) == 3 && args[0] == "check" && args[1] == "--config" {
return configuredResourceCheck(args[2], stdout, stderr)
}
if len(args) > 0 && args[0] == "dns-forwarder" {
return dnsForwarderCLI(args[1:], stdout, stderr)
}
if len(args) == 1 && args[0] == "target-config-check" {
path, _ := os.LookupEnv("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
policy, _ := os.LookupEnv(resourceStrategyIDsEnv)
if _, err := parseResourceStrategyIDs(policy); err != nil || policy != "" && path == "" {
fmt.Fprintln(stderr, "configerror: invalid generic strategy selection")
return 64
}
if path == "" {
path, _ = os.LookupEnv("ZAPRET_AUTOTUNE_QUALIFY_CONFIG")
}
if path != "" {
port, err := managedTargetSettings(os.LookupEnv)
if err != nil || port != 1080 {
fmt.Fprintln(stderr, "configerror: invalid generic target settings")
return 64
}
if _, _, err := resourceStaticConfig(os.LookupEnv); err != nil {
fmt.Fprintln(stderr, "configerror: invalid generic DNS/proxy configuration")
return 64
}
} else if _, err := parseManagedHealthConfig(os.LookupEnv); err != nil {
fmt.Fprintln(stderr, "configerror: invalid target health/service configuration")
return 64
}
if _, err := parseManagedDNSConfig(os.LookupEnv); err != nil {
fmt.Fprintln(stderr, "configerror: invalid managed DNS configuration")
return 64
}
return 0
}
if len(args) > 0 && args[0] == "managed" {
return managedCLI(args[1:], stdout, stderr, managedDependencies(player))
}
@@ -0,0 +1,385 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"os"
"os/exec"
"strconv"
"strings"
"syscall"
"time"
)
const managedAssetQueue = 200 // Serialized recovery never runs blockcheck concurrently.
const managedAssetTable = "zapret_autotune_asset_probe"
type managedAssetWorker struct {
pid int
done <-chan error
stop func() error
}
type managedAssetRuntime struct {
guard func() error
nft func(context.Context, string) (string, error)
start func([]string) (managedAssetWorker, error)
queue func(int) (bool, error)
curl subprocessRunner
proxy string
}
type managedAssetCleanupFailure struct{ Err error }
func (e *managedAssetCleanupFailure) Error() string { return "owned asset runtime cleanup failed" }
func (e *managedAssetCleanupFailure) Unwrap() error { return e.Err }
func newManagedAssetVerifier(target managedDeps) func(context.Context, storedCandidate, managedServiceInventory) ([]assetMeasurement, error) {
return newManagedAssetRuntime(target).verify
}
func newManagedAssetRuntime(target managedDeps) managedAssetRuntime {
proxy, _ := target.lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
proxy, proxyErr := managedProbeProxy(proxy)
r := managedAssetRuntime{proxy: proxy, curl: runResourceCurl}
r.guard = func() error {
if proxyErr != nil {
return proxyErr
}
if _, err := managedRuntimeGuard(target); err != nil {
return err
}
status, err := os.ReadFile("/proc/self/status")
if err != nil {
return err
}
return managedAssetCapabilityGuard(string(status))
}
r.nft = func(ctx context.Context, input string) (string, error) {
cmd := exec.CommandContext(ctx, "nft", "-f", "-")
cmd.Stdin = strings.NewReader(input)
var out strings.Builder
cmd.Stdout, cmd.Stderr = &out, io.Discard
err := cmd.Run()
return out.String(), err
}
r.start = func(args []string) (managedAssetWorker, error) {
// No daemon or Setpgid: retain monitor session ownership.
cmd := exec.Command("/opt/zapret2/nfq2/nfqws2", args...)
cmd.Stdout, cmd.Stderr = io.Discard, io.Discard
if err := cmd.Start(); err != nil {
return managedAssetWorker{}, err
}
done := make(chan error, 1)
go func() { done <- cmd.Wait() }()
return managedAssetWorker{pid: cmd.Process.Pid, done: done, stop: func() error {
err := cmd.Process.Kill()
if errors.Is(err, os.ErrProcessDone) {
return errors.New("asset worker exited before owned stop")
}
return err
}}, nil
}
r.queue = managedAssetQueueReady
return r
}
func managedAssetCapabilityGuard(status string) error {
const required = 1<<5 | 1<<12 | 1<<19 // KILL, NET_ADMIN, SYS_PTRACE
for _, line := range strings.Split(status, "\n") {
if strings.HasPrefix(line, "CapEff:") {
bits, err := strconv.ParseUint(strings.TrimSpace(strings.TrimPrefix(line, "CapEff:")), 16, 64)
if err == nil && bits&required == required {
return nil
}
}
}
return errors.New("asset verifier requires CAP_KILL, CAP_NET_ADMIN and CAP_SYS_PTRACE")
}
// Queue peer portid is a netlink socket id, not necessarily a process id.
func managedAssetQueueReady(pid int) (bool, error) {
queues, err := os.ReadFile("/proc/net/netfilter/nfnetlink_queue")
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
if err != nil {
return false, err
}
peer := ""
for _, line := range strings.Split(string(queues), "\n") {
f := strings.Fields(line)
if len(f) >= 2 && f[0] == strconv.Itoa(managedAssetQueue) {
peer = f[1]
}
}
if peer == "" {
return false, nil
}
if pid == 0 {
return true, nil
}
if err := syscall.Kill(pid, 0); err != nil {
return false, err
}
netlink, err := os.ReadFile("/proc/net/netlink")
if err != nil {
return false, err
}
inodes := map[string]bool{}
for _, line := range strings.Split(string(netlink), "\n") {
f := strings.Fields(line)
if len(f) >= 10 && f[1] == "12" && f[2] == peer {
inodes["socket:["+f[9]+"]"] = true
}
}
return managedAssetQueueFDReady(fmt.Sprintf("/proc/%d/fd", pid), inodes, os.Readlink)
}
func managedAssetQueueFDReady(dir string, inodes map[string]bool, readlink func(string) (string, error)) (bool, error) {
fds, err := os.ReadDir(dir)
if err != nil {
return false, err
}
denied := false
for _, fd := range fds {
link, err := readlink(dir + "/" + fd.Name())
if err == nil && inodes[link] {
return true, nil
}
if errors.Is(err, os.ErrPermission) {
denied = true
} else if err != nil && !errors.Is(err, os.ErrNotExist) {
return false, err
}
}
if denied {
return false, errors.New("asset queue ownership inspection denied")
}
return false, errors.New("asset queue belongs to another process")
}
const managedAssetRules = `create table inet zapret_autotune_asset_probe
add chain inet zapret_autotune_asset_probe output { type filter hook output priority -151; policy accept; }
add chain inet zapret_autotune_asset_probe input { type filter hook input priority -151; policy accept; }
add rule inet zapret_autotune_asset_probe output meta mark & 0x10000000 == 0 tcp dport 443 queue num 200
add rule inet zapret_autotune_asset_probe input meta mark & 0x10000000 == 0 tcp sport 443 queue num 200
`
func (r managedAssetRuntime) verify(ctx context.Context, c storedCandidate, inventory managedServiceInventory) (measurements []assetMeasurement, err error) {
if c.Group != "assets" || len(c.Endpoints) == 0 || len(c.Endpoints) > 2 || strings.TrimSpace(c.Args) == "" {
return nil, &managedRecoveryFatal{Err: errors.New("invalid asset candidate")}
}
if e := (&candidate{Args: c.Args}).validateArgs(); e != nil {
return nil, &managedRecoveryFatal{Err: e}
}
for _, arg := range strings.Fields(c.Args) {
if strings.HasPrefix(arg, "--lua-init=") && arg != "--lua-init=@/opt/zapret2/lua/zapret-lib.lua" && arg != "--lua-init=@/opt/zapret2/lua/zapret-antidpi.lua" {
return nil, &managedRecoveryFatal{Err: errors.New("asset verifier only permits fixed Lua init files")}
}
}
seen := map[string]bool{}
for _, host := range c.Endpoints {
if !assetEndpoint(host) || seen[host] {
return nil, &managedRecoveryFatal{Err: errors.New("invalid asset endpoints")}
}
seen[host] = true
if _, e := privateResourceURL(inventory.images[host], host); e != nil {
return nil, e
}
}
if e := ctx.Err(); e != nil {
return nil, e
}
err = r.withOwnedWorker(ctx, c.Endpoints, c.Args, func(probeCtx context.Context) error {
for _, host := range c.Endpoints {
for _, protocol := range []string{"tls12", "tls13"} {
m, e := collectAssetMeasurement(probeCtx, r.proxy, inventory.images[host], host, protocol, c.Args, r.curl)
if e != nil {
if ctx.Err() != nil {
return ctx.Err()
}
if managedAssetTransferNonpass(e) {
return errCandidateNonpass
}
return e
}
measurements = append(measurements, m)
}
}
return nil
})
if err != nil {
return nil, err
}
return measurements, nil
}
// Shared, serialized session lifecycle: only an atomically created table is
// owned, and success is returned only after queue removal and worker join.
func (r managedAssetRuntime) withOwnedWorker(ctx context.Context, endpoints []string, strategyArgs string, capture func(context.Context) error) (err error) {
if err := ctx.Err(); err != nil {
return err
}
if e := r.guard(); e != nil {
return &managedRecoveryFatal{Err: e}
}
tables, e := r.nft(ctx, "list tables\n")
if e != nil {
return &managedRecoveryFatal{Err: e}
}
if strings.Contains(tables, "table inet "+managedAssetTable) {
return &managedRecoveryFatal{Err: errors.New("asset nft table already exists")}
}
occupied, e := r.queue(0)
if e != nil || occupied {
return &managedRecoveryFatal{Err: qualificationOperation(errors.Join(e, errors.New("asset queue unavailable")), stageWorkerQueue)}
}
args := managedProbeWorkerArgs(endpoints, strategyArgs)
worker, e := r.start(args)
if e != nil {
return &managedRecoveryFatal{Err: qualificationOperation(e, stageWorkerStart)}
}
joined, table := false, false
defer func() {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
var cleanupErr error
// Remove queues before killing the consumer.
if table {
_, cleanupErr = r.nft(cleanup, "delete table inet "+managedAssetTable+"\n")
cleanupErr = qualificationOperation(cleanupErr, stageNFTDelete)
}
if !joined {
select {
case e := <-worker.done:
joined = true
cleanupErr = errors.Join(cleanupErr, qualificationOperation(errors.Join(e, errors.New("asset worker exited unexpectedly")), stageStopJoin))
default:
}
}
if !joined {
cleanupErr = errors.Join(cleanupErr, qualificationOperation(worker.stop(), stageStopJoin))
select {
case e := <-worker.done:
joined = true
if e != nil {
var exit *exec.ExitError
if !errors.As(e, &exit) || !exit.ProcessState.Sys().(syscall.WaitStatus).Signaled() || exit.ProcessState.Sys().(syscall.WaitStatus).Signal() != syscall.SIGKILL {
cleanupErr = errors.Join(cleanupErr, qualificationOperation(e, stageStopJoin))
}
}
case <-cleanup.Done():
cleanupErr = errors.Join(cleanupErr, qualificationOperation(errors.New("asset worker join timeout"), stageStopJoin))
}
}
if cleanupErr != nil {
err = &managedRecoveryFatal{Err: errors.Join(err, &managedAssetCleanupFailure{Err: cleanupErr})}
}
}()
readyCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
tick := time.NewTicker(25 * time.Millisecond)
defer tick.Stop()
for {
select {
case e := <-worker.done:
joined = true
return &managedRecoveryFatal{Err: qualificationOperation(errors.Join(e, errors.New("asset worker exited before READY")), stageWorkerExit)}
default:
}
ready, e := r.queue(worker.pid)
if e != nil {
return &managedRecoveryFatal{Err: qualificationOperation(e, stageWorkerQueue)}
}
if ready {
break
}
select {
case <-readyCtx.Done():
if ctx.Err() != nil {
return ctx.Err()
}
return &managedRecoveryFatal{Err: qualificationOperation(errors.New("asset queue READY timeout"), stageWorkerQueue)}
case <-tick.C:
}
}
// nft's transaction is atomic; failed creation owns no state to delete.
if _, e := r.nft(ctx, managedAssetRules); e != nil {
return &managedRecoveryFatal{Err: e}
}
table = true
// Recheck ownership after installing the queue and before the first capture.
if ready, e := r.queue(worker.pid); e != nil || !ready {
return &managedRecoveryFatal{Err: qualificationOperation(errors.Join(e, errors.New("owned queue lost before capture")), stageWorkerQueue)}
}
probeCtx, stopProbe := context.WithCancel(ctx)
watchDone := make(chan struct{})
exited := make(chan error, 1)
go func() {
defer close(watchDone)
select {
case e := <-worker.done:
exited <- qualificationOperation(errors.Join(e, errors.New("asset worker exited during probe")), stageWorkerExit)
stopProbe()
case <-probeCtx.Done():
}
}()
defer func() {
stopProbe()
<-watchDone
select {
case e := <-exited:
joined = true
err = &managedRecoveryFatal{Err: errors.Join(err, e)}
default:
}
}()
return capture(probeCtx)
}
func managedProbeWorkerArgs(endpoints []string, strategyArgs string) []string {
args := []string{"--qnum=200", "--fwmark=0x10000000", "--lua-init=@/opt/zapret2/lua/zapret-lib.lua", "--lua-init=@/opt/zapret2/lua/zapret-antidpi.lua", "--filter-tcp=443", "--hostlist-domains=" + strings.Join(endpoints, ",")}
return append(args, strings.Fields(strategyArgs)...)
}
// Collector errors conservatively carry a fatal wrapper. Only known transport
// failures and genuine rejected responses are nonpasses; malformed evidence,
// missing executables, unknown errors and worker failures remain infrastructure.
func managedAssetTransferNonpass(err error) bool {
if err == nil {
return false
}
switch e := err.(type) {
case *managedRecoveryFatal:
return false
case *resourceError:
return managedResourceTransferNonpass(e, false, true)
case imageBodyRejection:
return true
case *resourcePeerTLSRejection:
return true
case *exec.ExitError:
switch e.ExitCode() {
case 6, 28, 35, 51, 52, 55, 56, 60:
return true
}
return false
case interface{ Unwrap() []error }:
children := e.Unwrap()
if len(children) == 0 {
return false
}
for _, child := range children {
if !managedAssetTransferNonpass(child) {
return false
}
}
return true
case interface{ Unwrap() error }:
return managedAssetTransferNonpass(e.Unwrap())
default:
return err == context.DeadlineExceeded || err.Error() == "resource origin verification failed" || err.Error() == "image HTTP status"
}
}
@@ -0,0 +1,429 @@
package main
import (
"bytes"
"context"
"errors"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
)
func assetRuntimeFixture(t *testing.T) (managedAssetRuntime, *[]string, chan error) {
t.Helper()
events := []string{}
done := make(chan error, 1)
r := managedAssetRuntime{proxy: "socks5://127.0.0.1:1080", guard: func() error { return nil }}
r.nft = func(_ context.Context, input string) (string, error) {
if input == "list tables\n" {
events = append(events, "inventory")
return "", nil
}
if strings.HasPrefix(input, "delete") {
events = append(events, "delete")
return "", nil
}
if len(events) == 0 || events[len(events)-1] != "ready" {
t.Fatal("queue activated before READY", events)
}
if !strings.Contains(input, "hook input") || !strings.Contains(input, "hook output") || !strings.Contains(input, "meta mark & 0x10000000 == 0") {
t.Fatal(input)
}
events = append(events, "activate")
return "", nil
}
r.queue = func(pid int) (bool, error) {
if pid == 0 {
return false, nil
}
events = append(events, "ready")
return true, nil
}
r.start = func(args []string) (managedAssetWorker, error) {
if strings.Contains(strings.Join(args, " "), "--daemon") {
t.Fatal(args)
}
events = append(events, "start")
return managedAssetWorker{pid: 123, done: done, stop: func() error { events = append(events, "join"); done <- nil; return nil }}, nil
}
r.curl = func(_ context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
events = append(events, "curl")
config, err := io.ReadAll(stdin)
if err != nil {
return err
}
raw, err := strconv.Unquote(strings.TrimSuffix(strings.TrimPrefix(string(config), "url = "), "\n"))
if err != nil {
return err
}
var diag bytes.Buffer
resourceWire(raw, "", out, &diag)
tls := diag.String()
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
tls = strings.ReplaceAll(tls, "TLSv1.2", "TLSv1.3")
}
_, err = io.WriteString(diagnostic, tls)
return err
}
return r, &events, done
}
func TestManagedAssetTargetQueueOwnershipCapability(t *testing.T) {
// nfqws drops UID and becomes nondumpable. The queue ownership check must
// still be able to read its socket symlinks, without sharing host processes.
raw, err := os.ReadFile("../../deploy/managed-target/compose.yaml")
if err != nil {
t.Fatal(err)
}
manifest := string(raw)
_, capabilities, ok := strings.Cut(manifest, "cap_add: [")
if !ok {
t.Fatal("target capabilities missing")
}
capabilities, _, ok = strings.Cut(capabilities, "]")
if !ok || !strings.Contains(","+strings.ReplaceAll(capabilities, " ", "")+",", ",SYS_PTRACE,") {
t.Fatal("queue socket ownership requires SYS_PTRACE for dropped-UID workers")
}
if !strings.Contains(manifest, "privileged: false") || strings.Contains(manifest, "pid:") {
t.Fatal("queue inspection must remain confined to an unprivileged private PID namespace")
}
}
func TestManagedAssetCapabilityGuard(t *testing.T) {
const required uint64 = 1<<5 | 1<<12 | 1<<19
for _, bits := range []uint64{required, required | 1<<13, required &^ (1 << 5), required &^ (1 << 12), required &^ (1 << 19), 0} {
err := managedAssetCapabilityGuard("CapEff:\t" + strconv.FormatUint(bits, 16) + "\n")
if (err == nil) != (bits&required == required) {
t.Fatalf("capabilities %x: %v", bits, err)
}
}
for _, status := range []string{"", "CapEff: invalid", "CapPrm: 81020"} {
if err := managedAssetCapabilityGuard(status); err == nil {
t.Fatalf("accepted %q", status)
}
}
}
func TestManagedAssetQueueFDReady(t *testing.T) {
dir := t.TempDir()
for _, fd := range []string{"1", "2"} {
if err := os.WriteFile(filepath.Join(dir, fd), nil, 0600); err != nil {
t.Fatal(err)
}
}
ioErr := errors.New("proc read failure")
for _, tc := range []struct {
name string
firstErr error
secondLink string
wantReady bool
wantErr string
}{
{"foreign", nil, "socket:[other]", false, "asset queue belongs to another process"},
{"denied", os.ErrPermission, "socket:[other]", false, "asset queue ownership inspection denied"},
{"missing transient", os.ErrNotExist, "socket:[owned]", true, ""},
{"match despite denial", os.ErrPermission, "socket:[owned]", true, ""},
{"missing foreign", os.ErrNotExist, "socket:[other]", false, "asset queue belongs to another process"},
{"read failure", ioErr, "socket:[other]", false, ioErr.Error()},
} {
t.Run(tc.name, func(t *testing.T) {
ready, err := managedAssetQueueFDReady(dir, map[string]bool{"socket:[owned]": true}, func(path string) (string, error) {
if filepath.Base(path) == "1" {
return "socket:[other]", tc.firstErr
}
return tc.secondLink, nil
})
if ready != tc.wantReady || (err == nil) != (tc.wantErr == "") || err != nil && err.Error() != tc.wantErr {
t.Fatalf("ready=%v err=%v", ready, err)
}
})
}
if _, err := managedAssetQueueFDReady(filepath.Join(dir, "absent"), nil, os.Readlink); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("directory error lost: %v", err)
}
}
func TestManagedAssetRuntimeOwnedCollector(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
if err := d.transaction.Apply(d.db, []int64{front}); err != nil {
t.Fatal(err)
}
before := map[string][]byte{}
entries, err := os.ReadDir(d.transaction.Dir)
if err != nil {
t.Fatal(err)
}
for _, entry := range entries {
if entry.IsDir() {
continue
}
data, err := os.ReadFile(filepath.Join(d.transaction.Dir, entry.Name()))
if err != nil {
t.Fatal(err)
}
before[entry.Name()] = data
}
r, events, _ := assetRuntimeFixture(t)
c := storedCandidate{Group: "assets", Endpoints: []string{"i.ytimg.com", "yt3.ggpht.com"}, Args: managedAssetSeeds[0]}
m, err := r.verify(context.Background(), c, managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image", "yt3.ggpht.com": "https://yt3.ggpht.com/image"}})
if err != nil || len(m) != 4 {
t.Fatal(m, err)
}
count := 0
for _, e := range *events {
if e == "curl" {
count++
}
}
if count != 12 || strings.Join((*events)[len(*events)-2:], ",") != "delete,join" {
t.Fatal(*events)
}
for _, measurement := range m {
if measurement.Args != c.Args {
t.Fatal("unbound strategy")
}
if _, _, err := validateAssetTransfers(measurement); err != nil {
t.Fatal(err)
}
}
for name, data := range before {
after, err := os.ReadFile(filepath.Join(d.transaction.Dir, name))
if err != nil || !bytes.Equal(data, after) {
t.Fatal("probe mutated managed artifact", name, err)
}
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
}
func TestManagedAssetRuntimeFailedAtomicNFTInstallNeverDeletesForeignTable(t *testing.T) {
for _, mode := range []string{"preexisting", "raced-create", "install-failure"} {
t.Run(mode, func(t *testing.T) {
r, events, _ := assetRuntimeFixture(t)
r.nft = func(_ context.Context, input string) (string, error) {
if input == "list tables\n" {
if mode == "preexisting" {
return "table inet " + managedAssetTable, nil
}
return "", nil
}
if strings.HasPrefix(input, "delete") {
t.Fatal("deleted foreign table after failed atomic install")
}
if !strings.HasPrefix(input, "create table inet "+managedAssetTable+"\n") {
t.Fatal("table creation must reject racing existing table", input)
}
return "", errors.New(mode)
}
c := storedCandidate{Group: "assets", Endpoints: []string{"i.ytimg.com"}, Args: managedAssetSeeds[0]}
m, err := r.verify(context.Background(), c, managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image"}})
var fatal *managedRecoveryFatal
if len(m) != 0 || !errors.As(err, &fatal) || errors.Is(err, errCandidateNonpass) {
t.Fatal(m, err)
}
if mode == "preexisting" {
if len(*events) != 0 {
t.Fatal("started worker over preexisting table", *events)
}
} else if strings.Join(*events, ",") != "start,ready,join" {
t.Fatal("failed install did not join owned worker", *events)
}
})
}
}
func TestManagedAssetRuntimeCancelFailureAndCleanup(t *testing.T) {
for _, mode := range []string{"cancel-ready", "cancel-curl", "worker", "nonpass", "cleanup"} {
t.Run(mode, func(t *testing.T) {
r, events, done := assetRuntimeFixture(t)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
if mode == "cancel-ready" {
r.queue = func(pid int) (bool, error) {
if pid != 0 {
cancel()
}
return false, nil
}
}
if mode == "cancel-curl" {
r.curl = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
cancel()
return context.Canceled
}
}
if mode == "worker" {
r.curl = func(ctx context.Context, _ []string, _ io.Reader, _, _ io.Writer) error {
done <- errors.New("worker failure")
<-ctx.Done()
return ctx.Err()
}
}
if mode == "nonpass" {
r.curl = func(_ context.Context, _ []string, _ io.Reader, out, diagnostic io.Writer) error {
resourceWire("https://i.ytimg.com/image", "cn", out, diagnostic)
return nil
}
}
if mode == "cleanup" {
nft := r.nft
r.nft = func(ctx context.Context, input string) (string, error) {
out, err := nft(ctx, input)
if strings.HasPrefix(input, "delete") {
err = errors.New("cleanup failed")
}
return out, err
}
}
c := storedCandidate{Group: "assets", Endpoints: []string{"i.ytimg.com"}, Args: managedAssetSeeds[0]}
m, err := r.verify(ctx, c, managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image"}})
if err == nil || len(m) != 0 {
t.Fatal(m, err)
}
var fatal *managedRecoveryFatal
if (mode == "worker" || mode == "cleanup") && !errors.As(err, &fatal) {
t.Fatal("missing fatal", err)
}
if mode == "nonpass" && !errors.Is(err, errCandidateNonpass) {
t.Fatal(err)
}
if strings.HasPrefix(mode, "cancel") && !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
if mode == "cancel-ready" && (*events)[len(*events)-1] != "join" {
t.Fatal(*events)
}
if mode != "cancel-ready" && !strings.Contains(strings.Join(*events, ","), "delete") {
t.Fatal(*events)
}
})
}
}
func TestManagedAssetSeedsFallbackAndNoQualification(t *testing.T) {
for _, pass := range []bool{true, false} {
t.Run(strconv.FormatBool(pass), func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
if err := d.transaction.Apply(d.db, []int64{front, avatar, ggc}); err != nil {
t.Fatal(err)
}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image"}}, nil
}
calls := 0
d.verifyAssets = func(_ context.Context, c storedCandidate, _ managedServiceInventory) ([]assetMeasurement, error) {
calls++
if !pass || calls == 1 {
return nil, errCandidateNonpass
}
return []assetMeasurement{unitImageMeasurement(t, "i.ytimg.com", "tls12", c.Args), unitImageMeasurement(t, "i.ytimg.com", "tls13", c.Args)}, nil
}
err := repairManagedAssets(context.Background(), d, "i.ytimg.com")
if !pass {
if !errors.Is(err, errNoQualifiedStrategy) || calls != len(managedAssetSeeds) {
t.Fatal(calls, err)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front, avatar, ggc})
} else {
if err != nil || calls != 2 {
t.Fatal(calls, err)
}
ids, err := d.transaction.ActiveCandidates(d.db)
if err != nil || len(ids) != 4 {
t.Fatal(ids, err)
}
for _, required := range []int64{front, avatar, ggc} {
found := false
for _, id := range ids {
found = found || id == required
}
if !found {
t.Fatal(ids)
}
}
}
})
}
}
func TestManagedAssetPartialOverlapAndStageHealth(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
old := storedCandidate{Group: "assets", Endpoints: []string{"i.ytimg.com", "yt3.ggpht.com"}, Args: "--lua-desync=fake"}
id, err := saveManagedAssetCandidate(d.db, old, []assetMeasurement{
unitImageMeasurement(t, "i.ytimg.com", "tls12", old.Args), unitImageMeasurement(t, "i.ytimg.com", "tls13", old.Args),
unitImageMeasurement(t, "yt3.ggpht.com", "tls12", old.Args), unitImageMeasurement(t, "yt3.ggpht.com", "tls13", old.Args),
})
if err != nil {
t.Fatal(err)
}
unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=multisplit", true)
if err := d.transaction.Apply(d.db, []int64{front, id}); err != nil {
t.Fatal(err)
}
r, _, _ := assetRuntimeFixture(t)
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image", "yt3.ggpht.com": "https://yt3.ggpht.com/image"}, proxy: r.proxy, runner: r.curl, resourceErr: &managedResourceFailure{"media", "unknown", nil}}, nil
}
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
d.health = func(context.Context) (string, error) {
t.Fatal("unrelated media gated asset stage")
return "DEGRADED", nil
}
d.verifyAssets = func(_ context.Context, c storedCandidate, _ managedServiceInventory) ([]assetMeasurement, error) {
if c.ID != 0 {
return nil, errCandidateNonpass
}
if len(c.Endpoints) != 2 {
t.Fatal("partial overlap lost sibling", c)
}
return []assetMeasurement{
unitImageMeasurement(t, "i.ytimg.com", "tls12", c.Args), unitImageMeasurement(t, "i.ytimg.com", "tls13", c.Args),
unitImageMeasurement(t, "yt3.ggpht.com", "tls12", c.Args), unitImageMeasurement(t, "yt3.ggpht.com", "tls13", c.Args),
}, nil
}
if err := repairManagedAssets(context.Background(), d, "i.ytimg.com"); err != nil {
t.Fatal(err)
}
ids, err := d.transaction.ActiveCandidates(d.db)
if err != nil || len(ids) != 2 {
t.Fatal(ids, err)
}
replacement, err := readStrategyCandidate(d.db, ids[1])
if err != nil || len(replacement.Endpoints) != 2 {
t.Fatal(replacement, err)
}
}
func TestManagedAssetMalformedAndUnsafeCleanupNeverRestore(t *testing.T) {
for _, unsafe := range []bool{false, true} {
d, front, events := recoveryFixture(t, true)
if err := d.transaction.Apply(d.db, []int64{front}); err != nil {
t.Fatal(err)
}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image"}}, nil
}
calls := 0
d.verifyAssets = func(context.Context, storedCandidate, managedServiceInventory) ([]assetMeasurement, error) {
calls++
err := errors.New("malformed collector metrics")
if unsafe {
return nil, &managedRecoveryFatal{Err: &managedAssetCleanupFailure{Err: err}}
}
return nil, &managedRecoveryFatal{Err: err}
}
err := repairManagedAssets(context.Background(), d, "i.ytimg.com")
var fatal *managedRecoveryFatal
if !errors.As(err, &fatal) || errors.Is(err, errNoQualifiedStrategy) || calls != 1 {
t.Fatal(calls, err)
}
if unsafe && (*events)[len(*events)-1] != "stop" {
t.Fatal("unsafe restoration", *events)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
}
}
@@ -0,0 +1,541 @@
package main
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"io"
"time"
)
var errManagedAssetVerifierUnwired = errors.New("owned temporary asset verifier unavailable")
// Dependency-only construction fails closed. The production monitor adapter
// replaces this callback with newManagedAssetVerifier, never a root/404 probe.
func unwiredManagedAssetVerifier(context.Context, storedCandidate, managedServiceInventory) ([]assetMeasurement, error) {
return nil, errManagedAssetVerifierUnwired
}
type managedResourceFailure struct {
ProbeKind, Host string
Err error
}
func (e *managedResourceFailure) Error() string { return e.PublicError() }
func (e *managedResourceFailure) PublicError() string {
host := e.Host
if !discoveryHostname(host) {
host = "unknown"
}
kind := e.ProbeKind
if kind != "thumbnail" && kind != "avatar" && kind != "media" && kind != "front" {
kind = "unknown"
}
return "DEGRADED: " + kind + " " + host
}
func (e *managedResourceFailure) Unwrap() error { return e.Err }
// Only a typed collector's complete known-negative error set may lose its
// generic fatal wrapper at a live-health boundary. Missing resources remain
// DEGRADED; joined guard/infrastructure/cancellation failures retain their type.
func managedResourceHealthError(ctx context.Context, err error) error {
// Only a direct, known parser absence is missing inventory. Never flatten
// joined guard/infrastructure failures into an unrelated image absence.
if ctx.Err() == nil {
if failure, ok := err.(*managedResourceFailure); ok && failure.ProbeKind == "avatar" && failure.Host == "yt3.ggpht.com" {
if code, ok := failure.Err.(avatarMetadataError); ok && (code == "markerAbsent" || code == "jsonInvalid" || code == "noAllowedAvatar") {
return &managedResourceFailure{ProbeKind: failure.ProbeKind, Host: failure.Host}
}
}
}
var failure *managedResourceFailure
var collector *resourceError
if ctx.Err() == nil && errors.As(err, &failure) && (failure.ProbeKind == "media" || assetEndpoint(failure.Host) && failure.ProbeKind == managedImageProbeKind(failure.Host)) && errors.As(failure.Err, &collector) && managedResourceTransferNonpass(err, assetEndpoint(failure.Host), assetEndpoint(failure.Host)) {
return &managedResourceFailure{ProbeKind: failure.ProbeKind, Host: failure.Host}
}
return managedRecoveryContextError(ctx, err)
}
// Ephemeral inventory: never serialize URLs, page bodies or Player responses.
type managedServiceInventory struct {
images map[string]string
media mediaResource
resourceErr error
proxy string
runner subprocessRunner
}
func managedServiceResources(cfg healthConfig, player func(context.Context) ([]byte, error), page playerRunner) func(context.Context) (managedServiceInventory, error) {
return func(ctx context.Context) (managedServiceInventory, error) {
i := managedServiceInventory{images: make(map[string]string), proxy: cfg.SOCKSProxy, runner: runResourceCurl}
if _, err := requiredYoutubeAdapter(); err != nil {
return i, err
}
if err := ctx.Err(); err != nil {
return i, managedRecoveryContextError(ctx, err)
}
proxy, err := managedProbeProxy(cfg.SOCKSProxy)
if err != nil {
return i, &managedRecoveryFatal{Err: err}
}
i.proxy = proxy
body, err := player(ctx)
if err != nil {
return i, &managedResourceFailure{"front", managedFrontendHost, err}
}
if err := ctx.Err(); err != nil {
return i, managedRecoveryContextError(ctx, err)
}
var p struct {
VideoDetails struct {
Thumbnail struct {
Thumbnails []struct {
URL string `json:"url"`
} `json:"thumbnails"`
} `json:"thumbnail"`
} `json:"videoDetails"`
}
if json.Unmarshal(body, &p) == nil {
for _, t := range p.VideoDetails.Thumbnail.Thumbnails {
if _, e := privateResourceURL(t.URL, "i.ytimg.com"); e == nil {
i.images["i.ytimg.com"] = t.URL
break
}
}
}
i.media, _, err = playerMediaResource(body)
if err != nil {
if errors.Is(err, errServiceAdapterUnavailable) {
return i, err
}
i.resourceErr = &managedResourceFailure{"media", "unknown", err}
}
for attempt := 0; attempt < 2; attempt++ {
if err := ctx.Err(); err != nil {
return i, managedRecoveryContextError(ctx, err)
}
origin := "https://www.youtube.com/"
if attempt == 1 {
if !videoIDPattern.MatchString(cfg.VideoID) {
return i, &managedRecoveryFatal{Err: errors.New("invalid video ID")}
}
origin = "https://www.youtube.com/watch?v=" + cfg.VideoID
}
bounded, cancel := context.WithTimeout(ctx, 35*time.Second)
var output bytes.Buffer
out := &playerBoundedWriter{dst: &output, limit: 5 << 20}
// Fixed trusted WWW origin; no redirects, credentials or diagnostics.
err = page(bounded, []string{"--disable", "--silent", "--show-error", "--fail", "--noproxy", "", "--proxy", proxy, "--proto", "=https", "--max-time", "30", "--max-filesize", "5242880", origin}, nil, out, io.Discard)
requestErr := bounded.Err()
cancel()
if ctx.Err() != nil {
return i, managedRecoveryContextError(ctx, err)
}
if err != nil || requestErr != nil || out.exceeded {
cause := errors.Join(err, requestErr)
if out.exceeded {
cause = errors.Join(cause, errors.New("fresh metadata exceeds limit"))
}
return i, managedResourceHealthError(ctx, &managedResourceFailure{"avatar", "yt3.ggpht.com", resourceFailure(ctx, cause)})
}
raw, parseErr := avatarResourceFromPage(output.Bytes())
if parseErr == nil {
i.images["yt3.ggpht.com"] = raw
return i, nil
}
// Only missing structural metadata permits one fresh watch request.
var missing avatarMetadataError
if attempt == 1 || !errors.As(parseErr, &missing) || (missing != "markerAbsent" && missing != "noAllowedAvatar") {
return i, managedResourceHealthError(ctx, &managedResourceFailure{"avatar", "yt3.ggpht.com", parseErr})
}
}
return i, nil
}
}
func managedServiceHealth(proxy string, front func(context.Context) (string, error), inventory func(context.Context) (managedServiceInventory, error), runner subprocessRunner) func(context.Context) (string, error) {
return func(ctx context.Context) (string, error) {
if _, err := requiredYoutubeAdapter(); err != nil {
return "DEGRADED", err
}
status, err := front(ctx)
if err != nil || status != "OK" {
return "DEGRADED", &managedResourceFailure{"front", managedFrontendHost, err}
}
i, err := inventory(ctx)
var fatal *managedRecoveryFatal
if ctx.Err() != nil || errors.As(err, &fatal) {
return "DEGRADED", err
}
missing := err
for _, host := range []string{"i.ytimg.com", "yt3.ggpht.com"} {
kind := "thumbnail"
if host == "yt3.ggpht.com" {
kind = "avatar"
}
if i.images[host] == "" {
if missing == nil {
missing = &managedResourceFailure{kind, host, nil}
}
continue
}
for _, protocol := range []string{"tls12", "tls13"} {
if _, err := collectAssetMeasurement(ctx, proxy, i.images[host], host, protocol, "", runner); err != nil {
// At the live-health boundary, known bad image bodies/origins
// are resource damage. Preserve mixed guard/cancellation failures.
if ctx.Err() == nil && !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) && managedAssetHealthNonpass(err) {
err = nil
}
return "DEGRADED", &managedResourceFailure{kind, host, err}
}
}
}
if missing != nil {
return "DEGRADED", missing
}
if i.resourceErr != nil {
return "DEGRADED", i.resourceErr
}
if i.media.url == "" {
return "DEGRADED", &managedResourceFailure{"media", "unknown", nil}
} // SABR unknown, not ready.
for _, protocol := range []string{"tls12", "tls13"} {
if _, err := collectMediaSamples(ctx, proxy, i.media, protocol, runner); err != nil {
return "DEGRADED", &managedResourceFailure{"media", i.media.host, err}
}
}
return "OK", nil
}
}
func managedFullDesiredSet(db *sql.DB, current, replacements []int64) ([]int64, error) {
replaceGroups := map[string]bool{}
for _, id := range replacements {
c, err := readStrategyCandidate(db, id)
if err != nil {
return nil, err
}
replaceGroups[c.Group] = true
}
var desired []int64
for _, id := range current {
c, err := readStrategyCandidate(db, id)
if err != nil {
return nil, err
}
if !replaceGroups[c.Group] {
desired = append(desired, id)
}
}
return append(desired, replacements...), nil
}
func repairManagedAssets(ctx context.Context, d managedRecoveryDeps, host string) (err error) {
if d.resources == nil || d.verifyAssets == nil {
return errManagedAssetVerifierUnwired
}
i, err := d.resources(ctx)
var fatal *managedRecoveryFatal
if ctx.Err() != nil || errors.As(err, &fatal) {
return managedRecoveryContextError(ctx, err)
}
if err != nil && i.images[host] == "" {
return err
}
if i.images[host] == "" {
return &managedResourceFailure{managedImageProbeKind(host), host, nil}
}
m := *d.transaction
current, err := m.ActiveCandidates(d.db)
if err != nil {
return &managedRecoveryFatal{Err: err}
}
saved, err := listQualifiedCandidates(d.db, "assets", 10)
if err != nil {
return &managedRecoveryFatal{Err: err}
}
// Saved definitions first, then a bounded readonly empirical seed inventory.
// Include overlapping siblings in seeds; never discard a partial host set.
endpoints := []string{host}
for _, id := range current {
old, e := readStrategyCandidate(d.db, id)
if e != nil {
return &managedRecoveryFatal{Err: e}
}
if old.Group == "assets" {
for _, a := range old.Endpoints {
if a == host {
endpoints = append([]string(nil), old.Endpoints...)
}
}
}
}
for _, args := range managedAssetSeeds {
saved = append(saved, storedCandidate{Group: "assets", Endpoints: endpoints, Args: args})
}
for _, c := range saved {
covers := false
for _, endpoint := range c.Endpoints {
covers = covers || endpoint == host
}
if !covers {
continue
}
var desired []int64
for _, id := range current {
old, e := readStrategyCandidate(d.db, id)
if e != nil {
return &managedRecoveryFatal{Err: e}
}
overlap := false
for _, a := range old.Endpoints {
for _, b := range c.Endpoints {
overlap = overlap || a == b
}
}
if old.Group != "assets" || !overlap {
desired = append(desired, id)
}
// A partially overlapping multi-host candidate cannot lose its sibling.
if old.Group == "assets" && overlap {
for _, a := range old.Endpoints {
found := false
for _, b := range c.Endpoints {
found = found || a == b
}
if !found {
covers = false
}
}
}
}
if !covers {
continue
}
if e := (&candidate{Args: c.Args}).validateArgs(); e != nil {
return &managedRecoveryFatal{Err: e}
}
// All temporary workers must be joined by Verify before restoration/apply.
e := d.run(ctx, managedZapretScript, "stop")
var measurements []assetMeasurement
if e == nil {
measurements, e = d.verifyAssets(ctx, c, i)
}
var unsafeCleanup *managedAssetCleanupFailure
if errors.As(e, &unsafeCleanup) {
// Never reactivate the permanent service over an unjoined worker or
// residual owned queue. The fatal monitor exit retains last-good files.
return e
}
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
restoreErr := d.run(cleanup, managedZapretScript, "start")
cancel()
if restoreErr != nil {
return &managedRecoveryFatal{Err: errors.Join(e, restoreErr)}
}
if e != nil {
if errors.As(e, &fatal) {
return e
}
if ctx.Err() != nil || errors.Is(e, context.Canceled) {
return managedRecoveryContextError(ctx, e)
}
if errors.Is(e, errCandidateNonpass) {
continue
}
return e
}
if e = ctx.Err(); e != nil {
return e
}
id, e := saveManagedAssetCandidate(d.db, c, measurements)
if e != nil {
return &managedRecoveryFatal{Err: e}
}
m.Health = func() (string, error) {
// Asset-stage apply must not be vetoed by unrelated media damage.
// Global service health still gates READY after recovery.
var status string
var healthErr error
if d.frontHealth == nil {
status, healthErr = d.health(ctx)
} else {
status, healthErr = managedAssetApplyHealth(ctx, d, c.Endpoints)
}
var failure *managedResourceFailure
var fatal *managedRecoveryFatal
if ctx.Err() != nil {
return "DEGRADED", managedRecoveryContextError(ctx, healthErr)
}
// Apply treats a non-OK status with no error as a rejection. Never
// erase a safety failure or cancellation joined to ordinary damage.
if !errors.As(healthErr, &fatal) && !errors.Is(healthErr, context.Canceled) && !errors.Is(healthErr, context.DeadlineExceeded) && errors.As(healthErr, &failure) {
return "DEGRADED", nil
}
return status, healthErr
}
m.Reload = func() error {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
return d.run(cleanup, managedZapretScript, "restart")
}
e = m.Apply(d.db, append(desired, id))
if e == nil {
return nil
}
var rejection *managedHealthRejection
var rollback *managedRollbackError
if errors.As(e, &fatal) || errors.As(e, &rollback) {
return &managedRecoveryFatal{Err: e}
}
if ctx.Err() != nil || errors.Is(e, context.Canceled) || errors.Is(e, context.DeadlineExceeded) {
return managedRecoveryContextError(ctx, e)
}
if !errors.As(e, &rejection) {
return &managedRecoveryFatal{Err: e}
}
}
return errNoQualifiedStrategy
}
var managedAssetSeeds = [...]string{
"--payload=tls_client_hello --lua-desync=fake:blob=fake_default_tls:tcp_ts=-1000:repeats=1 --lua-desync=multidisorder:pos=1,midsld",
"--payload=tls_client_hello --lua-desync=hostfakesplit:disorder_after:tcp_ts=-1000:repeats=1 --lua-desync=multidisorder:pos=1,midsld",
"--payload=tls_client_hello --lua-desync=multidisorder:pos=1,midsld",
"--payload=tls_client_hello --lua-desync=hostfakesplit:disorder_after:tcp_md5:repeats=1 --lua-desync=multidisorder:pos=1,midsld",
"--payload=tls_client_hello --lua-desync=multisplit:pos=1,midsld",
}
func managedAssetApplyHealth(ctx context.Context, d managedRecoveryDeps, endpoints []string) (string, error) {
status, err := d.frontHealth(ctx)
if err != nil || status != "OK" {
return "DEGRADED", err
}
i, err := d.resources(ctx)
var fatal *managedRecoveryFatal
if ctx.Err() != nil || errors.As(err, &fatal) {
return "DEGRADED", managedRecoveryContextError(ctx, err)
}
if err != nil {
for _, host := range endpoints {
if i.images[host] == "" {
return "DEGRADED", err
}
}
}
// Production's normal service collector uses the same guarded SOCKS proxy.
// An adapter installs this callback with a candidate-independent collector.
if i.resourceErr != nil && len(i.images) == 0 {
return "DEGRADED", i.resourceErr
}
return managedAssetInventoryHealth(ctx, i, endpoints)
}
func managedAssetInventoryHealth(ctx context.Context, i managedServiceInventory, endpoints []string) (string, error) {
if i.runner == nil || i.proxy == "" {
return "DEGRADED", errors.New("asset health collector unavailable")
}
for _, host := range endpoints {
if i.images[host] == "" {
return "DEGRADED", &managedResourceFailure{managedImageProbeKind(host), host, nil}
}
for _, protocol := range []string{"tls12", "tls13"} {
if _, err := collectAssetMeasurement(ctx, i.proxy, i.images[host], host, protocol, "", i.runner); err != nil {
if ctx.Err() != nil {
return "DEGRADED", managedRecoveryContextError(ctx, err)
}
if managedAssetHealthNonpass(err) {
return "DEGRADED", nil
}
return "DEGRADED", &managedResourceFailure{managedImageProbeKind(host), host, err}
}
}
}
return "OK", nil
}
// Collector wrappers are conservatively fatal. At the post-activation health
// boundary, rejected origins/bodies are ordinary damage, not infrastructure.
// Every joined leaf must be a known nonpass; mixed safety failures stay fatal.
func managedAssetHealthNonpass(err error) bool {
switch e := err.(type) {
case *managedRecoveryFatal:
return false
case *resourceError:
cause := e.cause
if fatal, ok := cause.(*managedRecoveryFatal); ok {
cause = fatal.Err
}
return managedAssetHealthNonpass(cause)
case interface{ Unwrap() []error }:
children := e.Unwrap()
if len(children) == 0 {
return false
}
for _, child := range children {
if !managedAssetHealthNonpass(child) {
return false
}
}
return true
case interface{ Unwrap() error }:
return managedAssetHealthNonpass(e.Unwrap())
default:
if err == nil || err == context.Canceled {
return false
}
return managedAssetTransferNonpass(err)
}
}
func managedImageProbeKind(host string) string {
if host == "i.ytimg.com" {
return "thumbnail"
}
if host == "yt3.ggpht.com" {
return "avatar"
}
return "unknown"
}
func saveManagedAssetCandidate(db *sql.DB, c storedCandidate, measurements []assetMeasurement) (int64, error) {
if len(measurements) != 2*len(c.Endpoints) {
return 0, fmt.Errorf("incomplete asset verification matrix")
}
seen := map[string]bool{}
for _, m := range measurements {
member := false
for _, host := range c.Endpoints {
member = member || host == m.Endpoint
}
key := m.Endpoint + "/" + m.Protocol
if !member || seen[key] || m.Args != c.Args {
return 0, fmt.Errorf("asset verification matrix definition mismatch")
}
seen[key] = true
if _, _, err := validateAssetTransfers(m); err != nil {
return 0, err
}
}
tx, err := db.Begin()
if err != nil {
return 0, err
}
defer tx.Rollback()
id, err := insertStrategyCandidate(tx, "assets", c.Endpoints, c.Args, nil)
if err != nil {
return 0, err
}
for _, m := range measurements {
if err := saveAssetEvidence(tx, id, m, time.Now()); err != nil {
return 0, err
}
}
if err := tx.Commit(); err != nil {
return 0, err
}
return id, nil
}
@@ -0,0 +1,173 @@
package main
import (
"context"
"errors"
"io"
"os/exec"
"strings"
"testing"
)
func TestAvatarPageClassification(t *testing.T) {
for _, tc := range []struct{ body, code string }{
{`<html></html>`, "markerAbsent"},
{`var ytInitialData = {broken};`, "jsonInvalid"},
{`var ytInitialData = {"url":"https://yt3.ggpht.com.attacker.invalid/avatar"};`, "noAllowedAvatar"},
{`var ytInitialData = {"url":"http://yt3.ggpht.com/avatar"};`, "noAllowedAvatar"},
{`var ytInitialData = {"url":"https://user@yt3.ggpht.com/avatar"};`, "noAllowedAvatar"},
{`var ytInitialData = {"url":"https://yt3.ggpht.com/avatar?private=fixture"};`, "valid"},
} {
t.Run(tc.code, func(t *testing.T) {
raw, err := avatarResourceFromPage([]byte(tc.body))
if tc.code == "valid" {
if err != nil || raw == "" {
t.Fatal("missing validated resource", err)
}
return
}
var classified avatarMetadataError
if raw != "" || !errors.As(err, &classified) || string(classified) != tc.code {
t.Fatal("unexpected classification", err)
}
})
}
}
func TestManagedAvatarInventoryFallback(t *testing.T) {
exit := func(code string) error { return exec.Command("sh", "-c", "exit "+code).Run() }
for _, tc := range []struct {
name, home, watch, video string
pageErr, watchErr error
calls int
avatar, fatal bool
}{
{name: "homeNoAvatarWatchHasValidatedURL", home: `var ytInitialData = {};`, watch: `var ytInitialData = {"url":"https://yt3.ggpht.com/fresh?private=fixture"};`, calls: 2, avatar: true},
{name: "homeAlreadyValid", home: `var ytInitialData = {"url":"https://yt3.ggpht.com/fresh"};`, calls: 1, avatar: true},
{name: "bothMissing", home: `var ytInitialData = {};`, watch: `<html></html>`, calls: 2},
{name: "malformedHome", home: `var ytInitialData = {bad};`, calls: 1},
{name: "malformedWatch", home: `var ytInitialData = {};`, watch: `var ytInitialData = {bad};`, calls: 2},
{name: "watchHTTPMissing", home: `var ytInitialData = {};`, watchErr: exit("22"), calls: 2},
{name: "watchTimeout", home: `var ytInitialData = {};`, watchErr: exit("28"), calls: 2},
{name: "watchWrongCN", home: `var ytInitialData = {};`, watchErr: exit("60"), calls: 2},
{name: "watchMixedGuard", home: `var ytInitialData = {};`, watchErr: errors.Join(exit("28"), errors.New("owned proxy guard failed")), calls: 2, fatal: true},
{name: "untrustedWatchURL", watch: `var ytInitialData = {"url":"https://yt3.ggpht.com.evil.invalid/avatar"};`, calls: 2},
{name: "invalidVideo", video: "bad&injection", calls: 1, fatal: true},
{name: "timeoutNoFallback", pageErr: exit("28"), calls: 1},
{name: "wrongCNNoFallback", pageErr: exit("60"), calls: 1},
{name: "mixedGuardNoFallback", pageErr: errors.Join(exit("28"), errors.New("owned proxy guard failed")), calls: 1, fatal: true},
{name: "oversizedHomeNoFallback", home: strings.Repeat("x", (5<<20)+1), calls: 1, fatal: true},
} {
t.Run(tc.name, func(t *testing.T) {
cfg := healthConfig{SOCKSProxy: "socks5://127.0.0.1:11080", VideoID: "dQw4w9WgXcQ"}
if tc.video != "" {
cfg.VideoID = tc.video
}
calls := 0
resources := managedServiceResources(cfg, func(context.Context) ([]byte, error) {
return []byte(`{"playabilityStatus":{"status":"OK"},"videoDetails":{"thumbnail":{"thumbnails":[{"url":"https://i.ytimg.com/fresh"}]}},"streamingData":{"formats":[{"url":"https://fresh.googlevideo.com/private","mimeType":"video/mp4"}]}}`), nil
}, func(ctx context.Context, args []string, _ io.Reader, out, diagnostic io.Writer) error {
calls++
if _, ok := ctx.Deadline(); !ok || diagnostic != io.Discard {
t.Fatal("unbounded metadata or diagnostics")
}
joined := strings.Join(args, " ")
if strings.Contains(joined, "private=") || strings.Contains(joined, "--insecure") || strings.Contains(joined, "--location") || !strings.Contains(joined, "--proto =https") || !strings.Contains(joined, "--max-filesize 5242880") || !strings.Contains(joined, "--proxy socks5h://127.0.0.1:11080") {
t.Fatal("unsafe metadata request")
}
assertManagedDNSArgs(t, args, "socks5h://127.0.0.1:11080")
body, origin := tc.home, "https://www.youtube.com/"
if calls == 2 {
body, origin = tc.watch, "https://www.youtube.com/watch?v="+cfg.VideoID
}
if args[len(args)-1] != origin {
t.Fatal("unexpected metadata origin")
}
if tc.pageErr != nil {
return tc.pageErr
}
if calls == 2 && tc.watchErr != nil {
return tc.watchErr
}
_, err := io.WriteString(out, body)
return err
})
i, err := resources(context.Background())
var fatal *managedRecoveryFatal
if calls != tc.calls || (i.images["yt3.ggpht.com"] != "") != tc.avatar || errors.As(err, &fatal) != tc.fatal {
t.Fatal("fallback or guard contract", calls, err)
}
if (err == nil) != tc.avatar || i.images["i.ytimg.com"] == "" || i.media.url == "" || i.resourceErr != nil {
t.Fatal("missing avatar masked or partial inventory lost", err)
}
if err != nil && strings.Contains(err.Error(), "private") {
t.Fatal("private metadata leaked")
}
})
}
}
func TestManagedAvatarMetadataHealthClassification(t *testing.T) {
for _, code := range []avatarMetadataError{"markerAbsent", "jsonInvalid", "noAllowedAvatar"} {
failure := &managedResourceFailure{"avatar", "yt3.ggpht.com", code}
err := managedResourceHealthError(context.Background(), failure)
var missing *managedResourceFailure
if !errors.As(err, &missing) || missing.Err != nil {
t.Fatal("known metadata absence not normalized", code)
}
for _, mixed := range []error{
errors.Join(failure, errors.New("owned proxy guard failed")),
&managedResourceFailure{"avatar", "yt3.ggpht.com", errors.Join(code, errors.New("collector unavailable"))},
&managedResourceFailure{"avatar", "yt3.ggpht.com", errors.Join(code, context.Canceled)},
} {
err := managedResourceHealthError(context.Background(), mixed)
if !errors.As(err, &missing) || missing.Err == nil {
t.Fatal("mixed safety failure normalized", code)
}
}
}
unknown := avatarMetadataError("unknown")
err := managedResourceHealthError(context.Background(), &managedResourceFailure{"avatar", "yt3.ggpht.com", unknown})
if !errors.Is(err, unknown) {
t.Fatal("unknown parser error normalized")
}
}
func TestManagedAvatarInventoryCancellationNoFallback(t *testing.T) {
for _, timing := range []string{"before-player", "after-player", "during-home"} {
t.Run(timing, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
if timing == "before-player" {
cancel()
}
players, pages := 0, 0
resources := managedServiceResources(healthConfig{VideoID: "dQw4w9WgXcQ", SOCKSProxy: "socks5h://127.0.0.1:11080"}, func(context.Context) ([]byte, error) {
players++
if timing == "after-player" {
cancel()
}
return []byte(`{"playabilityStatus":{"status":"OK"}}`), nil
}, func(ctx context.Context, args []string, _ io.Reader, out, _ io.Writer) error {
pages++
assertManagedDNSArgs(t, args, "socks5h://127.0.0.1:11080")
if args[len(args)-1] != "https://www.youtube.com/" {
t.Fatal("watch fallback after cancellation", args)
}
cancel()
_, err := io.WriteString(out, `var ytInitialData = {};`)
return err
})
i, err := resources(ctx)
wantPlayers, wantPages := 1, 0
if timing == "before-player" {
wantPlayers = 0
} else if timing == "during-home" {
wantPages = 1
}
if players != wantPlayers || pages != wantPages || !errors.Is(err, context.Canceled) || i.images["yt3.ggpht.com"] != "" {
t.Fatal("cancellation lost or request after cancellation", players, pages, err)
}
})
}
}
+261 -54
View File
@@ -6,6 +6,7 @@ import (
"database/sql"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
@@ -30,6 +31,7 @@ type managedIdentity struct {
type managedDeps struct {
lookup func(string) (string, bool)
openDB func() (*sql.DB, error)
openWriteDB func(managedDeps) (*sql.DB, error)
player playerRunner
run func(string, ...string) error
script, directory string
@@ -41,6 +43,13 @@ func managedDependencies(player playerRunner) managedDeps {
return managedDeps{
lookup: os.LookupEnv, player: player,
openDB: func() (*sql.DB, error) { return openHistoryStore(databasePath()) },
openWriteDB: func(d managedDeps) (*sql.DB, error) {
path, err := managedMonitorDatabase(d)
if err != nil {
return nil, err
}
return openStore(path)
},
script: "/opt/zapret2/init.d/sysv/zapret2", directory: managedTargetDirectory,
verifyRuntime: verifyManagedRuntime, verifySOCKS: verifyManagedSOCKS,
run: func(path string, args ...string) error {
@@ -55,32 +64,82 @@ func managedDependencies(player playerRunner) managedDeps {
}
func managedArguments(args []string) (string, []int64, error) {
invalid := fmt.Errorf("managed accepts init, reconcile, status, active-ids, or apply with one or both candidate flags (full desired set); no path flags")
if len(args) == 1 && (args[0] == "init" || args[0] == "reconcile" || args[0] == "status" || args[0] == "active-ids") {
invalid := fmt.Errorf("managed accepts init, reconcile, status [--assets], active-ids, or apply with frontend/googlevideo/asset candidate flags (full desired set); no path flags")
if len(args) == 2 && args[0] == "status" && args[1] == "--assets" {
return "status", nil, nil
}
if len(args) == 1 && (args[0] == "init" || args[0] == "reconcile" || args[0] == "status" || args[0] == "active-ids" || args[0] == "experimental-request-check") {
return args[0], nil, nil
}
if (len(args) != 3 && len(args) != 5) || args[0] != "apply" {
if len(args) < 3 || len(args)%2 != 1 || args[0] != "apply" {
return "", nil, invalid
}
selected := map[string]int64{}
selected := map[string][]int64{}
seen := map[int64]bool{}
scopeSeen := false
experiment := false
request := false
var scope GooglevideoScope
for i := 1; i < len(args); i += 2 {
key := args[i]
if selected[key] != 0 || (key != "--frontend-candidate" && key != "--googlevideo-candidate") || args[i+1] == "" || strings.Trim(args[i+1], "0123456789") != "" {
if key == "--googlevideo-request" {
if request || args[i+1] != "startup" {
return "", nil, invalid
}
request = true
continue
}
if key == "--googlevideo-experiment" {
if experiment || args[i+1] != "native-circular" {
return "", nil, invalid
}
experiment = true
continue
}
if key == "--googlevideo-scope" {
scope = GooglevideoScope(args[i+1])
if scopeSeen || !scope.valid() || scope == "" {
return "", nil, invalid
}
scopeSeen = true
continue
}
if (len(selected[key]) != 0 && key != "--asset-candidate") || (key != "--frontend-candidate" && key != "--googlevideo-candidate" && key != "--asset-candidate") {
return "", nil, invalid
}
id, err := strconv.ParseInt(args[i+1], 10, 64)
if err != nil || id <= 0 {
values := strings.Split(args[i+1], ",")
if key != "--asset-candidate" && len(values) != 1 {
return "", nil, invalid
}
for _, value := range values {
if value == "" || strings.Trim(value, "0123456789") != "" {
return "", nil, invalid
}
id, err := strconv.ParseInt(value, 10, 64)
if err != nil || id <= 0 || seen[id] {
return "", nil, invalid
}
seen[id] = true
selected[key] = append(selected[key], id)
}
if len(selected["--asset-candidate"]) > 2 {
return "", nil, invalid
}
selected[key] = id
}
var ids []int64
for _, key := range []string{"--frontend-candidate", "--googlevideo-candidate"} {
if id := selected[key]; id != 0 {
ids = append(ids, id)
if experiment {
if !scopeSeen || scope != GooglevideoScopeExperimental || len(selected) != 0 {
return "", nil, invalid
}
return "experiment", nil, nil
}
if len(ids) == 2 && ids[0] == ids[1] {
if request || scope == GooglevideoScopeExperimental {
return "", nil, invalid
}
for _, key := range []string{"--frontend-candidate", "--googlevideo-candidate", "--asset-candidate"} {
ids = append(ids, selected[key]...)
}
if len(ids) == 0 || (scopeSeen && len(selected["--googlevideo-candidate"]) == 0) {
return "", nil, invalid
}
return "apply", ids, nil
@@ -196,7 +255,7 @@ func managedCheckIdentity(m *managedTransaction) error {
}
// Status and supervisor reads share the same locked, DB-validated snapshot.
func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB, error), activeOnly bool) error {
func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB, error), activeOnly bool, requestAssets ...bool) error {
return m.locked(func() error {
j, err := m.load()
if err != nil {
@@ -212,30 +271,57 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
if managedHash(b) != j.Hash && (j.State != "prepared" || managedHash(b) != j.Next) {
return errManagedDrift
}
if j.Version == 4 {
if activeOnly {
return fmt.Errorf("v4 deployment has no legacy active IDs")
}
if err := m.validateConfigRef(nil, j.Ref, j.Candidates, j.Hash, j.Scope); err != nil {
return err
}
if j.State == "prepared" {
if err := m.validateConfigRef(nil, j.PendingRef, j.PendingCandidates, j.Next, j.PendingScope); err != nil {
return err
}
}
_, err := fmt.Fprintf(out, "managed-test generation=%d hash=%s state=%s kind=%s\nLive resource health: NOT CHECKED; configured resources only, not production READY.\n", j.Generation, j.Hash, j.State, j.Ref.Kind)
return err
}
groups := map[string]bool{}
known := j.State == "committed" && j.Version == 2 && string(b) == managedBaseline && len(j.Candidates) == 0
if j.State == "committed" && j.Version == 2 && len(j.Candidates) != 0 {
operatorCoverage := map[string]bool{}
assetHosts := map[string]bool{}
known := j.State == "committed" && j.Version >= 2 && string(b) == managedBaseline && len(j.Candidates) == 0
if j.State == "committed" && j.Version >= 2 && len(j.Candidates) != 0 {
db, err := openDB()
if db != nil {
defer db.Close()
}
if err == nil && db != nil {
rendered, err := renderManagedConfig(db, j.Candidates)
rendered, err := renderStoredManagedConfigScope(db, j.Candidates, j.Scope)
if err == nil && managedHash(rendered) == j.Hash {
known = true
for _, id := range j.Candidates {
c, err := readStrategyCandidate(db, id)
if err != nil || !c.Qualified {
c, err := readStrategyCandidatePolicy(db, id, false)
if err != nil || (!c.Qualified && !experimentalGooglevideo(j.Scope, c)) {
known = false
break
}
groups[c.Group] = true
operatorCoverage[c.Group] = true
if experimentalGooglevideo(j.Scope, c) {
groups[c.Group] = false
}
if c.Group == "assets" {
for _, host := range c.Endpoints {
assetHosts[host] = true
}
}
}
}
}
}
if !known {
groups = map[string]bool{}
operatorCoverage = map[string]bool{}
// Unvalidated journal references are not active recovery candidates.
j.Candidates = nil
}
@@ -247,14 +333,35 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
missing = append(missing, group)
}
}
if groups["assets"] {
active = append(active, "assets")
}
if (groups["assets"] || (len(requestAssets) != 0 && requestAssets[0])) && (!known || len(assetHosts) != 2) {
missing = append(missing, "assets")
}
// Coverage labels active profiles, not qualification or the full host inventory.
covered := []string{}
for _, group := range []string{"frontend", "assets", "googlevideo"} {
if operatorCoverage[group] {
covered = append(covered, group)
}
}
scope := j.Scope
if !known {
scope = "unknown"
} else if operatorCoverage["googlevideo"] && scope == "" {
scope = GooglevideoScopeExact
}
if activeOnly {
if !known {
return fmt.Errorf("active candidate IDs unresolved; deployment preserved by hash")
}
return json.NewEncoder(out).Encode(struct {
IDs []int64 `json:"active_candidate_ids"`
Missing []string `json:"unresolved_groups"`
}{append([]int64{}, j.Candidates...), missing})
IDs []int64 `json:"active_candidate_ids"`
Missing []string `json:"unresolved_groups"`
Covered []string `json:"covered_groups"`
Scope GooglevideoScope `json:"googlevideo_scope"`
}{append([]int64{}, j.Candidates...), missing, covered, scope})
}
groupText := strings.Join(active, ",")
if groupText == "" {
@@ -267,30 +374,29 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
if known {
reportIDs = j.Candidates
}
_, err = fmt.Fprintf(out, "managed-test generation=%d hash=%s state=%s groups=%s active_candidate_ids=%v unresolved_groups=%s\nLive health: NOT CHECKED (Player API is not GGC/video playback)\n", j.Generation, j.Hash, j.State, groupText, reportIDs, strings.Join(missing, ","))
coverageText := strings.Join(covered, ",")
if coverageText == "" {
coverageText = "none"
if !known {
coverageText = "unknown"
}
}
_, err = fmt.Fprintf(out, "managed-test generation=%d hash=%s state=%s groups=%s active_candidate_ids=%v unresolved_groups=%s covered=%s\nLive health: NOT CHECKED (Player API is not GGC/video playback)\nCoverage: active config profiles only, not qualification or all-host/asset-inventory success.\n", j.Generation, j.Hash, j.State, groupText, reportIDs, strings.Join(missing, ","), coverageText)
if err == nil && j.Version == 3 {
if known && scope == GooglevideoScopeExperimental {
_, err = fmt.Fprintf(out, "Googlevideo scope: %s (experimental/unqualified; one authorized composite profile, not all hosts checked)\n", scope)
} else {
_, err = fmt.Fprintf(out, "Googlevideo scope: %s (suffix is authorized extrapolation, not all-host qualification)\n", scope)
}
}
return err
})
}
func managedRuntimeGuard(d managedDeps) (int, error) {
role, _ := d.lookup("ZAPRET_AUTOTUNE_TARGET_ROLE")
if role != "managed-test" {
return 2, fmt.Errorf("ZAPRET_AUTOTUNE_TARGET_ROLE must be managed-test")
}
for key, expected := range map[string]string{
"ZAPRET_AUTOTUNE_TARGET_ID": managedTargetID,
"ZAPRET_AUTOTUNE_TARGET_DIR": managedTargetDirectory,
"ZAPRET_CONFIG": managedTargetConfig,
} {
if value, _ := d.lookup(key); value != expected {
return 2, fmt.Errorf("%s does not match the fresh managed target contract", key)
}
}
portText, _ := d.lookup("SOCKS_PORT")
port, portErr := strconv.Atoi(portText)
proxy, _ := d.lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
if portErr != nil || port < 1 || port > 65535 || strconv.Itoa(port) != portText || proxy != "socks5://127.0.0.1:"+portText {
return 2, fmt.Errorf("managed SOCKS proxy must match the target SOCKS_PORT listener")
port, err := managedTargetSettings(d.lookup)
if err != nil {
return 2, err
}
if d.verifyRuntime == nil || d.verifySOCKS == nil {
return 1, fmt.Errorf("managed runtime validators unavailable")
@@ -304,7 +410,37 @@ func managedRuntimeGuard(d managedDeps) (int, error) {
return 0, nil
}
func managedTargetSettings(lookup func(string) (string, bool)) (int, error) {
role, _ := lookup("ZAPRET_AUTOTUNE_TARGET_ROLE")
if role != "managed-test" {
return 2, fmt.Errorf("ZAPRET_AUTOTUNE_TARGET_ROLE must be managed-test")
}
for key, expected := range map[string]string{
"ZAPRET_AUTOTUNE_TARGET_ID": managedTargetID,
"ZAPRET_AUTOTUNE_TARGET_DIR": managedTargetDirectory,
"ZAPRET_CONFIG": managedTargetConfig,
} {
if value, _ := lookup(key); value != expected {
return 2, fmt.Errorf("%s does not match the fresh managed target contract", key)
}
}
portText, _ := lookup("SOCKS_PORT")
port, portErr := strconv.Atoi(portText)
proxy, _ := lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
proxy, proxyErr := managedProbeProxy(proxy)
if portErr != nil || port < 1 || port > 65535 || strconv.Itoa(port) != portText || proxyErr != nil || proxy != "socks5h://127.0.0.1:"+portText {
return 2, fmt.Errorf("managed SOCKS proxy must match the target SOCKS_PORT listener")
}
return port, nil
}
func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
if len(args) == 3 && args[0] == "resource-apply" && args[1] == "--resource-config" {
return resourceApplyCLI(args[2], stdout, stderr, d)
}
if len(args) == 1 && args[0] == "resource-status" {
return resourceStatusCLI(stdout, d)
}
if len(args) > 0 && args[0] == "monitor" {
return managedMonitorCLI(args[1:], stdout, stderr, d)
}
@@ -323,31 +459,73 @@ func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
return 1
}
m := &managedTransaction{Dir: dir, Name: "config", Reload: func() error { return d.run(d.script, "restart") }, Health: func() (string, error) { return "", fmt.Errorf("health not configured") }}
if command == "init" {
if command == "experimental-request-check" {
err = managedExperimentalRequest(dir, false, nil)
} else if command == "init" {
err = managedInitialize(m)
} else if err = managedCheckIdentity(m); err == nil {
if command == "reconcile" {
err = managedPrestartReconcile(m, d)
} else if command == "status" || command == "active-ids" {
err = managedReport(m, stdout, d.openDB, command == "active-ids")
j, loadErr := m.load()
if loadErr != nil {
err = loadErr
} else if j.Version == 4 {
config, _ := d.lookup("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
s, e := openResourceApplySnapshot(config, d)
if e != nil {
err = errResourcePlan
} else {
m.resource = s
err = managedReport(m, stdout, d.openDB, command == "active-ids")
s.close()
}
} else {
err = managedReport(m, stdout, d.openDB, command == "active-ids", len(args) == 2 && args[1] == "--assets")
}
} else if command == "experiment" {
startup := false
for i := 1; i < len(args); i += 2 {
startup = startup || args[i] == "--googlevideo-request"
}
if startup {
err = managedExperimentalRequest(dir, true, func() error { return managedExperimentalSuffix(m, d) })
} else {
err = managedExperimentalSuffix(m, d)
}
} else {
expected := map[int64]string{}
var scopes []GooglevideoScope
for i := 1; i < len(args); i += 2 {
id, _ := strconv.ParseInt(args[i+1], 10, 64)
expected[id] = strings.TrimSuffix(strings.TrimPrefix(args[i], "--"), "-candidate")
if args[i] == "--googlevideo-scope" {
scopes = append(scopes, GooglevideoScope(args[i+1]))
continue
}
group := strings.TrimSuffix(strings.TrimPrefix(args[i], "--"), "-candidate")
if group == "asset" {
group = "assets"
}
for _, value := range strings.Split(args[i+1], ",") {
id, _ := strconv.ParseInt(value, 10, 64)
expected[id] = group
}
}
err = managedActivate(m, d, ids, expected)
m.forbidExperimentalApply = true
err = managedActivate(m, d, ids, expected, scopes...)
}
}
if err != nil {
fmt.Fprintln(stderr, "error:", err)
if errors.Is(err, errManagedScopeUnresolved) {
return 78
}
return 1
}
if command == "init" || command == "reconcile" || command == "apply" {
if command == "init" || command == "reconcile" || command == "apply" || command == "experiment" {
if _, err := fmt.Fprintln(stdout, "managed-test "+command+": OK"); err != nil {
return 1
}
if command == "apply" {
if command == "apply" || command == "experiment" {
if _, err := fmt.Fprintln(stdout, "Player API only; not full-profile/GGC/playback health. Use managed status for unresolved groups."); err != nil {
return 1
}
@@ -359,6 +537,27 @@ func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
// Startup may activate only a known baseline or DB-qualified rendered config.
// Reconcile restores bytes before its fixed target-local restart hook runs.
func managedPrestartReconcile(m *managedTransaction, d managedDeps) (err error) {
j, loadErr := m.load()
if loadErr != nil {
return loadErr
}
if j.Version == 4 {
config, _ := d.lookup("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
s, e := openResourceApplySnapshot(config, d)
if e != nil {
return errResourcePlan
}
defer s.close()
m.resource = s
defer func() { m.resource = nil }()
checks := productionResourceCheckDeps()
m.RollbackHealth = func() (string, error) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
return resourceApplyHealth(ctx, s, checks)
}
return m.Reconcile()
}
path, err := managedMonitorDatabase(d)
if err != nil {
return err
@@ -380,7 +579,7 @@ func managedPrestartReconcile(m *managedTransaction, d managedDeps) (err error)
return err
}
baselineHash := managedHash([]byte(managedBaseline))
if j.Version != 2 || (len(j.Candidates) == 0 && j.Hash != baselineHash) || (j.State == "prepared" && len(j.PendingCandidates) == 0 && j.Next != baselineHash) {
if j.Version < 2 || (len(j.Candidates) == 0 && j.Hash != baselineHash) || (j.State == "prepared" && len(j.PendingCandidates) == 0 && j.Next != baselineHash) {
return fmt.Errorf("startup candidate state unresolved; refusing activation")
}
return nil
@@ -395,16 +594,24 @@ func managedPrestartReconcile(m *managedTransaction, d managedDeps) (err error)
return err
}
func managedActivate(m *managedTransaction, d managedDeps, ids []int64, expected map[int64]string) error {
var errManagedScopeUnresolved = errors.New("managed deployment policy unresolved; explicit reconcile or fresh managed state required")
func managedActivate(m *managedTransaction, d managedDeps, ids []int64, expected map[int64]string, scopes ...GooglevideoScope) error {
if _, err := requiredYoutubeAdapter(); err != nil {
return err
}
info, err := os.Stat(d.script)
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0111 == 0 {
return fmt.Errorf("target zapret daemon init executable unavailable")
}
cfg, err := parseHealthConfig(d.lookup)
cfg, err := parseManagedHealthConfig(d.lookup)
if err != nil {
return err
}
m.Health = func() (string, error) {
if _, err := requiredYoutubeAdapter(); err != nil {
return "", err
}
portText, _ := d.lookup("SOCKS_PORT")
port, err := strconv.Atoi(portText)
if err != nil {
@@ -430,6 +637,9 @@ func managedActivate(m *managedTransaction, d managedDeps, ids []int64, expected
return fmt.Errorf("candidate database missing")
}
defer db.Close()
if _, err := m.ActiveCandidates(db); err != nil {
return fmt.Errorf("%w: %v", errManagedScopeUnresolved, err)
}
for _, id := range ids {
group := expected[id]
c, err := readStrategyCandidate(db, id)
@@ -437,11 +647,8 @@ func managedActivate(m *managedTransaction, d managedDeps, ids []int64, expected
return fmt.Errorf("invalid %s candidate ID", group)
}
}
if _, err := renderManagedConfig(db, ids); err != nil {
return err
}
if err := m.Reconcile(db); err != nil {
return err
}
return m.Apply(db, ids)
return m.Apply(db, ids, scopes...)
}
+141 -3
View File
@@ -46,8 +46,8 @@ func managedCLIFixture(t *testing.T) (managedDeps, map[string]string, *int) {
}
return nil
},
player: func(_ context.Context, args []string, stdout, _ io.Writer) error {
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5://127.0.0.1:11080|") {
player: func(_ context.Context, args []string, _ io.Reader, stdout, _ io.Writer) error {
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5h://127.0.0.1:11080|") {
t.Fatalf("unsafe health argv: %q", args)
}
_, err := io.WriteString(stdout, `{"playabilityStatus":{"status":"OK"}}`)
@@ -212,7 +212,7 @@ func TestManagedCLIApplyAndRollback(t *testing.T) {
}
switch mode {
case "health":
d.player = func(_ context.Context, _ []string, stdout, _ io.Writer) error {
d.player = func(_ context.Context, _ []string, _ io.Reader, stdout, _ io.Writer) error {
_, err := io.WriteString(stdout, `{"playabilityStatus":{"status":"ERROR","secret":"do-not-print"}}`)
return err
}
@@ -458,6 +458,144 @@ func TestManagedCLILegacyIDsUnresolved(t *testing.T) {
}
}
func TestManagedStatusOperatorCoverage(t *testing.T) {
for _, mode := range []string{"qualified", "experimental", "missing-candidate", "missing-db", "render-mismatch", "invalid-hash", "legacy", "pending"} {
t.Run(mode, func(t *testing.T) {
db := testStore(t)
front := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=fake", true)
ids := []int64{front}
scope := GooglevideoScopeExact
if mode == "qualified" {
ids = append(ids, managedCandidateFixture(t, db, "googlevideo", "rr1.googlevideo.com", "--lua-desync=fake", true))
} else {
asset := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
ggc, err := insertStrategyCandidate(tx, "googlevideo", []string{"rr1---sn-alpha.googlevideo.com"}, managedGooglevideoArgs, nil)
if err != nil {
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
ids = append(ids, asset, ggc)
scope = GooglevideoScopeExperimental
c, err := readStrategyCandidate(db, ggc)
if err != nil || c.Qualified || c.Mode != QualificationCold {
t.Fatalf("experimental candidate: %+v %v", c, err)
}
var evidence int
if err := db.QueryRow("SELECT count(*) FROM candidate_evidence WHERE candidate_id=?", ggc).Scan(&evidence); err != nil || evidence != 0 {
t.Fatalf("fabricated evidence: %d %v", evidence, err)
}
}
m, _ := managedFixture(t)
if err := m.Apply(db, ids, scope); err != nil {
t.Fatal(err)
}
j, err := m.load()
if err != nil {
t.Fatal(err)
}
openDB := func() (*sql.DB, error) { return openHistoryStore(databasePath()) }
switch mode {
case "missing-candidate":
j.Candidates[len(ids)-1] = 999999
case "missing-db":
openDB = func() (*sql.DB, error) { return nil, errors.New("unavailable") }
case "render-mismatch":
b := []byte("# not the rendered candidates\n" + managedBaseline)
if err := os.WriteFile(filepath.Join(m.Dir, m.Name), b, 0600); err != nil {
t.Fatal(err)
}
j.Hash = managedHash(b)
case "invalid-hash":
j.Hash = managedHash([]byte("wrong hash"))
case "legacy":
j.Version, j.Scope = 2, ""
case "pending":
j.State, j.Next = "prepared", j.Hash
j.PendingCandidates, j.PendingScope = append([]int64{}, ids...), scope
}
if err := m.journal(j); err != nil {
t.Fatal(err)
}
before, err := os.ReadFile(filepath.Join(m.Dir, m.Name))
if err != nil {
t.Fatal(err)
}
journalBefore, err := os.ReadFile(filepath.Join(m.Dir, managedJournalName))
if err != nil {
t.Fatal(err)
}
m.Health = func() (string, error) { t.Fatal("status checked live health"); return "", nil }
m.Reload = func() error { t.Fatal("status reloaded target"); return nil }
var out bytes.Buffer
err = managedReport(m, &out, openDB, false)
if mode == "invalid-hash" {
if !errors.Is(err, errManagedDrift) || out.Len() != 0 {
t.Fatalf("drift trusted: %s %v", out.String(), err)
}
} else {
if err != nil || !strings.Contains(out.String(), "Live health: NOT CHECKED") || strings.Contains(out.String(), "READY") {
t.Fatalf("status: %s %v", out.String(), err)
}
want := []string{"groups=unknown ", "covered=unknown", "active_candidate_ids=[] ", "unresolved_groups=frontend,googlevideo"}
if mode == "qualified" {
want = []string{"groups=frontend,googlevideo ", "covered=frontend,googlevideo", "unresolved_groups= covered="}
} else if mode == "experimental" {
want = []string{"groups=frontend,assets ", "covered=frontend,assets,googlevideo", "unresolved_groups=googlevideo,assets ", "experimental/unqualified", "one authorized composite profile", "not all hosts checked"}
}
for _, text := range want {
if !strings.Contains(out.String(), text) {
t.Fatalf("missing %q: %s", text, out.String())
}
}
}
out.Reset()
err = managedReport(m, &out, openDB, true)
if mode == "qualified" || mode == "experimental" {
var state struct {
IDs []int64 `json:"active_candidate_ids"`
Missing []string `json:"unresolved_groups"`
Covered []string `json:"covered_groups"`
Scope GooglevideoScope `json:"googlevideo_scope"`
}
if err != nil || json.Unmarshal(out.Bytes(), &state) != nil || !reflect.DeepEqual(state.IDs, ids) || state.Scope != scope {
t.Fatalf("JSON: %s %v", out.String(), err)
}
covered, missing := []string{"frontend", "googlevideo"}, []string(nil)
if mode == "experimental" {
covered, missing = []string{"frontend", "assets", "googlevideo"}, []string{"googlevideo", "assets"}
}
if !reflect.DeepEqual(state.Covered, covered) || !reflect.DeepEqual(state.Missing, missing) {
t.Fatalf("coverage changed qualification: %+v", state)
}
encoded, err := json.Marshal(state)
if err != nil {
t.Fatal(err)
}
var roundtrip map[string]json.RawMessage
if err := json.Unmarshal(encoded, &roundtrip); err != nil || len(roundtrip) != 4 {
t.Fatalf("JSON roundtrip: %s %v", encoded, err)
}
} else if err == nil || out.Len() != 0 {
t.Fatalf("invalid IDs trusted: %s %v", out.String(), err)
}
after, err := os.ReadFile(filepath.Join(m.Dir, m.Name))
if err != nil || !bytes.Equal(before, after) {
t.Fatal("status changed config", err)
}
journalAfter, err := os.ReadFile(filepath.Join(m.Dir, managedJournalName))
if err != nil || !bytes.Equal(journalBefore, journalAfter) {
t.Fatal("status changed journal", err)
}
})
}
}
func TestManagedMountpoint(t *testing.T) {
for _, path := range []string{"/", filepath.Dir(managedTargetDirectory), managedTargetDirectory + "-other"} {
if managedMountpoint("1 2 0:3 / "+path+" rw - tmpfs tmpfs rw\n", managedTargetDirectory) {
@@ -0,0 +1,65 @@
package main
import (
"encoding/json"
"os"
"reflect"
"testing"
)
// check_config.py provides sanitized actual `docker compose config` output.
// This bridges deployment interpolation to the production config parser offline.
func TestManagedGooglevideoComposeForwarding(t *testing.T) {
path := os.Getenv("AUTOTUNE_COMPOSE_CONFIG_FIXTURE")
if path == "" {
t.Skip("run deploy/managed-target/check_config.py for Compose fixture")
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatal("compose fixture unavailable")
}
var cases []struct {
Environment map[string]string `json:"environment"`
IDs []string `json:"ids"`
}
if json.Unmarshal(data, &cases) != nil || len(cases) < 6 {
t.Fatal("invalid compose fixture")
}
for i, c := range cases {
lookup := func(key string) (string, bool) { value, ok := c.Environment[key]; return value, ok }
mode, ids, err := managedGooglevideoConfig(lookup, c.Environment["ZAPRET_AUTOTUNE_VIDEO_ID"])
if err != nil || string(mode) != c.Environment["ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE"] || !reflect.DeepEqual(ids, c.IDs) {
t.Fatalf("compose factory case %d mismatch", i)
}
}
}
func TestManagedGooglevideoBlankInventoryDefault(t *testing.T) {
for _, value := range []string{"", " ", "\n", "[]", "bad", `["aaaaaaaaaaa"]`} {
lookup := func(key string) (string, bool) {
switch key {
case "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE":
return "1", true
case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE":
return "adaptive", true
case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS":
return value, true
}
return "", false
}
mode, ids, err := managedGooglevideoConfig(lookup, "dQw4w9WgXcQ")
if value != "" && value != `["aaaaaaaaaaa"]` {
if err == nil {
t.Fatal("invalid nonempty inventory accepted")
}
continue
}
want := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"}
if value != "" {
want = []string{"aaaaaaaaaaa"}
}
if err != nil || mode != QualificationAdaptive || !reflect.DeepEqual(ids, want) {
t.Fatal("inventory default/override mismatch")
}
}
}
+87 -3
View File
@@ -15,25 +15,72 @@ const managedFirewallConfig = "INIT_APPLY_FW=1\n" +
// renderManagedConfig accepts identifiers, not caller-supplied strategies or scope.
// Qualification is the existing complete endpoint × TLS evidence rule.
func renderManagedConfig(db *sql.DB, ids []int64) ([]byte, error) {
return renderManagedConfigScope(db, ids, GooglevideoScopeExact)
}
type GooglevideoScope string
const (
GooglevideoScopeExact GooglevideoScope = "exact"
GooglevideoScopeSuffix GooglevideoScope = "suffix"
GooglevideoScopeExperimental GooglevideoScope = "experimental_googlevideo_suffix"
)
func (s GooglevideoScope) valid() bool {
return s == "" || s == GooglevideoScopeExact || s == GooglevideoScopeSuffix || s == GooglevideoScopeExperimental
}
// Operator ownership is not qualification. Only the existing audited native
// composite may be owned without evidence, and only under the durable policy.
func experimentalGooglevideo(scope GooglevideoScope, c storedCandidate) bool {
return scope == GooglevideoScopeExperimental && c.Group == "googlevideo" && c.Mode == QualificationCold && validateNativeGooglevideoArgs(c.Args) == nil
}
func renderManagedConfigScope(db *sql.DB, ids []int64, scope GooglevideoScope) ([]byte, error) {
return renderManagedConfigPolicy(db, ids, scope, true)
}
// Only journal verification may reconstruct bytes without fresh qualification.
func renderStoredManagedConfigScope(db *sql.DB, ids []int64, scope GooglevideoScope) ([]byte, error) {
return renderManagedConfigPolicy(db, ids, scope, false)
}
func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope, fresh bool) ([]byte, error) {
if !scope.valid() {
return nil, fmt.Errorf("unknown googlevideo scope %q", scope)
}
if db == nil {
return nil, fmt.Errorf("candidate definitions database required")
}
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil {
return nil, err
}
if version < 3 || version > 7 {
return nil, fmt.Errorf("unsupported candidate schema version %d", version)
}
var profiles []string
groups := map[string]bool{}
seen := map[int64]bool{}
assets := map[string]bool{}
var otherScopes []string
nativeSuffix := false
for _, id := range ids {
if id <= 0 || seen[id] {
return nil, fmt.Errorf("duplicate candidate %d", id)
}
seen[id] = true
c, err := readStrategyCandidate(db, id)
c, err := readStrategyCandidatePolicy(db, id, fresh)
if err != nil {
return nil, err
}
if !c.Qualified {
if !c.Qualified && !experimentalGooglevideo(scope, c) {
return nil, fmt.Errorf("candidate %d is not qualified", id)
}
if groups[c.Group] {
if c.Group == "assets" && version < 6 {
return nil, fmt.Errorf("assets require schema 6")
}
if groups[c.Group] && c.Group != "assets" {
return nil, fmt.Errorf("duplicate candidate group %q", c.Group)
}
v := candidate{Group: c.Group, Domains: append([]string(nil), c.Endpoints...), Args: c.Args}
@@ -46,6 +93,16 @@ func renderManagedConfig(db *sql.DB, ids []int64) ([]byte, error) {
}
domains := append([]string(nil), v.Domains...)
for i, host := range domains {
if c.Group != "assets" {
otherScopes = append(otherScopes, host)
}
if c.Group == "assets" {
if assets[host] {
return nil, fmt.Errorf("overlapping asset endpoint")
}
assets[host] = true
domains[i] = "^" + host
}
if c.Group == "googlevideo" {
if !strings.HasSuffix(host, ".googlevideo.com") {
return nil, fmt.Errorf("GGC scope must be an exact googlevideo host")
@@ -53,9 +110,36 @@ func renderManagedConfig(db *sql.DB, ids []int64) ([]byte, error) {
domains[i] = "^" + host
}
}
if c.Group == "googlevideo" && scope == GooglevideoScopeSuffix {
nativeSuffix = c.Mode == QualificationAdaptive
if version < 7 || validateGooglevideoSuffixEvidencePolicy(db, c, fresh) != nil {
return nil, fmt.Errorf("suffix requires representative qualified hosts and verified media evidence")
}
// Authorized extrapolation, not evidence that all suffix hosts passed.
domains = []string{"googlevideo.com"}
}
if c.Group == "googlevideo" && scope == GooglevideoScopeExperimental {
if !experimentalGooglevideo(scope, c) {
return nil, fmt.Errorf("experimental suffix requires audited native candidate identity")
}
domains = []string{"googlevideo.com"}
}
groups[c.Group] = true
profiles = append(profiles, "--filter-tcp=443 --filter-l7=tls --hostlist-domains="+strings.Join(domains, ",")+" "+c.Args)
}
if scope == GooglevideoScopeSuffix && (!groups["googlevideo"] || nativeSuffix && !groups["frontend"] || !nativeSuffix && len(assets) != 2) {
return nil, fmt.Errorf("suffix requires qualified googlevideo and frontend (legacy cold scope requires both assets)")
}
if scope == GooglevideoScopeExperimental && (!groups["googlevideo"] || !groups["frontend"] || len(assets) == 0) {
return nil, fmt.Errorf("experimental suffix requires frontend and existing qualified assets")
}
for host := range assets {
for _, scope := range otherScopes {
if host == scope || strings.HasSuffix(host, "."+scope) {
return nil, fmt.Errorf("asset endpoint overlaps another profile")
}
}
}
if len(profiles) == 0 {
return nil, fmt.Errorf("qualified candidate required")
}
+87
View File
@@ -0,0 +1,87 @@
package main
import (
"context"
"io"
"strings"
"testing"
)
func assertManagedDNSArgs(t *testing.T, args []string, proxy string) {
t.Helper()
joined := strings.Join(args, "|")
if !strings.Contains(joined, "--noproxy||--proxy|"+proxy+"|") {
t.Fatalf("missing managed remote DNS proxy: %q", args)
}
for _, arg := range args {
if arg == "-k" || arg == "--insecure" || strings.HasPrefix(arg, "--resolve") || strings.HasPrefix(arg, "--connect-to") {
t.Fatalf("DNS pin or TLS bypass: %q", args)
}
}
}
func TestManagedProbeProxy(t *testing.T) {
for _, raw := range []string{"socks5://127.0.0.1:1080", "socks5h://127.0.0.1:1080", "socks5://localhost:1080", "socks5h://[::1]:1080"} {
got, err := managedProbeProxy(raw)
if err != nil || got != strings.Replace(raw, "socks5://", "socks5h://", 1) {
t.Fatal(raw, got, err)
}
}
for _, raw := range []string{"", "http://127.0.0.1:1080", "socks5h://proxy.example:1080", "socks5h://user:secret@127.0.0.1:1080", "socks5h://127.0.0.1:0", "socks5h://127.0.0.1:65536", "socks5h://127.0.0.1:1080/", "socks5h://127.0.0.1:1080?", "socks5h://127.0.0.1:1080#", "socks5h://127.0.0.1:1080\n", "socks5h://127.0.0.1:1080\x7f"} {
if got, err := managedProbeProxy(raw); err == nil || got != "" || strings.Contains(err.Error(), "secret") {
t.Fatal("unsafe proxy accepted or leaked", got, err)
}
}
}
func TestManagedDNSProductionPlayerAndGuard(t *testing.T) {
for _, scheme := range []string{"socks5", "socks5h"} {
t.Run(scheme, func(t *testing.T) {
target, env, _ := managedCLIFixture(t)
env["ZAPRET_AUTOTUNE_SOCKS_PROXY"] = scheme + "://127.0.0.1:11080"
owned := 0
target.verifySOCKS = func(port int) error {
if port != 11080 {
t.Fatal(port)
}
owned++
return nil
}
requests := 0
target.player = func(_ context.Context, args []string, _ io.Reader, out, _ io.Writer) error {
assertManagedDNSArgs(t, args, "socks5h://127.0.0.1:11080")
if args[len(args)-1] != playerURL {
t.Fatal("Player hostname changed", args)
}
requests++
_, err := io.WriteString(out, `{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"https://fresh.googlevideo.com/video?private=secret"}}`)
return err
}
r, _, _ := recoveryFixture(t, false)
d, err := managedRecoveryDependencies(target, r.db, r.scan, r.check, r.run)
if err != nil {
t.Fatal(err)
}
if status, err := d.health(context.Background()); err != nil || status != "OK" {
t.Fatal(status, err)
}
if host, err := d.discover(context.Background()); err != nil || host != "fresh.googlevideo.com" {
t.Fatal(host, err)
}
if requests != 2 || owned != 3 {
t.Fatal(requests, owned)
}
for _, proxy := range []string{scheme + "://127.0.0.1:1080", scheme + "://localhost:11080", scheme + "://[::1]:11080", scheme + "://proxy.example:11080"} {
env["ZAPRET_AUTOTUNE_SOCKS_PROXY"] = proxy
if _, err := managedRuntimeGuard(target); err == nil {
t.Fatal("unowned endpoint accepted", proxy)
}
}
})
}
lookup := configLookup(map[string]string{"ZAPRET_AUTOTUNE_SOCKS_PROXY": "socks5://127.0.0.1:11080"})
legacy, err := parseHealthConfig(lookup)
if err != nil || legacy.SOCKSProxy != "socks5://127.0.0.1:11080" || validDiscoveryProxy("socks5h://127.0.0.1:11080") {
t.Fatal("legacy proxy contract changed", legacy, err)
}
}
+150
View File
@@ -0,0 +1,150 @@
package main
import (
"crypto/sha256"
"errors"
"fmt"
"os"
"path/filepath"
"syscall"
)
const managedExperimentalRequestName = ".autotune-experimental-suffix-request"
func validExperimentalRequestStat(stat *syscall.Stat_t) bool {
return stat != nil && stat.Mode&(syscall.S_IFMT|07777) == syscall.S_IFREG|0600 && stat.Uid == uint32(os.Geteuid()) && stat.Nlink == 1 && stat.Size == 0
}
// A private directory excludes untrusted renames. Claim the validated inode
// before apply so no later cleanup can delete a replacement operator request.
// Failed claims stay private for diagnosis; never overwrite/retry them silently.
func managedExperimentalRequest(dir string, consume bool, apply func() error) error {
if err := managedDirectory(dir); err != nil {
return err
}
path := filepath.Join(dir, managedExperimentalRequestName)
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if errors.Is(err, syscall.ENOENT) {
return nil
}
invalid := fmt.Errorf("unsafe experimental startup request")
if err != nil {
return invalid
}
f := os.NewFile(uintptr(fd), "experimental request")
defer f.Close()
info, err := f.Stat()
if err != nil {
return invalid
}
stat, ok := info.Sys().(*syscall.Stat_t)
if !ok || !validExperimentalRequestStat(stat) {
return invalid
}
if !consume {
return nil
}
if apply == nil {
return invalid
}
claim, err := os.CreateTemp(dir, ".autotune-experimental-request-claim-*")
if err != nil {
return err
}
claimPath := claim.Name()
if err := claim.Close(); err != nil {
return err
}
if err := os.Rename(path, claimPath); err != nil {
return err
}
claimed, err := os.Lstat(claimPath)
if err != nil || !os.SameFile(info, claimed) {
return invalid
}
claimedStat, ok := claimed.Sys().(*syscall.Stat_t)
if !ok || !validExperimentalRequestStat(claimedStat) {
return invalid
}
transaction := &managedTransaction{Dir: dir}
if err := transaction.syncDir(); err != nil {
return err
}
if err := apply(); err != nil {
return err
}
claimed, err = os.Lstat(claimPath)
if err != nil || !os.SameFile(info, claimed) {
return invalid
}
if err := os.Remove(claimPath); err != nil {
return err
}
return transaction.syncDir()
}
// Runs before the PID-1-owned monitor starts; never races a recovery worker.
// A real Player-discovered identity is saved WITHOUT cold/adaptive evidence.
func managedExperimentalSuffix(m *managedTransaction, d managedDeps) error {
if _, err := requiredYoutubeAdapter(); err != nil {
return err
}
singleton, err := acquireManagedMonitorSingleton(d)
if err != nil {
return err
}
defer singleton.close()
lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua")
if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash {
return fmt.Errorf("native Lua identity mismatch")
}
cfg, err := parseManagedHealthConfig(d.lookup)
if err != nil {
return err
}
body, err := requestPlayer(cfg.VideoID, cfg.SOCKSProxy, d.player)
if err != nil {
return fmt.Errorf("fresh Player discovery unavailable")
}
host, err := playerHostname(body)
if err != nil {
return fmt.Errorf("fresh GGC identity unavailable")
}
if d.openWriteDB == nil {
return fmt.Errorf("experimental candidate writer unavailable")
}
db, err := d.openWriteDB(d)
if err != nil {
return err
}
defer db.Close()
current, err := m.ActiveCandidates(db)
if err != nil {
return err
}
tx, err := db.Begin()
if err != nil {
return err
}
id, err := insertStrategyCandidate(tx, "googlevideo", []string{host}, managedGooglevideoArgs, nil)
if err != nil {
_ = tx.Rollback()
return err
}
if err := tx.Commit(); err != nil {
return err
}
desired, err := managedFullDesiredSet(db, current, []int64{id})
if err != nil {
return err
}
expected := map[int64]string{}
for _, id := range desired {
c, err := readStrategyCandidate(db, id)
if err != nil {
return err
}
expected[id] = c.Group
}
return managedActivate(m, d, desired, expected, GooglevideoScopeExperimental)
}
@@ -0,0 +1,222 @@
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"os/exec"
"reflect"
"strings"
"testing"
"time"
)
func TestExperimentalProductionProbeCollectorClassification(t *testing.T) {
for _, mode := range []string{"bad-range", "healthy", "unknown", "infra", "mixed", "guard", "cancel", "metadata-timeout", "metadata-http", "metadata-malformed", "metadata-mixed", "metadata-cancel", "metadata-infra", "image-timeout", "image-mixed", "media-peer35", "media-peer60", "image-peer35", "image-peer60", "image-local58", "image-proxy7"} {
t.Run(mode, func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
target, _, _ := managedCLIFixture(t)
initManagedCLI(t, target)
d.transaction.Dir = target.directory
asset := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
id := managedCandidateFixture(t, d.db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
ids := []int64{front, asset, id}
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
before, err := d.transaction.load()
if err != nil {
t.Fatal(err)
}
d.service = true
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
d.resources = func(context.Context) (managedServiceInventory, error) {
r := experimentalResourceFixture()
if mode == "unknown" {
r = mediaResource{}
}
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/test", "yt3.ggpht.com": "https://yt3.ggpht.com/test"}, media: r}, nil
}
knownTimeout := exec.Command("sh", "-c", "exit 28").Run()
if strings.HasPrefix(mode, "metadata-") {
d.resources = managedServiceResources(healthConfig{SOCKSProxy: "socks5://127.0.0.1:11080", VideoID: "dQw4w9WgXcQ"}, func(context.Context) ([]byte, error) {
return []byte(fmt.Sprintf(`{"playabilityStatus":{"status":"OK"},"streamingData":{"formats":[{"url":%q,"mimeType":"video/mp4"}]}}`, experimentalResourceFixture().url)), nil
}, func(_ context.Context, args []string, _ io.Reader, out, diagnostic io.Writer) error {
origin := args[len(args)-1]
if (origin != "https://www.youtube.com/" && !(mode == "metadata-malformed" && origin == "https://www.youtube.com/watch?v=dQw4w9WgXcQ")) || diagnostic != io.Discard {
t.Fatal("metadata origin/diagnostics changed")
}
switch mode {
case "metadata-malformed":
_, err := io.WriteString(out, "<html>not playable metadata</html>")
return err
case "metadata-http":
return exec.Command("sh", "-c", "exit 22").Run()
case "metadata-mixed":
return errors.Join(knownTimeout, &managedRecoveryFatal{Err: errors.New("guard failed")})
case "metadata-cancel":
return errors.Join(knownTimeout, context.Canceled)
case "metadata-infra":
return errors.Join(knownTimeout, errors.New("collector unavailable"))
default:
return knownTimeout
}
})
i, _ := d.resources(context.Background())
if !freshExperimentalMedia(i.media) {
t.Fatal("metadata failure discarded fresh media")
}
}
calls := 0
d.recheckExperimental = func(_ context.Context, c storedCandidate, r mediaResource, _ *managedTransaction) error {
if c.ID != id || !freshExperimentalMedia(r) {
t.Fatal("recovery changed candidate or lost fresh media")
}
calls++
return nil
}
health := managedServiceHealth("socks5://127.0.0.1:11080", d.frontHealth, d.resources, func(_ context.Context, args []string, in io.Reader, out, diagnostic io.Writer) error {
if strings.HasPrefix(mode, "metadata-") {
t.Fatal("missing images must not become transfer evidence")
}
if mode == "image-timeout" {
return knownTimeout
}
if mode == "image-mixed" {
return errors.Join(knownTimeout, &managedRecoveryFatal{Err: errors.New("guard failed")})
}
config, _ := io.ReadAll(in)
raw := strings.TrimSuffix(strings.TrimPrefix(string(config), "url = \""), "\"\n")
isMedia := strings.Contains(raw, ".googlevideo.com/")
if strings.HasPrefix(mode, "media-peer") && isMedia || strings.HasPrefix(mode, "image-peer") && strings.Contains(raw, "yt3.ggpht.com/") {
return exec.Command("sh", "-c", "exit "+mode[len(mode)-2:]).Run()
}
if mode == "image-local58" {
return exec.Command("sh", "-c", "exit 58").Run()
}
if mode == "image-proxy7" {
return exec.Command("sh", "-c", "exit 7").Run()
}
var wire, diag bytes.Buffer
wireMode := ""
if strings.Contains(raw, ".googlevideo.com/") {
wireMode = "media"
if mode != "healthy" {
wireMode = "bad-range"
}
}
resourceWire(raw, wireMode, &wire, &diag)
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
diag = *bytes.NewBufferString(strings.ReplaceAll(diag.String(), "TLSv1.2", "TLSv1.3"))
}
_, _ = out.Write(wire.Bytes())
_, _ = diagnostic.Write(diag.Bytes())
return nil
})
d.health = func(ctx context.Context) (string, error) {
status, err := health(ctx)
if mode != "healthy" && status != "DEGRADED" {
t.Fatalf("resource failure became healthy: %s", status)
}
switch mode {
case "infra":
err = &managedResourceFailure{"media", experimentalResourceFixture().host, resourceFailure(ctx, errors.New("collector unavailable"))}
case "mixed":
err = errors.Join(err, &managedRecoveryFatal{Err: errors.New("guard failed")})
case "guard":
err = errors.Join(err, &managedRecoveryFatal{Err: errors.New("invalid media range or container")})
case "cancel":
err = errors.Join(err, context.Canceled)
}
return status, err
}
monitor := monitorRuntimeFixture(t)
monitor.db, monitor.target = d.db, target
monitor.threshold = 1
if err := wireManagedRecovery(&monitor, d, func(error) {}); err != nil {
t.Fatal(err)
}
err = monitor.probe(context.Background())
var fatal *managedRecoveryFatal
switch mode {
case "healthy":
if err != nil {
t.Fatal(err)
}
case "bad-range", "metadata-timeout", "metadata-http", "metadata-malformed", "image-timeout", "media-peer35", "media-peer60", "image-peer35", "image-peer60":
if err == nil || errors.As(err, &fatal) {
t.Fatalf("media damage killed monitor: %v", err)
}
case "unknown":
if err == nil || errors.Is(err, errNoQualifiedStrategy) {
t.Fatalf("unknown became green/absence: %v", err)
}
default:
if !errors.As(err, &fatal) {
t.Fatalf("safety failure normalized: %v", err)
}
}
if mode != "healthy" && mode != "unknown" {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
monitor.wait = func(ctx context.Context, _ time.Duration) error { cancel(); return ctx.Err() }
err = runManagedMonitor(ctx, monitor)
if mode == "bad-range" || mode == "metadata-timeout" || mode == "metadata-http" || mode == "metadata-malformed" || mode == "image-timeout" || strings.Contains(mode, "-peer") {
if !errors.Is(err, context.Canceled) || errors.As(err, &fatal) || calls != 1 {
t.Fatalf("monitor recovery unreachable: %v calls=%d", err, calls)
}
} else if !errors.As(err, &fatal) || calls != 0 {
t.Fatalf("fatal probe entered recovery: %v calls=%d", err, calls)
}
}
after, err := d.transaction.load()
if err != nil || !reflect.DeepEqual(before, after) {
t.Fatalf("probe/recovery changed journal: %v", err)
}
assertActiveCandidates(t, d.transaction, d.db, ids)
})
}
}
func TestManagedResourceHealthClassificationKeepsSafetyAndEvidenceBoundaries(t *testing.T) {
timeout := exec.Command("sh", "-c", "exit 28").Run()
httpFailure := exec.Command("sh", "-c", "exit 22").Run()
if managedGooglevideoTransferNonpass(resourceFailure(context.Background(), httpFailure)) {
t.Fatal("metadata HTTP nonpass became media rejection evidence")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
for _, tc := range []struct {
ctx context.Context
err error
}{
{context.Background(), &managedRecoveryFatal{Err: timeout}},
{context.Background(), errors.Join(resourceFailure(context.Background(), timeout), &managedRecoveryFatal{Err: timeout})},
{context.Background(), errors.Join(resourceFailure(context.Background(), timeout), errors.New("unknown collector failure"))},
{ctx, resourceFailure(context.Background(), timeout)},
} {
err := managedResourceHealthError(tc.ctx, &managedResourceFailure{"avatar", "yt3.ggpht.com", tc.err})
var failure *managedResourceFailure
if !errors.As(err, &failure) || failure.Err == nil {
t.Fatalf("safety failure normalized: %v", err)
}
if tc.ctx.Err() != nil && !errors.Is(err, context.Canceled) {
t.Fatal("parent cancellation lost")
}
}
}
func TestExperimentalProductionRecheckAdapterInstalled(t *testing.T) {
d, _, _ := recoveryFixture(t, true)
target, env, _ := managedCLIFixture(t)
env["ZAPRET_AUTOTUNE_YOUTUBE_SERVICE"] = "1"
adapted, err := managedMonitorRecoveryAdapters(target, d.db, d.scan, d.check, d.run)
if err != nil {
t.Fatal(err)
}
if !adapted.service || adapted.resources == nil || adapted.recheckExperimental == nil || adapted.run == nil || adapted.transaction == nil {
t.Fatal("production recheck dependency missing")
}
}
@@ -0,0 +1,218 @@
package main
import (
"context"
"crypto/sha256"
"errors"
"fmt"
"os"
"strconv"
"strings"
"time"
)
var errManagedExperimentalUnqualified = errors.New("experimental googlevideo suffix: operational health is not family qualification")
// URLs are trusted Player inventory, never rewritten or persisted.
func freshExperimentalMedia(r mediaResource) bool {
u, err := privateResourceURL(r.url, r.host)
if err != nil || !strings.HasSuffix(r.host, ".googlevideo.com") || u.Path != "/videoplayback" || (r.kind != "video/mp4" && r.kind != "audio/mp4") {
return false
}
q := u.Query()
expires, err := strconv.ParseInt(q.Get("expire"), 10, 64)
return err == nil && expires > time.Now().Unix()+120 && (q.Get("sig") != "" || q.Get("signature") != "" || q.Get("lsig") != "")
}
func experimentalMedia(ctx context.Context, d managedRecoveryDeps) (mediaResource, error) {
i, err := d.resources(ctx)
var fatal *managedRecoveryFatal
var failure *managedResourceFailure
// An unrelated metadata/image failure cannot conceal usable media, but a
// mixed safety/infrastructure error must not be turned into transfer absence.
unrelated := errors.As(err, &failure) && assetEndpoint(failure.Host) && (err == failure && failure.Err == nil || managedGooglevideoTransferNonpass(err))
if ctx.Err() != nil || errors.As(err, &fatal) && !unrelated || errors.Is(err, context.Canceled) || err != nil && !unrelated {
return mediaResource{}, managedRecoveryContextError(ctx, err)
}
if !freshExperimentalMedia(i.media) {
return mediaResource{}, &managedResourceFailure{"media", "unknown", nil}
}
return i.media, nil
}
func recoverManagedExperimental(ctx context.Context, d managedRecoveryDeps) (int64, error) {
if !d.service || d.resources == nil || d.recheckExperimental == nil {
return 0, errManagedGGCUnresolved
}
ids, err := d.transaction.ActiveCandidates(d.db)
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
j, err := d.transaction.load()
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
for _, id := range ids {
c, err := readStrategyCandidate(d.db, id)
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
if experimentalGooglevideo(j.Scope, c) {
r, err := experimentalMedia(ctx, d)
if err == nil {
err = d.recheckExperimental(ctx, c, r, d.transaction)
}
if err == nil && d.mediaSamples != nil && d.verifyGooglevideo != nil {
// One-host operational health never promotes the experiment. Only
// fresh representative inventory and a completed owned proof may.
qualified, err := qualifyManagedGooglevideo(ctx, d)
if err != nil {
return id, err
}
return qualified, nil
}
return id, managedRecoveryContextError(ctx, err)
}
}
return 0, &managedRecoveryFatal{Err: errors.New("owned experimental candidate missing")}
}
func newManagedExperimentalRecheck(target managedDeps, d managedRecoveryDeps) func(context.Context, storedCandidate, mediaResource, *managedTransaction) error {
runtime := newManagedAssetRuntime(target)
return func(ctx context.Context, c storedCandidate, resource mediaResource, transaction *managedTransaction) error {
// The adapter is constructed with path-only dependencies. Use this job's
// bound hooks for ownership checks and reconciliation, not that stale copy.
d := d
d.transaction = transaction
guard := func() error {
if err := runtime.guard(); err != nil {
return err
}
lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua")
if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash {
return errors.New("unverified native automation Lua source")
}
_, err = d.transaction.ActiveCandidates(d.db)
return err
}
restart := func(ctx context.Context) (err error) {
// Existing daemon protocol stops/joins before starting the same full set.
// Never activate a pending or drifted deployment after a partial stop.
defer func() {
if err != nil {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
if e := d.transaction.Reconcile(d.db); e != nil {
err = errors.Join(err, e)
} else {
err = errors.Join(err, d.run(cleanup, managedZapretScript, "start"))
}
}
}()
if err = d.run(ctx, managedZapretScript, "stop"); err != nil {
return err
}
if err = d.transaction.Reconcile(d.db); err != nil {
return err
}
return d.run(ctx, managedZapretScript, "start")
}
if !experimentalGooglevideo(GooglevideoScopeExperimental, c) {
return &managedRecoveryFatal{Err: errors.New("unaudited experimental candidate")}
}
ownedGuard := func() error {
if err := guard(); err != nil {
return err
}
j, err := d.transaction.load()
if err != nil {
return err
}
if j.Scope == GooglevideoScopeExperimental {
for _, id := range j.Candidates {
if id == c.ID {
return nil
}
}
}
return errors.New("experimental ownership changed")
}
return recheckDeployedExperimental(ctx, resource, ownedGuard, func(ctx context.Context) error {
if err := ownedGuard(); err != nil {
return err
}
return restart(ctx)
}, func(ctx context.Context) (mediaResource, error) {
return experimentalMedia(ctx, d)
}, func(ctx context.Context, r mediaResource, protocol string) error {
_, _, err := collectMediaTransfer(ctx, runtime.proxy, r, protocol, runtime.curl)
return err
})
}
}
// No temporary NFQ worker: successful learning stays in the deployed daemon RAM.
func recheckDeployedExperimental(ctx context.Context, resource mediaResource, guard func() error, restart func(context.Context) error, fresh func(context.Context) (mediaResource, error), transfer func(context.Context, mediaResource, string) error) error {
check := func(ctx context.Context, protocol string) error {
if err := ctx.Err(); err != nil {
return err
}
if err := guard(); err != nil {
return &managedRecoveryFatal{Err: err}
}
if !freshExperimentalMedia(resource) {
var err error
resource, err = fresh(ctx)
if err != nil {
return err
}
if !freshExperimentalMedia(resource) {
return &managedResourceFailure{"media", "unknown", nil}
}
}
return managedRecoveryContextError(ctx, transfer(ctx, resource, protocol))
}
sample := func() error {
for _, protocol := range []string{"tls12", "tls13"} {
for n := 0; n < 3; n++ {
if err := check(ctx, protocol); err != nil {
return err
}
}
}
return nil
}
err := sample()
if err == nil || !managedGooglevideoTransferNonpass(err) {
return err
}
if err := restart(ctx); err != nil {
return &managedRecoveryFatal{Err: managedRecoveryContextError(ctx, err)}
}
warm, cancel := context.WithTimeout(ctx, 120*time.Second)
defer cancel()
learned := map[string]bool{}
for n := 0; n < 10 && len(learned) < 2; n++ {
protocol := []string{"tls12", "tls13"}[n%2]
err = check(warm, protocol)
if ctx.Err() != nil {
return managedRecoveryContextError(ctx, err)
}
if warm.Err() != nil && managedRecoveryTimeoutSafe(err) {
break
}
if err != nil && !managedGooglevideoTransferNonpass(err) {
return err
}
if err == nil {
learned[protocol] = true
}
}
if len(learned) == 2 {
err = sample()
if err == nil || !managedGooglevideoTransferNonpass(err) {
return err
}
}
return fmt.Errorf("experimental media relearning exhausted: %w", errNoQualifiedStrategy)
}
@@ -0,0 +1,177 @@
package main
import (
"context"
"errors"
"fmt"
"reflect"
"testing"
"time"
)
func experimentalResourceFixture() mediaResource {
return mediaResource{url: fmt.Sprintf("https://rr2---sn-new.googlevideo.com/videoplayback?expire=%d&sig=private", time.Now().Add(time.Hour).Unix()), host: "rr2---sn-new.googlevideo.com", kind: "video/mp4"}
}
func TestExperimentalDeployedRecheck(t *testing.T) {
for _, mode := range []string{"healthy", "learn", "exhausted", "mixed", "cancel", "guard"} {
t.Run(mode, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
r := experimentalResourceFixture()
calls, restarts := 0, 0
protocols := []string{}
bad := errors.New("invalid media range or container")
infra := errors.New("collector unavailable")
err := recheckDeployedExperimental(ctx, r, func() error {
if mode == "guard" {
return infra
}
return nil
}, func(context.Context) error { restarts++; return nil }, func(context.Context) (mediaResource, error) {
t.Fatal("unexpected refresh")
return mediaResource{}, infra
}, func(_ context.Context, got mediaResource, protocol string) error {
if got != r {
t.Fatal("signed resource rewritten")
}
calls++
protocols = append(protocols, protocol)
switch mode {
case "learn":
if calls == 1 {
return resourceFailure(ctx, bad)
}
case "exhausted":
return resourceFailure(ctx, bad)
case "mixed":
return errors.Join(resourceFailure(ctx, bad), infra)
case "cancel":
cancel()
return context.Canceled
}
return nil
})
switch mode {
case "healthy":
if err != nil || restarts != 0 || !reflect.DeepEqual(protocols, []string{"tls12", "tls12", "tls12", "tls13", "tls13", "tls13"}) {
t.Fatalf("%v %d %v", err, restarts, protocols)
}
case "learn":
if err != nil || restarts != 1 || calls != 9 {
t.Fatalf("%v restarts=%d calls=%d", err, restarts, calls)
}
case "exhausted":
if !errors.Is(err, errNoQualifiedStrategy) || restarts != 1 || calls != 11 {
t.Fatalf("%v restarts=%d calls=%d", err, restarts, calls)
}
default:
if err == nil || errors.Is(err, errNoQualifiedStrategy) || restarts != 0 {
t.Fatalf("unsafe classification: %v restarts=%d", err, restarts)
}
}
})
}
}
func TestExperimentalServiceRecoveryPreservesOwnership(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
target, _, _ := managedCLIFixture(t)
initManagedCLI(t, target)
d.transaction.Dir = target.directory
asset := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
id := managedCandidateFixture(t, d.db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
ids := []int64{front, asset, id}
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
before, err := d.transaction.load()
if err != nil {
t.Fatal(err)
}
d.service = true
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
d.health = func(context.Context) (string, error) { return "OK", nil }
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{media: experimentalResourceFixture()}, &managedResourceFailure{"avatar", "yt3.ggpht.com", nil}
}
calls := 0
var boundTransaction *managedTransaction
d.recheckExperimental = func(_ context.Context, c storedCandidate, r mediaResource, transaction *managedTransaction) error {
calls++
boundTransaction = transaction
if transaction.Reload == nil || transaction.Health == nil {
t.Fatal("recheck retained path-only adapter transaction")
}
assertActiveCandidates(t, transaction, d.db, ids)
if c.ID != id || !freshExperimentalMedia(r) {
t.Fatal("wrong committed candidate/resource")
}
return nil
}
// Production adapters carry paths only until recoverManagedTarget binds the
// job's reload/health hooks. The recheck must receive that bound transaction.
d.transaction.Reload, d.transaction.Health = nil, nil
for _, adaptive := range []bool{false, true} {
d.adaptive = adaptive
result, err := recoverManagedTarget(context.Background(), d)
if err != nil || result.GooglevideoID != id || result.GGCError != nil {
t.Fatalf("%+v %v", result, err)
}
}
after, err := d.transaction.load()
if err != nil || !reflect.DeepEqual(before, after) || calls != 2 {
t.Fatalf("journal changed: %v calls=%d", err, calls)
}
c, err := readStrategyCandidate(d.db, id)
if err != nil || c.Qualified {
t.Fatalf("qualification forged: %+v %v", c, err)
}
monitor := managedMonitorDeps{db: d.db, target: target}
reports := 0
if err := wireManagedRecovery(&monitor, d, func(err error) {
if errors.Is(err, errManagedExperimentalUnqualified) {
reports++
}
}); err != nil {
t.Fatal(err)
}
if err := monitor.probe(context.Background()); err != nil || reports != 1 {
t.Fatalf("operational probe: %v reports=%d", err, reports)
}
assertActiveCandidates(t, boundTransaction, d.db, ids)
}
func TestExperimentalRefreshAndMixedFatalPrecedence(t *testing.T) {
bad := errors.New("invalid media range or container")
for _, err := range []error{
&managedRecoveryFatal{Err: bad},
fmt.Errorf("wrapped: %w", &managedRecoveryFatal{Err: bad}),
errors.Join(resourceFailure(context.Background(), bad), &managedRecoveryFatal{Err: bad}),
errors.Join(resourceFailure(context.Background(), bad), context.Canceled),
} {
if managedGooglevideoTransferNonpass(err) {
t.Fatalf("fatal accepted: %v", err)
}
}
refreshes, transfers := 0, 0
err := recheckDeployedExperimental(context.Background(), mediaResource{}, func() error { return nil }, func(context.Context) error {
t.Fatal("unknown media caused restart")
return nil
}, func(context.Context) (mediaResource, error) {
refreshes++
return experimentalResourceFixture(), nil
}, func(context.Context, mediaResource, string) error { transfers++; return nil })
if err != nil || refreshes != 1 || transfers != 6 {
t.Fatalf("refresh: %v %d %d", err, refreshes, transfers)
}
err = recheckDeployedExperimental(context.Background(), mediaResource{}, func() error { return nil }, func(context.Context) error {
t.Fatal("unknown media caused restart")
return nil
}, func(context.Context) (mediaResource, error) {
return mediaResource{}, &managedResourceFailure{"media", "unknown", nil}
}, func(context.Context, mediaResource, string) error { t.Fatal("unknown media transferred"); return nil })
if err == nil || errors.Is(err, errNoQualifiedStrategy) {
t.Fatalf("unknown treated as absence: %v", err)
}
}
@@ -0,0 +1,410 @@
package main
import (
"bytes"
"context"
"database/sql"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"reflect"
"strings"
"syscall"
"testing"
)
func TestExperimentalSuffixOwnershipNotQualification(t *testing.T) {
db := testStore(t)
// Existing real-definition fixture; no passing evidence is created for GGC.
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
id, err := insertStrategyCandidate(tx, "googlevideo", []string{"rr1---sn-alpha.googlevideo.com"}, managedGooglevideoArgs, nil)
if err != nil {
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
front := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=fake", true)
asset := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
current := []int64{front, asset}
ids := append(append([]int64{}, current...), id)
c, err := readStrategyCandidate(db, id)
if err != nil || c.Qualified {
t.Fatalf("fabricated qualification: %+v %v", c, err)
}
for _, scope := range []GooglevideoScope{GooglevideoScopeExact, GooglevideoScopeSuffix} {
if _, err := renderManagedConfigScope(db, ids, scope); err == nil {
t.Fatal("normal gate weakened")
}
}
b, err := renderManagedConfigScope(db, ids, GooglevideoScopeExperimental)
if err != nil || !strings.Contains(string(b), "--hostlist-domains=googlevideo.com ") {
t.Fatalf("render %s %v", b, err)
}
m, _ := managedFixture(t)
if err := m.Apply(db, current); err != nil {
t.Fatal(err)
}
if err := m.Apply(db, ids, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
j, err := m.load()
if err != nil || j.Scope != GooglevideoScopeExperimental || j.Version != 3 {
t.Fatalf("journal %+v %v", j, err)
}
if _, err := m.ActiveCandidates(db); err != nil {
t.Fatal(err)
}
if err := m.Apply(db, []int64{front, id}, GooglevideoScopeExperimental); err == nil {
t.Fatal("dropped asset")
}
if err := m.Apply(db, ids); err != nil {
t.Fatal("refresh erased policy", err)
}
m.Health = func() (string, error) { return "FAIL", nil }
if err := m.Apply(db, current, GooglevideoScopeExact); err == nil {
t.Fatal("failed apply committed")
}
j, err = m.load()
if err != nil || j.Scope != GooglevideoScopeExperimental || !reflect.DeepEqual(j.Candidates, ids) {
t.Fatalf("rollback %+v %v", j, err)
}
m.Health = func() (string, error) { return "OK", nil }
crash := errors.New("crash")
m.fault = func(point string) error {
if point == "replaced" {
return crash
}
return nil
}
if err := m.Apply(db, current, GooglevideoScopeExact); !errors.Is(err, crash) {
t.Fatal(err)
}
m.fault = nil
if err := m.Reconcile(db); err != nil {
t.Fatal(err)
}
j, err = m.load()
if err != nil || j.Scope != GooglevideoScopeExperimental {
t.Fatalf("reconcile %+v %v", j, err)
}
qualified, err := listQualifiedCandidates(db, "googlevideo", 10)
if err != nil || len(qualified) != 0 {
t.Fatalf("experiment leaked into qualified list %+v %v", qualified, err)
}
var evidence int
if err := db.QueryRow("SELECT count(*) FROM candidate_evidence WHERE candidate_id=?", id).Scan(&evidence); err != nil || evidence != 0 {
t.Fatal("fabricated evidence", evidence, err)
}
var out bytes.Buffer
if err := managedReport(m, &out, func() (*sql.DB, error) { return db, nil }, false); err != nil {
t.Fatal(err)
}
if !strings.Contains(out.String(), "unresolved_groups=googlevideo,assets") {
t.Fatalf("false qualified status: %s", out.String())
}
}
func TestExperimentalSuffixCLIExplicit(t *testing.T) {
command, ids, err := managedArguments([]string{"apply", "--googlevideo-experiment", "native-circular", "--googlevideo-scope", string(GooglevideoScopeExperimental)})
if err != nil || command != "experiment" || len(ids) != 0 {
t.Fatal(command, ids, err)
}
for _, args := range [][]string{
{"apply", "--googlevideo-experiment", "native-circular"},
{"apply", "--googlevideo-experiment", "native-circular", "--googlevideo-scope", "suffix"},
{"apply", "--googlevideo-experiment", "unknown", "--googlevideo-scope", string(GooglevideoScopeExperimental)},
{"apply", "--googlevideo-candidate", "3", "--googlevideo-scope", string(GooglevideoScopeExperimental)},
{"apply", "--frontend-candidate", "1", "--googlevideo-candidate", "3", "--asset-candidate", "2", "--googlevideo-scope", string(GooglevideoScopeExperimental)},
{"apply", "--googlevideo-request", "startup", "--googlevideo-candidate", "3"},
} {
if _, _, err := managedArguments(args); err == nil {
t.Fatal("implicit experiment", args)
}
}
}
func TestExperimentalSuffixCLIRejectsBypassBeforeWork(t *testing.T) {
d, _, _ := managedCLIFixture(t)
d.lookup = func(string) (string, bool) { t.Fatal("bypass reached runtime work"); return "", false }
d.player = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("bypass requested Player")
return nil
}
code, _, _ := callManaged(t, d, "apply", "--googlevideo-candidate", "3", "--googlevideo-scope", string(GooglevideoScopeExperimental))
if code != 2 {
t.Fatal(code)
}
}
func TestExperimentalRequestValidationAndConsumption(t *testing.T) {
for _, kind := range []string{"valid", "content", "mode", "directory", "symlink", "dangling", "hardlink", "fifo", "owner", "apply-failure", "replacement"} {
t.Run(kind, func(t *testing.T) {
dir := t.TempDir()
if err := os.Chmod(dir, 0700); err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, managedExperimentalRequestName)
if err := os.WriteFile(path, nil, 0600); err != nil {
t.Fatal(err)
}
switch kind {
case "content":
if err := os.WriteFile(path, []byte("not empty"), 0600); err != nil {
t.Fatal(err)
}
case "mode":
if err := os.Chmod(path, 0640); err != nil {
t.Fatal(err)
}
case "directory":
if err := os.Remove(path); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(path, 0700); err != nil {
t.Fatal(err)
}
case "symlink", "dangling":
target := filepath.Join(dir, "operator-file")
if err := os.Rename(path, target); err != nil {
t.Fatal(err)
}
if kind == "dangling" {
target += "-missing"
}
if err := os.Symlink(target, path); err != nil {
t.Fatal(err)
}
case "hardlink":
if err := os.Link(path, filepath.Join(dir, "operator-file")); err != nil {
t.Fatal(err)
}
case "fifo":
if err := os.Remove(path); err != nil {
t.Fatal(err)
}
if err := syscall.Mkfifo(path, 0600); err != nil {
t.Fatal(err)
}
case "owner":
if os.Geteuid() != 0 {
t.Skip("real wrong-UID fixture requires root; fstat owner unit check runs separately")
}
if err := os.Chown(path, 1, -1); err != nil {
t.Fatal(err)
}
}
before, err := os.Lstat(path)
if err != nil {
t.Fatal(err)
}
calls := 0
apply := func() error {
calls++
if kind == "replacement" {
return os.WriteFile(path, []byte("new request: retain"), 0600)
}
if kind == "apply-failure" {
return errors.New("apply failed")
}
return nil
}
if kind != "valid" && kind != "apply-failure" && kind != "replacement" {
if err := managedExperimentalRequest(dir, false, nil); err == nil {
t.Fatal("unsafe preflight accepted")
}
if err := managedExperimentalRequest(dir, true, apply); err == nil || calls != 0 {
t.Fatal("unsafe request applied", calls, err)
}
after, err := os.Lstat(path)
if err != nil || !os.SameFile(before, after) {
t.Fatal("unsafe operator file removed/replaced", err)
}
return
}
if err := managedExperimentalRequest(dir, false, nil); err != nil {
t.Fatal(err)
}
err = managedExperimentalRequest(dir, true, apply)
if kind == "apply-failure" {
if err == nil || calls != 1 {
t.Fatal("failure lost", calls, err)
}
claims, err := filepath.Glob(filepath.Join(dir, ".autotune-experimental-request-claim-*"))
if err != nil || len(claims) != 1 {
t.Fatal("failed claim lost", claims, err)
}
return
}
if err != nil || calls != 1 {
t.Fatal(calls, err)
}
if kind == "replacement" {
body, err := os.ReadFile(path)
if err != nil || string(body) != "new request: retain" {
t.Fatal("replacement deleted", err)
}
return
}
if err := managedExperimentalRequest(dir, true, apply); err != nil || calls != 1 {
t.Fatal("request repeated", calls, err)
}
})
}
}
func TestExperimentalRequestWrongOwnerStat(t *testing.T) {
stat := syscall.Stat_t{Mode: syscall.S_IFREG | 0600, Uid: uint32(os.Geteuid()), Nlink: 1}
if !validExperimentalRequestStat(&stat) {
t.Fatal("valid runtime-owned stat rejected")
}
stat.Uid++
if validExperimentalRequestStat(&stat) {
t.Fatal("foreign owner accepted")
}
}
func TestExperimentalSuffixCLIRejectsInheritedPolicy(t *testing.T) {
d, _, _ := managedCLIFixture(t)
initManagedCLI(t, d)
db := testStore(t)
front := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=fake", true)
asset := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
id := managedCandidateFixture(t, db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
m := &managedTransaction{Dir: d.directory, Name: "config", Reload: func() error { return nil }, Health: func() (string, error) { return "OK", nil }}
if err := m.Apply(db, []int64{front, asset, id}, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
d.player = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("inherited bypass requested Player")
return nil
}
d.openDB = func() (*sql.DB, error) { return db, nil }
code, _, diagnostic := callManaged(t, d, "apply", "--googlevideo-candidate", fmt.Sprint(id))
if code != 1 || !strings.Contains(diagnostic, "explicit guarded operator path") {
t.Fatal(code, diagnostic)
}
}
func TestExperimentalSuffixCLIInterleavedPolicyChange(t *testing.T) {
d, _, workers := managedCLIFixture(t)
initManagedCLI(t, d)
db := testStore(t)
front := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=fake", true)
asset := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
id := managedCandidateFixture(t, db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
operator := &managedTransaction{Dir: d.directory, Name: "config", Reload: func() error { return nil }, Health: func() (string, error) { return "OK", nil }}
if err := operator.Apply(db, []int64{front, asset}); err != nil {
t.Fatal(err)
}
var snapshot map[string][]byte
// No timing race: commit the operator policy after ordinary CLI dispatch,
// before its transaction reads the journal under the apply lock.
d.openDB = func() (*sql.DB, error) {
if err := operator.Apply(db, []int64{front, asset, id}, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
snapshot = map[string][]byte{}
for _, name := range []string{"config", managedJournalName, managedBackupName} {
b, err := operator.read(name)
if err != nil {
t.Fatal(err)
}
snapshot[name] = b
}
return db, nil
}
d.player = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("racing CLI requested Player")
return nil
}
code, _, diagnostic := callManaged(t, d, "apply", "--frontend-candidate", fmt.Sprint(front), "--googlevideo-candidate", fmt.Sprint(id), "--asset-candidate", fmt.Sprint(asset))
if code != 1 || !strings.Contains(diagnostic, "explicit guarded operator path") {
t.Fatal(code, diagnostic)
}
if *workers != 0 {
t.Fatal("unauthorized reload/worker started", *workers)
}
for name, before := range snapshot {
after, err := operator.read(name)
if err != nil || !bytes.Equal(before, after) {
t.Fatalf("racing CLI mutated %s: %v", name, err)
}
}
j, err := operator.load()
if err != nil || j.Scope != GooglevideoScopeExperimental || !reflect.DeepEqual(j.Candidates, []int64{front, asset, id}) {
t.Fatalf("operator deployment lost: %+v %v", j, err)
}
}
func TestExperimentalSuffixPreparedRollback(t *testing.T) {
db := testStore(t)
front := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=fake", true)
asset := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
id := managedCandidateFixture(t, db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
m, _ := managedFixture(t)
current := []int64{front, asset}
if err := m.Apply(db, current); err != nil {
t.Fatal(err)
}
prior, err := renderManagedConfig(db, current)
if err != nil {
t.Fatal(err)
}
crash := errors.New("prepared experiment interrupted")
m.fault = func(point string) error {
if point == "replaced" {
return crash
}
return nil
}
if err := m.Apply(db, []int64{front, asset, id}, GooglevideoScopeExperimental); !errors.Is(err, crash) {
t.Fatal(err)
}
j, err := m.load()
if err != nil || j.Scope != GooglevideoScopeExact || j.PendingScope != GooglevideoScopeExperimental || j.Hash != managedHash(prior) {
t.Fatalf("pending policy/hash: %+v %v", j, err)
}
m.fault = nil
if err := m.Reconcile(db); err != nil {
t.Fatal(err)
}
assertManagedContent(t, m, string(prior))
assertActiveCandidates(t, m, db, current)
j, err = m.load()
if err != nil || j.Scope != GooglevideoScopeExact || j.PendingScope != "" {
t.Fatalf("rollback policy: %+v %v", j, err)
}
}
func TestExperimentalSuffixCannotPromoteMonitorReadiness(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
target, _, _ := managedCLIFixture(t)
initManagedCLI(t, target)
d.transaction.Dir = target.directory
asset := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
id := managedCandidateFixture(t, d.db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
if err := d.transaction.Apply(d.db, []int64{front, asset, id}, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
monitor := managedMonitorDeps{db: d.db, target: target}
if err := wireManagedRecovery(&monitor, d, func(error) {}); err != nil {
t.Fatal(err)
}
if err := monitor.probe(context.Background()); !errors.Is(err, errManagedGGCUnresolved) {
t.Fatalf("false readiness: %v", err)
}
for _, adaptive := range []bool{false, true} {
d.adaptive = adaptive
result, err := recoverManagedTarget(context.Background(), d)
if !errors.Is(err, errManagedGGCUnresolved) || !errors.Is(result.GGCError, errManagedGGCUnresolved) {
t.Fatalf("recovery qualified experiment: %+v %v", result, err)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front, asset, id})
}
}
@@ -0,0 +1,87 @@
package main
import (
"context"
"crypto/sha256"
"database/sql"
"errors"
"fmt"
"os"
"reflect"
)
func newManagedGooglevideoApplyRecheck(target managedDeps, db *sql.DB) func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
r := newManagedAssetRuntime(target)
guard := r.guard
r.guard = func() error {
if err := guard(); err != nil {
return err
}
lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua")
if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash {
return errors.New("unverified native automation Lua source")
}
return nil
}
return managedGooglevideoApplyRecheck(r, db)
}
// Runs inside Apply's lock: never ActiveCandidates/Reconcile/restart here.
// The newly reloaded daemon learns in-place; a failure rolls back via Apply.
func managedGooglevideoApplyRecheck(r managedAssetRuntime, db *sql.DB) func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
return func(ctx context.Context, c storedCandidate, resources []mediaResource, m *managedTransaction) error {
j, err := m.load()
if err != nil {
return &managedRecoveryFatal{Err: err}
}
found := false
for _, id := range j.PendingCandidates {
found = found || id == c.ID
}
var hosts []string
for _, resource := range resources {
hosts = append(hosts, resource.host)
}
if j.State != "prepared" || j.PendingScope != GooglevideoScopeSuffix || !found || c.Mode != QualificationAdaptive || !reflect.DeepEqual(hosts, c.Endpoints) || validateGooglevideoSuffixEvidence(db, c) != nil {
return &managedRecoveryFatal{Err: errors.New("deployed media candidate ownership mismatch")}
}
guard := func() error {
if err := r.guard(); err != nil {
return err
}
if err := m.owns(j, false); err != nil {
return err
}
if err := validateManagedRefs(db, j.PendingCandidates, j.Next, j.PendingScope); err != nil {
return err
}
b, err := m.read(m.Name)
if err != nil {
return err
}
if managedHash(b) != j.Next {
return errManagedDrift
}
return nil
}
for _, resource := range resources {
err := recheckDeployedExperimental(ctx, resource, guard,
func(context.Context) error { return guard() }, // already reloaded; warm current daemon only
func(context.Context) (mediaResource, error) { return mediaResource{}, errCandidateNonpass },
func(ctx context.Context, resource mediaResource, protocol string) error {
_, _, err := collectMediaTransfer(ctx, r.proxy, resource, protocol, r.curl)
return err
})
if errors.Is(err, errNoQualifiedStrategy) && !managedRecoveryTimeoutSafe(err) {
var fatal *managedRecoveryFatal
if !errors.As(err, &fatal) {
return errCandidateNonpass
}
}
if err != nil {
return err
}
}
return nil
}
}
@@ -0,0 +1,191 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"testing"
"time"
)
func samplePlayerBody(host string, expires time.Time) []byte {
u, _ := json.Marshal(fmt.Sprintf("https://%s/videoplayback?expire=%d&sig=private-secret", host, expires.Unix()))
return []byte(fmt.Sprintf(`{"playabilityStatus":{"status":"OK"},"streamingData":{"formats":[{"mimeType":"video/mp4","url":%s}]}}`, u))
}
func TestManagedGooglevideoDiscoveryRotatesFreshInventory(t *testing.T) {
ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"}
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
for _, scenario := range []string{"rotating", "unavailable", "expired", "same-root", "unsafe-origin", "invalid-host", "control", "unknown", "mixed-safety", "cancel"} {
t.Run(scenario, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
calls := 0
collect := managedGooglevideoSamples(ids, func(ctx context.Context, id string) ([]byte, error) {
if _, ok := ctx.Deadline(); !ok {
t.Fatal("unbounded discovery")
}
if id != ids[calls%3] {
t.Fatal("not round robin")
}
calls++
host := hosts[(calls-1)%3]
expires := time.Now().Add(time.Hour)
if calls <= 3 {
switch scenario {
case "rotating":
host = hosts[0]
case "unavailable":
return nil, context.DeadlineExceeded
case "expired":
expires = time.Now().Add(5 * time.Minute)
case "unsafe-origin":
host += ":444"
case "invalid-host":
host = "not-a-ggc.googlevideo.com"
case "control":
host += "\n"
case "unknown":
return nil, errors.New("runner unavailable")
case "mixed-safety":
return nil, errors.Join(context.DeadlineExceeded, errors.New("cleanup failed"))
case "cancel":
cancel()
return nil, ctx.Err()
}
}
if scenario == "same-root" {
host = fmt.Sprintf("rr%d---sn-alpha-x.googlevideo.com", calls)
}
return samplePlayerBody(host, expires), nil
})
resources, err := collect(ctx)
if scenario == "unknown" || scenario == "mixed-safety" {
var fatal *managedRecoveryFatal
if !errors.As(err, &fatal) || calls != 1 || len(resources) != 0 {
t.Fatal(err, calls)
}
return
}
if scenario == "cancel" {
if !errors.Is(err, context.Canceled) || calls != 1 || len(resources) != 0 {
t.Fatal(err, calls)
}
return
}
if scenario == "same-root" {
var insufficient *InsufficientAdaptiveSamples
if !errors.As(err, &insufficient) || insufficient.Attempts != 9 || len(resources) != 0 || strings.Contains(err.Error(), "private-secret") {
t.Fatal(err, calls)
}
return
}
if err != nil || len(resources) != 3 || calls != 6 {
t.Fatal(err, calls, len(resources))
}
for i, r := range resources {
if r.host != hosts[i] || !strings.Contains(r.url, "private-secret") {
t.Fatal("changed signed inventory")
}
}
})
}
}
func TestManagedGooglevideoDiscoveryBoundsAndLateCancellation(t *testing.T) {
ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw", "aaaaaaaaaaa", "bbbbbbbbbbb"}
calls := 0
resources, err := managedGooglevideoSamples(ids, func(context.Context, string) ([]byte, error) {
calls++
return []byte(`{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"private-secret"}}`), nil
})(context.Background())
var insufficient *InsufficientAdaptiveSamples
if !errors.As(err, &insufficient) || calls != 15 || insufficient.Unsupported != 15 || len(resources) != 0 {
t.Fatal(err, calls)
}
for _, invalid := range [][]string{nil, {"bad"}, append(ids, "ccccccccccc")} {
_, err := managedGooglevideoSamples(invalid, func(context.Context, string) ([]byte, error) { t.Fatal("invalid config queried"); return nil, nil })(context.Background())
var fatal *managedRecoveryFatal
if !errors.As(err, &fatal) {
t.Fatal(err)
}
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
calls = 0
resources, err = managedGooglevideoSamples(ids[:3], func(context.Context, string) ([]byte, error) {
host := hosts[calls]
calls++
if calls == 3 {
cancel()
}
return samplePlayerBody(host, time.Now().Add(time.Hour)), nil
})(ctx)
if !errors.Is(err, context.Canceled) || len(resources) != 0 || calls != 3 {
t.Fatal(err, calls)
}
}
func TestExperimentalPromotionRequiresRepresentativeOwnedProof(t *testing.T) {
for _, enough := range []bool{false, true} {
t.Run(fmt.Sprint(enough), func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
r, _, resources, _ := googlevideoRuntimeFixture(t)
d.service = true
d.frontHealth = d.health
thumb := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
tx, err := d.db.Begin()
if err != nil {
t.Fatal(err)
}
experiment, err := insertStrategyCandidate(tx, "googlevideo", []string{resources[0].host}, managedGooglevideoArgs, nil, QualificationCold)
if err != nil {
_ = tx.Rollback()
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
old := []int64{front, thumb, avatar, experiment}
if err := d.transaction.Apply(d.db, old, GooglevideoScopeExperimental); err != nil {
t.Fatal(err)
}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{media: resources[0], images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/sample.jpg", "yt3.ggpht.com": "https://yt3.ggpht.com/sample.jpg"}, proxy: r.proxy, runner: adaptiveImageRunner}, nil
}
d.recheckExperimental = func(context.Context, storedCandidate, mediaResource, *managedTransaction) error { return nil }
d.mediaSamples = func(context.Context) ([]mediaResource, error) {
if enough {
return resources, nil
}
return resources[:1], nil
}
d.verifyGooglevideo = r.verifyGooglevideo
d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db)
id, err := recoverManagedExperimental(context.Background(), d)
if !enough {
if !errors.Is(err, errGooglevideoInsufficientSamples) || id != experiment {
t.Fatal(id, err)
}
assertActiveCandidates(t, d.transaction, d.db, old)
return
}
if err != nil || id == experiment {
t.Fatal(id, err)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front, thumb, avatar, id})
p, err := readGooglevideoProof(d.db, id)
if err != nil || p.Version != 2 || p.Adapter == nil || len(p.Hosts) != 3 {
t.Fatal(p, err)
}
c, err := readStrategyCandidate(d.db, experiment)
if err != nil || c.Qualified || c.Mode != QualificationCold {
t.Fatal("old row changed", c, err)
}
})
}
}
@@ -0,0 +1,286 @@
package main
import (
"context"
"errors"
"fmt"
"sort"
"strings"
"time"
"zapret-autotune/internal/adapter/youtube"
)
// Ordinary unresolved state, not a target/ownership guard failure.
type InsufficientAdaptiveSamples struct {
Attempts, DistinctHosts, Unavailable, Unsupported, Rejected int
}
func (e *InsufficientAdaptiveSamples) Error() string {
return fmt.Sprintf("DEGRADED: insufficient fresh ordinary signed GGC samples (attempts=%d distinct_hosts=%d unavailable=%d unsupported=%d rejected=%d)", e.Attempts, e.DistinctHosts, e.Unavailable, e.Unsupported, e.Rejected)
}
func (*InsufficientAdaptiveSamples) Unwrap() []error {
return []error{errGooglevideoInsufficientSamples}
}
// Sequential round-robin: at most 15 fresh requests, with no child goroutines.
const managedGooglevideoSampleRounds = 3
const managedGooglevideoDiscoveryTimeout = 3 * time.Minute
// Three hosts each need up to 120s warmup + six 35s transfers. Leave margin
// for daemon lifecycle as well; short-lived URLs are refetched, never rewritten.
const managedGooglevideoSampleTTL = youtube.InventoryTTL
func freshGooglevideoSample(r mediaResource) bool {
// The same exact GGC hostname syntax as the representative gate; reject bad
// inventory before counting it, without fetching any returned URL here.
prefix, root, ok := strings.Cut(strings.TrimSuffix(r.host, ".googlevideo.com"), "---sn-")
if !ok || !strings.HasSuffix(r.host, ".googlevideo.com") || !validDomain(r.host) || r.host != strings.ToLower(r.host) || root == "" || len(prefix) < 3 || !strings.HasPrefix(prefix, "rr") || strings.Trim(prefix[2:], "0123456789") != "" {
return false
}
if _, err := privateResourceURL(r.url, r.host); err != nil {
return false
}
return youtube.SignedSampleFresh(youtube.Resource{URL: r.url, Host: r.host, Kind: r.kind}, time.Now(), managedGooglevideoSampleTTL)
}
// Choose exactly three actual hosts passing the unchanged representative gate.
func representativeGooglevideoSamples(byHost map[string]mediaResource) []mediaResource {
hosts := make([]string, 0, len(byHost))
for host, r := range byHost {
if freshGooglevideoSample(r) {
hosts = append(hosts, host)
}
}
sort.Strings(hosts)
for i := 0; i < len(hosts); i++ {
for j := i + 1; j < len(hosts); j++ {
for k := j + 1; k < len(hosts); k++ {
if validateGooglevideoHosts([]string{hosts[i], hosts[j], hosts[k]}) == nil {
return []mediaResource{byHost[hosts[i]], byHost[hosts[j]], byHost[hosts[k]]}
}
}
}
}
return nil
}
func managedGooglevideoSamples(ids []string, request func(context.Context, string) ([]byte, error)) func(context.Context) ([]mediaResource, error) {
ids = append([]string(nil), ids...)
return func(ctx context.Context) ([]mediaResource, error) {
if _, err := requiredYoutubeAdapter(); err != nil {
return nil, err
}
if len(ids) == 0 || len(ids) > 5 || request == nil {
return nil, &managedRecoveryFatal{Err: errors.New("invalid sample collector configuration")}
}
for _, id := range ids {
if !videoIDPattern.MatchString(id) {
return nil, &managedRecoveryFatal{Err: errors.New("invalid sample video ID")}
}
}
bounded, cancel := context.WithTimeout(ctx, managedGooglevideoDiscoveryTimeout)
defer cancel()
stats := &InsufficientAdaptiveSamples{}
byHost := map[string]mediaResource{}
for round := 0; round < managedGooglevideoSampleRounds; round++ {
for _, id := range ids {
if ctx.Err() != nil {
return nil, managedRecoveryContextError(ctx, ctx.Err())
}
if bounded.Err() != nil {
return nil, stats
}
stats.Attempts++
body, err := request(bounded, id)
var fatal *managedRecoveryFatal
if ctx.Err() != nil || errors.As(err, &fatal) {
return nil, managedRecoveryContextError(ctx, err)
}
if err != nil {
if !managedResourceTransferNonpass(err, true, false) && !(bounded.Err() != nil && managedRecoveryTimeoutSafe(err)) {
return nil, &managedRecoveryFatal{Err: err}
}
stats.Unavailable++
if bounded.Err() != nil {
return nil, stats
}
continue
}
r, _, err := playerMediaResource(body)
if err != nil {
if errors.Is(err, errServiceAdapterUnavailable) {
return nil, err
}
stats.Unavailable++
continue
} // Missing/SABR is unknown, never fabricated.
if r.url == "" {
stats.Unsupported++
continue
}
if !freshGooglevideoSample(r) {
stats.Rejected++
continue
}
byHost[r.host] = r
stats.DistinctHosts = len(byHost)
if ctx.Err() != nil {
return nil, managedRecoveryContextError(ctx, ctx.Err())
}
if bounded.Err() != nil {
return nil, stats
}
if resources := representativeGooglevideoSamples(byHost); resources != nil {
return resources, nil
}
}
}
return nil, stats
}
}
func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) {
m := *d.transaction
journal, err := m.load()
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
if journal.Scope == GooglevideoScopeExperimental {
return recoverManagedExperimental(ctx, d)
}
return qualifyManagedGooglevideo(ctx, d)
}
// Both autonomous adaptive recovery and an operationally rechecked experiment
// use the same unchanged three-host owned proof and transactional apply gate.
func qualifyManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) {
m := *d.transaction
if !d.service || d.mediaSamples == nil || d.verifyGooglevideo == nil {
return 0, &InsufficientAdaptiveSamples{}
}
current, err := m.ActiveCandidates(d.db)
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
saved, err := listQualifiedCandidates(d.db, "googlevideo", 10)
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
// Revalidate saved native definitions first. Old proof never authorizes apply.
trials := []storedCandidate{}
for _, c := range saved {
if c.Mode == QualificationAdaptive && validateNativeGooglevideoArgs(c.Args) == nil {
trials = append(trials, c)
}
}
trials = append(trials, storedCandidate{Args: managedGooglevideoArgs})
for _, c := range trials {
resources, err := d.mediaSamples(ctx) // Fresh on every stage, URLs stay private.
if err != nil {
return 0, err
}
c.ID, c.Group, c.Mode = 0, "googlevideo", QualificationAdaptive
c.Endpoints = nil
for _, r := range resources {
c.Endpoints = append(c.Endpoints, r.host)
}
if validateGooglevideoHosts(c.Endpoints) != nil {
return 0, &InsufficientAdaptiveSamples{}
}
err = d.run(ctx, managedZapretScript, "stop")
var proof GooglevideoQualificationProof
if err == nil {
proof, err = d.verifyGooglevideo(ctx, c, resources)
}
var unsafe *managedAssetCleanupFailure
if errors.As(err, &unsafe) {
return 0, err
}
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
restoreErr := d.run(cleanup, managedZapretScript, "start")
cancel()
if restoreErr != nil {
return 0, &managedRecoveryFatal{Err: errors.Join(err, restoreErr)}
}
var fatal *managedRecoveryFatal
if ctx.Err() != nil || errors.As(err, &fatal) {
return 0, managedRecoveryContextError(ctx, err)
}
if errors.Is(err, errGooglevideoInsufficientSamples) {
return 0, &InsufficientAdaptiveSamples{}
}
if errors.Is(err, errCandidateNonpass) {
continue
}
if err != nil {
return 0, managedRecoveryContextError(ctx, err)
}
if err := ctx.Err(); err != nil {
return 0, err
}
tx, err := d.db.Begin()
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
id, err := insertStrategyCandidate(tx, "googlevideo", c.Endpoints, c.Args, nil, QualificationAdaptive)
if err == nil {
err = saveAdaptiveGooglevideoEvidence(tx, id, proof)
}
if err == nil {
err = tx.Commit()
} else {
_ = tx.Rollback()
}
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
desired, err := managedFullDesiredSet(d.db, current, []int64{id})
if err != nil {
return 0, &managedRecoveryFatal{Err: err}
}
m.Reload = func() error {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
return d.run(cleanup, managedZapretScript, "restart")
}
m.Health = func() (string, error) {
if d.frontHealth == nil || d.recheckGooglevideo == nil {
return "DEGRADED", errors.New("frontend and deployed media health required")
}
status, err := d.frontHealth(ctx)
if err != nil || status != "OK" {
return status, managedRecoveryContextError(ctx, err)
}
deployed, err := readStrategyCandidate(d.db, id)
if err == nil {
err = d.recheckGooglevideo(ctx, deployed, resources, &m)
}
if err != nil {
var fatal *managedRecoveryFatal
if ctx.Err() != nil || errors.As(err, &fatal) || errors.Is(err, context.Canceled) {
return "DEGRADED", managedRecoveryContextError(ctx, err)
}
if managedGooglevideoTransferNonpass(err) || errors.Is(err, errCandidateNonpass) {
return "DEGRADED", nil
}
return "DEGRADED", managedRecoveryContextError(ctx, err)
}
if ctx.Err() != nil {
return "DEGRADED", managedRecoveryContextError(ctx, ctx.Err())
}
return "OK", nil // Stage readiness only; whole-service monitor is unchanged.
}
err = m.Apply(d.db, desired, GooglevideoScopeSuffix)
var rollback *managedRollbackError
var rejection *managedHealthRejection
if err != nil && !errors.As(err, &rejection) && !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
return 0, &managedRecoveryFatal{Err: err}
}
if errors.As(err, &rollback) {
return 0, &managedRecoveryFatal{Err: err}
}
return id, err
}
return 0, errNoQualifiedStrategy
}
@@ -0,0 +1,239 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"image"
"image/png"
"io"
"reflect"
"strconv"
"strings"
"testing"
"time"
)
func TestManagedAdaptiveConfig(t *testing.T) {
for _, tc := range []struct {
mode, service, ids string
valid bool
}{
{"", "", "", true}, {"adaptive", "1", `["dQw4w9WgXcQ","aqz-KE-bpKQ","jNQXAC9IVRw"]`, true},
{"adaptive", "0", "", false}, {"unknown", "1", "", false},
{"adaptive", "1", `[]`, false}, {"adaptive", "1", `["bad"]`, false},
{"adaptive", "1", `["dQw4w9WgXcQ","dQw4w9WgXcQ"]`, false},
{"adaptive", "1", `["dQw4w9WgXcQ","aqz-KE-bpKQ","jNQXAC9IVRw","aaaaaaaaaaa","bbbbbbbbbbb","ccccccccccc"]`, false},
} {
lookup := func(key string) (string, bool) {
v := map[string]string{"ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE": tc.mode, "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE": tc.service, "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS": tc.ids}[key]
return v, v != ""
}
cfg, err := parseHealthConfig(lookup)
if (err == nil) != tc.valid {
t.Fatalf("%+v: %v", tc, err)
}
mode, ids, _ := managedGooglevideoConfig(lookup, cfg.VideoID)
if tc.mode == "" && (mode != QualificationCold || !reflect.DeepEqual(ids, []string{cfg.VideoID})) {
t.Fatal(cfg)
}
}
}
func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) {
for _, scenario := range []string{"qualified", "one-host", "unqualified-assets", "avatar-missing", "rollback", "apply-bad-body", "apply-mixed-guard", "apply-learns"} {
t.Run(scenario, func(t *testing.T) {
d, front, events := recoveryFixture(t, true)
r, _, resources, _ := googlevideoRuntimeFixture(t)
d.service, d.adaptive = true, true
d.frontHealth = d.health
assets := []int64{unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", scenario != "unqualified-assets"), unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)}
old := []int64{front}
if scenario != "unqualified-assets" {
old = append(old, assets...)
}
if err := d.transaction.Apply(d.db, old); err != nil {
t.Fatal(err)
}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/sample.jpg", "yt3.ggpht.com": "https://yt3.ggpht.com/sample.jpg"}, proxy: r.proxy, runner: adaptiveImageRunner}, nil
}
d.mediaSamples = func(context.Context) ([]mediaResource, error) {
if scenario == "one-host" {
return resources[:1], nil
}
return resources, nil
}
d.verifyGooglevideo = r.verifyGooglevideo
d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db)
applyCalls := 0
if strings.HasPrefix(scenario, "apply-") {
deployed := r
if scenario == "apply-mixed-guard" {
deployed.guard = func() error { return errors.Join(context.Canceled, errors.New("guard failed")) }
}
deployed.curl = func(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
applyCalls++
if scenario == "apply-learns" && applyCalls <= 2 {
return context.DeadlineExceeded
}
var body bytes.Buffer
err := r.curl(ctx, args, stdin, &body, diagnostic)
if err != nil {
return err
}
b := body.Bytes()
if scenario == "apply-bad-body" {
b = bytes.ReplaceAll(b, []byte("ftyp"), []byte("oops"))
}
_, err = out.Write(b)
return err
}
d.recheckGooglevideo = managedGooglevideoApplyRecheck(deployed, d.db)
}
if scenario == "avatar-missing" {
d.health = func(context.Context) (string, error) {
return "DEGRADED", &managedResourceFailure{"avatar", "yt3.ggpht.com", nil}
}
_, ids, err := managedGooglevideoConfig(func(key string) (string, bool) { return "1", key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" }, "dQw4w9WgXcQ")
if err != nil || len(ids) != 3 {
t.Fatal(ids, err)
}
calls := 0
d.mediaSamples = managedGooglevideoSamples(ids, func(_ context.Context, id string) ([]byte, error) {
if id != ids[calls%3] {
t.Fatal("wrong default video")
}
index := calls % 3
if calls < 3 {
index = 0
}
calls++
return samplePlayerBody(resources[index].host, time.Now().Add(time.Hour)), nil
})
}
if scenario == "rollback" {
d.recheckGooglevideo = func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
return errCandidateNonpass
}
}
result, err := recoverManagedTarget(context.Background(), d)
if scenario == "apply-mixed-guard" {
var fatal *managedRecoveryFatal
if !errors.As(err, &fatal) {
t.Fatal("mixed guard downgraded", err)
}
assertActiveCandidates(t, d.transaction, d.db, old)
return
}
if err != nil {
t.Fatal(err)
}
if scenario == "one-host" || scenario == "rollback" || scenario == "apply-bad-body" {
if result.GGCError == nil {
t.Fatal("unqualified stage succeeded")
}
assertActiveCandidates(t, d.transaction, d.db, old)
b, _ := d.transaction.read("config")
if strings.Contains(string(b), "--hostlist-domains=googlevideo.com") {
t.Fatal("unproven suffix")
}
return
}
if result.GGCError != nil || result.GooglevideoID == 0 {
t.Fatal(result)
}
c, err := readStrategyCandidate(d.db, result.GooglevideoID)
if err != nil || !c.Qualified || c.Mode != QualificationAdaptive {
t.Fatal(c, err)
}
p, err := readGooglevideoProof(d.db, c.ID)
if err != nil || p.SourceKind != NativeCircularPostWarmup {
t.Fatal(p, err)
}
j, err := d.transaction.load()
if err != nil || j.Scope != GooglevideoScopeSuffix {
t.Fatal(j, err)
}
assertActiveCandidates(t, d.transaction, d.db, append(old, c.ID))
if scenario == "apply-learns" && applyCalls < 20 {
t.Fatal("deployed warmup or full TLS transfers skipped", applyCalls)
}
if scenario == "avatar-missing" {
status, err := d.health(context.Background())
if status != "DEGRADED" || err == nil {
t.Fatal("whole-service health concealed")
}
}
if strings.Join(*events, ",") != "stop,start,restart" {
t.Fatal("restore must precede apply", *events)
}
})
}
}
func adaptiveImageRunner(_ context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
b, err := io.ReadAll(stdin)
if err != nil {
return err
}
raw, err := strconv.Unquote(strings.TrimSuffix(strings.TrimPrefix(string(b), "url = "), "\n"))
if err != nil {
return err
}
var body bytes.Buffer
if err := png.Encode(&body, image.NewRGBA(image.Rect(0, 0, 32, 32))); err != nil {
return err
}
fmt.Fprintf(out, "HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: %d\r\n\r\n", body.Len())
_, _ = out.Write(body.Bytes())
metrics, _ := json.Marshal(map[string]interface{}{"url_effective": raw, "ssl_verify_result": 0, "num_redirects": 0, "http_code": 200, "time_total": 0.1, "speed_download": 10000, "size_download": body.Len()})
fmt.Fprint(out, resourceTrailer, string(metrics))
tls := "TLSv1.2"
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
tls = "TLSv1.3"
}
fmt.Fprintln(diagnostic, "SSL connection using", tls)
return nil
}
func TestInsufficientAdaptiveSamplesCooldown(t *testing.T) {
err := &InsufficientAdaptiveSamples{}
if errors.Is(err, errNoQualifiedStrategy) || !errors.Is(err, errGooglevideoInsufficientSamples) || managedRecoveryCooldown(err, time.Minute) != time.Minute {
t.Fatal(err)
}
}
func TestGenuineNoQualifiedCooldown(t *testing.T) {
if managedRecoveryCooldown(errNoQualifiedStrategy, time.Minute) != 2*time.Hour {
t.Fatal("lost exhausted-search pause")
}
}
func TestManagedYouTubeDefaultSampleInventory(t *testing.T) {
for _, primary := range []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ"} {
lookup := func(key string) (string, bool) { return "1", key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" }
_, ids, err := managedGooglevideoConfig(lookup, primary)
want := []string{primary, "aqz-KE-bpKQ", "jNQXAC9IVRw"}
if primary == "aqz-KE-bpKQ" {
want = []string{primary, "jNQXAC9IVRw"}
}
if err != nil || !reflect.DeepEqual(ids, want) {
t.Fatal(ids, err)
}
}
_, ids, err := managedGooglevideoConfig(func(key string) (string, bool) {
switch key {
case "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE":
return "1", true
case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS":
return `["aaaaaaaaaaa"]`, true
}
return "", false
}, "dQw4w9WgXcQ")
if err != nil || !reflect.DeepEqual(ids, []string{"aaaaaaaaaaa"}) {
t.Fatal(ids, err)
}
}
@@ -0,0 +1,238 @@
package main
import (
"context"
"crypto/sha256"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"reflect"
"strings"
"time"
"zapret-autotune/internal/adapter/youtube"
)
// ponytail: one audited composite only. New native variants need an explicit
// semantic allowlist and regressions, not the generic shell-safe args validator.
const managedGooglevideoArgs = "--lua-init=@/opt/zapret2/lua/zapret-auto.lua --payload=all --in-range=-s34228 --lua-desync=circular:fails=1:retrans=2 --payload=tls_client_hello --lua-desync=fake:blob=fake_default_tls:tcp_ts=-1000:repeats=1:strategy=1 --lua-desync=multidisorder:pos=1,midsld:strategy=1 --lua-desync=fakeddisorder:pos=midsld:tcp_flags_set=SYN:strategy=2"
// Source inspected from the existing prod-derived prototype image. The default
// detector uses incoming seq >4096 for success and outgoing retrans <=32768;
// circular orchestrates only tagged instances, with contiguous strategy IDs.
// Incoming range must exceed 4096 (default x0-x0 cannot learn success).
const managedGooglevideoLuaHash = "aacfde0c95c3058f8e95f5d7d244398bdc03ebf846a8f17322129fb543366a3d"
const managedGooglevideoCollector = "owned-nfq-media-range-v1"
// Proof freshness is deliberately bounded to one day; expired proofs require a
// new owned run, never a refresh of their timestamp or cold summary rows.
const managedGooglevideoProofTTL = 24 * time.Hour
var errGooglevideoInsufficientSamples = errors.New("insufficient fresh ordinary signed GGC samples")
type GooglevideoRuntimeAttestation struct {
Collector string `json:"collector"`
LuaSHA256 string `json:"lua_sha256"`
RulesSHA256 string `json:"rules_sha256"`
WorkerArgs []string `json:"worker_args"`
CheckedAt time.Time `json:"checked_at"`
}
// Not serialized: DTO construction or unmarshalling cannot authorize a write.
// This binds the completed producer's exact facts, including failed warmups.
type googlevideoProducerSeal [32]byte
func googlevideoProofSeal(p GooglevideoQualificationProof) googlevideoProducerSeal {
raw, _ := json.Marshal(p)
return sha256.Sum256(raw)
}
func validateNativeGooglevideoArgs(args string) error {
if !reflect.DeepEqual(strings.Fields(args), strings.Fields(managedGooglevideoArgs)) {
return errors.New("unattested native circular semantics")
}
return nil
}
func (a *GooglevideoRuntimeAttestation) validate(c storedCandidate) error {
if err := a.validateStored(c); err != nil {
return err
}
if time.Since(a.CheckedAt) > managedGooglevideoProofTTL {
return errors.New("stale owned native attestation")
}
return nil
}
// Age affects new qualification, not reconstruction of journal-owned bytes.
func (a *GooglevideoRuntimeAttestation) validateStored(c storedCandidate) error {
if a == nil || a.Collector != managedGooglevideoCollector || a.LuaSHA256 != managedGooglevideoLuaHash || a.RulesSHA256 != fmt.Sprintf("%x", sha256.Sum256([]byte(managedAssetRules))) || !reflect.DeepEqual(a.WorkerArgs, managedProbeWorkerArgs(c.Endpoints, c.Args)) || a.CheckedAt.IsZero() || time.Since(a.CheckedAt) < -time.Minute {
return errors.New("missing, stale or mismatched owned native attestation")
}
return validateNativeGooglevideoArgs(c.Args)
}
// Caller gathers these resources from fresh Player queries for configured video
// samples. Never synthesize hosts by rewriting a signed URL. One-host inventory
// is an explicit unresolved result, before any worker or nft state is touched.
func newManagedGooglevideoVerifier(target managedDeps) func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error) {
r := newManagedAssetRuntime(target)
guard := r.guard
r.guard = func() error {
if err := guard(); err != nil {
return err
}
lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua")
if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash {
return errors.New("unverified native automation Lua source")
}
return nil
}
return r.verifyGooglevideo
}
func (r managedAssetRuntime) verifyGooglevideo(ctx context.Context, c storedCandidate, resources []mediaResource) (GooglevideoQualificationProof, error) {
p := GooglevideoQualificationProof{Version: 2, Mode: QualificationAdaptive, SourceKind: NativeCircularPostWarmup, Args: c.Args}
if c.Group != "googlevideo" || c.Mode != QualificationAdaptive {
return p, errors.New("adaptive googlevideo definition required")
}
if err := validateNativeGooglevideoArgs(c.Args); err != nil {
return p, err
}
module, err := requiredYoutubeAdapter()
if err != nil {
return p, err
}
binding := module.Binding()
p.Adapter = &binding
byHost := map[string]mediaResource{}
var samples []youtube.Resource
for _, resource := range resources {
if _, err := privateResourceURL(resource.url, resource.host); err != nil {
return p, errGooglevideoInsufficientSamples
}
byHost[resource.host] = resource
samples = append(samples, youtube.Resource{URL: resource.url, Host: resource.host, Kind: resource.kind})
}
if validateGooglevideoHosts(c.Endpoints) != nil {
return p, errGooglevideoInsufficientSamples
}
plan, err := module.PlanQualification(c.Endpoints, samples, time.Now())
if err != nil {
return p, errGooglevideoInsufficientSamples
}
if !reflect.DeepEqual(plan.Hosts, c.Endpoints) {
return p, errAdapterPlan
}
if err := validateAdapterPlan(binding, plan, byHost); err != nil {
return p, err
}
for _, host := range plan.Hosts {
p.Hosts = append(p.Hosts, GooglevideoHostProof{Host: host})
}
err = r.withOwnedWorker(ctx, c.Endpoints, c.Args, func(probeCtx context.Context) error {
hosts, err := executeAdapterPlan(probeCtx, binding, plan, byHost,
func(ctx context.Context, resource mediaResource, protocol string) (adapterMeasurement[GooglevideoPass], error) {
_, pass, err := collectMediaTransfer(ctx, r.proxy, resource, protocol, r.curl)
return adapterMeasurement[GooglevideoPass]{value: pass, bytes: pass.Bytes}, err
}, managedGooglevideoTransferNonpass)
for i, h := range hosts {
p.Hosts[i] = GooglevideoHostProof{Host: h.host, WarmupLimit: h.warmupLimit,
WarmupAttempts: h.warmupAttempts, WarmupFailures: h.warmupFailures, WarmupMS: h.warmupMS, Passes: h.samples}
}
return err
})
if err != nil {
return p, err
}
// Nothing attests until the owned consumer has been stopped and joined.
p.Attestation = &GooglevideoRuntimeAttestation{Collector: managedGooglevideoCollector, LuaSHA256: managedGooglevideoLuaHash, RulesSHA256: fmt.Sprintf("%x", sha256.Sum256([]byte(managedAssetRules))), WorkerArgs: managedProbeWorkerArgs(c.Endpoints, c.Args), CheckedAt: time.Now().UTC()}
if err := p.validate(c); err != nil {
return p, err
}
seal := googlevideoProofSeal(p)
p.producerSeal = &seal
return p, nil
}
func managedGooglevideoTransferNonpass(err error) bool {
return managedResourceTransferNonpass(err, false, false)
}
func managedResourceTransferNonpass(err error, metadata, image bool) bool {
// Collector normalization wraps known transfer failures as fatal for generic
// consumers. Only that boundary may unwrap the wrapper; guard/cleanup failures
// and mixed error sets must never authorize absence or a daemon restart.
if e, ok := err.(*resourceError); ok {
cause := e.cause
if fatal, ok := cause.(*managedRecoveryFatal); ok {
cause = fatal.Err
}
return managedResourceTransferNonpass(cause, metadata, image)
}
if _, ok := err.(*managedRecoveryFatal); ok {
return false
}
if joined, ok := err.(interface{ Unwrap() []error }); ok {
children := joined.Unwrap()
if len(children) == 0 {
return false
}
for _, child := range children {
if !managedResourceTransferNonpass(child, metadata, image) {
return false
}
}
return true
}
if wrapped, ok := err.(interface{ Unwrap() error }); ok {
return managedResourceTransferNonpass(wrapped.Unwrap(), metadata, image)
}
if _, ok := err.(imageBodyRejection); ok {
return image
}
// Metadata curl uses --fail. A non-2xx response is missing inventory,
// never media rejection or candidate-absence evidence.
if exit, ok := err.(*exec.ExitError); metadata && ok && exit.ExitCode() == 22 {
return true
}
if managedAssetTransferNonpass(err) {
return true
}
if err == nil {
return false
}
if err.Error() == "invalid media range or container" {
return true
}
return false
}
func validateGooglevideoHosts(hosts []string) error {
for _, host := range hosts {
if !validDomain(host) {
return errGooglevideoInsufficientSamples
}
}
module, err := requiredYoutubeAdapter()
if err != nil {
return err
}
if module.ValidateHosts(hosts) != nil {
return errGooglevideoInsufficientSamples
}
return nil
}
// Historical v1 reconstruction must not depend on a future registry default.
// This pinned contract remains available even when the execution module is absent.
func validateLegacyYoutubeV1Hosts(hosts []string) error {
for _, host := range hosts {
if !validDomain(host) {
return errGooglevideoInsufficientSamples
}
}
return (youtube.Module{}).ValidateHosts(hosts)
}
@@ -0,0 +1,297 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"strconv"
"strings"
"testing"
"time"
)
// No network: fake owned lifecycle and wire replies exercise the producer, not
// hardware readiness. The production factory adds the real guard/Lua hash gate.
func googlevideoRuntimeFixture(t *testing.T) (managedAssetRuntime, storedCandidate, []mediaResource, *[]string) {
t.Helper()
r, events, _ := assetRuntimeFixture(t)
c := storedCandidate{Group: "googlevideo", Mode: QualificationAdaptive, Args: managedGooglevideoArgs, Endpoints: []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}}
var resources []mediaResource
for _, host := range c.Endpoints {
resources = append(resources, mediaResource{url: fmt.Sprintf("https://%s/videoplayback?expire=%d&sig=private-secret", host, time.Now().Add(time.Hour).Unix()), host: host, kind: "video/mp4"})
}
r.curl = func(_ context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
*events = append(*events, "media")
if strings.Contains(strings.Join(args, " "), "private-secret") || !strings.Contains(strings.Join(args, " "), "--range 0-65535") {
t.Fatal("private URL in argv or no range")
}
b, err := io.ReadAll(stdin)
if err != nil {
return err
}
raw, err := strconv.Unquote(strings.TrimSuffix(strings.TrimPrefix(string(b), "url = "), "\n"))
if err != nil {
return err
}
body := make([]byte, 1024)
copy(body, []byte{0, 0, 0, 24, 'f', 't', 'y', 'p'})
fmt.Fprint(out, "HTTP/1.1 206 Partial Content\r\nContent-Type: video/mp4\r\nContent-Length: 1024\r\nContent-Range: bytes 0-1023/4096\r\n\r\n")
out.Write(body)
metrics, _ := json.Marshal(map[string]interface{}{"url_effective": raw, "ssl_verify_result": 0, "num_redirects": 0, "http_code": 206, "time_total": 0.1, "speed_download": 10000, "size_download": len(body)})
fmt.Fprint(out, resourceTrailer, string(metrics))
tls := "TLSv1.2"
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
tls = "TLSv1.3"
}
fmt.Fprintln(diagnostic, "SSL connection using", tls)
return nil
}
return r, c, resources, events
}
func TestManagedGooglevideoOwnedPostWarmupProducer(t *testing.T) {
r, c, resources, events := googlevideoRuntimeFixture(t)
runner, calls := r.curl, 0
r.curl = func(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
assertManagedDNSArgs(t, args, "socks5h://127.0.0.1:1080")
calls++
if calls <= 2 {
return context.DeadlineExceeded
}
return runner(ctx, args, stdin, out, diagnostic)
}
p, err := r.verifyGooglevideo(context.Background(), c, resources)
if err != nil || p.producerSeal == nil || p.validate(c) != nil {
t.Fatal(p, err)
}
if calls != 26 || p.Hosts[0].WarmupAttempts != 4 || p.Hosts[0].WarmupFailures != 2 || len(p.Hosts[0].Passes) != 6 {
t.Fatal("hidden warmup", p.Hosts, calls)
}
joined := strings.Join(*events, ",")
if strings.Count(joined, "start") != 1 || !strings.HasSuffix(joined, "delete,join") {
t.Fatal(joined)
}
raw, _ := json.Marshal(p)
if strings.Contains(string(raw), "private-secret") || strings.Contains(string(raw), "videoplayback") {
t.Fatal("durable URL leak")
}
for _, h := range p.Hosts {
for _, pass := range h.Passes {
if pass.RangeStart != 0 || pass.RangeEnd != 1023 || pass.ResourceBytes != 4096 {
t.Fatal(pass)
}
}
}
}
func TestManagedGooglevideoRefusesUnresolvedAndUnknownSemantics(t *testing.T) {
for _, mode := range []string{"missing", "one-host", "same-root", "rewritten", "expired", "unsigned", "no-incoming", "blocking-incoming", "detector-payload", "detector", "stage-gap", "ignored-stage", "unknown-init", "unknown-key", "out-range"} {
t.Run(mode, func(t *testing.T) {
r, c, resources, events := googlevideoRuntimeFixture(t)
switch mode {
case "missing":
resources = nil
case "one-host":
resources = resources[:1]
case "same-root":
c.Endpoints[1] = "rr2---sn-alpha.googlevideo.com"
case "rewritten":
resources[0].host = c.Endpoints[1]
case "expired":
resources[0].url = "https://" + resources[0].host + "/videoplayback?expire=1&sig=private-secret"
case "unsigned":
resources[0].url = strings.Split(resources[0].url, "&sig=")[0]
case "no-incoming":
c.Args = strings.Replace(c.Args, " --in-range=-s34228", "", 1)
case "blocking-incoming":
c.Args = strings.Replace(c.Args, "--in-range=-s34228", "--in-range=0", 1)
case "detector-payload":
c.Args = strings.Replace(c.Args, "--payload=all", "--payload=tls_client_hello", 1)
case "detector":
c.Args = strings.Replace(c.Args, "circular:fails=1", "circular:success_detector=cond_true:fails=1", 1)
case "stage-gap":
c.Args = strings.Replace(c.Args, "strategy=2", "strategy=3", 1)
case "ignored-stage":
c.Args = strings.Replace(c.Args, ":strategy=2", "", 1)
case "unknown-init":
c.Args += " --lua-init=@/opt/zapret2/lua/unknown.lua"
case "unknown-key":
c.Args = strings.Replace(c.Args, "circular:fails=1", "circular:unknown=1:fails=1", 1)
case "out-range":
c.Args += " --out-range=0"
}
p, err := r.verifyGooglevideo(context.Background(), c, resources)
if err == nil || p.producerSeal != nil || len(*events) != 0 {
t.Fatal("captured unresolved inventory/config", p, err, *events)
}
})
}
}
func TestManagedGooglevideoWarmupBoundAndCleanupAttestation(t *testing.T) {
for _, mode := range []string{"nonpass", "root404", "cleanup", "queue-lost", "worker-exit"} {
t.Run(mode, func(t *testing.T) {
r, c, resources, events := googlevideoRuntimeFixture(t)
calls := 0
if mode == "nonpass" {
r.curl = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
calls++
return context.DeadlineExceeded
}
}
if mode == "root404" {
r.curl = func(_ context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
calls++
b, err := io.ReadAll(stdin)
if err != nil {
return err
}
raw, err := strconv.Unquote(strings.TrimSuffix(strings.TrimPrefix(string(b), "url = "), "\n"))
if err != nil {
return err
}
var diag bytes.Buffer
resourceWire(raw, "root", out, &diag)
tls := diag.String()
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
tls = strings.ReplaceAll(tls, "TLSv1.2", "TLSv1.3")
}
io.WriteString(diagnostic, tls)
return nil
}
}
if mode == "cleanup" {
nft := r.nft
r.nft = func(ctx context.Context, input string) (string, error) {
if strings.HasPrefix(input, "delete") {
return "", errors.New("failed")
}
return nft(ctx, input)
}
}
if mode == "queue-lost" {
queue, n := r.queue, 0
r.queue = func(pid int) (bool, error) {
n++
if n == 3 {
return false, nil
}
return queue(pid)
}
}
if mode == "worker-exit" {
start := r.start
r.start = func(args []string) (managedAssetWorker, error) {
w, err := start(args)
ch := make(chan error, 1)
ch <- errors.New("dead")
w.done = ch
return w, err
}
}
p, err := r.verifyGooglevideo(context.Background(), c, resources)
if err == nil || p.Attestation != nil || p.producerSeal != nil {
t.Fatal("attested failed lifecycle", p, err)
}
if (mode == "nonpass" || mode == "root404") && (calls != 10 || p.Hosts[0].WarmupFailures != 10 || len(p.Hosts[0].Passes) != 0 || !strings.HasSuffix(strings.Join(*events, ","), "delete,join")) {
t.Fatal(p, calls, *events)
}
})
}
}
func TestManagedGooglevideoAdaptiveStorageAndReadGate(t *testing.T) {
r, c, resources, _ := googlevideoRuntimeFixture(t)
p, err := r.verifyGooglevideo(context.Background(), c, resources)
if err != nil {
t.Fatal(err)
}
db := testStore(t)
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
defer tx.Rollback()
id, err := insertStrategyCandidate(tx, c.Group, c.Endpoints, c.Args, nil, c.Mode)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(p)
var dto GooglevideoQualificationProof
if err := json.Unmarshal(raw, &dto); err != nil {
t.Fatal(err)
}
if dto.validate(c) != nil {
t.Fatal("DTO should be structurally valid")
}
if err := saveAdaptiveGooglevideoEvidence(tx, id, dto); err == nil {
t.Fatal("DTO alone authorized a write")
}
p.Hosts[0].WarmupFailures++
if err := saveAdaptiveGooglevideoEvidence(tx, id, p); err == nil {
t.Fatal("modified producer proof accepted")
}
p.Hosts[0].WarmupFailures--
if err := saveAdaptiveGooglevideoEvidence(tx, id, p); err != nil {
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
var n int
if err := db.QueryRow("SELECT count(*) FROM candidate_evidence WHERE candidate_id=?", id).Scan(&n); err != nil || n != 0 {
t.Fatal("manufactured cold metrics", n, err)
}
stored, err := readStrategyCandidate(db, id)
if err != nil || !stored.Qualified || stored.Mode != QualificationAdaptive {
t.Fatal(stored, err)
}
list, err := listQualifiedCandidates(db, "googlevideo", 1)
if err != nil || len(list) != 1 || list[0].ID != id {
t.Fatal(list, err)
}
if err := validateGooglevideoSuffixEvidence(db, stored); err != nil {
t.Fatal(err)
}
for _, mode := range []string{"malformed", "stale", "args", "attestation", "hidden-warmup", "media", "secret"} {
t.Run(mode, func(t *testing.T) {
proof, err := decodeGooglevideoProof(string(raw))
if err != nil {
t.Fatal(err)
}
switch mode {
case "stale":
proof.Attestation.CheckedAt = time.Now().Add(-25 * time.Hour)
case "args":
proof.Args = "--lua-desync=fake"
case "attestation":
proof.Attestation = nil
case "hidden-warmup":
proof.Hosts[0].WarmupAttempts = 0
case "media":
proof.Hosts[0].Passes[0].Status = 404
}
bad, _ := json.Marshal(proof)
if mode == "malformed" {
bad = []byte(`{"version":"bad"}`)
}
if mode == "secret" {
bad = append(bad[:len(bad)-1], []byte(`,"signed_url":"private-secret"}`)...)
}
if _, err := db.Exec("UPDATE googlevideo_scope_evidence SET proof=? WHERE candidate_id=?", string(bad), id); err != nil {
t.Fatal(err)
}
stored, err := readStrategyCandidate(db, id)
if err != nil || stored.Qualified {
t.Fatal(stored, err)
}
list, err := listQualifiedCandidates(db, "googlevideo", 1)
if err != nil || len(list) != 0 {
t.Fatal(list, err)
}
})
}
}
@@ -0,0 +1,69 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"testing"
"time"
)
func TestManagedGooglevideoFreshPrivateSamples(t *testing.T) {
ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"}
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
for _, scenario := range []string{"diverse", "duplicate", "expired", "sabr"} {
t.Run(scenario, func(t *testing.T) {
calls := 0
collect := managedGooglevideoSamples(ids, func(_ context.Context, id string) ([]byte, error) {
i := calls % len(ids)
calls++
if id != ids[i] {
t.Fatal("configured ID changed", id)
}
if scenario == "sabr" {
return []byte(`{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"private-secret"}}`), nil
}
host := hosts[i]
if scenario == "duplicate" {
host = hosts[0]
}
expires := time.Now().Add(time.Hour).Unix()
if scenario == "expired" {
expires = time.Now().Unix() + 30
}
url, _ := json.Marshal(fmt.Sprintf("https://%s/videoplayback?expire=%d&sig=private-secret", host, expires))
return []byte(fmt.Sprintf(`{"playabilityStatus":{"status":"OK"},"streamingData":{"adaptiveFormats":[{"mimeType":"video/mp4","url":%s}]}}`, url)), nil
})
for n := 0; n < 2; n++ {
resources, err := collect(context.Background())
if scenario != "diverse" {
var insufficient *InsufficientAdaptiveSamples
if !errors.As(err, &insufficient) || strings.Contains(err.Error(), "private-secret") || len(resources) != 0 {
t.Fatal("unresolved samples accepted or leaked", resources, err)
}
continue
}
var actual []string
for _, r := range resources {
actual = append(actual, r.host)
if !strings.Contains(r.url, "sig=private-secret") {
t.Fatal("signed URL changed")
}
}
if err != nil || !reflect.DeepEqual(actual, hosts) {
t.Fatal(actual, err)
}
}
want := 2 * len(ids)
if scenario != "diverse" {
want *= managedGooglevideoSampleRounds
}
if calls != want {
t.Fatal("samples were reused", calls)
}
})
}
}
+35 -3
View File
@@ -16,7 +16,7 @@ import (
// Provisioning owns executable, database, proxy and target paths; no path flags.
func managedMonitorConfig(args []string, lookup func(string) (string, bool)) (healthConfig, error) {
cfg, err := parseHealthConfig(lookup)
cfg, err := parseManagedHealthConfig(lookup)
if err != nil {
return cfg, err
}
@@ -57,6 +57,17 @@ func managedMonitorCLI(args []string, stdout, stderr io.Writer, target managedDe
}
func managedMonitorCLIContext(ctx context.Context, args []string, stdout, stderr io.Writer, target managedDeps, supervise func(context.Context, managedMonitorDeps) error) (code int) {
resourcePath, _ := target.lookup("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
if len(args) == 2 && args[0] == "--resource-config" && args[1] != "" {
return resourceMonitorCLI(ctx, args[1], stdout, stderr, target)
}
if resourcePath != "" {
if len(args) != 0 {
fmt.Fprintln(stderr, "configerror: generic monitor accepts only --resource-config FILE")
return 64
}
return resourceMonitorCLI(ctx, resourcePath, stdout, stderr, target)
}
cfg, err := managedMonitorConfig(args, target.lookup)
if err != nil {
fmt.Fprintln(stderr, "error: invalid managed monitor configuration")
@@ -111,13 +122,27 @@ func managedMonitorCLIContext(ctx context.Context, args []string, stdout, stderr
now: time.Now, wait: waitManagedMonitor,
reportRecoveryError: func(error) { fmt.Fprintln(stderr, "managed recovery failed (details suppressed)") },
}
if recovery.service {
monitor.reportProbeError = func(err error) {
var failure *managedResourceFailure
if errors.As(err, &failure) {
fmt.Fprintln(stderr, failure.PublicError())
} else {
fmt.Fprintln(stderr, "DEGRADED: service scope unresolved")
}
}
}
if err := wireManagedRecovery(&monitor, recovery, func(error) {
fmt.Fprintln(stderr, "managed GGC unresolved; monitoring continues (not playback health)")
}); err != nil {
fmt.Fprintln(stderr, "error: managed monitor wiring refused")
return 78
}
fmt.Fprintln(stdout, "managed monitor: Player API via owned loopback SOCKS; not playback health")
if recovery.service {
fmt.Fprintln(stdout, "managed monitor: service resource checks; SABR unknown; owned temporary asset verifier")
} else {
fmt.Fprintln(stdout, "managed monitor: Player API via owned loopback SOCKS; not playback health")
}
err = supervise(ctx, monitor)
if err != nil {
var fatal *managedRecoveryFatal
@@ -174,7 +199,14 @@ func managedMonitorRecoveryAdapters(target managedDeps, db *sql.DB, scanCommand
defer cancel()
return command(bounded, path, args...)
}
return managedRecoveryDependencies(target, db, scan, check, run)
d, err := managedRecoveryDependencies(target, db, scan, check, run)
if err == nil && d.service {
d.verifyAssets = newManagedAssetVerifier(target)
d.recheckExperimental = newManagedExperimentalRecheck(target, d)
d.verifyGooglevideo = newManagedGooglevideoVerifier(target)
d.recheckGooglevideo = newManagedGooglevideoApplyRecheck(target, db)
}
return d, err
}
// Joined cancellation plus failed cleanup/DB/lock close is not a graceful exit.
+59 -12
View File
@@ -26,10 +26,21 @@ func TestManagedPrestartHelper(t *testing.T) {
for len(args) > 0 && args[0] != "--" {
args = args[1:]
}
if len(args) < 3 {
if len(args) < 2 {
os.Exit(1)
}
args = args[2:] // -- managed COMMAND
args = args[1:] // -- COMMAND ...
if len(args) == 1 && args[0] == "target-config-check" {
// Use the production validator, not a fixture-only success response.
os.Exit(run(args, os.Stdout, os.Stderr))
}
if len(args) == 2 && args[0] == "dns-forwarder" && (args[1] == "mode" || args[1] == "address") {
os.Exit(run(args, os.Stdout, os.Stderr))
}
if len(args) < 2 || args[0] != "managed" {
os.Exit(1)
}
args = args[1:] // managed COMMAND
if args[0] == "monitor" {
os.Exit(78) // Stop after initial activation; no scans or network.
}
@@ -41,7 +52,7 @@ func TestManagedPrestartHelper(t *testing.T) {
}
func TestManagedEntrypointPrestartReconciliation(t *testing.T) {
for _, mode := range []string{"fresh", "baseline", "committed", "prepared", "drift", "unknown", "legacy", "corrupt-backup", "unqualified", "reload-failure"} {
for _, mode := range []string{"fresh", "baseline", "committed", "prepared", "drift", "unknown", "legacy", "corrupt-backup", "unqualified", "reload-failure", "invalid-dns", "invalid-service"} {
t.Run(mode, func(t *testing.T) {
d, env, _ := managedCLIFixture(t)
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
@@ -125,7 +136,7 @@ func TestManagedEntrypointPrestartReconciliation(t *testing.T) {
t.Fatal(err)
}
write("awk", "#!/bin/sh\nexit 0\n")
child := "#!/bin/sh\nsleeper=\ntrap '[ -z \"$sleeper\" ] || { kill \"$sleeper\" 2>/dev/null || true; wait \"$sleeper\" 2>/dev/null || true; }; exit 0' TERM INT\nwhile :; do sleep 0.1 & sleeper=$!; wait \"$sleeper\"; done\n"
child := "#!/bin/sh\n[ -z \"${SS_PASSWORD-}\" ] || exit 1\n[ \"$#\" -eq 2 ] || { [ \"$#\" -eq 4 ] && [ \"$3\" = -d ] && [ \"$4\" = \"$SS_DNS_SERVER\" ]; } || exit 1\n: > \"$FIXTURE_CHILD_STARTED\"\nsleeper=\ntrap '[ -z \"$sleeper\" ] || { kill \"$sleeper\" 2>/dev/null || true; wait \"$sleeper\" 2>/dev/null || true; }; exit 0' TERM INT\nwhile :; do sleep 0.1 & sleeper=$!; wait \"$sleeper\"; done\n"
write("ss-server", child)
write("ss-local", child)
log := filepath.Join(root, "log")
@@ -156,11 +167,35 @@ func TestManagedEntrypointPrestartReconciliation(t *testing.T) {
// The CLI guard sees fixed production bindings; only fake daemon reads
// use the fixture config, via its own script rather than CLI path flags.
cmd.Env = append(cmd.Env, "FIXTURE_CONFIG="+filepath.Join(d.directory, "config"))
childStarted := filepath.Join(root, "child-started")
// This legacy daemon fixture deliberately exercises explicit native DNS.
cmd.Env = append(cmd.Env, "FIXTURE_CHILD_STARTED="+childStarted, "ZAPRET_AUTOTUNE_DNS_MODE=native", "SS_DNS_SERVER=8.8.8.8")
wantExit := 78
if mode == "invalid-dns" {
cmd.Env = append(cmd.Env, "SS_DNS_SERVER=8.8.8.8\n--invalid")
wantExit = 64
}
if mode == "invalid-service" {
cmd.Env = append(cmd.Env, "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE=invalid")
wantExit = 64
}
output, err := cmd.CombinedOutput()
var exit *exec.ExitError
if ctx.Err() != nil || !errors.As(err, &exit) || exit.ExitCode() != 78 {
if ctx.Err() != nil || !errors.As(err, &exit) || exit.ExitCode() != wantExit {
t.Fatal(err, ctx.Err(), string(output))
}
if bytes.Contains(output, []byte(strings.Repeat("a", 32))) {
t.Fatal("startup exposed password")
}
if wantExit == 64 {
if _, err := os.Stat(childStarted); !errors.Is(err, os.ErrNotExist) {
t.Fatal("invalid configuration launched a service", err)
}
files, err := os.ReadDir(ss)
if err != nil || len(files) != 0 || !strings.Contains(string(output), "configerror:") {
t.Fatal("invalid configuration did not fail before config creation", err, string(output))
}
}
b, _ := os.ReadFile(log)
allowed := mode == "fresh" || mode == "baseline" || mode == "committed" || mode == "prepared"
if !allowed && mode != "reload-failure" && len(b) != 0 {
@@ -317,11 +352,11 @@ func TestManagedMonitorProductionAdapterBinding(t *testing.T) {
recovery, _, _ := recoveryFixture(t, false)
target, _, _ := managedCLIFixture(t)
target.directory = recovery.transaction.Dir
target.player = func(ctx context.Context, args []string, out, _ io.Writer) error {
target.player = func(ctx context.Context, args []string, _ io.Reader, out, _ io.Writer) error {
if ctx.Err() != nil {
t.Fatal(ctx.Err())
}
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5://127.0.0.1:11080|") {
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5h://127.0.0.1:11080|") {
t.Fatal(args)
}
_, err := io.WriteString(out, `{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"https://fresh.googlevideo.com/video?secret=never-print"}}`)
@@ -457,7 +492,7 @@ func TestManagedMonitorProductionAbsenceDiagnostics(t *testing.T) {
var fatal *managedRecoveryFatal
wantFatal := strings.Contains(mode, "nfq")
if errors.As(err, &fatal) != wantFatal || (!wantFatal && !errors.Is(err, context.Canceled)) {
t.Fatalf("mode=%s err=%v", mode, err)
t.Fatalf("mode=%s err=%v reported=%v", mode, err, reported)
}
if strings.Contains(errors.Join(err, reported).Error(), "secret-never-print") {
t.Fatal("diagnostics leaked")
@@ -545,9 +580,9 @@ func TestManagedMonitorCLIBindingAndCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
requests := 0
d.player = func(ctx context.Context, args []string, out, _ io.Writer) error {
d.player = func(ctx context.Context, args []string, _ io.Reader, out, _ io.Writer) error {
requests++
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5://127.0.0.1:11080|") {
if !strings.Contains(strings.Join(args, "|"), "--noproxy||--proxy|socks5h://127.0.0.1:11080|") {
t.Fatal(args)
}
if requests == 1 {
@@ -641,6 +676,14 @@ func TestManagedEntrypointSupervision(t *testing.T) {
write("ss-local", child)
zapret := write("zapret", "#!/bin/sh\necho zapret-$1 >> \"$LOG\"\nstage=$1\n"+startupFailure+"if [ \"$MODE:$1\" = ss-early:start ] && [ \"$(grep -c zapret-start \"$LOG\")\" -gt 1 ]; then\nuntil [ \"$(grep -c 'stopped-.*/ss-local' \"$LOG\")\" -ge \"$(grep -c zapret-start \"$LOG\")\" ]; do sleep 0.02; done\nsleep 0.02\nfi\nif [ \"$MODE:$1\" = stop-failure:stop ]; then exit 1; fi\nif [ \"$MODE:$1\" = stop-hang:stop ]; then trap '' TERM INT; exec sleep 300; fi\n")
monitor := write("autotune", "#!/bin/sh\nif [ \"$2\" != monitor ]; then\necho managed-$2 >> \"$LOG\"\nstage=$2\n"+startupFailure+"exit 0\nfi\nsleeper=\ntrap '"+joinSleeper+"echo monitor-stopped >> \"$LOG\"; case \"$MODE\" in *unsafe) exit 78;; *unknown) exit 7;; ss-technical) exit 75;; esac; exit 0' TERM INT\necho managed-monitor >> \"$LOG\"\nif [ \"$MODE\" = term-hang ]; then trap '' TERM INT; exec sleep 300; fi\ncase \"$MODE\" in fatal) exit 78;; technical|stop-failure|stop-hang) exit 75;; panic) exit 2;; killed) kill -KILL $$;; segv) exit 139;; zero) exit 0;; unknown) exit 7;; esac\nwhile :; do sleep 0.1 & sleeper=$!; wait \"$sleeper\"; done\n")
// This legacy lifecycle fixture has no DNS service. Implement only
// the explicit native getters; dedicated Python fixtures own DoH ACKs.
monitorData, err := os.ReadFile(monitor)
if err != nil {
t.Fatal(err)
}
getters := "if [ \"$1\" = dns-forwarder ]; then\ncase \"$2\" in mode) echo native;; address) echo;; *) exit 64;; esac\nexit 0\nfi\n"
write("autotune", strings.Replace(string(monitorData), "#!/bin/sh\n", "#!/bin/sh\n"+getters, 1))
data, err := os.ReadFile("../../managed-target-entrypoint.sh")
if err != nil {
t.Fatal(err)
@@ -659,13 +702,17 @@ func TestManagedEntrypointSupervision(t *testing.T) {
text = shortened
}
if mode == "stop-hang" {
text = strings.Replace(text, "timeout -k 5 30", "timeout -k 1 1", 1)
shortened := strings.Replace(text, "cleanup_command 30 "+zapret+" stop", "timeout -k 1 1 "+zapret+" stop", 1)
if shortened == text {
t.Fatal("fixture did not shorten stop watchdog")
}
text = shortened
}
entry := write("entrypoint", text)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "/bin/sh", entry)
cmd.Env = append(os.Environ(), "PATH="+root+":"+os.Getenv("PATH"), "LOG="+log, "MODE="+mode, "ZAPRET_CONFIG="+filepath.Join(root, "config"), "SS_PASSWORD="+strings.Repeat("a", 32))
cmd.Env = append(os.Environ(), "PATH="+root+":"+os.Getenv("PATH"), "LOG="+log, "MODE="+mode, "ZAPRET_CONFIG="+filepath.Join(root, "config"), "SS_PASSWORD="+strings.Repeat("a", 32), "ZAPRET_AUTOTUNE_DNS_MODE=native")
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
var shutdownSnapshot string
stopMonitorSession := func() {
+1 -1
View File
@@ -27,7 +27,7 @@ func TestManagedSingleGroupCLI(t *testing.T) {
t.Fatalf("apply: %d %s %s", code, out, diagnostic)
}
code, out, diagnostic := callManaged(t, d, "status")
if code != 0 || !strings.Contains(out, "groups="+group+" ") || !strings.Contains(out, "unresolved_groups="+missing+"\n") || !strings.Contains(out, "Live health: NOT CHECKED") {
if code != 0 || !strings.Contains(out, "groups="+group+" ") || !strings.Contains(out, "unresolved_groups="+missing+" covered="+group+"\n") || !strings.Contains(out, "Live health: NOT CHECKED") {
t.Fatalf("partial status: %d %s %s", code, out, diagnostic)
}
code, out, diagnostic = callManaged(t, d, "active-ids")
+154 -15
View File
@@ -30,24 +30,41 @@ type managedRecoveryDeps struct {
health func(context.Context) (string, error)
run func(context.Context, string, ...string) error
jobTimeout time.Duration // Zero uses the production budget; tests may shorten it.
service bool
frontHealth func(context.Context) (string, error)
resources func(context.Context) (managedServiceInventory, error)
// Verify owns temporary raw-config activation and must join/restore before
// returning. It must not touch permanent config or the transaction journal.
verifyAssets func(context.Context, storedCandidate, managedServiceInventory) ([]assetMeasurement, error)
adaptive bool
mediaSamples func(context.Context) ([]mediaResource, error)
verifyGooglevideo func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error)
recheckExperimental func(context.Context, storedCandidate, mediaResource, *managedTransaction) error
recheckGooglevideo func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error
}
// Target-local adapter: reuses bounded Player requests without invoking a CLI.
// The injected command runner receives only fixed script/argv from orchestration.
func managedRecoveryDependencies(target managedDeps, db *sql.DB, scan cacheScan, check cacheCheck, run func(context.Context, string, ...string) error) (managedRecoveryDeps, error) {
d := managedRecoveryDeps{db: db, scan: scan, check: check, run: run}
if _, err := requiredYoutubeAdapter(); err != nil {
return d, err
}
if target.lookup == nil || target.player == nil || db == nil || scan == nil || check == nil || run == nil {
return d, fmt.Errorf("complete target-local recovery adapter dependencies required")
}
if _, err := managedRuntimeGuard(target); err != nil {
return d, err
}
cfg, err := parseHealthConfig(target.lookup)
cfg, err := parseManagedHealthConfig(target.lookup)
if err != nil {
return d, err
}
d.transaction = &managedTransaction{Dir: target.directory, Name: "config"}
request := func(ctx context.Context) ([]byte, error) {
requestVideo := func(ctx context.Context, videoID string) ([]byte, error) {
if _, err := requiredYoutubeAdapter(); err != nil {
return nil, err
}
if err := ctx.Err(); err != nil {
return nil, err
}
@@ -56,14 +73,23 @@ func managedRecoveryDependencies(target managedDeps, db *sql.DB, scan cacheScan,
}
bounded, cancel := context.WithTimeout(ctx, 35*time.Second)
defer cancel()
body, err := requestPlayer(cfg.VideoID, cfg.SOCKSProxy, func(_ context.Context, args []string, out, diagnostics io.Writer) error {
return target.player(bounded, args, out, diagnostics)
body, err := requestPlayer(videoID, cfg.SOCKSProxy, func(_ context.Context, args []string, stdin io.Reader, out, diagnostics io.Writer) error {
return target.player(bounded, args, stdin, out, diagnostics)
})
if bounded.Err() != nil {
return nil, managedRecoveryContextError(ctx, errors.Join(err, bounded.Err()))
}
return body, err
}
request := func(ctx context.Context) ([]byte, error) { return requestVideo(ctx, cfg.VideoID) }
mode, videoIDs, err := managedGooglevideoConfig(target.lookup, cfg.VideoID)
if err != nil {
return d, err
}
d.adaptive = mode == QualificationAdaptive
if d.adaptive {
d.mediaSamples = managedGooglevideoSamples(videoIDs, requestVideo)
}
d.health = func(ctx context.Context) (string, error) {
body, err := request(ctx)
if err != nil {
@@ -95,6 +121,19 @@ func managedRecoveryDependencies(target managedDeps, db *sql.DB, scan cacheScan,
}
return playerHostname(body)
}
d.service, err = managedServiceEnabled(target.lookup)
if err != nil {
return d, err
}
if d.service {
d.frontHealth = d.health
// Construction is side-effect free; experiments may seek representative
// qualification without changing the legacy default exact-host mode.
d.mediaSamples = managedGooglevideoSamples(videoIDs, requestVideo)
d.resources = managedServiceResources(cfg, request, target.player)
d.health = managedServiceHealth(cfg.SOCKSProxy, d.frontHealth, d.resources, runResourceCurl)
d.verifyAssets = unwiredManagedAssetVerifier
}
return d, nil
}
@@ -221,16 +260,62 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
return nil
}
m.Health = func() (string, error) {
status, err := d.health(ctx)
health := d.health
if d.service && d.frontHealth != nil {
health = d.frontHealth
}
// Commit frontend/GGC progress independently; the monitor's overall
// service probe still requires media and both images before READY.
status, err := health(ctx)
if ctx.Err() != nil {
return "", managedRecoveryContextError(ctx, err)
}
return status, err
}
// Nested asset/GGC recovery reads the same transaction before binding its
// apply-specific hooks. Production adapters start with path-only transactions.
d.transaction = &m
if d.service {
_, healthErr := d.health(ctx)
var fatal *managedRecoveryFatal
var failure *managedResourceFailure
j, e := m.load()
if e != nil {
return result, &managedRecoveryFatal{Err: e}
}
if j.Scope == GooglevideoScopeExperimental {
healthErr = managedResourceHealthError(ctx, healthErr)
}
if ctx.Err() != nil || errors.As(healthErr, &fatal) || errors.Is(healthErr, context.Canceled) || errors.Is(healthErr, context.DeadlineExceeded) {
return result, managedRecoveryContextError(ctx, healthErr)
}
// A previously authorized native suffix stays adaptive after promotion,
// even if the operator's legacy environment still says "cold". Stored
// validation permits reconstruction, never fresh proof authorization.
if j.Scope == GooglevideoScopeSuffix {
for _, id := range j.Candidates {
c, err := readStrategyCandidatePolicy(d.db, id, false)
if err != nil {
return result, &managedRecoveryFatal{Err: err}
}
if c.Group == "googlevideo" && c.Mode == QualificationAdaptive && c.Qualified {
d.adaptive = true
}
}
}
if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) && j.Scope != GooglevideoScopeExperimental && !d.adaptive {
return result, repairManagedAssets(ctx, d, failure.Host)
}
}
ids, err := m.ActiveCandidates(d.db)
if err != nil {
return result, &managedRecoveryFatal{Err: err}
}
journal, err := m.load()
if err != nil {
return result, &managedRecoveryFatal{Err: err}
}
experimental := journal.Scope == GooglevideoScopeExperimental
for _, id := range ids {
c, e := readStrategyCandidate(d.db, id)
if e != nil {
@@ -262,6 +347,16 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
if err := ctx.Err(); err != nil {
return err
}
if d.service || experimental {
current, e := m.ActiveCandidates(d.db)
if e != nil {
return &managedRecoveryFatal{Err: e}
}
ids, e = managedFullDesiredSet(d.db, current, ids)
if e != nil {
return &managedRecoveryFatal{Err: e}
}
}
cleanupReload := m.Reload
first := true
m.Reload = func() error {
@@ -301,7 +396,11 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
}
healthy := false
if result.FrontendID != 0 {
status, e := d.health(ctx)
frontHealth := d.health
if d.service && d.frontHealth != nil {
frontHealth = d.frontHealth
}
status, e := frontHealth(ctx)
e = managedRecoveryContextError(ctx, e)
var fatal *managedRecoveryFatal
if errors.As(e, &fatal) || errors.Is(e, context.Canceled) {
@@ -372,11 +471,23 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
if e != nil {
return result, e
}
if len(current) != 1 || current[0] != result.FrontendID {
if experimental {
result.GooglevideoID, result.GGCError = recoverManagedExperimental(ctx, d)
return result, result.GGCError
}
if !d.service && (len(current) != 1 || current[0] != result.FrontendID) {
if e := apply([]int64{result.FrontendID}); e != nil {
return result, e
}
}
if d.adaptive {
result.GooglevideoID, result.GGCError = recoverManagedGooglevideo(ctx, d)
var fatal *managedRecoveryFatal
if errors.As(result.GGCError, &fatal) || ctx.Err() != nil {
return result, result.GGCError
}
return result, nil
}
host, e := d.discover(ctx)
e = managedRecoveryContextError(ctx, e)
var discoveryFatal *managedRecoveryFatal
@@ -578,6 +689,9 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
result, err := recoverManagedTarget(ctx, recovery)
if result.GGCError != nil {
reportGGC(result.GGCError)
if recovery.adaptive && err == nil {
return result.GGCError
}
}
// The supervisor reports ordinary recovery failures and retries only
// after its durable cooldown. Safety failures (rollback/cleanup/drift)
@@ -632,14 +746,6 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
if err := guard(); err != nil {
return &managedRecoveryFatal{Err: err}
}
status, err := recovery.health(ctx)
err = managedRecoveryContextError(ctx, err)
if err != nil {
return err
}
if status != "OK" {
return &managedHealthRejection{Status: status}
}
transaction := *recovery.transaction
transaction.Health = func() (string, error) { return recovery.health(ctx) }
// ActiveCandidates is read-only; never permit a probe to reload config.
@@ -649,6 +755,30 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
return &managedRecoveryFatal{Err: err}
}
// Keep Player API health separate from GGC scope (not playback health).
journal, err := transaction.load()
if err != nil {
return &managedRecoveryFatal{Err: err}
}
status, err := recovery.health(ctx)
if journal.Scope == GooglevideoScopeExperimental && recovery.service && recovery.recheckExperimental != nil {
err = managedResourceHealthError(ctx, err)
} else {
err = managedRecoveryContextError(ctx, err)
}
if err != nil {
return err
}
if status != "OK" {
return &managedHealthRejection{Status: status}
}
if journal.Scope == GooglevideoScopeExperimental {
if !recovery.service || recovery.recheckExperimental == nil {
return errManagedGGCUnresolved
}
// Operational service health is not family qualification.
reportGGC(errManagedExperimentalUnqualified)
return nil
}
// Two bounded Player requests reuse the existing health/discovery adapters.
host, err := recovery.discover(ctx)
err = managedRecoveryContextError(ctx, err)
@@ -667,6 +797,15 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
if err != nil {
return &managedRecoveryFatal{Err: err}
}
if candidate.Group == "googlevideo" && candidate.Mode == QualificationAdaptive && candidate.Qualified {
journal, e := transaction.load()
if e != nil {
return &managedRecoveryFatal{Err: e}
}
if journal.Scope == GooglevideoScopeSuffix {
return nil
}
}
if candidate.Group == "googlevideo" && len(candidate.Endpoints) == 1 && candidate.Endpoints[0] == host {
return nil
}
@@ -0,0 +1,52 @@
package main
import (
"context"
"database/sql"
"errors"
"os"
"testing"
)
func TestManagedRecoveryProductionTransactionHooks(t *testing.T) {
d, front, _ := recoveryFixture(t, false)
if path := os.Getenv("RC_PRIVATE_DB"); path != "" {
db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
if err != nil {
t.Fatal("private snapshot open failed")
}
defer db.Close()
ids, err := qualifiedCandidateIDs(db, "frontend")
if err != nil || len(ids) != 1 {
t.Fatal("private qualification unavailable")
}
if err := d.transaction.Apply(db, ids); err != nil {
t.Fatal("private offline apply failed")
}
d.db, front = db, ids[0]
} else {
front = managedCandidateFixture(t, d.db, "frontend", "youtube.com", "--lua-desync=fake", true)
if err := d.transaction.Apply(d.db, []int64{front}); err != nil {
t.Fatal(err)
}
}
// Production adapters supply only the target path; recovery binds hooks.
d.transaction = &managedTransaction{Dir: d.transaction.Dir, Name: "config"}
d.service, d.adaptive = true, true
d.frontHealth = d.health
samples := 0
d.mediaSamples = func(context.Context) ([]mediaResource, error) {
samples++
return nil, &InsufficientAdaptiveSamples{}
}
d.verifyGooglevideo = func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error) {
t.Fatal("unexpected verification")
return GooglevideoQualificationProof{}, nil
}
result, err := recoverManagedTarget(context.Background(), d)
var failure *InsufficientAdaptiveSamples
if err != nil || result.FrontendID != front || samples != 1 || !errors.As(result.GGCError, &failure) {
// Never expose private DB contents or error causes.
t.Fatalf("frontend_preserved=%t recovery_error_type=%T ggc_error_type=%T", result.FrontendID == front, err, result.GGCError)
}
}
@@ -0,0 +1,157 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"testing"
"time"
)
func TestManagedExpiredAdaptiveJournalRecovery(t *testing.T) {
for _, scenario := range []string{"committed", "prepared", "rollback"} {
t.Run(scenario, func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
r, c, resources, _ := googlevideoRuntimeFixture(t)
proof, err := r.verifyGooglevideo(context.Background(), c, resources)
if err != nil {
t.Fatal(err)
}
tx, err := d.db.Begin()
if err != nil {
t.Fatal(err)
}
id, err := insertStrategyCandidate(tx, c.Group, c.Endpoints, c.Args, nil, c.Mode)
if err == nil {
err = saveAdaptiveGooglevideoEvidence(tx, id, proof)
}
if err != nil {
_ = tx.Rollback()
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
ids := []int64{front, unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true), unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true), id}
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeSuffix); err != nil {
t.Fatal(err)
}
before, err := d.transaction.read("config")
if err != nil {
t.Fatal(err)
}
if scenario == "prepared" {
crash := errors.New("process died after replacement")
d.transaction.fault = func(point string) error {
if point == "replaced" {
return crash
}
return nil
}
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeSuffix); !errors.Is(err, crash) {
t.Fatal(err)
}
d.transaction.fault = nil
}
proof.Attestation.CheckedAt = time.Now().Add(-25 * time.Hour)
raw, err := json.Marshal(proof)
if err != nil {
t.Fatal(err)
}
if _, err := d.db.Exec("UPDATE googlevideo_scope_evidence SET proof=? WHERE candidate_id=?", string(raw), id); err != nil {
t.Fatal(err)
}
if err := d.transaction.Reconcile(d.db); err != nil {
t.Fatal("aged committed/pending refs prevented reconciliation", err)
}
assertActiveCandidates(t, d.transaction, d.db, ids)
stored, err := readStrategyCandidate(d.db, id)
if err != nil || stored.Qualified {
t.Fatal("aged proof authorized fresh qualification", stored, err)
}
if err := validateGooglevideoSuffixEvidence(d.db, stored); err == nil {
t.Fatal("aged proof authorized new suffix")
}
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeSuffix); err == nil {
t.Fatal("aged proof authorized new apply")
}
if err := d.transaction.applyRenderedDB(before, d.db, ids...); err == nil {
t.Fatal("pre-rendered bytes bypassed freshness")
}
d.service, d.adaptive = true, true
d.frontHealth = d.health
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/sample.jpg", "yt3.ggpht.com": "https://yt3.ggpht.com/sample.jpg"}, proxy: r.proxy, runner: adaptiveImageRunner}, nil
}
samples, verifies := 0, 0
d.mediaSamples = func(context.Context) ([]mediaResource, error) {
samples++
return resources, nil
}
d.verifyGooglevideo = func(ctx context.Context, c storedCandidate, resources []mediaResource) (GooglevideoQualificationProof, error) {
verifies++
return r.verifyGooglevideo(ctx, c, resources)
}
d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db)
if scenario == "rollback" {
d.recheckGooglevideo = func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
return errCandidateNonpass
}
}
result, err := recoverManagedTarget(context.Background(), d)
if err != nil || samples != 1 || verifies != 1 {
t.Fatal("aged active state blocked fresh owned verification", result, samples, verifies, err)
}
if scenario == "rollback" {
if result.GGCError == nil {
t.Fatal("unhealthy apply accepted")
}
assertActiveCandidates(t, d.transaction, d.db, ids)
after, err := d.transaction.read("config")
if err != nil || !reflect.DeepEqual(before, after) {
t.Fatal("aged rollback changed committed bytes", err)
}
} else {
if result.GGCError != nil || result.GooglevideoID <= id {
t.Fatal("fresh revalidation did not replace expired proof", result)
}
ids[len(ids)-1] = result.GooglevideoID
assertActiveCandidates(t, d.transaction, d.db, ids)
}
// Expiry alone is safe; malformed/invalid historical proof is not.
activeID := ids[len(ids)-1]
if _, err := d.db.Exec("UPDATE googlevideo_scope_evidence SET proof='{}' WHERE candidate_id=?", activeID); err != nil {
t.Fatal(err)
}
if _, err := d.transaction.ActiveCandidates(d.db); err == nil {
t.Fatal("invalid committed proof accepted")
}
if err := d.transaction.Reconcile(d.db); err == nil {
t.Fatal("invalid committed proof reconciled")
}
})
}
}
func TestManagedServiceDispatchSafetyPrecedesImageRepair(t *testing.T) {
for _, host := range []string{"i.ytimg.com", "yt3.ggpht.com"} {
for _, failure := range []error{&managedRecoveryFatal{Err: errors.New("guard failed")}, context.Canceled, context.DeadlineExceeded} {
t.Run(host+"/"+failure.Error(), func(t *testing.T) {
d, front, events := recoveryFixture(t, true)
d.service = true
mixed := errors.Join(&managedResourceFailure{managedImageProbeKind(host), host, nil}, failure)
d.health = func(context.Context) (string, error) { return "DEGRADED", mixed }
d.resources = func(context.Context) (managedServiceInventory, error) {
t.Fatal("mixed safety failure dispatched image repair")
return managedServiceInventory{}, nil
}
_, err := recoverManagedTarget(context.Background(), d)
if !errors.Is(err, failure) || len(*events) != 0 {
t.Fatal("joined safety failure swallowed/retried", err, *events)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
})
}
}
}
+352
View File
@@ -0,0 +1,352 @@
package main
import (
"context"
"errors"
"io"
"strings"
"testing"
)
func TestManagedServiceOptIn(t *testing.T) {
for _, value := range []string{"", "0", "1", "true", "secret"} {
enabled, err := managedServiceEnabled(func(key string) (string, bool) { return value, key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" && value != "" })
valid := value == "" || value == "0" || value == "1"
if (err == nil) != valid || (valid && enabled != (value == "1")) {
t.Fatalf("value %q: %v %v", value, enabled, err)
}
}
}
func TestManagedServiceMissingResourcesNeverGreen(t *testing.T) {
front := func(context.Context) (string, error) { return "OK", nil }
for _, host := range []string{"i.ytimg.com", "yt3.ggpht.com"} {
i := managedServiceInventory{images: map[string]string{}}
if host == "yt3.ggpht.com" {
i.images["i.ytimg.com"] = "https://i.ytimg.com/test"
}
// The first missing image must fail before any resource process is spawned.
if host == "yt3.ggpht.com" {
continue
}
health := managedServiceHealth("socks5://127.0.0.1:11080", front, func(context.Context) (managedServiceInventory, error) { return i, nil }, func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("unexpected transfer")
return nil
})
status, err := health(context.Background())
var failure *managedResourceFailure
if status != "DEGRADED" || !errors.As(err, &failure) || failure.Host != host {
t.Fatalf("%s %v", status, err)
}
}
if public := (&managedResourceFailure{"https://secret/?token=x", "secret/?token=x", errors.New("private")}).PublicError(); strings.Contains(public, "secret") || strings.Contains(public, "private") {
t.Fatal(public)
}
}
func TestManagedServiceFullDesiredSet(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
thumb := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
newFront := managedCandidateFixture(t, d.db, "frontend", "youtube.com", "--lua-desync=multisplit", true)
ids, err := managedFullDesiredSet(d.db, []int64{front, ggc, thumb, avatar}, []int64{newFront})
if err != nil {
t.Fatal(err)
}
if len(ids) != 4 || ids[0] != ggc || ids[1] != thumb || ids[2] != avatar || ids[3] != newFront {
t.Fatal(ids)
}
}
func TestManagedServiceSavedAssetRepairPreservesFullSet(t *testing.T) {
d, front, events := recoveryFixture(t, true)
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
old := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
saved := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=multisplit", true)
if err := d.transaction.Apply(d.db, []int64{front, ggc, old, avatar}); err != nil {
t.Fatal(err)
}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/fresh"}}, nil
}
joined := false
d.verifyAssets = func(_ context.Context, c storedCandidate, _ managedServiceInventory) ([]assetMeasurement, error) {
if c.ID != saved || len(*events) == 0 || (*events)[len(*events)-1] != "stop" {
t.Fatal("not saved-first under downtime", c.ID, *events)
}
joined = true
return []assetMeasurement{unitImageMeasurement(t, "i.ytimg.com", "tls12", c.Args), unitImageMeasurement(t, "i.ytimg.com", "tls13", c.Args)}, nil
}
d.health = func(context.Context) (string, error) {
if !joined {
t.Fatal("apply before verifier joined")
}
return "OK", nil
}
if err := repairManagedAssets(context.Background(), d, "i.ytimg.com"); err != nil {
t.Fatal(err)
}
ids, err := d.transaction.ActiveCandidates(d.db)
if err != nil || len(ids) != 4 {
t.Fatal(ids, err)
}
for _, required := range []int64{front, ggc, avatar} {
found := false
for _, id := range ids {
found = found || id == required
}
if !found {
t.Fatal("lost sibling", required, ids)
}
}
}
func TestManagedServiceAssetRollbackAndCancellation(t *testing.T) {
for _, cancelJob := range []bool{false, true} {
t.Run(map[bool]string{false: "false-media-rollback", true: "cancel-joins"}[cancelJob], func(t *testing.T) {
d, front, events := recoveryFixture(t, true)
old := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
if err := d.transaction.Apply(d.db, []int64{front, old, avatar}); err != nil {
t.Fatal(err)
}
unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=multisplit", true)
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/fresh"}}, nil
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
joined := false
d.verifyAssets = func(_ context.Context, c storedCandidate, _ managedServiceInventory) ([]assetMeasurement, error) {
joined = true
if cancelJob {
cancel()
return nil, context.Canceled
}
return []assetMeasurement{unitImageMeasurement(t, "i.ytimg.com", "tls12", c.Args), unitImageMeasurement(t, "i.ytimg.com", "tls13", c.Args)}, nil
}
d.health = func(context.Context) (string, error) {
return "DEGRADED", &managedResourceFailure{"media", "fresh.googlevideo.com", nil}
}
err := repairManagedAssets(ctx, d, "i.ytimg.com")
if err == nil || !joined {
t.Fatal("unverified success or unjoined verifier", err)
}
if cancelJob && (!errors.Is(err, context.Canceled) || (*events)[len(*events)-1] != "start") {
t.Fatal("missing bounded restoration", err, *events)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front, old, avatar})
})
}
}
func TestManagedServiceFreshPrivateInventory(t *testing.T) {
cfg := healthConfig{SOCKSProxy: "socks5://127.0.0.1:11080"}
resources := managedServiceResources(cfg, func(context.Context) ([]byte, error) {
return []byte(`{"playabilityStatus":{"status":"OK"},"videoDetails":{"thumbnail":{"thumbnails":[{"url":"https://i.ytimg.com/fresh?private=thumbnail"}]}},"streamingData":{}}`), nil
}, func(_ context.Context, args []string, _ io.Reader, out, _ io.Writer) error {
if args[len(args)-1] != "https://www.youtube.com/" || strings.Contains(strings.Join(args, " "), "private=") {
t.Fatal(args)
}
_, err := io.WriteString(out, `var ytInitialData = {"avatar":{"url":"https://yt3.ggpht.com/fresh?private=avatar"}};`)
return err
})
i, err := resources(context.Background())
if err != nil || len(i.images) != 2 || i.media.url != "" {
t.Fatal("fresh inventory or SABR unknown contract", err)
}
}
func TestManagedServicePartialInventoryChecksAndRepairsAvailableImage(t *testing.T) {
for _, host := range []string{"i.ytimg.com", "yt3.ggpht.com"} {
t.Run(host, func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
thumb := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
if err := d.transaction.Apply(d.db, []int64{front, ggc, thumb, avatar}); err != nil {
t.Fatal(err)
}
r, _, _ := assetRuntimeFixture(t)
d.service = true
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
missingHost := "yt3.ggpht.com"
if host == missingHost {
missingHost = "i.ytimg.com"
}
missing := &managedResourceFailure{managedImageProbeKind(missingHost), missingHost, errors.New("private metadata unavailable")}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{host: "https://" + host + "/image?token=private"}, proxy: r.proxy, runner: r.curl, resourceErr: &managedResourceFailure{"media", "unknown", nil}}, missing
}
repaired, checked := false, false
d.health = managedServiceHealth(r.proxy, d.frontHealth, d.resources, func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
checked = true
if !repaired {
resourceWire("https://"+host+"/image?token=private", "cn", out, diag)
return nil
}
return r.curl(ctx, args, in, out, diag)
})
status, err := d.health(context.Background())
var failure *managedResourceFailure
if status != "DEGRADED" || !checked || !errors.As(err, &failure) || failure.Host != host || failure.ProbeKind != managedImageProbeKind(host) || strings.Contains(err.Error(), "private") {
t.Fatal(status, checked, err)
}
d.verifyAssets = func(ctx context.Context, c storedCandidate, i managedServiceInventory) ([]assetMeasurement, error) {
m, err := r.verify(ctx, c, i)
repaired = err == nil
return m, err
}
if _, err := recoverManagedTarget(context.Background(), d); err != nil || !repaired {
t.Fatal("partial metadata blocked independent asset repair", err)
}
ids, err := d.transaction.ActiveCandidates(d.db)
if err != nil || len(ids) != 4 {
t.Fatal("lost full desired set", ids, err)
}
status, err = d.health(context.Background())
if status != "DEGRADED" || !errors.As(err, &failure) || failure.Host != missingHost {
t.Fatal("missing resources became green", status, err)
}
})
}
}
func TestManagedServiceBootstrapCommitsFrontendBeforeAssetRepair(t *testing.T) {
d, _, _ := recoveryFixture(t, false)
d.service = true
r, _, _ := assetRuntimeFixture(t)
frontChecks := 0
d.frontHealth = func(context.Context) (string, error) {
frontChecks++
if frontChecks == 1 {
return "DEGRADED", nil
}
return "OK", nil
}
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image", "yt3.ggpht.com": "https://yt3.ggpht.com/image"}, proxy: r.proxy, runner: r.curl, resourceErr: &managedResourceFailure{"media", "unknown", nil}}, nil
}
repaired := false
d.health = managedServiceHealth(r.proxy, d.frontHealth, d.resources, func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
if !repaired {
resourceWire("https://i.ytimg.com/image", "cn", out, diag)
return nil
}
return r.curl(ctx, args, in, out, diag)
})
result, err := recoverManagedTarget(context.Background(), d)
if err != nil || result.FrontendID == 0 || result.GooglevideoID == 0 {
t.Fatal("broken thumbnail vetoed frontend commit", result, err)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{result.FrontendID, result.GooglevideoID})
d.verifyAssets = func(ctx context.Context, c storedCandidate, i managedServiceInventory) ([]assetMeasurement, error) {
m, err := r.verify(ctx, c, i)
repaired = err == nil
return m, err
}
if _, err := recoverManagedTarget(context.Background(), d); err != nil || !repaired {
t.Fatal("thumbnail repair blocked by missing media", err)
}
ids, err := d.transaction.ActiveCandidates(d.db)
if err != nil || len(ids) != 3 || ids[0] != result.FrontendID || ids[1] != result.GooglevideoID {
t.Fatal("asset commit lost frontend/GGC", ids, err)
}
status, err := d.health(context.Background())
var failure *managedResourceFailure
if status != "DEGRADED" || !errors.As(err, &failure) || failure.ProbeKind != "media" {
t.Fatal("missing media became service-ready", status, err)
}
}
func TestManagedServiceAssetLocalHealthFailureRollsBack(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
old := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
prior := []int64{front, ggc, old, avatar}
if err := d.transaction.Apply(d.db, prior); err != nil {
t.Fatal(err)
}
before, err := d.transaction.read("config")
if err != nil {
t.Fatal(err)
}
r, _, _ := assetRuntimeFixture(t)
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image", "yt3.ggpht.com": "https://yt3.ggpht.com/image"}, proxy: r.proxy, runner: func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
input, err := io.ReadAll(in)
if err != nil {
return err
}
if strings.Contains(string(input), "i.ytimg.com") {
resourceWire("https://i.ytimg.com/image", "truncated", out, diag)
return nil
}
return r.curl(ctx, args, strings.NewReader(string(input)), out, diag)
}}, nil
}
i, _ := d.resources(context.Background())
if status, err := managedAssetInventoryHealth(context.Background(), i, []string{"yt3.ggpht.com"}); status != "OK" || err != nil {
t.Fatal("old sibling must remain healthy", status, err)
}
d.verifyAssets = r.verify
if err := repairManagedAssets(context.Background(), d, "i.ytimg.com"); !errors.Is(err, errNoQualifiedStrategy) {
t.Fatal(err)
}
assertActiveCandidates(t, d.transaction, d.db, prior)
after, err := d.transaction.read("config")
if err != nil || string(before) != string(after) {
t.Fatal("rejected asset changed prior config", err)
}
}
func TestManagedAssetHealthNonpassPreservesMixedSafetyFailures(t *testing.T) {
badBody := resourceFailure(context.Background(), imageBodyRejection("invalid image body"))
for _, tc := range []struct {
err error
nonpass bool
}{
{badBody, true},
{resourceFailure(context.Background(), errors.New("invalid image body")), false},
{&managedRecoveryFatal{Err: errors.New("invalid image body")}, false},
{errors.Join(badBody, &managedRecoveryFatal{Err: errors.New("guard failed")}), false},
{errors.Join(badBody, context.Canceled), false},
{&managedAssetCleanupFailure{Err: errors.New("cleanup failed")}, false},
} {
if managedAssetHealthNonpass(tc.err) != tc.nonpass {
t.Fatal(tc.err)
}
}
}
func TestManagedAssetRepairMixedFatalNeverRetries(t *testing.T) {
for _, stage := range []string{"verify", "health"} {
t.Run(stage, func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
r, _, _ := assetRuntimeFixture(t)
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/image"}, proxy: r.proxy, runner: r.curl}, nil
}
mixed := errors.Join(&managedRecoveryFatal{Err: errors.New("guard failed")}, errCandidateNonpass, &managedResourceFailure{"thumbnail", "i.ytimg.com", nil})
calls := 0
d.verifyAssets = func(ctx context.Context, c storedCandidate, i managedServiceInventory) ([]assetMeasurement, error) {
calls++
if stage == "verify" {
return nil, mixed
}
return r.verify(ctx, c, i)
}
d.frontHealth = func(context.Context) (string, error) { return "DEGRADED", mixed }
err := repairManagedAssets(context.Background(), d, "i.ytimg.com")
var fatal *managedRecoveryFatal
if !errors.As(err, &fatal) || calls != 1 {
t.Fatal("mixed safety failure retried as rejection", calls, err)
}
assertActiveCandidates(t, d.transaction, d.db, []int64{front})
})
}
}
+145
View File
@@ -0,0 +1,145 @@
package main
import (
"bytes"
"os"
"path/filepath"
"syscall"
"testing"
)
func managedWriterFixture(t *testing.T) (managedDeps, map[string]string) {
t.Helper()
d, env, _ := managedCLIFixture(t)
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
d.openWriteDB = managedDependencies(nil).openWriteDB
return d, env
}
func TestManagedCandidateWriterPreservesReadOnlyHistory(t *testing.T) {
d, _ := managedWriterFixture(t)
path := filepath.Join(d.directory, "candidates.db")
initial, err := openStore(path)
if err != nil {
t.Fatal(err)
}
if err := initial.Close(); err != nil {
t.Fatal(err)
}
db, err := d.openWriteDB(d)
if err != nil {
t.Fatal(err)
}
defer db.Close()
tx, err := db.Begin()
if err != nil {
t.Fatal(err)
}
id, err := insertStrategyCandidate(tx, "googlevideo", []string{"rr1---sn-alpha.googlevideo.com"}, managedGooglevideoArgs, nil)
if err != nil {
_ = tx.Rollback()
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
c, err := readStrategyCandidate(db, id)
if err != nil || c.Qualified {
t.Fatalf("experimental definition must remain unqualified: %+v %v", c, err)
}
t.Setenv("ZAPRET_AUTOTUNE_DB", path)
history, err := managedDependencies(nil).openDB()
if err != nil {
t.Fatal(err)
}
defer history.Close()
if _, err := history.Exec("CREATE TABLE managed_readonly_probe(value INTEGER)"); err == nil {
t.Fatal("ordinary managed history became writable")
}
if _, err := readStrategyCandidate(history, id); err != nil {
t.Fatal(err)
}
}
func TestManagedCandidateWriterRejectsUnsafeStoreBeforeMutation(t *testing.T) {
for _, suffix := range []string{"", "-wal", "-shm", "-journal"} {
for _, kind := range []string{"wrong-path", "symlink", "hardlink", "mode", "owner", "directory"} {
t.Run(suffix+"/"+kind, func(t *testing.T) {
d, env := managedWriterFixture(t)
path := filepath.Join(d.directory, "candidates.db"+suffix)
outside := filepath.Join(t.TempDir(), "unrelated")
data := []byte("operator-owned bytes; never migrate or chmod")
if err := os.WriteFile(outside, data, 0640); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err)
}
switch kind {
case "wrong-path":
env["ZAPRET_AUTOTUNE_DB"] = outside
case "symlink", "hardlink", "directory":
if err := os.Remove(path); err != nil {
t.Fatal(err)
}
var err error
if kind == "symlink" {
err = os.Symlink(outside, path)
} else if kind == "hardlink" {
err = os.Link(outside, path)
} else {
err = os.Mkdir(path, 0700)
}
if err != nil {
t.Fatal(err)
}
case "mode":
if err := os.Chmod(path, 0640); err != nil {
t.Fatal(err)
}
case "owner":
if os.Geteuid() != 0 {
t.Skip("wrong-UID fixture requires root")
}
if err := os.Chown(path, 1, -1); err != nil {
t.Fatal(err)
}
}
before, err := os.Lstat(path)
if err != nil {
t.Fatal(err)
}
outsideBefore, err := os.Stat(outside)
if err != nil {
t.Fatal(err)
}
if db, err := d.openWriteDB(d); err == nil {
_ = db.Close()
t.Fatal("unsafe writer accepted")
}
after, err := os.Lstat(path)
if err != nil || !os.SameFile(before, after) || before.Mode() != after.Mode() || before.Sys().(*syscall.Stat_t).Uid != after.Sys().(*syscall.Stat_t).Uid {
t.Fatal("unsafe file mutated", err)
}
outsideAfter, err := os.Stat(outside)
if err != nil || !os.SameFile(outsideBefore, outsideAfter) || outsideBefore.Mode() != outsideAfter.Mode() {
t.Fatal("unrelated file mutated", err)
}
for _, p := range []string{outside, path} {
if p == path && kind == "directory" {
continue
}
b, err := os.ReadFile(p)
if err != nil || !bytes.Equal(b, data) {
t.Fatal("file contents changed", err)
}
}
if suffix != "" {
if _, err := os.Lstat(filepath.Join(d.directory, "candidates.db")); !os.IsNotExist(err) {
t.Fatal("guard created database before rejecting sidecar", err)
}
}
})
}
}
}
+180 -31
View File
@@ -30,25 +30,35 @@ func (e *managedRollbackError) Error() string {
func (e *managedRollbackError) Unwrap() error { return e.Cause }
type managedJournal struct {
Version int `json:"version"`
Generation uint64 `json:"generation"`
ID string `json:"transaction_id"`
State string `json:"state"`
Hash string `json:"committed_hash"`
Next string `json:"pending_hash,omitempty"`
Candidates []int64 `json:"committed_candidate_ids,omitempty"`
PendingCandidates []int64 `json:"pending_candidate_ids,omitempty"`
Version int `json:"version"`
Generation uint64 `json:"generation"`
ID string `json:"transaction_id"`
State string `json:"state"`
Hash string `json:"committed_hash"`
Next string `json:"pending_hash,omitempty"`
Candidates []int64 `json:"committed_candidate_ids,omitempty"`
PendingCandidates []int64 `json:"pending_candidate_ids,omitempty"`
Scope GooglevideoScope `json:"committed_googlevideo_scope,omitempty"`
PendingScope GooglevideoScope `json:"pending_googlevideo_scope,omitempty"`
Ref *managedConfigRef `json:"committed_ref,omitempty"`
PendingRef *managedConfigRef `json:"pending_ref,omitempty"`
}
// Hooks take no commands or arguments. The target adapter binds Reload
// exclusively to /opt/zapret2/init.d/sysv/zapret2 restart and Health to
// health youtube over its configured loopback SOCKS, returning exactly "OK".
// exclusively to /opt/zapret2/init.d/sysv/zapret2 restart. Health is the
// adapter's owned protocol check, returning exactly "OK" (legacy YouTube or
// every required generic resource/TLS pair); optional rollback health is strict.
// Reload switches target-local firewall and daemons for apply and rollback.
// This engine does not manage SOCKS, health scheduling or DB active state.
type managedTransaction struct {
Dir, Name string
Reload func() error
Health func() (string, error)
Dir, Name string
Reload func() error
Health func() (string, error)
RollbackHealth func() (string, error)
resource *resourceApplySnapshot
// Ordinary candidate-ID CLI calls cannot inherit an operator experiment.
// Evaluated against the effective policy only while holding the journal lock.
forbidExperimentalApply bool
// Tests may simulate process death after durable boundaries. Such errors leave
// the journal intact for Reconcile, deliberately bypassing normal rollback.
fault func(string) error
@@ -181,7 +191,7 @@ func (m *managedTransaction) load() (managedJournal, error) {
if err != nil {
return j, err
}
if err := json.Unmarshal(b, &j); err != nil {
if err := strictPlanJSON(b, &j); err != nil {
return j, err
}
validHash := func(s string) bool {
@@ -189,7 +199,9 @@ func (m *managedTransaction) load() (managedJournal, error) {
return err == nil && len(b) == 32 && strings.ToLower(s) == s
}
validIDs := func(ids []int64) bool {
if len(ids) > 2 {
// Two legacy groups plus at most two disjoint inventory-bound assets.
// Group and scope policy remains in the DB-qualified renderer.
if len(ids) > 4 {
return false
}
seen := map[int64]bool{}
@@ -201,9 +213,12 @@ func (m *managedTransaction) load() (managedJournal, error) {
}
return true
}
if (j.Version != 1 && j.Version != 2) || j.ID == "" || !validHash(j.Hash) || (j.State != "committed" && j.State != "prepared") || (j.State == "prepared" && !validHash(j.Next)) || (j.State == "committed" && (j.Next != "" || len(j.PendingCandidates) != 0)) || !validIDs(j.Candidates) || !validIDs(j.PendingCandidates) || (j.Version == 1 && (len(j.Candidates) != 0 || len(j.PendingCandidates) != 0)) {
if (j.Version != 1 && j.Version != 2 && j.Version != 3 && j.Version != 4) || !j.Scope.valid() || !j.PendingScope.valid() || (j.Version < 3 && (j.Scope != "" || j.PendingScope != "")) || (j.Version == 3 && (j.Scope == "" || (j.State == "prepared" && j.PendingScope == ""))) || j.ID == "" || !validHash(j.Hash) || (j.State != "committed" && j.State != "prepared") || (j.State == "prepared" && !validHash(j.Next)) || (j.State == "committed" && (j.Next != "" || len(j.PendingCandidates) != 0 || j.PendingScope != "" || j.PendingRef != nil)) || !validIDs(j.Candidates) || !validIDs(j.PendingCandidates) || (j.Version == 1 && (len(j.Candidates) != 0 || len(j.PendingCandidates) != 0)) {
return j, fmt.Errorf("invalid managed journal")
}
if j.Version < 4 && (j.Ref != nil || j.PendingRef != nil) || j.Version == 4 && (!validManagedConfigRef(j.Ref, j.Candidates, j.Scope, j.Hash) || j.State == "prepared" && !validManagedConfigRef(j.PendingRef, j.PendingCandidates, j.PendingScope, j.Next)) {
return j, fmt.Errorf("invalid managed journal references")
}
return j, nil
}
@@ -227,18 +242,26 @@ func (m *managedTransaction) Initialize() error {
func (m *managedTransaction) checkpoint(point string) error {
if m.fault != nil {
return m.fault(point)
if err := m.fault(point); err != nil {
return err
}
}
if m.resource != nil {
return m.resource.guard()
}
return nil
}
func (m *managedTransaction) Apply(db *sql.DB, ids []int64) error {
func (m *managedTransaction) Apply(db *sql.DB, ids []int64, scopes ...GooglevideoScope) error {
ids = append([]int64(nil), ids...)
b, err := renderManagedConfig(db, ids)
if err != nil {
return err
if db == nil || len(scopes) > 1 || (len(scopes) == 1 && (!scopes[0].valid() || scopes[0] == "")) {
return fmt.Errorf("candidate database and optional explicit scope required")
}
return m.applyRenderedDB(b, db, ids...)
var scope *GooglevideoScope
if len(scopes) == 1 {
scope = &scopes[0]
}
return m.applyRenderedPolicy(nil, db, scope, ids...)
}
// ActiveCandidates uses the journal as active-state authority. The DB supplies
@@ -255,6 +278,9 @@ func (m *managedTransaction) ActiveCandidates(db *sql.DB) ([]int64, error) {
if j.State != "committed" {
return fmt.Errorf("incomplete transaction: reconcile first")
}
if j.Version == 4 && j.Ref.Kind == "generic" {
return fmt.Errorf("generic deployment has no legacy candidate IDs")
}
b, err := m.read(m.Name)
if err != nil {
return err
@@ -269,7 +295,7 @@ func (m *managedTransaction) ActiveCandidates(db *sql.DB) ([]int64, error) {
if db == nil {
return fmt.Errorf("candidate database required")
}
rendered, err := renderManagedConfig(db, j.Candidates)
rendered, err := renderStoredManagedConfigScope(db, j.Candidates, j.Scope)
if err != nil {
return err
}
@@ -286,6 +312,13 @@ func (m *managedTransaction) ActiveCandidates(db *sql.DB) ([]int64, error) {
func committedManagedJournal(j managedJournal) managedJournal {
j.State, j.Hash, j.Next = "committed", j.Next, ""
j.Candidates, j.PendingCandidates = j.PendingCandidates, nil
if j.Version == 3 {
j.Scope, j.PendingScope = j.PendingScope, ""
}
if j.Version == 4 {
j.Scope, j.PendingScope = j.PendingScope, ""
j.Ref, j.PendingRef = j.PendingRef, nil
}
return j
}
@@ -306,14 +339,18 @@ func (m *managedTransaction) applyRendered(b []byte, ids ...int64) error {
return m.applyRenderedDB(b, nil, ids...)
}
func validateManagedRefs(db *sql.DB, ids []int64, hash string) error {
func validateManagedRefs(db *sql.DB, ids []int64, hash string, scopes ...GooglevideoScope) error {
if len(ids) == 0 {
return nil
}
if db == nil {
return fmt.Errorf("candidate database required")
}
b, err := renderManagedConfig(db, ids)
var scope GooglevideoScope
if len(scopes) != 0 {
scope = scopes[0]
}
b, err := renderStoredManagedConfigScope(db, ids, scope)
if err != nil || managedHash(b) != hash {
return fmt.Errorf("managed candidate state unresolved")
}
@@ -321,6 +358,14 @@ func validateManagedRefs(db *sql.DB, ids []int64, hash string) error {
}
func (m *managedTransaction) applyRenderedDB(b []byte, db *sql.DB, ids ...int64) error {
return m.applyRenderedPolicy(b, db, nil, ids...)
}
func (m *managedTransaction) applyRenderedPolicy(b []byte, db *sql.DB, requested *GooglevideoScope, ids ...int64) error {
return m.applyPolicy(b, db, requested, nil, ids...)
}
func (m *managedTransaction) applyPolicy(b []byte, db *sql.DB, requested *GooglevideoScope, generic *managedConfigRef, ids ...int64) error {
return m.locked(func() error {
j, err := m.load()
if err != nil {
@@ -329,10 +374,74 @@ func (m *managedTransaction) applyRenderedDB(b []byte, db *sql.DB, ids ...int64)
if j.State != "committed" {
return fmt.Errorf("incomplete transaction: reconcile first")
}
if err := validateManagedRefs(db, j.Candidates, j.Hash); err != nil {
if err := m.validateConfigRef(db, j.Ref, j.Candidates, j.Hash, j.Scope); err != nil {
return err
}
if err := validateManagedRefs(db, ids, managedHash(b)); err != nil {
if generic == nil && j.Version == 4 {
return fmt.Errorf("generic journal cannot authorize legacy apply")
}
if generic != nil {
if m.resource == nil || *generic != m.resource.ref || m.resource.guard() != nil || m.resource.validate(generic, managedHash(b)) != nil {
return errResourcePlan
}
if j.Version == 4 && j.Ref.Kind == "generic" {
if reflect.DeepEqual(j.Ref, generic) {
current, e := m.read(m.Name)
if e != nil || managedHash(current) != j.Hash {
return errManagedDrift
}
return nil
}
}
if !(j.Version == 4 && j.Ref.Kind == "generic") && (len(j.Candidates) != 0 || j.Hash != managedHash([]byte(managedBaseline))) {
return fmt.Errorf("generic apply requires disabled baseline")
}
if m.resource.fresh() != nil {
return errResourcePlan
}
}
scope := j.Scope
if scope == "" {
scope = GooglevideoScopeExact
}
if requested != nil {
scope = *requested
}
if generic != nil {
scope = ""
}
if m.forbidExperimentalApply && scope == GooglevideoScopeExperimental {
return fmt.Errorf("experimental CLI apply requires explicit guarded operator path")
}
if requested != nil && scope == GooglevideoScopeExperimental {
for _, id := range j.Candidates {
c, err := readStrategyCandidatePolicy(db, id, false)
if err != nil {
return err
}
if c.Group != "googlevideo" {
found := false
for _, desired := range ids {
found = found || desired == id
}
if !found {
return fmt.Errorf("experimental suffix must preserve current frontend/assets")
}
}
}
}
if b == nil {
b, err = renderManagedConfigScope(db, ids, scope)
if err != nil {
return err
}
} else if len(ids) != 0 {
fresh, err := renderManagedConfigScope(db, ids, scope)
if err != nil || managedHash(fresh) != managedHash(b) {
return fmt.Errorf("fresh qualified candidate state required")
}
}
if err := m.validateConfigRef(db, generic, ids, managedHash(b), scope); err != nil {
return err
}
prior, err := m.read(m.Name)
@@ -374,7 +483,22 @@ func (m *managedTransaction) applyRenderedDB(b []byte, db *sql.DB, ids ...int64)
return err
}
j.Generation++
j.Version = 2
if generic != nil {
if j.Version != 4 {
j.Ref = &managedConfigRef{Kind: "baseline"}
}
j.Version = 4
j.Scope, j.PendingScope = "", ""
j.PendingRef = generic
} else if requested != nil || j.Version == 3 {
j.Version = 3
if j.Scope == "" {
j.Scope = GooglevideoScopeExact
}
j.PendingScope = scope
} else {
j.Version = 2
}
j.ID, j.State, j.Next = hex.EncodeToString(id), "prepared", managedHash(b)
j.PendingCandidates = append([]int64(nil), ids...)
if err := m.journal(j); err != nil {
@@ -403,7 +527,12 @@ func (m *managedTransaction) applyRenderedDB(b []byte, db *sql.DB, ids ...int64)
}
}
if err == nil {
err = m.Reload()
if generic != nil {
err = m.resource.guard()
}
if err == nil {
err = m.Reload()
}
}
if err == nil {
if fault := m.checkpoint("reloaded"); fault != nil {
@@ -424,6 +553,15 @@ func (m *managedTransaction) applyRenderedDB(b []byte, db *sql.DB, ids ...int64)
}
if err == nil {
committed := committedManagedJournal(j)
if generic != nil {
err = m.resource.guard()
}
if err != nil {
if rollback := m.restore(j); rollback != nil {
return &managedRollbackError{Cause: err, Rollback: rollback}
}
return err
}
current, err = m.read(m.Name)
if err == nil && managedHash(current) != j.Next {
return errManagedDrift
@@ -475,6 +613,15 @@ func (m *managedTransaction) restore(j managedJournal) error {
if err := m.Reload(); err != nil {
return err
}
if m.RollbackHealth != nil {
status, err := m.RollbackHealth()
if err != nil {
return err
}
if status != "OK" {
return &managedHealthRejection{Status: status}
}
}
if err := m.owns(j, false); err != nil {
return err
}
@@ -487,6 +634,8 @@ func (m *managedTransaction) restore(j managedJournal) error {
}
j.State, j.Next = "committed", ""
j.PendingCandidates = nil
j.PendingScope = ""
j.PendingRef = nil
return m.journal(j)
}
@@ -502,10 +651,10 @@ func (m *managedTransaction) Reconcile(databases ...*sql.DB) error {
if len(databases) != 0 {
db = databases[0]
}
if err := validateManagedRefs(db, j.Candidates, j.Hash); err != nil {
if err := m.validateConfigRef(db, j.Ref, j.Candidates, j.Hash, j.Scope); err != nil {
return err
}
if err := validateManagedRefs(db, j.PendingCandidates, j.Next); err != nil {
if err := m.validateConfigRef(db, j.PendingRef, j.PendingCandidates, j.Next, j.PendingScope); err != nil {
return err
}
if j.State == "prepared" {
+5 -1
View File
@@ -39,6 +39,7 @@ type managedMonitorDeps struct {
now func() time.Time
wait func(context.Context, time.Duration) error
reportRecoveryError func(error)
reportProbeError func(error)
}
// A startup reconciliation can be consumed once, never during live recovery.
@@ -140,7 +141,7 @@ func runManagedMonitor(ctx context.Context, d managedMonitorDeps) (err error) {
if ctx.Err() != nil {
return ctx.Err()
}
if errors.Is(err, errNoQualifiedStrategy) {
if errors.Is(err, errNoQualifiedStrategy) || errors.Is(err, errGooglevideoInsufficientSamples) {
if err := pauseMonitorRecovery(d.db, d.serviceID, d.now(), managedRecoveryCooldown(err, d.cooldown)); err != nil {
return err
}
@@ -155,6 +156,9 @@ func runManagedMonitor(ctx context.Context, d managedMonitorDeps) (err error) {
return err
}
probeErr := d.probe(ctx)
if probeErr != nil && d.reportProbeError != nil {
d.reportProbeError(probeErr)
}
var fatalProbe *managedRecoveryFatal
if errors.As(probeErr, &fatalProbe) {
return probeErr
+8 -4
View File
@@ -287,6 +287,7 @@ func monitorMigrationSnapshot(t *testing.T, db *sql.DB) string {
func TestMonitorV3MigrationPreservesHistory(t *testing.T) {
db := testStore(t)
saveProposal(t, db, "--lua-desync=fake")
restoreV5CandidateSchema(t, db)
if _, err := db.Exec(`INSERT INTO googlevideo_tune_runs (started_at, finished_at, state, error)
VALUES ('start', 'finish', 'FAILED', 'original failure');
DROP TABLE monitor_state; PRAGMA user_version = 3;`); err != nil {
@@ -299,14 +300,16 @@ func TestMonitorV3MigrationPreservesHistory(t *testing.T) {
}
}
var version, count int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 5 {
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 7 {
t.Fatalf("version=%d err=%v", version, err)
}
if err := db.QueryRow("SELECT count(*) FROM monitor_state").Scan(&count); err != nil || count != 0 {
t.Fatalf("monitor count=%d err=%v", count, err)
}
if after := monitorMigrationSnapshot(t, db); before != after {
t.Fatal("v3 history/candidate data changed")
// The additive v7 candidate column contributes one trailing cold default;
// compare the legacy row projection, not the new field's presence.
if after := strings.ReplaceAll(monitorMigrationSnapshot(t, db), `, "cold"}`, `}`); before != after {
t.Fatalf("v3 history/candidate data changed:\nbefore=%s\nafter=%s", before, after)
}
}
@@ -391,6 +394,7 @@ func TestMonitorCrashReconciliation(t *testing.T) {
func TestMonitorV4MigrationPreservesUnverifiableLease(t *testing.T) {
db := testStore(t)
restoreV5CandidateSchema(t, db)
if _, err := db.Exec("DROP TABLE monitor_state;" + monitorSchema); err != nil {
t.Fatal(err)
}
@@ -408,7 +412,7 @@ func TestMonitorV4MigrationPreservesUnverifiableLease(t *testing.T) {
}
}
var version int
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 5 {
if err := db.QueryRow("PRAGMA user_version").Scan(&version); err != nil || version != 7 {
t.Fatalf("version=%d err=%v", version, err)
}
clear, err := reconcileMonitorRecovery(db, "service", time.Minute, func(int) (int64, error) {
@@ -34,7 +34,7 @@ func (w playerCancelWriter) Write(p []byte) (int, error) {
func canceledPlayerProcess(t *testing.T, cancel context.CancelFunc, extra error, overflow string) playerRunner {
t.Helper()
return func(ctx context.Context, _ []string, out, diagnostics io.Writer) error {
return func(ctx context.Context, _ []string, _ io.Reader, out, diagnostics io.Writer) error {
cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestPlayerCancellationChild$")
cmd.Env = append(os.Environ(), "AUTOTUNE_PLAYER_CANCEL_CHILD=1")
cmd.Stdout = playerCancelWriter{cancel: cancel}
@@ -52,8 +52,8 @@ func TestPlayerRequestCancellationCause(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
runner := canceledPlayerProcess(t, cancel, nil, "")
_, err := requestPlayer("abcdefghijk", "socks5://127.0.0.1:11080", func(_ context.Context, args []string, out, diagnostics io.Writer) error {
return runner(ctx, args, out, diagnostics)
_, err := requestPlayer("abcdefghijk", "socks5://127.0.0.1:11080", func(_ context.Context, args []string, stdin io.Reader, out, diagnostics io.Writer) error {
return runner(ctx, args, stdin, out, diagnostics)
})
var child *exec.ExitError
if !errors.As(err, &child) || !managedRecoveryTimeoutSafe(err) {
@@ -67,12 +67,17 @@ func TestPlayerRequestCancellationCause(t *testing.T) {
func TestPlayerRequestPublicErrorsOpaque(t *testing.T) {
const secret = "https://user:password@example.com/video?signature=private"
cause := errors.New(secret)
runner := func(_ context.Context, _ []string, _, diagnostics io.Writer) error {
runner := func(_ context.Context, args []string, stdin io.Reader, _, diagnostics io.Writer) error {
payload, err := io.ReadAll(stdin)
if err != nil || !bytes.Contains(payload, []byte("abcdefghijk")) || strings.Contains(strings.Join(args, " "), "abcdefghijk") {
t.Fatal("private POST body missing from stdin or exposed in argv")
}
io.WriteString(diagnostics, secret)
return cause
diagnostics.Write(payload)
return errors.Join(cause, errors.New(string(payload)))
}
_, err := requestPlayer("abcdefghijk", "socks5://127.0.0.1:11080", runner)
if !errors.Is(err, cause) || strings.Contains(err.Error(), secret) || managedRecoveryTimeoutSafe(err) {
if !errors.Is(err, cause) || strings.Contains(err.Error(), secret) || strings.Contains(err.Error(), "abcdefghijk") || managedRecoveryTimeoutSafe(err) {
t.Fatal("opaque error lost its unsafe cause or exposed it")
}
t.Setenv("ZAPRET_AUTOTUNE_VIDEO_ID", "abcdefghijk")
@@ -84,7 +89,7 @@ func TestPlayerRequestPublicErrorsOpaque(t *testing.T) {
} else {
code = discoverYoutube([]string{"--video-id", "abcdefghijk", "--proxy", "socks5://127.0.0.1:11080"}, &out, &diagnostics, runner)
}
if code != 1 || out.Len() != 0 || strings.Contains(diagnostics.String(), secret) || !strings.Contains(diagnostics.String(), "player request failed") {
if code != 1 || out.Len() != 0 || strings.Contains(diagnostics.String(), secret) || strings.Contains(diagnostics.String(), "abcdefghijk") || !strings.Contains(diagnostics.String(), "player request failed") {
t.Fatalf("%s: code=%d diagnostics=%q", command, code, diagnostics.String())
}
}
@@ -141,7 +146,7 @@ func TestManagedMonitorSIGTERMChild(t *testing.T) {
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer cancel()
d.player = func(ctx context.Context, _ []string, _, diagnostics io.Writer) error {
d.player = func(ctx context.Context, _ []string, _ io.Reader, _, diagnostics io.Writer) error {
cmd := exec.CommandContext(ctx, os.Args[0], "-test.run=^TestPlayerCancellationChild$")
cmd.Env = append(os.Environ(), "AUTOTUNE_PLAYER_CANCEL_CHILD=1")
cmd.Stdout, cmd.Stderr = os.Stdout, diagnostics
+74
View File
@@ -0,0 +1,74 @@
package main
import (
"bytes"
"context"
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func playerCurlStub(t *testing.T, script string) string {
t.Helper()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "curl"), []byte("#!/bin/sh\n"+script), 0700); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
return dir
}
func TestPlayerPOSTPrivateStdin(t *testing.T) {
dir := playerCurlStub(t, `printf '%s\n' "$@" > "$PLAYER_CAPTURE.args"
cat > "$PLAYER_CAPTURE.body"
printf '%s' "$PLAYER_RESPONSE"
`)
t.Setenv("PLAYER_CAPTURE", filepath.Join(dir, "request"))
t.Setenv("PLAYER_RESPONSE", discoveryFixture)
body, err := requestPlayer("SECRET12345", "socks5://127.0.0.1:11080", runPlayerCurl)
if err != nil || string(body) != discoveryFixture {
t.Fatal("real runner did not preserve the response", err)
}
args, err := os.ReadFile(filepath.Join(dir, "request.args"))
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(args), "SECRET12345") || !strings.Contains(string(args), "--json\n@-\n") {
t.Fatal("POST body must travel only on stdin")
}
payload, err := os.ReadFile(filepath.Join(dir, "request.body"))
const want = `{"context":{"client":{"androidSdkVersion":30,"clientName":"ANDROID","clientVersion":"21.02.35","osName":"Android","osVersion":"11","userAgent":"com.google.youtube/21.02.35(Linux;U;Android11)gzip"}},"videoId":"SECRET12345"}`
if err != nil || string(payload) != want {
t.Fatal("POST bytes changed", err)
}
}
func TestPlayerPOSTInputBound(t *testing.T) {
for _, id := range []string{"", strings.Repeat("S", playerResponseLimit+1), "SECRET\n1234"} {
_, err := requestPlayer(id, "socks5://127.0.0.1:11080", func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("invalid input invoked curl")
return nil
})
if err == nil || strings.Contains(err.Error(), "SECRET") {
t.Fatal("invalid POST input must fail privately before execution")
}
}
}
func TestPlayerCurlStdinCancellation(t *testing.T) {
playerCurlStub(t, "cat > /dev/null\nprintf ready\nexec sleep 60\n")
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
var diagnostics bytes.Buffer
err := runPlayerCurl(ctx, []string{"--json", "@-"}, strings.NewReader(`{"videoId":"SECRET12345"}`), playerCancelWriter{cancel: cancel}, &diagnostics)
if err == nil || !errors.Is(ctx.Err(), context.Canceled) || !managedRecoveryTimeoutSafe(err) {
t.Fatal("real curl runner must join a canceled child", err)
}
if strings.Contains(diagnostics.String(), "SECRET12345") {
t.Fatal("stdin leaked into diagnostics")
}
}
@@ -0,0 +1,257 @@
package main
import (
"errors"
"fmt"
"io"
"syscall"
)
// These enums are closed: no caller-controlled strings enter diagnostics.
type qualificationStage uint8
const (
stageUnknown qualificationStage = iota
stagePreDNSGuard
stagePostDNSGuard
stageDNS
stageTransfer
stageAttestation
stageWorkerStart
stageWorkerQueue
stageWorkerExit
stageNFTDelete
stageStopJoin
stageRestore
stageFinalStoreGuard
stageACKWrite
)
type qualificationReason uint8
const (
reasonUnknown qualificationReason = iota
reasonExecution
reasonOutputBound
reasonMetrics
reasonOrigin
reasonTLSEvidence
reasonFraming
reasonBody
reasonMIME
reasonWorkerIdentity
reasonRules
reasonFiles
reasonDefinition
reasonRuntime
reasonCollector
reasonGuard
reasonWorkerScan
reasonEngine
reasonArgv
reasonParent
reasonStarttime
reasonQueue
reasonVersion
reasonRulesRead
reasonMode
reasonPlanStore
reasonContext
reasonQueuePresentOrUnreadable
reasonNFTRead
reasonNFTPresent
reasonBaseline
reasonPriorCleanupVeto
reasonWorkerScanRead
reasonWorkerScanReadlink
reasonWorkerScanForeign
)
func (s qualificationStage) String() string {
names := [...]string{"unknown", "pre_dns_guard", "post_dns_guard", "dns", "transfer", "attestation", "worker_start", "worker_queue", "worker_exit", "nft_delete", "stop_join", "restore", "final_store_guard", "ack_write"}
if int(s) >= len(names) {
return names[0]
}
return names[s]
}
func (r qualificationReason) String() string {
names := [...]string{"unknown", "execution", "output_bound", "metrics", "origin", "tls_evidence", "framing", "body", "mime", "worker_identity", "rules", "files", "definition", "runtime", "collector", "guard", "worker_scan", "engine", "argv", "parent", "starttime", "queue", "version", "rules_read", "mode", "plan_store", "context", "queue_present_or_unreadable", "nft_read", "nft_present", "baseline", "prior_cleanup_veto", "worker_scan_read", "worker_scan_readlink", "worker_scan_foreign"}
if int(r) >= len(names) {
return names[0]
}
return names[r]
}
type qualificationDiagnostic struct {
Err error
Stage qualificationStage
Reason qualificationReason
Candidate, Resource, Attempt int
TLS uint8
Errno qualificationErrno
ScanPID, OwnedPID int
}
type qualificationErrno uint8
const (
errnoNone qualificationErrno = iota
errnoEACCES
errnoEPERM
errnoESRCH
errnoENOENT
errnoOther
)
func (e qualificationErrno) String() string {
names := [...]string{"none", "eacces", "eperm", "esrch", "enoent", "other"}
if int(e) >= len(names) {
return "other"
}
return names[e]
}
func qualificationScanErrno(err error) qualificationErrno {
for _, item := range []struct {
err error
code qualificationErrno
}{{syscall.EACCES, errnoEACCES}, {syscall.EPERM, errnoEPERM}, {syscall.ESRCH, errnoESRCH}, {syscall.ENOENT, errnoENOENT}} {
if errors.Is(err, item.err) {
return item.code
}
}
if err == nil {
return errnoNone
}
return errnoOther
}
func workerScanReason(reason qualificationReason) bool {
return reason == reasonWorkerScanRead || reason == reasonWorkerScanReadlink || reason == reasonWorkerScanForeign
}
// Normalize classification without retaining private predicate errors.
func qualificationPlanPredicate(err error, fallback qualificationReason) error {
d := &qualificationDiagnostic{Err: errResourcePlan, Reason: qualificationErrorReason(err, fallback)}
copyQualificationScanMetadata(d, err)
return d
}
func copyQualificationScanMetadata(d *qualificationDiagnostic, err error) {
var source *qualificationDiagnostic
if workerScanReason(d.Reason) && errors.As(err, &source) {
d.Errno, d.ScanPID, d.OwnedPID = source.Errno, source.ScanPID, source.OwnedPID
}
}
func (e *qualificationDiagnostic) Error() string { return "qualification operation failed" }
func (e *qualificationDiagnostic) Unwrap() error { return e.Err }
func qualificationOperation(err error, stage qualificationStage) error {
if err == nil {
return nil
}
d := &qualificationDiagnostic{Err: err, Stage: stage, Reason: qualificationErrorReason(err, reasonUnknown)}
copyQualificationScanMetadata(d, err)
return d
}
func qualificationPredicate(err error, reason qualificationReason) error {
d := &qualificationDiagnostic{Err: err, Reason: reason}
copyQualificationScanMetadata(d, err)
return d
}
func qualificationErrorReason(err error, fallback qualificationReason) qualificationReason {
var diagnostic *qualificationDiagnostic
if errors.As(err, &diagnostic) && diagnostic.Reason != reasonUnknown {
return diagnostic.Reason
}
return fallback
}
func attestationMismatch(a, b qualificationAttestation) qualificationReason {
switch {
case a.PID != b.PID || a.StartTime != b.StartTime || a.WorkerArgsHash != b.WorkerArgsHash || a.Queue != b.Queue:
return reasonWorkerIdentity
case a.RulesHash != b.RulesHash:
return reasonRules
case a.LuaHash != b.LuaHash:
return reasonFiles
case a.Definition != b.Definition:
return reasonDefinition
case a.Runtime != b.Runtime:
return reasonRuntime
case a.Collector != b.Collector:
return reasonCollector
default:
return reasonUnknown
}
}
// Traverse only for reporting, never for classification. Bound even pathological
// error trees; output contains enums and ordinals, never Error() or type names.
func writeQualificationDiagnostics(w io.Writer, err error) {
budget := 32
var visit func(error, qualificationDiagnostic, bool)
visit = func(err error, inherited qualificationDiagnostic, cleanup bool) {
if err == nil || budget == 0 {
return
}
budget--
switch e := err.(type) {
case *qualificationDiagnostic:
if e.Candidate != 0 {
inherited.Candidate = e.Candidate
}
if e.Resource != 0 {
inherited.Resource = e.Resource
}
if e.Attempt != 0 {
inherited.Attempt = e.Attempt
}
if e.TLS != 0 {
inherited.TLS = e.TLS
}
if e.Stage != stageUnknown && (e.Stage != stageAttestation || inherited.Stage == stageUnknown) {
inherited.Stage = e.Stage
}
if e.Reason != reasonUnknown {
inherited.Reason = e.Reason
}
if workerScanReason(e.Reason) {
inherited.Errno, inherited.ScanPID, inherited.OwnedPID = e.Errno, e.ScanPID, e.OwnedPID
}
cleanup = cleanup || e.Stage == stageNFTDelete || e.Stage == stageStopJoin || e.Stage == stageRestore
visit(e.Err, inherited, cleanup)
return
case *resourceError:
inherited.Reason = e.reason
case interface{ Unwrap() []error }:
for _, child := range e.Unwrap() {
visit(child, inherited, cleanup)
}
return
}
if e, ok := err.(interface{ Unwrap() error }); ok {
visit(e.Unwrap(), inherited, cleanup)
return
}
role, tls := "primary", "none"
if cleanup {
role = "cleanup"
}
if inherited.TLS == 12 {
tls = "tls12"
} else if inherited.TLS == 13 {
tls = "tls13"
}
fmt.Fprintf(w, "qualification diagnostic: role=%s stage=%s reason=%s candidate=%d resource=%d attempt=%d tls=%s", role, inherited.Stage, inherited.Reason, inherited.Candidate, inherited.Resource, inherited.Attempt, tls)
if workerScanReason(inherited.Reason) {
fmt.Fprintf(w, " errno=%s scan_pid=%d owned_pid=%d", inherited.Errno, inherited.ScanPID, inherited.OwnedPID)
}
fmt.Fprintln(w)
}
visit(err, qualificationDiagnostic{}, false)
}
@@ -0,0 +1,239 @@
package main
import (
"bytes"
"context"
"errors"
"io"
"strings"
"testing"
)
func TestQualificationDiagnosticStagesAndPrivacy(t *testing.T) {
secret := errors.New("PRIVATE https://secret.example/path Authorization IP env")
for _, mode := range []string{"transfer", "next_guard", "post_guard", "attestation", "metrics", "mime", "cleanup"} {
t.Run(mode, func(t *testing.T) {
q, s, registry, _, calls := qualifierFixture(t)
original := q.checks.runner
q.checks.runner = func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
if *calls == 3 && (mode == "transfer" || mode == "cleanup") {
return secret
}
var wire bytes.Buffer
err := original(ctx, args, in, &wire, diag)
if *calls == 4 {
if mode == "metrics" {
wire.WriteString("PRIVATE")
}
if mode == "mime" {
wire = *bytes.NewBufferString(strings.Replace(wire.String(), "text/plain", "text/plain; PRIVATE", 1))
}
}
_, _ = out.Write(wire.Bytes())
return err
}
guards := 0
q.checks.guard = func() error {
guards++
if (mode == "next_guard" && guards == 9) || (mode == "post_guard" && guards == 8) {
return secret
}
return nil
}
attest := q.attest
q.attest = func(ctx context.Context, pid int, args []string, d strategyDefinition) (qualificationAttestation, error) {
a, err := attest(ctx, pid, args, d)
if mode == "attestation" && *calls == 4 {
a.RulesHash = strings.Repeat("b", 64)
}
return a, err
}
if mode == "cleanup" {
q.restore = func(context.Context) error { return secret }
}
_, err := q.complete(context.Background(), s, registry)
if err == nil {
t.Fatal("failure accepted")
}
var diagnostic bytes.Buffer
writeQualificationDiagnostics(&diagnostic, err)
got := diagnostic.String()
if strings.Contains(got, "PRIVATE") || strings.Contains(got, "example") || strings.Contains(got, "SECRET") {
t.Fatal(got)
}
want := map[string]string{"transfer": "stage=transfer reason=execution candidate=1 resource=1 attempt=1 tls=tls13", "cleanup": "stage=transfer reason=execution candidate=1 resource=1 attempt=1 tls=tls13", "next_guard": "stage=pre_dns_guard reason=unknown candidate=1 resource=1 attempt=2 tls=tls13", "post_guard": "stage=post_dns_guard reason=unknown candidate=1 resource=1 attempt=1 tls=tls13", "attestation": "stage=pre_dns_guard reason=rules", "metrics": "stage=transfer reason=metrics", "mime": "stage=transfer reason=mime"}[mode]
if !strings.Contains(got, want) {
t.Fatalf("want %s, got %s", want, got)
}
if mode == "cleanup" {
if !strings.Contains(got, "role=cleanup stage=restore") || qualificationCleanupVerified(err, func(context.Context) error { t.Fatal("cleanup retry"); return nil }) {
t.Fatal(got)
}
}
if mode == "transfer" && !errors.Is(err, secret) {
t.Fatal("lost original error")
}
})
}
}
func TestQualificationDiagnosticsDoNotChangeNegatives(t *testing.T) {
for _, mode := range []string{"status500", "mime_mismatch"} {
t.Run(mode, func(t *testing.T) {
q, s, registry, _, calls := qualifierFixture(t)
if mode == "mime_mismatch" {
source, err := q.source()
if err != nil {
t.Fatal(err)
}
for i := range source.config.Services {
for j := range source.config.Services[i].Resources {
source.config.Services[i].Resources[j].Expect.ContentType = "image/png"
}
}
q.source = func() (resourceMonitorSource, error) { return source, nil }
} else {
original := q.checks.runner
q.checks.runner = func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
var wire bytes.Buffer
err := original(ctx, args, in, &wire, diag)
_, _ = io.WriteString(out, strings.ReplaceAll(strings.ReplaceAll(wire.String(), "200 OK", "500 OK"), `"http_code":200`, `"http_code":500`))
return err
}
}
proof, err := q.complete(context.Background(), s, registry)
if err != nil || *calls != 12 {
t.Fatalf("%d %v", *calls, err)
}
for _, check := range proof.matrix.Runs[0].Checks {
if check.Outcome != "expectation_failed" || check.Attempts != 3 {
t.Fatal(check)
}
}
})
}
}
func TestQualificationDiagnosticVetoAndFinalStages(t *testing.T) {
secret := errors.New("PRIVATE sentinel")
for _, stage := range []qualificationStage{stageFinalStoreGuard, stageACKWrite, stageNFTDelete, stageStopJoin, stageWorkerQueue, stageWorkerExit} {
err := qualificationOperation(secret, stage)
var out bytes.Buffer
writeQualificationDiagnostics(&out, err)
if !strings.Contains(out.String(), "stage="+stage.String()) || strings.Contains(out.String(), "PRIVATE") || !errors.Is(err, secret) {
t.Fatal(out.String())
}
}
cancel := qualificationOperation(context.Canceled, stagePreDNSGuard)
if !managedCancellationOnly(cancel, context.Canceled) {
t.Fatal("pure cancellation changed")
}
for _, veto := range []error{secret, &managedRecoveryFatal{Err: context.Canceled}, &managedAssetCleanupFailure{Err: secret}} {
if managedCancellationOnly(errors.Join(cancel, veto), context.Canceled) {
t.Fatal("mixed veto lost")
}
}
if resourceRejectionOnly(qualificationOperation(&resourcePeerTLSRejection{error: secret}, stageTransfer)) {
t.Fatal("arbitrary rejection unwrapping added")
}
}
func TestQualificationFinalizeDiagnosticsAndACKGate(t *testing.T) {
secret := errors.New("PRIVATE sentinel")
for _, mode := range []string{"cleanup", "owned_cleanup", "restore", "guard", "ack", "success"} {
t.Run(mode, func(t *testing.T) {
var primary error
if mode == "cleanup" {
primary = &qualificationCleanupFailure{Err: secret}
}
if mode == "owned_cleanup" {
primary = errors.Join(context.Canceled, &managedRecoveryFatal{Err: &managedAssetCleanupFailure{Err: secret}})
}
restores, guards, acks := 0, 0, 0
err := qualificationFinalize(primary, func(context.Context) error {
restores++
if mode == "restore" {
return secret
}
return nil
}, func() error {
guards++
if mode == "guard" {
return secret
}
return nil
}, func() error {
acks++
if mode == "ack" {
return secret
}
return nil
})
if mode == "success" {
if err != nil || acks != 1 {
t.Fatal(err, acks)
}
return
}
var out bytes.Buffer
writeQualificationDiagnostics(&out, err)
stage := stageRestore
if mode == "guard" {
stage = stageFinalStoreGuard
}
if mode == "ack" {
stage = stageACKWrite
}
if !strings.Contains(out.String(), "stage="+stage.String()) || strings.Contains(out.String(), "PRIVATE") {
t.Fatal(out.String())
}
if mode != "ack" && acks != 0 {
t.Fatal("ACK passed failed gate")
}
if mode == "cleanup" || mode == "owned_cleanup" {
if restores != 0 || guards != 0 || acks != 0 || !strings.Contains(out.String(), "reason=prior_cleanup_veto") {
t.Fatal("cleanup failure retried or misdiagnosed", out.String())
}
}
})
}
}
func TestQualificationOpaqueAttestationReasonUnderPostDNSGuard(t *testing.T) {
secret := errors.New("PRIVATE attestation path env argv")
q, s, registry, _, transfers := qualifierFixture(t)
attest, calls := q.attest, 0
q.attest = func(ctx context.Context, pid int, args []string, d strategyDefinition) (qualificationAttestation, error) {
calls++
if calls == 3 { // Initial attestation, pre-DNS, then post-DNS.
return qualificationAttestation{}, qualificationPredicate(secret, reasonWorkerScan)
}
return attest(ctx, pid, args, d)
}
proof, err := q.complete(context.Background(), s, registry)
if !errors.Is(err, errResourcePlan) || !errors.Is(err, secret) || proof.seal != "" || *transfers != 0 {
t.Fatal("ownership veto or execution order changed", err, *transfers)
}
var out bytes.Buffer
writeQualificationDiagnostics(&out, err)
if strings.Contains(out.String(), "PRIVATE") || strings.Contains(out.String(), "reason=unknown") ||
strings.Count(out.String(), "stage=post_dns_guard reason=worker_scan candidate=1 resource=1 attempt=1 tls=tls12") != 2 {
t.Fatal(out.String())
}
}
func TestQualificationAttestationErrorRetainsOwnershipVeto(t *testing.T) {
q, s, registry, _, _ := qualifierFixture(t)
attest, calls := q.attest, 0
q.attest = func(ctx context.Context, pid int, args []string, d strategyDefinition) (qualificationAttestation, error) {
calls++
if calls == 2 {
return qualificationAttestation{}, context.Canceled
}
return attest(ctx, pid, args, d)
}
_, err := q.complete(context.Background(), s, registry)
if !errors.Is(err, context.Canceled) || !errors.Is(err, errResourcePlan) || managedCancellationOnly(err, context.Canceled) {
t.Fatal("attestation ownership veto changed", err)
}
}
@@ -0,0 +1,198 @@
package main
import (
"bytes"
"context"
"errors"
"go/ast"
"go/format"
"go/parser"
"go/token"
"reflect"
"strings"
"testing"
)
func TestQualificationRestorePredicatesAndOrder(t *testing.T) {
secret := errors.New("PRIVATE nft process baseline sentinel")
for _, reason := range []qualificationReason{reasonContext, reasonGuard, reasonMode, reasonPlanStore, reasonRuntime, reasonWorkerScan, reasonQueuePresentOrUnreadable, reasonNFTRead, reasonNFTPresent, reasonBaseline, reasonUnknown} {
t.Run(reason.String(), func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
if reason == reasonContext {
cancel()
}
var order []string
q := resourceQualifier{guard: func() error {
order = append(order, "guard")
if reason == reasonGuard {
return secret
}
if reason == reasonMode || reason == reasonPlanStore || reason == reasonRuntime {
return qualificationPredicate(secret, reason)
}
return nil
}}
q.runtime.nft = func(_ context.Context, input string) (string, error) {
order = append(order, "nft")
if input != "list tables\n" {
t.Fatal(input)
}
if reason == reasonNFTRead {
return "", secret
}
if reason == reasonNFTPresent {
return "table inet " + managedAssetTable, nil
}
return "", nil
}
err := q.restoreBaseline(ctx, func() error {
order = append(order, "worker_queue")
if reason == reasonWorkerScan || reason == reasonQueuePresentOrUnreadable {
return qualificationPredicate(secret, reason)
}
return nil
}, func() error {
order = append(order, "baseline")
if reason == reasonBaseline {
return errResourcePlan
}
return nil
})
want := []string{"guard", "worker_queue", "nft", "baseline"}
switch reason {
case reasonContext:
want = nil
case reasonGuard, reasonMode, reasonPlanStore, reasonRuntime:
want = want[:1]
case reasonWorkerScan, reasonQueuePresentOrUnreadable:
want = want[:2]
case reasonNFTRead, reasonNFTPresent:
want = want[:3]
}
if !reflect.DeepEqual(order, want) {
t.Fatal(order, want)
}
if reason == reasonUnknown {
if err != nil {
t.Fatal(err)
}
return
}
if !errors.Is(err, errResourcePlan) || errors.Is(err, secret) {
t.Fatal("original restore classification changed", err)
}
var out bytes.Buffer
writeQualificationDiagnostics(&out, qualificationOperation(err, stageRestore))
if !strings.Contains(out.String(), "role=cleanup stage=restore reason="+reason.String()) || strings.Contains(out.String(), "PRIVATE") {
t.Fatal(out.String())
}
})
}
}
// Check the actual production factory, not just injected diagnostic errors. This
// pins opaque predicate order/strictness without starting workers or changing nft.
func TestQualificationProductionPredicateMap(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "resource_qualify.go", nil, 0)
if err != nil {
t.Fatal(err)
}
text := func(n ast.Node) string {
var b bytes.Buffer
if err := format.Node(&b, fset, n); err != nil {
t.Fatal(err)
}
return b.String()
}
var attest *ast.FuncLit
var restore, guard, noWorker, onlyWorker, scanWorkers string
ast.Inspect(f, func(n ast.Node) bool {
if fn, ok := n.(*ast.FuncDecl); ok {
switch fn.Name.Name {
case "qualificationNoWorker":
noWorker = text(fn.Body)
case "qualificationOnlyWorker":
onlyWorker = text(fn.Body)
case "qualificationScanWorkers":
scanWorkers = text(fn.Body)
}
}
a, ok := n.(*ast.AssignStmt)
if !ok || len(a.Lhs) != 1 || len(a.Rhs) != 1 {
return true
}
if text(a.Lhs[0]) == "q.attest" {
attest, _ = a.Rhs[0].(*ast.FuncLit)
}
if text(a.Lhs[0]) == "q.restore" {
restore = text(a.Rhs[0])
}
if text(a.Lhs[0]) == "q.guard" && guard == "" {
guard = text(a.Rhs[0])
}
return true
})
if attest == nil || !strings.Contains(restore, "q.restoreBaseline(ctx, qualificationNoWorker, func() error { return qualificationBaseline(d.directory) })") {
t.Fatal("production restore wiring changed", restore)
}
for _, check := range []string{
"if e := qualificationMode(config, strategies, d); e != nil {\n\t\treturn qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonMode))",
"if e := s.guard(); e != nil {\n\t\treturn qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonPlanStore))",
"if e := r.guard(); e != nil {\n\t\treturn qualificationPredicate(e, qualificationErrorReason(e, reasonRuntime))",
} {
if !strings.Contains(guard, check) {
t.Fatal("production guard predicate changed", guard)
}
}
if !strings.Contains(noWorker, "qualificationOnlyWorker(0)") || !strings.Contains(noWorker, "qualificationPredicate(e, qualificationErrorReason(e, reasonWorkerScan))") ||
!strings.Contains(noWorker, "managedAssetQueueReady(0)") || !strings.Contains(noWorker, "e != nil || occupied") || !strings.Contains(noWorker, "qualificationPredicate(errResourcePlan, reasonQueuePresentOrUnreadable)") {
t.Fatal("production worker/queue predicate changed", noWorker)
}
if !strings.Contains(onlyWorker, "qualificationScanWorkers(owned, os.ReadDir, os.Readlink)") ||
!strings.Contains(scanWorkers, "if errors.Is(e, os.ErrNotExist) {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn &qualificationDiagnostic{Err: errResourcePlan, Reason: reasonWorkerScanReadlink") {
t.Fatal("unexpected procfs readlink errors must remain fail-closed", onlyWorker, scanWorkers)
}
var got []string
for _, stmt := range attest.Body.List {
branch, ok := stmt.(*ast.IfStmt)
if !ok {
continue
}
predicate := text(branch.Cond)
if branch.Init != nil {
predicate = text(branch.Init) + "; " + predicate
}
ret, ok := branch.Body.List[0].(*ast.ReturnStmt)
if !ok || len(ret.Results) != 2 {
t.Fatal("unexpected attestation branch", text(branch))
}
got = append(got, predicate+" => "+text(ret.Results[1]))
}
want := []string{
"e := q.guard(); e != nil => qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonGuard))",
"e := qualificationOnlyWorker(pid); e != nil => qualificationPlanPredicate(e, reasonWorkerScan)",
`engine == nil || engine.validate() != nil || engine.running(fmt.Sprintf("/proc/%d/exe", pid)) != nil => qualificationPredicate(errResourcePlan, reasonEngine)`,
`e != nil || string(cmd) != strings.Join(append([]string{qualificationEngine}, args...), "\x00")+"\x00" => qualificationPredicate(errResourcePlan, reasonArgv)`,
"e != nil => qualificationPredicate(e, reasonParent)",
"end < 0 => qualificationPredicate(errResourcePlan, reasonParent)",
"len(tail) < 2 || tail[1] != strconv.Itoa(os.Getpid()) => qualificationPredicate(errResourcePlan, reasonParent)",
"e != nil => qualificationPredicate(e, reasonStarttime)",
"e != nil || !ready => qualificationPredicate(errResourcePlan, reasonQueue)",
"e != nil => qualificationPredicate(e, reasonFiles)",
"version.Run() != nil => qualificationPredicate(errResourcePlan, reasonVersion)",
"!valid => qualificationPredicate(errResourcePlan, reasonVersion)",
`e != nil || strings.TrimSpace(rules) == "" => qualificationPredicate(errResourcePlan, reasonRulesRead)`,
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("production attestation predicates:\n%v\nwant:\n%v", got, want)
}
// The factory's strict mode check must precede any store, worker or nft access.
q := newResourceQualifier("", "", managedDeps{lookup: func(string) (string, bool) { return "", false }}, nil)
defer q.close()
_, err = q.attest(context.Background(), 0, nil, strategyDefinition{})
if !errors.Is(err, errResourcePlan) || qualificationErrorReason(err, reasonUnknown) != reasonMode {
t.Fatal(err)
}
}
@@ -0,0 +1,168 @@
package main
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
)
func TestQualificationWorkerScanBranches(t *testing.T) {
secret := errors.New("PRIVATE path argv propertyID sentinel")
dir := t.TempDir()
for _, name := range []string{"0-nonnumeric", "41"} {
if err := os.Mkdir(filepath.Join(dir, name), 0700); err != nil {
t.Fatal(err)
}
}
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
name string
err error
code qualificationErrno
}{
{"eacces", syscall.EACCES, errnoEACCES},
{"eperm", syscall.EPERM, errnoEPERM},
{"esrch", syscall.ESRCH, errnoESRCH},
{"enoent", syscall.ENOENT, errnoENOENT},
{"not_exist", os.ErrNotExist, errnoOther},
{"other", secret, errnoOther},
} {
for _, readDirFailure := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/readdir=%t", tc.name, readDirFailure), func(t *testing.T) {
wrapped := &os.PathError{Op: "PRIVATE", Path: secret.Error(), Err: errors.Join(tc.err, secret)}
calls := 0
err := qualificationScanWorkers(42, func(path string) ([]os.DirEntry, error) {
if path != "/proc" {
t.Fatal(path)
}
if readDirFailure {
return nil, wrapped
}
return entries, nil
}, func(path string) (string, error) {
calls++
if path != "/proc/41/exe" {
t.Fatal("nonnumeric entry scanned", path)
}
return "", wrapped
})
if !readDirFailure && errors.Is(wrapped, os.ErrNotExist) {
if err != nil || calls != 1 {
t.Fatal("wrapped ENOENT must skip", err, calls)
}
return
}
var d *qualificationDiagnostic
if !errors.As(err, &d) || d.Errno != tc.code || d.OwnedPID != 42 {
t.Fatal("missing typed metadata", err)
}
if readDirFailure {
if d.Reason != reasonWorkerScanRead || d.ScanPID != 0 || calls != 0 || !errors.Is(err, wrapped) {
t.Fatal("readdir classification changed", d, calls)
}
} else if d.Reason != reasonWorkerScanReadlink || d.ScanPID != 41 || !errors.Is(err, errResourcePlan) || errors.Is(err, wrapped) {
t.Fatal("readlink classification changed", d)
}
assertWorkerScanOutput(t, err, stageUnknown, d.Reason, tc.code, d.ScanPID, 42)
})
}
}
for _, tc := range []struct {
name, exe string
owned int
deny bool
}{
{"owned", "/PRIVATE/nfqws2", 41, false},
{"foreign", "/PRIVATE/nfqws2", 42, true},
// A foreign --version invocation has the same exe: never inspect argv
// or exempt it from the exclusive-worker predicate.
{"foreign_version", "/PRIVATE/nfqws2", 0, true},
{"unrelated", "/PRIVATE/other", 42, false},
} {
t.Run(tc.name, func(t *testing.T) {
err := qualificationScanWorkers(tc.owned, func(string) ([]os.DirEntry, error) { return entries, nil }, func(string) (string, error) { return tc.exe, nil })
if !tc.deny {
if err != nil {
t.Fatal(err)
}
return
}
if !errors.Is(err, errResourcePlan) {
t.Fatal("foreign worker accepted", err)
}
assertWorkerScanOutput(t, err, stageUnknown, reasonWorkerScanForeign, errnoNone, 41, tc.owned)
})
}
}
func assertWorkerScanOutput(t *testing.T, err error, stage qualificationStage, reason qualificationReason, errno qualificationErrno, scan, owned int) {
t.Helper()
var out bytes.Buffer
writeQualificationDiagnostics(&out, err)
want := fmt.Sprintf(" errno=%s scan_pid=%d owned_pid=%d", errno, scan, owned)
if !strings.Contains(out.String(), "stage="+stage.String()+" reason="+reason.String()) || !strings.Contains(out.String(), want) || strings.Contains(out.String(), "PRIVATE") {
t.Fatal(out.String())
}
}
func TestQualificationWorkerScanNormalization(t *testing.T) {
secret := errors.New("PRIVATE path argv propertyID sentinel")
for _, reason := range []qualificationReason{reasonWorkerScanRead, reasonWorkerScanReadlink, reasonWorkerScanForeign} {
for _, errno := range []qualificationErrno{errnoNone, errnoEACCES, errnoEPERM, errnoESRCH, errnoENOENT, errnoOther} {
scan := &qualificationDiagnostic{Err: secret, Reason: reason, Errno: errno, ScanPID: 41, OwnedPID: 42}
attestation := qualificationPlanPredicate(qualificationOperation(scan, stageAttestation), reasonWorkerScan)
if !errors.Is(attestation, errResourcePlan) || errors.Is(attestation, secret) {
t.Fatal("attestation classification changed", attestation)
}
assertWorkerScanOutput(t, qualificationOperation(attestation, stagePostDNSGuard), stagePostDNSGuard, reason, errno, 41, 42)
q := resourceQualifier{guard: func() error { return nil }}
q.runtime.nft = func(context.Context, string) (string, error) {
t.Fatal("restore passed scan veto")
return "", nil
}
restore := q.restoreBaseline(context.Background(), func() error {
return qualificationPredicate(scan, qualificationErrorReason(scan, reasonWorkerScan))
}, func() error { t.Fatal("unsafe baseline restore"); return nil })
if !errors.Is(restore, errResourcePlan) || errors.Is(restore, secret) {
t.Fatal("restore classification changed", restore)
}
assertWorkerScanOutput(t, qualificationOperation(restore, stageRestore), stageRestore, reason, errno, 41, 42)
}
}
var out bytes.Buffer
writeQualificationDiagnostics(&out, qualificationPredicate(secret, reasonEngine))
if strings.Contains(out.String(), "errno=") || strings.Contains(out.String(), "_pid=") || strings.Contains(out.String(), "PRIVATE") {
t.Fatal("scan metadata on non-scan diagnostic", out.String())
}
}
func TestQualificationWorkerScanPostDNSVeto(t *testing.T) {
q, s, registry, _, transfers := qualifierFixture(t)
attest, calls := q.attest, 0
q.attest = func(ctx context.Context, pid int, args []string, d strategyDefinition) (qualificationAttestation, error) {
calls++
if calls == 3 {
scan := &qualificationDiagnostic{Err: errors.New("PRIVATE path argv propertyID sentinel"), Reason: reasonWorkerScanReadlink, Errno: errnoEPERM, ScanPID: 41, OwnedPID: pid}
return qualificationAttestation{}, qualificationPlanPredicate(scan, reasonWorkerScan)
}
return attest(ctx, pid, args, d)
}
proof, err := q.complete(context.Background(), s, registry)
if !errors.Is(err, errResourcePlan) || proof.seal != "" || *transfers != 0 {
t.Fatal("ownership veto or execution order changed", err, *transfers)
}
var out bytes.Buffer
writeQualificationDiagnostics(&out, err)
if strings.Count(out.String(), "stage=post_dns_guard reason=worker_scan_readlink candidate=1 resource=1 attempt=1 tls=tls12 errno=eperm scan_pid=41 owned_pid=") != 2 || strings.Contains(out.String(), "PRIVATE") {
t.Fatal(out.String())
}
}
+1 -1
View File
@@ -66,7 +66,7 @@ func TestRecoverGooglevideo(t *testing.T) {
var order, checked []string
player := cachePlayer(t, &order)
if mode == "discovery-error" {
player = func(context.Context, []string, io.Writer, io.Writer) error {
player = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
order = append(order, "discover")
return fmt.Errorf("SECRET discovery failure")
}
+8 -1
View File
@@ -79,7 +79,7 @@ func (c *candidate) validate() error {
}
func (c *candidate) validateMeasurements() error {
if c.Group != "frontend" && c.Group != "googlevideo" {
if c.Group != "frontend" && c.Group != "googlevideo" && c.Group != "assets" {
return fmt.Errorf("unknown group")
}
if len(c.Domains) == 0 {
@@ -90,6 +90,9 @@ func (c *candidate) validateMeasurements() error {
return fmt.Errorf("invalid DNS hostname")
}
c.Domains[i] = strings.ToLower(domain)
if c.Group == "assets" && !assetEndpoint(c.Domains[i]) {
return fmt.Errorf("asset endpoint outside service inventory")
}
}
sort.Strings(c.Domains)
for _, metric := range []float64{c.SuccessRate, c.Coverage, c.MedianLatency, c.MedianSpeed} {
@@ -106,6 +109,10 @@ func (c *candidate) validateMeasurements() error {
return nil
}
func assetEndpoint(host string) bool {
return host == "i.ytimg.com" || host == "yt3.ggpht.com"
}
func (c *candidate) validateArgs() error {
// Only shell-inert, unquoted option values can be rendered verbatim.
for _, ch := range c.Args {
+468
View File
@@ -0,0 +1,468 @@
package main
import (
"bytes"
"context"
"database/sql"
"fmt"
"io"
"net"
"os"
"os/signal"
"sort"
"strconv"
"strings"
"syscall"
"time"
)
// v4 tags each side independently; generic references never impersonate SQL IDs.
type managedConfigRef struct {
Kind string `json:"kind"`
Registry string `json:"registry_hash,omitempty"`
Matrix string `json:"matrix_hash,omitempty"`
Source string `json:"source_fingerprint,omitempty"`
Plan string `json:"plan_digest,omitempty"`
Config string `json:"config_hash,omitempty"`
Policy string `json:"strategy_ids,omitempty"`
}
const resourceStrategyIDsEnv = "ZAPRET_AUTOTUNE_RESOURCE_STRATEGY_IDS"
// Policy is a canonical bounded allowlist, never qualification evidence.
func parseResourceStrategyIDs(value string) (string, error) {
if value == "" {
return "", nil
}
if len(value) > 4096 {
return "", errResourcePlan
}
ids := strings.Split(value, ",")
if len(ids) > 32 {
return "", errResourcePlan
}
seen := map[string]bool{}
for _, id := range ids {
if !resourceID.MatchString(id) || seen[id] {
return "", errResourcePlan
}
seen[id] = true
}
sort.Strings(ids)
return strings.Join(ids, ","), nil
}
func validManagedConfigRef(r *managedConfigRef, ids []int64, scope GooglevideoScope, hash string) bool {
if r == nil {
return false
}
switch r.Kind {
case "generic":
policy, e := parseResourceStrategyIDs(r.Policy)
return e == nil && policy == r.Policy && len(ids) == 0 && scope == "" && hashOK(r.Registry) && hashOK(r.Matrix) && hashOK(r.Source) && hashOK(r.Plan) && r.Config == hash
case "baseline", "legacy":
return r.Policy == "" && r.Registry == "" && r.Matrix == "" && r.Source == "" && r.Plan == "" && r.Config == "" && (r.Kind == "legacy" && len(ids) > 0 || r.Kind == "baseline" && len(ids) == 0 && scope == "" && hash == managedHash([]byte(managedBaseline)))
}
return false
}
func (m *managedTransaction) validateConfigRef(db *sql.DB, ref *managedConfigRef, ids []int64, hash string, scope GooglevideoScope) error {
if ref == nil {
return validateManagedRefs(db, ids, hash, scope)
}
if !validManagedConfigRef(ref, ids, scope, hash) {
return errResourcePlan
}
if ref.Kind != "generic" {
return validateManagedRefs(db, ids, hash, scope)
}
if m.resource == nil {
return errResourcePlan
}
return m.resource.validate(ref, hash)
}
type resourceApplySnapshot struct {
store *resourcePlanStore
source resourceMonitorSource
registry strategyRegistry
matrix qualificationMatrix
ref managedConfigRef
rendered []byte
guard func() error
now func() time.Time
close func()
}
// Historical reconstruction uses the proof's original completion horizon. Only
// new apply checks wall-clock freshness; expiry cannot erase committed identity.
func resourceApplyPlan(source resourceMonitorSource, key []byte, registry strategyRegistry, matrix qualificationMatrix, now time.Time, policies ...string) (resourcePlan, []byte, error) {
policy := ""
if len(policies) != 0 {
policy = policies[0]
}
canonical, e := parseResourceStrategyIDs(policy)
if e != nil || canonical != policy {
return resourcePlan{}, nil, errResourcePlan
}
if policy != "" {
for _, id := range strings.Split(policy, ",") {
found := false
for _, d := range registry.Strategies {
found = found || d.ID == id
}
if !found {
return resourcePlan{}, nil, errResourcePlan
}
}
}
p := deriveResourcePlan(source, key, registry, matrix, now, policy)
if p.Evidence != "authenticated" || len(p.Unassigned) != 0 || len(p.Assignments) == 0 {
return p, nil, errResourcePlan
}
for _, s := range source.config.Services {
for _, r := range s.Resources {
if r.TLS != "both" {
return p, nil, errResourcePlan
}
}
}
profiles := []string{}
seen := map[string]bool{}
for _, a := range p.Assignments {
d, e := canonicalStrategy(strategyDefinition{ID: a.Strategy, Args: a.Args, Selectors: a.Selectors})
if e != nil || d.Selectors.TLS != "both" || len(a.Hosts) == 0 {
return p, nil, errResourcePlan
}
hosts := []string{}
for _, host := range a.Hosts {
if !validDomain(host) || host != strings.ToLower(host) || net.ParseIP(host) != nil || seen[host] {
return p, nil, errResourcePlan
}
seen[host] = true
hosts = append(hosts, "^"+host)
}
profiles = append(profiles, "--filter-tcp=443 --filter-l7=tls --hostlist-domains="+strings.Join(hosts, ",")+" "+d.Args)
}
return p, []byte("# Autotuner-owned generic config; runtime acceptance pending.\nNFQWS2_ENABLE=1\n" + managedFirewallConfig + "NFQWS2_OPT='" + strings.Join(profiles, " --new ") + "'\n"), nil
}
func (s *resourceApplySnapshot) fresh() error {
p, b, e := resourceApplyPlan(s.source, s.store.files.key, s.registry, s.matrix, s.now(), s.ref.Policy)
if e != nil || planDigest(p.Assignments) != s.ref.Plan || managedHash(b) != s.ref.Config {
return errResourcePlan
}
return s.guard()
}
// Each journal side reconstructs independently inside the same immutable boundary.
func (s *resourceApplySnapshot) validate(ref *managedConfigRef, hash string) error {
if ref == nil || ref.Kind != "generic" || ref.Registry != s.ref.Registry || ref.Matrix != s.ref.Matrix || ref.Source != s.ref.Source || hash != ref.Config {
return errResourcePlan
}
p, b, e := s.historicalPlan(ref.Policy)
if e != nil || planDigest(p.Assignments) != ref.Plan || managedHash(b) != ref.Config {
return errResourcePlan
}
return s.guard()
}
func (s *resourceApplySnapshot) historicalPlan(policy string) (resourcePlan, []byte, error) {
var horizon int64
for _, run := range s.matrix.Runs {
if run.Completed > horizon {
horizon = run.Completed
}
}
return resourceApplyPlan(s.source, s.store.files.key, s.registry, s.matrix, time.UnixMilli(horizon), policy)
}
func (s *resourceApplySnapshot) selectPolicy(policy string) error {
p, b, e := s.historicalPlan(policy)
if e != nil {
return e
}
s.ref.Policy, s.ref.Plan, s.ref.Config = policy, planDigest(p.Assignments), managedHash(b)
s.rendered = b
return nil
}
// No imported proposal or version probe. Keep private proof/source/native files
// held across staging, health and commit and bind every selected candidate.
func openResourceApplySnapshot(config string, d managedDeps) (*resourceApplySnapshot, error) {
return openResourceApplySnapshotAt(config, d, "/opt/zapret2")
}
func openResourceApplySnapshotAt(config string, d managedDeps, root string) (*resourceApplySnapshot, error) {
store, e := openResourcePlanStore(d.directory, false)
if e != nil {
return nil, errResourcePlan
}
s := &resourceApplySnapshot{store: store, now: time.Now}
var engine *qualificationBinary
var sourceFile *os.File
lua := map[string]*os.File{}
s.close = func() {
for _, f := range lua {
f.Close()
}
if sourceFile != nil {
sourceFile.Close()
}
if engine != nil {
engine.file.Close()
}
store.close()
}
fail := func() (*resourceApplySnapshot, error) { s.close(); return nil, errResourcePlan }
s.registry, e = store.registry()
if e != nil {
return fail()
}
s.matrix, e = store.matrix(s.registry)
if e != nil {
return fail()
}
marker, markerBytes, e := store.files.openEntry(managedMarkerName)
if e != nil {
return fail()
}
store.held[managedMarkerName], store.snapshots[managedMarkerName] = marker, markerBytes
fd, e := syscall.Open(config, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if e != nil {
return fail()
}
sourceFile = os.NewFile(uintptr(fd), "resource source")
if resourcePrivateEntry(sourceFile) != nil {
return fail()
}
sourceBytes, e := io.ReadAll(io.LimitReader(sourceFile, 65537))
if e != nil || len(sourceBytes) > 65536 {
return fail()
}
s.source, e = readResourceMonitorSource(config, d.lookup, store.files.key)
if e != nil {
return fail()
}
engine, e = openQualificationEngine(root)
if e != nil {
return fail()
}
native, e := engine.filesHash()
if e != nil {
return fail()
}
// Hold Lua inodes as well as the qualification hash; equal-byte replacement
// during a transaction is still a trust change.
for _, name := range []string{"lua/zapret-lib.lua", "lua/zapret-antidpi.lua"} {
path := engine.root + "/" + name
f, e := os.Open(path)
if e != nil {
return fail()
}
lua[path] = f
}
p, rendered, e := s.historicalPlan("")
if e != nil {
return fail()
}
for _, run := range s.matrix.Runs {
if run.Attestation.LuaHash != native {
return fail()
}
}
s.rendered = rendered
s.ref = managedConfigRef{Kind: "generic", Registry: managedHash(store.snapshots[planRegistryName]), Matrix: managedHash(store.snapshots[planMatrixName]), Source: s.source.fingerprint, Plan: planDigest(p.Assignments), Config: managedHash(rendered)}
s.guard = func() error {
if store.guard() != nil || managedCheckIdentity(&managedTransaction{Dir: d.directory, Name: "config"}) != nil {
return errResourcePlan
}
current, e := os.Lstat(config)
held, he := sourceFile.Stat()
b, re := io.ReadAll(io.NewSectionReader(sourceFile, 0, 65537))
fresh, se := readResourceMonitorSource(config, d.lookup, store.files.key)
hash, ne := engine.filesHash()
if e != nil || he != nil || re != nil || se != nil || ne != nil || !os.SameFile(current, held) || resourcePrivateEntry(sourceFile) != nil || !bytes.Equal(b, sourceBytes) || fresh.fingerprint != s.source.fingerprint || hash != native {
return errResourcePlan
}
for path, f := range lua {
current, e := os.Lstat(path)
held, he := f.Stat()
if e != nil || he != nil || !os.SameFile(current, held) {
return errResourcePlan
}
}
return nil
}
if s.guard() != nil {
return fail()
}
return s, nil
}
func resourceApplyHealth(ctx context.Context, s *resourceApplySnapshot, checks resourceCheckDeps) (string, error) {
guard := checks.guard
checks.guard = func() error {
if s.guard() != nil {
return errResourcePlan
}
return guard()
}
passed := true
for _, service := range s.source.config.Services {
for _, spec := range service.Resources {
for _, tls := range []string{"tls12", "tls13"} {
result, e := runResourceProtocol(ctx, service.ID, spec, tls, s.source.server, s.source.proxy, checks)
if e != nil || ctx.Err() != nil {
return "DEGRADED", errResourcePlan
}
passed = passed && result.Outcome == "pass"
}
}
}
if s.guard() != nil {
return "BLOCKED", errResourcePlan
}
if !passed {
return "DEGRADED", errResourcePlan
}
return "OK", nil
}
func resourceApplyMode(config string, d managedDeps) error {
if os.Geteuid() != 0 || os.Getppid() != 1 || config == "" {
return errResourcePlan
}
if _, e := managedRuntimeGuard(d); e != nil {
return errResourcePlan
}
cmd, e := os.ReadFile("/proc/1/cmdline")
if e != nil || string(cmd) != "/bin/sh\x00/opt/managed-target-entrypoint.sh\x00" {
return errResourcePlan
}
env, e := os.ReadFile("/proc/1/environ")
if e != nil {
return errResourcePlan
}
values := map[string]string{}
for _, item := range strings.Split(string(env), "\x00") {
k, v, _ := strings.Cut(item, "=")
values[k] = v
}
if values["ZAPRET_AUTOTUNE_RESOURCE_APPLY"] != "1" || values["ZAPRET_AUTOTUNE_RESOURCE_CONFIG"] != config || values["ZAPRET_AUTOTUNE_QUALIFY_CONFIG"] != "" || values["ZAPRET_AUTOTUNE_STRATEGY_CONFIG"] != "" {
return errResourcePlan
}
requested, _ := d.lookup(resourceStrategyIDsEnv)
if _, e := parseResourceStrategyIDs(requested); e != nil || requested != values[resourceStrategyIDsEnv] {
return errResourcePlan
}
for _, ns := range []string{"pid", "net"} {
a, e := os.Readlink("/proc/1/ns/" + ns)
b, f := os.Readlink("/proc/self/ns/" + ns)
if e != nil || f != nil || a != b {
return errResourcePlan
}
}
if !resourceProcessOwned("/proc", os.Getpid(), "/usr/local/bin/zapret-autotune", []string{"/usr/local/bin/zapret-autotune", "managed", "resource-apply", "--resource-config", config}) {
return errResourcePlan
}
dir, e := os.Open("/run/zapret-autotune-ss")
if e != nil {
return errResourcePlan
}
defer dir.Close()
f, e := resourcePrivateFile(dir, "monitor-ready")
if e != nil {
return errResourcePlan
}
defer f.Close()
b, e := io.ReadAll(io.LimitReader(f, 32))
if e != nil || string(b) != strconv.Itoa(os.Getpid())+"\n" {
return errResourcePlan
}
return nil
}
func resourceApplyCLI(config string, stdout, stderr io.Writer, d managedDeps) int {
fmt.Fprintln(stderr, "controller-startup: resource_apply=entry")
blocked := func() int {
fmt.Fprintln(stderr, "BLOCKED: generic apply trust/health incomplete; deployment preserved or prepared for recovery")
return 78
}
if resourceApplyMode(config, d) != nil {
return blocked()
}
lock, e := acquireManagedMonitorSingleton(d)
if e != nil {
return blocked()
}
defer lock.close()
if qualificationNoWorker() != nil {
return blocked()
}
s, e := openResourceApplySnapshot(config, d)
if e != nil {
return blocked()
}
defer s.close()
signals, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
ctx, cancel := context.WithTimeout(signals, 15*time.Minute)
defer cancel()
checks := productionResourceCheckDeps()
m := &managedTransaction{Dir: d.directory, Name: "config", resource: s, Reload: func() error { return d.run(d.script, "restart") }}
m.Health = func() (string, error) { return resourceApplyHealth(ctx, s, checks) }
m.RollbackHealth = func() (string, error) {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer cancel()
return resourceApplyHealth(cleanup, s, checks)
}
if resourceSingletonGuard(lock, d.directory) != nil || m.Reconcile() != nil {
return blocked()
}
requested, _ := d.lookup(resourceStrategyIDsEnv)
policy, e := parseResourceStrategyIDs(requested)
if e != nil || s.selectPolicy(policy) != nil {
return blocked()
}
originalGuard := s.guard
s.guard = func() error {
if resourceApplyMode(config, d) != nil || resourceSingletonGuard(lock, d.directory) != nil {
return errResourcePlan
}
return originalGuard()
}
j, e := m.load()
if e != nil {
return blocked()
}
if (j.Version != 4 || j.Ref.Kind == "baseline") && qualificationNoWorker() != nil {
return blocked()
}
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e != nil {
return blocked()
}
fmt.Fprintln(stdout, "generic resource apply: committed; configured resources only, not production READY")
return 0
}
func resourceDeploymentIdentity(config string, d managedDeps) error {
m := &managedTransaction{Dir: d.directory, Name: "config"}
j, e := m.load()
if e != nil {
return errResourcePlan
}
if j.Version != 4 {
return nil
}
if j.State != "committed" {
return errResourcePlan
}
s, e := openResourceApplySnapshot(config, d)
if e != nil {
return errResourcePlan
}
defer s.close()
m.resource = s
return m.validateConfigRef(nil, j.Ref, j.Candidates, j.Hash, j.Scope)
}
+499
View File
@@ -0,0 +1,499 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
)
func completeApplyFixture(t *testing.T) (resourceMonitorSource, []byte, strategyRegistry, qualificationMatrix, time.Time) {
s, key, r, m, now := planFixture(t)
// e was intentionally unknown in the proposal fixture; this request covers
// only a/b/c/d, including a's cross-service duplicate resource.
s.config.Services = append(s.config.Services[:4:4], s.config.Services[5])
return s, key, r, m, now
}
func TestResourceApplyRender(t *testing.T) {
s, key, r, m, now := completeApplyFixture(t)
p, b, e := resourceApplyPlan(s, key, r, m, now)
if e != nil || !p.ProposalOnly || len(p.Assignments) != 3 {
t.Fatal(p, e)
}
if !strings.Contains(string(b), "--hostlist-domains=^a.example,^c.example") || strings.Count(string(b), " --new ") != 2 {
t.Fatal(string(b))
}
for i := 0; i < 30; i++ {
_, next, e := resourceApplyPlan(s, key, r, m, now)
if e != nil || !bytes.Equal(next, b) {
t.Fatal("nondeterministic render")
}
}
for _, kind := range []string{"negative", "unknown", "incomplete", "stale", "future", "config", "tls", "unsafe"} {
t.Run(kind, func(t *testing.T) {
s, key, r, m, now := completeApplyFixture(t)
switch kind {
case "negative":
m.Runs[0].Checks[len(m.Runs[0].Checks)-1].Outcome = "expectation_failed"
case "unknown":
m.Runs[0].Attestation.Joined = false
case "incomplete":
m.Runs[0].Checks = m.Runs[0].Checks[:len(m.Runs[0].Checks)-1]
case "stale":
now = now.Add(25 * time.Hour)
case "future":
now = now.Add(-time.Hour)
case "config":
s.fingerprint = strings.Repeat("f", 64)
case "tls":
s.config.Services[0].Resources[0].TLS = "tls12"
case "unsafe":
r.Strategies[0].Args = "--lua-desync=execute:cmd=id"
}
if _, _, e := resourceApplyPlan(s, key, r, m, now); e == nil {
t.Fatal("authorized invalid proof")
}
})
}
}
// Uses the production snapshot factory, private HMAC envelopes and shared
// transaction engine; native/HTTP fixtures perform no network operations.
func nativeApplyFixture(t *testing.T) (*managedTransaction, *resourceApplySnapshot, string, string) {
return nativeApplyAlternativesFixture(t, false)
}
func nativeApplyAlternativesFixture(t *testing.T, alternatives bool) (*managedTransaction, *resourceApplySnapshot, string, string) {
return nativeApplyProfilesFixture(t, alternatives, false)
}
func nativeApplyProfilesFixture(t *testing.T, alternatives, sameArgs bool) (*managedTransaction, *resourceApplySnapshot, string, string) {
t.Helper()
root, _, _ := qualificationBinaryFixture(t)
dir := t.TempDir()
if e := os.Chmod(dir, 0700); e != nil {
t.Fatal(e)
}
m := &managedTransaction{Dir: dir, Name: "config", Reload: func() error { return nil }, Health: func() (string, error) { return "OK", nil }}
if e := managedInitialize(m); e != nil {
t.Fatal(e)
}
store, e := openResourcePlanStore(dir, true)
if e != nil {
t.Fatal(e)
}
config := resourceCheckFile(t, strings.Replace(resourceCheckExample, `"tls":"tls12"`, `"tls":"both"`, 1))
d := managedDeps{directory: dir, lookup: resourceCheckLookup}
source, e := readResourceMonitorSource(config, d.lookup, store.files.key)
if e != nil {
t.Fatal(e)
}
_, _, registry, _, _ := planFixture(t)
registry.Strategies = registry.Strategies[:1]
if alternatives {
other := registry.Strategies[0]
other.ID = "zz-alternative"
other.Args = "--lua-desync=split:pos=2"
if sameArgs {
other.Args = registry.Strategies[0].Args
}
registry.Strategies = append(registry.Strategies, other)
}
b, _ := json.Marshal(registry)
if e := store.importDefinitions(b); e != nil {
t.Fatal(e)
}
engine, e := openQualificationEngine(root)
if e != nil {
t.Fatal(e)
}
hash, e := engine.filesHash()
engine.file.Close()
if e != nil {
t.Fatal(e)
}
now := time.Now()
def := registry.Strategies[0]
run := qualificationRun{ID: "run-test", Strategy: def.ID, Definition: planDigest(def), Config: source.fingerprint, Started: now.Add(-time.Minute).UnixMilli(), Completed: now.UnixMilli(), Attestation: qualificationAttestation{Collector: "owned-resource-nfq-v1", Runtime: "nfqws2-1.0.2", Definition: planDigest(def), LuaHash: hash, RulesHash: strings.Repeat("b", 64), WorkerArgsHash: strings.Repeat("c", 64), Queue: managedAssetQueue, PID: 123, StartTime: 456, Restored: true, Joined: true}}
for _, service := range source.config.Services {
for _, spec := range service.Resources {
for _, tls := range []string{"tls12", "tls13"} {
run.Checks = append(run.Checks, qualificationCheck{Service: service.ID, Resource: spec.ID, TLS: tls, Fingerprint: resourceRowFingerprint(store.files.key, source, spec), Outcome: "pass", Attempts: 3})
}
}
}
matrix := qualificationMatrix{Version: 1, Registry: planDigest(registry), Runs: []qualificationRun{run}}
if alternatives {
other := run
other.ID, other.Strategy = "run-alternative", registry.Strategies[1].ID
other.Definition = planDigest(registry.Strategies[1])
other.Attestation.Definition = other.Definition
matrix.Runs = append(matrix.Runs, other)
}
if e := store.saveCompleted(completedQualification{matrix: matrix, seal: planDigest(matrix)}); e != nil {
t.Fatal(e)
}
store.close()
s, e := openResourceApplySnapshotAt(config, d, root)
if e != nil {
t.Fatal(e)
}
t.Cleanup(s.close)
m.resource = s
return m, s, config, root
}
func TestResourceApplyLifecycle(t *testing.T) {
m, s, config, root := nativeApplyFixture(t)
reloads := 0
m.Reload = func() error { reloads++; return nil }
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e != nil {
t.Fatal(e)
}
j, e := m.load()
if e != nil || j.Version != 4 || j.State != "committed" || j.Ref.Kind != "generic" || j.PendingRef != nil || reloads != 1 {
t.Fatal(j, e, reloads)
}
if _, e := m.ActiveCandidates(nil); e == nil {
t.Fatal("generic treated as empty baseline")
}
s.now = func() time.Time { return time.Now().Add(25 * time.Hour) }
if s.fresh() == nil {
t.Fatal("new expired apply authorized")
}
if e := m.Reconcile(); e != nil {
t.Fatal("aged committed proof invalidated", e)
}
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e != nil || reloads != 1 {
t.Fatal("replay changed runtime", e, reloads)
}
before, _ := os.ReadFile(filepath.Join(m.Dir, "config"))
raw, _ := os.ReadFile(config)
if e := os.WriteFile(config, bytes.Replace(raw, []byte("example.com"), []byte("changed.com"), 1), 0600); e != nil {
t.Fatal(e)
}
if e := m.Reconcile(); e == nil {
t.Fatal("source drift reconciled")
}
after, _ := os.ReadFile(filepath.Join(m.Dir, "config"))
if !bytes.Equal(before, after) {
t.Fatal("drift erased deployment")
}
if _, e := openResourceApplySnapshotAt(config, managedDeps{directory: m.Dir, lookup: resourceCheckLookup}, root); e == nil {
t.Fatal("changed source accepted after reopen")
}
}
func TestResourceApplyCrashRecovery(t *testing.T) {
for _, point := range []string{"staged", "backup", "prepared", "replaced", "reloaded", "healthy"} {
t.Run(point, func(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
m.fault = func(p string) error {
if p == point {
return errors.New("process death")
}
return nil
}
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e == nil {
t.Fatal("fault ignored")
}
m.fault = nil
if e := m.Reconcile(); e != nil {
t.Fatal(e)
}
b, _ := m.read("config")
j, _ := m.load()
if string(b) != managedBaseline || j.State != "committed" || j.Ref != nil && j.Ref.Kind != "baseline" {
t.Fatal("pending promoted", j, string(b))
}
})
}
}
func TestResourceApplyRollback(t *testing.T) {
for _, failure := range []string{"reload", "health", "rollback-reload", "rollback-health"} {
t.Run(failure, func(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
calls := 0
m.Reload = func() error {
calls++
if failure == "reload" && calls == 1 || failure == "rollback-reload" && calls == 2 {
return errors.New("reload")
}
return nil
}
m.Health = func() (string, error) { return "DEGRADED", nil }
m.RollbackHealth = func() (string, error) {
if failure == "rollback-health" {
return "DEGRADED", nil
}
return "OK", nil
}
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e == nil {
t.Fatal("failure committed")
}
j, _ := m.load()
b, _ := m.read("config")
want := "committed"
if strings.HasPrefix(failure, "rollback-") {
want = "prepared"
}
if j.State != want || string(b) != managedBaseline {
t.Fatal(j, string(b))
}
})
}
}
func TestResourceApplyHeldMutation(t *testing.T) {
for _, kind := range []string{"matrix", "source", "lua", "directory"} {
t.Run(kind, func(t *testing.T) {
m, s, config, root := nativeApplyFixture(t)
m.fault = func(point string) error {
if point != "staged" {
return nil
}
path := config
switch kind {
case "matrix":
path = filepath.Join(m.Dir, planMatrixName)
case "lua":
path = filepath.Join(root, "lua/zapret-lib.lua")
case "directory":
return os.Rename(m.Dir, m.Dir+"-moved")
}
b, e := os.ReadFile(path)
if e != nil {
return e
}
if e := os.Remove(path); e != nil {
return e
}
return os.WriteFile(path, b, 0600)
}
reloads := 0
m.Reload = func() error { reloads++; return nil }
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e == nil || reloads != 0 {
t.Fatal("mutation authorized", e, reloads)
}
})
}
}
func TestResourceApplyJournalTags(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
for _, kind := range []string{"unknown-version", "mixed", "missing", "unknown-tag", "old-tag"} {
t.Run(kind, func(t *testing.T) {
j := managedJournal{Version: 4, ID: "fixture", State: "committed", Hash: s.ref.Config, Ref: &s.ref}
switch kind {
case "unknown-version":
j.Version = 5
case "mixed":
j.Candidates = []int64{1}
case "missing":
j.Ref = nil
case "unknown-tag":
r := s.ref
r.Kind = "other"
j.Ref = &r
case "old-tag":
j.Version = 3
j.Scope = GooglevideoScopeExact
}
if e := m.journal(j); e != nil {
t.Fatal(e)
}
if _, e := m.load(); e == nil {
t.Fatal("contradictory journal accepted")
}
})
}
}
func TestResourceApplyHealthCollector(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
_ = m
for _, mode := range []string{"", "verify", "status500"} {
calls := 0
checks := resourceCheckDeps{guard: func() error { return nil }, resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(_ context.Context, args []string, stdin io.Reader, out, diag io.Writer) error {
calls++
config, _ := io.ReadAll(stdin)
if !bytes.Contains(config, []byte("connect-to")) {
t.Fatal("missing pin")
}
raw, err := strconv.Unquote(strings.TrimPrefix(strings.Split(string(config), "\n")[0], "url = "))
if err != nil {
t.Fatal(err)
}
method, tls := "GET", "tls12"
for _, arg := range args {
if arg == "--head" {
method = "HEAD"
}
if arg == "--tlsv1.3" {
tls = "tls13"
}
if arg == "--insecure" {
t.Fatal("TLS bypass")
}
}
configuredWire(raw, method, tls, mode, out, diag)
return nil
}}
status, e := resourceApplyHealth(context.Background(), s, checks)
if calls != 4 || (mode == "" && (e != nil || status != "OK")) || (mode != "" && e == nil) {
t.Fatal(mode, calls, status, e)
}
}
}
func TestResourceApplyUnauthorizedCLI(t *testing.T) {
var out bytes.Buffer
d := managedDeps{}
if code := resourceApplyCLI("/private/SECRET_SENTINEL", &out, &out, d); code != 78 {
t.Fatal(code)
}
if got, want := out.String(), "controller-startup: resource_apply=entry\nBLOCKED: generic apply trust/health incomplete; deployment preserved or prepared for recovery\n"; got != want {
t.Fatalf("unexpected diagnostic: %q", got)
}
}
func TestResourceApplyFreshBeforeStaging(t *testing.T) {
for _, kind := range []string{"expired", "forged", "native", "proof", "unsupported"} {
t.Run(kind, func(t *testing.T) {
m, s, _, root := nativeApplyFixture(t)
ref := s.ref
switch kind {
case "expired":
s.now = func() time.Time { return time.Now().Add(25 * time.Hour) }
case "forged":
ref.Plan = strings.Repeat("a", 64)
case "native":
if e := os.WriteFile(filepath.Join(root, "lua/zapret-antidpi.lua"), []byte("drift"), 0755); e != nil {
t.Fatal(e)
}
case "proof":
if e := os.WriteFile(filepath.Join(m.Dir, planMatrixName), []byte(`{"payload":{},"mac":"forged"}`), 0600); e != nil {
t.Fatal(e)
}
case "unsupported":
s.source.config.Services[0].Resources[0].TLS = "tls13"
}
reloads, stages := 0, 0
m.Reload = func() error { reloads++; return nil }
m.fault = func(point string) error {
if point == "staged" {
stages++
}
return nil
}
if e := m.applyPolicy(s.rendered, nil, nil, &ref); e == nil || stages != 0 || reloads != 0 {
t.Fatal("unqualified staging", e, stages, reloads)
}
b, _ := m.read("config")
j, _ := m.load()
if string(b) != managedBaseline || j.Version != 2 || j.Generation != 0 {
t.Fatal("baseline changed", j)
}
})
}
}
func TestResourceApplyPreparedExpiredRecoveryHealth(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
m.fault = func(point string) error {
if point == "healthy" {
return errors.New("crash")
}
return nil
}
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e == nil {
t.Fatal("crash missed")
}
m.fault = nil
s.now = func() time.Time { return time.Now().Add(25 * time.Hour) }
m.RollbackHealth = func() (string, error) { return "DEGRADED", nil }
if e := m.Reconcile(); e == nil {
t.Fatal("unhealthy rollback finalized")
}
j, _ := m.load()
b, _ := m.read("config")
if j.State != "prepared" || string(b) != managedBaseline {
t.Fatal("baseline was not physically restored", j)
}
m.RollbackHealth = func() (string, error) { return "OK", nil }
if e := m.Reconcile(); e != nil {
t.Fatal("expiry blocked recovery", e)
}
}
func TestResourceApplyCancellationJoinsBeforeRollback(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
joined := false
checks := resourceCheckDeps{guard: func() error { return nil }, resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(ctx context.Context, _ []string, _ io.Reader, _, _ io.Writer) error {
cancel()
<-ctx.Done()
joined = true
return ctx.Err()
}}
m.Health = func() (string, error) { return resourceApplyHealth(ctx, s, checks) }
reloads := 0
m.Reload = func() error {
reloads++
if reloads == 2 && !joined {
t.Fatal("rollback raced health child")
}
return nil
}
m.RollbackHealth = func() (string, error) {
if !joined {
t.Fatal("rollback health raced child")
}
return "OK", nil
}
if e := m.applyPolicy(s.rendered, nil, nil, &s.ref); e == nil || reloads != 2 || !joined {
t.Fatal(e, reloads, joined)
}
j, _ := m.load()
if j.State != "committed" || j.Ref.Kind != "baseline" {
t.Fatal(j)
}
}
func TestResourceApplyRefusesActiveLegacy(t *testing.T) {
m, s, _, _ := nativeApplyFixture(t)
db := testStore(t)
id := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=multisplit", true)
m.resource = nil
if e := m.Apply(db, []int64{id}, GooglevideoScopeExact); e != nil {
t.Fatal(e)
}
before, _ := m.read("config")
journal, _ := m.read(managedJournalName)
m.resource = s
reloads := 0
m.Reload = func() error { reloads++; return nil }
if e := m.applyPolicy(s.rendered, db, nil, &s.ref); e == nil || reloads != 0 {
t.Fatal("active legacy maintenance authorized", e)
}
after, _ := m.read("config")
nextJournal, _ := m.read(managedJournalName)
if !bytes.Equal(before, after) || !bytes.Equal(journal, nextJournal) {
t.Fatal("legacy state changed")
}
}
+534
View File
@@ -0,0 +1,534 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"mime"
"net"
"net/netip"
"net/url"
"os"
"os/signal"
"regexp"
"sort"
"strings"
"syscall"
"time"
)
var errResourceConfig = errors.New("invalid resource configuration")
var resourceID = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,47}$`)
var resourceEnv = regexp.MustCompile(`^[A-Z_][A-Z0-9_]{0,127}$`)
var resourceHeader = regexp.MustCompile("^[!#$%&'*+.^_`|~0-9A-Za-z-]{1,64}$")
var resourceNumericHost = regexp.MustCompile(`(?i)^(?:[0-9]+|0x[0-9a-f]+)(?:\.(?:[0-9]+|0x[0-9a-f]+))*$`)
type resourceCheckConfig struct {
Version int `json:"version"`
Services []resourceCheckService `json:"services"`
}
type resourceCheckService struct {
ID string `json:"id"`
Resources []resourceCheckSpec `json:"resources"`
}
type resourceCheckSpec struct {
ID string `json:"id"`
Group string `json:"group"`
Kind string `json:"kind"`
URL string `json:"url,omitempty"`
URLEnv string `json:"url_env,omitempty"`
Method string `json:"method"`
TLS string `json:"tls"`
TimeoutMS int `json:"timeout_ms"`
IntervalMS *int `json:"interval_ms,omitempty"`
CooldownMS *int `json:"cooldown_ms,omitempty"`
Expect resourceCheckExpect `json:"expect"`
HeadersEnv map[string]string `json:"headers_env,omitempty"`
AllowPrivate bool `json:"allow_private,omitempty"`
AllowedNetworks []string `json:"allowed_networks,omitempty"`
host, privateConfig string
networks []netip.Prefix
}
type resourceCheckExpect struct {
Statuses []int `json:"statuses"`
ContentType string `json:"content_type,omitempty"`
MinBytes int `json:"min_bytes"`
MaxBytes int `json:"max_bytes"`
}
// encoding/json otherwise silently accepts duplicate keys and null scalars.
func uniqueResourceJSON(d *json.Decoder, depth int) error {
if depth > 16 {
return errResourceConfig
}
t, err := d.Token()
if err != nil || t == nil {
return errResourceConfig
}
if delim, ok := t.(json.Delim); ok {
switch delim {
case '{':
seen := map[string]bool{}
for d.More() {
key, err := d.Token()
if err != nil {
return errResourceConfig
}
s, ok := key.(string)
if !ok || seen[s] {
return errResourceConfig
}
seen[s] = true
if err := uniqueResourceJSON(d, depth+1); err != nil {
return err
}
}
case '[':
for d.More() {
if err := uniqueResourceJSON(d, depth+1); err != nil {
return err
}
}
default:
return errResourceConfig
}
if _, err := d.Token(); err != nil {
return errResourceConfig
}
}
return nil
}
func privateCurlValue(v string) string {
return "\"" + strings.NewReplacer("\\", "\\\\", "\"", "\\\"").Replace(v) + "\""
}
func safeResourceValue(v string) bool {
return len(v) <= 16<<10 && strings.IndexFunc(v, func(c rune) bool { return c < 32 || c == 127 }) < 0
}
func parseResourceChecks(data []byte, lookup func(string) (string, bool)) (resourceCheckConfig, error) {
var cfg resourceCheckConfig
fail := func() (resourceCheckConfig, error) { return resourceCheckConfig{}, errResourceConfig }
if len(data) > 1<<20 || lookup == nil {
return fail()
}
d := json.NewDecoder(bytes.NewReader(data))
if uniqueResourceJSON(d, 0) != nil {
return fail()
}
if _, err := d.Token(); err != io.EOF {
return fail()
}
d = json.NewDecoder(bytes.NewReader(data))
d.DisallowUnknownFields()
if d.Decode(&cfg) != nil || (cfg.Version != 1 && cfg.Version != 2) || len(cfg.Services) == 0 || len(cfg.Services) > 16 {
return fail()
}
if cfg.Version == 2 {
// Presence matters: an omitted minimum must not silently become zero.
var fields struct {
Services []struct {
Resources []struct {
Kind *string `json:"kind"`
Expect struct {
Statuses *[]int `json:"statuses"`
ContentType *string `json:"content_type"`
MinBytes *int `json:"min_bytes"`
MaxBytes *int `json:"max_bytes"`
} `json:"expect"`
} `json:"resources"`
} `json:"services"`
}
if json.Unmarshal(data, &fields) != nil {
return fail()
}
for _, s := range fields.Services {
for _, r := range s.Resources {
if r.Kind != nil && *r.Kind == "" || r.Expect.Statuses == nil || r.Expect.ContentType == nil || *r.Expect.ContentType == "" || r.Expect.MinBytes == nil || r.Expect.MaxBytes == nil {
return fail()
}
}
}
}
services, total := map[string]bool{}, 0
for si := range cfg.Services {
s := &cfg.Services[si]
if !resourceID.MatchString(s.ID) || services[s.ID] || len(s.Resources) == 0 {
return fail()
}
services[s.ID] = true
ids := map[string]bool{}
for ri := range s.Resources {
r := &s.Resources[ri]
if cfg.Version == 2 && r.Kind == "" {
r.Kind = "http"
}
total++
if total > 64 || !resourceID.MatchString(r.ID) || ids[r.ID] || !resourceID.MatchString(r.Group) || (r.Kind != "http" && (cfg.Version != 2 || r.Kind != "image" && r.Kind != "mp4_range")) {
return fail()
}
ids[r.ID] = true
for _, scheduling := range []struct {
value **int
fallback int
}{{&r.IntervalMS, 300000}, {&r.CooldownMS, 30000}} {
if *scheduling.value == nil {
v := scheduling.fallback
*scheduling.value = &v
}
if **scheduling.value < 100 || **scheduling.value > 86400000 {
return fail()
}
}
if (r.URL == "") == (r.URLEnv == "") {
return fail()
}
if r.URLEnv != "" {
if !resourceEnv.MatchString(r.URLEnv) {
return fail()
}
var ok bool
r.URL, ok = lookup(r.URLEnv)
if !ok || r.URL == "" {
return fail()
}
}
u, err := url.Parse(r.URL)
if err != nil {
return fail()
}
r.host = u.Hostname()
// curl normalizes legacy IPv4 spellings before matching --connect-to.
// Only canonical literals may bypass DNS without defeating that match.
if resourceNumericHost.MatchString(r.host) {
ip := net.ParseIP(r.host)
if ip == nil || ip.To4() == nil || ip.String() != r.host {
return fail()
}
}
if _, err := privateResourceURL(r.URL, r.host); err != nil {
return fail()
}
// Reject encoded controls too, without ever displaying the decoded path.
decoded, err := url.PathUnescape(r.URL)
if err != nil || !safeResourceValue(decoded) {
return fail()
}
if r.Method != "GET" && r.Method != "HEAD" {
return fail()
}
if r.TLS != "tls12" && r.TLS != "tls13" && r.TLS != "both" {
return fail()
}
if r.TimeoutMS < 100 || r.TimeoutMS > 30000 || r.Expect.MaxBytes < 1 || r.Expect.MaxBytes > 5<<20 || r.Expect.MinBytes < 0 || r.Expect.MinBytes > r.Expect.MaxBytes || (r.Method == "HEAD" && r.Expect.MinBytes != 0) {
return fail()
}
if len(r.Expect.Statuses) == 0 || len(r.Expect.Statuses) > 16 {
return fail()
}
statuses := map[int]bool{}
for _, status := range r.Expect.Statuses {
if status < 200 || status > 599 || status >= 300 && status < 400 || statuses[status] {
return fail()
}
statuses[status] = true
}
if r.Expect.ContentType != "" {
mt, params, err := mime.ParseMediaType(r.Expect.ContentType)
if err != nil || len(params) != 0 || mt != r.Expect.ContentType {
return fail()
}
}
if r.Kind != "http" {
status := 200
if r.Kind == "mp4_range" {
status = 206
}
if r.Method != "GET" || len(r.Expect.Statuses) != 1 || r.Expect.Statuses[0] != status {
return fail()
}
if r.Kind == "image" {
// ponytail: public GIF awaits an aggregate frame/pixel-bounded decoder;
// legacy image adapters retain their existing GIF contract.
if r.Expect.ContentType != "image/png" && r.Expect.ContentType != "image/jpeg" {
return fail()
}
} else if (r.Expect.ContentType != "video/mp4" && r.Expect.ContentType != "audio/mp4") || r.Expect.MinBytes < 16 || r.Expect.MaxBytes > 65536 {
return fail()
}
}
if len(r.HeadersEnv) > 16 {
return fail()
}
headers := map[string]bool{}
names := make([]string, 0, len(r.HeadersEnv))
for name := range r.HeadersEnv {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
env := r.HeadersEnv[name]
lower := strings.ToLower(name)
if !resourceHeader.MatchString(name) || headers[lower] || !resourceEnv.MatchString(env) {
return fail()
}
headers[lower] = true
switch lower {
case "host", "connection", "transfer-encoding", "content-length", "expect", "upgrade", "proxy-authorization", "proxy-connection", "te", "trailer", "range", "accept-encoding":
return fail()
}
value, ok := lookup(env)
if !ok || value == "" || !safeResourceValue(value) {
return fail()
}
r.privateConfig += "header = " + privateCurlValue(name+": "+value) + "\n"
}
if len(r.AllowedNetworks) > 16 || r.AllowPrivate && len(r.AllowedNetworks) == 0 {
return fail()
}
for _, raw := range r.AllowedNetworks {
prefix, err := netip.ParsePrefix(raw)
if err != nil || prefix != prefix.Masked() || prefix.Addr().Is4In6() {
return fail()
}
r.networks = append(r.networks, prefix)
}
}
}
return cfg, nil
}
func readResourceChecks(path string, lookup func(string) (string, bool)) (resourceCheckConfig, error) {
return readResourceChecksMode(path, lookup, false)
}
func readResourceChecksMode(path string, lookup func(string) (string, bool), owned bool) (resourceCheckConfig, error) {
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if err != nil {
return resourceCheckConfig{}, errResourceConfig
}
f := os.NewFile(uintptr(fd), "resource config")
defer f.Close()
info, err := f.Stat()
if err != nil || !info.Mode().IsRegular() || info.Size() > 1<<20 {
return resourceCheckConfig{}, errResourceConfig
}
if owned {
st, ok := info.Sys().(*syscall.Stat_t)
if !ok || (st.Uid != 0 && st.Uid != uint32(os.Geteuid())) || st.Nlink != 1 || info.Mode().Perm()&0022 != 0 {
return resourceCheckConfig{}, errResourceConfig
}
}
data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
if err != nil {
return resourceCheckConfig{}, errResourceConfig
}
if owned {
current, err := os.Lstat(path)
if err != nil || !os.SameFile(info, current) {
return resourceCheckConfig{}, errResourceConfig
}
}
return parseResourceChecks(data, lookup)
}
var resourceDeniedNetworks = []string{"0.0.0.0/8", "100.64.0.0/10", "192.0.0.0/24", "192.0.2.0/24", "198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "240.0.0.0/4", "2001:db8::/32", "2001::/23", "2002::/16", "64:ff9b::/96", "64:ff9b:1::/48", "100::/64"}
func approvedResourceIP(r resourceCheckSpec, ip net.IP) bool {
a, ok := netip.AddrFromSlice(ip)
if !ok {
return false
}
a = a.Unmap()
// Link-local and metadata are never enabled by allow_private.
if !a.IsGlobalUnicast() || a.IsLoopback() || a.IsLinkLocalUnicast() || a.IsLinkLocalMulticast() || a.String() == "100.100.100.200" || a.String() == "fd00:ec2::254" || a.String() == "168.63.129.16" {
return false
}
for _, raw := range resourceDeniedNetworks {
p := netip.MustParsePrefix(raw)
if p.Contains(a) {
return false
}
}
if a.Is6() && !a.IsPrivate() && !netip.MustParsePrefix("2000::/3").Contains(a) {
return false
}
if a.IsPrivate() || a.IsLoopback() {
if !r.AllowPrivate {
return false
}
}
if len(r.networks) != 0 {
for _, p := range r.networks {
if p.Contains(a) {
return true
}
}
return false
}
return true
}
type resourceCheckDeps struct {
// Qualification must distinguish infrastructure from verified peer negatives.
// Monitoring retains its existing coarse outcome behavior.
qualification bool
lookup func(string) (string, bool)
guard func() error
resolve func(context.Context, string, string) ([]net.IP, error)
runner subprocessRunner
}
func resolveResourceIPs(ctx context.Context, server, host string) ([]net.IP, error) {
return resolveResourceIPsWithDial(ctx, server, host, (&net.Dialer{}).DialContext)
}
func productionResourceCheckDeps() resourceCheckDeps {
d := managedDependencies(runPlayerCurl)
return resourceCheckDeps{lookup: os.LookupEnv, guard: func() error {
if _, err := managedRuntimeGuard(d); err != nil {
return err
}
// The provisioned local.json and owned SS chain use the target's fixed
// internal port, not an additional ss-local listener in this namespace.
if port, _ := d.lookup("SOCKS_PORT"); port != "1080" {
return errDNSForwarder
}
cfg, err := parseManagedDNSConfig(d.lookup)
if err != nil {
return err
}
return verifyResourceDNSRuntime(cfg)
}, resolve: resolveResourceIPs, runner: runResourceCurl}
}
type resourceCheckResult struct {
Service string `json:"service"`
Resource string `json:"resource"`
Group string `json:"group"`
Outcome string `json:"outcome"`
TLS string `json:"tls"`
Bytes int `json:"bytes"`
TimeMS float64 `json:"time_ms"`
Status int `json:"status"`
}
// A synchronous runner: subprocess cancellation joins before this returns.
// The guard is also called after DNS, immediately before the pinned transfer.
func runResourceProtocol(ctx context.Context, service string, r resourceCheckSpec, protocol, server, proxy string, d resourceCheckDeps) (resourceCheckResult, error) {
result := resourceCheckResult{Service: service, Resource: r.ID, Group: r.Group, TLS: protocol, Outcome: "dns_failed"}
if err := d.guard(); err != nil {
return result, qualificationOperation(err, stagePreDNSGuard)
}
probeCtx, cancel := context.WithTimeout(ctx, time.Duration(r.TimeoutMS)*time.Millisecond)
defer cancel()
ips, err := d.resolve(probeCtx, server, r.host)
if d.qualification && err != nil {
var dns *net.DNSError
knownTimeout := probeCtx.Err() != nil && managedRecoveryTimeoutSafe(err)
if !knownTimeout && (!errors.As(err, &dns) || !dns.IsNotFound) {
return result, qualificationOperation(err, stageDNS)
}
}
if d.qualification && err == nil && (len(ips) == 0 || len(ips) > 64) {
return result, qualificationOperation(errResourceConfig, stageDNS)
}
if err == nil && len(ips) > 0 && len(ips) <= 64 {
approved := true
for _, ip := range ips {
if !approvedResourceIP(r, ip) {
approved = false
}
}
result.Outcome = "destination_denied"
if approved {
if err := d.guard(); err != nil {
return result, qualificationOperation(err, stagePostDNSGuard)
}
pin := r.host + ":443:" + net.JoinHostPort(ips[0].String(), "443")
options := resourceTransferOptions{method: r.Method, limit: r.Expect.MaxBytes, timeout: time.Duration(r.TimeoutMS) * time.Millisecond, config: r.privateConfig + "connect-to = " + privateCurlValue(pin) + "\n", expect: &r.Expect}
switch r.Kind {
case "image":
options.response = resourceResponseImage
case "mp4_range":
options.response, options.mediaType = resourceResponseMP4, r.Expect.ContentType
}
transfer, _, err := collectResourceTransferOptions(probeCtx, proxy, r.URL, r.host, protocol, d.runner, options)
if d.qualification && err != nil && err != errResourceExpectation {
knownTimeout := probeCtx.Err() != nil && managedRecoveryTimeoutSafe(err)
if !knownTimeout && !resourceRejectionOnly(err) && !managedAssetTransferNonpass(err) {
return result, qualificationOperation(err, stageTransfer)
}
}
result.Outcome = "transfer_failed"
if err == nil || err == errResourceExpectation {
result.Bytes, result.TimeMS, result.Status = len(transfer.Body), transfer.LatencyMS, transfer.Status
result.Outcome = "expectation_failed"
if err == nil {
result.Outcome = "pass"
}
}
}
}
if probeCtx.Err() != nil {
result.Outcome = "timeout"
if ctx.Err() != nil {
result.Outcome = "cancelled"
}
}
return result, nil
}
func resourceCheckCLI(ctx context.Context, path string, stdout, stderr io.Writer, d resourceCheckDeps) int {
if d.lookup == nil {
fmt.Fprintln(stderr, "configerror: invalid resource check configuration")
return 2
}
cfg, err := readResourceChecks(path, d.lookup)
dns, dnsErr := parseManagedDNSConfig(d.lookup)
proxy, _ := d.lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
proxy, proxyErr := managedProbeProxy(proxy)
if err != nil || dnsErr != nil || proxyErr != nil || d.guard == nil || d.resolve == nil || d.runner == nil || (dns.Mode == "native" && dns.NativeServer == "") {
fmt.Fprintln(stderr, "configerror: invalid resource check configuration")
return 2
}
server := dns.serverAddress()
if dns.Mode == "native" {
server = net.JoinHostPort(server, "53")
}
code := 0
for _, s := range cfg.Services {
for _, r := range s.Resources {
protocols := []string{r.TLS}
if r.TLS == "both" {
protocols = []string{"tls12", "tls13"}
}
for _, protocol := range protocols {
if ctx.Err() != nil {
return 1
}
result, err := runResourceProtocol(ctx, s.ID, r, protocol, server, proxy, d)
if err != nil {
fmt.Fprintln(stderr, "blocked: resource check ownership unavailable")
return 2
}
if result.Outcome != "pass" {
code = 1
}
if json.NewEncoder(stdout).Encode(result) != nil {
return 1
}
}
}
}
return code
}
func configuredResourceCheck(path string, stdout, stderr io.Writer) int {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
return resourceCheckCLI(ctx, path, stdout, stderr, productionResourceCheckDeps())
}
+499
View File
@@ -0,0 +1,499 @@
package main
import (
"bytes"
"context"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
)
const resourceCheckExample = `{"version":1,"services":[{"id":"first","resources":[{"id":"page","group":"web","kind":"http","url":"https://example.com/private?secret=value","method":"GET","tls":"tls12","timeout_ms":1000,"expect":{"statuses":[200],"content_type":"text/plain","min_bytes":1,"max_bytes":100}}]},{"id":"second","resources":[{"id":"headers","group":"web","kind":"http","url_env":"CHECK_URL","method":"HEAD","tls":"both","timeout_ms":1000,"expect":{"statuses":[200],"min_bytes":0,"max_bytes":100},"headers_env":{"Authorization":"CHECK_TOKEN"}}]}]}`
func resourceCheckLookup(key string) (string, bool) {
values := map[string]string{"CHECK_URL": "https://example.org/secret", "CHECK_TOKEN": "Bearer private-token", "ZAPRET_AUTOTUNE_SOCKS_PROXY": "socks5h://127.0.0.1:1080"}
v, ok := values[key]
return v, ok
}
func resourceCheckFile(t *testing.T, data string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "checks.json")
if err := os.WriteFile(path, []byte(data), 0600); err != nil {
t.Fatal(err)
}
return path
}
func configuredWire(raw, method, protocol, mode string, out, diag io.Writer) {
body := "hello"
length := len(body)
if method == "HEAD" {
body, length = "", 999999999
}
status, verify, redirects, speed := 200, 0, 0, 50
if method == "HEAD" {
speed = 0
}
if mode == "redirect" {
status = 302
}
if mode == "status500" {
status = 500
}
if mode == "redirect-count" {
redirects = 1
}
if mode == "verify" {
verify = 60
}
if mode == "origin" {
raw = "https://evil.example/"
}
if mode == "overflow" {
body, length = strings.Repeat("a", 101), 101
}
if mode == "truncated" {
length++
}
if mode == "extra" {
body += "extra"
}
if mode == "chunked" {
fmt.Fprint(out, "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nTransfer-Encoding: chunked\r\n\r\n")
body = "5\r\nhello\r\n0\r\n\r\n"
} else {
fmt.Fprintf(out, "HTTP/1.1 %d OK\r\nContent-Type: text/plain\r\nContent-Length: %d\r\n\r\n", status, length)
}
fmt.Fprint(out, body)
metrics, _ := json.Marshal(map[string]interface{}{"url_effective": raw, "ssl_verify_result": verify, "num_redirects": redirects, "http_code": status, "time_total": 0.1, "speed_download": speed, "size_download": len(body)})
fmt.Fprint(out, resourceTrailer, string(metrics))
if mode == "tls" {
protocol = "tls13"
}
version := "1.2"
if protocol == "tls13" {
version = "1.3"
}
fmt.Fprintln(diag, "SSL connection using TLSv"+version)
}
func TestResourceCheckSchema(t *testing.T) {
if _, err := parseResourceChecks([]byte(resourceCheckExample), resourceCheckLookup); err != nil {
t.Fatal(err)
}
for _, data := range []string{
resourceCheckExample + ` {}`, strings.Replace(resourceCheckExample, `"version":1`, `"version":3`, 1),
strings.Replace(resourceCheckExample, `"version":1`, `"version":1,"version":1`, 1),
strings.Replace(resourceCheckExample, `"version":1`, `"unknown":1,"version":1`, 1),
strings.Replace(resourceCheckExample, `"id":"first"`, `"id":"First /"`, 1),
strings.Replace(resourceCheckExample, `"id":"second"`, `"id":"first"`, 1),
strings.Replace(resourceCheckExample, `"max_bytes":100`, `"max_bytes":0`, 1),
strings.Replace(resourceCheckExample, `"method":"GET"`, `"method":"POST"`, 1),
strings.Replace(resourceCheckExample, `"kind":"http"`, `"kind":"image"`, 1),
strings.Replace(resourceCheckExample, `"statuses":[200]`, `"statuses":[302]`, 1),
strings.Replace(resourceCheckExample, `"statuses":[200]`, `"statuses":[200,200]`, 1),
strings.Replace(resourceCheckExample, `"tls":"tls12"`, `"tls":"auto"`, 1),
strings.Replace(resourceCheckExample, `"timeout_ms":1000`, `"timeout_ms":30001`, 1),
strings.Replace(resourceCheckExample, `"headers_env":{"Authorization":"CHECK_TOKEN"}`, `"headers_env":{"Host":"CHECK_TOKEN"}`, 1),
strings.Replace(resourceCheckExample, `"CHECK_TOKEN"`, `"missing"`, 1),
strings.Replace(resourceCheckExample, `"CHECK_TOKEN"`, `"MISSING"`, 1),
strings.Replace(resourceCheckExample, `"CHECK_URL"`, `"MISSING"`, 1),
strings.Replace(resourceCheckExample, `"method":"GET"`, `"method":null`, 1),
strings.Replace(resourceCheckExample, `"kind":"http"`, `"allow_private":true,"kind":"http"`, 1),
strings.Replace(resourceCheckExample, `"kind":"http"`, `"allowed_networks":["10.0.0.1/24"],"kind":"http"`, 1),
} {
if _, err := parseResourceChecks([]byte(data), resourceCheckLookup); err == nil {
t.Fatal("invalid schema accepted")
}
}
for _, raw := range []string{"http://example.com/", "https://user:password@example.com/", "https://example.com:444/", "https://example.com/a%0d%0aHeader", "https://example.com/a\n", "https://example.com/#fragment", "https://example.com:/", "https://[::1]/", "https://2130706433/", "https://127.1/", "https://0x7f000001/", "https://0177.0.0.1/"} {
lookup := func(key string) (string, bool) {
if key == "CHECK_URL" {
return raw, true
}
return resourceCheckLookup(key)
}
if _, err := parseResourceChecks([]byte(resourceCheckExample), lookup); err == nil {
t.Fatal("invalid origin accepted")
}
}
lookup := func(key string) (string, bool) {
if key == "CHECK_TOKEN" {
return "secret\r\nHost: evil", true
}
return resourceCheckLookup(key)
}
if _, err := parseResourceChecks([]byte(resourceCheckExample), lookup); err == nil {
t.Fatal("header injection")
}
for _, name := range []string{"Range", "rAnGe", "Accept-Encoding", "Host", "Connection", "Content-Length", "Proxy-Authorization"} {
data := strings.Replace(resourceCheckExample, `"headers_env":{"Authorization":"CHECK_TOKEN"}`, `"headers_env":{"`+name+`":"CHECK_TOKEN"}`, 1)
if _, err := parseResourceChecks([]byte(data), resourceCheckLookup); err == nil {
t.Fatal("transport header accepted", name)
}
}
}
func TestResourceCheckFiles(t *testing.T) {
path := resourceCheckFile(t, resourceCheckExample)
link := filepath.Join(t.TempDir(), "link")
if err := os.Symlink(path, link); err != nil {
t.Fatal(err)
}
for _, p := range []string{link, t.TempDir(), resourceCheckFile(t, strings.Repeat("x", (1<<20)+1))} {
if _, err := readResourceChecks(p, resourceCheckLookup); err == nil {
t.Fatal("unsafe file accepted")
}
}
if _, err := readResourceChecks(path, resourceCheckLookup); err != nil {
t.Fatal(err)
}
info, _ := os.Stat(path)
if info.Mode().Perm() != 0600 {
t.Fatal("changed permissions")
}
}
func TestResourceCheckResolverPinnedServer(t *testing.T) {
var workers sync.WaitGroup
dial := func(ctx context.Context, network, address string) (net.Conn, error) {
if network != "tcp" || address != dnsListenAddress {
t.Error("DNS fallback", network, address)
}
client, server := net.Pipe()
workers.Add(1)
go func() {
defer workers.Done()
defer server.Close()
server.SetDeadline(time.Now().Add(time.Second))
var length [2]byte
if _, err := io.ReadFull(server, length[:]); err != nil {
t.Error(err)
return
}
query := make([]byte, int(binary.BigEndian.Uint16(length[:])))
if _, err := io.ReadFull(server, query); err != nil {
t.Error(err)
return
}
end, name, ok := dnsQuestion(query)
if !ok {
t.Error("invalid query")
return
}
if !bytes.Equal(name[:len(name)-4], []byte{7, 'e', 'x', 'a', 'm', 'p', 'l', 'e', 3, 'c', 'o', 'm', 0}) {
t.Error("search-domain expansion")
}
response := append([]byte(nil), query[:end]...)
response[2], response[3] = 0x81, 0x80
response[10], response[11] = 0, 0 // strip query EDNS
if binary.BigEndian.Uint16(query[end-4:end-2]) == 1 {
response[7] = 2
for _, ip := range [][]byte{{93, 184, 216, 34}, {10, 0, 0, 1}} {
response = append(response, 0xc0, 12, 0, 1, 0, 1, 0, 0, 0, 1, 0, 4)
response = append(response, ip...)
}
}
binary.BigEndian.PutUint16(length[:], uint16(len(response)))
server.Write(append(length[:], response...))
}()
return client, nil
}
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
ips, err := resolveResourceIPsWithDial(ctx, dnsListenAddress, "example.com", dial)
workers.Wait()
if err != nil || len(ips) != 2 {
t.Fatal("lost mixed DNS answers", ips, err)
}
for _, ip := range ips {
if ip.String() != "93.184.216.34" && ip.String() != "10.0.0.1" {
t.Fatal("unexpected answer", ip)
}
}
_, err = resolveResourceIPsWithDial(ctx, dnsListenAddress, "example.com", func(context.Context, string, string) (net.Conn, error) {
return nil, errors.New("forwarder unavailable")
})
if err == nil {
t.Fatal("DNS outage fell back")
}
}
func TestResourceCheckDestinationPolicy(t *testing.T) {
cfg, _ := parseResourceChecks([]byte(resourceCheckExample), resourceCheckLookup)
r := cfg.Services[0].Resources[0]
for _, raw := range []string{"127.0.0.1", "10.0.0.1", "172.16.0.1", "192.168.0.1", "169.254.169.254", "0.0.0.0", "224.0.0.1", "100.64.0.1", "100.100.100.200", "198.18.0.1", "::1", "::2", "fe80::1", "fc00::1", "fd00:ec2::254", "168.63.129.16", "::ffff:127.0.0.1", "2002:7f00:1::1"} {
if approvedResourceIP(r, net.ParseIP(raw)) {
t.Fatal("unsafe destination", raw)
}
}
if !approvedResourceIP(r, net.ParseIP("93.184.216.34")) {
t.Fatal("public denied")
}
data := strings.Replace(resourceCheckExample, `"kind":"http"`, `"allow_private":true,"allowed_networks":["10.20.0.0/16"],"kind":"http"`, 1)
cfg, err := parseResourceChecks([]byte(data), resourceCheckLookup)
if err != nil {
t.Fatal(err)
}
r = cfg.Services[0].Resources[0]
if !approvedResourceIP(r, net.ParseIP("10.20.1.2")) || approvedResourceIP(r, net.ParseIP("10.21.1.2")) || approvedResourceIP(r, net.ParseIP("93.184.216.34")) || approvedResourceIP(r, net.ParseIP("169.254.169.254")) {
t.Fatal("explicit subnet policy")
}
data = strings.Replace(resourceCheckExample, `"kind":"http"`, `"allowed_networks":["93.184.216.0/24"],"kind":"http"`, 1)
cfg, err = parseResourceChecks([]byte(data), resourceCheckLookup)
if err != nil {
t.Fatal(err)
}
r = cfg.Services[0].Resources[0]
if !approvedResourceIP(r, net.ParseIP("93.184.216.34")) || approvedResourceIP(r, net.ParseIP("8.8.8.8")) {
t.Fatal("public CIDR did not restrict")
}
data = strings.Replace(resourceCheckExample, `"kind":"http"`, `"allow_private":true,"allowed_networks":["0.0.0.0/0","::/0"],"kind":"http"`, 1)
cfg, err = parseResourceChecks([]byte(data), resourceCheckLookup)
if err != nil {
t.Fatal(err)
}
r = cfg.Services[0].Resources[0]
for _, raw := range []string{"127.0.0.1", "::1", "::ffff:127.0.0.1", "169.254.169.254", "fd00:ec2::254", "100.100.100.200", "168.63.129.16", "198.18.0.1", "fe80::1"} {
if approvedResourceIP(r, net.ParseIP(raw)) {
t.Fatal("private opt-in enabled metadata/reserved", raw)
}
}
}
func TestResourceCheckIsolationPrivacyAndPinning(t *testing.T) {
for _, mode := range []string{"pass", "mixed", "rebind", "runner", "expect", "guard"} {
t.Run(mode, func(t *testing.T) {
calls, resolves, guards := 0, 0, 0
var output, diagnostic bytes.Buffer
d := resourceCheckDeps{lookup: resourceCheckLookup, guard: func() error {
guards++
if mode == "guard" {
return errors.New("private-secret")
}
return nil
}, resolve: func(_ context.Context, server, host string) ([]net.IP, error) {
resolves++
if server != dnsListenAddress {
t.Fatal("wrong resolver")
}
if host == "example.com" && mode == "mixed" {
return []net.IP{net.ParseIP("93.184.216.34"), net.ParseIP("127.0.0.1")}, nil
}
if resolves == 3 && mode == "rebind" {
return []net.IP{net.ParseIP("10.0.0.1")}, nil
}
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(ctx context.Context, args []string, stdin io.Reader, out, diag io.Writer) error {
calls++
argv := strings.Join(args, " ")
if strings.Contains(argv, "example.") || strings.Contains(argv, "secret") || strings.Contains(argv, "private-token") || strings.Contains(argv, "93.184.") {
t.Fatal("private argv", args)
}
config, _ := io.ReadAll(stdin)
if !strings.Contains(string(config), `connect-to = "example.`) || !strings.Contains(string(config), `:443:93.184.216.34:443"`) || strings.Contains(argv, "--resolve") {
t.Fatal("not pinning CONNECT")
}
lines := strings.Split(string(config), "\n")
raw, err := strconv.Unquote(strings.TrimPrefix(lines[0], "url = "))
if err != nil {
t.Fatal(err)
}
method, protocol := "GET", "tls12"
if strings.Contains(argv, "--head") {
method = "HEAD"
if !strings.Contains(string(config), "Bearer private-token") {
t.Fatal("missing private header")
}
}
if strings.Contains(argv, "--tlsv1.3") {
protocol = "tls13"
}
if method == "GET" && mode == "runner" {
return errors.New("private-secret")
}
wireMode := ""
if mode == "expect" && method == "GET" {
wireMode = "redirect"
}
configuredWire(raw, method, protocol, wireMode, out, diag)
return nil
}}
code := resourceCheckCLI(context.Background(), resourceCheckFile(t, resourceCheckExample), &output, &diagnostic, d)
if mode == "pass" && (code != 0 || calls != 3 || resolves != 3) {
t.Fatalf("code=%d calls=%d output=%s", code, calls, output.String())
}
if mode != "pass" && code == 0 {
t.Fatal("false pass")
}
if mode == "rebind" && (calls != 2 || !strings.Contains(output.String(), `"outcome":"destination_denied"`)) {
t.Fatal("rebinding was not revalidated", output.String())
}
if mode != "guard" && strings.Count(output.String(), `"outcome":"pass"`) < 2 {
t.Fatal("failure stopped second service", output.String())
}
if mode == "guard" && (calls != 0 || resolves != 0 || guards != 1) {
t.Fatal("unguarded probe")
}
if strings.Contains(output.String()+diagnostic.String(), "secret") || strings.Contains(output.String(), "example.") || strings.Contains(output.String(), "Bearer") {
t.Fatal("private output")
}
})
}
}
func TestResourceCheckTransferContract(t *testing.T) {
for _, method := range []string{"GET", "HEAD"} {
for _, mode := range []string{"pass", "chunked", "redirect", "redirect-count", "verify", "origin", "tls", "overflow", "truncated", "extra"} {
t.Run(method+"/"+mode, func(t *testing.T) {
if method == "HEAD" && mode == "chunked" {
return
}
runner := func(_ context.Context, _ []string, _ io.Reader, out, diag io.Writer) error {
configuredWire("https://example.com/", method, "tls12", mode, out, diag)
return nil
}
transfer, _, err := collectResourceTransferOptions(context.Background(), "socks5h://127.0.0.1:1080", "https://example.com/", "example.com", "tls12", runner, resourceTransferOptions{method: method, limit: 100, timeout: time.Second})
valid := mode == "pass" || mode == "chunked" || method == "HEAD" && mode == "truncated"
if (err == nil) != valid {
t.Fatalf("err=%v", err)
}
if method == "HEAD" && err == nil && len(transfer.Body) != 0 {
t.Fatal("HEAD body")
}
})
}
}
}
func TestResourceCheckCancellationAndPrevalidation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
joined := false
d := resourceCheckDeps{lookup: resourceCheckLookup, guard: func() error { return nil }, resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(ctx context.Context, _ []string, _ io.Reader, _, _ io.Writer) error {
cancel()
<-ctx.Done()
joined = true
return ctx.Err()
}}
var output, diagnostic bytes.Buffer
if resourceCheckCLI(ctx, resourceCheckFile(t, resourceCheckExample), &output, &diagnostic, d) != 1 || !joined || strings.Count(output.String(), "cancelled") != 1 {
t.Fatal("cancel/join")
}
d.guard = func() error { t.Fatal("guard before all config validated"); return nil }
bad := strings.Replace(resourceCheckExample, `"CHECK_URL"`, `"MISSING"`, 1)
if resourceCheckCLI(context.Background(), resourceCheckFile(t, bad), &output, &diagnostic, d) != 2 {
t.Fatal("prevalidation")
}
// Runtime child deadlines are bounded and returned only after the child joins.
d.guard = func() error { return nil }
d.runner = func(ctx context.Context, _ []string, _ io.Reader, _, _ io.Writer) error {
<-ctx.Done()
return ctx.Err()
}
short := strings.ReplaceAll(resourceCheckExample, `"timeout_ms":1000`, `"timeout_ms":100`)
output.Reset()
if resourceCheckCLI(context.Background(), resourceCheckFile(t, short), &output, &diagnostic, d) != 1 || strings.Count(output.String(), `"outcome":"timeout"`) != 3 {
t.Fatal("timeouts", output.String())
}
}
// This loopback-only SOCKS fixture observes real curl's CONNECT destination.
// It rejects CONNECT immediately: no TLS or resource traffic leaves the fixture.
func TestResourceCheckCurlSOCKSPinnedDestination(t *testing.T) {
if _, err := exec.LookPath("curl"); err != nil {
t.Skip("curl unavailable")
}
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer listener.Close()
target := make(chan string, 1)
go func() {
conn, err := listener.Accept()
if err != nil {
target <- "accept failed"
return
}
defer conn.Close()
conn.SetDeadline(time.Now().Add(3 * time.Second))
fail := func() { target <- "invalid SOCKS framing" }
var hello [2]byte
if _, err := io.ReadFull(conn, hello[:]); err != nil || hello[0] != 5 {
fail()
return
}
methods := make([]byte, int(hello[1]))
if _, err := io.ReadFull(conn, methods); err != nil {
fail()
return
}
conn.Write([]byte{5, 0})
var header [4]byte
if _, err := io.ReadFull(conn, header[:]); err != nil || header[0] != 5 || header[1] != 1 {
fail()
return
}
var host string
switch header[3] {
case 1:
b := make([]byte, 4)
if _, err := io.ReadFull(conn, b); err != nil {
fail()
return
}
host = net.IP(b).String()
case 3:
var size [1]byte
if _, err := io.ReadFull(conn, size[:]); err != nil {
fail()
return
}
b := make([]byte, int(size[0]))
if _, err := io.ReadFull(conn, b); err != nil {
fail()
return
}
host = string(b)
default:
fail()
return
}
var port [2]byte
if _, err := io.ReadFull(conn, port[:]); err != nil || port != [2]byte{1, 187} {
fail()
return
}
target <- host
conn.Write([]byte{5, 5, 0, 1, 0, 0, 0, 0, 0, 0})
}()
ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
defer cancel()
_, _, err = collectResourceTransferOptions(ctx, "socks5h://"+listener.Addr().String(), "https://example.com/private?secret=value", "example.com", "tls12", runResourceCurl, resourceTransferOptions{method: "GET", limit: 100, timeout: 3 * time.Second, config: "connect-to = \"example.com:443:93.184.216.34:443\"\n"})
if err == nil {
t.Fatal("fixture unexpectedly transferred")
}
select {
case host := <-target:
if host != "93.184.216.34" {
t.Fatal("CONNECT not pinned", host)
}
case <-ctx.Done():
t.Fatal("SOCKS child not joined")
}
}
+537
View File
@@ -0,0 +1,537 @@
package main
import (
"bufio"
"bytes"
"context"
"crypto/sha256"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"image"
"io"
"math"
"mime"
"net/http"
"net/textproto"
"net/url"
"os/exec"
"regexp"
"strconv"
"strings"
"time"
)
// stdin is mandatory: ephemeral signed URLs must never enter argv or diagnostics.
type subprocessRunner func(context.Context, []string, io.Reader, io.Writer, io.Writer) error
func runResourceCurl(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
cmd := exec.CommandContext(ctx, "curl", args...)
cmd.Stdin, cmd.Stdout, cmd.Stderr = stdin, out, diagnostic
return cmd.Run()
}
type resourceError struct {
cause error
reason qualificationReason
}
// Created only after proxy, URL and TLS inputs have passed collector validation.
// Rejection is never qualification, nor evidence that local ownership failed.
type resourcePeerTLSRejection struct{ error }
// Created only after origin, verified TLS and HTTP framing have been decoded.
// A configured HTTP peer response rejection is not local ownership failure.
type resourcePeerResponseRejection struct{ error }
func (e *resourcePeerResponseRejection) Error() string { return "resource peer response rejected" }
func (e *resourcePeerResponseRejection) Unwrap() error { return e.error }
func (e *resourcePeerTLSRejection) Error() string { return "resource peer TLS rejected" }
func (e *resourcePeerTLSRejection) Unwrap() error { return e.error }
func resourceRejectionOnly(err error) bool {
switch err.(type) {
case *resourcePeerTLSRejection, *resourcePeerResponseRejection, imageBodyRejection:
return true
}
if e, ok := err.(*resourceError); ok {
return resourceRejectionOnly(e.cause)
}
if e, ok := err.(interface{ Unwrap() []error }); ok {
children := e.Unwrap()
if len(children) == 0 {
return false
}
for _, child := range children {
if !resourceRejectionOnly(child) {
return false
}
}
return true
}
return false
}
// Safe metadata classifications contain neither page content nor resource URLs.
type avatarMetadataError string
func (e avatarMetadataError) Error() string { return string(e) }
func (e *resourceError) PublicError() string { return "resource qualification failed" }
func (e *resourceError) Error() string { return e.PublicError() }
func (e *resourceError) Unwrap() error { return e.cause }
func resourceFailure(ctx context.Context, err error) error {
if ctx.Err() != nil {
err = errors.Join(err, ctx.Err())
}
if !managedRecoveryTimeoutSafe(err) && !resourceRejectionOnly(err) {
err = &managedRecoveryFatal{Err: err}
}
return &resourceError{cause: err}
}
func privateResourceURL(raw, host string) (*url.URL, error) {
if len(raw) == 0 || len(raw) > 16<<10 {
return nil, errors.New("invalid resource URL")
}
for _, c := range raw {
if c <= 32 || c == 127 {
return nil, errors.New("invalid resource URL")
}
}
u, err := url.Parse(raw)
if err != nil || u.Scheme != "https" || u.User != nil || u.Opaque != "" || u.Fragment != "" || u.Hostname() != host || (u.Port() != "" && u.Port() != "443") || strings.HasSuffix(u.Host, ":") || !discoveryHostname(host) {
return nil, errors.New("invalid resource origin")
}
return u, nil
}
func thumbnailResource(videoID string) (string, error) {
if !videoIDPattern.MatchString(videoID) {
return "", errors.New("invalid video ID")
}
return "https://i.ytimg.com/vi/" + videoID + "/hqdefault.jpg", nil
}
// Parse a fresh structural player/ytInitialData object, not HTML URL regexes.
// Callers extracting ytInitialData from a WWW page pass that JSON object here.
func avatarResource(metadata []byte) (string, error) {
if len(metadata) > playerResponseLimit {
return "", errors.New("metadata exceeds limit")
}
var root interface{}
if json.Unmarshal(metadata, &root) != nil {
return "", avatarMetadataError("jsonInvalid")
}
var visit func(interface{}, int) string
visit = func(v interface{}, depth int) string {
if depth > 64 {
return ""
}
switch x := v.(type) {
case map[string]interface{}:
if raw, ok := x["url"].(string); ok {
if _, err := privateResourceURL(raw, "yt3.ggpht.com"); err == nil {
return raw
}
}
for _, child := range x {
if raw := visit(child, depth+1); raw != "" {
return raw
}
}
case []interface{}:
for _, child := range x {
if raw := visit(child, depth+1); raw != "" {
return raw
}
}
}
return ""
}
if raw := visit(root, 0); raw != "" {
return raw, nil
}
return "", avatarMetadataError("noAllowedAvatar")
}
// The WWW page is transient; decode the embedded JSON structurally and never
// scan HTML for arbitrary URLs to fetch.
func avatarResourceFromPage(page []byte) (string, error) {
if len(page) > 5<<20 {
return "", errors.New("page exceeds limit")
}
found := false
for _, marker := range []string{"var ytInitialData = ", "ytInitialData = ", "window[\"ytInitialData\"] = "} {
if i := bytes.Index(page, []byte(marker)); i >= 0 {
found = true
var object json.RawMessage
if json.NewDecoder(bytes.NewReader(page[i+len(marker):])).Decode(&object) == nil {
return avatarResource(object)
}
}
}
if found {
return "", avatarMetadataError("jsonInvalid")
}
return "", avatarMetadataError("markerAbsent")
}
var resourceContentRange = regexp.MustCompile(`^bytes 0-([0-9]+)/([0-9]+)$`)
type mediaResource struct{ url, host, kind string }
type mediaCapability struct {
OrdinaryURL bool
SABRSupported bool
}
// SABR is deliberately unknown/not supported, never fabricated as readiness.
func playerMediaResource(body []byte) (mediaResource, mediaCapability, error) {
module, err := requiredYoutubeAdapter()
if err != nil {
return mediaResource{}, mediaCapability{}, err
}
resources, err := module.DiscoverResponse(body)
if err != nil {
return mediaResource{}, mediaCapability{}, err
}
for _, r := range resources {
if _, err := privateResourceURL(r.URL, r.Host); err == nil {
return mediaResource{r.URL, r.Host, r.Kind}, mediaCapability{OrdinaryURL: true}, nil
}
}
return mediaResource{}, mediaCapability{}, nil
}
const resourceTrailer = "\nAUTOTUNE_RESOURCE_METRICS:"
type resourceResponseKind uint8
// A completed generic mismatch retains measured facts, unlike unsafe transfers.
var errResourceExpectation = errors.New("resource expectation failed")
const (
resourceResponseGeneric resourceResponseKind = iota
resourceResponseImage
resourceResponseMP4
)
type resourceTransferOptions struct {
method string
limit int
timeout time.Duration
// Private curl config only: never placed in argv or public evidence.
config string
response resourceResponseKind
mediaType string
expect *resourceCheckExpect
}
func collectResourceTransferOptions(ctx context.Context, proxy, raw, host, protocol string, runner subprocessRunner, options resourceTransferOptions) (assetTransfer, http.Header, error) {
generic := options.response == resourceResponseGeneric
configured := generic || options.expect != nil
reason := reasonOrigin
fail := func(err error) (assetTransfer, http.Header, error) {
failure := resourceFailure(ctx, err)
failure.(*resourceError).reason = reason
return assetTransfer{}, nil, failure
}
if options.response > resourceResponseMP4 || (!generic && options.method != "GET") || (options.response == resourceResponseMP4 && options.mediaType != "video/mp4" && options.mediaType != "audio/mp4") {
return fail(errors.New("invalid response contract"))
}
proxy, proxyErr := managedProbeProxy(proxy)
if proxyErr != nil || runner == nil {
return fail(errors.New("invalid owned proxy or runner"))
}
if _, err := privateResourceURL(raw, host); err != nil {
return fail(err)
}
version, tlsFlag, maxTLS := uint16(0x0303), "--tlsv1.2", "1.2"
if protocol == "tls13" {
version, tlsFlag, maxTLS = 0x0304, "--tlsv1.3", "1.3"
} else if protocol != "tls12" {
return fail(errors.New("invalid TLS protocol"))
}
limit := 5 << 20
if options.response == resourceResponseMP4 {
limit = 65536
}
if configured {
limit = options.limit
}
// Qualification uses HTTP/1.1, not browser HTTP/2. --raw preserves native
// transfer framing so net/http, rather than curl and Go twice, decodes it.
args := []string{"--disable", "--silent", "--show-error", "--verbose", "--http1.1", "--raw", "--noproxy", "", "--proxy", proxy, "--proto", "=https", tlsFlag, "--tls-max", maxTLS, "--max-time", "30", "--max-filesize", strconv.Itoa(limit), "--suppress-connect-headers", "--include", "--write-out", resourceTrailer + "%{json}", "--config", "-"}
if options.response == resourceResponseMP4 {
args = append(args, "--range", "0-65535")
}
if configured {
if (options.method != "GET" && options.method != "HEAD") || options.limit < 1 || options.limit > 5<<20 || options.timeout < 100*time.Millisecond || options.timeout > 30*time.Second {
return fail(errors.New("invalid transfer options"))
}
if options.response == resourceResponseMP4 && options.limit > 65536 {
return fail(errors.New("invalid transfer options"))
}
args = append(args, "--max-time", strconv.FormatFloat(options.timeout.Seconds(), 'f', 3, 64))
if options.method == "HEAD" {
args = append(args, "--head", "--max-filesize", "0")
}
}
// curl config quotes are not JSON quotes; reject controls above, escape only these.
config := "url = \"" + strings.NewReplacer("\\", "\\\\", "\"", "\\\"").Replace(raw) + "\"\n"
config += options.config
timeout := 35 * time.Second
if configured {
timeout = options.timeout
}
childCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
var output, diagnostic bytes.Buffer
reason = reasonExecution
out := &playerBoundedWriter{dst: &output, limit: limit + (96 << 10)}
diag := &playerBoundedWriter{dst: &diagnostic, limit: 64 << 10}
err := runner(childCtx, args, strings.NewReader(config), out, diag)
if out.exceeded || diag.exceeded {
reason = reasonOutputBound
err = errors.Join(err, errors.New("resource output exceeds limit"))
}
if err != nil || childCtx.Err() != nil {
// Only actual curl peer negatives become typed resource damage. Mixed
// cancellation, output overflow and explicit safety errors veto this path.
if exit, ok := err.(*exec.ExitError); ok && childCtx.Err() == nil {
switch exit.ExitCode() {
case 35, 51, 60:
err = &resourcePeerTLSRejection{error: err}
}
}
return fail(errors.Join(err, childCtx.Err()))
}
reason = reasonMetrics
i := bytes.LastIndex(output.Bytes(), []byte(resourceTrailer))
if i < 0 {
return fail(errors.New("missing transfer metrics"))
}
var metrics struct {
URL string `json:"url_effective"`
Verify *int `json:"ssl_verify_result"`
Redirects *int `json:"num_redirects"`
Status int `json:"http_code"`
Time float64 `json:"time_total"`
Speed float64 `json:"speed_download"`
Size float64 `json:"size_download"`
}
if json.Unmarshal(output.Bytes()[i+len(resourceTrailer):], &metrics) != nil {
return fail(errors.New("invalid transfer metrics"))
}
reason = reasonOrigin
if _, err := privateResourceURL(metrics.URL, host); err != nil || metrics.URL != raw || metrics.Verify == nil || *metrics.Verify != 0 || metrics.Redirects == nil || *metrics.Redirects != 0 {
return fail(errors.New("resource origin verification failed"))
}
reason = reasonTLSEvidence
actualTLS := "SSL connection using TLSv1.2"
if version == 0x0304 {
actualTLS = "SSL connection using TLSv1.3"
}
otherTLS := "SSL connection using TLSv1.3"
if version == 0x0304 {
otherTLS = "SSL connection using TLSv1.2"
}
if !strings.Contains(diagnostic.String(), actualTLS) || strings.Contains(diagnostic.String(), otherTLS) {
return fail(errors.New("actual TLS version mismatch"))
}
reason = reasonFraming
wire := output.Bytes()[:i]
headerEnd := bytes.Index(wire, []byte("\r\n\r\n"))
statusEnd := bytes.Index(wire, []byte("\r\n"))
if headerEnd < 0 || headerEnd > 64<<10 || statusEnd < 0 {
return fail(errors.New("missing or oversized response headers"))
}
h, err := textproto.NewReader(bufio.NewReader(bytes.NewReader(wire[statusEnd+2 : headerEnd+4]))).ReadMIMEHeader()
if err != nil {
return fail(errors.New("invalid response headers"))
}
for _, key := range []string{"Content-Type", "Content-Length", "Transfer-Encoding", "Content-Range"} {
if len(h.Values(key)) > 1 {
return fail(errors.New("duplicate response framing headers"))
}
}
if h.Get("Transfer-Encoding") != "" && (h.Get("Transfer-Encoding") != "chunked" || h.Get("Content-Length") != "") {
return fail(errors.New("ambiguous response framing"))
}
r := bufio.NewReader(bytes.NewReader(wire))
request := &http.Request{Method: options.method}
response, err := http.ReadResponse(r, request)
if err != nil {
return fail(errors.New("invalid response headers"))
}
defer response.Body.Close()
// CONNECT headers are suppressed by curl. Interim/duplicate responses and
// unrequested versions are outside this contract, never heuristically skipped.
if response.ProtoMajor != 1 || response.ProtoMinor != 1 || response.StatusCode < 200 || (!configured && response.Header.Get("Content-Type") == "") {
return fail(errors.New("unsupported response contract"))
}
if generic && response.StatusCode >= 300 && response.StatusCode < 400 {
return fail(&resourcePeerResponseRejection{error: errors.New("redirect rejected")})
}
reason = reasonBody
body, err := io.ReadAll(io.LimitReader(response.Body, int64(limit+1)))
// With --raw curl counts the wire body, including chunk framing; the image
// limit and durable byte identity apply to the fully decoded payload instead.
wireSize := len(wire) - headerEnd - 4
_, remaining := r.Peek(1)
if err != nil || remaining != io.EOF || (!configured && len(body) == 0) || len(body) > limit || metrics.Size != float64(wireSize) || response.StatusCode != metrics.Status {
return fail(errors.New("incomplete or oversized resource body"))
}
reason = reasonMetrics
for _, n := range []float64{metrics.Time, metrics.Speed} {
if n < 0 || (!configured && n == 0) || math.IsNaN(n) || math.IsInf(n, 0) {
return fail(errors.New("invalid resource metrics"))
}
}
if configured && metrics.Time > options.timeout.Seconds()+1 {
return fail(errors.New("resource timing exceeds budget"))
}
reason = reasonMIME
contentType, _, err := mime.ParseMediaType(response.Header.Get("Content-Type"))
if err != nil {
contentType = ""
}
t := assetTransfer{EffectiveHost: host, SSLVerifyResult: metrics.Verify, Redirects: *metrics.Redirects, TLSVersion: version, Status: metrics.Status, ContentType: contentType, Body: body, LatencyMS: metrics.Time * 1000, SpeedKBPS: metrics.Speed / 1024}
if err != nil && options.expect != nil && (options.response == resourceResponseImage || options.response == resourceResponseMP4) {
// A malformed semantic MIME value is a completed typed mismatch only
// after the common wire/security checks; never retain the raw value.
return t, response.Header, errResourceExpectation
}
if err != nil && (!configured || response.Header.Get("Content-Type") != "") {
return fail(errors.New("invalid content type"))
}
if err := options.validateResponse(&t, response.Header); err != nil {
if err == errResourceExpectation {
return t, response.Header, err
}
return assetTransfer{}, nil, resourceFailure(ctx, err)
}
return t, response.Header, nil
}
// Adapter contracts run only after the common origin/TLS/framing/bounds checks.
// Generic v1 deliberately has no range or decoded-image expectation.
func (options resourceTransferOptions) validateResponse(t *assetTransfer, h http.Header) error {
if err := options.validateContract(t, h); err != nil {
if options.expect != nil {
return errResourceExpectation
}
return err
}
if e := options.expect; e != nil {
for _, status := range e.Statuses {
if t.Status == status && len(t.Body) >= e.MinBytes && len(t.Body) <= e.MaxBytes && (e.ContentType == "" || t.ContentType == e.ContentType) {
return nil
}
}
return errResourceExpectation
}
return nil
}
func (options resourceTransferOptions) validateContract(t *assetTransfer, h http.Header) error {
switch options.response {
case resourceResponseGeneric:
return nil
case resourceResponseImage:
if t.Status != 200 {
return errors.New("image HTTP status")
}
if options.expect != nil {
// Reject GIF before the legacy full decoder can allocate its frames.
_, format, err := image.DecodeConfig(bytes.NewReader(t.Body))
if err != nil || (format != "png" && format != "jpeg") {
return errors.New("invalid public image format")
}
}
return normalizeAssetBody(t)
case resourceResponseMP4:
parts := resourceContentRange.FindStringSubmatch(h.Get("Content-Range"))
var end, total int64
if len(parts) == 3 {
var endErr, totalErr error
end, endErr = strconv.ParseInt(parts[1], 10, 64)
total, totalErr = strconv.ParseInt(parts[2], 10, 64)
if endErr != nil || totalErr != nil {
return errors.New("invalid media range or container")
}
}
if t.Status != 206 || t.ContentType != options.mediaType || len(parts) != 3 || end < 15 || end > 65535 || end+1 != int64(len(t.Body)) || total <= end || len(t.Body) < 16 || string(t.Body[4:8]) != "ftyp" || binary.BigEndian.Uint32(t.Body[:4]) < 16 || int(binary.BigEndian.Uint32(t.Body[:4])) > len(t.Body) {
return errors.New("invalid media range or container")
}
t.Size, t.Format, t.BodyHash = len(t.Body), "mp4", fmt.Sprintf("%x", sha256.Sum256(t.Body))
return nil
default:
return errors.New("invalid response contract")
}
}
// The ordinary 3/3 sampler is shared by image and media qualification.
func sampleResource(ctx context.Context, transfer func(context.Context) (assetTransfer, error)) ([]assetTransfer, error) {
var samples []assetTransfer
for i := 0; i < 3; i++ {
t, err := transfer(ctx)
if err != nil {
return nil, err
}
samples = append(samples, t)
}
return samples, nil
}
func collectAssetMeasurement(ctx context.Context, proxy, raw, host, protocol, args string, runner subprocessRunner) (assetMeasurement, error) {
m := assetMeasurement{Source: "image-transfer", Endpoint: host, Protocol: protocol, Args: args}
if !assetEndpoint(host) {
return m, resourceFailure(ctx, errors.New("invalid image endpoint"))
}
samples, err := sampleResource(ctx, func(ctx context.Context) (assetTransfer, error) {
t, _, err := collectResourceTransferOptions(ctx, proxy, raw, host, protocol, runner, resourceTransferOptions{method: "GET", response: resourceResponseImage})
if err != nil {
return assetTransfer{}, resourceFailure(ctx, err)
}
t.Body = nil
return t, nil
})
m.Transfers = samples
return m, err
}
func collectMediaSamples(ctx context.Context, proxy string, resource mediaResource, protocol string, runner subprocessRunner) ([]assetTransfer, error) {
if !strings.HasSuffix(resource.host, ".googlevideo.com") || (resource.kind != "video/mp4" && resource.kind != "audio/mp4") {
return nil, resourceFailure(ctx, errors.New("ordinary media unavailable"))
}
return sampleResource(ctx, func(ctx context.Context) (assetTransfer, error) {
t, _, err := collectMediaTransfer(ctx, proxy, resource, protocol, runner)
return t, err
})
}
// Only normalized range facts leave this collector; URLs, headers and payloads
// remain transient. Used by both cold sampling and post-warmup sampling.
func collectMediaTransfer(ctx context.Context, proxy string, resource mediaResource, protocol string, runner subprocessRunner) (assetTransfer, GooglevideoPass, error) {
if !strings.HasSuffix(resource.host, ".googlevideo.com") || (resource.kind != "video/mp4" && resource.kind != "audio/mp4") {
return assetTransfer{}, GooglevideoPass{}, resourceFailure(ctx, errors.New("ordinary media unavailable"))
}
t, h, err := collectResourceTransferOptions(ctx, proxy, resource.url, resource.host, protocol, runner, resourceTransferOptions{method: "GET", response: resourceResponseMP4, mediaType: resource.kind})
if err != nil {
return assetTransfer{}, GooglevideoPass{}, err
}
// The shared core has validated and normalized the complete range contract.
_, totalText, _ := strings.Cut(h.Get("Content-Range"), "/")
total, err := strconv.ParseInt(totalText, 10, 64)
if err != nil {
return assetTransfer{}, GooglevideoPass{}, resourceFailure(ctx, errors.New("invalid media range or container"))
}
// Retain only normalized byte identity; no signed URL or payload.
t.Body = nil
p := GooglevideoPass{Protocol: protocol, VerifiedTLS: true, Status: t.Status, RangeStart: 0, RangeEnd: int64(t.Size) - 1, ResourceBytes: total, Bytes: int64(t.Size), FTYP: true, LatencyMS: t.LatencyMS}
return t, p, nil
}
@@ -0,0 +1,313 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"image"
"image/png"
"io"
"os"
"os/exec"
"strconv"
"strings"
"testing"
)
func resourceFixture() []byte {
var b bytes.Buffer
if err := png.Encode(&b, image.NewRGBA(image.Rect(0, 0, 1, 1))); err != nil {
panic(err)
}
return b.Bytes()
}
func resourceWire(raw, mode string, out, diagnostic io.Writer) {
body, contentType, status := resourceFixture(), "image/png", 200
header := ""
if mode == "media" || mode == "bad-range" {
body = []byte{0, 0, 0, 24, 'f', 't', 'y', 'p', 'i', 's', 'o', 'm', 0, 0, 0, 0, 'i', 's', 'o', 'm', 'm', 'p', '4', '2'}
contentType, status, header = "video/mp4", 206, "Content-Range: bytes 0-23/100\r\n"
if mode == "bad-range" {
header = "Content-Range: bytes 1-24/100\r\n"
}
}
verify, redirects, effective := 0, 0, raw
switch mode {
case "root":
body, contentType, status = []byte("not found"), "text/html", 404
case "html":
body, contentType, status = []byte("bad request"), "text/html", 400
case "cn":
verify = 60
case "redirect":
status, redirects = 302, 1
case "host":
effective = "https://other.ytimg.com/resource"
case "truncated":
body = body[:len(body)/2]
case "type":
contentType = "image/jpeg"
}
fmt.Fprintf(out, "HTTP/1.1 %d result\r\nContent-Type: %s\r\nContent-Length: %d\r\n%s\r\n", status, contentType, len(body), header)
out.Write(body)
metrics, _ := json.Marshal(map[string]interface{}{"url_effective": effective, "ssl_verify_result": verify, "num_redirects": redirects, "http_code": status, "time_total": 0.1, "speed_download": 10000, "size_download": len(body)})
fmt.Fprint(out, resourceTrailer, string(metrics))
fmt.Fprintln(diagnostic, "* SSL connection using TLSv1.2 / test")
}
// A real child exercises the stdin/argv protocol, without any socket or curl.
func TestResourceChild(t *testing.T) {
if os.Getenv("AUTOTUNE_RESOURCE_CHILD") != "1" {
return
}
assertManagedDNSArgs(t, os.Args[3:], "socks5h://127.0.0.1:1080")
b, err := io.ReadAll(io.LimitReader(os.Stdin, 32<<10))
if err != nil {
os.Exit(3)
}
raw, err := strconv.Unquote(strings.TrimSuffix(strings.TrimPrefix(string(b), "url = "), "\n"))
if err != nil || strings.ContainsAny(raw, "\r\n") {
os.Exit(4)
}
resourceWire(raw, "", os.Stdout, os.Stderr)
os.Exit(0)
}
func TestResourcePrivateChildAndEvidence(t *testing.T) {
raw := "https://yt3.ggpht.com/avatar?signature=private-secret&quote=\"&slash=\\"
calls := 0
runner := func(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
calls++
joined := strings.Join(args, " ")
if strings.Contains(joined, "private-secret") || strings.Contains(joined, "https://") || strings.Contains(joined, "-k") || args[0] != "--disable" {
t.Fatalf("unsafe argv: %v", args)
}
if !strings.Contains(joined, "--noproxy --proxy socks5h://127.0.0.1:1080") || !strings.Contains(joined, "--tlsv1.2 --tls-max 1.2") {
t.Fatal("missing explicit routing/TLS")
}
assertManagedDNSArgs(t, args, "socks5h://127.0.0.1:1080")
if !strings.Contains(joined, "--http1.1 --raw") || !strings.Contains(joined, "--suppress-connect-headers") {
t.Fatal("missing native HTTP/1.1 raw contract")
}
cmd := exec.CommandContext(ctx, os.Args[0], append([]string{"-test.run=^TestResourceChild$", "--"}, args...)...)
cmd.Env = append(os.Environ(), "AUTOTUNE_RESOURCE_CHILD=1")
cmd.Stdin, cmd.Stdout, cmd.Stderr = stdin, out, diagnostic
return cmd.Run()
}
m, err := collectAssetMeasurement(context.Background(), "socks5://127.0.0.1:1080", raw, "yt3.ggpht.com", "tls12", "--lua-desync=fake", runner)
if err != nil || calls != 3 {
t.Fatalf("calls=%d err=%v", calls, err)
}
if _, _, err := validateAssetTransfers(m); err != nil {
t.Fatal(err)
}
b, _ := json.Marshal(m)
if strings.Contains(string(b), "private-secret") || strings.Contains(string(b), "body\"") || len(m.Transfers[0].Body) != 0 || m.Transfers[0].Size != len(resourceFixture()) {
t.Fatal("durable evidence leaks payload or misses byte identity")
}
}
func TestResourceNativeFraming(t *testing.T) {
raw := "https://yt3.ggpht.com/avatar?signature=private-secret"
for _, mode := range []string{"length", "chunked", "close", "normalized-chunked", "h2", "empty", "truncated-header", "truncated-length", "truncated-chunk", "duplicate-length", "duplicate-type", "duplicate-transfer", "mixed-framing", "interim", "duplicate-response", "extra-bytes"} {
t.Run(mode, func(t *testing.T) {
payload := resourceFixture()
// A delimiter inside a body must not steal the final private footer.
payload = append(payload, []byte(resourceTrailer+`{"url_effective":"private-body-marker"}`)...)
header := fmt.Sprintf("HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: %d\r\n\r\n", len(payload))
body := payload
switch mode {
case "chunked", "normalized-chunked", "truncated-chunk":
header = "HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nTransfer-Encoding: chunked\r\n\r\n"
if mode != "normalized-chunked" {
var wire bytes.Buffer
fmt.Fprintf(&wire, "%x\r\n", len(payload[:20]))
wire.Write(payload[:20])
fmt.Fprint(&wire, "\r\n")
fmt.Fprintf(&wire, "%x\r\n", len(payload[20:]))
wire.Write(payload[20:])
fmt.Fprint(&wire, "\r\n0\r\n\r\n")
body = wire.Bytes()
if mode == "truncated-chunk" {
body = body[:len(body)-5]
}
}
case "close":
header = "HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nConnection: close\r\n\r\n"
case "h2":
header = strings.Replace(header, "HTTP/1.1 200 OK", "HTTP/2 200", 1)
case "empty":
header, body = "", nil
case "truncated-header":
header = "HTTP/1.1 200 OK\r\nContent-Type: image/png\r\n"
case "truncated-length":
body = payload[:len(payload)-1]
case "duplicate-length":
header = strings.Replace(header, "Content-Length:", fmt.Sprintf("Content-Length: %d\r\nContent-Length:", len(payload)), 1)
case "duplicate-type":
header = strings.Replace(header, "Content-Type:", "Content-Type: image/png\r\nContent-Type:", 1)
case "duplicate-transfer":
header = "HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nTransfer-Encoding: chunked\r\nTransfer-Encoding: chunked\r\n\r\n"
case "mixed-framing":
header = strings.Replace(header, "Content-Length:", "Transfer-Encoding: chunked\r\nContent-Length:", 1)
case "interim":
header = "HTTP/1.1 100 Continue\r\n\r\n" + header
case "duplicate-response":
header = "HTTP/1.1 200 Connection established\r\n\r\n" + header
case "extra-bytes":
body = append(append([]byte(nil), payload...), []byte("unframed bytes")...)
}
runner := func(_ context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
if !strings.Contains(strings.Join(args, " "), "--http1.1 --raw") {
t.Fatal("HTTP version/framing not pinned")
}
fmt.Fprint(out, header)
out.Write(body)
metrics, _ := json.Marshal(map[string]interface{}{"url_effective": raw, "ssl_verify_result": 0, "num_redirects": 0, "http_code": 200, "time_total": 0.1, "speed_download": 10000, "size_download": len(body)})
fmt.Fprint(out, resourceTrailer, string(metrics))
fmt.Fprintln(diagnostic, "SSL connection using TLSv1.2")
return nil
}
m, err := collectAssetMeasurement(context.Background(), "socks5://127.0.0.1:1080", raw, "yt3.ggpht.com", "tls12", "", runner)
valid := mode == "length" || mode == "chunked" || mode == "close"
if (err == nil) != valid {
t.Fatalf("valid=%v error=%v", valid, err)
}
if err != nil && strings.Contains(err.Error(), "private-secret") {
t.Fatal("private footer leaked")
}
if valid && (len(m.Transfers) != 3 || m.Transfers[0].Size != len(payload)) {
t.Fatal("wrong decoded body identity")
}
})
}
}
func TestResourceRejectionsAndCancellation(t *testing.T) {
for _, mode := range []string{"root", "html", "cn", "redirect", "host", "truncated", "type", "tls", "overflow", "unknown", "mixed", "cancel"} {
t.Run(mode, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
runner := func(_ context.Context, _ []string, stdin io.Reader, out, diagnostic io.Writer) error {
if mode == "cancel" {
cancel()
return context.Canceled
}
if mode == "unknown" {
return errors.New("private-secret")
}
if mode == "mixed" {
cancel()
return errors.Join(context.Canceled, errors.New("private-secret"))
}
if mode == "overflow" {
_, err := out.Write(make([]byte, 6<<20))
return err
}
resourceWire("https://yt3.ggpht.com/avatar?signature=private-secret", mode, out, diagnostic)
if mode == "tls" {
fmt.Fprintln(diagnostic, "SSL connection using TLSv1.3")
}
return nil
}
_, err := collectAssetMeasurement(ctx, "socks5://127.0.0.1:1080", "https://yt3.ggpht.com/avatar?signature=private-secret", "yt3.ggpht.com", "tls12", "", runner)
if err == nil || strings.Contains(err.Error(), "private-secret") {
t.Fatal("accepted failure or exposed secret", err)
}
var fatal *managedRecoveryFatal
if errors.As(err, &fatal) != (mode != "cancel" && mode != "truncated" && mode != "type") {
t.Fatal("unsafe cancellation classification", err)
}
if (mode == "cancel" || mode == "mixed") && !errors.Is(err, context.Canceled) {
t.Fatal("lost cancellation")
}
})
}
}
func TestResourceDiscoveryAndMedia(t *testing.T) {
if raw, err := avatarResourceFromPage([]byte(`<html><script>var ytInitialData = {"owner":{"thumbnail":{"thumbnails":[{"url":"https://yt3.ggpht.com/fresh"}]}}};</script></html>`)); err != nil || raw != "https://yt3.ggpht.com/fresh" {
t.Fatal("WWW structural parsing", err)
}
for _, raw := range []string{"http://yt3.ggpht.com/a", "https://yt3.ggpht.com.evil/a", "https://user@yt3.ggpht.com/a", "https://yt3.ggpht.com:444/a", "https://yt3.ggpht.com/a\nurl=bad"} {
if _, err := privateResourceURL(raw, "yt3.ggpht.com"); err == nil {
t.Fatal("accepted unsafe URL", raw)
}
}
if _, err := thumbnailResource("invalid"); err == nil {
t.Fatal("invalid thumbnail ID")
}
if raw, err := avatarResource([]byte(`{"owner":{"thumbnail":{"thumbnails":[{"url":"https://yt3.ggpht.com/fresh"}]}}}`)); err != nil || raw != "https://yt3.ggpht.com/fresh" {
t.Fatal("avatar parsing", err)
}
resource, capability, err := playerMediaResource([]byte(`{"playabilityStatus":{"status":"OK"},"streamingData":{"adaptiveFormats":[{"url":"https://rr1.unit.googlevideo.com/videoplayback?sig=private-secret","mimeType":"video/mp4; codecs=\"avc1\""}]}}`))
if err != nil || !capability.OrdinaryURL || capability.SABRSupported {
t.Fatal("media capability", err)
}
for _, mode := range []string{"media", "bad-range", "root"} {
runner := func(_ context.Context, args []string, _ io.Reader, out, diagnostic io.Writer) error {
assertManagedDNSArgs(t, args, "socks5h://127.0.0.1:1080")
if !strings.Contains(strings.Join(args, " "), "--range 0-65535") {
t.Fatal("missing bounded range")
}
resourceWire(resource.url, mode, out, diagnostic)
return nil
}
samples, err := collectMediaSamples(context.Background(), "socks5://127.0.0.1:1080", resource, "tls12", runner)
if (err == nil) != (mode == "media") {
t.Fatal(mode, err)
}
if err == nil && (len(samples) != 3 || samples[0].Body != nil || samples[0].Format != "mp4") {
t.Fatal("invalid normalized media samples")
}
}
_, capability, err = playerMediaResource([]byte(`{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"https://rr1.unit.googlevideo.com/sabr?sig=private"}}`))
if err != nil || capability.OrdinaryURL || capability.SABRSupported {
t.Fatal("SABR falsely green")
}
}
func TestResourceTLS13AndStrictDTO(t *testing.T) {
raw := "https://i.ytimg.com/vi/dQw4w9WgXcQ/hqdefault.jpg"
runner := func(_ context.Context, args []string, _ io.Reader, out, diagnostic io.Writer) error {
if !strings.Contains(strings.Join(args, " "), "--tlsv1.3 --tls-max 1.3") {
t.Fatal("TLS13 not pinned")
}
var private bytes.Buffer
resourceWire(raw, "", out, &private)
fmt.Fprintln(diagnostic, "* SSL connection using TLSv1.3 / test")
return nil
}
m, err := collectAssetMeasurement(context.Background(), "socks5://127.0.0.1:1080", raw, "i.ytimg.com", "tls13", "", runner)
if err != nil {
t.Fatal(err)
}
if _, _, err := validateAssetTransfers(m); err != nil {
t.Fatal(err)
}
encoded, _ := json.Marshal(m.Transfers[0])
var fields map[string]json.RawMessage
json.Unmarshal(encoded, &fields)
for key := range fields {
copy := make(map[string]json.RawMessage)
for k, v := range fields {
if k != key {
copy[k] = v
}
}
b, _ := json.Marshal(copy)
var transfer assetTransfer
if json.Unmarshal(b, &transfer) == nil {
t.Fatal("missing required field accepted", key)
}
}
fields["url"] = json.RawMessage(`"https://signed.invalid/?signature=private"`)
b, _ := json.Marshal(fields)
var transfer assetTransfer
if json.Unmarshal(b, &transfer) == nil {
t.Fatal("unknown private field accepted")
}
}
@@ -0,0 +1,127 @@
package main
import (
"bytes"
"context"
"fmt"
"io"
"reflect"
"strings"
"testing"
"time"
)
func TestResourceCoreMediaRangeIntegers(t *testing.T) {
for _, tc := range []struct {
name, contentRange string
total int64
}{
{"ordinary", "bytes 0-23/100", 100},
{"max-valid", "bytes 0-23/9223372036854775807", 9223372036854775807},
{"total-overflow", "bytes 0-23/9223372036854775808", 0},
{"total-huge", "bytes 0-23/999999999999999999999999999999999999", 0},
{"end-overflow", "bytes 0-9223372036854775808/9223372036854775807", 0},
{"nonnumeric", "bytes 0-23/abc", 0},
{"unknown", "bytes 0-23/*", 0},
{"signed", "bytes 0-23/+100", 0},
{"negative", "bytes 0-23/-100", 0},
{"signed-end", "bytes 0-+23/100", 0},
{"negative-end", "bytes 0--23/100", 0},
{"total-equals-end", "bytes 0-23/23", 0},
{"total-below-end", "bytes 0-23/22", 0},
{"end-body-mismatch", "bytes 0-24/100", 0},
} {
t.Run(tc.name, func(t *testing.T) {
raw := "https://rr1.unit.googlevideo.com/fresh?sig=private-secret"
runner := func(_ context.Context, _ []string, _ io.Reader, out, diag io.Writer) error {
var wire bytes.Buffer
resourceWire(raw, "media", &wire, diag)
fmt.Fprint(out, strings.Replace(wire.String(), "bytes 0-23/100", tc.contentRange, 1))
return nil
}
transfer, _, err := collectResourceTransferOptions(context.Background(), "socks5://127.0.0.1:1080", raw, "rr1.unit.googlevideo.com", "tls12", runner, resourceTransferOptions{method: "GET", response: resourceResponseMP4, mediaType: "video/mp4"})
valid := tc.total != 0
if (err == nil) != valid || !valid && !reflect.DeepEqual(transfer, assetTransfer{}) {
t.Fatalf("core valid=%v transfer=%+v err=%v", valid, transfer, err)
}
transfer, pass, err := collectMediaTransfer(context.Background(), "socks5://127.0.0.1:1080", mediaResource{raw, "rr1.unit.googlevideo.com", "video/mp4"}, "tls12", runner)
if (err == nil) != valid {
t.Fatalf("adapter valid=%v err=%v", valid, err)
}
if valid {
if pass.ResourceBytes != tc.total || pass.Bytes != 24 || pass.RangeEnd != 23 || transfer.Size != 24 {
t.Fatalf("incorrect normalized facts: %+v %+v", transfer, pass)
}
} else if !reflect.DeepEqual(transfer, assetTransfer{}) || !reflect.DeepEqual(pass, GooglevideoPass{}) {
t.Fatalf("rejected range produced evidence: %+v %+v", transfer, pass)
}
})
}
}
// Unlike the adapter golden test, these assertions call the shared execution
// entry directly: strict validation must not be left in a legacy-only branch.
func TestResourceCoreTypedContracts(t *testing.T) {
for _, mode := range []string{"pass", "status", "start", "end", "total", "malformed", "ftyp", "box", "decoder"} {
for _, kind := range []resourceResponseKind{resourceResponseGeneric, resourceResponseImage, resourceResponseMP4} {
t.Run(fmt.Sprintf("%d/%s", kind, mode), func(t *testing.T) {
options := resourceTransferOptions{method: "GET", limit: 1024, timeout: time.Second, response: kind, mediaType: "video/mp4"}
raw := "https://i.ytimg.com/fresh?sig=private-secret"
runner := func(_ context.Context, _ []string, _ io.Reader, out, diag io.Writer) error {
var wire bytes.Buffer
wireMode := ""
if kind == resourceResponseMP4 {
wireMode = "media"
}
if mode == "decoder" && kind != resourceResponseMP4 {
wireMode = "type"
}
resourceWire(raw, wireMode, &wire, diag)
data := wire.String()
switch mode {
case "status":
data = strings.ReplaceAll(data, "206", "200")
if kind != resourceResponseMP4 {
data = strings.ReplaceAll(data, "200", "201")
}
case "start":
data = strings.Replace(data, "bytes 0-23/100", "bytes 1-24/100", 1)
case "end":
data = strings.Replace(data, "bytes 0-23/100", "bytes 0-24/100", 1)
case "total":
data = strings.Replace(data, "bytes 0-23/100", "bytes 0-23/23", 1)
case "malformed":
data = strings.Replace(data, "bytes 0-23/100", "bytes 0-23/*", 1)
case "ftyp":
data = strings.Replace(data, "ftyp", "xxxx", 1)
case "box":
data = strings.Replace(data, string([]byte{0, 0, 0, 24}), string([]byte{0, 0, 0, 25}), 1)
}
fmt.Fprint(out, data)
return nil
}
transfer, _, err := collectResourceTransferOptions(context.Background(), "socks5://127.0.0.1:1080", raw, "i.ytimg.com", "tls12", runner, options)
valid := kind == resourceResponseGeneric || mode == "pass" || kind == resourceResponseImage && mode != "decoder" && mode != "status" || kind == resourceResponseMP4 && mode == "decoder"
if (err == nil) != valid {
t.Fatalf("valid=%v err=%v", valid, err)
}
if valid && kind != resourceResponseGeneric && (transfer.Size != len(transfer.Body) || transfer.BodyHash == "" || transfer.Format == "") {
t.Fatal("strict normalization not performed by core")
}
})
}
}
}
func TestResourceCoreGenericExpectationFacts(t *testing.T) {
raw := "https://i.ytimg.com/private"
runner := func(_ context.Context, _ []string, _ io.Reader, out, diag io.Writer) error {
resourceWire(raw, "", out, diag)
return nil
}
expect := resourceCheckExpect{Statuses: []int{201}, MaxBytes: 1024}
transfer, _, err := collectResourceTransferOptions(context.Background(), "socks5://127.0.0.1:1080", raw, "i.ytimg.com", "tls12", runner, resourceTransferOptions{method: "GET", limit: 1024, timeout: time.Second, expect: &expect})
if err != errResourceExpectation || transfer.Status != 200 || len(transfer.Body) != len(resourceFixture()) || transfer.LatencyMS != 100 {
t.Fatalf("core did not validate generic expectation with intact facts: %+v %v", transfer, err)
}
}
@@ -0,0 +1,193 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"net"
"reflect"
"strings"
"testing"
)
// Exercise the actual legacy adapters and generic producer with the same wire
// mutations. This test also runs unchanged against the pre-migration source.
func TestResourceCoreAdapterParity(t *testing.T) {
for _, adapter := range []string{"image", "media", "GET", "HEAD"} {
for _, protocol := range []string{"tls12", "tls13"} {
for _, mode := range []string{"pass", "verify", "origin", "redirect", "tls", "framing", "oversize", "range", "ftyp", "decoder"} {
t.Run(adapter+"/"+protocol+"/"+mode, func(t *testing.T) {
host := "i.ytimg.com"
if adapter == "media" {
host = "rr1.unit.googlevideo.com"
}
raw := "https://" + host + "/fresh?sig=private-secret"
calls := 0
runner := func(_ context.Context, args []string, stdin io.Reader, out, diag io.Writer) error {
calls++
argv := strings.Join(args, " ")
for _, required := range []string{"--disable --silent", "--http1.1 --raw", "--noproxy --proxy socks5h://127.0.0.1:1080", "--proto =https", "--suppress-connect-headers", "--config -"} {
if !strings.Contains(argv, required) {
t.Fatalf("missing %s", required)
}
}
if strings.Contains(argv, "private-secret") || strings.Contains(argv, host) || strings.Contains(argv, "--insecure") || strings.Contains(argv, "--location") {
t.Fatal("unsafe argv")
}
if strings.Contains(argv, "--range 0-65535") != (adapter == "media") || strings.Contains(argv, "--head") != (adapter == "HEAD") {
t.Fatal("method/range changed")
}
config, _ := io.ReadAll(stdin)
if !strings.Contains(string(config), raw) {
t.Fatal("URL not in private stdin")
}
if adapter == "GET" || adapter == "HEAD" {
if !strings.Contains(string(config), `connect-to = "i.ytimg.com:443:93.184.216.34:443"`) || !strings.Contains(string(config), "Authorization: private-header") {
t.Fatal("private header or CONNECT pin lost")
}
}
var wire, diagnostic bytes.Buffer
wireMode := ""
if adapter == "media" {
wireMode = "media"
}
if mode == "decoder" && adapter == "image" {
wireMode = "type"
}
resourceWire(raw, wireMode, &wire, &diagnostic)
data := wire.String()
if adapter == "HEAD" {
start := strings.Index(data, "\r\n\r\n") + 4
end := strings.LastIndex(data, resourceTrailer)
data = data[:start] + data[end:]
data = strings.Replace(data, fmt.Sprintf(`"size_download":%d`, len(resourceFixture())), `"size_download":0`, 1)
}
switch mode {
case "verify":
data = strings.Replace(data, `"ssl_verify_result":0`, `"ssl_verify_result":60`, 1)
case "origin":
data = strings.Replace(data, `"url_effective":"https://`, `"url_effective":"https://wrong.`, 1)
case "redirect":
data = strings.Replace(data, `"num_redirects":0`, `"num_redirects":1`, 1)
case "framing":
data = strings.Replace(data, "Content-Length:", "Content-Length: 1\r\nContent-Length:", 1)
case "oversize":
_, err := out.Write(make([]byte, 6<<20))
return err
case "range":
data = strings.Replace(data, "bytes 0-23/100", "bytes 1-24/100", 1)
case "ftyp":
data = strings.Replace(data, "ftyp", "xxxx", 1)
}
fmt.Fprint(out, data)
actual := "TLSv1.2"
if protocol == "tls13" {
actual = "TLSv1.3"
}
if mode == "tls" {
actual = "TLSv1.1"
}
fmt.Fprintln(diag, "SSL connection using "+actual)
return nil
}
valid := mode == "pass" || mode == "range" && adapter != "media" || mode == "ftyp" && adapter != "media" || mode == "decoder" && adapter != "image"
var err error
var transfer assetTransfer
switch adapter {
case "image":
var m assetMeasurement
m, err = collectAssetMeasurement(context.Background(), "socks5://127.0.0.1:1080", raw, host, protocol, "candidate", runner)
if err == nil {
transfer = m.Transfers[0]
if calls != 3 || m.Source != "image-transfer" || m.Endpoint != host || m.Protocol != protocol || m.Args != "candidate" {
t.Fatal("legacy measurement changed")
}
}
case "media":
var pass GooglevideoPass
transfer, pass, err = collectMediaTransfer(context.Background(), "socks5://127.0.0.1:1080", mediaResource{raw, host, "video/mp4"}, protocol, runner)
if err == nil && !reflect.DeepEqual(pass, GooglevideoPass{Protocol: protocol, VerifiedTLS: true, Status: 206, RangeStart: 0, RangeEnd: 23, ResourceBytes: 100, Bytes: 24, FTYP: true, LatencyMS: 100}) {
t.Fatalf("legacy pass changed: %+v", pass)
}
default:
d := resourceCheckDeps{guard: func() error { return nil }, resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: runner}
r := resourceCheckSpec{ID: "r", Group: "g", URL: raw, host: host, Method: adapter, TimeoutMS: 1000, Expect: resourceCheckExpect{Statuses: []int{200}, ContentType: "image/png", MaxBytes: 1024}, privateConfig: "header = \"Authorization: private-header\"\n"}
result, e := runResourceProtocol(context.Background(), "s", r, protocol, "dns", "socks5://127.0.0.1:1080", d)
err = e
if valid {
wantBytes := len(resourceFixture())
if adapter == "HEAD" {
wantBytes = 0
}
if result != (resourceCheckResult{Service: "s", Resource: "r", Group: "g", TLS: protocol, Outcome: "pass", Status: 200, Bytes: wantBytes, TimeMS: 100}) {
t.Fatalf("generic result changed: %+v", result)
}
} else if result.Outcome == "pass" {
t.Fatal("unsafe generic pass")
}
if err == nil && !valid {
err = fmt.Errorf("rejected transfer")
}
}
if (err == nil) != valid {
t.Fatalf("valid=%v err=%v", valid, err)
}
if valid && (adapter == "image" || adapter == "media") {
body, format, status, mime := resourceFixture(), "png", 200, "image/png"
if adapter == "media" {
body, format, status, mime = []byte{0, 0, 0, 24, 'f', 't', 'y', 'p', 'i', 's', 'o', 'm', 0, 0, 0, 0, 'i', 's', 'o', 'm', 'm', 'p', '4', '2'}, "mp4", 206, "video/mp4"
}
version := uint16(0x0303)
if protocol == "tls13" {
version = 0x0304
}
verify := 0
want := assetTransfer{EffectiveHost: host, SSLVerifyResult: &verify, TLSVersion: version, Status: status, ContentType: mime, Size: len(body), Format: format, BodyHash: fmt.Sprintf("%x", sha256.Sum256(body)), LatencyMS: 100, SpeedKBPS: 10000.0 / 1024}
if !reflect.DeepEqual(transfer, want) {
t.Fatalf("normalized facts changed: got %+v want %+v", transfer, want)
}
encoded, _ := json.Marshal(transfer)
if strings.Contains(string(encoded), "private-secret") {
t.Fatal("private evidence")
}
}
})
}
}
}
}
func TestResourceCoreGenericMismatchParity(t *testing.T) {
for _, field := range []string{"status", "mime", "bytes"} {
for _, qualification := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/qualification=%v", field, qualification), func(t *testing.T) {
raw := "https://i.ytimg.com/fresh?sig=private-secret"
r := resourceCheckSpec{ID: "r", Group: "g", URL: raw, host: "i.ytimg.com", Method: "GET", TimeoutMS: 1000, Expect: resourceCheckExpect{Statuses: []int{200}, ContentType: "image/png", MaxBytes: 1024}}
switch field {
case "status":
r.Expect.Statuses = []int{201}
case "mime":
r.Expect.ContentType = "image/jpeg"
case "bytes":
r.Expect.MinBytes = 1000
}
d := resourceCheckDeps{qualification: qualification, guard: func() error { return nil }, resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(_ context.Context, _ []string, _ io.Reader, out, diag io.Writer) error {
resourceWire(raw, "", out, diag)
return nil
}}
result, err := runResourceProtocol(context.Background(), "s", r, "tls12", "dns", "socks5://127.0.0.1:1080", d)
want := resourceCheckResult{Service: "s", Resource: "r", Group: "g", TLS: "tls12", Outcome: "expectation_failed", Bytes: len(resourceFixture()), TimeMS: 100, Status: 200}
if err != nil || result != want {
t.Fatalf("result=%+v err=%v", result, err)
}
})
}
}
}
+245
View File
@@ -0,0 +1,245 @@
package main
import (
"context"
"crypto/rand"
"encoding/binary"
"io"
"net"
"strings"
"time"
)
func resourceDNSQuery(host string, kind uint16) ([]byte, error) {
if !discoveryHostname(host) {
return nil, errDNSForwarder
}
query := make([]byte, 12)
if _, err := rand.Read(query[:2]); err != nil {
return nil, errDNSForwarder
}
query[2], query[5] = 1, 1
for _, label := range strings.Split(strings.ToLower(host), ".") {
query = append(query, byte(len(label)))
query = append(query, label...)
}
query = append(query, 0, byte(kind>>8), byte(kind), 0, 1)
if !validDNSQuery(query) {
return nil, errDNSForwarder
}
return query, nil
}
func resourceDNSExchange(ctx context.Context, server string, query []byte, dial dnsDialFunc) ([]byte, error) {
conn, err := dial(ctx, "tcp", server)
if err != nil {
return nil, errDNSForwarder
}
defer conn.Close()
deadline := time.Now().Add(5 * time.Second)
if d, ok := ctx.Deadline(); ok && d.Before(deadline) {
deadline = d
}
if conn.SetDeadline(deadline) != nil {
return nil, errDNSForwarder
}
// Closing the connection interrupts framed I/O, including contexts cancelled
// without a deadline. The callback is joined before returning.
done := make(chan struct{})
stop := context.AfterFunc(ctx, func() { conn.Close(); close(done) })
defer func() {
if !stop() {
<-done
}
}()
var size [2]byte
binary.BigEndian.PutUint16(size[:], uint16(len(query)))
if writeDNSFrame(conn, size[:], query) != nil {
return nil, errDNSForwarder
}
if _, err := io.ReadFull(conn, size[:]); err != nil {
return nil, errDNSForwarder
}
n := int(binary.BigEndian.Uint16(size[:]))
if n < 12 {
return nil, errDNSForwarder
}
response := make([]byte, n)
if _, err := io.ReadFull(conn, response); err != nil {
return nil, errDNSForwarder
}
if ctx.Err() != nil || !validDNSResponse(query, response) || response[2]&2 != 0 || response[3]&15 != 0 {
return nil, errDNSForwarder
}
return response, nil
}
func resourceDNSHost(name []byte) (string, bool) {
var labels []string
for i := 0; i < len(name); {
n := int(name[i])
i++
if n == 0 {
host := strings.ToLower(strings.Join(labels, "."))
return host, i == len(name) && discoveryHostname(host)
}
if n > 63 || i+n > len(name) {
return "", false
}
labels = append(labels, string(name[i:i+n]))
i += n
}
return "", false
}
// Only Answer records on a complete, unambiguous chain from the original
// question contribute addresses. Authority/Additional glue is never a target.
func resourceDNSAnswers(query, wire []byte, priorAliases map[string]bool) ([]net.IP, string, error) {
if !validDNSResponse(query, wire) {
return nil, "", errDNSForwarder
}
_, question, ok := dnsQuestion(query)
if !ok {
return nil, "", errDNSForwarder
}
host, ok := resourceDNSHost(question[:len(question)-4])
if !ok {
return nil, "", errDNSForwarder
}
end, ok := dnsQuestionEnd(wire)
if !ok {
return nil, "", errDNSForwarder
}
count := int(binary.BigEndian.Uint16(wire[6:8]))
if count > 128 {
return nil, "", errDNSForwarder
}
aliases := map[string]string{}
addresses := map[string][]net.IP{}
for i := 0; i < count; i++ {
header, name, ok := dnsName(wire, end)
if !ok || header+10 > len(wire) {
return nil, "", errDNSForwarder
}
owner, valid := resourceDNSHost(name)
if !valid {
return nil, "", errDNSForwarder
}
kind, class := binary.BigEndian.Uint16(wire[header:]), binary.BigEndian.Uint16(wire[header+2:])
start := header + 10
end = start + int(binary.BigEndian.Uint16(wire[header+8:]))
if end > len(wire) || class != 1 {
return nil, "", errDNSForwarder
}
switch kind {
case 1, 28:
size := 4
if kind == 28 {
size = 16
}
if end-start != size {
return nil, "", errDNSForwarder
}
addresses[owner] = append(addresses[owner], append(net.IP(nil), wire[start:end]...))
case 5:
last, target, ok := dnsName(wire, start)
if !ok || last != end {
return nil, "", errDNSForwarder
}
alias, ok := resourceDNSHost(target)
if !ok || aliases[owner] != "" {
return nil, "", errDNSForwarder
}
aliases[owner] = alias
}
}
seen := map[string]bool{}
for steps := 0; steps < 16; steps++ {
if seen[host] || priorAliases[host] {
return nil, "", errDNSForwarder
}
seen[host] = true
if alias := aliases[host]; alias != "" {
if len(addresses[host]) != 0 {
return nil, "", errDNSForwarder
}
priorAliases[host] = true
if len(priorAliases) > 16 {
return nil, "", errDNSForwarder
}
host = alias
continue
}
// Unrelated answer addresses/aliases are ambiguous, not a safe partial set.
for owner := range addresses {
if !seen[owner] {
return nil, "", errDNSForwarder
}
}
for owner := range aliases {
if !seen[owner] {
return nil, "", errDNSForwarder
}
}
if len(addresses[host]) > 0 {
return addresses[host], "", nil
}
if len(aliases) > 0 {
return nil, host, nil
}
return nil, "", nil // authenticated framing with NOERROR/NODATA
}
return nil, "", errDNSForwarder
}
func resolveResourceIPsWithDial(ctx context.Context, server, host string, dial dnsDialFunc) ([]net.IP, error) {
serverHost, port, err := net.SplitHostPort(server)
if err != nil || net.ParseIP(serverHost) == nil || (port != "53" && server != dnsListenAddress) || dial == nil {
return nil, errDNSForwarder
}
if ip := net.ParseIP(host); ip != nil {
return []net.IP{ip}, nil
}
var ips []net.IP
unique := map[string]bool{}
for _, kind := range []uint16{1, 28} {
name := host
seen := map[string]bool{}
priorAliases := map[string]bool{}
for steps := 0; ; steps++ {
if steps >= 16 || seen[strings.ToLower(name)] || ctx.Err() != nil {
return nil, errDNSForwarder
}
seen[strings.ToLower(name)] = true
query, err := resourceDNSQuery(name, kind)
if err != nil {
return nil, err
}
wire, err := resourceDNSExchange(ctx, server, query, dial)
if err != nil {
return nil, err
}
answers, alias, err := resourceDNSAnswers(query, wire, priorAliases)
if err != nil {
return nil, err
}
for _, ip := range answers {
if !unique[ip.String()] {
unique[ip.String()] = true
ips = append(ips, ip)
}
if len(ips) > 64 {
return nil, errDNSForwarder
}
}
if alias == "" {
if steps > 0 && len(answers) == 0 {
return nil, errDNSForwarder
}
break
}
name = alias
}
}
return ips, nil
}
+229
View File
@@ -0,0 +1,229 @@
package main
import (
"encoding/json"
"fmt"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"syscall"
)
// Generic checks do not inherit PID 1's anchors. Read them through procfs,
// without changing the stricter inherited-FD readiness/cleanup protocol.
func verifyResourceDNSRuntime(cfg managedDNSConfig) error {
if os.Geteuid() != 0 {
return errDNSForwarder
}
return verifyResourceDNSRuntimeAt(cfg, "/proc", filepath.Dir(dnsReadyPath), "/usr/local/bin/zapret-autotune")
}
func resourcePrivateFile(dir *os.File, name string) (*os.File, error) {
fd, err := syscall.Openat(int(dir.Fd()), name, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, errDNSForwarder
}
f := os.NewFile(uintptr(fd), name)
info, err := f.Stat()
if err != nil || !dnsOwnedAnchor(info, false) || info.Size() > 64<<10 {
f.Close()
return nil, errDNSForwarder
}
return f, nil
}
func resourceProcessOwned(proc string, pid int, exe string, args []string) bool {
base := filepath.Join(proc, strconv.Itoa(pid))
actual, err := os.Readlink(filepath.Join(base, "exe"))
if err != nil || actual != exe {
return false
}
cmd, err := os.ReadFile(filepath.Join(base, "cmdline"))
if err != nil || string(cmd) != strings.Join(args, "\x00")+"\x00" {
return false
}
status, err := os.ReadFile(filepath.Join(base, "status"))
if err != nil {
return false
}
uid := strconv.Itoa(os.Geteuid())
parent, owner := false, false
for _, line := range strings.Split(string(status), "\n") {
fields := strings.Fields(line)
if len(fields) == 2 && fields[0] == "PPid:" && fields[1] == "1" {
parent = true
}
if len(fields) == 5 && fields[0] == "Uid:" && fields[1] == uid && fields[2] == uid && fields[3] == uid && fields[4] == uid {
owner = true
}
}
if !parent || !owner {
return false
}
for _, ns := range []string{"pid", "net"} {
self, err := os.Readlink(filepath.Join(proc, "self/ns", ns))
if err != nil {
return false
}
other, err := os.Readlink(filepath.Join(base, "ns", ns))
if err != nil || other != self {
return false
}
}
return true
}
// All sockets matching a binding must be held by this process; finding just
// one matching inode is insufficient when another process owns UDP/TCP.
func resourceProcessSockets(proc string, pid, port int, udp, wildcard bool) bool {
fds, err := os.ReadDir(filepath.Join(proc, strconv.Itoa(pid), "fd"))
if err != nil {
return false
}
owned := map[string]bool{}
for _, fd := range fds {
link, err := os.Readlink(filepath.Join(proc, strconv.Itoa(pid), "fd", fd.Name()))
if err == nil {
owned[link] = true
}
}
tables := []string{"tcp"}
if udp {
tables = append(tables, "udp")
}
for _, table := range tables {
data, err := os.ReadFile(filepath.Join(proc, "net", table))
if err != nil {
return false
}
found := false
for _, line := range strings.Split(string(data), "\n") {
f := strings.Fields(line)
if len(f) < 10 {
continue
}
address := fmt.Sprintf("0100007F:%04X", port)
if !wildcard && f[1] == fmt.Sprintf("00000000:%04X", port) {
return false
}
match := f[1] == address || wildcard && f[1] == fmt.Sprintf("00000000:%04X", port)
if !match {
continue
}
if table == "tcp" && f[3] != "0A" {
continue
}
if table == "udp" && (f[3] != "07" || f[2] != "00000000:0000") {
return false
}
if !owned["socket:["+f[9]+"]"] {
return false
}
found = true
}
if !found {
return false
}
}
return true
}
func verifyResourceDNSRuntimeAt(cfg managedDNSConfig, proc, dir, executable string) error {
if cfg.Mode != "doh" && (cfg.Mode != "native" || cfg.NativeServer == "") {
return errDNSForwarder
}
if managedDirectory(dir) != nil {
return errDNSForwarder
}
fd, err := syscall.Open(dir, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
if err != nil {
return errDNSForwarder
}
parent := os.NewFile(uintptr(fd), "DNS private directory")
defer parent.Close()
info, err := parent.Stat()
if err != nil || !dnsOwnedAnchor(info, true) {
return errDNSForwarder
}
if cfg.Mode == "doh" {
ready, err := resourcePrivateFile(parent, "dns-ready")
if err != nil {
return err
}
defer ready.Close()
data, err := io.ReadAll(io.LimitReader(ready, 32))
if err != nil {
return errDNSForwarder
}
pid, err := strconv.Atoi(strings.TrimSuffix(string(data), "\n"))
if err != nil || pid <= 1 || string(data) != strconv.Itoa(pid)+"\n" {
return errDNSForwarder
}
anchorDir, err := os.Open(filepath.Join(proc, "1/fd/8"))
if err != nil {
return errDNSForwarder
}
defer anchorDir.Close()
anchor, err := os.Open(filepath.Join(proc, "1/fd/9"))
if err != nil {
return errDNSForwarder
}
defer anchor.Close()
anchorInfo, err := anchorDir.Stat()
if err != nil || !os.SameFile(info, anchorInfo) || dnsReadinessAction(anchorDir, anchor, pid, false) != nil {
return errDNSForwarder
}
if !resourceProcessOwned(proc, pid, executable, []string{"/usr/local/bin/zapret-autotune", "dns-forwarder", "serve"}) || !resourceProcessSockets(proc, pid, 5353, true, false) {
return errDNSForwarder
}
self, err := os.Stat(filepath.Join(proc, "self/exe"))
if err != nil {
return errDNSForwarder
}
peer, err := os.Stat(filepath.Join(proc, strconv.Itoa(pid), "exe"))
if err != nil || !os.SameFile(self, peer) {
return errDNSForwarder
}
}
local, err := resourcePrivateFile(parent, "local.json")
if err != nil {
return err
}
defer local.Close()
var localConfig struct {
Server string `json:"server"`
ServerPort int `json:"server_port"`
LocalPort int `json:"local_port"`
}
decoder := json.NewDecoder(io.LimitReader(local, 64<<10))
if decoder.Decode(&localConfig) != nil || localConfig.Server != "127.0.0.1" || localConfig.ServerPort != 8388 || localConfig.LocalPort != 1080 {
return errDNSForwarder
}
var extra interface{}
if decoder.Decode(&extra) != io.EOF {
return errDNSForwarder
}
processes, err := os.ReadDir(proc)
if err != nil {
return errDNSForwarder
}
localOwned, serverOwned := false, false
for _, entry := range processes {
pid, err := strconv.Atoi(entry.Name())
if err != nil || pid <= 1 {
continue
}
if resourceProcessOwned(proc, pid, "/usr/bin/ss-local", []string{"ss-local", "-c", dir + "/local.json"}) && resourceProcessSockets(proc, pid, 1080, false, true) {
localOwned = true
}
if resourceProcessOwned(proc, pid, "/usr/bin/ss-server", []string{"ss-server", "-c", dir + "/server.json", "-d", cfg.serverAddress()}) && resourceProcessSockets(proc, pid, 8388, false, true) {
serverOwned = true
}
}
if !localOwned || !serverOwned {
return errDNSForwarder
}
return nil
}
+316
View File
@@ -0,0 +1,316 @@
package main
import (
"context"
"encoding/binary"
"errors"
"io"
"net"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
)
func resourceDNSNameForTest(host string) []byte {
var name []byte
for _, s := range strings.Split(host, ".") {
name = append(name, byte(len(s)))
name = append(name, s...)
}
return append(name, 0)
}
func resourceDNSRecordForTest(owner string, kind uint16, data []byte) []byte {
r := resourceDNSNameForTest(owner)
r = append(r, byte(kind>>8), byte(kind), 0, 1, 0, 0, 0, 1, byte(len(data)>>8), byte(len(data)))
return append(r, data...)
}
func resourceDNSResponseForTest(q []byte, answers, additional [][]byte) []byte {
r := append([]byte(nil), q...)
r[2], r[3] = 0x81, 0x80
binary.BigEndian.PutUint16(r[6:8], uint16(len(answers)))
binary.BigEndian.PutUint16(r[10:12], uint16(len(additional)))
for _, rr := range answers {
r = append(r, rr...)
}
for _, rr := range additional {
r = append(r, rr...)
}
return r
}
func TestResourceDNSWireOnlyAndAliases(t *testing.T) {
for _, mode := range []string{"hosts", "alias", "follow", "loop", "cross-loop", "incomplete", "additional", "unrelated", "mixed", "mapped", "linklocal", "servfail", "wrong-id", "wrong-question", "bad-cname", "duplicate-cname", "alias-address", "bad-address", "truncated", "cancel"} {
t.Run(mode, func(t *testing.T) {
calls := 0
var workers sync.WaitGroup
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
dial := func(ctx context.Context, network, server string) (net.Conn, error) {
calls++
if network != "tcp" || server != dnsListenAddress {
t.Fatal("wrong DNS route")
}
client, peer := net.Pipe()
workers.Add(1)
go func() {
defer workers.Done()
defer peer.Close()
peer.SetDeadline(time.Now().Add(time.Second))
var size [2]byte
if _, err := io.ReadFull(peer, size[:]); err != nil {
return
}
q := make([]byte, int(binary.BigEndian.Uint16(size[:])))
if _, err := io.ReadFull(peer, q); err != nil {
return
}
end, question, ok := dnsQuestion(q)
if !ok {
t.Error("bad query")
return
}
host, _ := resourceDNSHost(question[:len(question)-4])
kind := binary.BigEndian.Uint16(q[end-4:])
if mode == "cancel" {
cancel()
var b [1]byte
peer.Read(b[:])
return
}
if mode == "truncated" {
peer.Write([]byte{0, 20, 1})
return
}
address := resourceDNSRecordForTest(host, 1, []byte{93, 184, 216, 34})
var answer, additional [][]byte
if kind == 1 {
answer = [][]byte{address}
}
switch mode {
case "alias":
if kind == 1 {
answer = [][]byte{resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("cdn.example")), resourceDNSRecordForTest("cdn.example", 1, []byte{93, 184, 216, 34})}
}
case "follow", "incomplete":
if host != "cdn.example" {
answer = [][]byte{resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("cdn.example"))}
} else if mode == "incomplete" {
answer = nil
} else {
answer = [][]byte{resourceDNSRecordForTest(host, kind, net.ParseIP("2606:4700::1111").To16())}
if kind == 1 {
answer = [][]byte{address}
}
}
case "loop":
answer = [][]byte{resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("cdn.example")), resourceDNSRecordForTest("cdn.example", 5, resourceDNSNameForTest(host))}
case "cross-loop":
if host == "example.com" {
answer = [][]byte{resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("cdn.example"))}
} else {
answer = [][]byte{resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("example.com")), resourceDNSRecordForTest("example.com", 1, []byte{93, 184, 216, 34})}
}
case "bad-cname":
answer = [][]byte{resourceDNSRecordForTest(host, 5, []byte{0xc0, 12, 0})}
case "duplicate-cname":
answer = [][]byte{resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("one.example")), resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("two.example"))}
case "alias-address":
answer = [][]byte{address, resourceDNSRecordForTest(host, 5, resourceDNSNameForTest("cdn.example"))}
case "bad-address":
answer = [][]byte{resourceDNSRecordForTest(host, 1, []byte{1, 2, 3})}
case "additional":
additional = [][]byte{resourceDNSRecordForTest("other.example", 1, []byte{127, 0, 0, 1})}
case "unrelated":
answer = append(answer, resourceDNSRecordForTest("other.example", 1, []byte{127, 0, 0, 1}))
case "mixed":
if kind == 1 {
answer = append(answer, resourceDNSRecordForTest(host, 1, []byte{10, 0, 0, 1}))
}
case "mapped":
if kind == 28 {
answer = [][]byte{resourceDNSRecordForTest(host, 28, net.ParseIP("::ffff:10.0.0.1").To16())}
}
case "linklocal":
if kind == 28 {
answer = [][]byte{resourceDNSRecordForTest(host, 28, net.ParseIP("fe80::1").To16())}
}
}
r := resourceDNSResponseForTest(q, answer, additional)
if mode == "servfail" {
r[3] = 0x82
}
if mode == "wrong-id" {
r[0] ^= 1
}
if mode == "wrong-question" {
r[13] ^= 1
}
binary.BigEndian.PutUint16(size[:], uint16(len(r)))
writeDNSFrame(peer, size[:], r)
}()
return client, nil
}
host := "example.com"
if mode == "hosts" {
host = "localhost"
}
ips, err := resolveResourceIPsWithDial(ctx, dnsListenAddress, host, dial)
workers.Wait()
valid := mode == "hosts" || mode == "alias" || mode == "follow" || mode == "additional" || mode == "mixed" || mode == "mapped" || mode == "linklocal"
if (err == nil) != valid || calls == 0 {
t.Fatalf("calls=%d ips=%v err=%v", calls, ips, err)
}
if mode == "hosts" && (calls != 2 || len(ips) != 1 || ips[0].String() != "93.184.216.34") {
t.Fatal("hosts file used", ips)
}
if mode == "additional" && len(ips) != 1 {
t.Fatal("additional glue collected", ips)
}
if mode == "mixed" || mode == "mapped" || mode == "linklocal" {
denied := false
for _, ip := range ips {
if !approvedResourceIP(resourceCheckSpec{}, ip) {
denied = true
}
}
if !denied {
t.Fatal("unsafe answer lost", ips)
}
}
})
}
for _, host := range []string{"localhost", "dev.home.zeusina.ru"} {
calls := 0
_, err := resolveResourceIPsWithDial(context.Background(), dnsListenAddress, host, func(context.Context, string, string) (net.Conn, error) { calls++; return nil, errors.New("offline") })
if err == nil || calls != 1 {
t.Fatal("system hosts/DNS bypass")
}
}
}
func resourceRuntimeFixture(t *testing.T) (managedDNSConfig, string, string, string) {
t.Helper()
root := t.TempDir()
proc, dir := filepath.Join(root, "proc"), filepath.Join(root, "private")
mkdir := func(p string) {
t.Helper()
if err := os.MkdirAll(p, 0700); err != nil {
t.Fatal(err)
}
}
write := func(p, s string) {
t.Helper()
if err := os.WriteFile(p, []byte(s), 0600); err != nil {
t.Fatal(err)
}
}
link := func(target, path string) {
t.Helper()
if err := os.Symlink(target, path); err != nil {
t.Fatal(err)
}
}
mkdir(dir)
mkdir(filepath.Join(proc, "net"))
mkdir(filepath.Join(proc, "1/fd"))
mkdir(filepath.Join(proc, "self/ns"))
executable := filepath.Join(root, "binary")
write(executable, "binary")
link(executable, filepath.Join(proc, "self/exe"))
for _, ns := range []string{"pid", "net"} {
link(ns+"[1]", filepath.Join(proc, "self/ns", ns))
}
write(filepath.Join(dir, "dns-ready"), "42\n")
link(dir, filepath.Join(proc, "1/fd/8"))
link(filepath.Join(dir, "dns-ready"), filepath.Join(proc, "1/fd/9"))
write(filepath.Join(dir, "local.json"), `{"server":"127.0.0.1","server_port":8388,"local_port":1080}`)
for _, p := range []struct {
pid int
exe string
args []string
sockets []string
}{
{42, executable, []string{"/usr/local/bin/zapret-autotune", "dns-forwarder", "serve"}, []string{"11", "12"}},
{43, "/usr/bin/ss-local", []string{"ss-local", "-c", dir + "/local.json"}, []string{"13"}},
{44, "/usr/bin/ss-server", []string{"ss-server", "-c", dir + "/server.json", "-d", dnsListenAddress}, []string{"14"}},
} {
base := filepath.Join(proc, strconv.Itoa(p.pid))
mkdir(filepath.Join(base, "fd"))
mkdir(filepath.Join(base, "ns"))
link(p.exe, filepath.Join(base, "exe"))
write(filepath.Join(base, "cmdline"), strings.Join(p.args, "\x00")+"\x00")
uid := strconv.Itoa(os.Geteuid())
write(filepath.Join(base, "status"), "PPid:\t1\nUid:\t"+strings.Join([]string{uid, uid, uid, uid}, "\t")+"\n")
for _, ns := range []string{"pid", "net"} {
link(ns+"[1]", filepath.Join(base, "ns", ns))
}
for i, inode := range p.sockets {
link("socket:["+inode+"]", filepath.Join(base, "fd", strconv.Itoa(i)))
}
}
write(filepath.Join(proc, "net/tcp"), "0: 0100007F:14E9 00000000:0000 0A 0 0 0 0 0 11\n1: 00000000:0438 00000000:0000 0A 0 0 0 0 0 13\n2: 00000000:20C4 00000000:0000 0A 0 0 0 0 0 14\n")
write(filepath.Join(proc, "net/udp"), "0: 0100007F:14E9 00000000:0000 07 0 0 0 0 0 12\n")
return managedDNSConfig{Mode: "doh"}, proc, dir, executable
}
func TestResourceDNSOwnedRuntime(t *testing.T) {
for _, mode := range []string{"valid", "missing", "symlink", "permissions", "stale", "unanchored", "foreign-udp", "foreign-tcp", "foreign-exe", "wrong-parent", "namespace", "ss-dns", "ss-remote"} {
t.Run(mode, func(t *testing.T) {
cfg, proc, dir, exe := resourceRuntimeFixture(t)
write := func(path, data string) {
if err := os.WriteFile(path, []byte(data), 0600); err != nil {
t.Fatal(err)
}
}
switch mode {
case "missing":
os.Remove(filepath.Join(dir, "dns-ready"))
case "symlink":
os.Rename(filepath.Join(dir, "dns-ready"), filepath.Join(dir, "other"))
os.Symlink("other", filepath.Join(dir, "dns-ready"))
case "permissions":
os.Chmod(filepath.Join(dir, "dns-ready"), 0644)
case "stale":
write(filepath.Join(dir, "dns-ready"), "45\n")
case "unanchored":
os.Remove(filepath.Join(proc, "1/fd/9"))
case "foreign-udp":
write(filepath.Join(proc, "net/udp"), "0: 0100007F:14E9 00000000:0000 07 0 0 0 0 0 99\n")
case "foreign-tcp":
write(filepath.Join(proc, "net/tcp"), "0: 0100007F:14E9 00000000:0000 0A 0 0 0 0 0 99\n")
case "foreign-exe":
os.Remove(filepath.Join(proc, "42/exe"))
os.Symlink("/usr/bin/other", filepath.Join(proc, "42/exe"))
case "wrong-parent":
write(filepath.Join(proc, "42/status"), "PPid: 45\nUid: 0 0 0 0\n")
case "namespace":
os.Remove(filepath.Join(proc, "42/ns/net"))
os.Symlink("net[other]", filepath.Join(proc, "42/ns/net"))
case "ss-dns":
write(filepath.Join(proc, "44/cmdline"), "ss-server\x00-c\x00"+dir+"/server.json\x00-d\x008.8.8.8\x00")
case "ss-remote":
write(filepath.Join(dir, "local.json"), `{"server":"8.8.8.8","server_port":8388,"local_port":1080}`)
}
err := verifyResourceDNSRuntimeAt(cfg, proc, dir, exe)
if (err == nil) != (mode == "valid") {
t.Fatalf("err=%v", err)
}
if mode != "valid" {
d := resourceCheckDeps{lookup: resourceCheckLookup, guard: func() error { return verifyResourceDNSRuntimeAt(cfg, proc, dir, exe) }, resolve: func(context.Context, string, string) ([]net.IP, error) {
t.Fatal("DNS probe before ownership")
return nil, nil
}, runner: func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Fatal("transfer before ownership")
return nil
}}
if code := resourceCheckCLI(context.Background(), resourceCheckFile(t, resourceCheckExample), io.Discard, io.Discard, d); code != 2 {
t.Fatal("ownership failure did not block", code)
}
}
})
}
}
+470
View File
@@ -0,0 +1,470 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"os"
"path/filepath"
"reflect"
"syscall"
"time"
)
// HEALTHY means only that the configured checks passed, not production READY.
type resourceMonitorSource struct {
config resourceCheckConfig
server, proxy, fingerprint string
}
func resourceStaticConfig(lookup func(string) (string, bool)) (string, string, error) {
dns, err := parseManagedDNSConfig(lookup)
raw, _ := lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
proxy, proxyErr := managedProbeProxy(raw)
if err != nil || proxyErr != nil || dns.Mode == "native" && dns.NativeServer == "" {
return "", "", errResourceConfig
}
server := dns.serverAddress()
if dns.Mode == "native" {
server = net.JoinHostPort(server, "53")
}
return server, proxy, nil
}
func readResourceMonitorSource(path string, lookup func(string) (string, bool), key []byte) (resourceMonitorSource, error) {
// Resolve each referenced environment value once in this immutable snapshot.
values := map[string]struct {
value string
ok bool
}{}
snapshot := func(name string) (string, bool) {
if v, ok := values[name]; ok {
return v.value, v.ok
}
v, ok := lookup(name)
if len(v) > 16384 {
return "", false
}
values[name] = struct {
value string
ok bool
}{v, ok}
return v, ok
}
cfg, err := readResourceChecksMode(path, snapshot, true)
server, proxy, staticErr := resourceStaticConfig(snapshot)
if err != nil || staticErr != nil {
return resourceMonitorSource{}, errResourceConfig
}
source := resourceMonitorSource{config: cfg, server: server, proxy: proxy}
b, _ := json.Marshal(cfg)
// Marshal omits the private curl headers; bind them explicitly as well.
parts := []string{string(b), server, proxy}
for _, s := range cfg.Services {
for _, r := range s.Resources {
parts = append(parts, r.privateConfig)
}
}
b, _ = json.Marshal(parts)
source.fingerprint = resourceMAC(key, b)
return source, nil
}
func resourceRowFingerprint(key []byte, source resourceMonitorSource, r resourceCheckSpec) string {
b, _ := json.Marshal(r)
b, _ = json.Marshal([]string{string(b), r.privateConfig, source.server, source.proxy})
if source.config.Version == 2 {
b, _ = json.Marshal([]string{"resource-check-v2", string(b)})
}
return resourceMAC(key, b)
}
func reconcileResourceRows(key []byte, source resourceMonitorSource, prior []resourceMonitorRow) []resourceMonitorRow {
old := map[string]resourceMonitorRow{}
for _, r := range prior {
old[r.Service+"/"+r.Resource] = r
}
rows := []resourceMonitorRow{}
for _, s := range source.config.Services {
for _, r := range s.Resources {
fp := resourceRowFingerprint(key, source, r)
row, ok := old[s.ID+"/"+r.ID]
if !ok || row.Fingerprint != fp {
row = resourceMonitorRow{Service: s.ID, Resource: r.ID, Fingerprint: fp, Health: "UNKNOWN", Outcome: "unknown"}
}
rows = append(rows, row)
}
}
return rows
}
type resourceMonitorDeps struct {
load func() (resourceMonitorState, error)
save func(resourceMonitorState) error
source func() (resourceMonitorSource, error)
guard func() error
probe func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error)
key []byte
now func() time.Time
wait func(context.Context, time.Duration) error
report func(bool)
}
// Sequential round robin with independent due times; reload at least every
// second while idle/blocked. No recovery leases or strategy mutations.
func runResourceMonitor(ctx context.Context, d resourceMonitorDeps) error {
cursor := 0
persistenceBlocked, failedMAC, attemptedMAC := false, "", ""
var shutdownError error
for ctx.Err() == nil {
state, stateErr := d.load()
if persistenceBlocked && stateErr == nil && state.MAC != failedMAC && state.MAC != attemptedMAC {
persistenceBlocked = false
}
source, sourceErr := d.source()
blocked := persistenceBlocked || stateErr != nil || sourceErr != nil || d.guard() != nil
dirty := false
if !blocked {
rows := reconcileResourceRows(d.key, source, state.Rows)
dirty = !reflect.DeepEqual(rows, state.Rows)
state.Rows = rows
if len(state.Rows) > 0 {
cursor %= len(state.Rows)
for n := 0; n < len(state.Rows); n++ {
i := (cursor + n) % len(state.Rows)
row := &state.Rows[i]
if row.NextDue > d.now().UnixMilli() {
continue
}
var spec resourceCheckSpec
for _, s := range source.config.Services {
if s.ID == row.Service {
for _, r := range s.Resources {
if r.ID == row.Resource {
spec = r
}
}
}
}
guard := func() error {
if d.guard() != nil {
return errResourceConfig
}
fresh, err := d.source()
if err != nil || fresh.fingerprint != source.fingerprint {
return errResourceConfig
}
return nil
}
outcome, err := d.probe(ctx, row.Service, spec, source, guard)
cursor = (i + 1) % len(state.Rows)
if err != nil {
if ctx.Err() == nil || !managedCancellationOnly(err, ctx.Err()) {
blocked = true
if ctx.Err() != nil {
shutdownError = err
}
}
break
}
// Cancellation joins the probe but is not completed evidence.
// Keep the previous health, failure count and due time intact.
if ctx.Err() != nil || outcome == "cancelled" {
break
}
dirty = true
row.Outcome, row.LastCompleted = outcome, d.now().UnixMilli()
delay := *spec.CooldownMS
if outcome == "pass" {
row.Health, row.Failures, delay = "HEALTHY", 0, *spec.IntervalMS
} else {
row.Health = "DEGRADED"
if row.Failures < ^uint64(0) {
row.Failures++
}
}
row.NextDue = row.LastCompleted + int64(delay)
break // persist each completed (and joined) check before another probe
}
}
}
dirty = dirty || state.Blocked != blocked
state.Blocked = blocked
if dirty {
state.Updated = d.now().UnixMilli()
}
if dirty && stateErr == nil && !persistenceBlocked {
if d.save(state) != nil {
blocked, persistenceBlocked, failedMAC, attemptedMAC = true, true, state.MAC, resourceStateMAC(d.key, state)
}
}
d.report(blocked)
if ctx.Err() != nil {
if persistenceBlocked {
return errors.Join(shutdownError, errResourceState)
}
return shutdownError
}
// Bound reloads, but don't delay an already due sibling by a cooldown.
delay := time.Second
if !blocked {
for _, r := range state.Rows {
milliseconds := r.NextDue - d.now().UnixMilli()
if milliseconds > 1000 {
milliseconds = 1000
}
remaining := time.Duration(milliseconds) * time.Millisecond
if remaining < 0 {
remaining = 0
}
if remaining < delay {
delay = remaining
}
}
}
if err := d.wait(ctx, delay); err != nil {
if ctx.Err() != nil {
if persistenceBlocked {
return errResourceState
}
return nil
}
return err
}
}
if persistenceBlocked {
return errResourceState
}
return nil
}
func probeResourceMonitor(ctx context.Context, service string, spec resourceCheckSpec, source resourceMonitorSource, guard func() error, check resourceCheckDeps) (string, error) {
check.guard = guard
protocols := []string{spec.TLS}
if spec.TLS == "both" {
protocols = []string{"tls12", "tls13"}
}
outcome := "pass"
for _, protocol := range protocols {
if ctx.Err() != nil {
return "cancelled", nil
}
result, err := runResourceProtocol(ctx, service, spec, protocol, source.server, source.proxy, check)
if err != nil {
return "", err
}
if result.Outcome != "pass" {
outcome = result.Outcome
}
}
return outcome, nil
}
func resourceMonitorCLI(ctx context.Context, path string, stdout, stderr io.Writer, target managedDeps) (code int) {
return resourceMonitorCLIWithCheck(ctx, path, stdout, stderr, target, productionResourceCheckDeps())
}
func resourceMonitorCLIWithCheck(ctx context.Context, path string, stdout, stderr io.Writer, target managedDeps, check resourceCheckDeps) (code int) {
port, settingsErr := managedTargetSettings(target.lookup)
if settingsErr != nil || port != 1080 {
fmt.Fprintln(stderr, "configerror: invalid generic target settings")
return 64
}
if _, _, err := resourceStaticConfig(target.lookup); err != nil {
fmt.Fprintln(stderr, "configerror: invalid generic DNS/proxy configuration")
return 64
}
var lock *managedMonitorSingleton
var store *resourceMonitorStore
defer func() {
if store != nil && store.close() != nil {
code = 78
}
if lock != nil && lock.close() != nil {
code = 78
}
}()
lastBlocked := false
report := func(blocked bool) {
if blocked && !lastBlocked {
fmt.Fprintln(stderr, "BLOCKED: generic resource monitor guard/state unavailable")
}
lastBlocked = blocked
}
fmt.Fprintln(stdout, "generic resource monitor: configured checks only; no recovery/apply; HEALTHY is not production READY")
// Guard failures before store/lock acquisition remain cancellable and alive.
for ctx.Err() == nil {
if lock == nil {
var err error
lock, err = acquireManagedMonitorSingleton(target)
if err != nil {
if errors.Is(err, syscall.EWOULDBLOCK) {
fmt.Fprintln(stderr, "BLOCKED: monitor singleton already held")
return 78
}
report(true)
if waitManagedMonitor(ctx, time.Second) != nil {
return 0
}
continue
}
}
if store == nil {
var err error
store, err = openResourceMonitorStore(target.directory, true)
if err != nil {
report(true)
if waitManagedMonitor(ctx, time.Second) != nil {
return 0
}
continue
}
start, err := readMonitorProcessStarttime(os.Getpid())
state, loadErr := store.load()
if err != nil || loadErr != nil {
fmt.Fprintln(stderr, "BLOCKED: generic monitor ownership unavailable")
return 78
}
state.Controller = &resourceMonitorOwner{PID: os.Getpid(), Start: start}
if store.save(state) != nil {
fmt.Fprintln(stderr, "BLOCKED: generic monitor ownership publication failed")
return 78
}
}
err := runResourceMonitor(ctx, resourceMonitorDeps{
load: store.load, save: store.save, key: store.key,
source: func() (resourceMonitorSource, error) {
return readResourceMonitorSource(path, target.lookup, store.key)
},
guard: func() error {
if resourceSingletonGuard(lock, target.directory) != nil {
return errResourceState
}
// Probe guards validate the held snapshot, never reload/adopt a new
// authenticated operator edit while an older check is in flight.
if store.snapshotGuard() != nil {
return errResourceState
}
if resourceManagedIdentity(target.directory) != nil {
return errResourceState
}
if resourceDeploymentIdentity(path, target) != nil {
return errResourceState
}
return check.guard()
},
probe: func(ctx context.Context, service string, spec resourceCheckSpec, source resourceMonitorSource, guard func() error) (string, error) {
return probeResourceMonitor(ctx, service, spec, source, guard, check)
},
now: time.Now, wait: waitManagedMonitor, report: report,
})
if err != nil {
fmt.Fprintln(stderr, "BLOCKED: generic monitor safety/finalization failure")
return 78
}
}
return 0
}
func resourceSingletonGuard(lock *managedMonitorSingleton, dir string) error {
path := filepath.Join(dir, managedMonitorLockName)
if managedPrivateFile(path) != nil {
return errResourceState
}
current, err := os.Lstat(path)
held, heldErr := lock.file.Stat()
if err != nil || heldErr != nil || !os.SameFile(current, held) {
return errResourceState
}
return nil
}
func resourceStatusCLI(stdout io.Writer, target managedDeps) int {
return resourceStatusCLIAt(stdout, target, "/proc")
}
func resourceStatusCLIAt(stdout io.Writer, target managedDeps, proc string) int {
blocked := func() int { fmt.Fprintln(stdout, `{"health":"BLOCKED","stale":true}`); return 1 }
store, err := openResourceMonitorStore(target.directory, false)
if err != nil {
return blocked()
}
defer store.close()
state, err := store.load()
path, _ := target.lookup("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
if err != nil || resourceMonitorLiveOwner(store, state, path, proc) != nil {
return blocked()
}
source, err := readResourceMonitorSource(path, target.lookup, store.key)
check := productionResourceCheckDeps()
if err != nil || check.guard() != nil || resourceManagedIdentity(target.directory) != nil {
return blocked()
}
if resourceDeploymentIdentity(path, target) != nil {
return blocked()
}
rows := reconcileResourceRows(store.key, source, state.Rows)
b1, _ := json.Marshal(rows)
b2, _ := json.Marshal(state.Rows)
stale := resourceSnapshotStale(state, time.Now().UnixMilli()) || string(b1) != string(b2)
health := "CHECKS_ONLY"
if stale {
health = "STALE"
}
if state.Blocked {
health = "BLOCKED"
}
// Source/runtime validation may take time; don't emit a cache whose live
// ownership disappeared or whose inode changed during those checks.
if resourceMonitorLiveOwner(store, state, path, proc) != nil {
return blocked()
}
if json.NewEncoder(stdout).Encode(struct {
Health string `json:"health"`
Stale bool `json:"stale"`
State resourceMonitorState `json:"state"`
}{health, stale, state}) != nil {
return 1
}
if stale || state.Blocked {
return 1
}
return 0
}
// Updated is a semantic-change timestamp, not a disk heartbeat. Evidence is
// stale only once its next scheduled check plus the maximum run budget is past.
func resourceSnapshotStale(state resourceMonitorState, now int64) bool {
if state.Updated > now {
return true
}
if len(state.Rows) == 0 {
return now-state.Updated > 65000
}
for _, row := range state.Rows {
if row.NextDue == 0 {
if now-state.Updated > 65000 {
return true
}
} else if now > row.NextDue && now-row.NextDue > 65000 {
return true
}
}
return false
}
func resourceManagedIdentity(dir string) error {
m := &managedTransaction{Dir: dir, Name: "config"}
if managedCheckIdentity(m) != nil {
return errResourceState
}
j, err := m.load()
if err != nil || j.State != "committed" {
return errResourceState
}
b, err := m.read("config")
if err != nil || managedHash(b) != j.Hash {
return errResourceState
}
return nil
}
@@ -0,0 +1,326 @@
package main
import (
"bytes"
"context"
"errors"
"io"
"net"
"os"
"path/filepath"
"reflect"
"testing"
"time"
)
func resourceCompletedFixture(t *testing.T) (resourceMonitorSource, []byte, resourceMonitorState) {
t.Helper()
source, key := resourceMonitorFixture(t)
state := resourceMonitorState{Version: 1, Updated: 5000, Rows: reconcileResourceRows(key, source, nil)}
for i := range state.Rows {
state.Rows[i].Health, state.Rows[i].Outcome, state.Rows[i].LastCompleted = "HEALTHY", "pass", 5000
state.Rows[i].NextDue = 9000
}
return source, key, state
}
func TestResourceMonitorCancelledRowsUnchangedAndJoined(t *testing.T) {
for _, mixed := range []bool{false, true} {
t.Run(map[bool]string{false: "curl-cancel", true: "cancel-and-guard"}[mixed], func(t *testing.T) {
source, key, state := resourceCompletedFixture(t)
prior := append([]resourceMonitorRow(nil), state.Rows...)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
joined, writes := false, 0
check := resourceCheckDeps{resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(ctx context.Context, _ []string, _ io.Reader, _ io.Writer, _ io.Writer) error {
cancel()
<-ctx.Done()
joined = true
return ctx.Err()
}}
d := resourceMonitorDeps{key: key, now: func() time.Time { return time.UnixMilli(10000) }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(s resourceMonitorState) error {
if !joined {
t.Error("persisted before joining curl")
}
writes++
state = s
return nil
}, source: func() (resourceMonitorSource, error) { return source, nil }, guard: func() error { return nil }, report: func(bool) {}, wait: func(context.Context, time.Duration) error { t.Error("wait after cancellation"); return nil }}
d.probe = func(ctx context.Context, service string, spec resourceCheckSpec, source resourceMonitorSource, guard func() error) (string, error) {
outcome, err := probeResourceMonitor(ctx, service, spec, source, guard, check)
if mixed {
return outcome, errors.Join(ctx.Err(), errResourceConfig)
}
return outcome, err
}
err := runResourceMonitor(ctx, d)
if mixed {
if !errors.Is(err, errResourceConfig) || !errors.Is(err, context.Canceled) || !state.Blocked || writes != 1 {
t.Fatal("lost mixed shutdown guard error", state, writes, err)
}
} else if err != nil || writes != 0 {
t.Fatal("cancel published evidence", writes, err)
}
if !joined || !reflect.DeepEqual(prior, state.Rows) {
t.Fatal("cancel changed completed rows or failed to join", state.Rows)
}
})
}
}
func TestResourceMonitorIdleNoPersistenceHeartbeat(t *testing.T) {
source, key, state := resourceCompletedFixture(t)
for i := range state.Rows {
state.Rows[i].NextDue = 305000
}
now := time.UnixMilli(10000)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
writes, waits := 0, 0
d := resourceMonitorDeps{key: key, now: func() time.Time { return now }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(s resourceMonitorState) error { writes++; state = s; return nil }, source: func() (resourceMonitorSource, error) { return source, nil }, guard: func() error { return nil }, report: func(bool) {}, probe: func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error) {
t.Error("probed resource before due")
return "pass", nil
}, wait: func(_ context.Context, d time.Duration) error {
now = now.Add(d)
waits++
if waits == 5 {
cancel()
}
return nil
}}
if err := runResourceMonitor(ctx, d); err != nil {
t.Fatal(err)
}
if writes != 0 || state.Updated != 5000 {
t.Fatal("idle disk heartbeat", writes, state.Updated)
}
if resourceSnapshotStale(state, 100000) || !resourceSnapshotStale(state, 370001) {
t.Fatal("staleness still depends on idle heartbeat")
}
}
func TestResourceMonitorBlockedTransitionsOnly(t *testing.T) {
source, key, state := resourceCompletedFixture(t)
for i := range state.Rows {
state.Rows[i].NextDue = 305000
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
tick, writes := 0, 0
d := resourceMonitorDeps{key: key, now: func() time.Time { return time.UnixMilli(10000 + int64(tick)*1000) }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(s resourceMonitorState) error { writes++; state = s; return nil }, source: func() (resourceMonitorSource, error) {
if tick < 3 {
return resourceMonitorSource{}, errResourceConfig
}
return source, nil
}, guard: func() error { return nil }, report: func(bool) {}, probe: func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error) {
t.Error("unexpected probe")
return "pass", nil
}, wait: func(context.Context, time.Duration) error {
tick++
if tick == 5 {
cancel()
}
return nil
}}
if err := runResourceMonitor(ctx, d); err != nil {
t.Fatal(err)
}
if writes != 2 || state.Blocked {
t.Fatal("blocked transition heartbeat or missing resume", writes, state)
}
}
func resourceBoundaryStore(t *testing.T, createState bool) (*resourceMonitorStore, resourceMonitorState) {
t.Helper()
dir := t.TempDir()
if err := os.Chmod(dir, 0700); err != nil {
t.Fatal(err)
}
s, err := openResourceMonitorStore(dir, true)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { s.close() })
v, err := s.load()
if err != nil {
t.Fatal(err)
}
if createState {
if err := s.save(v); err != nil {
t.Fatal(err)
}
}
return s, v
}
func replaceResourceEntry(t *testing.T, path string, data []byte) {
t.Helper()
if err := os.WriteFile(path+"-replacement", data, 0600); err != nil {
t.Fatal(err)
}
if err := os.Rename(path+"-replacement", path); err != nil {
t.Fatal(err)
}
}
func TestResourceMonitorHeldEntriesRejectReplacement(t *testing.T) {
for _, name := range []string{resourceStateName, resourceKeyName} {
for _, point := range []string{"before-load", "loaded", "staged", "publish"} {
t.Run(name+"/"+point, func(t *testing.T) {
s, v := resourceBoundaryStore(t, true)
path := filepath.Join(s.path, name)
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var replacement os.FileInfo
mutate := func() {
replaceResourceEntry(t, path, data)
replacement, err = os.Stat(path)
if err != nil {
t.Fatal(err)
}
}
if point == "before-load" {
mutate()
} else {
s.fault = func(p, _ string) {
if p == point {
mutate()
}
}
}
if point == "before-load" || point == "loaded" {
if _, err := s.load(); err == nil {
t.Fatal("load adopted identical replaced inode")
}
}
if err := s.save(v); err == nil {
t.Fatal("save clobbered identical replacement")
}
current, err := os.Stat(path)
if err != nil || replacement == nil || !os.SameFile(current, replacement) {
t.Fatal("foreign inode changed", err)
}
got, err := os.ReadFile(path)
if err != nil || !bytes.Equal(got, data) {
t.Fatal("foreign bytes changed", err)
}
})
}
}
}
func TestResourceMonitorMidStageCorruptionPreserved(t *testing.T) {
for _, name := range []string{resourceStateName, resourceKeyName, "stage", "stage-in-place"} {
t.Run(name, func(t *testing.T) {
s, v := resourceBoundaryStore(t, true)
bad := []byte("untrusted corrupt operator bytes")
path := ""
s.fault = func(point, stage string) {
if point != "staged" {
return
}
path = filepath.Join(s.path, name)
if name == "stage" || name == "stage-in-place" {
path = stage
}
if name == "stage-in-place" {
if err := os.WriteFile(path, bad, 0600); err != nil {
t.Fatal(err)
}
} else {
replaceResourceEntry(t, path, bad)
}
}
if err := s.save(v); err == nil {
t.Fatal("corruption overwritten")
}
got, err := os.ReadFile(path)
if err != nil || !bytes.Equal(got, bad) {
t.Fatal("unknown stage/file removed or clobbered", err)
}
})
}
}
func TestResourceMonitorInitialPublicationExclusive(t *testing.T) {
s, v := resourceBoundaryStore(t, false)
path := filepath.Join(s.path, resourceStateName)
bad := []byte("new foreign entry created just before initial publication")
s.fault = func(point, _ string) {
if point == "publish" {
if err := os.WriteFile(path, bad, 0600); err != nil {
t.Fatal(err)
}
}
}
if err := s.save(v); err == nil {
t.Fatal("exclusive initial publish overwrote raced entry")
}
got, err := os.ReadFile(path)
if err != nil || !bytes.Equal(got, bad) {
t.Fatal("new entry clobbered", err)
}
if _, err := s.load(); err == nil {
t.Fatal("new entry silently adopted")
}
}
func TestResourceMonitorPublicationRetainsNewAnchor(t *testing.T) {
s, v := resourceBoundaryStore(t, true)
old := s.stateFile
v.Updated = 10000
if err := s.save(v); err != nil {
t.Fatal(err)
}
if old == s.stateFile {
t.Fatal("old inode retained instead of published inode")
}
if _, err := old.Stat(); err == nil {
t.Fatal("old anchor not closed after verified publication")
}
if s.entryMatches(resourceStateName, s.stateFile) != nil {
t.Fatal("published state not anchored")
}
if loaded, err := s.load(); err != nil || loaded.Updated != 10000 {
t.Fatal(loaded, err)
}
}
func TestResourceMonitorOriginalIdentityRepair(t *testing.T) {
s, v := resourceBoundaryStore(t, true)
path := filepath.Join(s.path, resourceStateName)
valid, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
original, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(`{"version":99}`), 0600); err != nil {
t.Fatal(err)
}
if _, err := s.load(); err == nil {
t.Fatal("corrupt original trusted")
}
if err := s.save(v); err == nil {
t.Fatal("corrupt original overwritten")
}
if err := os.WriteFile(path, valid, 0600); err != nil {
t.Fatal(err)
}
if _, err := s.load(); err != nil {
t.Fatal("valid original-identity repair blocked", err)
}
current, err := os.Stat(path)
if err != nil || !os.SameFile(current, original) {
t.Fatal("repair changed inode", err)
}
if err := s.save(v); err != nil {
t.Fatal("repaired original failed publish", err)
}
}
@@ -0,0 +1,159 @@
package main
import (
"bytes"
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
)
func TestResourceMonitorFinalPublicationStageGuard(t *testing.T) {
for _, existing := range []bool{false, true} {
for _, mutation := range []string{"byte", "truncate", "append", "growth", "path-replace", "key-replace", "key-in-place", "state-replace"} {
name := "initial/"
if existing {
name = "existing/"
}
t.Run(name+mutation, func(t *testing.T) {
s, v := resourceBoundaryStore(t, existing)
path := filepath.Join(s.path, resourceStateName)
oldBytes, oldErr := os.ReadFile(path)
oldInfo, _ := os.Stat(path)
if existing && oldErr != nil {
t.Fatal(oldErr)
}
v.Updated = 10000
var changedPath string
var changedBytes []byte
var changedInfo os.FileInfo
s.fault = func(point, stage string) {
if point != "publish" {
return
}
data, err := os.ReadFile(stage)
if err != nil {
t.Fatal(err)
}
changedPath = stage
switch mutation {
case "byte":
data[0] = '['
case "truncate":
data = data[:len(data)-1]
case "append":
data = append(data, ' ')
case "growth":
data = append(data, bytes.Repeat([]byte{'x'}, 65537)...)
case "key-replace", "key-in-place":
changedPath = filepath.Join(s.path, resourceKeyName)
data, err = os.ReadFile(changedPath)
if err != nil {
t.Fatal(err)
}
if mutation == "key-in-place" {
data[0] ^= 1
}
case "state-replace":
changedPath = path
if existing {
data = append([]byte(nil), oldBytes...)
}
}
changedBytes = append([]byte(nil), data...)
if mutation == "path-replace" || mutation == "key-replace" || mutation == "state-replace" {
replaceResourceEntry(t, changedPath, data)
} else if err := os.WriteFile(changedPath, data, 0600); err != nil {
t.Fatal(err)
}
changedInfo, err = os.Stat(changedPath)
if err != nil {
t.Fatal(err)
}
}
if err := s.save(v); !errors.Is(err, errResourceState) {
t.Fatal("unsafe stage published", err)
}
current, err := os.ReadFile(changedPath)
info, infoErr := os.Stat(changedPath)
if err != nil || infoErr != nil || !bytes.Equal(current, changedBytes) || !os.SameFile(info, changedInfo) {
t.Fatal("unknown artifact changed/deleted", err, infoErr)
}
if mutation != "state-replace" {
if existing {
current, err := os.ReadFile(path)
info, infoErr := os.Stat(path)
if err != nil || infoErr != nil || !bytes.Equal(current, oldBytes) || !os.SameFile(info, oldInfo) {
t.Fatal("trusted old state lost before final guard", err, infoErr)
}
} else if _, err := os.Lstat(path); !errors.Is(err, os.ErrNotExist) {
t.Fatal("initial corrupt state published", err)
}
}
if existing && s.entryMatches(resourceStateName, s.stateFile) != nil && mutation != "state-replace" {
t.Fatal("old trusted anchor lost")
}
})
}
}
}
func TestResourceMonitorPublicationFailureBlocksFurtherProbes(t *testing.T) {
source, _, _ := resourceCompletedFixture(t)
s, _ := resourceBoundaryStore(t, true)
state, err := s.load()
if err != nil {
t.Fatal(err)
}
state.Updated = 5000
state.Rows = reconcileResourceRows(s.key, source, nil)
for i := range state.Rows {
state.Rows[i].Health, state.Rows[i].Outcome = "HEALTHY", "pass"
state.Rows[i].LastCompleted, state.Rows[i].NextDue = 5000, 9000
}
if err := s.save(state); err != nil {
t.Fatal(err)
}
path := filepath.Join(s.path, resourceStateName)
old, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
oldInfo, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
s.fault = func(point, stage string) {
if point == "publish" {
if err := os.WriteFile(stage, []byte("unknown stage corruption"), 0600); err != nil {
t.Fatal(err)
}
}
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
probes, waits, blocked := 0, 0, false
d := resourceMonitorDeps{key: s.key, now: func() time.Time { return time.UnixMilli(10000) }, load: s.load, save: s.save, source: func() (resourceMonitorSource, error) { return source, nil }, guard: s.snapshotGuard, report: func(v bool) { blocked = v }, probe: func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error) {
probes++
return "pass", nil
}, wait: func(context.Context, time.Duration) error {
waits++
if waits == 3 {
cancel()
}
return nil
}}
if err := runResourceMonitor(ctx, d); !errors.Is(err, errResourceState) {
t.Fatal("lost unsafe publication failure", err)
}
if probes != 1 || !blocked {
t.Fatal("continued probing after failed publication", probes, blocked)
}
current, err := os.ReadFile(path)
currentInfo, infoErr := os.Stat(path)
if err != nil || infoErr != nil || !bytes.Equal(current, old) || !os.SameFile(currentInfo, oldInfo) {
t.Fatal("old completed snapshot lost", err, infoErr)
}
}
@@ -0,0 +1,63 @@
package main
import (
"os"
"path/filepath"
"strconv"
)
// A signed cache is not a live ownership claim. Inspect only procfs metadata,
// never the controller's private FD contents, and fail closed on uncertainty.
func resourceMonitorLiveOwner(store *resourceMonitorStore, state resourceMonitorState, config, proc string) error {
owner := state.Controller
if owner == nil || owner.PID <= 1 || owner.Start <= 0 || config == "" {
return errResourceState
}
base := filepath.Join(proc, strconv.Itoa(owner.PID))
identity := func() bool {
stat, err := os.ReadFile(filepath.Join(base, "stat"))
start, parseErr := parseMonitorProcessStat(owner.PID, string(stat))
if err != nil || parseErr != nil || start != owner.Start {
return false
}
exe, err := os.Readlink(filepath.Join(proc, "self/exe"))
self, selfErr := os.Stat(filepath.Join(proc, "self/exe"))
other, otherErr := os.Stat(filepath.Join(base, "exe"))
if err != nil || selfErr != nil || otherErr != nil || !os.SameFile(self, other) {
return false
}
return resourceProcessOwned(proc, owner.PID, exe, []string{exe, "managed", "monitor", "--resource-config", config}) ||
resourceProcessOwned(proc, owner.PID, exe, []string{exe, "managed", "monitor"})
}
if !identity() || store.snapshotGuard() != nil {
return errResourceState
}
// Include the singleton and directory so another target's controller cannot
// lend apparent liveness to a copied authenticated snapshot.
anchors := []string{store.path, store.entryPath(resourceKeyName), store.entryPath(resourceStateName), store.entryPath(managedMonitorLockName)}
fds, err := os.ReadDir(filepath.Join(base, "fd"))
if err != nil {
return errResourceState
}
for _, anchor := range anchors {
current, err := os.Lstat(anchor)
if err != nil {
return errResourceState
}
matched := false
for _, fd := range fds {
held, err := os.Stat(filepath.Join(base, "fd", fd.Name()))
if err == nil && os.SameFile(current, held) {
matched = true
break
}
}
if !matched {
return errResourceState
}
}
if !identity() || store.snapshotGuard() != nil || managedPrivateFile(store.entryPath(managedMonitorLockName)) != nil {
return errResourceState
}
return nil
}
@@ -0,0 +1,149 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
)
func TestResourceStatusLiveOwnership(t *testing.T) {
for _, change := range []string{"state", "key", "directory", "lock", "dead", "pid-reuse", "exe", "argv", "uid", "parent", "namespace", "missing-owner"} {
t.Run(change, func(t *testing.T) {
dir := filepath.Join(t.TempDir(), "state")
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
store, err := openResourceMonitorStore(dir, true)
if err != nil {
t.Fatal(err)
}
defer store.close()
write := func(path, contents string) {
t.Helper()
if err := os.WriteFile(path, []byte(contents), 0600); err != nil {
t.Fatal(err)
}
}
link := func(path, target string) {
t.Helper()
if err := os.Symlink(target, path); err != nil {
t.Fatal(err)
}
}
lockPath := filepath.Join(dir, managedMonitorLockName)
write(lockPath, "")
lock, err := os.Open(lockPath)
if err != nil {
t.Fatal(err)
}
defer lock.Close()
source, _ := resourceMonitorFixture(t)
state := resourceMonitorState{Version: 1, Updated: time.Now().UnixMilli(), Controller: &resourceMonitorOwner{PID: 42, Start: 123}, Rows: reconcileResourceRows(store.key, source, nil)}
for i := range state.Rows {
state.Rows[i].Health, state.Rows[i].Outcome = "HEALTHY", "pass"
state.Rows[i].LastCompleted, state.Rows[i].NextDue = state.Updated, state.Updated+60000
}
if err := store.save(state); err != nil {
t.Fatal(err)
}
state, err = store.load()
if err != nil {
t.Fatal(err)
}
proc := t.TempDir()
for _, sub := range []string{"self/ns", "42/ns", "42/fd"} {
if err := os.MkdirAll(filepath.Join(proc, sub), 0700); err != nil {
t.Fatal(err)
}
}
exe, err := os.Executable()
if err != nil {
t.Fatal(err)
}
link(filepath.Join(proc, "self/exe"), exe)
link(filepath.Join(proc, "42/exe"), exe)
for _, ns := range []string{"pid", "net"} {
link(filepath.Join(proc, "self/ns", ns), ns+"[1]")
link(filepath.Join(proc, "42/ns", ns), ns+"[1]")
}
fields := append([]string{"S"}, strings.Fields(strings.Repeat("0 ", 18))...)
fields = append(fields, "123")
stat := "42 (monitor) " + strings.Join(fields, " ")
write(filepath.Join(proc, "42/stat"), stat)
uid := strconv.Itoa(os.Geteuid())
write(filepath.Join(proc, "42/status"), "PPid:\t1\nUid:\t"+strings.Join([]string{uid, uid, uid, uid}, "\t")+"\n")
config := "/private/checks.json"
write(filepath.Join(proc, "42/cmdline"), strings.Join([]string{exe, "managed", "monitor", "--resource-config", config}, "\x00")+"\x00")
for i, f := range []*os.File{store.dir, store.keyFile, store.stateFile, lock} {
link(filepath.Join(proc, "42/fd", strconv.Itoa(i)), "/proc/self/fd/"+strconv.FormatUint(uint64(f.Fd()), 10))
}
if err := resourceMonitorLiveOwner(store, state, config, proc); err != nil {
t.Fatal("valid live owner refused", err)
}
if resourceSnapshotStale(state, time.Now().UnixMilli()) {
t.Fatal("fixture already stale")
}
var preservedPath string
var preserved []byte
switch change {
case "state", "key", "lock":
name := map[string]string{"state": resourceStateName, "key": resourceKeyName, "lock": managedMonitorLockName}[change]
preservedPath = filepath.Join(dir, name)
preserved, err = os.ReadFile(preservedPath)
if err != nil {
t.Fatal(err)
}
write(preservedPath+".replacement", string(preserved))
if err := os.Rename(preservedPath+".replacement", preservedPath); err != nil {
t.Fatal(err)
}
case "directory":
if err := os.Rename(dir, dir+".old"); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
case "dead":
os.Remove(filepath.Join(proc, "42/stat"))
case "pid-reuse":
write(filepath.Join(proc, "42/stat"), strings.TrimSuffix(stat, "123")+"124")
case "exe":
os.Remove(filepath.Join(proc, "42/exe"))
link(filepath.Join(proc, "42/exe"), "/bin/sh")
case "argv":
write(filepath.Join(proc, "42/cmdline"), exe+"\x00managed\x00resource-status\x00")
case "uid":
write(filepath.Join(proc, "42/status"), "PPid:\t1\nUid:\t9999\t9999\t9999\t9999\n")
case "parent":
write(filepath.Join(proc, "42/status"), "PPid:\t2\nUid:\t"+strings.Join([]string{uid, uid, uid, uid}, "\t")+"\n")
case "namespace":
os.Remove(filepath.Join(proc, "42/ns/net"))
link(filepath.Join(proc, "42/ns/net"), "net[2]")
case "missing-owner":
state.Controller = nil
if err := store.save(state); err != nil {
t.Fatal(err)
}
}
if resourceMonitorLiveOwner(store, state, config, proc) == nil {
t.Fatal("invalid live ownership accepted")
}
target := managedDeps{directory: dir, lookup: func(name string) (string, bool) { return config, name == "ZAPRET_AUTOTUNE_RESOURCE_CONFIG" }}
var out bytes.Buffer
if resourceStatusCLIAt(&out, target, proc) != 1 || out.String() != "{\"health\":\"BLOCKED\",\"stale\":true}\n" {
t.Fatal("cached status hid live failure", out.String())
}
if preservedPath != "" {
got, err := os.ReadFile(preservedPath)
if err != nil || !bytes.Equal(got, preserved) {
t.Fatal("foreign entry modified", err)
}
}
})
}
}
@@ -0,0 +1,408 @@
package main
import (
"bytes"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"io"
"os"
"path/filepath"
"strconv"
"syscall"
)
const resourceStateName = ".resource-monitor-state.json"
const resourceKeyName = ".resource-monitor-key"
var errResourceState = errors.New("resource state blocked")
type resourceMonitorRow struct {
Service string `json:"service"`
Resource string `json:"resource"`
Fingerprint string `json:"fingerprint"`
Health string `json:"health"`
Outcome string `json:"outcome"`
Failures uint64 `json:"failures"`
LastCompleted int64 `json:"last_completed_ms"`
NextDue int64 `json:"next_due_ms"`
}
type resourceMonitorState struct {
Version int `json:"version"`
Updated int64 `json:"updated_ms"`
Blocked bool `json:"blocked"`
Rows []resourceMonitorRow `json:"rows"`
Controller *resourceMonitorOwner `json:"controller,omitempty"`
MAC string `json:"mac"`
}
type resourceMonitorOwner struct {
PID int `json:"pid"`
Start int64 `json:"starttime"`
}
type resourceMonitorStore struct {
dir *os.File
path string
key []byte
previous []byte
existed bool
keyFile, stateFile *os.File
// Deterministic filesystem interleavings in tests; nil in production.
fault func(point, stage string)
}
func resourceMAC(key, data []byte) string {
m := hmac.New(sha256.New, key)
m.Write(data)
return hex.EncodeToString(m.Sum(nil))
}
func resourceStateMAC(key []byte, s resourceMonitorState) string {
s.MAC = ""
b, _ := json.Marshal(s)
return resourceMAC(key, b)
}
func (s *resourceMonitorStore) guard() error {
if managedDirectory(s.path) != nil {
return errResourceState
}
current, err := os.Stat(s.path)
held, heldErr := s.dir.Stat()
if err != nil || heldErr != nil || !os.SameFile(current, held) {
return errResourceState
}
return nil
}
func (s *resourceMonitorStore) entryPath(name string) string {
return filepath.Join("/proc/self/fd", strconv.FormatUint(uint64(s.dir.Fd()), 10), name)
}
func (s *resourceMonitorStore) close() error {
var errs []error
for _, f := range []*os.File{s.stateFile, s.keyFile, s.dir} {
if f != nil {
errs = append(errs, f.Close())
}
}
s.stateFile, s.keyFile, s.dir = nil, nil, nil
return errors.Join(errs...)
}
func resourcePrivateEntry(f *os.File) error {
info, err := f.Stat()
if err != nil {
return errResourceState
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok || !info.Mode().IsRegular() || info.Mode().Perm() != 0600 || st.Uid != uint32(os.Geteuid()) || st.Nlink != 1 || info.Size() > 65536 {
return errResourceState
}
return nil
}
func (s *resourceMonitorStore) entryMatches(name string, held *os.File) error {
current, err := os.Lstat(s.entryPath(name))
anchored, heldErr := held.Stat()
if err != nil || heldErr != nil || !os.SameFile(current, anchored) || resourcePrivateEntry(held) != nil {
return errResourceState
}
return nil
}
func (s *resourceMonitorStore) openEntry(name string) (*os.File, []byte, error) {
if s.guard() != nil {
return nil, nil, errResourceState
}
fd, err := syscall.Openat(int(s.dir.Fd()), name, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, nil, err
}
f := os.NewFile(uintptr(fd), name)
if resourcePrivateEntry(f) != nil {
f.Close()
return nil, nil, errResourceState
}
b, err := io.ReadAll(io.LimitReader(f, 65537))
if err != nil || len(b) > 65536 || s.entryMatches(name, f) != nil {
f.Close()
return nil, nil, errResourceState
}
return f, b, nil
}
func (s *resourceMonitorStore) trustedRead(name string, held *os.File) ([]byte, error) {
f, b, err := s.openEntry(name)
if err != nil {
return nil, err
}
defer f.Close()
if held == nil || s.entryMatches(name, held) != nil {
return nil, errResourceState
}
readInfo, err := f.Stat()
heldInfo, heldErr := held.Stat()
if err != nil || heldErr != nil || !os.SameFile(readInfo, heldInfo) {
return nil, errResourceState
}
return b, nil
}
// Both content and held inode identities must still match the trusted snapshot.
// In particular, an identical signed replacement is not silently adopted.
func (s *resourceMonitorStore) snapshotGuard() error {
key, err := s.trustedRead(resourceKeyName, s.keyFile)
if err != nil || !hmac.Equal(key, s.key) {
return errResourceState
}
if s.existed {
b, err := s.trustedRead(resourceStateName, s.stateFile)
if err != nil || !bytes.Equal(b, s.previous) {
return errResourceState
}
if _, err := s.decode(b); err != nil {
return errResourceState
}
} else {
f, _, err := s.openEntry(resourceStateName)
if f != nil {
f.Close()
}
if !errors.Is(err, os.ErrNotExist) {
return errResourceState
}
}
return s.guard()
}
func openResourceMonitorStore(path string, create bool) (*resourceMonitorStore, error) {
if managedDirectory(path) != nil {
return nil, errResourceState
}
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, errResourceState
}
s := &resourceMonitorStore{path: path, dir: os.NewFile(uintptr(fd), "resource state directory")}
fail := func() (*resourceMonitorStore, error) { s.close(); return nil, errResourceState }
stateFile, state, stateErr := s.openEntry(resourceStateName)
s.stateFile = stateFile
if stateErr != nil && !errors.Is(stateErr, os.ErrNotExist) {
return fail()
}
keyFile, key, err := s.openEntry(resourceKeyName)
s.keyFile = keyFile
if errors.Is(err, os.ErrNotExist) && errors.Is(stateErr, os.ErrNotExist) && create {
key = make([]byte, 32)
if _, err = rand.Read(key); err != nil {
return fail()
}
keyFD, err := syscall.Openat(fd, resourceKeyName, syscall.O_RDWR|syscall.O_CREAT|syscall.O_EXCL|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0600)
if err != nil {
return fail()
}
f := os.NewFile(uintptr(keyFD), "resource key")
s.keyFile = f
_, writeErr := f.Write(key)
syncErr := f.Sync()
if errors.Join(writeErr, syncErr, s.dir.Sync()) != nil {
return fail()
}
} else if err != nil {
return fail()
}
if len(key) != 32 {
return fail()
}
s.key = append([]byte(nil), key...)
s.previous, s.existed = state, stateErr == nil
if s.existed {
if _, err := s.decode(state); err != nil {
return fail()
}
}
if s.snapshotGuard() != nil {
return fail()
}
return s, nil
}
func (s *resourceMonitorStore) decode(b []byte) (resourceMonitorState, error) {
var v resourceMonitorState
d := json.NewDecoder(bytes.NewReader(b))
if uniqueResourceJSON(d, 0) != nil {
return v, errResourceState
}
if _, err := d.Token(); err != io.EOF {
return v, errResourceState
}
d = json.NewDecoder(bytes.NewReader(b))
d.DisallowUnknownFields()
if d.Decode(&v) != nil || v.Version != 1 || v.Updated < 0 || len(v.Rows) > 64 || !hmac.Equal([]byte(v.MAC), []byte(resourceStateMAC(s.key, v))) {
return v, errResourceState
}
seen := map[string]bool{}
services := map[string]bool{}
for _, r := range v.Rows {
id := r.Service + "/" + r.Resource
fp, err := hex.DecodeString(r.Fingerprint)
if !resourceID.MatchString(r.Service) || !resourceID.MatchString(r.Resource) || seen[id] || err != nil || len(fp) != 32 || r.LastCompleted < 0 || r.NextDue < 0 {
return v, errResourceState
}
seen[id] = true
services[r.Service] = true
if len(services) > 16 || r.LastCompleted > v.Updated || r.LastCompleted > 253402300799999 || (r.Health == "UNKNOWN" && r.NextDue != 0) || (r.Health != "UNKNOWN" && (r.NextDue < r.LastCompleted || r.NextDue-r.LastCompleted > 86400000)) {
return v, errResourceState
}
if r.Health != "UNKNOWN" && r.Health != "HEALTHY" && r.Health != "DEGRADED" {
return v, errResourceState
}
switch r.Outcome {
case "unknown", "pass", "dns_failed", "destination_denied", "transfer_failed", "expectation_failed", "timeout", "cancelled":
default:
return v, errResourceState
}
if (r.Health == "UNKNOWN" && (r.Outcome != "unknown" || r.LastCompleted != 0 || r.Failures != 0)) || (r.Health == "HEALTHY" && (r.Outcome != "pass" || r.Failures != 0)) || (r.Health == "DEGRADED" && (r.Outcome == "unknown" || r.Outcome == "pass" || r.Failures == 0)) {
return v, errResourceState
}
}
return v, nil
}
func (s *resourceMonitorStore) load() (resourceMonitorState, error) {
if s.guard() != nil {
return resourceMonitorState{}, errResourceState
}
key, err := s.trustedRead(resourceKeyName, s.keyFile)
if err != nil || !hmac.Equal(key, s.key) {
return resourceMonitorState{}, errResourceState
}
b, err := s.trustedRead(resourceStateName, s.stateFile)
if errors.Is(err, os.ErrNotExist) && !s.existed {
return resourceMonitorState{Version: 1, Rows: []resourceMonitorRow{}}, nil
}
if err != nil {
return resourceMonitorState{}, errResourceState
}
v, err := s.decode(b)
if err != nil {
return v, err
}
if s.fault != nil {
s.fault("loaded", "")
}
// Authenticated operator repair on the original held inode can be reloaded;
// replaced files remain preserved/BLOCKED until deliberately repaired.
key, err = s.trustedRead(resourceKeyName, s.keyFile)
if err != nil || !hmac.Equal(key, s.key) || s.entryMatches(resourceStateName, s.stateFile) != nil {
return v, errResourceState
}
current, err := s.trustedRead(resourceStateName, s.stateFile)
if err != nil || !bytes.Equal(current, b) {
return v, errResourceState
}
s.previous = b
return v, nil
}
func (s *resourceMonitorStore) save(v resourceMonitorState) error {
if s.snapshotGuard() != nil {
return errResourceState
}
v.MAC = resourceStateMAC(s.key, v)
b, err := json.Marshal(v)
if err != nil {
return errResourceState
}
if _, err := s.decode(b); err != nil {
return errResourceState
}
// Keep the staged inode open from creation through publication; reopening a
// path after rename must never adopt a foreign replacement as our own inode.
f, err := os.CreateTemp(s.entryPath(""), ".resource-monitor-stage-")
if err != nil {
return errResourceState
}
stage := f.Name()
stageGuard := func() bool {
if s.entryMatches(filepath.Base(stage), f) != nil {
return false
}
contents, err := io.ReadAll(io.NewSectionReader(f, 0, 65537))
// Exact bytes bind the entire validated update, including its HMAC;
// truncation, append and in-place edits must fail before publication.
return err == nil && bytes.Equal(contents, b) && s.entryMatches(filepath.Base(stage), f) == nil
}
adopted := false
defer func() {
// Delete only the known staged inode, never an operator replacement.
info, err := os.Lstat(stage)
held, heldErr := f.Stat()
if err == nil && heldErr == nil && os.SameFile(info, held) && resourcePrivateEntry(f) == nil {
contents, readErr := io.ReadAll(io.NewSectionReader(f, 0, 65537))
if readErr == nil && bytes.Equal(contents, b) {
os.Remove(stage)
}
}
if !adopted {
f.Close()
}
}()
if _, err := f.Write(b); err != nil {
return errResourceState
}
if f.Sync() != nil {
return errResourceState
}
if s.fault != nil {
s.fault("staged", stage)
}
if s.snapshotGuard() != nil || !stageGuard() {
return errResourceState
}
if s.fault != nil {
s.fault("publish", stage)
}
// Detect benign/operator replacements immediately before publication. An
// actively hostile same-UID actor racing between this final check and rename
// is outside the guarantee: POSIX rename has no inode compare-and-swap.
if s.existed {
if s.snapshotGuard() != nil || !stageGuard() {
return errResourceState
}
if os.Rename(stage, s.entryPath(resourceStateName)) != nil {
return errResourceState
}
} else {
// Exclusive initial publication cannot clobber a newly appeared entry.
key, err := s.trustedRead(resourceKeyName, s.keyFile)
if err != nil || !hmac.Equal(key, s.key) || s.guard() != nil || !stageGuard() {
return errResourceState
}
if os.Link(stage, s.entryPath(resourceStateName)) != nil {
return errResourceState
}
stageInfo, stageErr := os.Lstat(stage)
heldInfo, heldErr := f.Stat()
if stageErr != nil || heldErr != nil || !os.SameFile(stageInfo, heldInfo) {
return errResourceState
}
if os.Remove(stage) != nil {
return errResourceState
}
}
if s.entryMatches(resourceStateName, f) != nil {
return errResourceState
}
published, err := s.trustedRead(resourceStateName, f)
if err != nil || !bytes.Equal(published, b) {
return errResourceState
}
old := s.stateFile
s.stateFile, s.previous, s.existed, adopted = f, b, true, true
var closeErr error
if old != nil {
closeErr = old.Close()
}
if errors.Join(closeErr, s.dir.Sync()) != nil {
return errResourceState
}
return nil
}
@@ -0,0 +1,541 @@
package main
import (
"bytes"
"context"
"errors"
"io"
"net"
"os"
"path/filepath"
"reflect"
"strings"
"syscall"
"testing"
"time"
)
func resourceMonitorFixture(t *testing.T) (resourceMonitorSource, []byte) {
t.Helper()
key := bytes.Repeat([]byte{7}, 32)
path := resourceCheckFile(t, resourceCheckExample)
s, err := readResourceMonitorSource(path, resourceCheckLookup, key)
if err != nil {
t.Fatal(err)
}
return s, key
}
func TestResourceMonitorSchedule(t *testing.T) {
source, key := resourceMonitorFixture(t)
a, b, cooldown := 1000, 4000, 200
source.config.Services[0].Resources[0].IntervalMS = &a
source.config.Services[0].Resources[0].CooldownMS = &cooldown
source.config.Services[1].Resources[0].IntervalMS = &b
state := resourceMonitorState{Version: 1}
now := time.UnixMilli(10000)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var visits []string
var elapsed []int64
d := resourceMonitorDeps{key: key, now: func() time.Time { return now }, source: func() (resourceMonitorSource, error) { return source, nil }, guard: func() error { return nil }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(v resourceMonitorState) error { state = v; return nil }, report: func(bool) {}, wait: func(_ context.Context, d time.Duration) error { now = now.Add(d); return nil }}
d.probe = func(_ context.Context, service string, _ resourceCheckSpec, _ resourceMonitorSource, guard func() error) (string, error) {
if err := guard(); err != nil {
return "", err
}
visits = append(visits, service)
elapsed = append(elapsed, now.UnixMilli()-10000)
if len(visits) == 8 {
cancel()
}
if len(visits) == 1 {
return "timeout", nil
}
return "pass", nil
}
if err := runResourceMonitor(ctx, d); err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(visits, []string{"first", "second", "first", "first", "first", "first", "second", "first"}) || !reflect.DeepEqual(elapsed, []int64{0, 0, 200, 1200, 2200, 3200, 4000, 4200}) {
t.Fatalf("unfair/due schedule: %v %v", visits, elapsed)
}
if state.Rows[0].Health != "HEALTHY" || state.Rows[0].Failures != 0 || state.Rows[1].NextDue != 18000 {
t.Fatal(state)
}
}
func TestResourceMonitorBlockedRepairAndPersistenceFailure(t *testing.T) {
for _, kind := range []string{"source", "ownership", "storage", "save"} {
t.Run(kind, func(t *testing.T) {
source, key := resourceMonitorFixture(t)
state := resourceMonitorState{Version: 1, MAC: "initial"}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
tick, probes := 0, 0
d := resourceMonitorDeps{key: key, now: func() time.Time { return time.UnixMilli(10000 + int64(tick)*1000) }, report: func(bool) {}, wait: func(_ context.Context, _ time.Duration) error {
tick++
if tick == 4 {
cancel()
}
return nil
}, source: func() (resourceMonitorSource, error) {
if kind == "source" && tick < 2 {
return resourceMonitorSource{}, errResourceConfig
}
return source, nil
}, guard: func() error {
if kind == "ownership" && tick < 2 {
return errResourceConfig
}
return nil
}, load: func() (resourceMonitorState, error) {
if kind == "storage" && tick < 2 {
return state, errResourceState
}
return state, nil
}, save: func(v resourceMonitorState) error {
if kind == "save" {
return errResourceState
}
state = v
return nil
}}
d.probe = func(_ context.Context, _ string, _ resourceCheckSpec, _ resourceMonitorSource, g func() error) (string, error) {
if err := g(); err != nil {
return "", err
}
probes++
return "pass", nil
}
if err := runResourceMonitor(ctx, d); (err != nil) != (kind == "save") {
t.Fatal(err)
}
if kind == "save" {
if probes != 1 {
t.Fatalf("probed after persistence failure: %d", probes)
}
} else if probes != 2 {
t.Fatalf("did not resume safely: %d", probes)
}
})
}
}
func TestResourceMonitorConfigReconcilePrivacy(t *testing.T) {
source, key := resourceMonitorFixture(t)
rows := reconcileResourceRows(key, source, nil)
rows[0].Health, rows[0].Outcome, rows[0].LastCompleted = "HEALTHY", "pass", 10000
rows[1].Health, rows[1].Outcome, rows[1].LastCompleted = "HEALTHY", "pass", 10000
old := rows[1].Fingerprint
source.config.Services[0].Resources[0].URL = "https://example.com/changed-secret"
changed := reconcileResourceRows(key, source, rows)
if changed[0].Health != "UNKNOWN" || changed[1].Health != "HEALTHY" || changed[1].Fingerprint != old {
t.Fatal(changed)
}
source.config.Services = source.config.Services[1:]
if got := reconcileResourceRows(key, source, changed); len(got) != 1 || got[0].Service != "second" {
t.Fatal(got)
}
}
func TestResourceMonitorStore(t *testing.T) {
source, key := resourceMonitorFixture(t)
dir := t.TempDir()
if err := os.Chmod(dir, 0700); err != nil {
t.Fatal(err)
}
s, err := openResourceMonitorStore(dir, true)
if err != nil {
t.Fatal(err)
}
key = s.key
state, err := s.load()
if err != nil || len(state.Rows) != 0 {
t.Fatal(state, err)
}
state.Updated = 10000
state.Rows = reconcileResourceRows(key, source, nil)
if err := s.save(state); err != nil {
t.Fatal(err)
}
s.close()
s, err = openResourceMonitorStore(dir, false)
if err != nil {
t.Fatal(err)
}
defer s.close()
loaded, err := s.load()
if err != nil || !reflect.DeepEqual(loaded.Rows, state.Rows) {
t.Fatal(loaded, err)
}
data, err := os.ReadFile(filepath.Join(dir, resourceStateName))
if err != nil {
t.Fatal(err)
}
for _, secret := range []string{"example.com", "private-token", "secret=value", "example.org", "Authorization"} {
if bytes.Contains(data, []byte(secret)) {
t.Fatal("secret serialized")
}
}
for _, name := range []string{resourceStateName, resourceKeyName} {
info, err := os.Stat(filepath.Join(dir, name))
if err != nil || info.Mode().Perm() != 0600 {
t.Fatal(info, err)
}
}
// Corruption is preserved; restoring a valid signed version resumes loading.
bad := []byte(`{"version":99}`)
if err := os.WriteFile(filepath.Join(dir, resourceStateName), bad, 0600); err != nil {
t.Fatal(err)
}
if _, err := s.load(); err == nil {
t.Fatal("corrupt state trusted")
}
if err := s.save(state); err == nil {
t.Fatal("corrupt state overwritten")
}
got, _ := os.ReadFile(filepath.Join(dir, resourceStateName))
if !bytes.Equal(got, bad) {
t.Fatal("evidence changed")
}
if err := os.WriteFile(filepath.Join(dir, resourceStateName), data, 0600); err != nil {
t.Fatal(err)
}
if _, err := s.load(); err != nil {
t.Fatal("valid repair refused", err)
}
if err := os.Remove(filepath.Join(dir, resourceKeyName)); err != nil {
t.Fatal(err)
}
if _, err := openResourceMonitorStore(dir, true); err == nil {
t.Fatal("missing key silently regenerated")
}
}
func TestResourceMonitorUnsafeStorePreserved(t *testing.T) {
for _, kind := range []string{"symlink", "hardlink", "permissions", "wrong-owner", "directory-replaced"} {
t.Run(kind, func(t *testing.T) {
dir := filepath.Join(t.TempDir(), "state")
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
s, err := openResourceMonitorStore(dir, true)
if err != nil {
t.Fatal(err)
}
defer s.close()
path := filepath.Join(dir, resourceKeyName)
original, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
switch kind {
case "symlink":
if err := os.Rename(path, path+"-original"); err != nil {
t.Fatal(err)
}
if err := os.Symlink(path+"-original", path); err != nil {
t.Fatal(err)
}
case "hardlink":
if err := os.Link(path, path+"-alias"); err != nil {
t.Fatal(err)
}
case "permissions":
if err := os.Chmod(path, 0644); err != nil {
t.Fatal(err)
}
case "wrong-owner":
if os.Geteuid() != 0 {
t.Skip("requires chown capability")
}
if err := os.Chown(path, 12345, -1); err != nil {
t.Fatal(err)
}
case "directory-replaced":
if err := os.Rename(dir, dir+"-old"); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
path = filepath.Join(dir+"-old", resourceKeyName)
}
if _, err := s.load(); err == nil {
t.Fatal("unsafe storage accepted")
}
if err := s.save(resourceMonitorState{Version: 1}); err == nil {
t.Fatal("unsafe storage overwritten")
}
data, err := os.ReadFile(path)
if err != nil || !bytes.Equal(data, original) {
t.Fatal("key changed", err)
}
})
}
}
func TestResourceMonitorReloadBeforeDNSAndTransfer(t *testing.T) {
for _, phase := range []string{"before-dns", "after-dns", "replace-after-dns", "env-after-dns", "owner-after-dns"} {
t.Run(phase, func(t *testing.T) {
key := bytes.Repeat([]byte{1}, 32)
path := resourceCheckFile(t, resourceCheckExample)
envChanged, ownerLost := false, false
lookup := func(k string) (string, bool) {
if envChanged && k == "CHECK_TOKEN" {
return "changed secret", true
}
return resourceCheckLookup(k)
}
source, err := readResourceMonitorSource(path, lookup, key)
if err != nil {
t.Fatal(err)
}
transfers, dnsCalls := 0, 0
guard := func() error {
if ownerLost {
return errResourceConfig
}
fresh, err := readResourceMonitorSource(path, lookup, key)
if err != nil || fresh.fingerprint != source.fingerprint {
return errResourceConfig
}
return nil
}
mutate := func() {
switch phase {
case "env-after-dns":
envChanged = true
case "owner-after-dns":
ownerLost = true
case "replace-after-dns":
data := strings.Replace(resourceCheckExample, "secret=value", "secret=rotated", 1)
if err := os.WriteFile(path+"-next", []byte(data), 0600); err != nil {
t.Fatal(err)
}
if err := os.Rename(path+"-next", path); err != nil {
t.Fatal(err)
}
default:
if err := os.WriteFile(path, []byte(`{"version":1}`), 0600); err != nil {
t.Fatal(err)
}
}
}
if phase == "before-dns" {
mutate()
}
check := resourceCheckDeps{resolve: func(context.Context, string, string) ([]net.IP, error) {
dnsCalls++
mutate()
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}, runner: func(context.Context, []string, io.Reader, io.Writer, io.Writer) error { transfers++; return nil }}
_, err = probeResourceMonitor(context.Background(), "first", source.config.Services[0].Resources[0], source, guard, check)
if err == nil || transfers != 0 || (phase == "before-dns" && dnsCalls != 0) {
t.Fatal("unsafe probe", transfers, dnsCalls, err)
}
})
}
}
func TestResourceMonitorBothTLSAndCancellationJoin(t *testing.T) {
source, _ := resourceMonitorFixture(t)
spec := source.config.Services[1].Resources[0]
check := resourceCheckDeps{resolve: func(context.Context, string, string) ([]net.IP, error) {
return []net.IP{net.ParseIP("93.184.216.34")}, nil
}}
calls := 0
check.runner = func(_ context.Context, args []string, _ io.Reader, out, diag io.Writer) error {
calls++
protocol := "tls12"
mode := ""
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
protocol = "tls13"
mode = "verify"
}
configuredWire(spec.URL, spec.Method, protocol, mode, out, diag)
return nil
}
outcome, err := probeResourceMonitor(context.Background(), "second", spec, source, func() error { return nil }, check)
if err != nil || outcome == "pass" || calls != 2 {
t.Fatal(outcome, calls, err)
}
ctx, cancel := context.WithCancel(context.Background())
joined := false
check.runner = func(ctx context.Context, _ []string, _ io.Reader, _ io.Writer, _ io.Writer) error {
cancel()
<-ctx.Done()
joined = true
return ctx.Err()
}
outcome, err = probeResourceMonitor(ctx, "second", spec, source, func() error { return nil }, check)
if err != nil || outcome != "cancelled" || !joined {
t.Fatal("did not join cancellation", outcome, err)
}
}
func TestResourceMonitorSchemaScheduling(t *testing.T) {
for _, field := range []string{"interval_ms", "cooldown_ms"} {
for _, value := range []string{"0", "-1", "99", "86400001", "null", "\"1000\""} {
data := strings.Replace(resourceCheckExample, `"timeout_ms":1000`, `"timeout_ms":1000,"`+field+`":`+value, 1)
if _, err := parseResourceChecks([]byte(data), resourceCheckLookup); err == nil {
t.Fatal(field, value)
}
}
}
}
func TestResourceMonitorSingletonAndGenericDispatch(t *testing.T) {
d, env, _ := managedCLIFixture(t)
lock, err := acquireManagedMonitorSingleton(d)
if err != nil {
t.Fatal(err)
}
defer lock.close()
if _, err := acquireManagedMonitorSingleton(d); !errors.Is(err, syscall.EWOULDBLOCK) {
t.Fatal("singleton conflict ignored", err)
}
// Generic dispatch precedes invalid YouTube/adaptive settings. A cancelled
// generic startup needs no runtime/network, database or valid source JSON.
env["SOCKS_PORT"] = "1080"
env["ZAPRET_AUTOTUNE_SOCKS_PROXY"] = "socks5h://127.0.0.1:1080"
env["ZAPRET_AUTOTUNE_RESOURCE_CONFIG"] = "/missing-invalid-config"
env["ZAPRET_AUTOTUNE_VIDEO_ID"] = "bad"
env["ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE"] = "invalid"
ctx, cancel := context.WithCancel(context.Background())
cancel()
var out, diag bytes.Buffer
code := managedMonitorCLIContext(ctx, nil, &out, &diag, d, func(context.Context, managedMonitorDeps) error { t.Fatal("YouTube supervisor invoked"); return nil })
if code != 0 || !strings.Contains(out.String(), "generic resource monitor") {
t.Fatal(code, out.String(), diag.String())
}
delete(env, "ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
if code := managedMonitorCLIContext(ctx, nil, &out, &diag, d, nil); code != 64 {
t.Fatal("legacy invalid config no longer rejected", code)
}
}
func TestResourceMonitorInvalidInitialConfigAliveAndTERM(t *testing.T) {
d, env, _ := managedCLIFixture(t)
env["SOCKS_PORT"] = "1080"
env["ZAPRET_AUTOTUNE_SOCKS_PROXY"] = "socks5h://127.0.0.1:1080"
d.verifySOCKS = func(int) error { return nil }
initManagedCLI(t, d)
path := resourceCheckFile(t, `{"invalid":true}`)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
started := make(chan struct{}, 1)
check := resourceCheckDeps{guard: func() error {
select {
case started <- struct{}{}:
default:
}
return nil
}, resolve: func(context.Context, string, string) ([]net.IP, error) {
t.Error("DNS with invalid source")
return nil, nil
}, runner: func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
t.Error("transfer with invalid source")
return nil
}}
var out, diag bytes.Buffer
done := make(chan int, 1)
go func() { done <- resourceMonitorCLIWithCheck(ctx, path, &out, &diag, d, check) }()
// Source error short-circuits the runtime guard, so wait for authenticated
// BLOCKED persistence rather than relying on a diagnostic log race.
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
s, err := openResourceMonitorStore(d.directory, false)
if err == nil {
state, err := s.load()
s.close()
if err == nil && state.Blocked {
break
}
}
time.Sleep(10 * time.Millisecond)
}
select {
case code := <-done:
t.Fatalf("invalid source terminated controller: %d", code)
default:
}
cancel()
select {
case code := <-done:
if code != 0 {
t.Fatal(code, diag.String())
}
case <-time.After(2 * time.Second):
t.Fatal("TERM did not join controller")
}
if !strings.Contains(diag.String(), "BLOCKED") {
t.Fatal("no blocked status", diag.String())
}
}
func TestResourceMonitorConcurrentEnvironmentReload(t *testing.T) {
path := resourceCheckFile(t, resourceCheckExample)
t.Setenv("CHECK_URL", "https://example.org/private")
t.Setenv("CHECK_TOKEN", "initial-secret")
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://127.0.0.1:1080")
key := bytes.Repeat([]byte{3}, 32)
baseline, err := readResourceMonitorSource(path, os.LookupEnv, key)
if err != nil {
t.Fatal(err)
}
done := make(chan struct{})
go func() {
defer close(done)
for i := 0; i < 100; i++ {
os.Setenv("CHECK_TOKEN", "rotated-secret")
os.Setenv("CHECK_TOKEN", "initial-secret")
}
}()
for i := 0; i < 100; i++ {
s, err := readResourceMonitorSource(path, os.LookupEnv, key)
if err != nil {
t.Fatal(err)
}
if s.fingerprint == "" {
t.Fatal("empty fingerprint")
}
}
<-done
if err := os.Setenv("CHECK_TOKEN", "rotated-secret"); err != nil {
t.Fatal(err)
}
fresh, err := readResourceMonitorSource(path, os.LookupEnv, key)
if err != nil || fresh.fingerprint == baseline.fingerprint {
t.Fatal("secret rotation not bound", err)
}
}
func TestResourceMonitorTargetConfigBoundary(t *testing.T) {
_, env, _ := managedCLIFixture(t)
for k, v := range env {
t.Setenv(k, v)
}
t.Setenv("SOCKS_PORT", "1080")
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://127.0.0.1:1080")
t.Setenv("ZAPRET_AUTOTUNE_RESOURCE_CONFIG", resourceCheckFile(t, `{"invalid":true}`))
t.Setenv("ZAPRET_AUTOTUNE_VIDEO_ID", "invalid")
t.Setenv("ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE", "invalid")
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 0 {
t.Fatal("bad resource JSON/YouTube blocked safe generic startup", code)
}
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://foreign.example:1080")
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 64 {
t.Fatal("bad proxy allowed", code)
}
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://127.0.0.1:1080")
t.Setenv("ZAPRET_AUTOTUNE_DNS_MODE", "invalid")
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 64 {
t.Fatal("bad DNS allowed", code)
}
t.Setenv("ZAPRET_AUTOTUNE_DNS_MODE", "doh")
t.Setenv("ZAPRET_AUTOTUNE_TARGET_ROLE", "foreign")
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 64 {
t.Fatal("bad target role allowed", code)
}
}
@@ -0,0 +1,135 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"os/exec"
"strings"
"testing"
"time"
)
func TestResourcePeerTLSCollectorBoundaries(t *testing.T) {
for _, code := range []int{7, 35, 51, 58, 60, 65} {
t.Run(fmt.Sprint(code), func(t *testing.T) {
negative := exec.Command("sh", "-c", fmt.Sprintf("exit %d", code)).Run()
for _, mixed := range []error{nil, context.Canceled, errors.New("unknown infrastructure"), &managedRecoveryFatal{Err: errors.New("guard failed")}} {
runner := func(_ context.Context, args []string, _ io.Reader, _, _ io.Writer) error {
if strings.Contains(strings.Join(args, " "), "--insecure") {
t.Fatal("certificate bypass")
}
return errors.Join(negative, mixed)
}
// Use curl's actual exit type for the unmixed production path.
if mixed == nil {
runner = func(context.Context, []string, io.Reader, io.Writer, io.Writer) error { return negative }
}
m, err := collectAssetMeasurement(context.Background(), "socks5://127.0.0.1:1080", "https://yt3.ggpht.com/fresh?private=secret", "yt3.ggpht.com", "tls12", "", runner)
if err == nil || len(m.Transfers) != 0 || strings.Contains(err.Error(), "secret") {
t.Fatal("accepted/leaked rejection", err)
}
peer := code == 35 || code == 51 || code == 60
var fatal *managedRecoveryFatal
if errors.As(err, &fatal) != (!peer || mixed != nil) {
t.Fatalf("code=%d mixed=%v fatal=%v", code, mixed, err)
}
if managedAssetTransferNonpass(err) != (peer && mixed == nil) || managedAssetHealthNonpass(err) != (peer && mixed == nil) {
t.Fatal("unsafe nonpass classification", err)
}
}
})
}
}
func TestManagedAvatarPeerTLSMonitorNoQualifiedCooldown(t *testing.T) {
for _, tc := range []struct {
code int
stage string
}{{35, "verify"}, {60, "verify"}, {35, "apply"}, {60, "apply"}} {
t.Run(fmt.Sprintf("%d/%s", tc.code, tc.stage), func(t *testing.T) {
d, front, _ := recoveryFixture(t, true)
r, events, _ := assetRuntimeFixture(t)
good := r.curl
negative := exec.Command("sh", "-c", fmt.Sprintf("exit %d", tc.code)).Run()
reject := func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
if strings.Contains(strings.Join(args, " "), "--insecure") {
t.Fatal("certificate bypass")
}
return negative
}
if tc.stage == "verify" {
r.curl = reject
}
thumbnail := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
ggc := managedCandidateFixture(t, d.db, "googlevideo", "fresh.googlevideo.com", "--lua-desync=fake", true)
ids := []int64{front, thumbnail, ggc}
target, _, _ := managedCLIFixture(t)
initManagedCLI(t, target)
d.transaction.Dir = target.directory
if err := d.transaction.Apply(d.db, ids); err != nil {
t.Fatal(err)
}
before, err := d.transaction.read("config")
if err != nil {
t.Fatal(err)
}
d.service = true
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
d.resources = func(context.Context) (managedServiceInventory, error) {
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/fresh", "yt3.ggpht.com": "https://yt3.ggpht.com/fresh"}, proxy: r.proxy, runner: reject}, nil
}
d.verifyAssets = r.verify
d.health = managedServiceHealth(r.proxy, d.frontHealth, d.resources, func(ctx context.Context, args []string, in io.Reader, out, diag io.Writer) error {
config, _ := io.ReadAll(in)
if strings.Contains(string(config), "yt3.ggpht.com") {
return negative
}
return good(ctx, args, strings.NewReader(string(config)), out, diag)
})
monitor := monitorRuntimeFixture(t)
monitor.db, monitor.target, monitor.threshold = d.db, target, 1
if err := wireManagedRecovery(&monitor, d, func(error) {}); err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
monitor.reportRecoveryError = func(err error) {
if !errors.Is(err, errNoQualifiedStrategy) {
t.Fatal("not genuine candidate exhaustion", err)
}
}
waits := 0
monitor.wait = func(ctx context.Context, _ time.Duration) error {
state, err := readMonitorState(d.db, monitor.serviceID)
if err != nil || state.RecoveryRunning || state.CooldownUntil.Before(monitor.now().Add(2*time.Hour)) {
t.Fatal("absence pause not durable", state, err)
}
waits++
if waits == 2 {
cancel()
return ctx.Err()
}
return nil
}
if err := runManagedMonitor(ctx, monitor); !errors.Is(err, context.Canceled) {
t.Fatal("peer rejection killed monitor", err)
}
joins := 0
for _, event := range *events {
if event == "join" {
joins++
}
}
if joins != len(managedAssetSeeds) {
t.Fatal("candidate worker not joined", joins, *events)
}
assertActiveCandidates(t, d.transaction, d.db, ids)
after, err := d.transaction.read("config")
if err != nil || string(before) != string(after) {
t.Fatal("peer-negative apply changed committed config", err)
}
})
}
}
+339
View File
@@ -0,0 +1,339 @@
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"io"
"sort"
"strings"
"time"
)
var errResourcePlan = errors.New("invalid private qualification data")
// Input groups are metadata. Only the HTTPS collector's selectors are supported.
type strategySelectors struct {
Transport string `json:"transport"`
Ports string `json:"ports"`
L7 string `json:"l7"`
TLS string `json:"tls"`
}
type strategyDefinition struct {
ID string `json:"id"`
Args string `json:"args"`
Selectors strategySelectors `json:"selectors"`
}
type strategyRegistry struct {
Version int `json:"version"`
Strategies []strategyDefinition `json:"strategies"`
}
func strictPlanJSON(b []byte, dst interface{}) error {
if len(b) > 65536 {
return errResourcePlan
}
d := json.NewDecoder(bytes.NewReader(b))
if uniqueResourceJSON(d, 0) != nil {
return errResourcePlan
}
if _, err := d.Token(); err != io.EOF {
return errResourcePlan
}
d = json.NewDecoder(bytes.NewReader(b))
d.DisallowUnknownFields()
if d.Decode(dst) != nil {
return errResourcePlan
}
return nil
}
func planDigest(v interface{}) string {
b, _ := json.Marshal(v)
h := sha256.Sum256(b)
return hex.EncodeToString(h[:])
}
func canonicalStrategy(d strategyDefinition) (strategyDefinition, error) {
d.Args = strings.Join(strings.Fields(d.Args), " ")
if !resourceID.MatchString(d.ID) || d.Args == "" || len(d.Args) > 4096 || (&candidate{Args: d.Args}).validateArgs() != nil {
return d, errResourcePlan
}
if d.Selectors.Transport != "tcp" || d.Selectors.Ports != "443" || d.Selectors.L7 != "tls" || (d.Selectors.TLS != "both" && d.Selectors.TLS != "tls12" && d.Selectors.TLS != "tls13") {
return d, errResourcePlan
}
for _, arg := range strings.Fields(d.Args) {
option, value, _ := strings.Cut(arg, "=")
if option == "--lua-init" {
return d, errResourcePlan
}
if option != "--lua-desync" {
continue
}
parts := strings.Split(value, ":")
switch parts[0] {
case "fake", "multisplit", "multidisorder", "fakeddisorder", "split", "disorder":
default:
return d, errResourcePlan
}
for _, part := range parts[1:] {
key, v, ok := strings.Cut(part, "=")
if !ok || v == "" || strings.ContainsAny(v, "/@%") || strings.Contains(v, "..") {
return d, errResourcePlan
}
switch key {
case "pos", "repeats", "tcp_ts", "tcp_flags_set", "seqovl":
case "blob":
if v != "fake_default_tls" {
return d, errResourcePlan
}
default:
return d, errResourcePlan
}
}
}
return d, nil
}
func parseStrategyRegistry(b []byte) (strategyRegistry, error) {
var r strategyRegistry
if strictPlanJSON(b, &r) != nil || r.Version != 1 || len(r.Strategies) == 0 || len(r.Strategies) > 32 {
return r, errResourcePlan
}
seen := map[string]bool{}
for i, d := range r.Strategies {
v, err := canonicalStrategy(d)
if err != nil || seen[v.ID] {
return r, errResourcePlan
}
seen[v.ID] = true
r.Strategies[i] = v
}
sort.Slice(r.Strategies, func(i, j int) bool { return r.Strategies[i].ID < r.Strategies[j].ID })
return r, nil
}
type qualificationCheck struct {
Service string `json:"service"`
Resource string `json:"resource"`
Fingerprint string `json:"fingerprint"`
TLS string `json:"tls"`
Outcome string `json:"outcome"`
Attempts int `json:"attempts"`
}
type qualificationRun struct {
ID string `json:"id"`
Strategy string `json:"strategy"`
Definition string `json:"definition"`
Config string `json:"config"`
Started int64 `json:"started_ms"`
Completed int64 `json:"completed_ms"`
Attestation qualificationAttestation `json:"attestation"`
Checks []qualificationCheck `json:"checks"`
}
type qualificationAttestation struct {
Collector string `json:"collector"`
Runtime string `json:"runtime"`
Definition string `json:"definition"`
LuaHash string `json:"lua_hash"`
RulesHash string `json:"rules_hash"`
WorkerArgsHash string `json:"worker_args_hash"`
Queue int `json:"queue"`
PID int `json:"pid"`
StartTime int64 `json:"starttime"`
Restored bool `json:"restored"`
Joined bool `json:"joined"`
}
type qualificationMatrix struct {
Version int `json:"version"`
Registry string `json:"registry"`
Runs []qualificationRun `json:"runs"`
}
// Only the owned producer (or internal test fixtures) constructs this capability.
// JSON imports cannot authorize writes, and post-seal DTO mutation is rejected.
type completedQualification struct {
matrix qualificationMatrix
seal string
}
func hashOK(s string) bool {
b, e := hex.DecodeString(s)
return e == nil && len(b) == 32 && strings.ToLower(s) == s
}
func validateMatrix(m qualificationMatrix, registry strategyRegistry) error {
if m.Version != 1 || m.Registry != planDigest(registry) || len(m.Runs) > 32 {
return errResourcePlan
}
defs := map[string]strategyDefinition{}
for _, d := range registry.Strategies {
defs[d.ID] = d
}
ids, strategies := map[string]bool{}, map[string]bool{}
for _, r := range m.Runs {
d, ok := defs[r.Strategy]
a := r.Attestation
if !ok || !resourceID.MatchString(r.ID) || ids[r.ID] || strategies[r.Strategy] || r.Definition != planDigest(d) || !hashOK(r.Config) || r.Started <= 0 || r.Completed < r.Started || r.Completed-r.Started > 86400000 || len(r.Checks) == 0 || len(r.Checks) > 128 {
return errResourcePlan
}
ids[r.ID], strategies[r.Strategy] = true, true
if a.Collector != "owned-resource-nfq-v1" || a.Runtime != "nfqws2-1.0.2" || a.Definition != r.Definition || !hashOK(a.LuaHash) || !hashOK(a.RulesHash) || !hashOK(a.WorkerArgsHash) || a.Queue != managedAssetQueue || a.PID <= 0 || a.StartTime <= 0 || !a.Restored || !a.Joined {
return errResourcePlan
}
seen := map[string]bool{}
for _, c := range r.Checks {
id := c.Service + "/" + c.Resource + "/" + c.TLS
if !resourceID.MatchString(c.Service) || !resourceID.MatchString(c.Resource) || !hashOK(c.Fingerprint) || seen[id] || c.Attempts != 3 || (c.TLS != "tls12" && c.TLS != "tls13") {
return errResourcePlan
}
seen[id] = true
switch c.Outcome {
case "pass", "dns_failed", "destination_denied", "transfer_failed", "expectation_failed", "timeout":
default:
return errResourcePlan
}
}
}
return nil
}
type domainCompatibility struct {
Host string `json:"host"`
Validated []string `json:"validated"`
Negative []string `json:"negative"`
Unknown []string `json:"unknown"`
}
type runtimeAssignment struct {
Strategy string `json:"strategy"`
Args string `json:"args"`
Selectors strategySelectors `json:"selectors"`
Hosts []string `json:"hosts"`
}
type resourcePlan struct {
ProposalOnly bool `json:"proposal_only"`
Policy string `json:"strategy_ids,omitempty"`
Evidence string `json:"evidence"`
Domains []domainCompatibility `json:"domains"`
Assignments []runtimeAssignment `json:"assignments"`
Unassigned []string `json:"unassigned"`
}
func deriveResourcePlan(source resourceMonitorSource, key []byte, registry strategyRegistry, matrix qualificationMatrix, now time.Time, policy ...string) resourcePlan {
p := resourcePlan{ProposalOnly: true, Evidence: "unknown", Domains: []domainCompatibility{}, Assignments: []runtimeAssignment{}, Unassigned: []string{}}
if len(policy) != 0 {
p.Policy = policy[0]
}
hosts := map[string][]qualificationCheck{}
for _, s := range source.config.Services {
for _, r := range s.Resources {
protocols := []string{r.TLS}
if r.TLS == "both" {
protocols = []string{"tls12", "tls13"}
}
for _, tls := range protocols {
host := strings.ToLower(r.host)
hosts[host] = append(hosts[host], qualificationCheck{Service: s.ID, Resource: r.ID, Fingerprint: resourceRowFingerprint(key, source, r), TLS: tls})
}
}
}
runs := map[string]qualificationRun{}
if validateMatrix(matrix, registry) == nil {
p.Evidence = "authenticated"
for _, r := range matrix.Runs {
if r.Config == source.fingerprint && r.Completed <= now.UnixMilli() && now.UnixMilli()-r.Completed <= 86400000 {
runs[r.Strategy] = r
}
}
}
names := []string{}
for host := range hosts {
names = append(names, host)
}
sort.Strings(names)
for _, host := range names {
row := domainCompatibility{Host: host, Validated: []string{}, Negative: []string{}, Unknown: []string{}}
for _, d := range registry.Strategies {
r, exists := runs[d.ID]
checks := map[string]qualificationCheck{}
for _, c := range r.Checks {
checks[c.Service+"/"+c.Resource+"/"+c.TLS] = c
}
complete, failed := exists, false
for _, expected := range hosts[host] {
c, ok := checks[expected.Service+"/"+expected.Resource+"/"+expected.TLS]
if !ok || c.Fingerprint != expected.Fingerprint || (d.Selectors.TLS != "both" && d.Selectors.TLS != expected.TLS) {
complete = false
continue
}
if c.Outcome != "pass" {
failed = true
}
}
if !complete {
row.Unknown = append(row.Unknown, d.ID)
} else if failed {
row.Negative = append(row.Negative, d.ID)
} else {
row.Validated = append(row.Validated, d.ID)
}
}
p.Domains = append(p.Domains, row)
}
// Merge only equal canonical args/selectors, retaining per-candidate alternatives.
// Greatest uncovered coverage is deterministic greedy, not globally optimal.
profiles := map[string]runtimeAssignment{}
coverage := map[string]map[string]bool{}
for _, d := range registry.Strategies {
if len(policy) != 0 && policy[0] != "" && !strings.Contains(","+policy[0]+",", ","+d.ID+",") {
continue
}
profile := planDigest(struct {
Args string
Selectors strategySelectors
}{d.Args, d.Selectors})
a, exists := profiles[profile]
if !exists || d.ID < a.Strategy {
profiles[profile] = runtimeAssignment{Strategy: d.ID, Args: d.Args, Selectors: d.Selectors}
}
if coverage[profile] == nil {
coverage[profile] = map[string]bool{}
}
for _, row := range p.Domains {
for _, id := range row.Validated {
if id == d.ID {
coverage[profile][row.Host] = true
}
}
}
}
assigned := map[string]bool{}
for {
best := ""
var chosen []string
for profile, a := range profiles {
uncovered := []string{}
for _, host := range names {
if coverage[profile][host] && !assigned[host] {
uncovered = append(uncovered, host)
}
}
if len(uncovered) > len(chosen) || len(uncovered) > 0 && len(uncovered) == len(chosen) && (best == "" || a.Strategy < profiles[best].Strategy) {
best, chosen = profile, uncovered
}
}
if len(chosen) == 0 {
break
}
a := profiles[best]
a.Hosts = chosen
p.Assignments = append(p.Assignments, a)
for _, host := range chosen {
assigned[host] = true
}
}
for _, host := range names {
if !assigned[host] {
p.Unassigned = append(p.Unassigned, host)
}
}
return p
}
+300
View File
@@ -0,0 +1,300 @@
package main
import (
"bytes"
"crypto/hmac"
"crypto/rand"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"syscall"
"time"
)
const planKeyName = ".resource-plan-key"
const planRegistryName = ".resource-plan-strategies.json"
const planMatrixName = ".resource-plan-matrix.json"
// Reuse held-directory/private-entry checks, not monitor publication/lifecycle.
// Entries are write-once O_EXCL: no replacement, repair or clobber of foreign files.
type resourcePlanStore struct {
files *resourceMonitorStore
held map[string]*os.File
snapshots map[string][]byte
}
type planEnvelope struct {
Payload json.RawMessage `json:"payload"`
MAC string `json:"mac"`
}
func openResourcePlanStore(path string, create bool) (*resourcePlanStore, error) {
if managedDirectory(path) != nil {
return nil, errResourcePlan
}
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
if err != nil {
return nil, err
}
s := &resourcePlanStore{files: &resourceMonitorStore{path: path, dir: os.NewFile(uintptr(fd), "plan directory")}, held: map[string]*os.File{}, snapshots: map[string][]byte{}}
fail := func(err error) (*resourcePlanStore, error) { s.close(); return nil, err }
for _, name := range []string{planKeyName, planRegistryName, planMatrixName} {
f, b, e := s.files.openEntry(name)
if e != nil && !errors.Is(e, os.ErrNotExist) {
return fail(errResourcePlan)
}
if e == nil {
s.held[name] = f
s.snapshots[name] = b
}
}
key := s.snapshots[planKeyName]
if key == nil && create && s.held[planRegistryName] == nil && s.held[planMatrixName] == nil {
key = make([]byte, 32)
if _, err = rand.Read(key); err != nil {
return fail(errResourcePlan)
}
if s.publish(planKeyName, key) != nil {
return fail(errResourcePlan)
}
}
if len(key) != 32 {
if !create && key == nil && s.held[planRegistryName] == nil && s.held[planMatrixName] == nil {
return fail(os.ErrNotExist)
}
return fail(errResourcePlan)
}
s.files.key = append([]byte(nil), key...)
if s.guard() != nil {
return fail(errResourcePlan)
}
return s, nil
}
func (s *resourcePlanStore) close() {
for _, f := range s.held {
f.Close()
}
s.files.dir.Close()
}
func (s *resourcePlanStore) guard() error {
if s.files.guard() != nil {
return errResourcePlan
}
names := map[string]bool{planKeyName: true, planRegistryName: true, planMatrixName: true}
for name := range s.held {
names[name] = true
}
for name := range names {
f := s.held[name]
if f == nil {
g, _, e := s.files.openEntry(name)
if g != nil {
g.Close()
}
if !errors.Is(e, os.ErrNotExist) {
return errResourcePlan
}
continue
}
b, e := s.files.trustedRead(name, f)
if e != nil || !bytes.Equal(b, s.snapshots[name]) {
return errResourcePlan
}
}
return nil
}
func (s *resourcePlanStore) publish(name string, b []byte) error {
if len(b) > 65536 || s.guard() != nil {
return errResourcePlan
}
fd, e := syscall.Openat(int(s.files.dir.Fd()), name, syscall.O_RDWR|syscall.O_CREAT|syscall.O_EXCL|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0600)
if e != nil {
return errResourcePlan
}
f := os.NewFile(uintptr(fd), name)
// Keep incomplete entries on failure: never delete a possible foreign inode.
if _, e = f.Write(b); e != nil {
f.Close()
return errResourcePlan
}
if f.Sync() != nil || s.files.entryMatches(name, f) != nil {
f.Close()
return errResourcePlan
}
read, e := s.files.trustedRead(name, f)
if e != nil || !bytes.Equal(read, b) {
f.Close()
return errResourcePlan
}
s.held[name], s.snapshots[name] = f, append([]byte(nil), b...)
if s.files.dir.Sync() != nil || s.guard() != nil {
return errResourcePlan
}
return nil
}
func (s *resourcePlanStore) writeAuthenticated(name string, v interface{}) error {
b, e := json.Marshal(v)
if e != nil {
return errResourcePlan
}
envelope, e := json.Marshal(planEnvelope{Payload: b, MAC: resourceMAC(s.files.key, b)})
if e != nil {
return errResourcePlan
}
return s.publish(name, envelope)
}
func (s *resourcePlanStore) readAuthenticated(name string, dst interface{}) error {
if s.guard() != nil {
return errResourcePlan
}
if s.held[name] == nil {
return os.ErrNotExist
}
var envelope planEnvelope
if strictPlanJSON(s.snapshots[name], &envelope) != nil || !hmac.Equal([]byte(envelope.MAC), []byte(resourceMAC(s.files.key, envelope.Payload))) || strictPlanJSON(envelope.Payload, dst) != nil {
return errResourcePlan
}
return s.guard()
}
func (s *resourcePlanStore) registry() (strategyRegistry, error) {
var r strategyRegistry
if e := s.readAuthenticated(planRegistryName, &r); e != nil {
return r, e
}
b, _ := json.Marshal(r)
canonical, e := parseStrategyRegistry(b)
if e != nil || planDigest(canonical) != planDigest(r) {
return r, errResourcePlan
}
return r, nil
}
func (s *resourcePlanStore) importDefinitions(b []byte) error {
r, e := parseStrategyRegistry(b)
if e != nil {
return e
}
return s.writeAuthenticated(planRegistryName, r)
}
func (s *resourcePlanStore) saveCompleted(p completedQualification) error {
r, e := s.registry()
if e != nil || p.seal == "" || p.seal != planDigest(p.matrix) || validateMatrix(p.matrix, r) != nil {
return errResourcePlan
}
return s.writeAuthenticated(planMatrixName, p.matrix)
}
func (s *resourcePlanStore) matrix(r strategyRegistry) (qualificationMatrix, error) {
var m qualificationMatrix
e := s.readAuthenticated(planMatrixName, &m)
if e != nil {
return m, e
}
if validateMatrix(m, r) != nil {
return m, errResourcePlan
}
return m, nil
}
func resourcePlanCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
if os.Geteuid() != 0 {
fmt.Fprintln(stderr, "blocked: root target role required")
return 78
}
if _, err := managedTargetSettings(d.lookup); err != nil || d.directory != managedTargetDirectory {
fmt.Fprintln(stderr, "blocked: managed target required")
return 78
}
if len(args) == 6 && args[0] == "qualify" && args[1] == "--config" && args[3] == "--strategies" && args[5] == "--isolated" {
return qualifyResourceCLI(args[2], args[4], stdout, stderr, d)
}
if len(args) == 3 && args[0] == "qualify" && args[1] == "--config" {
fmt.Fprintln(stderr, "unknown: qualification requires --isolated and PID 1 isolated startup; no runtime changes made")
return 78
}
if len(args) == 5 && args[0] == "qualify" && args[1] == "--config" && args[3] == "--strategies" {
fmt.Fprintln(stderr, "unknown: qualification requires --isolated and PID 1 isolated startup; no runtime changes made")
return 78
}
if len(args) == 3 && args[0] == "strategy-import" && args[1] == "--strategies" {
fd, e := syscall.Open(args[2], syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if e != nil {
fmt.Fprintln(stderr, "configerror: definitions unavailable")
return 64
}
f := os.NewFile(uintptr(fd), "definitions")
defer f.Close()
if resourcePrivateEntry(f) != nil {
fmt.Fprintln(stderr, "configerror: definitions require owned regular 0600 file")
return 64
}
b, e := io.ReadAll(io.LimitReader(f, 65537))
current, statErr := os.Lstat(args[2])
held, heldErr := f.Stat()
if e != nil || statErr != nil || heldErr != nil || !os.SameFile(current, held) || resourcePrivateEntry(f) != nil {
return 64
}
s, e := openResourcePlanStore(d.directory, true)
if e != nil {
fmt.Fprintln(stderr, "blocked: private registry unavailable")
return 78
}
defer s.close()
if s.importDefinitions(b) != nil {
fmt.Fprintln(stderr, "blocked: invalid definitions or immutable registry already exists")
return 78
}
fmt.Fprintln(stdout, "strategy definitions imported; qualification UNKNOWN")
return 0
}
if len(args) != 3 || args[0] != "plan" || args[1] != "--config" {
fmt.Fprintln(stderr, "usage: plan --config FILE | strategy-import --strategies FILE | qualify --config FILE --strategies FILE --isolated")
return 64
}
return resourcePlanProposal(args[2], stdout, stderr, d)
}
// Internal reader separated only to test the real CLI trust boundary in a
// temporary owned directory, without touching the provisioned target.
func resourcePlanProposal(configPath string, stdout, stderr io.Writer, d managedDeps) int {
key := make([]byte, 32)
registry := strategyRegistry{Version: 1, Strategies: []strategyDefinition{}}
var matrix qualificationMatrix
s, e := openResourcePlanStore(d.directory, false)
blocked := e != nil && !errors.Is(e, os.ErrNotExist)
if e == nil {
defer s.close()
key = s.files.key
registry, e = s.registry()
if e != nil {
blocked = !errors.Is(e, os.ErrNotExist)
registry = strategyRegistry{Version: 1, Strategies: []strategyDefinition{}}
}
if e == nil {
matrix, e = s.matrix(registry)
if e != nil {
blocked = !errors.Is(e, os.ErrNotExist)
matrix = qualificationMatrix{}
}
}
}
source, e := readResourceMonitorSource(configPath, d.lookup, key)
if e != nil {
fmt.Fprintln(stderr, "configerror: resource configuration unavailable")
return 64
}
p := deriveResourcePlan(source, key, registry, matrix, time.Now())
if blocked {
p.Evidence = "blocked"
}
if s != nil && s.guard() != nil {
p = deriveResourcePlan(source, key, registry, qualificationMatrix{}, time.Now())
p.Evidence = "blocked"
}
if json.NewEncoder(stdout).Encode(p) != nil {
return 1
}
if blocked || p.Evidence == "blocked" {
return 78
}
return 0
}
+331
View File
@@ -0,0 +1,331 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
func planFixture(t *testing.T) (resourceMonitorSource, []byte, strategyRegistry, qualificationMatrix, time.Time) {
t.Helper()
key := bytes.Repeat([]byte{7}, 32)
now := time.Unix(1700000000, 0)
source := resourceMonitorSource{server: "127.0.0.1:5353", proxy: "socks5h://127.0.0.1:1080", fingerprint: resourceMAC(key, []byte("config-secret")), config: resourceCheckConfig{Version: 1}}
for i, host := range []string{"a.example", "b.example", "c.example", "d.example", "e.example", "a.example"} {
service := "alpha"
group := "group-a"
if i == 2 {
service = "beta"
group = "group-b"
}
if i == 5 {
service = "gamma"
group = "group-c"
}
source.config.Services = append(source.config.Services, resourceCheckService{ID: service + string(rune('a'+i)), Resources: []resourceCheckSpec{{ID: "resource", Group: group, host: host, TLS: "both", URL: "https://" + host + "/SECRET?token=SECRET", privateConfig: "header = SECRET"}}})
}
registry := strategyRegistry{Version: 1}
for i, args := range []string{"--lua-desync=multisplit:pos=1", "--lua-desync=multidisorder:pos=1", "--lua-desync=fake:repeats=2"} {
registry.Strategies = append(registry.Strategies, strategyDefinition{ID: "s" + string(rune('1'+i)), Args: args, Selectors: strategySelectors{Transport: "tcp", Ports: "443", L7: "tls", TLS: "both"}})
}
matrix := qualificationMatrix{Version: 1, Registry: planDigest(registry), Runs: []qualificationRun{}}
for _, d := range registry.Strategies {
digest := planDigest(d)
r := qualificationRun{ID: "run-" + d.ID, Strategy: d.ID, Definition: digest, Config: source.fingerprint, Started: now.Add(-time.Minute).UnixMilli(), Completed: now.UnixMilli(), Attestation: qualificationAttestation{Collector: "owned-resource-nfq-v1", Runtime: "nfqws2-1.0.2", Definition: digest, LuaHash: strings.Repeat("a", 64), RulesHash: strings.Repeat("b", 64), WorkerArgsHash: strings.Repeat("c", 64), Queue: 200, PID: 123, StartTime: 456, Restored: true, Joined: true}}
for _, s := range source.config.Services {
for _, spec := range s.Resources {
if spec.host == "e.example" {
continue
}
pass := d.ID == "s1" && (spec.host == "a.example" || spec.host == "c.example") || d.ID == "s2" && spec.host == "b.example" || d.ID == "s3" && spec.host == "d.example"
for _, tls := range []string{"tls12", "tls13"} {
outcome := "expectation_failed"
if pass {
outcome = "pass"
}
r.Checks = append(r.Checks, qualificationCheck{Service: s.ID, Resource: spec.ID, Fingerprint: resourceRowFingerprint(key, source, spec), TLS: tls, Outcome: outcome, Attempts: 3})
}
}
}
matrix.Runs = append(matrix.Runs, r)
}
return source, key, registry, matrix, now
}
func cloneMatrix(m qualificationMatrix) qualificationMatrix {
b, _ := json.Marshal(m)
var v qualificationMatrix
json.Unmarshal(b, &v)
return v
}
func TestResourcePlanCoverage(t *testing.T) {
source, key, registry, matrix, now := planFixture(t)
p := deriveResourcePlan(source, key, registry, matrix, now)
want := [][]string{{"a.example", "c.example"}, {"b.example"}, {"d.example"}}
if len(p.Assignments) != 3 || !reflect.DeepEqual(p.Unassigned, []string{"e.example"}) {
t.Fatalf("%+v", p)
}
for i, a := range p.Assignments {
if !reflect.DeepEqual(a.Hosts, want[i]) {
t.Fatalf("%+v", p)
}
}
for i := 0; i < 100; i++ {
if !reflect.DeepEqual(p, deriveResourcePlan(source, key, registry, matrix, now)) {
t.Fatal("nondeterministic")
}
}
b, _ := json.Marshal(matrix)
out, _ := json.Marshal(p)
if strings.Contains(string(b)+string(out), "SECRET") || strings.Contains(string(b), "https://") {
t.Fatal("secret serialization")
}
// One bad URL on the same host in another service/group invalidates only a.
bad := cloneMatrix(matrix)
bad.Runs[0].Checks[len(bad.Runs[0].Checks)-1].Outcome = "expectation_failed"
changed := deriveResourcePlan(source, key, registry, bad, now)
if len(changed.Domains[0].Validated) != 0 || !reflect.DeepEqual(p.Domains[1:], changed.Domains[1:]) {
t.Fatalf("subset failure leaked: %+v", changed)
}
partial := cloneMatrix(matrix)
partial.Runs[0].Checks = partial.Runs[0].Checks[:len(partial.Runs[0].Checks)-1]
changed = deriveResourcePlan(source, key, registry, partial, now)
if len(changed.Domains[0].Validated) != 0 || len(changed.Domains[0].Unknown) != 1 {
t.Fatal("partial TLS qualified")
}
for _, mutate := range []func(*qualificationMatrix){
func(m *qualificationMatrix) { m.Runs[0].Config = strings.Repeat("c", 64) },
func(m *qualificationMatrix) {
m.Runs[0].Completed = now.Add(-25 * time.Hour).UnixMilli()
m.Runs[0].Started = m.Runs[0].Completed - 1
},
func(m *qualificationMatrix) { m.Runs[0].Attestation.Joined = false },
func(m *qualificationMatrix) { m.Runs[0].Checks = append(m.Runs[0].Checks, m.Runs[0].Checks[0]) },
func(m *qualificationMatrix) { m.Runs[1].ID = m.Runs[0].ID },
func(m *qualificationMatrix) { m.Runs[0].Definition = strings.Repeat("c", 64) },
} {
m := cloneMatrix(matrix)
mutate(&m)
if len(deriveResourcePlan(source, key, registry, m, now).Domains[0].Validated) != 0 {
t.Fatal("invalid evidence qualified")
}
}
source.fingerprint = strings.Repeat("d", 64)
if len(deriveResourcePlan(source, key, registry, matrix, now).Assignments) != 0 {
t.Fatal("changed config qualified")
}
}
func TestResourcePlanSelectorsAndAlternatives(t *testing.T) {
source, key, registry, matrix, now := planFixture(t)
registry.Strategies[1].Args = registry.Strategies[0].Args
matrix.Registry = planDigest(registry)
matrix.Runs[1].Definition = planDigest(registry.Strategies[1])
matrix.Runs[1].Attestation.Definition = matrix.Runs[1].Definition
// Add a shared alternative on c without removing original evidence.
for i := range matrix.Runs[1].Checks {
if matrix.Runs[1].Checks[i].Service == "betac" {
matrix.Runs[1].Checks[i].Outcome = "pass"
}
}
p := deriveResourcePlan(source, key, registry, matrix, now)
if len(p.Assignments) != 2 || !reflect.DeepEqual(p.Assignments[0].Hosts, []string{"a.example", "b.example", "c.example"}) || len(p.Domains[2].Validated) != 2 {
t.Fatalf("merge/alternatives: %+v", p)
}
registry.Strategies[1].Selectors.TLS = "tls12"
matrix.Registry = planDigest(registry)
matrix.Runs[1].Definition = planDigest(registry.Strategies[1])
matrix.Runs[1].Attestation.Definition = matrix.Runs[1].Definition
// Hosts a,c require both; host b only tls12.
source.config.Services[1].Resources[0].TLS = "tls12"
for i := range matrix.Runs[1].Checks {
c := &matrix.Runs[1].Checks[i]
if c.Service == "alphab" {
c.Fingerprint = resourceRowFingerprint(key, source, source.config.Services[1].Resources[0])
}
}
p = deriveResourcePlan(source, key, registry, matrix, now)
if len(p.Assignments) != 3 || p.Assignments[0].Selectors == p.Assignments[1].Selectors {
t.Fatalf("selectors merged: %+v", p)
}
}
func TestResourcePlanRegistryValidation(t *testing.T) {
_, _, r, _, _ := planFixture(t)
d := r.Strategies[0]
d.Args = " \t--payload=tls_client_hello\n--lua-desync=multisplit:pos=1 "
v, e := canonicalStrategy(d)
if e != nil || v.Args != "--payload=tls_client_hello --lua-desync=multisplit:pos=1" {
t.Fatalf("canonical: %+v %v", v, e)
}
for _, args := range []string{"--lua-init=@/opt/zapret2/lua/evil.lua", "--qnum=200", "--user=root", "--fwmark=1", "--filter-mark=1", "--lua-desync=execute:cmd=id", "--lua-desync=fake:blob=/etc/passwd", "--lua-desync=fake:blob=secret.lua", "--lua-desync=fake:repeats=1;id"} {
d.Args = args
if _, e := canonicalStrategy(d); e == nil {
t.Fatalf("accepted %s", args)
}
}
for _, b := range []string{`{"version":1,"version":1,"strategies":[]}`, `{"version":1,"strategies":[],"evidence":true}`} {
if _, e := parseStrategyRegistry([]byte(b)); e == nil {
t.Fatal("import accepted proof/duplicate")
}
}
}
func TestResourcePlanPrivateStore(t *testing.T) {
_, _, registry, matrix, _ := planFixture(t)
dir := t.TempDir()
os.Chmod(dir, 0700)
s, e := openResourcePlanStore(dir, true)
if e != nil {
t.Fatal(e)
}
defer s.close()
b, _ := json.Marshal(registry)
if s.importDefinitions(b) != nil {
t.Fatal("import")
}
if s.importDefinitions(b) == nil {
t.Fatal("clobbered immutable registry")
}
proof := completedQualification{matrix: cloneMatrix(matrix), seal: planDigest(matrix)}
proof.matrix.Runs[0].Checks[0].Outcome = "expectation_failed"
if s.saveCompleted(proof) == nil {
t.Fatal("post-seal mutation")
}
if s.saveCompleted(completedQualification{matrix: matrix}) == nil {
t.Fatal("unsealed DTO")
}
if s.saveCompleted(completedQualification{matrix: matrix, seal: planDigest(matrix)}) != nil {
t.Fatal("fixture save")
}
loaded, e := s.matrix(registry)
if e != nil || !reflect.DeepEqual(loaded, matrix) {
t.Fatalf("load: %v", e)
}
second, e := openResourcePlanStore(dir, false)
if e != nil {
t.Fatal(e)
}
second.close()
path := filepath.Join(dir, planMatrixName)
raw, e := os.ReadFile(path)
if e != nil {
t.Fatal(e)
}
raw = bytes.Replace(raw, []byte(`"outcome":"pass"`), []byte(`"outcome":"timeout"`), 1)
os.WriteFile(path, raw, 0600)
if _, e = s.matrix(registry); e == nil {
t.Fatal("held mutation accepted")
}
reopened, e := openResourcePlanStore(dir, false)
if e != nil {
t.Fatal(e)
}
defer reopened.close()
if _, e = reopened.matrix(registry); e == nil {
t.Fatal("forged HMAC accepted")
}
}
func TestResourcePlanForeignEntries(t *testing.T) {
for _, kind := range []string{"file", "symlink", "replacement"} {
t.Run(kind, func(t *testing.T) {
dir := t.TempDir()
os.Chmod(dir, 0700)
path := filepath.Join(dir, planRegistryName)
if kind == "file" {
os.WriteFile(path, []byte("FOREIGN"), 0600)
} else if kind == "symlink" {
os.Symlink("foreign", path)
}
s, e := openResourcePlanStore(dir, true)
if kind != "replacement" {
if e == nil {
s.close()
t.Fatal("adopted foreign registry without key")
}
return
}
if e != nil {
t.Fatal(e)
}
defer s.close()
_, _, r, _, _ := planFixture(t)
b, _ := json.Marshal(r)
if s.importDefinitions(b) != nil {
t.Fatal("import")
}
os.Rename(path, path+".old")
os.WriteFile(path, s.snapshots[planRegistryName], 0600)
if _, e = s.registry(); e == nil {
t.Fatal("adopted identical foreign replacement")
}
if s.importDefinitions(b) == nil {
t.Fatal("clobbered replacement")
}
})
}
}
func TestResourcePlanCLIUnknownAndForged(t *testing.T) {
dir := t.TempDir()
os.Chmod(dir, 0700)
path := filepath.Join(dir, "resources.json")
config := `{"version":1,"services":[{"id":"service","resources":[{"id":"resource","group":"metadata","kind":"http","url":"https://a.example/SECRET","method":"GET","tls":"both","timeout_ms":1000,"expect":{"statuses":[200],"min_bytes":0,"max_bytes":100}}]}]}`
if err := os.WriteFile(path, []byte(config), 0600); err != nil {
t.Fatal(err)
}
env := map[string]string{"ZAPRET_AUTOTUNE_SOCKS_PROXY": "socks5h://127.0.0.1:1080", "ZAPRET_AUTOTUNE_DNS_MODE": "native", "SS_DNS_SERVER": "1.1.1.1"}
d := managedDeps{directory: dir, lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
var out, diagnostics bytes.Buffer
code := resourcePlanProposal(path, &out, &diagnostics, d)
if code != 0 {
t.Fatalf("unknown reader: %d %s", code, diagnostics.String())
}
var p resourcePlan
if json.Unmarshal(out.Bytes(), &p) != nil || p.Evidence != "unknown" || len(p.Assignments) != 0 || len(p.Unassigned) != 1 || strings.Contains(out.String(), "SECRET") {
t.Fatalf("%s", out.String())
}
s, err := openResourcePlanStore(dir, true)
if err != nil {
t.Fatal(err)
}
_, _, registry, matrix, _ := planFixture(t)
definitions, _ := json.Marshal(registry)
if s.importDefinitions(definitions) != nil {
t.Fatal("import")
}
source, err := readResourceMonitorSource(path, d.lookup, s.files.key)
if err != nil {
t.Fatal(err)
}
for i := range matrix.Runs {
r := &matrix.Runs[i]
r.Config = source.fingerprint
r.Started = time.Now().Add(-time.Minute).UnixMilli()
r.Completed = time.Now().UnixMilli()
r.Checks = nil
for _, tls := range []string{"tls12", "tls13"} {
r.Checks = append(r.Checks, qualificationCheck{Service: "service", Resource: "resource", TLS: tls, Outcome: "pass", Fingerprint: resourceRowFingerprint(s.files.key, source, source.config.Services[0].Resources[0]), Attempts: 3})
}
}
if s.saveCompleted(completedQualification{matrix: matrix, seal: planDigest(matrix)}) != nil {
t.Fatal("sealed fixture")
}
s.close()
out.Reset()
code = resourcePlanProposal(path, &out, &diagnostics, d)
if code != 0 || json.Unmarshal(out.Bytes(), &p) != nil || p.Evidence != "authenticated" || len(p.Assignments) != 1 || len(p.Domains[0].Validated) != 3 {
t.Fatalf("authenticated CLI %d %s", code, out.String())
}
// Public JSON PASS and monitor observations cannot become private proof.
if err := os.WriteFile(filepath.Join(dir, planMatrixName), []byte(`{"pass":true,"health":"HEALTHY"}`), 0600); err != nil {
t.Fatal(err)
}
out.Reset()
diagnostics.Reset()
code = resourcePlanProposal(path, &out, &diagnostics, d)
if code != 78 || !strings.Contains(out.String(), `"evidence":"blocked"`) || strings.Contains(out.String(), `"hosts":[`) {
t.Fatalf("forged CLI %d %s %s", code, out.String(), diagnostics.String())
}
}
+684
View File
@@ -0,0 +1,684 @@
package main
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"os"
"os/exec"
"os/signal"
"path/filepath"
"reflect"
"sort"
"strconv"
"strings"
"syscall"
"time"
)
const qualificationEngine = "/opt/zapret2/nfq2/nfqws2"
var errQualificationCapacity = errors.New("qualification matrix exceeds private capacity")
// A fresh disabled baseline needs no daemon stop/config transaction. Deliberately
// refuse existing active profiles: maintenance pause/restore is not implemented.
// ponytail: isolated startup baseline only; add job-bound profile rollback before
// supporting an already active desired set, never just invoke a daemon stop.
func qualificationBaseline(dir string) error {
if resourceManagedIdentity(dir) != nil {
return errResourcePlan
}
m := &managedTransaction{Dir: dir, Name: "config"}
j, e := m.load()
if e != nil || j.State != "committed" || len(j.Candidates) != 0 {
return errResourcePlan
}
b, e := m.read("config")
if e != nil || string(b) != managedBaseline {
return errResourcePlan
}
return nil
}
func qualificationMode(config, strategies string, d managedDeps) error {
if os.Geteuid() != 0 || os.Getppid() != 1 {
return errResourcePlan
}
if _, e := managedRuntimeGuard(d); e != nil {
return e
}
cmd, e := os.ReadFile("/proc/1/cmdline")
if e != nil || string(cmd) != "/bin/sh\x00/opt/managed-target-entrypoint.sh\x00" {
return errResourcePlan
}
env, e := os.ReadFile("/proc/1/environ")
if e != nil {
return errResourcePlan
}
values := map[string]string{}
for _, s := range strings.Split(string(env), "\x00") {
k, v, _ := strings.Cut(s, "=")
values[k] = v
}
if config == "" || strategies == "" || values["ZAPRET_AUTOTUNE_QUALIFY_CONFIG"] != config || values["ZAPRET_AUTOTUNE_STRATEGY_CONFIG"] != strategies {
return errResourcePlan
}
for _, ns := range []string{"pid", "net"} {
a, e := os.Readlink("/proc/1/ns/" + ns)
b, f := os.Readlink("/proc/self/ns/" + ns)
if e != nil || f != nil || a != b {
return errResourcePlan
}
}
if !resourceProcessOwned("/proc", os.Getpid(), "/usr/local/bin/zapret-autotune", []string{"/usr/local/bin/zapret-autotune", "qualify", "--config", config, "--strategies", strategies, "--isolated"}) {
return errResourcePlan
}
dir, e := os.Open("/run/zapret-autotune-ss")
if e != nil {
return errResourcePlan
}
defer dir.Close()
ready, e := resourcePrivateFile(dir, "monitor-ready")
if e != nil {
return e
}
defer ready.Close()
b, e := io.ReadAll(io.LimitReader(ready, 32))
if e != nil || string(b) != strconv.Itoa(os.Getpid())+"\n" {
return errResourcePlan
}
return qualificationBaseline(d.directory)
}
func qualificationFilesHash() (string, error) {
engine, e := openQualificationEngine("/opt/zapret2")
if e != nil {
return "", e
}
defer engine.file.Close()
return engine.filesHash()
}
func (engine *qualificationBinary) filesHash() (string, error) {
// Fixed image files only. Bind both executable and Lua inputs before and after
// every candidate; never accept a caller-supplied native hash/version label.
if engine.validate() != nil {
return "", errResourcePlan
}
hashes := []string{engine.hash}
for _, path := range []string{filepath.Join(engine.root, "lua/zapret-lib.lua"), filepath.Join(engine.root, "lua/zapret-antidpi.lua")} {
fd, e := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if e != nil {
return "", errResourcePlan
}
f := os.NewFile(uintptr(fd), path)
info, e := f.Stat()
if e != nil || !info.Mode().IsRegular() || info.Size() > 32<<20 || info.Mode().Perm()&0022 != 0 {
f.Close()
return "", errResourcePlan
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok || st.Uid != 0 {
f.Close()
return "", errResourcePlan
}
b, e := io.ReadAll(io.LimitReader(f, 32<<20+1))
current, se := os.Lstat(path)
f.Close()
if e != nil || se != nil || !os.SameFile(info, current) || int64(len(b)) != info.Size() {
return "", errResourcePlan
}
hashes = append(hashes, planDigest(b))
}
return planDigest(hashes), nil
}
func (s *resourcePlanStore) protectBaseline() error {
for _, name := range []string{"config", managedMarkerName, managedJournalName, managedBackupName} {
f, b, e := s.files.openEntry(name)
if e != nil {
return errResourcePlan
}
s.held[name], s.snapshots[name] = f, b
}
return s.guard()
}
func qualificationNoWorker() error {
if e := qualificationOnlyWorker(0); e != nil {
return qualificationPredicate(e, qualificationErrorReason(e, reasonWorkerScan))
}
occupied, e := managedAssetQueueReady(0)
if e != nil || occupied {
return qualificationPredicate(errResourcePlan, reasonQueuePresentOrUnreadable)
}
return nil
}
func (q resourceQualifier) restoreBaseline(ctx context.Context, noWorker, baseline func() error) error {
if ctx.Err() != nil {
return qualificationPredicate(errResourcePlan, reasonContext)
}
if e := q.guard(); e != nil {
return qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonGuard))
}
if e := noWorker(); e != nil {
return qualificationPlanPredicate(e, reasonUnknown)
}
tables, e := q.runtime.nft(ctx, "list tables\n")
if e != nil {
return qualificationPredicate(errResourcePlan, reasonNFTRead)
}
if strings.Contains(tables, "table inet "+managedAssetTable) {
return qualificationPredicate(errResourcePlan, reasonNFTPresent)
}
if e := baseline(); e != nil {
return qualificationPredicate(e, reasonBaseline)
}
return nil
}
func qualificationOnlyWorker(owned int) error {
return qualificationScanWorkers(owned, os.ReadDir, os.Readlink)
}
func qualificationScanWorkers(owned int, readDir func(string) ([]os.DirEntry, error), readlink func(string) (string, error)) error {
entries, e := readDir("/proc")
if e != nil {
return &qualificationDiagnostic{Err: e, Reason: reasonWorkerScanRead, Errno: qualificationScanErrno(e), OwnedPID: owned}
}
for _, entry := range entries {
pid, e := strconv.Atoi(entry.Name())
if e != nil {
continue
}
exe, e := readlink(filepath.Join("/proc", entry.Name(), "exe"))
if e != nil {
if errors.Is(e, os.ErrNotExist) {
continue
}
return &qualificationDiagnostic{Err: errResourcePlan, Reason: reasonWorkerScanReadlink, Errno: qualificationScanErrno(e), ScanPID: pid, OwnedPID: owned}
}
if filepath.Base(exe) == "nfqws2" && entry.Name() != strconv.Itoa(owned) {
return &qualificationDiagnostic{Err: errResourcePlan, Reason: reasonWorkerScanForeign, ScanPID: pid, OwnedPID: owned}
}
}
return nil
}
func qualificationWorkerArgs(endpoints []string, args string) []string {
base := managedProbeWorkerArgs(endpoints, "")
base = append(base, "--filter-l7=tls") // Supported by pinned 1.0.2; generic caller only.
return append(base, strings.Fields(args)...)
}
// All seams are package-private and supplied by the runtime factory, not JSON.
type resourceQualifier struct {
runtime managedAssetRuntime
checks resourceCheckDeps
source func() (resourceMonitorSource, error)
restore func(context.Context) error
attest func(context.Context, int, []string, strategyDefinition) (qualificationAttestation, error)
guard func() error
now func() time.Time
filesHash func() (string, error)
close func()
}
func newResourceQualifier(config, strategies string, d managedDeps, s *resourcePlanStore) resourceQualifier {
r := newManagedAssetRuntime(d)
q := resourceQualifier{runtime: r, checks: productionResourceCheckDeps(), now: time.Now}
var engine *qualificationBinary
q.close = func() {
if engine != nil {
engine.file.Close()
}
}
q.filesHash = func() (string, error) {
if engine == nil {
var e error
engine, e = openQualificationEngine("/opt/zapret2")
if e != nil {
return "", e
}
}
return engine.filesHash()
}
start, queue := r.start, r.queue
r.start = func(args []string) (managedAssetWorker, error) {
if engine == nil || engine.validate() != nil {
return managedAssetWorker{}, errResourcePlan
}
return start(args)
}
r.queue = func(pid int) (bool, error) {
if engine == nil || engine.validate() != nil {
return false, errResourcePlan
}
ready, e := queue(pid)
// withOwnedWorker checks READY before creating the nft table.
if e == nil && ready && pid != 0 && engine.running(fmt.Sprintf("/proc/%d/exe", pid)) != nil {
return false, errResourcePlan
}
return ready, e
}
q.runtime = r
q.source = func() (resourceMonitorSource, error) { return readResourceMonitorSource(config, d.lookup, s.files.key) }
q.guard = func() error {
if e := qualificationMode(config, strategies, d); e != nil {
return qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonMode))
}
if e := s.guard(); e != nil {
return qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonPlanStore))
}
if e := r.guard(); e != nil {
return qualificationPredicate(e, qualificationErrorReason(e, reasonRuntime))
}
return nil
}
q.restore = func(ctx context.Context) error {
return q.restoreBaseline(ctx, qualificationNoWorker, func() error { return qualificationBaseline(d.directory) })
}
q.attest = func(ctx context.Context, pid int, args []string, definition strategyDefinition) (qualificationAttestation, error) {
if e := q.guard(); e != nil {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, qualificationErrorReason(e, reasonGuard))
}
if e := qualificationOnlyWorker(pid); e != nil {
return qualificationAttestation{}, qualificationPlanPredicate(e, reasonWorkerScan)
}
if engine == nil || engine.validate() != nil || engine.running(fmt.Sprintf("/proc/%d/exe", pid)) != nil {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonEngine)
}
cmd, e := os.ReadFile(fmt.Sprintf("/proc/%d/cmdline", pid))
if e != nil || string(cmd) != strings.Join(append([]string{qualificationEngine}, args...), "\x00")+"\x00" {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonArgv)
}
stat, e := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
if e != nil {
return qualificationAttestation{}, qualificationPredicate(e, reasonParent)
}
end := strings.LastIndex(string(stat), ") ")
if end < 0 {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonParent)
}
tail := strings.Fields(string(stat)[end+2:])
if len(tail) < 2 || tail[1] != strconv.Itoa(os.Getpid()) {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonParent)
}
start, e := readMonitorProcessStarttime(pid)
if e != nil {
return qualificationAttestation{}, qualificationPredicate(e, reasonStarttime)
}
ready, e := r.queue(pid)
if e != nil || !ready {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonQueue)
}
hash, e := q.filesHash()
if e != nil {
return qualificationAttestation{}, qualificationPredicate(e, reasonFiles)
}
versionCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
var output bytes.Buffer
version := exec.CommandContext(versionCtx, engine.path, "--version")
version.Stdout = &playerBoundedWriter{dst: &output, limit: 4096}
version.Stderr = io.Discard
if version.Run() != nil {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonVersion)
}
fields := strings.Fields(output.String())
valid := false
for i, f := range fields {
if f == "version" && i+1 < len(fields) && (fields[i+1] == "1.0.2" || fields[i+1] == "v1.0.2") {
valid = true
}
}
if !valid {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonVersion)
}
rules, e := r.nft(ctx, "list table inet "+managedAssetTable+"\n")
if e != nil || strings.TrimSpace(rules) == "" {
return qualificationAttestation{}, qualificationPredicate(errResourcePlan, reasonRulesRead)
}
return qualificationAttestation{Collector: "owned-resource-nfq-v1", Runtime: "nfqws2-1.0.2", Definition: planDigest(definition), LuaHash: hash, RulesHash: planDigest(rules), WorkerArgsHash: planDigest(args), Queue: managedAssetQueue, PID: pid, StartTime: start}, nil
}
return q
}
func (q resourceQualifier) complete(ctx context.Context, s *resourcePlanStore, registry strategyRegistry) (completedQualification, error) {
fail := func(e error) (completedQualification, error) { return completedQualification{}, e }
if q.guard == nil || q.restore == nil || q.attest == nil || q.source == nil || q.now == nil || q.filesHash == nil || q.guard() != nil {
return fail(errResourcePlan)
}
if s.held[planMatrixName] != nil {
return fail(errResourcePlan)
}
trustedRegistry, registryErr := s.registry()
if registryErr != nil || planDigest(trustedRegistry) != planDigest(registry) {
return fail(errResourcePlan)
}
source, e := q.source()
if e != nil {
return fail(e)
}
hosts := map[string]bool{}
for _, service := range source.config.Services {
for _, spec := range service.Resources {
if !validDomain(spec.host) || net.ParseIP(spec.host) != nil || spec.host != strings.ToLower(spec.host) {
return fail(errResourcePlan)
}
hosts[spec.host] = true
}
}
endpoints := []string{}
for host := range hosts {
endpoints = append(endpoints, host)
}
sort.Strings(endpoints)
matrix := qualificationMatrix{Version: 1, Registry: planDigest(registry), Runs: []qualificationRun{}}
// Check the worst-case immutable envelope size before launching any worker.
estimate := qualificationMatrix{Version: 1, Registry: matrix.Registry}
for _, definition := range registry.Strategies {
run := qualificationRun{ID: "run-" + strings.Repeat("a", 32), Strategy: definition.ID, Definition: planDigest(definition), Config: source.fingerprint, Started: 9223372036854775807, Completed: 9223372036854775807, Attestation: qualificationAttestation{Collector: "owned-resource-nfq-v1", Runtime: "nfqws2-1.0.2", Definition: planDigest(definition), LuaHash: strings.Repeat("a", 64), RulesHash: strings.Repeat("a", 64), WorkerArgsHash: strings.Repeat("a", 64), Queue: 200, PID: 2147483647, StartTime: 9223372036854775807, Restored: true, Joined: true}}
for _, service := range source.config.Services {
for _, spec := range service.Resources {
protocols := []string{spec.TLS}
if spec.TLS == "both" {
protocols = []string{"tls12", "tls13"}
}
for _, protocol := range protocols {
run.Checks = append(run.Checks, qualificationCheck{Service: service.ID, Resource: spec.ID, TLS: protocol, Fingerprint: resourceRowFingerprint(s.files.key, source, spec), Outcome: "expectation_failed", Attempts: 3})
}
}
}
estimate.Runs = append(estimate.Runs, run)
}
b, _ := json.Marshal(estimate)
// Include the exact authenticated envelope overhead, using maximal timestamp
// widths and the longest permitted outcome. Args/selectors are fixed digests.
envelope, _ := json.Marshal(planEnvelope{Payload: b, MAC: strings.Repeat("a", 64)})
if len(envelope) > 65536 {
return fail(errQualificationCapacity)
}
stable := func() error {
if ctx.Err() != nil {
return ctx.Err()
}
if q.guard() != nil {
return errResourcePlan
}
fresh, e := q.source()
if e != nil || fresh.fingerprint != source.fingerprint {
return errResourcePlan
}
return nil
}
for candidateIndex, definition := range registry.Strategies {
if _, e := canonicalStrategy(definition); e != nil {
return fail(e)
}
if e := stable(); e != nil {
return fail(e)
}
nativeBefore, nativeErr := q.filesHash()
if nativeErr != nil || !hashOK(nativeBefore) {
return fail(errResourcePlan)
}
id := make([]byte, 16)
if _, e := rand.Read(id); e != nil {
return fail(e)
}
run := qualificationRun{ID: "run-" + hex.EncodeToString(id), Strategy: definition.ID, Definition: planDigest(definition), Config: source.fingerprint, Started: q.now().UnixMilli()}
runtime := q.runtime
originalStart := runtime.start
pid := 0
var args []string
runtime.start = func(actual []string) (managedAssetWorker, error) {
if !reflect.DeepEqual(actual, managedProbeWorkerArgs(endpoints, definition.Args)) {
return managedAssetWorker{}, errResourcePlan
}
args = qualificationWorkerArgs(endpoints, definition.Args)
worker, e := originalStart(args)
pid = worker.pid
return worker, e
}
err := runtime.withOwnedWorker(ctx, endpoints, definition.Args, func(captureCtx context.Context) error {
if !reflect.DeepEqual(args, qualificationWorkerArgs(endpoints, definition.Args)) {
return errResourcePlan
}
a, e := q.attest(captureCtx, pid, args, definition)
if e != nil {
return qualificationOperation(e, stageAttestation)
}
if a.LuaHash != nativeBefore {
return &qualificationDiagnostic{Err: errResourcePlan, Stage: stageAttestation, Reason: reasonFiles}
}
run.Attestation = a
checks := q.checks
checks.qualification = true
originalGuard := checks.guard
checks.guard = func() error {
if e := stable(); e != nil {
return e
}
if e := originalGuard(); e != nil {
return e
}
current, e := q.attest(captureCtx, pid, args, definition)
if e != nil {
return qualificationOperation(errors.Join(errResourcePlan, e), stageAttestation)
}
if current != a {
return &qualificationDiagnostic{Err: errResourcePlan, Stage: stageAttestation, Reason: attestationMismatch(a, current)}
}
return nil
}
resourceIndex := 0
for _, service := range source.config.Services {
for _, spec := range service.Resources {
resourceIndex++
protocols := []string{spec.TLS}
if spec.TLS == "both" {
protocols = []string{"tls12", "tls13"}
}
for _, protocol := range protocols {
// Unsupported TLS policy is unknown for this candidate, never PASS. Do
// still measure the full required set; planner applies selector eligibility.
cell := qualificationCheck{Service: service.ID, Resource: spec.ID, Fingerprint: resourceRowFingerprint(s.files.key, source, spec), TLS: protocol, Outcome: "pass"}
for repeat := 0; repeat < 3; repeat++ {
result, e := runResourceProtocol(captureCtx, service.ID, spec, protocol, source.server, source.proxy, checks)
if e == nil && captureCtx.Err() != nil {
e = captureCtx.Err()
}
if e == nil && result.Outcome == "cancelled" {
e = context.Canceled
}
if e != nil {
tls := uint8(12)
if protocol == "tls13" {
tls = 13
}
return &qualificationDiagnostic{Err: e, Candidate: candidateIndex + 1, Resource: resourceIndex, Attempt: repeat + 1, TLS: tls}
}
if result.Outcome != "pass" {
cell.Outcome = result.Outcome
}
cell.Attempts++
}
run.Checks = append(run.Checks, cell)
}
}
}
return stable()
})
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
restoreErr := q.restore(cleanup)
if restoreErr != nil {
restoreErr = &qualificationDiagnostic{Err: restoreErr, Stage: stageRestore, Candidate: candidateIndex + 1}
}
cancel()
if err != nil || restoreErr != nil {
err = qualificationOperation(err, stageUnknown)
if diagnostic, ok := err.(*qualificationDiagnostic); ok {
diagnostic.Candidate = candidateIndex + 1
}
if restoreErr != nil {
return fail(&qualificationCleanupFailure{Err: errors.Join(err, restoreErr)})
}
return fail(errors.Join(err, restoreErr))
}
hash, hashErr := q.filesHash()
if hashErr != nil || hash != run.Attestation.LuaHash {
return fail(errResourcePlan)
}
if e := stable(); e != nil {
return fail(e)
}
run.Attestation.Joined, run.Attestation.Restored = true, true
run.Completed = q.now().UnixMilli()
matrix.Runs = append(matrix.Runs, run)
}
if e := stable(); e != nil {
return fail(e)
}
if validateMatrix(matrix, registry) != nil {
return fail(errResourcePlan)
}
return completedQualification{matrix: matrix, seal: planDigest(matrix)}, nil
}
func qualifyResourceCLI(config, strategies string, stdout, stderr io.Writer, d managedDeps) (code int) {
if qualificationMode(config, strategies, d) != nil {
fmt.Fprintln(stderr, "blocked: qualification requires PID 1 isolated startup with disabled baseline")
return 78
}
// Refuse existing/foreign proof state before even creating a singleton inode.
prior, priorErr := openResourcePlanStore(d.directory, false)
if priorErr == nil {
exists := prior.held[planMatrixName] != nil
prior.close()
if exists {
fmt.Fprintln(stderr, "blocked: immutable matrix already exists; no runtime changes made")
return 78
}
} else if !errors.Is(priorErr, os.ErrNotExist) {
fmt.Fprintln(stderr, "blocked: qualification private state")
return 78
}
lock, e := acquireManagedMonitorSingleton(d)
if e != nil {
fmt.Fprintln(stderr, "blocked: another controller owns target")
return 78
}
defer lock.close()
if qualificationNoWorker() != nil {
fmt.Fprintln(stderr, "blocked: conflicting NFQ worker/queue")
return 78
}
s, e := openResourcePlanStore(d.directory, true)
if e != nil {
fmt.Fprintln(stderr, "blocked: qualification private state")
return 78
}
defer s.close()
if s.held[planMatrixName] != nil {
fmt.Fprintln(stderr, "blocked: immutable matrix already exists; no runtime changes made")
return 78
}
if s.protectBaseline() != nil {
fmt.Fprintln(stderr, "blocked: known baseline snapshot unavailable")
return 78
}
fd, e := syscall.Open(strategies, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if e != nil {
return 64
}
f := os.NewFile(uintptr(fd), "definitions")
defer f.Close()
if resourcePrivateEntry(f) != nil {
return 64
}
b, e := io.ReadAll(io.LimitReader(f, 65537))
info, se := os.Lstat(strategies)
held, he := f.Stat()
if e != nil || se != nil || he != nil || !os.SameFile(info, held) {
return 64
}
registry, e := parseStrategyRegistry(b)
if e != nil {
return 64
}
if s.held[planRegistryName] == nil {
if s.importDefinitions(b) != nil {
return 78
}
} else {
existing, e := s.registry()
if e != nil || planDigest(existing) != planDigest(registry) {
return 78
}
}
signals, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
ctx, cancel := context.WithTimeout(signals, 15*time.Minute)
defer cancel()
q := newResourceQualifier(config, strategies, d, s)
defer q.close()
var sessionErr error
defer func() {
if code != 0 && code != 78 {
return
}
if e := qualificationFinalize(sessionErr, q.restore, s.guard, func() error {
return writeQualificationCompletion(qualificationCompletionDir, s.files.key, os.Getpid(), code)
}); e != nil {
writeQualificationDiagnostics(stderr, e)
code = 78
}
}()
originalGuard := q.guard
q.guard = func() error {
current, e := os.Lstat(strategies)
heldInfo, he := f.Stat()
read, re := io.ReadAll(io.NewSectionReader(f, 0, 65537))
if e != nil || he != nil || re != nil || !os.SameFile(current, heldInfo) || resourcePrivateEntry(f) != nil || !bytes.Equal(read, b) {
return errResourcePlan
}
return originalGuard()
}
proof, e := q.complete(ctx, s, registry)
sessionErr = e
if errors.Is(e, errQualificationCapacity) {
fmt.Fprintln(stderr, "unknown: qualification matrix exceeds 64 KiB capacity; no worker launched")
return 64
}
if e != nil || ctx.Err() != nil {
writeQualificationDiagnostics(stderr, e)
fmt.Fprintln(stderr, "unknown: qualification incomplete or ownership/cleanup blocked; no completed matrix published")
if signals.Err() != nil && managedCancellationOnly(e, signals.Err()) {
return 0
}
return 78
}
fresh, sourceErr := q.source()
if sourceErr != nil || q.guard() != nil || ctx.Err() != nil {
fmt.Fprintln(stderr, "unknown: source changed before publication")
return 78
}
for _, run := range proof.matrix.Runs {
if run.Config != fresh.fingerprint {
fmt.Fprintln(stderr, "unknown: source changed before publication")
return 78
}
}
if s.saveCompleted(proof) != nil {
fmt.Fprintln(stderr, "blocked: completed matrix publication refused")
return 78
}
return resourcePlanProposal(config, stdout, stderr, d)
}
@@ -0,0 +1,133 @@
package main
import (
"io"
"os"
"path/filepath"
"syscall"
)
// Only the packaged nfq2 alias is followed. All target parents and the final
// binary remain no-symlink, root-owned and not group/other writable. As with
// the runtime contract, a malicious root racing checks is outside the boundary.
type qualificationBinary struct {
root, path string
file *os.File
info, alias os.FileInfo
hash string
}
func qualificationTrusted(info os.FileInfo) bool {
st, ok := info.Sys().(*syscall.Stat_t)
return ok && st.Uid == 0 && info.Mode().Perm()&0022 == 0
}
func qualificationParents(path, boundary string) error {
for dir := filepath.Dir(path); ; dir = filepath.Dir(dir) {
info, e := os.Lstat(dir)
if e != nil || !info.IsDir() || !qualificationTrusted(info) {
return errResourcePlan
}
if dir == boundary {
return nil
}
if dir == "/" {
return errResourcePlan
}
}
}
func qualificationEnginePath(root string) (string, os.FileInfo, error) {
alias := filepath.Join(root, "nfq2/nfqws2")
boundary := root // Private fixture seam; production also checks /opt and /.
if root == "/opt/zapret2" {
boundary = "/"
}
if qualificationParents(alias, boundary) != nil {
return "", nil, errResourcePlan
}
info, e := os.Lstat(alias)
if e != nil || info.Mode()&os.ModeSymlink == 0 {
return "", nil, errResourcePlan
}
st, ok := info.Sys().(*syscall.Stat_t)
if !ok || st.Uid != 0 {
return "", nil, errResourcePlan
}
link, e := os.Readlink(alias)
if e != nil {
return "", nil, errResourcePlan
}
// Check the raw spelling before Join can erase symlink/.. traversal. Only
// the platform observed in the pinned qualification package is supported.
if link != "../binaries/linux-x86_64/nfqws2" {
return "", nil, errResourcePlan
}
path := filepath.Join(root, "binaries/linux-x86_64/nfqws2")
if qualificationParents(path, boundary) != nil {
return "", nil, errResourcePlan
}
return path, info, nil
}
func qualificationBinaryHash(f *os.File) (os.FileInfo, string, error) {
info, e := f.Stat()
if e != nil || !info.Mode().IsRegular() || !qualificationTrusted(info) || info.Mode().Perm()&0111 == 0 || info.Size() > 32<<20 {
return nil, "", errResourcePlan
}
b, e := io.ReadAll(io.NewSectionReader(f, 0, 32<<20+1))
if e != nil || int64(len(b)) != info.Size() {
return nil, "", errResourcePlan
}
return info, planDigest(b), nil
}
func openQualificationEngine(root string) (*qualificationBinary, error) {
path, alias, e := qualificationEnginePath(root)
if e != nil {
return nil, e
}
fd, e := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
if e != nil {
return nil, errResourcePlan
}
f := os.NewFile(uintptr(fd), path)
info, hash, e := qualificationBinaryHash(f)
b := &qualificationBinary{root: root, path: path, alias: alias, file: f, info: info, hash: hash}
if e != nil || b.validate() != nil {
f.Close()
return nil, errResourcePlan
}
return b, nil
}
func (b *qualificationBinary) validate() error {
path, alias, e := qualificationEnginePath(b.root)
if e != nil || path != b.path || !os.SameFile(alias, b.alias) {
return errResourcePlan
}
current, e := os.Lstat(path)
if e != nil || !current.Mode().IsRegular() || !os.SameFile(current, b.info) || !qualificationTrusted(current) {
return errResourcePlan
}
info, hash, e := qualificationBinaryHash(b.file)
if e != nil || !os.SameFile(info, b.info) || hash != b.hash {
return errResourcePlan
}
return nil
}
func (b *qualificationBinary) running(procExe string) error {
// /proc/PID/exe is a kernel-owned link, not a catalogue path. Compare the
// opened running inode and bytes, never the alias spelling from readlink.
f, e := os.Open(procExe)
if e != nil {
return errResourcePlan
}
defer f.Close()
info, hash, e := qualificationBinaryHash(f)
if e != nil || !os.SameFile(info, b.info) || hash != b.hash {
return errResourcePlan
}
return nil
}
@@ -0,0 +1,265 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"testing"
)
func qualificationBinaryFixture(t *testing.T) (string, string, string) {
t.Helper()
if os.Geteuid() != 0 {
t.Skip("root-owned package fixture requires root; run with sudo")
}
root := t.TempDir()
for _, dir := range []string{"nfq2", "binaries/linux-x86_64", "lua"} {
if e := os.MkdirAll(filepath.Join(root, dir), 0755); e != nil {
t.Fatal(e)
}
}
path := filepath.Join(root, "binaries/linux-x86_64/nfqws2")
for _, name := range []string{path, filepath.Join(root, "lua/zapret-lib.lua"), filepath.Join(root, "lua/zapret-antidpi.lua")} {
if e := os.WriteFile(name, []byte("fixed package bytes"), 0755); e != nil {
t.Fatal(e)
}
}
alias := filepath.Join(root, "nfq2/nfqws2")
if e := os.Symlink("../binaries/linux-x86_64/nfqws2", alias); e != nil {
t.Fatal(e)
}
return root, path, alias
}
func TestQualificationPackagedAlias(t *testing.T) {
root, path, _ := qualificationBinaryFixture(t)
b, e := openQualificationEngine(root)
if e != nil {
t.Fatal(e)
}
defer b.file.Close()
want := planDigest([]string{planDigest([]byte("fixed package bytes")), planDigest([]byte("fixed package bytes")), planDigest([]byte("fixed package bytes"))})
if hash, e := b.filesHash(); e != nil || hash != want || b.path != path {
t.Fatal("packaged alias hash", hash, e)
}
proc := filepath.Join(root, "proc-exe")
if e := os.Symlink(path, proc); e != nil {
t.Fatal(e)
}
if b.running(proc) != nil {
t.Fatal("canonical running inode rejected")
}
other := filepath.Join(root, "other")
if e := os.WriteFile(other, []byte("fixed package bytes"), 0755); e != nil {
t.Fatal(e)
}
if b.running(other) == nil {
t.Fatal("same hash, different running inode accepted")
}
}
func TestQualificationPackagedAliasRejects(t *testing.T) {
for _, mode := range []string{"escape", "final-symlink", "writable", "untrusted", "parent-writable", "parent-symlink", "missing"} {
t.Run(mode, func(t *testing.T) {
root, path, alias := qualificationBinaryFixture(t)
var e error
switch mode {
case "escape":
e = os.Remove(alias)
if e == nil {
e = os.Symlink("/bin/nfqws2", alias)
}
case "final-symlink":
e = os.Remove(path)
if e == nil {
e = os.Symlink("/bin/true", path)
}
case "writable":
e = os.Chmod(path, 0775)
case "untrusted":
e = os.Chown(path, 1000, 1000)
case "parent-writable":
e = os.Chmod(filepath.Dir(path), 0777)
case "parent-symlink":
dir := filepath.Dir(path)
e = os.Rename(dir, dir+"-real")
if e == nil {
e = os.Symlink(dir+"-real", dir)
}
case "missing":
e = os.Remove(path)
}
if e != nil {
t.Fatal(e)
}
if b, e := openQualificationEngine(root); e == nil {
b.file.Close()
t.Fatal("unsafe package accepted")
}
})
}
}
func TestQualificationPackagedAliasRawSpelling(t *testing.T) {
for _, link := range []string{
"../binaries/linux-x86_64/./nfqws2",
"../binaries//linux-x86_64/nfqws2",
"../binaries/linux-x86_64/../linux-x86_64/nfqws2",
"../binaries/unsupported/nfqws2",
"../binaries/linux-x86_64/nested/nfqws2",
} {
t.Run(link, func(t *testing.T) {
root, _, alias := qualificationBinaryFixture(t)
if e := os.Remove(alias); e != nil {
t.Fatal(e)
}
if e := os.Symlink(link, alias); e != nil {
t.Fatal(e)
}
if b, e := openQualificationEngine(root); e == nil {
b.file.Close()
t.Fatal("non-package spelling accepted")
}
})
}
}
func TestQualificationConcealedTraversalBeforeWorker(t *testing.T) {
root, _, alias := qualificationBinaryFixture(t)
attacker := t.TempDir()
marker := filepath.Join(attacker, "executed")
for _, dir := range []string{"subdir", "linux-x86_64"} {
if e := os.Mkdir(filepath.Join(attacker, dir), 0755); e != nil {
t.Fatal(e)
}
}
outside := filepath.Join(attacker, "linux-x86_64/nfqws2")
if e := os.WriteFile(outside, []byte(fmt.Sprintf("#!/bin/sh\n: > %q\n", marker)), 0755); e != nil {
t.Fatal(e)
}
if e := os.Symlink(filepath.Join(attacker, "subdir"), filepath.Join(root, "binaries/redirect")); e != nil {
t.Fatal(e)
}
if e := os.Remove(alias); e != nil {
t.Fatal(e)
}
if e := os.Symlink("../binaries/redirect/../linux-x86_64/nfqws2", alias); e != nil {
t.Fatal(e)
}
resolved, e := os.Stat(alias)
outsideInfo, outsideErr := os.Stat(outside)
if e != nil || outsideErr != nil || !os.SameFile(resolved, outsideInfo) {
t.Fatal("fixture must resolve outside package", e, outsideErr)
}
q, s, registry, events, calls := qualifierFixture(t)
q.filesHash = func() (string, error) {
b, e := openQualificationEngine(root)
if e != nil {
return "", e
}
defer b.file.Close()
return b.filesHash()
}
starts := 0
q.runtime.start = func([]string) (managedAssetWorker, error) {
starts++
if e := exec.Command(alias).Run(); e != nil {
t.Fatal(e)
}
return managedAssetWorker{}, errResourcePlan
}
proof, e := q.complete(context.Background(), s, registry)
if !errors.Is(e, errResourcePlan) || starts != 0 || *calls != 0 || len(*events) != 0 || proof.seal != "" {
t.Fatalf("traversal rejection too late: err=%v starts=%d calls=%d events=%v", e, starts, *calls, *events)
}
if _, e := os.Stat(marker); !errors.Is(e, os.ErrNotExist) {
t.Fatal("outside executable ran", e)
}
}
func TestQualificationPackagedAliasProcExe(t *testing.T) {
root, path, alias := qualificationBinaryFixture(t)
bytes, e := os.ReadFile("/bin/sleep")
if e != nil {
t.Fatal(e)
}
if e := os.WriteFile(path, bytes, 0755); e != nil {
t.Fatal(e)
}
b, e := openQualificationEngine(root)
if e != nil {
t.Fatal(e)
}
defer b.file.Close()
cmd := exec.Command(alias, "60")
if e := cmd.Start(); e != nil {
t.Fatal(e)
}
defer func() { cmd.Process.Kill(); cmd.Wait() }()
if b.running(fmt.Sprintf("/proc/%d/exe", cmd.Process.Pid)) != nil {
t.Fatal("real packaged worker inode rejected")
}
if _, e := b.filesHash(); e != nil {
t.Fatal("valid pinned alias changed after start", e)
}
if b.running("/proc/self/exe") == nil {
t.Fatal("foreign running executable accepted")
}
}
func TestQualificationPackagedAliasMutationBeforeStart(t *testing.T) {
for _, mode := range []string{"resolution", "alias-replaced", "binary-replaced", "bytes", "lua-symlink", "lua-writable"} {
t.Run(mode, func(t *testing.T) {
root, path, alias := qualificationBinaryFixture(t)
b, e := openQualificationEngine(root)
if e != nil {
t.Fatal(e)
}
defer b.file.Close()
if _, e := b.filesHash(); e != nil {
t.Fatal(e)
}
switch mode {
case "resolution", "alias-replaced":
link := "../binaries/linux-x86_64/nfqws2"
if mode == "resolution" {
link = "../binaries/other/nfqws2"
if e := os.Mkdir(filepath.Join(root, "binaries/other"), 0755); e != nil {
t.Fatal(e)
}
if e := os.WriteFile(filepath.Join(root, "binaries/other/nfqws2"), []byte("fixed package bytes"), 0755); e != nil {
t.Fatal(e)
}
}
e = os.Rename(alias, alias+"-old")
if e == nil {
e = os.Symlink(link, alias)
}
case "binary-replaced":
e = os.Rename(path, path+"-old")
if e == nil {
e = os.WriteFile(path, []byte("fixed package bytes"), 0755)
}
case "bytes":
e = os.WriteFile(path, []byte("mutated package"), 0755)
case "lua-symlink":
lua := filepath.Join(root, "lua/zapret-lib.lua")
e = os.Remove(lua)
if e == nil {
e = os.Symlink(path, lua)
}
case "lua-writable":
e = os.Chmod(filepath.Join(root, "lua/zapret-lib.lua"), 0666)
}
if e != nil {
t.Fatal(e)
}
if _, e := b.filesHash(); e == nil {
t.Fatal("changed package accepted before start")
}
})
}
}
@@ -0,0 +1,178 @@
package main
import (
"bytes"
"context"
"errors"
"io"
"os"
"strconv"
"strings"
"syscall"
"time"
)
const qualificationCompletionName = "qualification-completed.json"
const qualificationCompletionDir = "/run/zapret-autotune-ss"
// This acknowledges controller cleanup, NOT candidate qualification. It cannot
// authorize a PASS, matrix write or acceptance of ordinary monitor exit 78.
type qualificationCompletion struct {
Version int `json:"version"`
PID int `json:"pid"`
Code int `json:"code"`
CleanupComplete bool `json:"cleanup_complete"`
}
type qualificationCleanupFailure struct{ Err error }
func (e *qualificationCleanupFailure) Error() string { return "qualification cleanup unverified" }
func (e *qualificationCleanupFailure) Unwrap() error { return e.Err }
func qualificationCompletionStore(path string, key []byte) (*resourcePlanStore, error) {
if managedDirectory(path) != nil || len(key) != 32 {
return nil, errResourcePlan
}
fd, e := syscall.Open(path, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
if e != nil {
return nil, e
}
s := &resourcePlanStore{files: &resourceMonitorStore{path: path, dir: os.NewFile(uintptr(fd), "qualification completion directory"), key: append([]byte(nil), key...)}, held: map[string]*os.File{}, snapshots: map[string][]byte{}}
f, b, e := s.files.openEntry(qualificationCompletionName)
if e == nil {
s.held[qualificationCompletionName], s.snapshots[qualificationCompletionName] = f, b
} else if !errors.Is(e, os.ErrNotExist) {
s.close()
return nil, errResourcePlan
}
return s, nil
}
func writeQualificationCompletion(path string, key []byte, pid, code int) error {
if pid <= 1 || (code != 0 && code != 78) {
return errResourcePlan
}
s, e := qualificationCompletionStore(path, key)
if e != nil {
return e
}
defer s.close()
return s.writeAuthenticated(qualificationCompletionName, qualificationCompletion{Version: 1, PID: pid, Code: code, CleanupComplete: true})
}
func readQualificationCompletion(path string, key []byte, pid, code int) error {
s, e := qualificationCompletionStore(path, key)
if e != nil {
return e
}
defer s.close()
var ack qualificationCompletion
if s.readAuthenticated(qualificationCompletionName, &ack) != nil || ack.Version != 1 || ack.PID != pid || pid <= 1 || ack.Code != code || (code != 0 && code != 78) || !ack.CleanupComplete {
return errResourcePlan
}
return nil
}
func qualificationCleanupVerified(err error, restore func(context.Context) error) bool {
var ownedCleanup *managedAssetCleanupFailure
var restoration *qualificationCleanupFailure
if errors.As(err, &ownedCleanup) || errors.As(err, &restoration) {
return false
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
return restore(ctx) == nil
}
// Same cleanup gate and ordering as the controller's completion defer. Keeping
// operations separate makes ACK refusal diagnosable without exposing errors.
func qualificationFinalize(err error, restore func(context.Context) error, guard, ack func() error) error {
var restoreErr error
if !qualificationCleanupVerified(err, func(ctx context.Context) error {
restoreErr = restore(ctx)
return restoreErr
}) {
if restoreErr == nil {
return qualificationOperation(qualificationPredicate(errResourcePlan, reasonPriorCleanupVeto), stageRestore)
}
return qualificationOperation(restoreErr, stageRestore)
}
if e := guard(); e != nil {
return qualificationOperation(e, stageFinalStoreGuard)
}
return qualificationOperation(ack(), stageACKWrite)
}
func qualificationCompletionCLI(args []string, d managedDeps) int {
if len(args) != 2 || os.Geteuid() != 0 || os.Getppid() != 1 {
return 78
}
pid, e := strconv.Atoi(args[0])
code, ce := strconv.Atoi(args[1])
if e != nil || ce != nil || strconv.Itoa(pid) != args[0] || strconv.Itoa(code) != args[1] {
return 78
}
cmd, e := os.ReadFile("/proc/1/cmdline")
if e != nil || !bytes.Equal(cmd, []byte("/bin/sh\x00/opt/managed-target-entrypoint.sh\x00")) {
return 78
}
env, e := os.ReadFile("/proc/1/environ")
if e != nil {
return 78
}
values := map[string]string{}
for _, field := range strings.Split(string(env), "\x00") {
k, v, _ := strings.Cut(field, "=")
values[k] = v
}
if values["ZAPRET_AUTOTUNE_QUALIFY_CONFIG"] == "" || values["ZAPRET_AUTOTUNE_STRATEGY_CONFIG"] == "" {
return 78
}
if _, e := managedRuntimeGuard(d); e != nil {
return 78
}
lock, e := acquireManagedMonitorSingleton(d)
if e != nil {
return 78
}
defer lock.close()
s, e := openResourcePlanStore(d.directory, false)
if e != nil {
return 78
}
defer s.close()
if s.protectBaseline() != nil || qualificationBaseline(d.directory) != nil {
return 78
}
dns, e := parseManagedDNSConfig(d.lookup)
if e != nil || verifyResourceDNSRuntime(dns) != nil || qualificationNoWorker() != nil {
return 78
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
r := newManagedAssetRuntime(d)
tables, e := r.nft(ctx, "list tables\n")
if e != nil || strings.Contains(tables, "table inet "+managedAssetTable) {
return 78
}
// The private session ACK ties completion to this PID 1 launch, not a stale
// target-volume status. The completion itself lives only on this boot's tmpfs.
dir, e := os.Open(qualificationCompletionDir)
if e != nil {
return 78
}
defer dir.Close()
f, e := resourcePrivateFile(dir, "monitor-ready")
if e != nil {
return 78
}
defer f.Close()
b, e := io.ReadAll(io.LimitReader(f, 32))
if e != nil || string(b) != strconv.Itoa(pid)+"\n" {
return 78
}
if readQualificationCompletion(qualificationCompletionDir, s.files.key, pid, code) != nil || s.guard() != nil {
return 78
}
return 0
}
@@ -0,0 +1,102 @@
package main
import (
"bytes"
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
)
func TestResourceQualifierCompletionOwnership(t *testing.T) {
dir := t.TempDir()
if e := os.Chmod(dir, 0700); e != nil {
t.Fatal(e)
}
key := bytes.Repeat([]byte{9}, 32)
if e := writeQualificationCompletion(dir, key, 123, 78); e != nil {
t.Fatal(e)
}
if e := readQualificationCompletion(dir, key, 123, 78); e != nil {
t.Fatal(e)
}
for _, mismatch := range []struct {
pid, code int
key []byte
}{{124, 78, key}, {123, 0, key}, {123, 78, bytes.Repeat([]byte{8}, 32)}} {
if readQualificationCompletion(dir, mismatch.key, mismatch.pid, mismatch.code) == nil {
t.Fatal("accepted foreign launch/status/key")
}
}
if writeQualificationCompletion(dir, key, 123, 78) == nil {
t.Fatal("clobbered write-once completion")
}
path := filepath.Join(dir, qualificationCompletionName)
raw, e := os.ReadFile(path)
if e != nil {
t.Fatal(e)
}
if e := os.WriteFile(path, bytes.Replace(raw, []byte(`"cleanup_complete":true`), []byte(`"cleanup_complete":false`), 1), 0600); e != nil {
t.Fatal(e)
}
if readQualificationCompletion(dir, key, 123, 78) == nil {
t.Fatal("accepted forged completion")
}
}
func TestResourceQualifierCompletionCleanupVeto(t *testing.T) {
for _, tc := range []struct {
err error
safe bool
}{
{nil, true}, {errors.New("source changed, proof UNKNOWN"), true}, {context.Canceled, true},
{&managedRecoveryFatal{Err: &managedAssetCleanupFailure{Err: errors.New("worker join failed")}}, false},
{errors.Join(context.Canceled, &qualificationCleanupFailure{Err: errors.New("baseline restoration failed")}), false},
} {
called := false
got := qualificationCleanupVerified(tc.err, func(context.Context) error { called = true; return nil })
if got != tc.safe || called != tc.safe {
t.Fatalf("cleanup ACK=%v called=%v for %T", got, called, tc.err)
}
}
if qualificationCleanupVerified(nil, func(context.Context) error { return errors.New("live baseline guard failed") }) {
t.Fatal("acknowledged unverified cleanup")
}
}
func TestResourceQualifierCompletionForeignFile(t *testing.T) {
dir := t.TempDir()
os.Chmod(dir, 0700)
path := filepath.Join(dir, qualificationCompletionName)
if e := os.WriteFile(path, []byte("FOREIGN"), 0600); e != nil {
t.Fatal(e)
}
key := bytes.Repeat([]byte{9}, 32)
if writeQualificationCompletion(dir, key, 123, 78) == nil || readQualificationCompletion(dir, key, 123, 78) == nil {
t.Fatal("adopted foreign completion")
}
b, e := os.ReadFile(path)
if e != nil || string(b) != "FOREIGN" {
t.Fatal("changed foreign completion")
}
}
func TestResourceQualifierOwnedCleanupFailureCannotAcknowledge(t *testing.T) {
q, s, r, _, _ := qualifierFixture(t)
nft := q.runtime.nft
q.runtime.nft = func(ctx context.Context, input string) (string, error) {
if strings.HasPrefix(input, "delete table") {
return "", errors.New("owned nft cleanup failed")
}
return nft(ctx, input)
}
proof, err := q.complete(context.Background(), s, r)
if err == nil || proof.seal != "" {
t.Fatal("unsafe lifecycle sealed proof")
}
if qualificationCleanupVerified(err, q.restore) {
t.Fatal("unsafe owned cleanup authorized completed ACK")
}
}
Loaded 100 of 125 files, more files were not shown because too many files have changed in this diff. Show more