diff --git a/GOOGLEVIDEO_DISCOVERY.md b/GOOGLEVIDEO_DISCOVERY.md new file mode 100644 index 0000000..bf0c46c --- /dev/null +++ b/GOOGLEVIDEO_DISCOVERY.md @@ -0,0 +1,34 @@ +# Bounded representative Googlevideo discovery + +The YouTube default inventory is the configured primary plus `aqz-KE-bpKQ` and +`jNQXAC9IVRw`, deduplicated. Explicit configured inventories override it. The +inventory (one to five IDs) is queried freshly +in round-robin order for up to three rounds, bounded by three minutes. No new +manual CDN addresses, host rewriting or IP pinning is added. +With only one configured video, diversity depends on that video's real Player +backend rotation; if it does not rotate, qualification remains unresolved. + +Discovery deduplicates actual ordinary signed MP4 media hosts, refetches expired +inventory and selects three hosts passing the existing root/prefix diversity +gate. URLs need twenty minutes remaining for the three sequential owned native +warmups and verified TLS transfers. URLs stay transient; unresolved diagnostics +contain only attempt/host and rejection counts. Discovery is not media proof. + +Production adaptive recovery uses this collector. Existing experimental suffixes +retain their one-host operational recheck; promotion requires fresh three-host +inventory and the unchanged version-1 owned native proof factory. Insufficient +inventory leaves the committed experimental profile intact and retries after +the configured cooldown, not the exhausted-strategy two-hour pause. Native GGC +scope requires qualified frontend and any supplied asset candidates remain +qualified and nonoverlapping; missing avatar inventory is not its prerequisite. +Apply checks frontend and all three actual media resources in the newly deployed +daemon, including bounded learning and three verified transfers per TLS version. +Whole-service monitoring still reports avatar degradation independently. + +Single-CDN version-2 qualification has been withdrawn. Maximum strategy catalogs, +new video discovery sources and controller stay-alive changes are separate work. +The three-CDN requirement was reaffirmed on 2026-10-13. The common suffix remains +authorized extrapolative coverage; only exact measured endpoints have transfer +evidence. This candidate is offline unit/fixture-verified only: owned factory, +autonomous discovery/promotion and deployed learning still need a scoped actual +three-CDN field test. No remote network checks or deployment accompany this update. diff --git a/MANAGED_TARGET.md b/MANAGED_TARGET.md index 0f761a1..21b99f2 100644 --- a/MANAGED_TARGET.md +++ b/MANAGED_TARGET.md @@ -1,5 +1,38 @@ # Fresh managed-test target contract (isolated acceptance, not production approval) +## Autonomous three-CDN source candidate (2026-10-13): field acceptance pending + +The user's three-CDN requirement is retained; the proposed single-CDN relaxation +was withdrawn. Service-mode discovery needs no manual domains: missing/empty +`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` uses primary `ZAPRET_AUTOTUNE_VIDEO_ID` +(default `dQw4w9WgXcQ`) plus `aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated. +A nonempty JSON array of 1..5 unique VideoIDs overrides this inventory. VideoIDs +are discovery inputs, not proof of distinct CDNs. + +Fresh round-robin Player requests are bounded to three rounds (at most 15 requests) +and three minutes. Qualification selects three actual signed MP4 media hosts, +with at least two roots and two prefixes, and twenty-minute remaining URL TTL. +Only the existing version-1 owned native proof factory authorizes qualification: +post-warmup actual media 3/3 on both TLS protocols for each measured endpoint. +The common `googlevideo.com` suffix is an authorized extrapolative scope, not +measured evidence for every Googlevideo host or cold/uniform behavior. + +Experimental promotion uses the same proof and fresh deployed three-host media +checks before committing apply. Insufficient inventory preserves the current +experiment and uses configured retry cooldown; genuine exhausted strategy search +still waits at least two hours. Qualified frontend is required; avatar success +does not block this media stage, but whole-service health remains DEGRADED until +actual avatar checks pass. Legacy service-off/exact-host behavior is unchanged. + +This source candidate has only offline unit/fixture verification, not a newly +deployed image or field-qualified autonomous promotion. The 2026-10-12 measured +results below remain historical exact-endpoint evidence. Build from the committed +source archive and field-test the owned factory, three actual CDNs, deployed +relearning/apply, rollback and whole-service health before acceptance. Maximum +strategy collection and PID1 global-failure stay-alive remain unfinished future +increments. Arbitrary declarative service configuration is future work; no new +generic policy framework is supplied. See [GOOGLEVIDEO_DISCOVERY.md](GOOGLEVIDEO_DISCOVERY.md). + `managed init|reconcile|status|active-ids|apply` is exclusively for a **new dedicated container** which owns its entire config. The existing/dev/production container must refuse these commands. A role env or CLI-created identity is not enough. Provisioning diff --git a/PROGRESS.md b/PROGRESS.md index c247a40..d144046 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,9 +1,40 @@ # Прогресс реализации -Срез на **2026-10-12**. Этот файл описывает фактическое состояние кода, а не весь объём требований из `Requirements.md` и `Architecture.md`. +Срез на **2026-10-13**. Этот файл описывает фактическое состояние кода, а не весь объём требований из `Requirements.md` и `Architecture.md`. ## Текущий статус +### Autonomous three-CDN source candidate (2026-10-13): offline only + +- User reaffirmed **three actual CDN checks**, discovered by the program without + manual domains; the earlier single-CDN/version-2 proposal is withdrawn. Missing + or empty service-mode VideoID inventory selects primary (default `dQw4w9WgXcQ`) + plus `aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated. A nonempty 1..5 unique-ID + override is respected; three videos do not prove three CDN hosts. +- Fresh discovery runs up to three round-robin rounds, at most 15 requests in + three minutes; it deduplicates actual signed ordinary MP4 hosts, requires + twenty-minute remaining URL TTL, and selects three hosts across at least two + roots and two prefixes. The unchanged version-1 owned native proof factory + requires actual post-warmup media 3/3 on both TLS protocols at each endpoint. + Common Googlevideo suffix authorization is extrapolative, not all-host evidence. +- Existing experiments can now seek promotion through that shared qualification + path, then frontend and fresh deployed three-host media checks under the current + transaction/ownership guards. Insufficient inventory leaves the experiment + committed and retries at configured cooldown; genuine strategy exhaustion keeps + its minimum two-hour pause. Missing avatar does not block the media stage; + whole-service monitoring remains DEGRADED until actual avatar health passes. + Legacy service-off/exact-host behavior and cancellation safety remain unchanged. +- This is an offline unit/fixture-tested source candidate, **not deployed or + field-accepted**. Renewed actual three-CDN discovery, owned factory proof, + deployed learning/apply/rollback and avatar qualification remain field gates. + Earlier statements below that automatic qualification is unimplemented describe + those historical RCs, not this new source; their measured results are not new + factory evidence. No remote network checks or deployment accompany this update. +- Maximum strategy collection and PID1 global-failure stay-alive are still future + increments; broad maximum-profile live staging is deferred. Arbitrary declarative + service configuration remains a future note, not a new framework. Details: + [GOOGLEVIDEO_DISCOVERY.md](GOOGLEVIDEO_DISCOVERY.md). + ### Final managed RC field acceptance (2026-10-12): DEGRADED, not READY - Supplied final field results, not network checks rerun by this docs update: source `fd25cb89398300bb9849db4b5511af75bf86eccf`, correct managed image `sha256:73e955e7fdc6efaef88c62ebe4f7afeab71c29d29ba3c88b2a2b7547bb647da5`, active container `a4e845a6d7…`, official managed PID 1, isolated SOCKS `127.0.0.1:11082`. The earlier `37823…` prototype image is **not active** and is not this acceptance image. Private sanitized proof: `/home/agent/zapret2-autotune-rc-fd25cb8/proof/final.safe.json`. The later docs commit is not the tested image source. diff --git a/cmd/zapret-autotune/config.go b/cmd/zapret-autotune/config.go index ab5e9e0..21323e1 100644 --- a/cmd/zapret-autotune/config.go +++ b/cmd/zapret-autotune/config.go @@ -75,9 +75,17 @@ func managedGooglevideoConfig(lookup func(string) (string, bool), primary string if mode == QualificationAdaptive && !service { return "", nil, fmt.Errorf("adaptive googlevideo requires ZAPRET_AUTOTUNE_YOUTUBE_SERVICE=1") } - // Only the configured primary is sampled by default; no implicit public playlist. ids := []string{primary} - if raw, ok := lookup("ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS"); ok { + // YouTube-specific public discovery samples, not CDN addresses. Explicit + // inventories below override these defaults; health/discovery primary stays put. + if service { + for _, id := range []string{"aqz-KE-bpKQ", "jNQXAC9IVRw"} { + if id != primary { + ids = append(ids, id) + } + } + } + if raw, ok := lookup("ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS"); ok && raw != "" { if json.Unmarshal([]byte(raw), &ids) != nil || len(ids) == 0 || len(ids) > 5 { return "", nil, fmt.Errorf("GOOGLEVIDEO_VIDEO_IDS requires a JSON array of 1..5 video IDs") } diff --git a/cmd/zapret-autotune/managed_compose_config_test.go b/cmd/zapret-autotune/managed_compose_config_test.go new file mode 100644 index 0000000..45ff9a4 --- /dev/null +++ b/cmd/zapret-autotune/managed_compose_config_test.go @@ -0,0 +1,65 @@ +package main + +import ( + "encoding/json" + "os" + "reflect" + "testing" +) + +// check_config.py provides sanitized actual `docker compose config` output. +// This bridges deployment interpolation to the production config parser offline. +func TestManagedGooglevideoComposeForwarding(t *testing.T) { + path := os.Getenv("AUTOTUNE_COMPOSE_CONFIG_FIXTURE") + if path == "" { + t.Skip("run deploy/managed-target/check_config.py for Compose fixture") + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal("compose fixture unavailable") + } + var cases []struct { + Environment map[string]string `json:"environment"` + IDs []string `json:"ids"` + } + if json.Unmarshal(data, &cases) != nil || len(cases) < 6 { + t.Fatal("invalid compose fixture") + } + for i, c := range cases { + lookup := func(key string) (string, bool) { value, ok := c.Environment[key]; return value, ok } + mode, ids, err := managedGooglevideoConfig(lookup, c.Environment["ZAPRET_AUTOTUNE_VIDEO_ID"]) + if err != nil || string(mode) != c.Environment["ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE"] || !reflect.DeepEqual(ids, c.IDs) { + t.Fatalf("compose factory case %d mismatch", i) + } + } +} + +func TestManagedGooglevideoBlankInventoryDefault(t *testing.T) { + for _, value := range []string{"", " ", "\n", "[]", "bad", `["aaaaaaaaaaa"]`} { + lookup := func(key string) (string, bool) { + switch key { + case "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE": + return "1", true + case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE": + return "adaptive", true + case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS": + return value, true + } + return "", false + } + mode, ids, err := managedGooglevideoConfig(lookup, "dQw4w9WgXcQ") + if value != "" && value != `["aaaaaaaaaaa"]` { + if err == nil { + t.Fatal("invalid nonempty inventory accepted") + } + continue + } + want := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"} + if value != "" { + want = []string{"aaaaaaaaaaa"} + } + if err != nil || mode != QualificationAdaptive || !reflect.DeepEqual(ids, want) { + t.Fatal("inventory default/override mismatch") + } + } +} diff --git a/cmd/zapret-autotune/managed_config.go b/cmd/zapret-autotune/managed_config.go index 4a87982..f8a8563 100644 --- a/cmd/zapret-autotune/managed_config.go +++ b/cmd/zapret-autotune/managed_config.go @@ -64,6 +64,7 @@ func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope, seen := map[int64]bool{} assets := map[string]bool{} var otherScopes []string + nativeSuffix := false for _, id := range ids { if id <= 0 || seen[id] { return nil, fmt.Errorf("duplicate candidate %d", id) @@ -110,6 +111,7 @@ func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope, } } if c.Group == "googlevideo" && scope == GooglevideoScopeSuffix { + nativeSuffix = c.Mode == QualificationAdaptive if version < 7 || validateGooglevideoSuffixEvidencePolicy(db, c, fresh) != nil { return nil, fmt.Errorf("suffix requires representative qualified hosts and verified media evidence") } @@ -125,8 +127,8 @@ func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope, groups[c.Group] = true profiles = append(profiles, "--filter-tcp=443 --filter-l7=tls --hostlist-domains="+strings.Join(domains, ",")+" "+c.Args) } - if scope == GooglevideoScopeSuffix && (!groups["googlevideo"] || len(assets) != 2) { - return nil, fmt.Errorf("suffix requires googlevideo and both qualified asset profiles") + if scope == GooglevideoScopeSuffix && (!groups["googlevideo"] || nativeSuffix && !groups["frontend"] || !nativeSuffix && len(assets) != 2) { + return nil, fmt.Errorf("suffix requires qualified googlevideo and frontend (legacy cold scope requires both assets)") } if scope == GooglevideoScopeExperimental && (!groups["googlevideo"] || !groups["frontend"] || len(assets) == 0) { return nil, fmt.Errorf("experimental suffix requires frontend and existing qualified assets") diff --git a/cmd/zapret-autotune/managed_experimental_recovery.go b/cmd/zapret-autotune/managed_experimental_recovery.go index 9749b1c..8c9ed47 100644 --- a/cmd/zapret-autotune/managed_experimental_recovery.go +++ b/cmd/zapret-autotune/managed_experimental_recovery.go @@ -62,6 +62,15 @@ func recoverManagedExperimental(ctx context.Context, d managedRecoveryDeps) (int if err == nil { err = d.recheckExperimental(ctx, c, r, d.transaction) } + if err == nil && d.mediaSamples != nil && d.verifyGooglevideo != nil { + // One-host operational health never promotes the experiment. Only + // fresh representative inventory and a completed owned proof may. + qualified, err := qualifyManagedGooglevideo(ctx, d) + if err != nil { + return id, err + } + return qualified, nil + } return id, managedRecoveryContextError(ctx, err) } } diff --git a/cmd/zapret-autotune/managed_googlevideo_apply.go b/cmd/zapret-autotune/managed_googlevideo_apply.go new file mode 100644 index 0000000..9f9ae0a --- /dev/null +++ b/cmd/zapret-autotune/managed_googlevideo_apply.go @@ -0,0 +1,87 @@ +package main + +import ( + "context" + "crypto/sha256" + "database/sql" + "errors" + "fmt" + "os" + "reflect" +) + +func newManagedGooglevideoApplyRecheck(target managedDeps, db *sql.DB) func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error { + r := newManagedAssetRuntime(target) + guard := r.guard + r.guard = func() error { + if err := guard(); err != nil { + return err + } + lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua") + if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash { + return errors.New("unverified native automation Lua source") + } + return nil + } + return managedGooglevideoApplyRecheck(r, db) +} + +// Runs inside Apply's lock: never ActiveCandidates/Reconcile/restart here. +// The newly reloaded daemon learns in-place; a failure rolls back via Apply. +func managedGooglevideoApplyRecheck(r managedAssetRuntime, db *sql.DB) func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error { + return func(ctx context.Context, c storedCandidate, resources []mediaResource, m *managedTransaction) error { + j, err := m.load() + if err != nil { + return &managedRecoveryFatal{Err: err} + } + found := false + for _, id := range j.PendingCandidates { + found = found || id == c.ID + } + var hosts []string + for _, resource := range resources { + hosts = append(hosts, resource.host) + } + if j.State != "prepared" || j.PendingScope != GooglevideoScopeSuffix || !found || c.Mode != QualificationAdaptive || !reflect.DeepEqual(hosts, c.Endpoints) || validateGooglevideoSuffixEvidence(db, c) != nil { + return &managedRecoveryFatal{Err: errors.New("deployed media candidate ownership mismatch")} + } + guard := func() error { + if err := r.guard(); err != nil { + return err + } + if err := m.owns(j, false); err != nil { + return err + } + if err := validateManagedRefs(db, j.PendingCandidates, j.Next, j.PendingScope); err != nil { + return err + } + b, err := m.read(m.Name) + if err != nil { + return err + } + if managedHash(b) != j.Next { + return errManagedDrift + } + return nil + } + for _, resource := range resources { + err := recheckDeployedExperimental(ctx, resource, guard, + func(context.Context) error { return guard() }, // already reloaded; warm current daemon only + func(context.Context) (mediaResource, error) { return mediaResource{}, errCandidateNonpass }, + func(ctx context.Context, resource mediaResource, protocol string) error { + _, _, err := collectMediaTransfer(ctx, r.proxy, resource, protocol, r.curl) + return err + }) + if errors.Is(err, errNoQualifiedStrategy) && !managedRecoveryTimeoutSafe(err) { + var fatal *managedRecoveryFatal + if !errors.As(err, &fatal) { + return errCandidateNonpass + } + } + if err != nil { + return err + } + } + return nil + } +} diff --git a/cmd/zapret-autotune/managed_googlevideo_discovery_test.go b/cmd/zapret-autotune/managed_googlevideo_discovery_test.go new file mode 100644 index 0000000..ba2c655 --- /dev/null +++ b/cmd/zapret-autotune/managed_googlevideo_discovery_test.go @@ -0,0 +1,191 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "strings" + "testing" + "time" +) + +func samplePlayerBody(host string, expires time.Time) []byte { + u, _ := json.Marshal(fmt.Sprintf("https://%s/videoplayback?expire=%d&sig=private-secret", host, expires.Unix())) + return []byte(fmt.Sprintf(`{"playabilityStatus":{"status":"OK"},"streamingData":{"formats":[{"mimeType":"video/mp4","url":%s}]}}`, u)) +} + +func TestManagedGooglevideoDiscoveryRotatesFreshInventory(t *testing.T) { + ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"} + hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"} + for _, scenario := range []string{"rotating", "unavailable", "expired", "same-root", "unsafe-origin", "invalid-host", "control", "unknown", "mixed-safety", "cancel"} { + t.Run(scenario, func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + calls := 0 + collect := managedGooglevideoSamples(ids, func(ctx context.Context, id string) ([]byte, error) { + if _, ok := ctx.Deadline(); !ok { + t.Fatal("unbounded discovery") + } + if id != ids[calls%3] { + t.Fatal("not round robin") + } + calls++ + host := hosts[(calls-1)%3] + expires := time.Now().Add(time.Hour) + if calls <= 3 { + switch scenario { + case "rotating": + host = hosts[0] + case "unavailable": + return nil, context.DeadlineExceeded + case "expired": + expires = time.Now().Add(5 * time.Minute) + case "unsafe-origin": + host += ":444" + case "invalid-host": + host = "not-a-ggc.googlevideo.com" + case "control": + host += "\n" + case "unknown": + return nil, errors.New("runner unavailable") + case "mixed-safety": + return nil, errors.Join(context.DeadlineExceeded, errors.New("cleanup failed")) + case "cancel": + cancel() + return nil, ctx.Err() + } + } + if scenario == "same-root" { + host = fmt.Sprintf("rr%d---sn-alpha-x.googlevideo.com", calls) + } + return samplePlayerBody(host, expires), nil + }) + resources, err := collect(ctx) + if scenario == "unknown" || scenario == "mixed-safety" { + var fatal *managedRecoveryFatal + if !errors.As(err, &fatal) || calls != 1 || len(resources) != 0 { + t.Fatal(err, calls) + } + return + } + if scenario == "cancel" { + if !errors.Is(err, context.Canceled) || calls != 1 || len(resources) != 0 { + t.Fatal(err, calls) + } + return + } + if scenario == "same-root" { + var insufficient *InsufficientAdaptiveSamples + if !errors.As(err, &insufficient) || insufficient.Attempts != 9 || len(resources) != 0 || strings.Contains(err.Error(), "private-secret") { + t.Fatal(err, calls) + } + return + } + if err != nil || len(resources) != 3 || calls != 6 { + t.Fatal(err, calls, len(resources)) + } + for i, r := range resources { + if r.host != hosts[i] || !strings.Contains(r.url, "private-secret") { + t.Fatal("changed signed inventory") + } + } + }) + } +} + +func TestManagedGooglevideoDiscoveryBoundsAndLateCancellation(t *testing.T) { + ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw", "aaaaaaaaaaa", "bbbbbbbbbbb"} + calls := 0 + resources, err := managedGooglevideoSamples(ids, func(context.Context, string) ([]byte, error) { + calls++ + return []byte(`{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"private-secret"}}`), nil + })(context.Background()) + var insufficient *InsufficientAdaptiveSamples + if !errors.As(err, &insufficient) || calls != 15 || insufficient.Unsupported != 15 || len(resources) != 0 { + t.Fatal(err, calls) + } + for _, invalid := range [][]string{nil, {"bad"}, append(ids, "ccccccccccc")} { + _, err := managedGooglevideoSamples(invalid, func(context.Context, string) ([]byte, error) { t.Fatal("invalid config queried"); return nil, nil })(context.Background()) + var fatal *managedRecoveryFatal + if !errors.As(err, &fatal) { + t.Fatal(err) + } + } + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"} + calls = 0 + resources, err = managedGooglevideoSamples(ids[:3], func(context.Context, string) ([]byte, error) { + host := hosts[calls] + calls++ + if calls == 3 { + cancel() + } + return samplePlayerBody(host, time.Now().Add(time.Hour)), nil + })(ctx) + if !errors.Is(err, context.Canceled) || len(resources) != 0 || calls != 3 { + t.Fatal(err, calls) + } +} + +func TestExperimentalPromotionRequiresRepresentativeOwnedProof(t *testing.T) { + for _, enough := range []bool{false, true} { + t.Run(fmt.Sprint(enough), func(t *testing.T) { + d, front, _ := recoveryFixture(t, true) + r, _, resources, _ := googlevideoRuntimeFixture(t) + d.service = true + d.frontHealth = d.health + thumb := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true) + avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true) + tx, err := d.db.Begin() + if err != nil { + t.Fatal(err) + } + experiment, err := insertStrategyCandidate(tx, "googlevideo", []string{resources[0].host}, managedGooglevideoArgs, nil, QualificationCold) + if err != nil { + _ = tx.Rollback() + t.Fatal(err) + } + if err := tx.Commit(); err != nil { + t.Fatal(err) + } + old := []int64{front, thumb, avatar, experiment} + if err := d.transaction.Apply(d.db, old, GooglevideoScopeExperimental); err != nil { + t.Fatal(err) + } + d.resources = func(context.Context) (managedServiceInventory, error) { + return managedServiceInventory{media: resources[0], images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/sample.jpg", "yt3.ggpht.com": "https://yt3.ggpht.com/sample.jpg"}, proxy: r.proxy, runner: adaptiveImageRunner}, nil + } + d.recheckExperimental = func(context.Context, storedCandidate, mediaResource, *managedTransaction) error { return nil } + d.mediaSamples = func(context.Context) ([]mediaResource, error) { + if enough { + return resources, nil + } + return resources[:1], nil + } + d.verifyGooglevideo = r.verifyGooglevideo + d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db) + id, err := recoverManagedExperimental(context.Background(), d) + if !enough { + if !errors.Is(err, errGooglevideoInsufficientSamples) || id != experiment { + t.Fatal(id, err) + } + assertActiveCandidates(t, d.transaction, d.db, old) + return + } + if err != nil || id == experiment { + t.Fatal(id, err) + } + assertActiveCandidates(t, d.transaction, d.db, []int64{front, thumb, avatar, id}) + p, err := readGooglevideoProof(d.db, id) + if err != nil || p.Version != 1 || len(p.Hosts) != 3 { + t.Fatal(p, err) + } + c, err := readStrategyCandidate(d.db, experiment) + if err != nil || c.Qualified || c.Mode != QualificationCold { + t.Fatal("old row changed", c, err) + } + }) + } +} diff --git a/cmd/zapret-autotune/managed_googlevideo_recovery.go b/cmd/zapret-autotune/managed_googlevideo_recovery.go index 7c9986c..14479a9 100644 --- a/cmd/zapret-autotune/managed_googlevideo_recovery.go +++ b/cmd/zapret-autotune/managed_googlevideo_recovery.go @@ -3,80 +3,139 @@ package main import ( "context" "errors" + "fmt" "sort" "strconv" + "strings" "time" ) // Ordinary unresolved state, not a target/ownership guard failure. -type InsufficientAdaptiveSamples struct{} +type InsufficientAdaptiveSamples struct { + Attempts, DistinctHosts, Unavailable, Unsupported, Rejected int +} -func (*InsufficientAdaptiveSamples) Error() string { - return "DEGRADED: insufficient fresh ordinary signed GGC samples" +func (e *InsufficientAdaptiveSamples) Error() string { + return fmt.Sprintf("DEGRADED: insufficient fresh ordinary signed GGC samples (attempts=%d distinct_hosts=%d unavailable=%d unsupported=%d rejected=%d)", e.Attempts, e.DistinctHosts, e.Unavailable, e.Unsupported, e.Rejected) } func (*InsufficientAdaptiveSamples) Unwrap() []error { - return []error{errGooglevideoInsufficientSamples, errNoQualifiedStrategy} + return []error{errGooglevideoInsufficientSamples} +} + +// Sequential round-robin: at most 15 fresh requests, with no child goroutines. +const managedGooglevideoSampleRounds = 3 +const managedGooglevideoDiscoveryTimeout = 3 * time.Minute + +// Three hosts each need up to 120s warmup + six 35s transfers. Leave margin +// for daemon lifecycle as well; short-lived URLs are refetched, never rewritten. +const managedGooglevideoSampleTTL = 20 * time.Minute + +func freshGooglevideoSample(r mediaResource) bool { + // The same exact GGC hostname syntax as the representative gate; reject bad + // inventory before counting it, without fetching any returned URL here. + prefix, root, ok := strings.Cut(strings.TrimSuffix(r.host, ".googlevideo.com"), "---sn-") + if !ok || !strings.HasSuffix(r.host, ".googlevideo.com") || !validDomain(r.host) || r.host != strings.ToLower(r.host) || root == "" || len(prefix) < 3 || !strings.HasPrefix(prefix, "rr") || strings.Trim(prefix[2:], "0123456789") != "" { + return false + } + u, err := privateResourceURL(r.url, r.host) + if err != nil || u.Path != "/videoplayback" || (r.kind != "video/mp4" && r.kind != "audio/mp4") { + return false + } + q := u.Query() + expires, err := strconv.ParseInt(q.Get("expire"), 10, 64) + return err == nil && expires > time.Now().Add(managedGooglevideoSampleTTL).Unix() && (q.Get("sig") != "" || q.Get("signature") != "" || q.Get("lsig") != "") +} + +// Choose exactly three actual hosts passing the unchanged representative gate. +func representativeGooglevideoSamples(byHost map[string]mediaResource) []mediaResource { + hosts := make([]string, 0, len(byHost)) + for host, r := range byHost { + if freshGooglevideoSample(r) { + hosts = append(hosts, host) + } + } + sort.Strings(hosts) + for i := 0; i < len(hosts); i++ { + for j := i + 1; j < len(hosts); j++ { + for k := j + 1; k < len(hosts); k++ { + if validateGooglevideoHosts([]string{hosts[i], hosts[j], hosts[k]}) == nil { + return []mediaResource{byHost[hosts[i]], byHost[hosts[j]], byHost[hosts[k]]} + } + } + } + } + return nil } func managedGooglevideoSamples(ids []string, request func(context.Context, string) ([]byte, error)) func(context.Context) ([]mediaResource, error) { + ids = append([]string(nil), ids...) return func(ctx context.Context) ([]mediaResource, error) { - byHost := map[string]mediaResource{} + if len(ids) == 0 || len(ids) > 5 || request == nil { + return nil, &managedRecoveryFatal{Err: errors.New("invalid sample collector configuration")} + } for _, id := range ids { - body, err := request(ctx, id) - var fatal *managedRecoveryFatal - if ctx.Err() != nil || errors.As(err, &fatal) { - return nil, managedRecoveryContextError(ctx, err) + if !videoIDPattern.MatchString(id) { + return nil, &managedRecoveryFatal{Err: errors.New("invalid sample video ID")} } - if err != nil { - continue + } + bounded, cancel := context.WithTimeout(ctx, managedGooglevideoDiscoveryTimeout) + defer cancel() + stats := &InsufficientAdaptiveSamples{} + byHost := map[string]mediaResource{} + for round := 0; round < managedGooglevideoSampleRounds; round++ { + for _, id := range ids { + if ctx.Err() != nil { + return nil, managedRecoveryContextError(ctx, ctx.Err()) + } + if bounded.Err() != nil { + return nil, stats + } + stats.Attempts++ + body, err := request(bounded, id) + var fatal *managedRecoveryFatal + if ctx.Err() != nil || errors.As(err, &fatal) { + return nil, managedRecoveryContextError(ctx, err) + } + if err != nil { + if !managedResourceTransferNonpass(err, true, false) && !(bounded.Err() != nil && managedRecoveryTimeoutSafe(err)) { + return nil, &managedRecoveryFatal{Err: err} + } + stats.Unavailable++ + if bounded.Err() != nil { + return nil, stats + } + continue + } + r, _, err := playerMediaResource(body) + if err != nil { + stats.Unavailable++ + continue + } // Missing/SABR is unknown, never fabricated. + if r.url == "" { + stats.Unsupported++ + continue + } + if !freshGooglevideoSample(r) { + stats.Rejected++ + continue + } + byHost[r.host] = r + stats.DistinctHosts = len(byHost) + if ctx.Err() != nil { + return nil, managedRecoveryContextError(ctx, ctx.Err()) + } + if bounded.Err() != nil { + return nil, stats + } + if resources := representativeGooglevideoSamples(byHost); resources != nil { + return resources, nil + } } - r, _, err := playerMediaResource(body) - if err != nil { - continue - } // Missing/SABR is unknown, never fabricated. - u, err := privateResourceURL(r.url, r.host) - if err != nil || u.Path != "/videoplayback" { - continue - } - q := u.Query() - expires, err := strconv.ParseInt(q.Get("expire"), 10, 64) - if err != nil || expires <= time.Now().Unix()+120 || (q.Get("sig") == "" && q.Get("signature") == "" && q.Get("lsig") == "") { - continue - } - byHost[r.host] = r } - hosts := make([]string, 0, len(byHost)) - for host := range byHost { - hosts = append(hosts, host) - } - sort.Strings(hosts) - if validateGooglevideoHosts(hosts) != nil { - return nil, &InsufficientAdaptiveSamples{} - } - resources := make([]mediaResource, 0, len(hosts)) - for _, host := range hosts { - resources = append(resources, byHost[host]) - } - return resources, nil + return nil, stats } } -func managedAdaptiveAssetsQualified(d managedRecoveryDeps, ids []int64) (bool, error) { - covered := map[string]bool{} - for _, id := range ids { - c, err := readStrategyCandidate(d.db, id) - if err != nil { - return false, err - } - if c.Group == "assets" && c.Qualified { - for _, host := range c.Endpoints { - covered[host] = true - } - } - } - return covered["i.ytimg.com"] && covered["yt3.ggpht.com"], nil -} - func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) { m := *d.transaction journal, err := m.load() @@ -86,6 +145,13 @@ func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int6 if journal.Scope == GooglevideoScopeExperimental { return recoverManagedExperimental(ctx, d) } + return qualifyManagedGooglevideo(ctx, d) +} + +// Both autonomous adaptive recovery and an operationally rechecked experiment +// use the same unchanged three-host owned proof and transactional apply gate. +func qualifyManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) { + m := *d.transaction if !d.service || d.mediaSamples == nil || d.verifyGooglevideo == nil { return 0, &InsufficientAdaptiveSamples{} } @@ -93,20 +159,6 @@ func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int6 if err != nil { return 0, &managedRecoveryFatal{Err: err} } - ok, err := managedAdaptiveAssetsQualified(d, current) - if err != nil { - return 0, &managedRecoveryFatal{Err: err} - } - if !ok { - return 0, &managedResourceFailure{"thumbnail", "unknown", nil} - } - status, err := managedAssetApplyHealth(ctx, d, []string{"i.ytimg.com", "yt3.ggpht.com"}) - if err != nil { - return 0, err - } - if status != "OK" { - return 0, &managedHealthRejection{Status: status} - } saved, err := listQualifiedCandidates(d.db, "googlevideo", 10) if err != nil { return 0, &managedRecoveryFatal{Err: err} @@ -189,16 +241,31 @@ func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int6 return d.run(cleanup, managedZapretScript, "restart") } m.Health = func() (string, error) { - status, err := d.health(ctx) // Fresh media byte health, not hostname-only. - var failure *managedResourceFailure - var fatal *managedRecoveryFatal - if ctx.Err() != nil { + if d.frontHealth == nil || d.recheckGooglevideo == nil { + return "DEGRADED", errors.New("frontend and deployed media health required") + } + status, err := d.frontHealth(ctx) + if err != nil || status != "OK" { + return status, managedRecoveryContextError(ctx, err) + } + deployed, err := readStrategyCandidate(d.db, id) + if err == nil { + err = d.recheckGooglevideo(ctx, deployed, resources, &m) + } + if err != nil { + var fatal *managedRecoveryFatal + if ctx.Err() != nil || errors.As(err, &fatal) || errors.Is(err, context.Canceled) { + return "DEGRADED", managedRecoveryContextError(ctx, err) + } + if managedGooglevideoTransferNonpass(err) || errors.Is(err, errCandidateNonpass) { + return "DEGRADED", nil + } return "DEGRADED", managedRecoveryContextError(ctx, err) } - if errors.As(err, &failure) && !errors.As(err, &fatal) && !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) { - return "DEGRADED", nil + if ctx.Err() != nil { + return "DEGRADED", managedRecoveryContextError(ctx, ctx.Err()) } - return status, err + return "OK", nil // Stage readiness only; whole-service monitor is unchanged. } err = m.Apply(d.db, desired, GooglevideoScopeSuffix) var rollback *managedRollbackError diff --git a/cmd/zapret-autotune/managed_googlevideo_recovery_test.go b/cmd/zapret-autotune/managed_googlevideo_recovery_test.go index 6959484..0e397ef 100644 --- a/cmd/zapret-autotune/managed_googlevideo_recovery_test.go +++ b/cmd/zapret-autotune/managed_googlevideo_recovery_test.go @@ -13,6 +13,7 @@ import ( "strconv" "strings" "testing" + "time" ) func TestManagedAdaptiveConfig(t *testing.T) { @@ -42,7 +43,7 @@ func TestManagedAdaptiveConfig(t *testing.T) { } func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) { - for _, scenario := range []string{"qualified", "one-host", "unqualified-assets", "rollback"} { + for _, scenario := range []string{"qualified", "one-host", "unqualified-assets", "avatar-missing", "rollback", "apply-bad-body", "apply-mixed-guard", "apply-learns"} { t.Run(scenario, func(t *testing.T) { d, front, events := recoveryFixture(t, true) r, _, resources, _ := googlevideoRuntimeFixture(t) @@ -66,16 +67,71 @@ func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) { return resources, nil } d.verifyGooglevideo = r.verifyGooglevideo - if scenario == "rollback" { + d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db) + applyCalls := 0 + if strings.HasPrefix(scenario, "apply-") { + deployed := r + if scenario == "apply-mixed-guard" { + deployed.guard = func() error { return errors.Join(context.Canceled, errors.New("guard failed")) } + } + deployed.curl = func(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error { + applyCalls++ + if scenario == "apply-learns" && applyCalls <= 2 { + return context.DeadlineExceeded + } + var body bytes.Buffer + err := r.curl(ctx, args, stdin, &body, diagnostic) + if err != nil { + return err + } + b := body.Bytes() + if scenario == "apply-bad-body" { + b = bytes.ReplaceAll(b, []byte("ftyp"), []byte("oops")) + } + _, err = out.Write(b) + return err + } + d.recheckGooglevideo = managedGooglevideoApplyRecheck(deployed, d.db) + } + if scenario == "avatar-missing" { d.health = func(context.Context) (string, error) { - return "DEGRADED", &managedResourceFailure{"media", resources[0].host, nil} + return "DEGRADED", &managedResourceFailure{"avatar", "yt3.ggpht.com", nil} + } + _, ids, err := managedGooglevideoConfig(func(key string) (string, bool) { return "1", key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" }, "dQw4w9WgXcQ") + if err != nil || len(ids) != 3 { + t.Fatal(ids, err) + } + calls := 0 + d.mediaSamples = managedGooglevideoSamples(ids, func(_ context.Context, id string) ([]byte, error) { + if id != ids[calls%3] { + t.Fatal("wrong default video") + } + index := calls % 3 + if calls < 3 { + index = 0 + } + calls++ + return samplePlayerBody(resources[index].host, time.Now().Add(time.Hour)), nil + }) + } + if scenario == "rollback" { + d.recheckGooglevideo = func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error { + return errCandidateNonpass } } result, err := recoverManagedTarget(context.Background(), d) + if scenario == "apply-mixed-guard" { + var fatal *managedRecoveryFatal + if !errors.As(err, &fatal) { + t.Fatal("mixed guard downgraded", err) + } + assertActiveCandidates(t, d.transaction, d.db, old) + return + } if err != nil { t.Fatal(err) } - if scenario != "qualified" { + if scenario == "one-host" || scenario == "rollback" || scenario == "apply-bad-body" { if result.GGCError == nil { t.Fatal("unqualified stage succeeded") } @@ -102,6 +158,15 @@ func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) { t.Fatal(j, err) } assertActiveCandidates(t, d.transaction, d.db, append(old, c.ID)) + if scenario == "apply-learns" && applyCalls < 20 { + t.Fatal("deployed warmup or full TLS transfers skipped", applyCalls) + } + if scenario == "avatar-missing" { + status, err := d.health(context.Background()) + if status != "DEGRADED" || err == nil { + t.Fatal("whole-service health concealed") + } + } if strings.Join(*events, ",") != "stop,start,restart" { t.Fatal("restore must precede apply", *events) } @@ -136,7 +201,39 @@ func adaptiveImageRunner(_ context.Context, args []string, stdin io.Reader, out, func TestInsufficientAdaptiveSamplesCooldown(t *testing.T) { err := &InsufficientAdaptiveSamples{} - if !errors.Is(err, errNoQualifiedStrategy) || !errors.Is(err, errGooglevideoInsufficientSamples) { + if errors.Is(err, errNoQualifiedStrategy) || !errors.Is(err, errGooglevideoInsufficientSamples) || managedRecoveryCooldown(err, time.Minute) != time.Minute { t.Fatal(err) } } + +func TestGenuineNoQualifiedCooldown(t *testing.T) { + if managedRecoveryCooldown(errNoQualifiedStrategy, time.Minute) != 2*time.Hour { + t.Fatal("lost exhausted-search pause") + } +} + +func TestManagedYouTubeDefaultSampleInventory(t *testing.T) { + for _, primary := range []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ"} { + lookup := func(key string) (string, bool) { return "1", key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" } + _, ids, err := managedGooglevideoConfig(lookup, primary) + want := []string{primary, "aqz-KE-bpKQ", "jNQXAC9IVRw"} + if primary == "aqz-KE-bpKQ" { + want = []string{primary, "jNQXAC9IVRw"} + } + if err != nil || !reflect.DeepEqual(ids, want) { + t.Fatal(ids, err) + } + } + _, ids, err := managedGooglevideoConfig(func(key string) (string, bool) { + switch key { + case "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE": + return "1", true + case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS": + return `["aaaaaaaaaaa"]`, true + } + return "", false + }, "dQw4w9WgXcQ") + if err != nil || !reflect.DeepEqual(ids, []string{"aaaaaaaaaaa"}) { + t.Fatal(ids, err) + } +} diff --git a/cmd/zapret-autotune/managed_googlevideo_samples_test.go b/cmd/zapret-autotune/managed_googlevideo_samples_test.go index bf42643..ccf5175 100644 --- a/cmd/zapret-autotune/managed_googlevideo_samples_test.go +++ b/cmd/zapret-autotune/managed_googlevideo_samples_test.go @@ -57,7 +57,11 @@ func TestManagedGooglevideoFreshPrivateSamples(t *testing.T) { t.Fatal(actual, err) } } - if calls != 2*len(ids) { + want := 2 * len(ids) + if scenario != "diverse" { + want *= managedGooglevideoSampleRounds + } + if calls != want { t.Fatal("samples were reused", calls) } }) diff --git a/cmd/zapret-autotune/managed_monitor_cli.go b/cmd/zapret-autotune/managed_monitor_cli.go index 18d0c20..48b777b 100644 --- a/cmd/zapret-autotune/managed_monitor_cli.go +++ b/cmd/zapret-autotune/managed_monitor_cli.go @@ -192,9 +192,8 @@ func managedMonitorRecoveryAdapters(target managedDeps, db *sql.DB, scanCommand if err == nil && d.service { d.verifyAssets = newManagedAssetVerifier(target) d.recheckExperimental = newManagedExperimentalRecheck(target, d) - if d.adaptive { - d.verifyGooglevideo = newManagedGooglevideoVerifier(target) - } + d.verifyGooglevideo = newManagedGooglevideoVerifier(target) + d.recheckGooglevideo = newManagedGooglevideoApplyRecheck(target, db) } return d, err } diff --git a/cmd/zapret-autotune/managed_recovery.go b/cmd/zapret-autotune/managed_recovery.go index 0476cef..ec29060 100644 --- a/cmd/zapret-autotune/managed_recovery.go +++ b/cmd/zapret-autotune/managed_recovery.go @@ -40,6 +40,7 @@ type managedRecoveryDeps struct { mediaSamples func(context.Context) ([]mediaResource, error) verifyGooglevideo func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error) recheckExperimental func(context.Context, storedCandidate, mediaResource, *managedTransaction) error + recheckGooglevideo func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error } // Target-local adapter: reuses bounded Player requests without invoking a CLI. @@ -120,6 +121,9 @@ func managedRecoveryDependencies(target managedDeps, db *sql.DB, scan cacheScan, } if d.service { d.frontHealth = d.health + // Construction is side-effect free; experiments may seek representative + // qualification without changing the legacy default exact-host mode. + d.mediaSamples = managedGooglevideoSamples(videoIDs, requestVideo) d.resources = managedServiceResources(cfg, request, target.player) d.health = managedServiceHealth(cfg.SOCKSProxy, d.frontHealth, d.resources, runResourceCurl) d.verifyAssets = unwiredManagedAssetVerifier @@ -279,7 +283,21 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma if ctx.Err() != nil || errors.As(healthErr, &fatal) || errors.Is(healthErr, context.Canceled) || errors.Is(healthErr, context.DeadlineExceeded) { return result, managedRecoveryContextError(ctx, healthErr) } - if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) && j.Scope != GooglevideoScopeExperimental { + // A previously authorized native suffix stays adaptive after promotion, + // even if the operator's legacy environment still says "cold". Stored + // validation permits reconstruction, never fresh proof authorization. + if j.Scope == GooglevideoScopeSuffix { + for _, id := range j.Candidates { + c, err := readStrategyCandidatePolicy(d.db, id, false) + if err != nil { + return result, &managedRecoveryFatal{Err: err} + } + if c.Group == "googlevideo" && c.Mode == QualificationAdaptive && c.Qualified { + d.adaptive = true + } + } + } + if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) && j.Scope != GooglevideoScopeExperimental && !d.adaptive { return result, repairManagedAssets(ctx, d, failure.Host) } } @@ -773,7 +791,7 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe if err != nil { return &managedRecoveryFatal{Err: err} } - if recovery.adaptive && candidate.Group == "googlevideo" && candidate.Mode == QualificationAdaptive && candidate.Qualified { + if candidate.Group == "googlevideo" && candidate.Mode == QualificationAdaptive && candidate.Qualified { journal, e := transaction.load() if e != nil { return &managedRecoveryFatal{Err: e} diff --git a/cmd/zapret-autotune/managed_recovery_hooks_test.go b/cmd/zapret-autotune/managed_recovery_hooks_test.go index 5253985..11d5a50 100644 --- a/cmd/zapret-autotune/managed_recovery_hooks_test.go +++ b/cmd/zapret-autotune/managed_recovery_hooks_test.go @@ -34,17 +34,18 @@ func TestManagedRecoveryProductionTransactionHooks(t *testing.T) { d.transaction = &managedTransaction{Dir: d.transaction.Dir, Name: "config"} d.service, d.adaptive = true, true d.frontHealth = d.health + samples := 0 d.mediaSamples = func(context.Context) ([]mediaResource, error) { - t.Fatal("asset qualification must precede private media discovery") - return nil, nil + samples++ + return nil, &InsufficientAdaptiveSamples{} } d.verifyGooglevideo = func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error) { t.Fatal("unexpected verification") return GooglevideoQualificationProof{}, nil } result, err := recoverManagedTarget(context.Background(), d) - var failure *managedResourceFailure - if err != nil || result.FrontendID != front || !errors.As(result.GGCError, &failure) { + var failure *InsufficientAdaptiveSamples + if err != nil || result.FrontendID != front || samples != 1 || !errors.As(result.GGCError, &failure) { // Never expose private DB contents or error causes. t.Fatalf("frontend_preserved=%t recovery_error_type=%T ggc_error_type=%T", result.FrontendID == front, err, result.GGCError) } diff --git a/cmd/zapret-autotune/managed_recovery_regression_test.go b/cmd/zapret-autotune/managed_recovery_regression_test.go index 569139f..b3ab794 100644 --- a/cmd/zapret-autotune/managed_recovery_regression_test.go +++ b/cmd/zapret-autotune/managed_recovery_regression_test.go @@ -93,9 +93,10 @@ func TestManagedExpiredAdaptiveJournalRecovery(t *testing.T) { verifies++ return r.verifyGooglevideo(ctx, c, resources) } + d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db) if scenario == "rollback" { - d.health = func(context.Context) (string, error) { - return "DEGRADED", &managedResourceFailure{"media", resources[0].host, nil} + d.recheckGooglevideo = func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error { + return errCandidateNonpass } } result, err := recoverManagedTarget(context.Background(), d) diff --git a/cmd/zapret-autotune/monitor_runtime.go b/cmd/zapret-autotune/monitor_runtime.go index 1b7cc13..ae1bdaa 100644 --- a/cmd/zapret-autotune/monitor_runtime.go +++ b/cmd/zapret-autotune/monitor_runtime.go @@ -141,7 +141,7 @@ func runManagedMonitor(ctx context.Context, d managedMonitorDeps) (err error) { if ctx.Err() != nil { return ctx.Err() } - if errors.Is(err, errNoQualifiedStrategy) { + if errors.Is(err, errNoQualifiedStrategy) || errors.Is(err, errGooglevideoInsufficientSamples) { if err := pauseMonitorRecovery(d.db, d.serviceID, d.now(), managedRecoveryCooldown(err, d.cooldown)); err != nil { return err } diff --git a/deploy/managed-target/.env.example b/deploy/managed-target/.env.example index 3c46dd9..7039f68 100644 --- a/deploy/managed-target/.env.example +++ b/deploy/managed-target/.env.example @@ -3,7 +3,8 @@ ZAPRET_AUTOTUNE_HOST_SOCKS_PORT=11080 ZAPRET_AUTOTUNE_YOUTUBE_SERVICE=0 ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE=cold ZAPRET_AUTOTUNE_VIDEO_ID=dQw4w9WgXcQ -# Optional JSON array: 1..5 unique 11-character VideoIDs (not signed URLs). +# Missing/empty selects automatic YouTube inventory: primary + two public samples. +# Nonempty JSON overrides it: 1..5 unique 11-character VideoIDs (not signed URLs). # ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS='["dQw4w9WgXcQ","aqz-KE-bpKQ","jNQXAC9IVRw"]' # Optional target ss-server native DNS; empty retains the base resolver behavior. SS_DNS_SERVER= diff --git a/deploy/managed-target/README.md b/deploy/managed-target/README.md index 49e095b..0d136cd 100644 --- a/deploy/managed-target/README.md +++ b/deploy/managed-target/README.md @@ -5,6 +5,29 @@ reconfigure, or stop the existing `zapret2-ss-autotune` container. The target starts from a disabled managed baseline; `managed monitor` bootstraps the frontend/API strategy, then discovers and applies an exact-host GGC strategy. +## New autonomous three-CDN source candidate (2026-10-13): offline only + +Service-mode source now discovers representative inventory automatically and can +promote an existing experiment only through the unchanged version-1 owned native +proof factory: **three actual media hosts**, at least two roots and two prefixes, +post-warmup 3/3 on both TLS protocols. Discovery is bounded to three rounds, +15 requests maximum and three minutes; signed MP4 URLs need twenty minutes left. +VideoIDs are not CDN evidence. No manual domain list or single-CDN relaxation is +introduced. Missing/empty inventory defaults and explicit overrides are below. + +Apply requires qualified frontend and fresh deployed checks of all three media +resources, not avatar readiness. Whole-service health remains DEGRADED until avatar +checks pass. Insufficient inventory preserves the committed experiment and uses +configured cooldown; true strategy exhaustion retains its minimum two-hour pause. +Suffix authorization remains extrapolative, not proof of every Googlevideo host. + +This source is offline unit/fixture-verified only; no new image deployment or +autonomous three-CDN field acceptance is claimed. The historical results below +do not close that gate. Maximum strategy collection and PID1 global-failure +stay-alive are unfinished future increments, as is arbitrary declarative service +configuration. See [MANAGED_TARGET.md](../../MANAGED_TARGET.md) and +[GOOGLEVIDEO_DISCOVERY.md](../../GOOGLEVIDEO_DISCOVERY.md). + ## Recorded Googlevideo suffix experiment (2026-10-12) ### Final field follow-up: DEGRADED / NOT READY @@ -133,9 +156,11 @@ Isolated, accelerated shell supervision fixtures (no network): Compose forwards `ZAPRET_AUTOTUNE_YOUTUBE_SERVICE` (`0` or `1`, default `0`), `ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE` (`cold` or `adaptive`, default `cold`), `ZAPRET_AUTOTUNE_VIDEO_ID` (default `dQw4w9WgXcQ`) and -`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` (JSON array of 1..5 unique VideoIDs; -default only the primary ID). Explicit empty/invalid settings are rejected, -not silently replaced. Adaptive requires service `1`. See `.env.example`. +`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` (optional JSON array of 1..5 unique VideoIDs). +Missing/empty inventory selects automatic service-mode samples: primary plus +`aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated. A nonempty explicit inventory +overrides those defaults, including an intentional single video. Invalid nonempty +settings are rejected. Adaptive requires service `1`. See `.env.example`. Service `0` preserves legacy Player API/exact-host behavior. Adaptive qualification is **post-warmup**, distinct from cold qualification; diff --git a/deploy/managed-target/check_config.py b/deploy/managed-target/check_config.py index 6ef795a..b189818 100644 --- a/deploy/managed-target/check_config.py +++ b/deploy/managed-target/check_config.py @@ -45,9 +45,21 @@ with tempfile.TemporaryDirectory(prefix='target-config-') as tmp: assert json.loads(path.read_text())['password'] == base['SS_PASSWORD'] # Render privately, never print the config/password; config does not deploy. compose_env = {k: v for k, v in base.items() if not k.startswith('ZAPRET_AUTOTUNE_')} + factory_cases = [] for overrides, service, mode, ids in ( ({}, '0', 'cold', ['dQw4w9WgXcQ']), ({'ZAPRET_AUTOTUNE_VIDEO_ID': 'aqz-KE-bpKQ'}, '0', 'cold', ['aqz-KE-bpKQ']), + ({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive'}, + '1', 'adaptive', ['dQw4w9WgXcQ', 'aqz-KE-bpKQ', 'jNQXAC9IVRw']), + ({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive', + 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS': ''}, + '1', 'adaptive', ['dQw4w9WgXcQ', 'aqz-KE-bpKQ', 'jNQXAC9IVRw']), + ({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive', + 'ZAPRET_AUTOTUNE_VIDEO_ID': 'aqz-KE-bpKQ'}, + '1', 'adaptive', ['aqz-KE-bpKQ', 'jNQXAC9IVRw']), + ({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive', + 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS': '["aaaaaaaaaaa"]'}, + '1', 'adaptive', ['aaaaaaaaaaa']), ({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS': '["dQw4w9WgXcQ","aqz-KE-bpKQ","jNQXAC9IVRw"]', 'SS_DNS_SERVER': '8.8.8.8'}, '1', 'adaptive', ['dQw4w9WgXcQ', 'aqz-KE-bpKQ', 'jNQXAC9IVRw']), @@ -60,7 +72,15 @@ with tempfile.TemporaryDirectory(prefix='target-config-') as tmp: settings = json.loads(result.stdout)['services']['target']['environment'] assert settings['ZAPRET_AUTOTUNE_YOUTUBE_SERVICE'] == service assert settings['ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE'] == mode - assert json.loads(settings['ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS']) == ids + assert settings['ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS'] == overrides.get('ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS', '') assert settings['SS_DNS_SERVER'] == overrides.get('SS_DNS_SERVER', '') assert settings['ZAPRET_AUTOTUNE_SOCKS_PROXY'] == 'socks5h://127.0.0.1:1080' + factory_cases.append({'environment': {k: v for k, v in settings.items() + if k.startswith('ZAPRET_AUTOTUNE_')}, 'ids': ids}) + # Only sanitized fixture settings cross into Go; never persist the password. + fixture = root / 'compose-factory.json' + fixture.write_text(json.dumps(factory_cases)) + subprocess.run(['go', 'test', './cmd/zapret-autotune', '-run', '^TestManagedGooglevideoComposeForwarding$', + '-count=1'], cwd=ROOT, env=dict(env, AUTOTUNE_COMPOSE_CONFIG_FIXTURE=str(fixture)), + check=True, timeout=60) print('target config fixtures passed') diff --git a/deploy/managed-target/check_static.py b/deploy/managed-target/check_static.py index 7b9efed..c5fbb5b 100644 --- a/deploy/managed-target/check_static.py +++ b/deploy/managed-target/check_static.py @@ -26,7 +26,7 @@ for setting in ( 'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE: ${ZAPRET_AUTOTUNE_YOUTUBE_SERVICE-0}', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE: ${ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE-cold}', 'ZAPRET_AUTOTUNE_VIDEO_ID: ${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}', - "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS-[\"${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}\"]}'", + "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS:-}'", ): assert setting in compose, setting assert entrypoint.index('target-config-check || exit 64') < entrypoint.index('cat > "$config_dir/server.json"') diff --git a/deploy/managed-target/compose.yaml b/deploy/managed-target/compose.yaml index b569861..9b610ed 100644 --- a/deploy/managed-target/compose.yaml +++ b/deploy/managed-target/compose.yaml @@ -16,7 +16,7 @@ services: ZAPRET_AUTOTUNE_YOUTUBE_SERVICE: ${ZAPRET_AUTOTUNE_YOUTUBE_SERVICE-0} ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE: ${ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE-cold} ZAPRET_AUTOTUNE_VIDEO_ID: ${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ} - ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS-["${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}"]}' + ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS:-}' SOCKS_PORT: "1080" ZAPRET_AUTOTUNE_TARGET_ROLE: managed-test ZAPRET_AUTOTUNE_TARGET_ID: zapret-autotune-managed-v1