feat: discover and qualify Googlevideo CDNs autonomously
Retain strict three-host native qualification, discover fresh representative inventory automatically, and recheck deployed media before promotion. Document offline-only verification and outstanding field gates. Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
9984f21466
commit
3fc8563995
22 files changed
+798
-105
No files matched your search
@@ -0,0 +1,34 @@
|
||||
# Bounded representative Googlevideo discovery
|
||||
|
||||
The YouTube default inventory is the configured primary plus `aqz-KE-bpKQ` and
|
||||
`jNQXAC9IVRw`, deduplicated. Explicit configured inventories override it. The
|
||||
inventory (one to five IDs) is queried freshly
|
||||
in round-robin order for up to three rounds, bounded by three minutes. No new
|
||||
manual CDN addresses, host rewriting or IP pinning is added.
|
||||
With only one configured video, diversity depends on that video's real Player
|
||||
backend rotation; if it does not rotate, qualification remains unresolved.
|
||||
|
||||
Discovery deduplicates actual ordinary signed MP4 media hosts, refetches expired
|
||||
inventory and selects three hosts passing the existing root/prefix diversity
|
||||
gate. URLs need twenty minutes remaining for the three sequential owned native
|
||||
warmups and verified TLS transfers. URLs stay transient; unresolved diagnostics
|
||||
contain only attempt/host and rejection counts. Discovery is not media proof.
|
||||
|
||||
Production adaptive recovery uses this collector. Existing experimental suffixes
|
||||
retain their one-host operational recheck; promotion requires fresh three-host
|
||||
inventory and the unchanged version-1 owned native proof factory. Insufficient
|
||||
inventory leaves the committed experimental profile intact and retries after
|
||||
the configured cooldown, not the exhausted-strategy two-hour pause. Native GGC
|
||||
scope requires qualified frontend and any supplied asset candidates remain
|
||||
qualified and nonoverlapping; missing avatar inventory is not its prerequisite.
|
||||
Apply checks frontend and all three actual media resources in the newly deployed
|
||||
daemon, including bounded learning and three verified transfers per TLS version.
|
||||
Whole-service monitoring still reports avatar degradation independently.
|
||||
|
||||
Single-CDN version-2 qualification has been withdrawn. Maximum strategy catalogs,
|
||||
new video discovery sources and controller stay-alive changes are separate work.
|
||||
The three-CDN requirement was reaffirmed on 2026-10-13. The common suffix remains
|
||||
authorized extrapolative coverage; only exact measured endpoints have transfer
|
||||
evidence. This candidate is offline unit/fixture-verified only: owned factory,
|
||||
autonomous discovery/promotion and deployed learning still need a scoped actual
|
||||
three-CDN field test. No remote network checks or deployment accompany this update.
|
||||
@@ -1,5 +1,38 @@
|
||||
# Fresh managed-test target contract (isolated acceptance, not production approval)
|
||||
|
||||
## Autonomous three-CDN source candidate (2026-10-13): field acceptance pending
|
||||
|
||||
The user's three-CDN requirement is retained; the proposed single-CDN relaxation
|
||||
was withdrawn. Service-mode discovery needs no manual domains: missing/empty
|
||||
`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` uses primary `ZAPRET_AUTOTUNE_VIDEO_ID`
|
||||
(default `dQw4w9WgXcQ`) plus `aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated.
|
||||
A nonempty JSON array of 1..5 unique VideoIDs overrides this inventory. VideoIDs
|
||||
are discovery inputs, not proof of distinct CDNs.
|
||||
|
||||
Fresh round-robin Player requests are bounded to three rounds (at most 15 requests)
|
||||
and three minutes. Qualification selects three actual signed MP4 media hosts,
|
||||
with at least two roots and two prefixes, and twenty-minute remaining URL TTL.
|
||||
Only the existing version-1 owned native proof factory authorizes qualification:
|
||||
post-warmup actual media 3/3 on both TLS protocols for each measured endpoint.
|
||||
The common `googlevideo.com` suffix is an authorized extrapolative scope, not
|
||||
measured evidence for every Googlevideo host or cold/uniform behavior.
|
||||
|
||||
Experimental promotion uses the same proof and fresh deployed three-host media
|
||||
checks before committing apply. Insufficient inventory preserves the current
|
||||
experiment and uses configured retry cooldown; genuine exhausted strategy search
|
||||
still waits at least two hours. Qualified frontend is required; avatar success
|
||||
does not block this media stage, but whole-service health remains DEGRADED until
|
||||
actual avatar checks pass. Legacy service-off/exact-host behavior is unchanged.
|
||||
|
||||
This source candidate has only offline unit/fixture verification, not a newly
|
||||
deployed image or field-qualified autonomous promotion. The 2026-10-12 measured
|
||||
results below remain historical exact-endpoint evidence. Build from the committed
|
||||
source archive and field-test the owned factory, three actual CDNs, deployed
|
||||
relearning/apply, rollback and whole-service health before acceptance. Maximum
|
||||
strategy collection and PID1 global-failure stay-alive remain unfinished future
|
||||
increments. Arbitrary declarative service configuration is future work; no new
|
||||
generic policy framework is supplied. See [GOOGLEVIDEO_DISCOVERY.md](GOOGLEVIDEO_DISCOVERY.md).
|
||||
|
||||
`managed init|reconcile|status|active-ids|apply` is exclusively for a **new dedicated container**
|
||||
which owns its entire config. The existing/dev/production container must refuse
|
||||
these commands. A role env or CLI-created identity is not enough. Provisioning
|
||||
|
||||
+32
-1
@@ -1,9 +1,40 @@
|
||||
# Прогресс реализации
|
||||
|
||||
Срез на **2026-10-12**. Этот файл описывает фактическое состояние кода, а не весь объём требований из `Requirements.md` и `Architecture.md`.
|
||||
Срез на **2026-10-13**. Этот файл описывает фактическое состояние кода, а не весь объём требований из `Requirements.md` и `Architecture.md`.
|
||||
|
||||
## Текущий статус
|
||||
|
||||
### Autonomous three-CDN source candidate (2026-10-13): offline only
|
||||
|
||||
- User reaffirmed **three actual CDN checks**, discovered by the program without
|
||||
manual domains; the earlier single-CDN/version-2 proposal is withdrawn. Missing
|
||||
or empty service-mode VideoID inventory selects primary (default `dQw4w9WgXcQ`)
|
||||
plus `aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated. A nonempty 1..5 unique-ID
|
||||
override is respected; three videos do not prove three CDN hosts.
|
||||
- Fresh discovery runs up to three round-robin rounds, at most 15 requests in
|
||||
three minutes; it deduplicates actual signed ordinary MP4 hosts, requires
|
||||
twenty-minute remaining URL TTL, and selects three hosts across at least two
|
||||
roots and two prefixes. The unchanged version-1 owned native proof factory
|
||||
requires actual post-warmup media 3/3 on both TLS protocols at each endpoint.
|
||||
Common Googlevideo suffix authorization is extrapolative, not all-host evidence.
|
||||
- Existing experiments can now seek promotion through that shared qualification
|
||||
path, then frontend and fresh deployed three-host media checks under the current
|
||||
transaction/ownership guards. Insufficient inventory leaves the experiment
|
||||
committed and retries at configured cooldown; genuine strategy exhaustion keeps
|
||||
its minimum two-hour pause. Missing avatar does not block the media stage;
|
||||
whole-service monitoring remains DEGRADED until actual avatar health passes.
|
||||
Legacy service-off/exact-host behavior and cancellation safety remain unchanged.
|
||||
- This is an offline unit/fixture-tested source candidate, **not deployed or
|
||||
field-accepted**. Renewed actual three-CDN discovery, owned factory proof,
|
||||
deployed learning/apply/rollback and avatar qualification remain field gates.
|
||||
Earlier statements below that automatic qualification is unimplemented describe
|
||||
those historical RCs, not this new source; their measured results are not new
|
||||
factory evidence. No remote network checks or deployment accompany this update.
|
||||
- Maximum strategy collection and PID1 global-failure stay-alive are still future
|
||||
increments; broad maximum-profile live staging is deferred. Arbitrary declarative
|
||||
service configuration remains a future note, not a new framework. Details:
|
||||
[GOOGLEVIDEO_DISCOVERY.md](GOOGLEVIDEO_DISCOVERY.md).
|
||||
|
||||
### Final managed RC field acceptance (2026-10-12): DEGRADED, not READY
|
||||
|
||||
- Supplied final field results, not network checks rerun by this docs update: source `fd25cb89398300bb9849db4b5511af75bf86eccf`, correct managed image `sha256:73e955e7fdc6efaef88c62ebe4f7afeab71c29d29ba3c88b2a2b7547bb647da5`, active container `a4e845a6d7…`, official managed PID 1, isolated SOCKS `127.0.0.1:11082`. The earlier `37823…` prototype image is **not active** and is not this acceptance image. Private sanitized proof: `/home/agent/zapret2-autotune-rc-fd25cb8/proof/final.safe.json`. The later docs commit is not the tested image source.
|
||||
|
||||
@@ -75,9 +75,17 @@ func managedGooglevideoConfig(lookup func(string) (string, bool), primary string
|
||||
if mode == QualificationAdaptive && !service {
|
||||
return "", nil, fmt.Errorf("adaptive googlevideo requires ZAPRET_AUTOTUNE_YOUTUBE_SERVICE=1")
|
||||
}
|
||||
// Only the configured primary is sampled by default; no implicit public playlist.
|
||||
ids := []string{primary}
|
||||
if raw, ok := lookup("ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS"); ok {
|
||||
// YouTube-specific public discovery samples, not CDN addresses. Explicit
|
||||
// inventories below override these defaults; health/discovery primary stays put.
|
||||
if service {
|
||||
for _, id := range []string{"aqz-KE-bpKQ", "jNQXAC9IVRw"} {
|
||||
if id != primary {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
if raw, ok := lookup("ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS"); ok && raw != "" {
|
||||
if json.Unmarshal([]byte(raw), &ids) != nil || len(ids) == 0 || len(ids) > 5 {
|
||||
return "", nil, fmt.Errorf("GOOGLEVIDEO_VIDEO_IDS requires a JSON array of 1..5 video IDs")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// check_config.py provides sanitized actual `docker compose config` output.
|
||||
// This bridges deployment interpolation to the production config parser offline.
|
||||
func TestManagedGooglevideoComposeForwarding(t *testing.T) {
|
||||
path := os.Getenv("AUTOTUNE_COMPOSE_CONFIG_FIXTURE")
|
||||
if path == "" {
|
||||
t.Skip("run deploy/managed-target/check_config.py for Compose fixture")
|
||||
}
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal("compose fixture unavailable")
|
||||
}
|
||||
var cases []struct {
|
||||
Environment map[string]string `json:"environment"`
|
||||
IDs []string `json:"ids"`
|
||||
}
|
||||
if json.Unmarshal(data, &cases) != nil || len(cases) < 6 {
|
||||
t.Fatal("invalid compose fixture")
|
||||
}
|
||||
for i, c := range cases {
|
||||
lookup := func(key string) (string, bool) { value, ok := c.Environment[key]; return value, ok }
|
||||
mode, ids, err := managedGooglevideoConfig(lookup, c.Environment["ZAPRET_AUTOTUNE_VIDEO_ID"])
|
||||
if err != nil || string(mode) != c.Environment["ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE"] || !reflect.DeepEqual(ids, c.IDs) {
|
||||
t.Fatalf("compose factory case %d mismatch", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedGooglevideoBlankInventoryDefault(t *testing.T) {
|
||||
for _, value := range []string{"", " ", "\n", "[]", "bad", `["aaaaaaaaaaa"]`} {
|
||||
lookup := func(key string) (string, bool) {
|
||||
switch key {
|
||||
case "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE":
|
||||
return "1", true
|
||||
case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE":
|
||||
return "adaptive", true
|
||||
case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS":
|
||||
return value, true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
mode, ids, err := managedGooglevideoConfig(lookup, "dQw4w9WgXcQ")
|
||||
if value != "" && value != `["aaaaaaaaaaa"]` {
|
||||
if err == nil {
|
||||
t.Fatal("invalid nonempty inventory accepted")
|
||||
}
|
||||
continue
|
||||
}
|
||||
want := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"}
|
||||
if value != "" {
|
||||
want = []string{"aaaaaaaaaaa"}
|
||||
}
|
||||
if err != nil || mode != QualificationAdaptive || !reflect.DeepEqual(ids, want) {
|
||||
t.Fatal("inventory default/override mismatch")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -64,6 +64,7 @@ func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope,
|
||||
seen := map[int64]bool{}
|
||||
assets := map[string]bool{}
|
||||
var otherScopes []string
|
||||
nativeSuffix := false
|
||||
for _, id := range ids {
|
||||
if id <= 0 || seen[id] {
|
||||
return nil, fmt.Errorf("duplicate candidate %d", id)
|
||||
@@ -110,6 +111,7 @@ func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope,
|
||||
}
|
||||
}
|
||||
if c.Group == "googlevideo" && scope == GooglevideoScopeSuffix {
|
||||
nativeSuffix = c.Mode == QualificationAdaptive
|
||||
if version < 7 || validateGooglevideoSuffixEvidencePolicy(db, c, fresh) != nil {
|
||||
return nil, fmt.Errorf("suffix requires representative qualified hosts and verified media evidence")
|
||||
}
|
||||
@@ -125,8 +127,8 @@ func renderManagedConfigPolicy(db *sql.DB, ids []int64, scope GooglevideoScope,
|
||||
groups[c.Group] = true
|
||||
profiles = append(profiles, "--filter-tcp=443 --filter-l7=tls --hostlist-domains="+strings.Join(domains, ",")+" "+c.Args)
|
||||
}
|
||||
if scope == GooglevideoScopeSuffix && (!groups["googlevideo"] || len(assets) != 2) {
|
||||
return nil, fmt.Errorf("suffix requires googlevideo and both qualified asset profiles")
|
||||
if scope == GooglevideoScopeSuffix && (!groups["googlevideo"] || nativeSuffix && !groups["frontend"] || !nativeSuffix && len(assets) != 2) {
|
||||
return nil, fmt.Errorf("suffix requires qualified googlevideo and frontend (legacy cold scope requires both assets)")
|
||||
}
|
||||
if scope == GooglevideoScopeExperimental && (!groups["googlevideo"] || !groups["frontend"] || len(assets) == 0) {
|
||||
return nil, fmt.Errorf("experimental suffix requires frontend and existing qualified assets")
|
||||
|
||||
@@ -62,6 +62,15 @@ func recoverManagedExperimental(ctx context.Context, d managedRecoveryDeps) (int
|
||||
if err == nil {
|
||||
err = d.recheckExperimental(ctx, c, r, d.transaction)
|
||||
}
|
||||
if err == nil && d.mediaSamples != nil && d.verifyGooglevideo != nil {
|
||||
// One-host operational health never promotes the experiment. Only
|
||||
// fresh representative inventory and a completed owned proof may.
|
||||
qualified, err := qualifyManagedGooglevideo(ctx, d)
|
||||
if err != nil {
|
||||
return id, err
|
||||
}
|
||||
return qualified, nil
|
||||
}
|
||||
return id, managedRecoveryContextError(ctx, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
)
|
||||
|
||||
func newManagedGooglevideoApplyRecheck(target managedDeps, db *sql.DB) func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
|
||||
r := newManagedAssetRuntime(target)
|
||||
guard := r.guard
|
||||
r.guard = func() error {
|
||||
if err := guard(); err != nil {
|
||||
return err
|
||||
}
|
||||
lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua")
|
||||
if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash {
|
||||
return errors.New("unverified native automation Lua source")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return managedGooglevideoApplyRecheck(r, db)
|
||||
}
|
||||
|
||||
// Runs inside Apply's lock: never ActiveCandidates/Reconcile/restart here.
|
||||
// The newly reloaded daemon learns in-place; a failure rolls back via Apply.
|
||||
func managedGooglevideoApplyRecheck(r managedAssetRuntime, db *sql.DB) func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
|
||||
return func(ctx context.Context, c storedCandidate, resources []mediaResource, m *managedTransaction) error {
|
||||
j, err := m.load()
|
||||
if err != nil {
|
||||
return &managedRecoveryFatal{Err: err}
|
||||
}
|
||||
found := false
|
||||
for _, id := range j.PendingCandidates {
|
||||
found = found || id == c.ID
|
||||
}
|
||||
var hosts []string
|
||||
for _, resource := range resources {
|
||||
hosts = append(hosts, resource.host)
|
||||
}
|
||||
if j.State != "prepared" || j.PendingScope != GooglevideoScopeSuffix || !found || c.Mode != QualificationAdaptive || !reflect.DeepEqual(hosts, c.Endpoints) || validateGooglevideoSuffixEvidence(db, c) != nil {
|
||||
return &managedRecoveryFatal{Err: errors.New("deployed media candidate ownership mismatch")}
|
||||
}
|
||||
guard := func() error {
|
||||
if err := r.guard(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := m.owns(j, false); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateManagedRefs(db, j.PendingCandidates, j.Next, j.PendingScope); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := m.read(m.Name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if managedHash(b) != j.Next {
|
||||
return errManagedDrift
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for _, resource := range resources {
|
||||
err := recheckDeployedExperimental(ctx, resource, guard,
|
||||
func(context.Context) error { return guard() }, // already reloaded; warm current daemon only
|
||||
func(context.Context) (mediaResource, error) { return mediaResource{}, errCandidateNonpass },
|
||||
func(ctx context.Context, resource mediaResource, protocol string) error {
|
||||
_, _, err := collectMediaTransfer(ctx, r.proxy, resource, protocol, r.curl)
|
||||
return err
|
||||
})
|
||||
if errors.Is(err, errNoQualifiedStrategy) && !managedRecoveryTimeoutSafe(err) {
|
||||
var fatal *managedRecoveryFatal
|
||||
if !errors.As(err, &fatal) {
|
||||
return errCandidateNonpass
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func samplePlayerBody(host string, expires time.Time) []byte {
|
||||
u, _ := json.Marshal(fmt.Sprintf("https://%s/videoplayback?expire=%d&sig=private-secret", host, expires.Unix()))
|
||||
return []byte(fmt.Sprintf(`{"playabilityStatus":{"status":"OK"},"streamingData":{"formats":[{"mimeType":"video/mp4","url":%s}]}}`, u))
|
||||
}
|
||||
|
||||
func TestManagedGooglevideoDiscoveryRotatesFreshInventory(t *testing.T) {
|
||||
ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw"}
|
||||
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
|
||||
for _, scenario := range []string{"rotating", "unavailable", "expired", "same-root", "unsafe-origin", "invalid-host", "control", "unknown", "mixed-safety", "cancel"} {
|
||||
t.Run(scenario, func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
calls := 0
|
||||
collect := managedGooglevideoSamples(ids, func(ctx context.Context, id string) ([]byte, error) {
|
||||
if _, ok := ctx.Deadline(); !ok {
|
||||
t.Fatal("unbounded discovery")
|
||||
}
|
||||
if id != ids[calls%3] {
|
||||
t.Fatal("not round robin")
|
||||
}
|
||||
calls++
|
||||
host := hosts[(calls-1)%3]
|
||||
expires := time.Now().Add(time.Hour)
|
||||
if calls <= 3 {
|
||||
switch scenario {
|
||||
case "rotating":
|
||||
host = hosts[0]
|
||||
case "unavailable":
|
||||
return nil, context.DeadlineExceeded
|
||||
case "expired":
|
||||
expires = time.Now().Add(5 * time.Minute)
|
||||
case "unsafe-origin":
|
||||
host += ":444"
|
||||
case "invalid-host":
|
||||
host = "not-a-ggc.googlevideo.com"
|
||||
case "control":
|
||||
host += "\n"
|
||||
case "unknown":
|
||||
return nil, errors.New("runner unavailable")
|
||||
case "mixed-safety":
|
||||
return nil, errors.Join(context.DeadlineExceeded, errors.New("cleanup failed"))
|
||||
case "cancel":
|
||||
cancel()
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
}
|
||||
if scenario == "same-root" {
|
||||
host = fmt.Sprintf("rr%d---sn-alpha-x.googlevideo.com", calls)
|
||||
}
|
||||
return samplePlayerBody(host, expires), nil
|
||||
})
|
||||
resources, err := collect(ctx)
|
||||
if scenario == "unknown" || scenario == "mixed-safety" {
|
||||
var fatal *managedRecoveryFatal
|
||||
if !errors.As(err, &fatal) || calls != 1 || len(resources) != 0 {
|
||||
t.Fatal(err, calls)
|
||||
}
|
||||
return
|
||||
}
|
||||
if scenario == "cancel" {
|
||||
if !errors.Is(err, context.Canceled) || calls != 1 || len(resources) != 0 {
|
||||
t.Fatal(err, calls)
|
||||
}
|
||||
return
|
||||
}
|
||||
if scenario == "same-root" {
|
||||
var insufficient *InsufficientAdaptiveSamples
|
||||
if !errors.As(err, &insufficient) || insufficient.Attempts != 9 || len(resources) != 0 || strings.Contains(err.Error(), "private-secret") {
|
||||
t.Fatal(err, calls)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || len(resources) != 3 || calls != 6 {
|
||||
t.Fatal(err, calls, len(resources))
|
||||
}
|
||||
for i, r := range resources {
|
||||
if r.host != hosts[i] || !strings.Contains(r.url, "private-secret") {
|
||||
t.Fatal("changed signed inventory")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedGooglevideoDiscoveryBoundsAndLateCancellation(t *testing.T) {
|
||||
ids := []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ", "jNQXAC9IVRw", "aaaaaaaaaaa", "bbbbbbbbbbb"}
|
||||
calls := 0
|
||||
resources, err := managedGooglevideoSamples(ids, func(context.Context, string) ([]byte, error) {
|
||||
calls++
|
||||
return []byte(`{"playabilityStatus":{"status":"OK"},"streamingData":{"serverAbrStreamingUrl":"private-secret"}}`), nil
|
||||
})(context.Background())
|
||||
var insufficient *InsufficientAdaptiveSamples
|
||||
if !errors.As(err, &insufficient) || calls != 15 || insufficient.Unsupported != 15 || len(resources) != 0 {
|
||||
t.Fatal(err, calls)
|
||||
}
|
||||
for _, invalid := range [][]string{nil, {"bad"}, append(ids, "ccccccccccc")} {
|
||||
_, err := managedGooglevideoSamples(invalid, func(context.Context, string) ([]byte, error) { t.Fatal("invalid config queried"); return nil, nil })(context.Background())
|
||||
var fatal *managedRecoveryFatal
|
||||
if !errors.As(err, &fatal) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
hosts := []string{"rr1---sn-alpha.googlevideo.com", "rr2---sn-beta.googlevideo.com", "rr5---sn-gamma.googlevideo.com"}
|
||||
calls = 0
|
||||
resources, err = managedGooglevideoSamples(ids[:3], func(context.Context, string) ([]byte, error) {
|
||||
host := hosts[calls]
|
||||
calls++
|
||||
if calls == 3 {
|
||||
cancel()
|
||||
}
|
||||
return samplePlayerBody(host, time.Now().Add(time.Hour)), nil
|
||||
})(ctx)
|
||||
if !errors.Is(err, context.Canceled) || len(resources) != 0 || calls != 3 {
|
||||
t.Fatal(err, calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExperimentalPromotionRequiresRepresentativeOwnedProof(t *testing.T) {
|
||||
for _, enough := range []bool{false, true} {
|
||||
t.Run(fmt.Sprint(enough), func(t *testing.T) {
|
||||
d, front, _ := recoveryFixture(t, true)
|
||||
r, _, resources, _ := googlevideoRuntimeFixture(t)
|
||||
d.service = true
|
||||
d.frontHealth = d.health
|
||||
thumb := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
|
||||
avatar := unitAssetCandidate(t, d.db, "yt3.ggpht.com", "--lua-desync=fake", true)
|
||||
tx, err := d.db.Begin()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
experiment, err := insertStrategyCandidate(tx, "googlevideo", []string{resources[0].host}, managedGooglevideoArgs, nil, QualificationCold)
|
||||
if err != nil {
|
||||
_ = tx.Rollback()
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := []int64{front, thumb, avatar, experiment}
|
||||
if err := d.transaction.Apply(d.db, old, GooglevideoScopeExperimental); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d.resources = func(context.Context) (managedServiceInventory, error) {
|
||||
return managedServiceInventory{media: resources[0], images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/sample.jpg", "yt3.ggpht.com": "https://yt3.ggpht.com/sample.jpg"}, proxy: r.proxy, runner: adaptiveImageRunner}, nil
|
||||
}
|
||||
d.recheckExperimental = func(context.Context, storedCandidate, mediaResource, *managedTransaction) error { return nil }
|
||||
d.mediaSamples = func(context.Context) ([]mediaResource, error) {
|
||||
if enough {
|
||||
return resources, nil
|
||||
}
|
||||
return resources[:1], nil
|
||||
}
|
||||
d.verifyGooglevideo = r.verifyGooglevideo
|
||||
d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db)
|
||||
id, err := recoverManagedExperimental(context.Background(), d)
|
||||
if !enough {
|
||||
if !errors.Is(err, errGooglevideoInsufficientSamples) || id != experiment {
|
||||
t.Fatal(id, err)
|
||||
}
|
||||
assertActiveCandidates(t, d.transaction, d.db, old)
|
||||
return
|
||||
}
|
||||
if err != nil || id == experiment {
|
||||
t.Fatal(id, err)
|
||||
}
|
||||
assertActiveCandidates(t, d.transaction, d.db, []int64{front, thumb, avatar, id})
|
||||
p, err := readGooglevideoProof(d.db, id)
|
||||
if err != nil || p.Version != 1 || len(p.Hosts) != 3 {
|
||||
t.Fatal(p, err)
|
||||
}
|
||||
c, err := readStrategyCandidate(d.db, experiment)
|
||||
if err != nil || c.Qualified || c.Mode != QualificationCold {
|
||||
t.Fatal("old row changed", c, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -3,80 +3,139 @@ package main
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Ordinary unresolved state, not a target/ownership guard failure.
|
||||
type InsufficientAdaptiveSamples struct{}
|
||||
type InsufficientAdaptiveSamples struct {
|
||||
Attempts, DistinctHosts, Unavailable, Unsupported, Rejected int
|
||||
}
|
||||
|
||||
func (*InsufficientAdaptiveSamples) Error() string {
|
||||
return "DEGRADED: insufficient fresh ordinary signed GGC samples"
|
||||
func (e *InsufficientAdaptiveSamples) Error() string {
|
||||
return fmt.Sprintf("DEGRADED: insufficient fresh ordinary signed GGC samples (attempts=%d distinct_hosts=%d unavailable=%d unsupported=%d rejected=%d)", e.Attempts, e.DistinctHosts, e.Unavailable, e.Unsupported, e.Rejected)
|
||||
}
|
||||
func (*InsufficientAdaptiveSamples) Unwrap() []error {
|
||||
return []error{errGooglevideoInsufficientSamples, errNoQualifiedStrategy}
|
||||
return []error{errGooglevideoInsufficientSamples}
|
||||
}
|
||||
|
||||
// Sequential round-robin: at most 15 fresh requests, with no child goroutines.
|
||||
const managedGooglevideoSampleRounds = 3
|
||||
const managedGooglevideoDiscoveryTimeout = 3 * time.Minute
|
||||
|
||||
// Three hosts each need up to 120s warmup + six 35s transfers. Leave margin
|
||||
// for daemon lifecycle as well; short-lived URLs are refetched, never rewritten.
|
||||
const managedGooglevideoSampleTTL = 20 * time.Minute
|
||||
|
||||
func freshGooglevideoSample(r mediaResource) bool {
|
||||
// The same exact GGC hostname syntax as the representative gate; reject bad
|
||||
// inventory before counting it, without fetching any returned URL here.
|
||||
prefix, root, ok := strings.Cut(strings.TrimSuffix(r.host, ".googlevideo.com"), "---sn-")
|
||||
if !ok || !strings.HasSuffix(r.host, ".googlevideo.com") || !validDomain(r.host) || r.host != strings.ToLower(r.host) || root == "" || len(prefix) < 3 || !strings.HasPrefix(prefix, "rr") || strings.Trim(prefix[2:], "0123456789") != "" {
|
||||
return false
|
||||
}
|
||||
u, err := privateResourceURL(r.url, r.host)
|
||||
if err != nil || u.Path != "/videoplayback" || (r.kind != "video/mp4" && r.kind != "audio/mp4") {
|
||||
return false
|
||||
}
|
||||
q := u.Query()
|
||||
expires, err := strconv.ParseInt(q.Get("expire"), 10, 64)
|
||||
return err == nil && expires > time.Now().Add(managedGooglevideoSampleTTL).Unix() && (q.Get("sig") != "" || q.Get("signature") != "" || q.Get("lsig") != "")
|
||||
}
|
||||
|
||||
// Choose exactly three actual hosts passing the unchanged representative gate.
|
||||
func representativeGooglevideoSamples(byHost map[string]mediaResource) []mediaResource {
|
||||
hosts := make([]string, 0, len(byHost))
|
||||
for host, r := range byHost {
|
||||
if freshGooglevideoSample(r) {
|
||||
hosts = append(hosts, host)
|
||||
}
|
||||
}
|
||||
sort.Strings(hosts)
|
||||
for i := 0; i < len(hosts); i++ {
|
||||
for j := i + 1; j < len(hosts); j++ {
|
||||
for k := j + 1; k < len(hosts); k++ {
|
||||
if validateGooglevideoHosts([]string{hosts[i], hosts[j], hosts[k]}) == nil {
|
||||
return []mediaResource{byHost[hosts[i]], byHost[hosts[j]], byHost[hosts[k]]}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func managedGooglevideoSamples(ids []string, request func(context.Context, string) ([]byte, error)) func(context.Context) ([]mediaResource, error) {
|
||||
ids = append([]string(nil), ids...)
|
||||
return func(ctx context.Context) ([]mediaResource, error) {
|
||||
byHost := map[string]mediaResource{}
|
||||
if len(ids) == 0 || len(ids) > 5 || request == nil {
|
||||
return nil, &managedRecoveryFatal{Err: errors.New("invalid sample collector configuration")}
|
||||
}
|
||||
for _, id := range ids {
|
||||
body, err := request(ctx, id)
|
||||
var fatal *managedRecoveryFatal
|
||||
if ctx.Err() != nil || errors.As(err, &fatal) {
|
||||
return nil, managedRecoveryContextError(ctx, err)
|
||||
if !videoIDPattern.MatchString(id) {
|
||||
return nil, &managedRecoveryFatal{Err: errors.New("invalid sample video ID")}
|
||||
}
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
bounded, cancel := context.WithTimeout(ctx, managedGooglevideoDiscoveryTimeout)
|
||||
defer cancel()
|
||||
stats := &InsufficientAdaptiveSamples{}
|
||||
byHost := map[string]mediaResource{}
|
||||
for round := 0; round < managedGooglevideoSampleRounds; round++ {
|
||||
for _, id := range ids {
|
||||
if ctx.Err() != nil {
|
||||
return nil, managedRecoveryContextError(ctx, ctx.Err())
|
||||
}
|
||||
if bounded.Err() != nil {
|
||||
return nil, stats
|
||||
}
|
||||
stats.Attempts++
|
||||
body, err := request(bounded, id)
|
||||
var fatal *managedRecoveryFatal
|
||||
if ctx.Err() != nil || errors.As(err, &fatal) {
|
||||
return nil, managedRecoveryContextError(ctx, err)
|
||||
}
|
||||
if err != nil {
|
||||
if !managedResourceTransferNonpass(err, true, false) && !(bounded.Err() != nil && managedRecoveryTimeoutSafe(err)) {
|
||||
return nil, &managedRecoveryFatal{Err: err}
|
||||
}
|
||||
stats.Unavailable++
|
||||
if bounded.Err() != nil {
|
||||
return nil, stats
|
||||
}
|
||||
continue
|
||||
}
|
||||
r, _, err := playerMediaResource(body)
|
||||
if err != nil {
|
||||
stats.Unavailable++
|
||||
continue
|
||||
} // Missing/SABR is unknown, never fabricated.
|
||||
if r.url == "" {
|
||||
stats.Unsupported++
|
||||
continue
|
||||
}
|
||||
if !freshGooglevideoSample(r) {
|
||||
stats.Rejected++
|
||||
continue
|
||||
}
|
||||
byHost[r.host] = r
|
||||
stats.DistinctHosts = len(byHost)
|
||||
if ctx.Err() != nil {
|
||||
return nil, managedRecoveryContextError(ctx, ctx.Err())
|
||||
}
|
||||
if bounded.Err() != nil {
|
||||
return nil, stats
|
||||
}
|
||||
if resources := representativeGooglevideoSamples(byHost); resources != nil {
|
||||
return resources, nil
|
||||
}
|
||||
}
|
||||
r, _, err := playerMediaResource(body)
|
||||
if err != nil {
|
||||
continue
|
||||
} // Missing/SABR is unknown, never fabricated.
|
||||
u, err := privateResourceURL(r.url, r.host)
|
||||
if err != nil || u.Path != "/videoplayback" {
|
||||
continue
|
||||
}
|
||||
q := u.Query()
|
||||
expires, err := strconv.ParseInt(q.Get("expire"), 10, 64)
|
||||
if err != nil || expires <= time.Now().Unix()+120 || (q.Get("sig") == "" && q.Get("signature") == "" && q.Get("lsig") == "") {
|
||||
continue
|
||||
}
|
||||
byHost[r.host] = r
|
||||
}
|
||||
hosts := make([]string, 0, len(byHost))
|
||||
for host := range byHost {
|
||||
hosts = append(hosts, host)
|
||||
}
|
||||
sort.Strings(hosts)
|
||||
if validateGooglevideoHosts(hosts) != nil {
|
||||
return nil, &InsufficientAdaptiveSamples{}
|
||||
}
|
||||
resources := make([]mediaResource, 0, len(hosts))
|
||||
for _, host := range hosts {
|
||||
resources = append(resources, byHost[host])
|
||||
}
|
||||
return resources, nil
|
||||
return nil, stats
|
||||
}
|
||||
}
|
||||
|
||||
func managedAdaptiveAssetsQualified(d managedRecoveryDeps, ids []int64) (bool, error) {
|
||||
covered := map[string]bool{}
|
||||
for _, id := range ids {
|
||||
c, err := readStrategyCandidate(d.db, id)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if c.Group == "assets" && c.Qualified {
|
||||
for _, host := range c.Endpoints {
|
||||
covered[host] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return covered["i.ytimg.com"] && covered["yt3.ggpht.com"], nil
|
||||
}
|
||||
|
||||
func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) {
|
||||
m := *d.transaction
|
||||
journal, err := m.load()
|
||||
@@ -86,6 +145,13 @@ func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int6
|
||||
if journal.Scope == GooglevideoScopeExperimental {
|
||||
return recoverManagedExperimental(ctx, d)
|
||||
}
|
||||
return qualifyManagedGooglevideo(ctx, d)
|
||||
}
|
||||
|
||||
// Both autonomous adaptive recovery and an operationally rechecked experiment
|
||||
// use the same unchanged three-host owned proof and transactional apply gate.
|
||||
func qualifyManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) {
|
||||
m := *d.transaction
|
||||
if !d.service || d.mediaSamples == nil || d.verifyGooglevideo == nil {
|
||||
return 0, &InsufficientAdaptiveSamples{}
|
||||
}
|
||||
@@ -93,20 +159,6 @@ func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int6
|
||||
if err != nil {
|
||||
return 0, &managedRecoveryFatal{Err: err}
|
||||
}
|
||||
ok, err := managedAdaptiveAssetsQualified(d, current)
|
||||
if err != nil {
|
||||
return 0, &managedRecoveryFatal{Err: err}
|
||||
}
|
||||
if !ok {
|
||||
return 0, &managedResourceFailure{"thumbnail", "unknown", nil}
|
||||
}
|
||||
status, err := managedAssetApplyHealth(ctx, d, []string{"i.ytimg.com", "yt3.ggpht.com"})
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if status != "OK" {
|
||||
return 0, &managedHealthRejection{Status: status}
|
||||
}
|
||||
saved, err := listQualifiedCandidates(d.db, "googlevideo", 10)
|
||||
if err != nil {
|
||||
return 0, &managedRecoveryFatal{Err: err}
|
||||
@@ -189,16 +241,31 @@ func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int6
|
||||
return d.run(cleanup, managedZapretScript, "restart")
|
||||
}
|
||||
m.Health = func() (string, error) {
|
||||
status, err := d.health(ctx) // Fresh media byte health, not hostname-only.
|
||||
var failure *managedResourceFailure
|
||||
var fatal *managedRecoveryFatal
|
||||
if ctx.Err() != nil {
|
||||
if d.frontHealth == nil || d.recheckGooglevideo == nil {
|
||||
return "DEGRADED", errors.New("frontend and deployed media health required")
|
||||
}
|
||||
status, err := d.frontHealth(ctx)
|
||||
if err != nil || status != "OK" {
|
||||
return status, managedRecoveryContextError(ctx, err)
|
||||
}
|
||||
deployed, err := readStrategyCandidate(d.db, id)
|
||||
if err == nil {
|
||||
err = d.recheckGooglevideo(ctx, deployed, resources, &m)
|
||||
}
|
||||
if err != nil {
|
||||
var fatal *managedRecoveryFatal
|
||||
if ctx.Err() != nil || errors.As(err, &fatal) || errors.Is(err, context.Canceled) {
|
||||
return "DEGRADED", managedRecoveryContextError(ctx, err)
|
||||
}
|
||||
if managedGooglevideoTransferNonpass(err) || errors.Is(err, errCandidateNonpass) {
|
||||
return "DEGRADED", nil
|
||||
}
|
||||
return "DEGRADED", managedRecoveryContextError(ctx, err)
|
||||
}
|
||||
if errors.As(err, &failure) && !errors.As(err, &fatal) && !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) {
|
||||
return "DEGRADED", nil
|
||||
if ctx.Err() != nil {
|
||||
return "DEGRADED", managedRecoveryContextError(ctx, ctx.Err())
|
||||
}
|
||||
return status, err
|
||||
return "OK", nil // Stage readiness only; whole-service monitor is unchanged.
|
||||
}
|
||||
err = m.Apply(d.db, desired, GooglevideoScopeSuffix)
|
||||
var rollback *managedRollbackError
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestManagedAdaptiveConfig(t *testing.T) {
|
||||
@@ -42,7 +43,7 @@ func TestManagedAdaptiveConfig(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) {
|
||||
for _, scenario := range []string{"qualified", "one-host", "unqualified-assets", "rollback"} {
|
||||
for _, scenario := range []string{"qualified", "one-host", "unqualified-assets", "avatar-missing", "rollback", "apply-bad-body", "apply-mixed-guard", "apply-learns"} {
|
||||
t.Run(scenario, func(t *testing.T) {
|
||||
d, front, events := recoveryFixture(t, true)
|
||||
r, _, resources, _ := googlevideoRuntimeFixture(t)
|
||||
@@ -66,16 +67,71 @@ func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) {
|
||||
return resources, nil
|
||||
}
|
||||
d.verifyGooglevideo = r.verifyGooglevideo
|
||||
if scenario == "rollback" {
|
||||
d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db)
|
||||
applyCalls := 0
|
||||
if strings.HasPrefix(scenario, "apply-") {
|
||||
deployed := r
|
||||
if scenario == "apply-mixed-guard" {
|
||||
deployed.guard = func() error { return errors.Join(context.Canceled, errors.New("guard failed")) }
|
||||
}
|
||||
deployed.curl = func(ctx context.Context, args []string, stdin io.Reader, out, diagnostic io.Writer) error {
|
||||
applyCalls++
|
||||
if scenario == "apply-learns" && applyCalls <= 2 {
|
||||
return context.DeadlineExceeded
|
||||
}
|
||||
var body bytes.Buffer
|
||||
err := r.curl(ctx, args, stdin, &body, diagnostic)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
b := body.Bytes()
|
||||
if scenario == "apply-bad-body" {
|
||||
b = bytes.ReplaceAll(b, []byte("ftyp"), []byte("oops"))
|
||||
}
|
||||
_, err = out.Write(b)
|
||||
return err
|
||||
}
|
||||
d.recheckGooglevideo = managedGooglevideoApplyRecheck(deployed, d.db)
|
||||
}
|
||||
if scenario == "avatar-missing" {
|
||||
d.health = func(context.Context) (string, error) {
|
||||
return "DEGRADED", &managedResourceFailure{"media", resources[0].host, nil}
|
||||
return "DEGRADED", &managedResourceFailure{"avatar", "yt3.ggpht.com", nil}
|
||||
}
|
||||
_, ids, err := managedGooglevideoConfig(func(key string) (string, bool) { return "1", key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" }, "dQw4w9WgXcQ")
|
||||
if err != nil || len(ids) != 3 {
|
||||
t.Fatal(ids, err)
|
||||
}
|
||||
calls := 0
|
||||
d.mediaSamples = managedGooglevideoSamples(ids, func(_ context.Context, id string) ([]byte, error) {
|
||||
if id != ids[calls%3] {
|
||||
t.Fatal("wrong default video")
|
||||
}
|
||||
index := calls % 3
|
||||
if calls < 3 {
|
||||
index = 0
|
||||
}
|
||||
calls++
|
||||
return samplePlayerBody(resources[index].host, time.Now().Add(time.Hour)), nil
|
||||
})
|
||||
}
|
||||
if scenario == "rollback" {
|
||||
d.recheckGooglevideo = func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
|
||||
return errCandidateNonpass
|
||||
}
|
||||
}
|
||||
result, err := recoverManagedTarget(context.Background(), d)
|
||||
if scenario == "apply-mixed-guard" {
|
||||
var fatal *managedRecoveryFatal
|
||||
if !errors.As(err, &fatal) {
|
||||
t.Fatal("mixed guard downgraded", err)
|
||||
}
|
||||
assertActiveCandidates(t, d.transaction, d.db, old)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if scenario != "qualified" {
|
||||
if scenario == "one-host" || scenario == "rollback" || scenario == "apply-bad-body" {
|
||||
if result.GGCError == nil {
|
||||
t.Fatal("unqualified stage succeeded")
|
||||
}
|
||||
@@ -102,6 +158,15 @@ func TestManagedAdaptiveRecoveryProductionPath(t *testing.T) {
|
||||
t.Fatal(j, err)
|
||||
}
|
||||
assertActiveCandidates(t, d.transaction, d.db, append(old, c.ID))
|
||||
if scenario == "apply-learns" && applyCalls < 20 {
|
||||
t.Fatal("deployed warmup or full TLS transfers skipped", applyCalls)
|
||||
}
|
||||
if scenario == "avatar-missing" {
|
||||
status, err := d.health(context.Background())
|
||||
if status != "DEGRADED" || err == nil {
|
||||
t.Fatal("whole-service health concealed")
|
||||
}
|
||||
}
|
||||
if strings.Join(*events, ",") != "stop,start,restart" {
|
||||
t.Fatal("restore must precede apply", *events)
|
||||
}
|
||||
@@ -136,7 +201,39 @@ func adaptiveImageRunner(_ context.Context, args []string, stdin io.Reader, out,
|
||||
|
||||
func TestInsufficientAdaptiveSamplesCooldown(t *testing.T) {
|
||||
err := &InsufficientAdaptiveSamples{}
|
||||
if !errors.Is(err, errNoQualifiedStrategy) || !errors.Is(err, errGooglevideoInsufficientSamples) {
|
||||
if errors.Is(err, errNoQualifiedStrategy) || !errors.Is(err, errGooglevideoInsufficientSamples) || managedRecoveryCooldown(err, time.Minute) != time.Minute {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenuineNoQualifiedCooldown(t *testing.T) {
|
||||
if managedRecoveryCooldown(errNoQualifiedStrategy, time.Minute) != 2*time.Hour {
|
||||
t.Fatal("lost exhausted-search pause")
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedYouTubeDefaultSampleInventory(t *testing.T) {
|
||||
for _, primary := range []string{"dQw4w9WgXcQ", "aqz-KE-bpKQ"} {
|
||||
lookup := func(key string) (string, bool) { return "1", key == "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE" }
|
||||
_, ids, err := managedGooglevideoConfig(lookup, primary)
|
||||
want := []string{primary, "aqz-KE-bpKQ", "jNQXAC9IVRw"}
|
||||
if primary == "aqz-KE-bpKQ" {
|
||||
want = []string{primary, "jNQXAC9IVRw"}
|
||||
}
|
||||
if err != nil || !reflect.DeepEqual(ids, want) {
|
||||
t.Fatal(ids, err)
|
||||
}
|
||||
}
|
||||
_, ids, err := managedGooglevideoConfig(func(key string) (string, bool) {
|
||||
switch key {
|
||||
case "ZAPRET_AUTOTUNE_YOUTUBE_SERVICE":
|
||||
return "1", true
|
||||
case "ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS":
|
||||
return `["aaaaaaaaaaa"]`, true
|
||||
}
|
||||
return "", false
|
||||
}, "dQw4w9WgXcQ")
|
||||
if err != nil || !reflect.DeepEqual(ids, []string{"aaaaaaaaaaa"}) {
|
||||
t.Fatal(ids, err)
|
||||
}
|
||||
}
|
||||
@@ -57,7 +57,11 @@ func TestManagedGooglevideoFreshPrivateSamples(t *testing.T) {
|
||||
t.Fatal(actual, err)
|
||||
}
|
||||
}
|
||||
if calls != 2*len(ids) {
|
||||
want := 2 * len(ids)
|
||||
if scenario != "diverse" {
|
||||
want *= managedGooglevideoSampleRounds
|
||||
}
|
||||
if calls != want {
|
||||
t.Fatal("samples were reused", calls)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -192,9 +192,8 @@ func managedMonitorRecoveryAdapters(target managedDeps, db *sql.DB, scanCommand
|
||||
if err == nil && d.service {
|
||||
d.verifyAssets = newManagedAssetVerifier(target)
|
||||
d.recheckExperimental = newManagedExperimentalRecheck(target, d)
|
||||
if d.adaptive {
|
||||
d.verifyGooglevideo = newManagedGooglevideoVerifier(target)
|
||||
}
|
||||
d.verifyGooglevideo = newManagedGooglevideoVerifier(target)
|
||||
d.recheckGooglevideo = newManagedGooglevideoApplyRecheck(target, db)
|
||||
}
|
||||
return d, err
|
||||
}
|
||||
|
||||
@@ -40,6 +40,7 @@ type managedRecoveryDeps struct {
|
||||
mediaSamples func(context.Context) ([]mediaResource, error)
|
||||
verifyGooglevideo func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error)
|
||||
recheckExperimental func(context.Context, storedCandidate, mediaResource, *managedTransaction) error
|
||||
recheckGooglevideo func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error
|
||||
}
|
||||
|
||||
// Target-local adapter: reuses bounded Player requests without invoking a CLI.
|
||||
@@ -120,6 +121,9 @@ func managedRecoveryDependencies(target managedDeps, db *sql.DB, scan cacheScan,
|
||||
}
|
||||
if d.service {
|
||||
d.frontHealth = d.health
|
||||
// Construction is side-effect free; experiments may seek representative
|
||||
// qualification without changing the legacy default exact-host mode.
|
||||
d.mediaSamples = managedGooglevideoSamples(videoIDs, requestVideo)
|
||||
d.resources = managedServiceResources(cfg, request, target.player)
|
||||
d.health = managedServiceHealth(cfg.SOCKSProxy, d.frontHealth, d.resources, runResourceCurl)
|
||||
d.verifyAssets = unwiredManagedAssetVerifier
|
||||
@@ -279,7 +283,21 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
|
||||
if ctx.Err() != nil || errors.As(healthErr, &fatal) || errors.Is(healthErr, context.Canceled) || errors.Is(healthErr, context.DeadlineExceeded) {
|
||||
return result, managedRecoveryContextError(ctx, healthErr)
|
||||
}
|
||||
if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) && j.Scope != GooglevideoScopeExperimental {
|
||||
// A previously authorized native suffix stays adaptive after promotion,
|
||||
// even if the operator's legacy environment still says "cold". Stored
|
||||
// validation permits reconstruction, never fresh proof authorization.
|
||||
if j.Scope == GooglevideoScopeSuffix {
|
||||
for _, id := range j.Candidates {
|
||||
c, err := readStrategyCandidatePolicy(d.db, id, false)
|
||||
if err != nil {
|
||||
return result, &managedRecoveryFatal{Err: err}
|
||||
}
|
||||
if c.Group == "googlevideo" && c.Mode == QualificationAdaptive && c.Qualified {
|
||||
d.adaptive = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) && j.Scope != GooglevideoScopeExperimental && !d.adaptive {
|
||||
return result, repairManagedAssets(ctx, d, failure.Host)
|
||||
}
|
||||
}
|
||||
@@ -773,7 +791,7 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
|
||||
if err != nil {
|
||||
return &managedRecoveryFatal{Err: err}
|
||||
}
|
||||
if recovery.adaptive && candidate.Group == "googlevideo" && candidate.Mode == QualificationAdaptive && candidate.Qualified {
|
||||
if candidate.Group == "googlevideo" && candidate.Mode == QualificationAdaptive && candidate.Qualified {
|
||||
journal, e := transaction.load()
|
||||
if e != nil {
|
||||
return &managedRecoveryFatal{Err: e}
|
||||
|
||||
@@ -34,17 +34,18 @@ func TestManagedRecoveryProductionTransactionHooks(t *testing.T) {
|
||||
d.transaction = &managedTransaction{Dir: d.transaction.Dir, Name: "config"}
|
||||
d.service, d.adaptive = true, true
|
||||
d.frontHealth = d.health
|
||||
samples := 0
|
||||
d.mediaSamples = func(context.Context) ([]mediaResource, error) {
|
||||
t.Fatal("asset qualification must precede private media discovery")
|
||||
return nil, nil
|
||||
samples++
|
||||
return nil, &InsufficientAdaptiveSamples{}
|
||||
}
|
||||
d.verifyGooglevideo = func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error) {
|
||||
t.Fatal("unexpected verification")
|
||||
return GooglevideoQualificationProof{}, nil
|
||||
}
|
||||
result, err := recoverManagedTarget(context.Background(), d)
|
||||
var failure *managedResourceFailure
|
||||
if err != nil || result.FrontendID != front || !errors.As(result.GGCError, &failure) {
|
||||
var failure *InsufficientAdaptiveSamples
|
||||
if err != nil || result.FrontendID != front || samples != 1 || !errors.As(result.GGCError, &failure) {
|
||||
// Never expose private DB contents or error causes.
|
||||
t.Fatalf("frontend_preserved=%t recovery_error_type=%T ggc_error_type=%T", result.FrontendID == front, err, result.GGCError)
|
||||
}
|
||||
|
||||
@@ -93,9 +93,10 @@ func TestManagedExpiredAdaptiveJournalRecovery(t *testing.T) {
|
||||
verifies++
|
||||
return r.verifyGooglevideo(ctx, c, resources)
|
||||
}
|
||||
d.recheckGooglevideo = managedGooglevideoApplyRecheck(r, d.db)
|
||||
if scenario == "rollback" {
|
||||
d.health = func(context.Context) (string, error) {
|
||||
return "DEGRADED", &managedResourceFailure{"media", resources[0].host, nil}
|
||||
d.recheckGooglevideo = func(context.Context, storedCandidate, []mediaResource, *managedTransaction) error {
|
||||
return errCandidateNonpass
|
||||
}
|
||||
}
|
||||
result, err := recoverManagedTarget(context.Background(), d)
|
||||
|
||||
@@ -141,7 +141,7 @@ func runManagedMonitor(ctx context.Context, d managedMonitorDeps) (err error) {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
if errors.Is(err, errNoQualifiedStrategy) {
|
||||
if errors.Is(err, errNoQualifiedStrategy) || errors.Is(err, errGooglevideoInsufficientSamples) {
|
||||
if err := pauseMonitorRecovery(d.db, d.serviceID, d.now(), managedRecoveryCooldown(err, d.cooldown)); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -3,7 +3,8 @@ ZAPRET_AUTOTUNE_HOST_SOCKS_PORT=11080
|
||||
ZAPRET_AUTOTUNE_YOUTUBE_SERVICE=0
|
||||
ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE=cold
|
||||
ZAPRET_AUTOTUNE_VIDEO_ID=dQw4w9WgXcQ
|
||||
# Optional JSON array: 1..5 unique 11-character VideoIDs (not signed URLs).
|
||||
# Missing/empty selects automatic YouTube inventory: primary + two public samples.
|
||||
# Nonempty JSON overrides it: 1..5 unique 11-character VideoIDs (not signed URLs).
|
||||
# ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS='["dQw4w9WgXcQ","aqz-KE-bpKQ","jNQXAC9IVRw"]'
|
||||
# Optional target ss-server native DNS; empty retains the base resolver behavior.
|
||||
SS_DNS_SERVER=
|
||||
@@ -5,6 +5,29 @@ reconfigure, or stop the existing `zapret2-ss-autotune` container. The target
|
||||
starts from a disabled managed baseline; `managed monitor` bootstraps the
|
||||
frontend/API strategy, then discovers and applies an exact-host GGC strategy.
|
||||
|
||||
## New autonomous three-CDN source candidate (2026-10-13): offline only
|
||||
|
||||
Service-mode source now discovers representative inventory automatically and can
|
||||
promote an existing experiment only through the unchanged version-1 owned native
|
||||
proof factory: **three actual media hosts**, at least two roots and two prefixes,
|
||||
post-warmup 3/3 on both TLS protocols. Discovery is bounded to three rounds,
|
||||
15 requests maximum and three minutes; signed MP4 URLs need twenty minutes left.
|
||||
VideoIDs are not CDN evidence. No manual domain list or single-CDN relaxation is
|
||||
introduced. Missing/empty inventory defaults and explicit overrides are below.
|
||||
|
||||
Apply requires qualified frontend and fresh deployed checks of all three media
|
||||
resources, not avatar readiness. Whole-service health remains DEGRADED until avatar
|
||||
checks pass. Insufficient inventory preserves the committed experiment and uses
|
||||
configured cooldown; true strategy exhaustion retains its minimum two-hour pause.
|
||||
Suffix authorization remains extrapolative, not proof of every Googlevideo host.
|
||||
|
||||
This source is offline unit/fixture-verified only; no new image deployment or
|
||||
autonomous three-CDN field acceptance is claimed. The historical results below
|
||||
do not close that gate. Maximum strategy collection and PID1 global-failure
|
||||
stay-alive are unfinished future increments, as is arbitrary declarative service
|
||||
configuration. See [MANAGED_TARGET.md](../../MANAGED_TARGET.md) and
|
||||
[GOOGLEVIDEO_DISCOVERY.md](../../GOOGLEVIDEO_DISCOVERY.md).
|
||||
|
||||
## Recorded Googlevideo suffix experiment (2026-10-12)
|
||||
|
||||
### Final field follow-up: DEGRADED / NOT READY
|
||||
@@ -133,9 +156,11 @@ Isolated, accelerated shell supervision fixtures (no network):
|
||||
Compose forwards `ZAPRET_AUTOTUNE_YOUTUBE_SERVICE` (`0` or `1`, default `0`),
|
||||
`ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE` (`cold` or `adaptive`, default `cold`),
|
||||
`ZAPRET_AUTOTUNE_VIDEO_ID` (default `dQw4w9WgXcQ`) and
|
||||
`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` (JSON array of 1..5 unique VideoIDs;
|
||||
default only the primary ID). Explicit empty/invalid settings are rejected,
|
||||
not silently replaced. Adaptive requires service `1`. See `.env.example`.
|
||||
`ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS` (optional JSON array of 1..5 unique VideoIDs).
|
||||
Missing/empty inventory selects automatic service-mode samples: primary plus
|
||||
`aqz-KE-bpKQ` and `jNQXAC9IVRw`, deduplicated. A nonempty explicit inventory
|
||||
overrides those defaults, including an intentional single video. Invalid nonempty
|
||||
settings are rejected. Adaptive requires service `1`. See `.env.example`.
|
||||
Service `0` preserves legacy Player API/exact-host behavior.
|
||||
|
||||
Adaptive qualification is **post-warmup**, distinct from cold qualification;
|
||||
|
||||
@@ -45,9 +45,21 @@ with tempfile.TemporaryDirectory(prefix='target-config-') as tmp:
|
||||
assert json.loads(path.read_text())['password'] == base['SS_PASSWORD']
|
||||
# Render privately, never print the config/password; config does not deploy.
|
||||
compose_env = {k: v for k, v in base.items() if not k.startswith('ZAPRET_AUTOTUNE_')}
|
||||
factory_cases = []
|
||||
for overrides, service, mode, ids in (
|
||||
({}, '0', 'cold', ['dQw4w9WgXcQ']),
|
||||
({'ZAPRET_AUTOTUNE_VIDEO_ID': 'aqz-KE-bpKQ'}, '0', 'cold', ['aqz-KE-bpKQ']),
|
||||
({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive'},
|
||||
'1', 'adaptive', ['dQw4w9WgXcQ', 'aqz-KE-bpKQ', 'jNQXAC9IVRw']),
|
||||
({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive',
|
||||
'ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS': ''},
|
||||
'1', 'adaptive', ['dQw4w9WgXcQ', 'aqz-KE-bpKQ', 'jNQXAC9IVRw']),
|
||||
({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive',
|
||||
'ZAPRET_AUTOTUNE_VIDEO_ID': 'aqz-KE-bpKQ'},
|
||||
'1', 'adaptive', ['aqz-KE-bpKQ', 'jNQXAC9IVRw']),
|
||||
({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive',
|
||||
'ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS': '["aaaaaaaaaaa"]'},
|
||||
'1', 'adaptive', ['aaaaaaaaaaa']),
|
||||
({'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE': '1', 'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE': 'adaptive',
|
||||
'ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS': '["dQw4w9WgXcQ","aqz-KE-bpKQ","jNQXAC9IVRw"]',
|
||||
'SS_DNS_SERVER': '8.8.8.8'}, '1', 'adaptive', ['dQw4w9WgXcQ', 'aqz-KE-bpKQ', 'jNQXAC9IVRw']),
|
||||
@@ -60,7 +72,15 @@ with tempfile.TemporaryDirectory(prefix='target-config-') as tmp:
|
||||
settings = json.loads(result.stdout)['services']['target']['environment']
|
||||
assert settings['ZAPRET_AUTOTUNE_YOUTUBE_SERVICE'] == service
|
||||
assert settings['ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE'] == mode
|
||||
assert json.loads(settings['ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS']) == ids
|
||||
assert settings['ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS'] == overrides.get('ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS', '')
|
||||
assert settings['SS_DNS_SERVER'] == overrides.get('SS_DNS_SERVER', '')
|
||||
assert settings['ZAPRET_AUTOTUNE_SOCKS_PROXY'] == 'socks5h://127.0.0.1:1080'
|
||||
factory_cases.append({'environment': {k: v for k, v in settings.items()
|
||||
if k.startswith('ZAPRET_AUTOTUNE_')}, 'ids': ids})
|
||||
# Only sanitized fixture settings cross into Go; never persist the password.
|
||||
fixture = root / 'compose-factory.json'
|
||||
fixture.write_text(json.dumps(factory_cases))
|
||||
subprocess.run(['go', 'test', './cmd/zapret-autotune', '-run', '^TestManagedGooglevideoComposeForwarding$',
|
||||
'-count=1'], cwd=ROOT, env=dict(env, AUTOTUNE_COMPOSE_CONFIG_FIXTURE=str(fixture)),
|
||||
check=True, timeout=60)
|
||||
print('target config fixtures passed')
|
||||
@@ -26,7 +26,7 @@ for setting in (
|
||||
'ZAPRET_AUTOTUNE_YOUTUBE_SERVICE: ${ZAPRET_AUTOTUNE_YOUTUBE_SERVICE-0}',
|
||||
'ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE: ${ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE-cold}',
|
||||
'ZAPRET_AUTOTUNE_VIDEO_ID: ${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}',
|
||||
"ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS-[\"${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}\"]}'",
|
||||
"ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS:-}'",
|
||||
):
|
||||
assert setting in compose, setting
|
||||
assert entrypoint.index('target-config-check || exit 64') < entrypoint.index('cat > "$config_dir/server.json"')
|
||||
|
||||
@@ -16,7 +16,7 @@ services:
|
||||
ZAPRET_AUTOTUNE_YOUTUBE_SERVICE: ${ZAPRET_AUTOTUNE_YOUTUBE_SERVICE-0}
|
||||
ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE: ${ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE-cold}
|
||||
ZAPRET_AUTOTUNE_VIDEO_ID: ${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}
|
||||
ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS-["${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}"]}'
|
||||
ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS:-}'
|
||||
SOCKS_PORT: "1080"
|
||||
ZAPRET_AUTOTUNE_TARGET_ROLE: managed-test
|
||||
ZAPRET_AUTOTUNE_TARGET_ID: zapret-autotune-managed-v1
|
||||
|
||||
Reference in new issue
Block a user