feat: recover experimental Googlevideo media autonomously
Preserve operator-owned profiles while rechecking fresh media and bounded native relearning in the deployed daemon. Keep mixed safety failures fatal and expose active coverage separately from qualification. Document source-only acceptance and retain existing recovery budgets and privacy boundaries. Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
1a35cb3019
commit
a60f62de22
14 files changed
+878
-41
No files matched your search
+22
-4
@@ -419,7 +419,25 @@ do not blindly retry it. Use the matched whole-state/image rollback above.
|
|||||||
Candidate-ID CLI apply cannot select/inherit the experimental policy; explicit
|
Candidate-ID CLI apply cannot select/inherit the experimental policy; explicit
|
||||||
native-circular authorization is required. Internal journal recovery is unchanged.
|
native-circular authorization is required. Internal journal recovery is unchanged.
|
||||||
|
|
||||||
Resource probes still run. This policy remains `DEGRADED`/GGC-unresolved even
|
Resource probes still run. In service mode (legacy or adaptive), healthy actual
|
||||||
when a bounded media check succeeds: neither playback READY nor family
|
image/media probes short-circuit recovery, but operational health does not mean
|
||||||
qualification is inferred. Automatic frontend/image repairs preserve the
|
family qualification or playback READY. Experimental media recovery preserves
|
||||||
operator suffix; automatic GGC recovery does not replace or qualify it.
|
the operator suffix and all committed profile IDs: a validated transfer failure
|
||||||
|
permits one guarded restart of that same deployed daemon, then at most 10 native
|
||||||
|
relearning attempts within 120s and a final fresh signed media 3/3 for each of
|
||||||
|
TLS1.2 and TLS1.3. Learning stays in the persistent daemon's RAM; no temporary
|
||||||
|
NFQ worker or replacement/qualified candidate is created. Expiring media is
|
||||||
|
rediscovered, never rewritten or persisted. Missing inventory remains degraded,
|
||||||
|
not strategy absence; mixed, guard, infrastructure and cancellation errors do
|
||||||
|
not authorize relearning. Unrelated missing image inventory may coexist with
|
||||||
|
usable media, but successful media recovery does not repair or qualify it.
|
||||||
|
|
||||||
|
`managed status` reports active-profile `covered=` separately from qualified
|
||||||
|
`groups` and `unresolved_groups`; `managed active-ids` adds `covered_groups` and
|
||||||
|
`googlevideo_scope` to its JSON. An experimental Googlevideo profile can be
|
||||||
|
covered yet qualification-unresolved; one image profile does not establish full
|
||||||
|
asset-inventory success. Status is read-only and reports live health NOT CHECKED.
|
||||||
|
The new autonomous recovery/classifier/status behavior is source-only pending
|
||||||
|
field checks, not a deployment or production-acceptance claim. Generic production
|
||||||
|
media behavior and signed-URL/SS secret privacy remain unchanged, as do the
|
||||||
|
persisted minimum two-hour strategy-absence pause and 90-minute whole-job budget.
|
||||||
+6
-4
@@ -6,18 +6,20 @@
|
|||||||
|
|
||||||
### Googlevideo suffix RC — browser-verified experiment (2026-10-12)
|
### Googlevideo suffix RC — browser-verified experiment (2026-10-12)
|
||||||
|
|
||||||
- Source RC `eade051c2bc335b71b1616d697326ca4c8019d70`, immutable image `sha256:fdb60593e29d7c337a413a47aaddfbcaaeabdeb877042c58f506d4abdc98a13b`, isolated host SOCKS `127.0.0.1:11082`. Recorded journal: generation3, active candidates [1,2,3], scope `experimental_googlevideo_suffix`; `www.youtube.com` and `i.ytimg.com` profiles preserved. Older targets were unchanged. These are recorded field observations, not a new probe/deployment by this documentation commit; its SHA is not the tested image source.
|
- Initial source RC `eade051c2bc335b71b1616d697326ca4c8019d70`, immutable image `sha256:fdb60593e29d7c337a413a47aaddfbcaaeabdeb877042c58f506d4abdc98a13b`, isolated host SOCKS `127.0.0.1:11082`. Historical initial journal: generation3, active candidates [1,2,3], scope `experimental_googlevideo_suffix`; `www.youtube.com` and `i.ytimg.com` profiles preserved. Older targets were unchanged. These are recorded field observations, not current candidate IDs or a new probe/deployment by this source update; its SHA is not the tested image source.
|
||||||
- Fresh signed media on `rr16---sn-n8v7znsz.googlevideo.com` returned HTTP206, 65536 bytes, MP4 `ftyp`, TLS1.2. On **2026-10-12 the user confirmed browser videos work** with this live RC and accepted one shared profile for Googlevideo, including initial learning. Browser images also appeared, but that does not qualify every image route.
|
- Fresh signed media on `rr16---sn-n8v7znsz.googlevideo.com` returned HTTP206, 65536 bytes, MP4 `ftyp`, TLS1.2. On **2026-10-12 the user confirmed browser videos work** with this live RC and accepted one shared profile for Googlevideo, including initial learning. Browser images also appeared, but that does not qualify every image route.
|
||||||
- One suffix profile is a **composite native circular A/E profile with two strategies and per-host learning**, not one uniform static strategy. Explicit operator experimental policy is durable across restart with hash/rollback guards; it is neither family-qualified nor `READY`, and no all-host cold 3/3 result is claimed.
|
- One suffix profile is a **composite native circular A/E profile with two strategies and per-host learning**, not one uniform static strategy. Explicit operator experimental policy is durable across restart with hash/rollback guards; it is neither family-qualified nor `READY`, and no all-host cold 3/3 result is claimed.
|
||||||
- Service health already checks actual fresh image resources and media with both TLS protocols. Experimental recovery intentionally returns without automatically qualifying or replacing the operator-approved experiment: the remaining gap is autonomous experimental recovery/autoqualification, not missing probe implementation. Operator scope coverage and qualification are separate states.
|
- Source now wires autonomous experimental media recovery in either configured service mode. Healthy actual-resource probes short-circuit recovery; a validated media transfer failure permits one guarded restart of the same committed profiles and bounded native relearning in the persistent deployed daemon (10 attempts / 120s), followed by fresh signed media 3/3 on both TLS protocols. Learning stays in daemon RAM, not a temporary verification worker; expired inventory is refreshed, never rewritten or persisted. Missing inventory is degraded, not strategy absence; guard/infrastructure/cancellation and mixed failures remain fail-closed. No experimental candidate is replaced or promoted to qualified.
|
||||||
|
- Status distinguishes active-profile coverage (`covered=` / JSON `covered_groups`, `googlevideo_scope`) from qualified `groups` and `unresolved_groups`. Experimental Googlevideo remains unresolved for qualification even when operational media is healthy; a thumbnail profile does not mean the full asset inventory is qualified. Status itself performs no live health check. New recovery/classifier/status code is pending field acceptance; no new deployment, network test, family-qualified/`READY` or production claim is made. Generic production media handling, signed-URL/SS privacy, persisted minimum two-hour strategy-absence pause and the separate 90-minute whole-job budget are unchanged.
|
||||||
|
- Local source verification: fresh `go test ./... -count=1`, affected experimental/resource-classifier/status/monitor regression tests under `-race`, and `git diff --check` passed. These are unit/fixture checks only, not new field evidence.
|
||||||
|
|
||||||
Remaining acceptance gates (not asserted defects):
|
Remaining acceptance gates (not asserted defects):
|
||||||
|
|
||||||
1. Representative fresh signed media evidence: at least three unique hosts across at least two roots/prefixes, post-warmup 3/3 for both TLS protocols plus actual media transfer. Do not impose uniform static/cold behavior on the accepted learning experiment or label it normally qualified without evidence.
|
1. Representative fresh signed media evidence: at least three unique hosts across at least two roots/prefixes, post-warmup 3/3 for both TLS protocols plus actual media transfer. Do not impose uniform static/cold behavior on the accepted learning experiment or label it normally qualified without evidence.
|
||||||
2. Qualify the avatar route/profile: only the thumbnail image profile is currently active; browser-visible images do not close this gate.
|
2. Qualify the avatar route/profile: only the thumbnail image profile is currently active; browser-visible images do not close this gate.
|
||||||
3. Field-accept actual restart/recreation recovery after the all-Googlevideo suffix change, including renewed native learning; older exact-host lifecycle PASS does not cover this RC.
|
3. Field-accept actual restart/recreation recovery after the all-Googlevideo suffix change, including renewed native learning; older exact-host lifecycle PASS does not cover this RC.
|
||||||
4. Resolve autonomous recovery/autoqualification policy for the approved experiment without silently replacing or promoting its operator-approved scope.
|
4. Field-accept the new autonomous same-profile experimental recovery and resource-error classifier without replacing or promoting operator-approved scope. Automatic family qualification is not implemented.
|
||||||
5. Reconcile service status with those results: distinguish operator-covered suffix from unresolved qualification/assets; no family-qualified/`READY` or blanket production claim yet.
|
5. Field-check the new covered-versus-qualified status fields against the live journal and asset inventory; no family-qualified/`READY` or blanket production claim yet.
|
||||||
|
|
||||||
### Earlier isolated-target acceptance (2026-10-11)
|
### Earlier isolated-target acceptance (2026-10-11)
|
||||||
|
|
||||||
|
|||||||
@@ -38,6 +38,18 @@ func (e *managedResourceFailure) PublicError() string {
|
|||||||
}
|
}
|
||||||
func (e *managedResourceFailure) Unwrap() error { return e.Err }
|
func (e *managedResourceFailure) Unwrap() error { return e.Err }
|
||||||
|
|
||||||
|
// Only a typed collector's complete known-negative error set may lose its
|
||||||
|
// generic fatal wrapper at a live-health boundary. Missing resources remain
|
||||||
|
// DEGRADED; joined guard/infrastructure/cancellation failures retain their type.
|
||||||
|
func managedResourceHealthError(ctx context.Context, err error) error {
|
||||||
|
var failure *managedResourceFailure
|
||||||
|
var collector *resourceError
|
||||||
|
if ctx.Err() == nil && errors.As(err, &failure) && (failure.ProbeKind == "media" || assetEndpoint(failure.Host) && failure.ProbeKind == managedImageProbeKind(failure.Host)) && errors.As(failure.Err, &collector) && managedResourceTransferNonpass(err, assetEndpoint(failure.Host)) {
|
||||||
|
return &managedResourceFailure{ProbeKind: failure.ProbeKind, Host: failure.Host}
|
||||||
|
}
|
||||||
|
return managedRecoveryContextError(ctx, err)
|
||||||
|
}
|
||||||
|
|
||||||
// Ephemeral inventory: never serialize URLs, page bodies or Player responses.
|
// Ephemeral inventory: never serialize URLs, page bodies or Player responses.
|
||||||
type managedServiceInventory struct {
|
type managedServiceInventory struct {
|
||||||
images map[string]string
|
images map[string]string
|
||||||
@@ -85,7 +97,11 @@ func managedServiceResources(cfg healthConfig, player func(context.Context) ([]b
|
|||||||
return i, managedRecoveryContextError(ctx, err)
|
return i, managedRecoveryContextError(ctx, err)
|
||||||
}
|
}
|
||||||
if err != nil || bounded.Err() != nil || out.exceeded {
|
if err != nil || bounded.Err() != nil || out.exceeded {
|
||||||
return i, &managedResourceFailure{"avatar", "yt3.ggpht.com", resourceFailure(bounded, errors.Join(err, bounded.Err(), errors.New("fresh root metadata request failed")))}
|
cause := errors.Join(err, bounded.Err())
|
||||||
|
if cause == nil || out.exceeded {
|
||||||
|
cause = errors.Join(cause, errors.New("fresh root metadata request failed"))
|
||||||
|
}
|
||||||
|
return i, managedResourceHealthError(ctx, &managedResourceFailure{"avatar", "yt3.ggpht.com", resourceFailure(bounded, cause)})
|
||||||
}
|
}
|
||||||
if raw, e := avatarResourceFromPage(output.Bytes()); e == nil {
|
if raw, e := avatarResourceFromPage(output.Bytes()); e == nil {
|
||||||
i.images["yt3.ggpht.com"] = raw
|
i.images["yt3.ggpht.com"] = raw
|
||||||
|
|||||||
@@ -272,6 +272,7 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
|
|||||||
return errManagedDrift
|
return errManagedDrift
|
||||||
}
|
}
|
||||||
groups := map[string]bool{}
|
groups := map[string]bool{}
|
||||||
|
operatorCoverage := map[string]bool{}
|
||||||
assetHosts := map[string]bool{}
|
assetHosts := map[string]bool{}
|
||||||
known := j.State == "committed" && j.Version >= 2 && string(b) == managedBaseline && len(j.Candidates) == 0
|
known := j.State == "committed" && j.Version >= 2 && string(b) == managedBaseline && len(j.Candidates) == 0
|
||||||
if j.State == "committed" && j.Version >= 2 && len(j.Candidates) != 0 {
|
if j.State == "committed" && j.Version >= 2 && len(j.Candidates) != 0 {
|
||||||
@@ -290,6 +291,7 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
groups[c.Group] = true
|
groups[c.Group] = true
|
||||||
|
operatorCoverage[c.Group] = true
|
||||||
if experimentalGooglevideo(j.Scope, c) {
|
if experimentalGooglevideo(j.Scope, c) {
|
||||||
groups[c.Group] = false
|
groups[c.Group] = false
|
||||||
}
|
}
|
||||||
@@ -304,6 +306,7 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
|
|||||||
}
|
}
|
||||||
if !known {
|
if !known {
|
||||||
groups = map[string]bool{}
|
groups = map[string]bool{}
|
||||||
|
operatorCoverage = map[string]bool{}
|
||||||
// Unvalidated journal references are not active recovery candidates.
|
// Unvalidated journal references are not active recovery candidates.
|
||||||
j.Candidates = nil
|
j.Candidates = nil
|
||||||
}
|
}
|
||||||
@@ -321,14 +324,29 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
|
|||||||
if (groups["assets"] || (len(requestAssets) != 0 && requestAssets[0])) && (!known || len(assetHosts) != 2) {
|
if (groups["assets"] || (len(requestAssets) != 0 && requestAssets[0])) && (!known || len(assetHosts) != 2) {
|
||||||
missing = append(missing, "assets")
|
missing = append(missing, "assets")
|
||||||
}
|
}
|
||||||
|
// Coverage labels active profiles, not qualification or the full host inventory.
|
||||||
|
covered := []string{}
|
||||||
|
for _, group := range []string{"frontend", "assets", "googlevideo"} {
|
||||||
|
if operatorCoverage[group] {
|
||||||
|
covered = append(covered, group)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
scope := j.Scope
|
||||||
|
if !known {
|
||||||
|
scope = "unknown"
|
||||||
|
} else if operatorCoverage["googlevideo"] && scope == "" {
|
||||||
|
scope = GooglevideoScopeExact
|
||||||
|
}
|
||||||
if activeOnly {
|
if activeOnly {
|
||||||
if !known {
|
if !known {
|
||||||
return fmt.Errorf("active candidate IDs unresolved; deployment preserved by hash")
|
return fmt.Errorf("active candidate IDs unresolved; deployment preserved by hash")
|
||||||
}
|
}
|
||||||
return json.NewEncoder(out).Encode(struct {
|
return json.NewEncoder(out).Encode(struct {
|
||||||
IDs []int64 `json:"active_candidate_ids"`
|
IDs []int64 `json:"active_candidate_ids"`
|
||||||
Missing []string `json:"unresolved_groups"`
|
Missing []string `json:"unresolved_groups"`
|
||||||
}{append([]int64{}, j.Candidates...), missing})
|
Covered []string `json:"covered_groups"`
|
||||||
|
Scope GooglevideoScope `json:"googlevideo_scope"`
|
||||||
|
}{append([]int64{}, j.Candidates...), missing, covered, scope})
|
||||||
}
|
}
|
||||||
groupText := strings.Join(active, ",")
|
groupText := strings.Join(active, ",")
|
||||||
if groupText == "" {
|
if groupText == "" {
|
||||||
@@ -341,9 +359,20 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
|
|||||||
if known {
|
if known {
|
||||||
reportIDs = j.Candidates
|
reportIDs = j.Candidates
|
||||||
}
|
}
|
||||||
_, err = fmt.Fprintf(out, "managed-test generation=%d hash=%s state=%s groups=%s active_candidate_ids=%v unresolved_groups=%s\nLive health: NOT CHECKED (Player API is not GGC/video playback)\n", j.Generation, j.Hash, j.State, groupText, reportIDs, strings.Join(missing, ","))
|
coverageText := strings.Join(covered, ",")
|
||||||
|
if coverageText == "" {
|
||||||
|
coverageText = "none"
|
||||||
|
if !known {
|
||||||
|
coverageText = "unknown"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "managed-test generation=%d hash=%s state=%s groups=%s active_candidate_ids=%v unresolved_groups=%s covered=%s\nLive health: NOT CHECKED (Player API is not GGC/video playback)\nCoverage: active config profiles only, not qualification or all-host/asset-inventory success.\n", j.Generation, j.Hash, j.State, groupText, reportIDs, strings.Join(missing, ","), coverageText)
|
||||||
if err == nil && j.Version == 3 {
|
if err == nil && j.Version == 3 {
|
||||||
_, err = fmt.Fprintf(out, "Googlevideo scope: %s (suffix is authorized extrapolation, not all-host qualification)\n", j.Scope)
|
if known && scope == GooglevideoScopeExperimental {
|
||||||
|
_, err = fmt.Fprintf(out, "Googlevideo scope: %s (experimental/unqualified; one authorized composite profile, not all hosts checked)\n", scope)
|
||||||
|
} else {
|
||||||
|
_, err = fmt.Fprintf(out, "Googlevideo scope: %s (suffix is authorized extrapolation, not all-host qualification)\n", scope)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return err
|
return err
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -458,6 +458,144 @@ func TestManagedCLILegacyIDsUnresolved(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestManagedStatusOperatorCoverage(t *testing.T) {
|
||||||
|
for _, mode := range []string{"qualified", "experimental", "missing-candidate", "missing-db", "render-mismatch", "invalid-hash", "legacy", "pending"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
db := testStore(t)
|
||||||
|
front := managedCandidateFixture(t, db, "frontend", "www.youtube.com", "--lua-desync=fake", true)
|
||||||
|
ids := []int64{front}
|
||||||
|
scope := GooglevideoScopeExact
|
||||||
|
if mode == "qualified" {
|
||||||
|
ids = append(ids, managedCandidateFixture(t, db, "googlevideo", "rr1.googlevideo.com", "--lua-desync=fake", true))
|
||||||
|
} else {
|
||||||
|
asset := unitAssetCandidate(t, db, "i.ytimg.com", "--lua-desync=fake", true)
|
||||||
|
tx, err := db.Begin()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ggc, err := insertStrategyCandidate(tx, "googlevideo", []string{"rr1---sn-alpha.googlevideo.com"}, managedGooglevideoArgs, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := tx.Commit(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ids = append(ids, asset, ggc)
|
||||||
|
scope = GooglevideoScopeExperimental
|
||||||
|
c, err := readStrategyCandidate(db, ggc)
|
||||||
|
if err != nil || c.Qualified || c.Mode != QualificationCold {
|
||||||
|
t.Fatalf("experimental candidate: %+v %v", c, err)
|
||||||
|
}
|
||||||
|
var evidence int
|
||||||
|
if err := db.QueryRow("SELECT count(*) FROM candidate_evidence WHERE candidate_id=?", ggc).Scan(&evidence); err != nil || evidence != 0 {
|
||||||
|
t.Fatalf("fabricated evidence: %d %v", evidence, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
m, _ := managedFixture(t)
|
||||||
|
if err := m.Apply(db, ids, scope); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
j, err := m.load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
openDB := func() (*sql.DB, error) { return openHistoryStore(databasePath()) }
|
||||||
|
switch mode {
|
||||||
|
case "missing-candidate":
|
||||||
|
j.Candidates[len(ids)-1] = 999999
|
||||||
|
case "missing-db":
|
||||||
|
openDB = func() (*sql.DB, error) { return nil, errors.New("unavailable") }
|
||||||
|
case "render-mismatch":
|
||||||
|
b := []byte("# not the rendered candidates\n" + managedBaseline)
|
||||||
|
if err := os.WriteFile(filepath.Join(m.Dir, m.Name), b, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
j.Hash = managedHash(b)
|
||||||
|
case "invalid-hash":
|
||||||
|
j.Hash = managedHash([]byte("wrong hash"))
|
||||||
|
case "legacy":
|
||||||
|
j.Version, j.Scope = 2, ""
|
||||||
|
case "pending":
|
||||||
|
j.State, j.Next = "prepared", j.Hash
|
||||||
|
j.PendingCandidates, j.PendingScope = append([]int64{}, ids...), scope
|
||||||
|
}
|
||||||
|
if err := m.journal(j); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
before, err := os.ReadFile(filepath.Join(m.Dir, m.Name))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
journalBefore, err := os.ReadFile(filepath.Join(m.Dir, managedJournalName))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
m.Health = func() (string, error) { t.Fatal("status checked live health"); return "", nil }
|
||||||
|
m.Reload = func() error { t.Fatal("status reloaded target"); return nil }
|
||||||
|
var out bytes.Buffer
|
||||||
|
err = managedReport(m, &out, openDB, false)
|
||||||
|
if mode == "invalid-hash" {
|
||||||
|
if !errors.Is(err, errManagedDrift) || out.Len() != 0 {
|
||||||
|
t.Fatalf("drift trusted: %s %v", out.String(), err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if err != nil || !strings.Contains(out.String(), "Live health: NOT CHECKED") || strings.Contains(out.String(), "READY") {
|
||||||
|
t.Fatalf("status: %s %v", out.String(), err)
|
||||||
|
}
|
||||||
|
want := []string{"groups=unknown ", "covered=unknown", "active_candidate_ids=[] ", "unresolved_groups=frontend,googlevideo"}
|
||||||
|
if mode == "qualified" {
|
||||||
|
want = []string{"groups=frontend,googlevideo ", "covered=frontend,googlevideo", "unresolved_groups= covered="}
|
||||||
|
} else if mode == "experimental" {
|
||||||
|
want = []string{"groups=frontend,assets ", "covered=frontend,assets,googlevideo", "unresolved_groups=googlevideo,assets ", "experimental/unqualified", "one authorized composite profile", "not all hosts checked"}
|
||||||
|
}
|
||||||
|
for _, text := range want {
|
||||||
|
if !strings.Contains(out.String(), text) {
|
||||||
|
t.Fatalf("missing %q: %s", text, out.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.Reset()
|
||||||
|
err = managedReport(m, &out, openDB, true)
|
||||||
|
if mode == "qualified" || mode == "experimental" {
|
||||||
|
var state struct {
|
||||||
|
IDs []int64 `json:"active_candidate_ids"`
|
||||||
|
Missing []string `json:"unresolved_groups"`
|
||||||
|
Covered []string `json:"covered_groups"`
|
||||||
|
Scope GooglevideoScope `json:"googlevideo_scope"`
|
||||||
|
}
|
||||||
|
if err != nil || json.Unmarshal(out.Bytes(), &state) != nil || !reflect.DeepEqual(state.IDs, ids) || state.Scope != scope {
|
||||||
|
t.Fatalf("JSON: %s %v", out.String(), err)
|
||||||
|
}
|
||||||
|
covered, missing := []string{"frontend", "googlevideo"}, []string(nil)
|
||||||
|
if mode == "experimental" {
|
||||||
|
covered, missing = []string{"frontend", "assets", "googlevideo"}, []string{"googlevideo", "assets"}
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(state.Covered, covered) || !reflect.DeepEqual(state.Missing, missing) {
|
||||||
|
t.Fatalf("coverage changed qualification: %+v", state)
|
||||||
|
}
|
||||||
|
encoded, err := json.Marshal(state)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var roundtrip map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(encoded, &roundtrip); err != nil || len(roundtrip) != 4 {
|
||||||
|
t.Fatalf("JSON roundtrip: %s %v", encoded, err)
|
||||||
|
}
|
||||||
|
} else if err == nil || out.Len() != 0 {
|
||||||
|
t.Fatalf("invalid IDs trusted: %s %v", out.String(), err)
|
||||||
|
}
|
||||||
|
after, err := os.ReadFile(filepath.Join(m.Dir, m.Name))
|
||||||
|
if err != nil || !bytes.Equal(before, after) {
|
||||||
|
t.Fatal("status changed config", err)
|
||||||
|
}
|
||||||
|
journalAfter, err := os.ReadFile(filepath.Join(m.Dir, managedJournalName))
|
||||||
|
if err != nil || !bytes.Equal(journalBefore, journalAfter) {
|
||||||
|
t.Fatal("status changed journal", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestManagedMountpoint(t *testing.T) {
|
func TestManagedMountpoint(t *testing.T) {
|
||||||
for _, path := range []string{"/", filepath.Dir(managedTargetDirectory), managedTargetDirectory + "-other"} {
|
for _, path := range []string{"/", filepath.Dir(managedTargetDirectory), managedTargetDirectory + "-other"} {
|
||||||
if managedMountpoint("1 2 0:3 / "+path+" rw - tmpfs tmpfs rw\n", managedTargetDirectory) {
|
if managedMountpoint("1 2 0:3 / "+path+" rw - tmpfs tmpfs rw\n", managedTargetDirectory) {
|
||||||
|
|||||||
@@ -0,0 +1,211 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os/exec"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExperimentalProductionProbeCollectorClassification(t *testing.T) {
|
||||||
|
for _, mode := range []string{"bad-range", "healthy", "unknown", "infra", "mixed", "guard", "cancel", "metadata-timeout", "metadata-http", "metadata-malformed", "metadata-mixed", "metadata-cancel", "metadata-infra", "image-timeout", "image-mixed"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, front, _ := recoveryFixture(t, true)
|
||||||
|
target, _, _ := managedCLIFixture(t)
|
||||||
|
initManagedCLI(t, target)
|
||||||
|
d.transaction.Dir = target.directory
|
||||||
|
asset := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
|
||||||
|
id := managedCandidateFixture(t, d.db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
|
||||||
|
ids := []int64{front, asset, id}
|
||||||
|
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeExperimental); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
before, err := d.transaction.load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
d.service = true
|
||||||
|
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
|
||||||
|
d.resources = func(context.Context) (managedServiceInventory, error) {
|
||||||
|
r := experimentalResourceFixture()
|
||||||
|
if mode == "unknown" {
|
||||||
|
r = mediaResource{}
|
||||||
|
}
|
||||||
|
return managedServiceInventory{images: map[string]string{"i.ytimg.com": "https://i.ytimg.com/test", "yt3.ggpht.com": "https://yt3.ggpht.com/test"}, media: r}, nil
|
||||||
|
}
|
||||||
|
knownTimeout := exec.Command("sh", "-c", "exit 28").Run()
|
||||||
|
if strings.HasPrefix(mode, "metadata-") {
|
||||||
|
d.resources = managedServiceResources(healthConfig{SOCKSProxy: "socks5://127.0.0.1:11080"}, func(context.Context) ([]byte, error) {
|
||||||
|
return []byte(fmt.Sprintf(`{"playabilityStatus":{"status":"OK"},"streamingData":{"formats":[{"url":%q,"mimeType":"video/mp4"}]}}`, experimentalResourceFixture().url)), nil
|
||||||
|
}, func(_ context.Context, args []string, out, diagnostic io.Writer) error {
|
||||||
|
if args[len(args)-1] != "https://www.youtube.com/" || diagnostic != io.Discard {
|
||||||
|
t.Fatal("metadata origin/diagnostics changed")
|
||||||
|
}
|
||||||
|
switch mode {
|
||||||
|
case "metadata-malformed":
|
||||||
|
_, err := io.WriteString(out, "<html>not playable metadata</html>")
|
||||||
|
return err
|
||||||
|
case "metadata-http":
|
||||||
|
return exec.Command("sh", "-c", "exit 22").Run()
|
||||||
|
case "metadata-mixed":
|
||||||
|
return errors.Join(knownTimeout, &managedRecoveryFatal{Err: errors.New("guard failed")})
|
||||||
|
case "metadata-cancel":
|
||||||
|
return errors.Join(knownTimeout, context.Canceled)
|
||||||
|
case "metadata-infra":
|
||||||
|
return errors.Join(knownTimeout, errors.New("collector unavailable"))
|
||||||
|
default:
|
||||||
|
return knownTimeout
|
||||||
|
}
|
||||||
|
})
|
||||||
|
i, _ := d.resources(context.Background())
|
||||||
|
if !freshExperimentalMedia(i.media) {
|
||||||
|
t.Fatal("metadata failure discarded fresh media")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
calls := 0
|
||||||
|
d.recheckExperimental = func(_ context.Context, c storedCandidate, r mediaResource) error {
|
||||||
|
if c.ID != id || !freshExperimentalMedia(r) {
|
||||||
|
t.Fatal("recovery changed candidate or lost fresh media")
|
||||||
|
}
|
||||||
|
calls++
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
health := managedServiceHealth("socks5://127.0.0.1:11080", d.frontHealth, d.resources, func(_ context.Context, args []string, in io.Reader, out, diagnostic io.Writer) error {
|
||||||
|
if strings.HasPrefix(mode, "metadata-") {
|
||||||
|
t.Fatal("missing images must not become transfer evidence")
|
||||||
|
}
|
||||||
|
if mode == "image-timeout" {
|
||||||
|
return knownTimeout
|
||||||
|
}
|
||||||
|
if mode == "image-mixed" {
|
||||||
|
return errors.Join(knownTimeout, &managedRecoveryFatal{Err: errors.New("guard failed")})
|
||||||
|
}
|
||||||
|
config, _ := io.ReadAll(in)
|
||||||
|
raw := strings.TrimSuffix(strings.TrimPrefix(string(config), "url = \""), "\"\n")
|
||||||
|
var wire, diag bytes.Buffer
|
||||||
|
wireMode := ""
|
||||||
|
if strings.Contains(raw, ".googlevideo.com/") {
|
||||||
|
wireMode = "media"
|
||||||
|
if mode != "healthy" {
|
||||||
|
wireMode = "bad-range"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
resourceWire(raw, wireMode, &wire, &diag)
|
||||||
|
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
|
||||||
|
diag = *bytes.NewBufferString(strings.ReplaceAll(diag.String(), "TLSv1.2", "TLSv1.3"))
|
||||||
|
}
|
||||||
|
_, _ = out.Write(wire.Bytes())
|
||||||
|
_, _ = diagnostic.Write(diag.Bytes())
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
d.health = func(ctx context.Context) (string, error) {
|
||||||
|
status, err := health(ctx)
|
||||||
|
if mode != "healthy" && status != "DEGRADED" {
|
||||||
|
t.Fatalf("resource failure became healthy: %s", status)
|
||||||
|
}
|
||||||
|
switch mode {
|
||||||
|
case "infra":
|
||||||
|
err = &managedResourceFailure{"media", experimentalResourceFixture().host, resourceFailure(ctx, errors.New("collector unavailable"))}
|
||||||
|
case "mixed":
|
||||||
|
err = errors.Join(err, &managedRecoveryFatal{Err: errors.New("guard failed")})
|
||||||
|
case "guard":
|
||||||
|
err = errors.Join(err, &managedRecoveryFatal{Err: errors.New("invalid media range or container")})
|
||||||
|
case "cancel":
|
||||||
|
err = errors.Join(err, context.Canceled)
|
||||||
|
}
|
||||||
|
return status, err
|
||||||
|
}
|
||||||
|
monitor := monitorRuntimeFixture(t)
|
||||||
|
monitor.db, monitor.target = d.db, target
|
||||||
|
monitor.threshold = 1
|
||||||
|
if err := wireManagedRecovery(&monitor, d, func(error) {}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
err = monitor.probe(context.Background())
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
switch mode {
|
||||||
|
case "healthy":
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "bad-range", "metadata-timeout", "metadata-http", "metadata-malformed", "image-timeout":
|
||||||
|
if err == nil || errors.As(err, &fatal) {
|
||||||
|
t.Fatalf("media damage killed monitor: %v", err)
|
||||||
|
}
|
||||||
|
case "unknown":
|
||||||
|
if err == nil || errors.Is(err, errNoQualifiedStrategy) {
|
||||||
|
t.Fatalf("unknown became green/absence: %v", err)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
if !errors.As(err, &fatal) {
|
||||||
|
t.Fatalf("safety failure normalized: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode != "healthy" && mode != "unknown" {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
monitor.wait = func(ctx context.Context, _ time.Duration) error { cancel(); return ctx.Err() }
|
||||||
|
err = runManagedMonitor(ctx, monitor)
|
||||||
|
if mode == "bad-range" || mode == "metadata-timeout" || mode == "metadata-http" || mode == "metadata-malformed" || mode == "image-timeout" {
|
||||||
|
if !errors.Is(err, context.Canceled) || errors.As(err, &fatal) || calls != 1 {
|
||||||
|
t.Fatalf("monitor recovery unreachable: %v calls=%d", err, calls)
|
||||||
|
}
|
||||||
|
} else if !errors.As(err, &fatal) || calls != 0 {
|
||||||
|
t.Fatalf("fatal probe entered recovery: %v calls=%d", err, calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
after, err := d.transaction.load()
|
||||||
|
if err != nil || !reflect.DeepEqual(before, after) {
|
||||||
|
t.Fatalf("probe/recovery changed journal: %v", err)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, ids)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedResourceHealthClassificationKeepsSafetyAndEvidenceBoundaries(t *testing.T) {
|
||||||
|
timeout := exec.Command("sh", "-c", "exit 28").Run()
|
||||||
|
httpFailure := exec.Command("sh", "-c", "exit 22").Run()
|
||||||
|
if managedGooglevideoTransferNonpass(resourceFailure(context.Background(), httpFailure)) {
|
||||||
|
t.Fatal("metadata HTTP nonpass became media rejection evidence")
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
for _, tc := range []struct {
|
||||||
|
ctx context.Context
|
||||||
|
err error
|
||||||
|
}{
|
||||||
|
{context.Background(), &managedRecoveryFatal{Err: timeout}},
|
||||||
|
{context.Background(), errors.Join(resourceFailure(context.Background(), timeout), &managedRecoveryFatal{Err: timeout})},
|
||||||
|
{context.Background(), errors.Join(resourceFailure(context.Background(), timeout), errors.New("unknown collector failure"))},
|
||||||
|
{ctx, resourceFailure(context.Background(), timeout)},
|
||||||
|
} {
|
||||||
|
err := managedResourceHealthError(tc.ctx, &managedResourceFailure{"avatar", "yt3.ggpht.com", tc.err})
|
||||||
|
var failure *managedResourceFailure
|
||||||
|
if !errors.As(err, &failure) || failure.Err == nil {
|
||||||
|
t.Fatalf("safety failure normalized: %v", err)
|
||||||
|
}
|
||||||
|
if tc.ctx.Err() != nil && !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatal("parent cancellation lost")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExperimentalProductionRecheckAdapterInstalled(t *testing.T) {
|
||||||
|
d, _, _ := recoveryFixture(t, true)
|
||||||
|
target, env, _ := managedCLIFixture(t)
|
||||||
|
env["ZAPRET_AUTOTUNE_YOUTUBE_SERVICE"] = "1"
|
||||||
|
adapted, err := managedMonitorRecoveryAdapters(target, d.db, d.scan, d.check, d.run)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !adapted.service || adapted.resources == nil || adapted.recheckExperimental == nil || adapted.run == nil || adapted.transaction == nil {
|
||||||
|
t.Fatal("production recheck dependency missing")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,205 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
var errManagedExperimentalUnqualified = errors.New("experimental googlevideo suffix: operational health is not family qualification")
|
||||||
|
|
||||||
|
// URLs are trusted Player inventory, never rewritten or persisted.
|
||||||
|
func freshExperimentalMedia(r mediaResource) bool {
|
||||||
|
u, err := privateResourceURL(r.url, r.host)
|
||||||
|
if err != nil || !strings.HasSuffix(r.host, ".googlevideo.com") || u.Path != "/videoplayback" || (r.kind != "video/mp4" && r.kind != "audio/mp4") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
q := u.Query()
|
||||||
|
expires, err := strconv.ParseInt(q.Get("expire"), 10, 64)
|
||||||
|
return err == nil && expires > time.Now().Unix()+120 && (q.Get("sig") != "" || q.Get("signature") != "" || q.Get("lsig") != "")
|
||||||
|
}
|
||||||
|
|
||||||
|
func experimentalMedia(ctx context.Context, d managedRecoveryDeps) (mediaResource, error) {
|
||||||
|
i, err := d.resources(ctx)
|
||||||
|
var fatal *managedRecoveryFatal
|
||||||
|
var failure *managedResourceFailure
|
||||||
|
// An unrelated metadata/image failure cannot conceal usable media, but a
|
||||||
|
// mixed safety/infrastructure error must not be turned into transfer absence.
|
||||||
|
unrelated := errors.As(err, &failure) && assetEndpoint(failure.Host) && (err == failure && failure.Err == nil || managedGooglevideoTransferNonpass(err))
|
||||||
|
if ctx.Err() != nil || errors.As(err, &fatal) && !unrelated || errors.Is(err, context.Canceled) || err != nil && !unrelated {
|
||||||
|
return mediaResource{}, managedRecoveryContextError(ctx, err)
|
||||||
|
}
|
||||||
|
if !freshExperimentalMedia(i.media) {
|
||||||
|
return mediaResource{}, &managedResourceFailure{"media", "unknown", nil}
|
||||||
|
}
|
||||||
|
return i.media, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func recoverManagedExperimental(ctx context.Context, d managedRecoveryDeps) (int64, error) {
|
||||||
|
if !d.service || d.resources == nil || d.recheckExperimental == nil {
|
||||||
|
return 0, errManagedGGCUnresolved
|
||||||
|
}
|
||||||
|
ids, err := d.transaction.ActiveCandidates(d.db)
|
||||||
|
if err != nil {
|
||||||
|
return 0, &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
j, err := d.transaction.load()
|
||||||
|
if err != nil {
|
||||||
|
return 0, &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
for _, id := range ids {
|
||||||
|
c, err := readStrategyCandidate(d.db, id)
|
||||||
|
if err != nil {
|
||||||
|
return 0, &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
if experimentalGooglevideo(j.Scope, c) {
|
||||||
|
r, err := experimentalMedia(ctx, d)
|
||||||
|
if err == nil {
|
||||||
|
err = d.recheckExperimental(ctx, c, r)
|
||||||
|
}
|
||||||
|
return id, managedRecoveryContextError(ctx, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0, &managedRecoveryFatal{Err: errors.New("owned experimental candidate missing")}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newManagedExperimentalRecheck(target managedDeps, d managedRecoveryDeps) func(context.Context, storedCandidate, mediaResource) error {
|
||||||
|
runtime := newManagedAssetRuntime(target)
|
||||||
|
guard := func() error {
|
||||||
|
if err := runtime.guard(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
lua, err := os.ReadFile("/opt/zapret2/lua/zapret-auto.lua")
|
||||||
|
if err != nil || fmt.Sprintf("%x", sha256.Sum256(lua)) != managedGooglevideoLuaHash {
|
||||||
|
return errors.New("unverified native automation Lua source")
|
||||||
|
}
|
||||||
|
_, err = d.transaction.ActiveCandidates(d.db)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
restart := func(ctx context.Context) (err error) {
|
||||||
|
// Existing daemon protocol stops/joins before starting the same full set.
|
||||||
|
// Never activate a pending or drifted deployment after a partial stop.
|
||||||
|
defer func() {
|
||||||
|
if err != nil {
|
||||||
|
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if e := d.transaction.Reconcile(d.db); e != nil {
|
||||||
|
err = errors.Join(err, e)
|
||||||
|
} else {
|
||||||
|
err = errors.Join(err, d.run(cleanup, managedZapretScript, "start"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
if err = d.run(ctx, managedZapretScript, "stop"); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err = d.transaction.Reconcile(d.db); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return d.run(ctx, managedZapretScript, "start")
|
||||||
|
}
|
||||||
|
return func(ctx context.Context, c storedCandidate, resource mediaResource) error {
|
||||||
|
if !experimentalGooglevideo(GooglevideoScopeExperimental, c) {
|
||||||
|
return &managedRecoveryFatal{Err: errors.New("unaudited experimental candidate")}
|
||||||
|
}
|
||||||
|
ownedGuard := func() error {
|
||||||
|
if err := guard(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
j, err := d.transaction.load()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if j.Scope == GooglevideoScopeExperimental {
|
||||||
|
for _, id := range j.Candidates {
|
||||||
|
if id == c.ID {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return errors.New("experimental ownership changed")
|
||||||
|
}
|
||||||
|
return recheckDeployedExperimental(ctx, resource, ownedGuard, func(ctx context.Context) error {
|
||||||
|
if err := ownedGuard(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return restart(ctx)
|
||||||
|
}, func(ctx context.Context) (mediaResource, error) {
|
||||||
|
return experimentalMedia(ctx, d)
|
||||||
|
}, func(ctx context.Context, r mediaResource, protocol string) error {
|
||||||
|
_, _, err := collectMediaTransfer(ctx, runtime.proxy, r, protocol, runtime.curl)
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// No temporary NFQ worker: successful learning stays in the deployed daemon RAM.
|
||||||
|
func recheckDeployedExperimental(ctx context.Context, resource mediaResource, guard func() error, restart func(context.Context) error, fresh func(context.Context) (mediaResource, error), transfer func(context.Context, mediaResource, string) error) error {
|
||||||
|
check := func(ctx context.Context, protocol string) error {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := guard(); err != nil {
|
||||||
|
return &managedRecoveryFatal{Err: err}
|
||||||
|
}
|
||||||
|
if !freshExperimentalMedia(resource) {
|
||||||
|
var err error
|
||||||
|
resource, err = fresh(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !freshExperimentalMedia(resource) {
|
||||||
|
return &managedResourceFailure{"media", "unknown", nil}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return managedRecoveryContextError(ctx, transfer(ctx, resource, protocol))
|
||||||
|
}
|
||||||
|
sample := func() error {
|
||||||
|
for _, protocol := range []string{"tls12", "tls13"} {
|
||||||
|
for n := 0; n < 3; n++ {
|
||||||
|
if err := check(ctx, protocol); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
err := sample()
|
||||||
|
if err == nil || !managedGooglevideoTransferNonpass(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := restart(ctx); err != nil {
|
||||||
|
return &managedRecoveryFatal{Err: managedRecoveryContextError(ctx, err)}
|
||||||
|
}
|
||||||
|
warm, cancel := context.WithTimeout(ctx, 120*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
learned := map[string]bool{}
|
||||||
|
for n := 0; n < 10 && len(learned) < 2; n++ {
|
||||||
|
protocol := []string{"tls12", "tls13"}[n%2]
|
||||||
|
err = check(warm, protocol)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return managedRecoveryContextError(ctx, err)
|
||||||
|
}
|
||||||
|
if warm.Err() != nil && managedRecoveryTimeoutSafe(err) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil && !managedGooglevideoTransferNonpass(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
learned[protocol] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(learned) == 2 {
|
||||||
|
err = sample()
|
||||||
|
if err == nil || !managedGooglevideoTransferNonpass(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fmt.Errorf("experimental media relearning exhausted: %w", errNoQualifiedStrategy)
|
||||||
|
}
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func experimentalResourceFixture() mediaResource {
|
||||||
|
return mediaResource{url: fmt.Sprintf("https://rr2---sn-new.googlevideo.com/videoplayback?expire=%d&sig=private", time.Now().Add(time.Hour).Unix()), host: "rr2---sn-new.googlevideo.com", kind: "video/mp4"}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExperimentalDeployedRecheck(t *testing.T) {
|
||||||
|
for _, mode := range []string{"healthy", "learn", "exhausted", "mixed", "cancel", "guard"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
r := experimentalResourceFixture()
|
||||||
|
calls, restarts := 0, 0
|
||||||
|
protocols := []string{}
|
||||||
|
bad := errors.New("invalid media range or container")
|
||||||
|
infra := errors.New("collector unavailable")
|
||||||
|
err := recheckDeployedExperimental(ctx, r, func() error {
|
||||||
|
if mode == "guard" {
|
||||||
|
return infra
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}, func(context.Context) error { restarts++; return nil }, func(context.Context) (mediaResource, error) {
|
||||||
|
t.Fatal("unexpected refresh")
|
||||||
|
return mediaResource{}, infra
|
||||||
|
}, func(_ context.Context, got mediaResource, protocol string) error {
|
||||||
|
if got != r {
|
||||||
|
t.Fatal("signed resource rewritten")
|
||||||
|
}
|
||||||
|
calls++
|
||||||
|
protocols = append(protocols, protocol)
|
||||||
|
switch mode {
|
||||||
|
case "learn":
|
||||||
|
if calls == 1 {
|
||||||
|
return resourceFailure(ctx, bad)
|
||||||
|
}
|
||||||
|
case "exhausted":
|
||||||
|
return resourceFailure(ctx, bad)
|
||||||
|
case "mixed":
|
||||||
|
return errors.Join(resourceFailure(ctx, bad), infra)
|
||||||
|
case "cancel":
|
||||||
|
cancel()
|
||||||
|
return context.Canceled
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
switch mode {
|
||||||
|
case "healthy":
|
||||||
|
if err != nil || restarts != 0 || !reflect.DeepEqual(protocols, []string{"tls12", "tls12", "tls12", "tls13", "tls13", "tls13"}) {
|
||||||
|
t.Fatalf("%v %d %v", err, restarts, protocols)
|
||||||
|
}
|
||||||
|
case "learn":
|
||||||
|
if err != nil || restarts != 1 || calls != 9 {
|
||||||
|
t.Fatalf("%v restarts=%d calls=%d", err, restarts, calls)
|
||||||
|
}
|
||||||
|
case "exhausted":
|
||||||
|
if !errors.Is(err, errNoQualifiedStrategy) || restarts != 1 || calls != 11 {
|
||||||
|
t.Fatalf("%v restarts=%d calls=%d", err, restarts, calls)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
if err == nil || errors.Is(err, errNoQualifiedStrategy) || restarts != 0 {
|
||||||
|
t.Fatalf("unsafe classification: %v restarts=%d", err, restarts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExperimentalServiceRecoveryPreservesOwnership(t *testing.T) {
|
||||||
|
d, front, _ := recoveryFixture(t, true)
|
||||||
|
target, _, _ := managedCLIFixture(t)
|
||||||
|
initManagedCLI(t, target)
|
||||||
|
d.transaction.Dir = target.directory
|
||||||
|
asset := unitAssetCandidate(t, d.db, "i.ytimg.com", "--lua-desync=fake", true)
|
||||||
|
id := managedCandidateFixture(t, d.db, "googlevideo", "rr1---sn-alpha.googlevideo.com", managedGooglevideoArgs, false)
|
||||||
|
ids := []int64{front, asset, id}
|
||||||
|
if err := d.transaction.Apply(d.db, ids, GooglevideoScopeExperimental); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
before, err := d.transaction.load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
d.service = true
|
||||||
|
d.frontHealth = func(context.Context) (string, error) { return "OK", nil }
|
||||||
|
d.health = func(context.Context) (string, error) { return "OK", nil }
|
||||||
|
d.resources = func(context.Context) (managedServiceInventory, error) {
|
||||||
|
return managedServiceInventory{media: experimentalResourceFixture()}, &managedResourceFailure{"avatar", "yt3.ggpht.com", nil}
|
||||||
|
}
|
||||||
|
calls := 0
|
||||||
|
d.recheckExperimental = func(_ context.Context, c storedCandidate, r mediaResource) error {
|
||||||
|
calls++
|
||||||
|
if c.ID != id || !freshExperimentalMedia(r) {
|
||||||
|
t.Fatal("wrong committed candidate/resource")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for _, adaptive := range []bool{false, true} {
|
||||||
|
d.adaptive = adaptive
|
||||||
|
result, err := recoverManagedTarget(context.Background(), d)
|
||||||
|
if err != nil || result.GooglevideoID != id || result.GGCError != nil {
|
||||||
|
t.Fatalf("%+v %v", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
after, err := d.transaction.load()
|
||||||
|
if err != nil || !reflect.DeepEqual(before, after) || calls != 2 {
|
||||||
|
t.Fatalf("journal changed: %v calls=%d", err, calls)
|
||||||
|
}
|
||||||
|
c, err := readStrategyCandidate(d.db, id)
|
||||||
|
if err != nil || c.Qualified {
|
||||||
|
t.Fatalf("qualification forged: %+v %v", c, err)
|
||||||
|
}
|
||||||
|
monitor := managedMonitorDeps{db: d.db, target: target}
|
||||||
|
reports := 0
|
||||||
|
if err := wireManagedRecovery(&monitor, d, func(err error) {
|
||||||
|
if errors.Is(err, errManagedExperimentalUnqualified) {
|
||||||
|
reports++
|
||||||
|
}
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := monitor.probe(context.Background()); err != nil || reports != 1 {
|
||||||
|
t.Fatalf("operational probe: %v reports=%d", err, reports)
|
||||||
|
}
|
||||||
|
assertActiveCandidates(t, d.transaction, d.db, ids)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExperimentalRefreshAndMixedFatalPrecedence(t *testing.T) {
|
||||||
|
bad := errors.New("invalid media range or container")
|
||||||
|
for _, err := range []error{
|
||||||
|
&managedRecoveryFatal{Err: bad},
|
||||||
|
fmt.Errorf("wrapped: %w", &managedRecoveryFatal{Err: bad}),
|
||||||
|
errors.Join(resourceFailure(context.Background(), bad), &managedRecoveryFatal{Err: bad}),
|
||||||
|
errors.Join(resourceFailure(context.Background(), bad), context.Canceled),
|
||||||
|
} {
|
||||||
|
if managedGooglevideoTransferNonpass(err) {
|
||||||
|
t.Fatalf("fatal accepted: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
refreshes, transfers := 0, 0
|
||||||
|
err := recheckDeployedExperimental(context.Background(), mediaResource{}, func() error { return nil }, func(context.Context) error {
|
||||||
|
t.Fatal("unknown media caused restart")
|
||||||
|
return nil
|
||||||
|
}, func(context.Context) (mediaResource, error) {
|
||||||
|
refreshes++
|
||||||
|
return experimentalResourceFixture(), nil
|
||||||
|
}, func(context.Context, mediaResource, string) error { transfers++; return nil })
|
||||||
|
if err != nil || refreshes != 1 || transfers != 6 {
|
||||||
|
t.Fatalf("refresh: %v %d %d", err, refreshes, transfers)
|
||||||
|
}
|
||||||
|
err = recheckDeployedExperimental(context.Background(), mediaResource{}, func() error { return nil }, func(context.Context) error {
|
||||||
|
t.Fatal("unknown media caused restart")
|
||||||
|
return nil
|
||||||
|
}, func(context.Context) (mediaResource, error) {
|
||||||
|
return mediaResource{}, &managedResourceFailure{"media", "unknown", nil}
|
||||||
|
}, func(context.Context, mediaResource, string) error { t.Fatal("unknown media transferred"); return nil })
|
||||||
|
if err == nil || errors.Is(err, errNoQualifiedStrategy) {
|
||||||
|
t.Fatalf("unknown treated as absence: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -78,16 +78,15 @@ func managedAdaptiveAssetsQualified(d managedRecoveryDeps, ids []int64) (bool, e
|
|||||||
}
|
}
|
||||||
|
|
||||||
func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) {
|
func recoverManagedGooglevideo(ctx context.Context, d managedRecoveryDeps) (int64, error) {
|
||||||
if !d.service || d.mediaSamples == nil || d.verifyGooglevideo == nil {
|
|
||||||
return 0, &InsufficientAdaptiveSamples{}
|
|
||||||
}
|
|
||||||
m := *d.transaction
|
m := *d.transaction
|
||||||
journal, err := m.load()
|
journal, err := m.load()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, &managedRecoveryFatal{Err: err}
|
return 0, &managedRecoveryFatal{Err: err}
|
||||||
}
|
}
|
||||||
if journal.Scope == GooglevideoScopeExperimental {
|
if journal.Scope == GooglevideoScopeExperimental {
|
||||||
// Explicit operator policy remains owned, never auto-qualified/replaced.
|
return recoverManagedExperimental(ctx, d)
|
||||||
|
}
|
||||||
|
if !d.service || d.mediaSamples == nil || d.verifyGooglevideo == nil {
|
||||||
return 0, &InsufficientAdaptiveSamples{}
|
return 0, &InsufficientAdaptiveSamples{}
|
||||||
}
|
}
|
||||||
current, err := m.ActiveCandidates(d.db)
|
current, err := m.ActiveCandidates(d.db)
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
|
"os/exec"
|
||||||
"reflect"
|
"reflect"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -185,6 +186,43 @@ func (r managedAssetRuntime) verifyGooglevideo(ctx context.Context, c storedCand
|
|||||||
}
|
}
|
||||||
|
|
||||||
func managedGooglevideoTransferNonpass(err error) bool {
|
func managedGooglevideoTransferNonpass(err error) bool {
|
||||||
|
return managedResourceTransferNonpass(err, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func managedResourceTransferNonpass(err error, metadata bool) bool {
|
||||||
|
// Collector normalization wraps known transfer failures as fatal for generic
|
||||||
|
// consumers. Only that boundary may unwrap the wrapper; guard/cleanup failures
|
||||||
|
// and mixed error sets must never authorize absence or a daemon restart.
|
||||||
|
if e, ok := err.(*resourceError); ok {
|
||||||
|
cause := e.cause
|
||||||
|
if fatal, ok := cause.(*managedRecoveryFatal); ok {
|
||||||
|
cause = fatal.Err
|
||||||
|
}
|
||||||
|
return managedResourceTransferNonpass(cause, metadata)
|
||||||
|
}
|
||||||
|
if _, ok := err.(*managedRecoveryFatal); ok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if joined, ok := err.(interface{ Unwrap() []error }); ok {
|
||||||
|
children := joined.Unwrap()
|
||||||
|
if len(children) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, child := range children {
|
||||||
|
if !managedResourceTransferNonpass(child, metadata) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if wrapped, ok := err.(interface{ Unwrap() error }); ok {
|
||||||
|
return managedResourceTransferNonpass(wrapped.Unwrap(), metadata)
|
||||||
|
}
|
||||||
|
// Metadata curl uses --fail. A non-2xx response is missing inventory,
|
||||||
|
// never media rejection or candidate-absence evidence.
|
||||||
|
if exit, ok := err.(*exec.ExitError); metadata && ok && exit.ExitCode() == 22 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
if managedAssetTransferNonpass(err) {
|
if managedAssetTransferNonpass(err) {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
@@ -194,9 +232,6 @@ func managedGooglevideoTransferNonpass(err error) bool {
|
|||||||
if err.Error() == "invalid media range or container" {
|
if err.Error() == "invalid media range or container" {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
if e, ok := err.(interface{ Unwrap() error }); ok {
|
|
||||||
return managedGooglevideoTransferNonpass(e.Unwrap())
|
|
||||||
}
|
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -191,6 +191,7 @@ func managedMonitorRecoveryAdapters(target managedDeps, db *sql.DB, scanCommand
|
|||||||
d, err := managedRecoveryDependencies(target, db, scan, check, run)
|
d, err := managedRecoveryDependencies(target, db, scan, check, run)
|
||||||
if err == nil && d.service {
|
if err == nil && d.service {
|
||||||
d.verifyAssets = newManagedAssetVerifier(target)
|
d.verifyAssets = newManagedAssetVerifier(target)
|
||||||
|
d.recheckExperimental = newManagedExperimentalRecheck(target, d)
|
||||||
if d.adaptive {
|
if d.adaptive {
|
||||||
d.verifyGooglevideo = newManagedGooglevideoVerifier(target)
|
d.verifyGooglevideo = newManagedGooglevideoVerifier(target)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -488,7 +488,7 @@ func TestManagedMonitorProductionAbsenceDiagnostics(t *testing.T) {
|
|||||||
var fatal *managedRecoveryFatal
|
var fatal *managedRecoveryFatal
|
||||||
wantFatal := strings.Contains(mode, "nfq")
|
wantFatal := strings.Contains(mode, "nfq")
|
||||||
if errors.As(err, &fatal) != wantFatal || (!wantFatal && !errors.Is(err, context.Canceled)) {
|
if errors.As(err, &fatal) != wantFatal || (!wantFatal && !errors.Is(err, context.Canceled)) {
|
||||||
t.Fatalf("mode=%s err=%v", mode, err)
|
t.Fatalf("mode=%s err=%v reported=%v", mode, err, reported)
|
||||||
}
|
}
|
||||||
if strings.Contains(errors.Join(err, reported).Error(), "secret-never-print") {
|
if strings.Contains(errors.Join(err, reported).Error(), "secret-never-print") {
|
||||||
t.Fatal("diagnostics leaked")
|
t.Fatal("diagnostics leaked")
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ func TestManagedSingleGroupCLI(t *testing.T) {
|
|||||||
t.Fatalf("apply: %d %s %s", code, out, diagnostic)
|
t.Fatalf("apply: %d %s %s", code, out, diagnostic)
|
||||||
}
|
}
|
||||||
code, out, diagnostic := callManaged(t, d, "status")
|
code, out, diagnostic := callManaged(t, d, "status")
|
||||||
if code != 0 || !strings.Contains(out, "groups="+group+" ") || !strings.Contains(out, "unresolved_groups="+missing+"\n") || !strings.Contains(out, "Live health: NOT CHECKED") {
|
if code != 0 || !strings.Contains(out, "groups="+group+" ") || !strings.Contains(out, "unresolved_groups="+missing+" covered="+group+"\n") || !strings.Contains(out, "Live health: NOT CHECKED") {
|
||||||
t.Fatalf("partial status: %d %s %s", code, out, diagnostic)
|
t.Fatalf("partial status: %d %s %s", code, out, diagnostic)
|
||||||
}
|
}
|
||||||
code, out, diagnostic = callManaged(t, d, "active-ids")
|
code, out, diagnostic = callManaged(t, d, "active-ids")
|
||||||
|
|||||||
@@ -35,10 +35,11 @@ type managedRecoveryDeps struct {
|
|||||||
resources func(context.Context) (managedServiceInventory, error)
|
resources func(context.Context) (managedServiceInventory, error)
|
||||||
// Verify owns temporary raw-config activation and must join/restore before
|
// Verify owns temporary raw-config activation and must join/restore before
|
||||||
// returning. It must not touch permanent config or the transaction journal.
|
// returning. It must not touch permanent config or the transaction journal.
|
||||||
verifyAssets func(context.Context, storedCandidate, managedServiceInventory) ([]assetMeasurement, error)
|
verifyAssets func(context.Context, storedCandidate, managedServiceInventory) ([]assetMeasurement, error)
|
||||||
adaptive bool
|
adaptive bool
|
||||||
mediaSamples func(context.Context) ([]mediaResource, error)
|
mediaSamples func(context.Context) ([]mediaResource, error)
|
||||||
verifyGooglevideo func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error)
|
verifyGooglevideo func(context.Context, storedCandidate, []mediaResource) (GooglevideoQualificationProof, error)
|
||||||
|
recheckExperimental func(context.Context, storedCandidate, mediaResource) error
|
||||||
}
|
}
|
||||||
|
|
||||||
// Target-local adapter: reuses bounded Player requests without invoking a CLI.
|
// Target-local adapter: reuses bounded Player requests without invoking a CLI.
|
||||||
@@ -267,11 +268,18 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
|
|||||||
if d.service {
|
if d.service {
|
||||||
_, healthErr := d.health(ctx)
|
_, healthErr := d.health(ctx)
|
||||||
var fatal *managedRecoveryFatal
|
var fatal *managedRecoveryFatal
|
||||||
|
var failure *managedResourceFailure
|
||||||
|
j, e := m.load()
|
||||||
|
if e != nil {
|
||||||
|
return result, &managedRecoveryFatal{Err: e}
|
||||||
|
}
|
||||||
|
if j.Scope == GooglevideoScopeExperimental {
|
||||||
|
healthErr = managedResourceHealthError(ctx, healthErr)
|
||||||
|
}
|
||||||
if ctx.Err() != nil || errors.As(healthErr, &fatal) || errors.Is(healthErr, context.Canceled) || errors.Is(healthErr, context.DeadlineExceeded) {
|
if ctx.Err() != nil || errors.As(healthErr, &fatal) || errors.Is(healthErr, context.Canceled) || errors.Is(healthErr, context.DeadlineExceeded) {
|
||||||
return result, managedRecoveryContextError(ctx, healthErr)
|
return result, managedRecoveryContextError(ctx, healthErr)
|
||||||
}
|
}
|
||||||
var failure *managedResourceFailure
|
if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) && j.Scope != GooglevideoScopeExperimental {
|
||||||
if errors.As(healthErr, &failure) && assetEndpoint(failure.Host) {
|
|
||||||
return result, repairManagedAssets(ctx, d, failure.Host)
|
return result, repairManagedAssets(ctx, d, failure.Host)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -440,9 +448,7 @@ func recoverManagedTarget(ctx context.Context, d managedRecoveryDeps) (result ma
|
|||||||
return result, e
|
return result, e
|
||||||
}
|
}
|
||||||
if experimental {
|
if experimental {
|
||||||
// Operator extrapolation survives recovery in either configured mode.
|
result.GooglevideoID, result.GGCError = recoverManagedExperimental(ctx, d)
|
||||||
// Frontend/asset repairs remain available, but cannot qualify this policy.
|
|
||||||
result.GGCError = errManagedGGCUnresolved
|
|
||||||
return result, result.GGCError
|
return result, result.GGCError
|
||||||
}
|
}
|
||||||
if !d.service && (len(current) != 1 || current[0] != result.FrontendID) {
|
if !d.service && (len(current) != 1 || current[0] != result.FrontendID) {
|
||||||
@@ -716,14 +722,6 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
|
|||||||
if err := guard(); err != nil {
|
if err := guard(); err != nil {
|
||||||
return &managedRecoveryFatal{Err: err}
|
return &managedRecoveryFatal{Err: err}
|
||||||
}
|
}
|
||||||
status, err := recovery.health(ctx)
|
|
||||||
err = managedRecoveryContextError(ctx, err)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if status != "OK" {
|
|
||||||
return &managedHealthRejection{Status: status}
|
|
||||||
}
|
|
||||||
transaction := *recovery.transaction
|
transaction := *recovery.transaction
|
||||||
transaction.Health = func() (string, error) { return recovery.health(ctx) }
|
transaction.Health = func() (string, error) { return recovery.health(ctx) }
|
||||||
// ActiveCandidates is read-only; never permit a probe to reload config.
|
// ActiveCandidates is read-only; never permit a probe to reload config.
|
||||||
@@ -737,8 +735,25 @@ func wireManagedRecovery(monitor *managedMonitorDeps, recovery managedRecoveryDe
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return &managedRecoveryFatal{Err: err}
|
return &managedRecoveryFatal{Err: err}
|
||||||
}
|
}
|
||||||
|
status, err := recovery.health(ctx)
|
||||||
|
if journal.Scope == GooglevideoScopeExperimental && recovery.service && recovery.recheckExperimental != nil {
|
||||||
|
err = managedResourceHealthError(ctx, err)
|
||||||
|
} else {
|
||||||
|
err = managedRecoveryContextError(ctx, err)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if status != "OK" {
|
||||||
|
return &managedHealthRejection{Status: status}
|
||||||
|
}
|
||||||
if journal.Scope == GooglevideoScopeExperimental {
|
if journal.Scope == GooglevideoScopeExperimental {
|
||||||
return errManagedGGCUnresolved
|
if !recovery.service || recovery.recheckExperimental == nil {
|
||||||
|
return errManagedGGCUnresolved
|
||||||
|
}
|
||||||
|
// Operational service health is not family qualification.
|
||||||
|
reportGGC(errManagedExperimentalUnqualified)
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
// Two bounded Player requests reuse the existing health/discovery adapters.
|
// Two bounded Player requests reuse the existing health/discovery adapters.
|
||||||
host, err := recovery.discover(ctx)
|
host, err := recovery.discover(ctx)
|
||||||
|
|||||||
Reference in new issue
Block a user