feat: add guarded periodic resource monitoring
Add checks-only scheduling, read-only status, private authenticated persistence and guarded PID1 lifecycle. Keep strategy selection/recovery and remote fault acceptance out of scope. Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
cfce8170d7
commit
4efc066625
16 files changed
+2302
-98
No files matched your search
@@ -0,0 +1,66 @@
|
|||||||
|
# Periodic configured resource checks
|
||||||
|
|
||||||
|
`managed monitor --resource-config FILE` runs resource-config v1 periodically.
|
||||||
|
PID 1 selects the same mode when `ZAPRET_AUTOTUNE_RESOURCE_CONFIG` is nonempty;
|
||||||
|
unset preserves the existing YouTube monitor. The modes share one singleton
|
||||||
|
lock. Supply an absolute container-visible config path (for example mount a
|
||||||
|
read-only operator config directory using a Compose override). Configuration
|
||||||
|
must be a regular root/runtime-owned, non-writable-by-others file without
|
||||||
|
symlink/hardlink aliases. Environment references are resolved on every reload.
|
||||||
|
|
||||||
|
Each resource accepts `interval_ms` (default 300000) and `cooldown_ms` (default
|
||||||
|
30000), both integers in [100, 86400000]. Checks are sequential round robin
|
||||||
|
with independent due times; `tls: both` requires both passes. Ordinary DNS,
|
||||||
|
TLS, timeout or expectation failures mark only that resource DEGRADED and
|
||||||
|
retry after its cooldown. There is **no strategy selection, recovery or apply**,
|
||||||
|
and no no-strategy two-hour pause. HEALTHY means configured checks passed,
|
||||||
|
not playback health, generic recovery or whole-production READY: HEALTHY is
|
||||||
|
**CHECKS_ONLY**, never READY. The scheduling, status, persistence and lifecycle
|
||||||
|
base is unit/fixture-verified; isolated remote fault acceptance remains pending.
|
||||||
|
|
||||||
|
Invalid resource JSON initially or after reload is BLOCKED, with no DNS or
|
||||||
|
transfers until repaired. Bad static target role, proxy or DNS settings still
|
||||||
|
fail startup validation; resource JSON is deliberately not part of
|
||||||
|
`target-config-check`. Ownership and fresh source snapshots are checked before
|
||||||
|
DNS and again before transfer; existing DNS/IP policy, pinned SOCKS transfers,
|
||||||
|
TLS verification and private stdin are unchanged. No cached addresses bypass
|
||||||
|
DNS policy. Existing zapret reconciliation remains mandatory and generic
|
||||||
|
startup never consumes experimental apply requests.
|
||||||
|
|
||||||
|
State is independent of recovery SQL leases, fixed in the private target
|
||||||
|
directory as `.resource-monitor-state.json` and `.resource-monitor-key`.
|
||||||
|
Both are private 0600 files. Fingerprints HMAC the resolved configuration
|
||||||
|
(including secrets); state is also authenticated and never contains URLs,
|
||||||
|
headers, addresses or raw errors. Missing state initializes UNKNOWN. A missing
|
||||||
|
key with existing state, corrupt or unsafe file is preserved and BLOCKED,
|
||||||
|
never silently replaced. Restore the original key and a valid version-1 signed
|
||||||
|
state from a trusted backup to repair it. The controller holds directory, key
|
||||||
|
and state inode anchors: an identical signed replacement is still BLOCKED and
|
||||||
|
preserved. Repair on the original held inode is reloaded only after version/MAC
|
||||||
|
validation; replacing an inode requires deliberate operator repair/restart.
|
||||||
|
A persistence failure pauses probes;
|
||||||
|
deliberate repair requires a different valid authenticated snapshot, not just
|
||||||
|
retrying a failed write. Never chmod/adopt unknown files to bypass validation.
|
||||||
|
This is not isolation from a hostile same-UID actor: that actor can read the
|
||||||
|
HMAC key and race filesystem publication; root/container administrators can
|
||||||
|
also read private data. HMAC fingerprints protect stored secret values from
|
||||||
|
unkeyed guessing, not from an actor with access to the key.
|
||||||
|
|
||||||
|
`managed resource-status` is read-only and reports BLOCKED or STALE explicitly
|
||||||
|
(a check overdue by >65s, unknown state age >65s, or changed configuration).
|
||||||
|
Idle reloads do not write a disk heartbeat: only configured state changes,
|
||||||
|
BLOCKED transitions and completed checks are flushed. Cancelled checks join
|
||||||
|
their children but do not change prior health, counts or completion/due times.
|
||||||
|
It reads the config path from
|
||||||
|
`ZAPRET_AUTOTUNE_RESOURCE_CONFIG`; use that environment setting also for manual
|
||||||
|
monitor/status invocations. Status rows report only configured-check health.
|
||||||
|
|
||||||
|
TERM cancels/joins transfers before finalizing a completed snapshot and exits
|
||||||
|
0 on safe cleanup. PID 1 retains DNS through monitor/SS cleanup. In generic
|
||||||
|
mode transient child restart-budget exhaustion pauses until the rolling window
|
||||||
|
opens, rather than terminating the container. Unknown/unsafe cleanup stays
|
||||||
|
BLOCKED pending operator repair/restart; no unsafe daemon sourcing is allowed.
|
||||||
|
Generic cleanup shares an 80-second budget, including watchdog setup and
|
||||||
|
command kill escalation, with bounded final reaping; it is designed to fit
|
||||||
|
within a 90-second Docker stop limit (the supplied Compose grants 240 seconds).
|
||||||
|
These changes do not claim to fix legacy YouTube exit 78.
|
||||||
@@ -21,6 +21,10 @@ Commands:
|
|||||||
health youtube One-shot Player API health through configured loopback SOCKS
|
health youtube One-shot Player API health through configured loopback SOCKS
|
||||||
managed init|reconcile|status|apply|monitor
|
managed init|reconcile|status|apply|monitor
|
||||||
Guarded activation/monitoring in the dedicated managed-test target only
|
Guarded activation/monitoring in the dedicated managed-test target only
|
||||||
|
managed monitor --resource-config FILE
|
||||||
|
Periodic configured checks only (no recovery/apply)
|
||||||
|
managed resource-status
|
||||||
|
Read-only configured-check snapshot (explicit stale/blocked)
|
||||||
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
|
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
|
||||||
check --config FILE
|
check --config FILE
|
||||||
Check configured HTTPS resources (checks only; no strategy selection)
|
Check configured HTTPS resources (checks only; no strategy selection)
|
||||||
@@ -48,7 +52,18 @@ func runWithPlayer(args []string, stdout, stderr io.Writer, player playerRunner)
|
|||||||
return dnsForwarderCLI(args[1:], stdout, stderr)
|
return dnsForwarderCLI(args[1:], stdout, stderr)
|
||||||
}
|
}
|
||||||
if len(args) == 1 && args[0] == "target-config-check" {
|
if len(args) == 1 && args[0] == "target-config-check" {
|
||||||
if _, err := parseManagedHealthConfig(os.LookupEnv); err != nil {
|
path, _ := os.LookupEnv("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
|
||||||
|
if path != "" {
|
||||||
|
port, err := managedTargetSettings(os.LookupEnv)
|
||||||
|
if err != nil || port != 1080 {
|
||||||
|
fmt.Fprintln(stderr, "configerror: invalid generic target settings")
|
||||||
|
return 64
|
||||||
|
}
|
||||||
|
if _, _, err := resourceStaticConfig(os.LookupEnv); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "configerror: invalid generic DNS/proxy configuration")
|
||||||
|
return 64
|
||||||
|
}
|
||||||
|
} else if _, err := parseManagedHealthConfig(os.LookupEnv); err != nil {
|
||||||
fmt.Fprintln(stderr, "configerror: invalid target health/service configuration")
|
fmt.Fprintln(stderr, "configerror: invalid target health/service configuration")
|
||||||
return 64
|
return 64
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -379,25 +379,9 @@ func managedReport(m *managedTransaction, out io.Writer, openDB func() (*sql.DB,
|
|||||||
}
|
}
|
||||||
|
|
||||||
func managedRuntimeGuard(d managedDeps) (int, error) {
|
func managedRuntimeGuard(d managedDeps) (int, error) {
|
||||||
role, _ := d.lookup("ZAPRET_AUTOTUNE_TARGET_ROLE")
|
port, err := managedTargetSettings(d.lookup)
|
||||||
if role != "managed-test" {
|
if err != nil {
|
||||||
return 2, fmt.Errorf("ZAPRET_AUTOTUNE_TARGET_ROLE must be managed-test")
|
return 2, err
|
||||||
}
|
|
||||||
for key, expected := range map[string]string{
|
|
||||||
"ZAPRET_AUTOTUNE_TARGET_ID": managedTargetID,
|
|
||||||
"ZAPRET_AUTOTUNE_TARGET_DIR": managedTargetDirectory,
|
|
||||||
"ZAPRET_CONFIG": managedTargetConfig,
|
|
||||||
} {
|
|
||||||
if value, _ := d.lookup(key); value != expected {
|
|
||||||
return 2, fmt.Errorf("%s does not match the fresh managed target contract", key)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
portText, _ := d.lookup("SOCKS_PORT")
|
|
||||||
port, portErr := strconv.Atoi(portText)
|
|
||||||
proxy, _ := d.lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
|
|
||||||
proxy, proxyErr := managedProbeProxy(proxy)
|
|
||||||
if portErr != nil || port < 1 || port > 65535 || strconv.Itoa(port) != portText || proxyErr != nil || proxy != "socks5h://127.0.0.1:"+portText {
|
|
||||||
return 2, fmt.Errorf("managed SOCKS proxy must match the target SOCKS_PORT listener")
|
|
||||||
}
|
}
|
||||||
if d.verifyRuntime == nil || d.verifySOCKS == nil {
|
if d.verifyRuntime == nil || d.verifySOCKS == nil {
|
||||||
return 1, fmt.Errorf("managed runtime validators unavailable")
|
return 1, fmt.Errorf("managed runtime validators unavailable")
|
||||||
@@ -411,7 +395,34 @@ func managedRuntimeGuard(d managedDeps) (int, error) {
|
|||||||
return 0, nil
|
return 0, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func managedTargetSettings(lookup func(string) (string, bool)) (int, error) {
|
||||||
|
role, _ := lookup("ZAPRET_AUTOTUNE_TARGET_ROLE")
|
||||||
|
if role != "managed-test" {
|
||||||
|
return 2, fmt.Errorf("ZAPRET_AUTOTUNE_TARGET_ROLE must be managed-test")
|
||||||
|
}
|
||||||
|
for key, expected := range map[string]string{
|
||||||
|
"ZAPRET_AUTOTUNE_TARGET_ID": managedTargetID,
|
||||||
|
"ZAPRET_AUTOTUNE_TARGET_DIR": managedTargetDirectory,
|
||||||
|
"ZAPRET_CONFIG": managedTargetConfig,
|
||||||
|
} {
|
||||||
|
if value, _ := lookup(key); value != expected {
|
||||||
|
return 2, fmt.Errorf("%s does not match the fresh managed target contract", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
portText, _ := lookup("SOCKS_PORT")
|
||||||
|
port, portErr := strconv.Atoi(portText)
|
||||||
|
proxy, _ := lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
|
||||||
|
proxy, proxyErr := managedProbeProxy(proxy)
|
||||||
|
if portErr != nil || port < 1 || port > 65535 || strconv.Itoa(port) != portText || proxyErr != nil || proxy != "socks5h://127.0.0.1:"+portText {
|
||||||
|
return 2, fmt.Errorf("managed SOCKS proxy must match the target SOCKS_PORT listener")
|
||||||
|
}
|
||||||
|
return port, nil
|
||||||
|
}
|
||||||
|
|
||||||
func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
||||||
|
if len(args) == 1 && args[0] == "resource-status" {
|
||||||
|
return resourceStatusCLI(stdout, d)
|
||||||
|
}
|
||||||
if len(args) > 0 && args[0] == "monitor" {
|
if len(args) > 0 && args[0] == "monitor" {
|
||||||
return managedMonitorCLI(args[1:], stdout, stderr, d)
|
return managedMonitorCLI(args[1:], stdout, stderr, d)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -57,6 +57,17 @@ func managedMonitorCLI(args []string, stdout, stderr io.Writer, target managedDe
|
|||||||
}
|
}
|
||||||
|
|
||||||
func managedMonitorCLIContext(ctx context.Context, args []string, stdout, stderr io.Writer, target managedDeps, supervise func(context.Context, managedMonitorDeps) error) (code int) {
|
func managedMonitorCLIContext(ctx context.Context, args []string, stdout, stderr io.Writer, target managedDeps, supervise func(context.Context, managedMonitorDeps) error) (code int) {
|
||||||
|
resourcePath, _ := target.lookup("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
|
||||||
|
if len(args) == 2 && args[0] == "--resource-config" && args[1] != "" {
|
||||||
|
return resourceMonitorCLI(ctx, args[1], stdout, stderr, target)
|
||||||
|
}
|
||||||
|
if resourcePath != "" {
|
||||||
|
if len(args) != 0 {
|
||||||
|
fmt.Fprintln(stderr, "configerror: generic monitor accepts only --resource-config FILE")
|
||||||
|
return 64
|
||||||
|
}
|
||||||
|
return resourceMonitorCLI(ctx, resourcePath, stdout, stderr, target)
|
||||||
|
}
|
||||||
cfg, err := managedMonitorConfig(args, target.lookup)
|
cfg, err := managedMonitorConfig(args, target.lookup)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
fmt.Fprintln(stderr, "error: invalid managed monitor configuration")
|
fmt.Fprintln(stderr, "error: invalid managed monitor configuration")
|
||||||
|
|||||||
@@ -702,7 +702,11 @@ func TestManagedEntrypointSupervision(t *testing.T) {
|
|||||||
text = shortened
|
text = shortened
|
||||||
}
|
}
|
||||||
if mode == "stop-hang" {
|
if mode == "stop-hang" {
|
||||||
text = strings.Replace(text, "timeout -k 5 30", "timeout -k 1 1", 1)
|
shortened := strings.Replace(text, "cleanup_command 30 "+zapret+" stop", "timeout -k 1 1 "+zapret+" stop", 1)
|
||||||
|
if shortened == text {
|
||||||
|
t.Fatal("fixture did not shorten stop watchdog")
|
||||||
|
}
|
||||||
|
text = shortened
|
||||||
}
|
}
|
||||||
entry := write("entrypoint", text)
|
entry := write("entrypoint", text)
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"regexp"
|
"regexp"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
@@ -42,6 +43,8 @@ type resourceCheckSpec struct {
|
|||||||
Method string `json:"method"`
|
Method string `json:"method"`
|
||||||
TLS string `json:"tls"`
|
TLS string `json:"tls"`
|
||||||
TimeoutMS int `json:"timeout_ms"`
|
TimeoutMS int `json:"timeout_ms"`
|
||||||
|
IntervalMS *int `json:"interval_ms,omitempty"`
|
||||||
|
CooldownMS *int `json:"cooldown_ms,omitempty"`
|
||||||
Expect resourceCheckExpect `json:"expect"`
|
Expect resourceCheckExpect `json:"expect"`
|
||||||
HeadersEnv map[string]string `json:"headers_env,omitempty"`
|
HeadersEnv map[string]string `json:"headers_env,omitempty"`
|
||||||
AllowPrivate bool `json:"allow_private,omitempty"`
|
AllowPrivate bool `json:"allow_private,omitempty"`
|
||||||
@@ -139,6 +142,18 @@ func parseResourceChecks(data []byte, lookup func(string) (string, bool)) (resou
|
|||||||
return fail()
|
return fail()
|
||||||
}
|
}
|
||||||
ids[r.ID] = true
|
ids[r.ID] = true
|
||||||
|
for _, scheduling := range []struct {
|
||||||
|
value **int
|
||||||
|
fallback int
|
||||||
|
}{{&r.IntervalMS, 300000}, {&r.CooldownMS, 30000}} {
|
||||||
|
if *scheduling.value == nil {
|
||||||
|
v := scheduling.fallback
|
||||||
|
*scheduling.value = &v
|
||||||
|
}
|
||||||
|
if **scheduling.value < 100 || **scheduling.value > 86400000 {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
}
|
||||||
if (r.URL == "") == (r.URLEnv == "") {
|
if (r.URL == "") == (r.URLEnv == "") {
|
||||||
return fail()
|
return fail()
|
||||||
}
|
}
|
||||||
@@ -202,7 +217,13 @@ func parseResourceChecks(data []byte, lookup func(string) (string, bool)) (resou
|
|||||||
return fail()
|
return fail()
|
||||||
}
|
}
|
||||||
headers := map[string]bool{}
|
headers := map[string]bool{}
|
||||||
for name, env := range r.HeadersEnv {
|
names := make([]string, 0, len(r.HeadersEnv))
|
||||||
|
for name := range r.HeadersEnv {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
for _, name := range names {
|
||||||
|
env := r.HeadersEnv[name]
|
||||||
lower := strings.ToLower(name)
|
lower := strings.ToLower(name)
|
||||||
if !resourceHeader.MatchString(name) || headers[lower] || !resourceEnv.MatchString(env) {
|
if !resourceHeader.MatchString(name) || headers[lower] || !resourceEnv.MatchString(env) {
|
||||||
return fail()
|
return fail()
|
||||||
@@ -234,6 +255,10 @@ func parseResourceChecks(data []byte, lookup func(string) (string, bool)) (resou
|
|||||||
}
|
}
|
||||||
|
|
||||||
func readResourceChecks(path string, lookup func(string) (string, bool)) (resourceCheckConfig, error) {
|
func readResourceChecks(path string, lookup func(string) (string, bool)) (resourceCheckConfig, error) {
|
||||||
|
return readResourceChecksMode(path, lookup, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func readResourceChecksMode(path string, lookup func(string) (string, bool), owned bool) (resourceCheckConfig, error) {
|
||||||
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
|
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return resourceCheckConfig{}, errResourceConfig
|
return resourceCheckConfig{}, errResourceConfig
|
||||||
@@ -244,10 +269,22 @@ func readResourceChecks(path string, lookup func(string) (string, bool)) (resour
|
|||||||
if err != nil || !info.Mode().IsRegular() || info.Size() > 1<<20 {
|
if err != nil || !info.Mode().IsRegular() || info.Size() > 1<<20 {
|
||||||
return resourceCheckConfig{}, errResourceConfig
|
return resourceCheckConfig{}, errResourceConfig
|
||||||
}
|
}
|
||||||
|
if owned {
|
||||||
|
st, ok := info.Sys().(*syscall.Stat_t)
|
||||||
|
if !ok || (st.Uid != 0 && st.Uid != uint32(os.Geteuid())) || st.Nlink != 1 || info.Mode().Perm()&0022 != 0 {
|
||||||
|
return resourceCheckConfig{}, errResourceConfig
|
||||||
|
}
|
||||||
|
}
|
||||||
data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
|
data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return resourceCheckConfig{}, errResourceConfig
|
return resourceCheckConfig{}, errResourceConfig
|
||||||
}
|
}
|
||||||
|
if owned {
|
||||||
|
current, err := os.Lstat(path)
|
||||||
|
if err != nil || !os.SameFile(info, current) {
|
||||||
|
return resourceCheckConfig{}, errResourceConfig
|
||||||
|
}
|
||||||
|
}
|
||||||
return parseResourceChecks(data, lookup)
|
return parseResourceChecks(data, lookup)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -329,6 +366,52 @@ type resourceCheckResult struct {
|
|||||||
Status int `json:"status"`
|
Status int `json:"status"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A synchronous runner: subprocess cancellation joins before this returns.
|
||||||
|
// The guard is also called after DNS, immediately before the pinned transfer.
|
||||||
|
func runResourceProtocol(ctx context.Context, service string, r resourceCheckSpec, protocol, server, proxy string, d resourceCheckDeps) (resourceCheckResult, error) {
|
||||||
|
result := resourceCheckResult{Service: service, Resource: r.ID, Group: r.Group, TLS: protocol, Outcome: "dns_failed"}
|
||||||
|
if err := d.guard(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
probeCtx, cancel := context.WithTimeout(ctx, time.Duration(r.TimeoutMS)*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
ips, err := d.resolve(probeCtx, server, r.host)
|
||||||
|
if err == nil && len(ips) > 0 && len(ips) <= 64 {
|
||||||
|
approved := true
|
||||||
|
for _, ip := range ips {
|
||||||
|
if !approvedResourceIP(r, ip) {
|
||||||
|
approved = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result.Outcome = "destination_denied"
|
||||||
|
if approved {
|
||||||
|
if err := d.guard(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
pin := r.host + ":443:" + net.JoinHostPort(ips[0].String(), "443")
|
||||||
|
options := resourceTransferOptions{generic: true, method: r.Method, limit: r.Expect.MaxBytes, timeout: time.Duration(r.TimeoutMS) * time.Millisecond, config: r.privateConfig + "connect-to = " + privateCurlValue(pin) + "\n"}
|
||||||
|
transfer, _, err := collectResourceTransferOptions(probeCtx, proxy, r.URL, r.host, protocol, false, d.runner, options)
|
||||||
|
result.Outcome = "transfer_failed"
|
||||||
|
if err == nil {
|
||||||
|
result.Bytes, result.TimeMS, result.Status = len(transfer.Body), transfer.LatencyMS, transfer.Status
|
||||||
|
result.Outcome = "expectation_failed"
|
||||||
|
for _, status := range r.Expect.Statuses {
|
||||||
|
if status == transfer.Status && len(transfer.Body) >= r.Expect.MinBytes && (r.Expect.ContentType == "" || transfer.ContentType == r.Expect.ContentType) {
|
||||||
|
result.Outcome = "pass"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if probeCtx.Err() != nil {
|
||||||
|
result.Outcome = "timeout"
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
result.Outcome = "cancelled"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
func resourceCheckCLI(ctx context.Context, path string, stdout, stderr io.Writer, d resourceCheckDeps) int {
|
func resourceCheckCLI(ctx context.Context, path string, stdout, stderr io.Writer, d resourceCheckDeps) int {
|
||||||
if d.lookup == nil {
|
if d.lookup == nil {
|
||||||
fmt.Fprintln(stderr, "configerror: invalid resource check configuration")
|
fmt.Fprintln(stderr, "configerror: invalid resource check configuration")
|
||||||
@@ -357,57 +440,11 @@ func resourceCheckCLI(ctx context.Context, path string, stdout, stderr io.Writer
|
|||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if d.guard() != nil {
|
result, err := runResourceProtocol(ctx, s.ID, r, protocol, server, proxy, d)
|
||||||
|
if err != nil {
|
||||||
fmt.Fprintln(stderr, "blocked: resource check ownership unavailable")
|
fmt.Fprintln(stderr, "blocked: resource check ownership unavailable")
|
||||||
return 2
|
return 2
|
||||||
}
|
}
|
||||||
result := resourceCheckResult{Service: s.ID, Resource: r.ID, Group: r.Group, TLS: protocol, Outcome: "dns_failed"}
|
|
||||||
probeCtx, cancel := context.WithTimeout(ctx, time.Duration(r.TimeoutMS)*time.Millisecond)
|
|
||||||
ips, err := d.resolve(probeCtx, server, r.host)
|
|
||||||
if err == nil && len(ips) > 0 && len(ips) <= 64 {
|
|
||||||
approved := true
|
|
||||||
for _, ip := range ips {
|
|
||||||
if !approvedResourceIP(r, ip) {
|
|
||||||
approved = false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
result.Outcome = "destination_denied"
|
|
||||||
if approved {
|
|
||||||
if d.guard() != nil {
|
|
||||||
cancel()
|
|
||||||
fmt.Fprintln(stderr, "blocked: resource check ownership unavailable")
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
// --connect-to changes the SOCKS CONNECT destination even for socks5h;
|
|
||||||
// URL authority, HTTP Host, SNI and certificate verification stay original.
|
|
||||||
pin := r.host + ":443:" + net.JoinHostPort(ips[0].String(), "443")
|
|
||||||
options := resourceTransferOptions{generic: true, method: r.Method, limit: r.Expect.MaxBytes, timeout: time.Duration(r.TimeoutMS) * time.Millisecond, config: r.privateConfig + "connect-to = " + privateCurlValue(pin) + "\n"}
|
|
||||||
transfer, _, err := collectResourceTransferOptions(probeCtx, proxy, r.URL, r.host, protocol, false, d.runner, options)
|
|
||||||
result.Outcome = "transfer_failed"
|
|
||||||
if err == nil {
|
|
||||||
result.Bytes, result.TimeMS, result.Status = len(transfer.Body), transfer.LatencyMS, transfer.Status
|
|
||||||
result.Outcome = "expectation_failed"
|
|
||||||
statusOK := false
|
|
||||||
for _, status := range r.Expect.Statuses {
|
|
||||||
if status == transfer.Status {
|
|
||||||
statusOK = true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if statusOK {
|
|
||||||
if len(transfer.Body) >= r.Expect.MinBytes && (r.Expect.ContentType == "" || transfer.ContentType == r.Expect.ContentType) {
|
|
||||||
result.Outcome = "pass"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if probeCtx.Err() != nil {
|
|
||||||
result.Outcome = "timeout"
|
|
||||||
if ctx.Err() != nil {
|
|
||||||
result.Outcome = "cancelled"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
cancel()
|
|
||||||
if result.Outcome != "pass" {
|
if result.Outcome != "pass" {
|
||||||
code = 1
|
code = 1
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,441 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// HEALTHY means only that the configured checks passed, not production READY.
|
||||||
|
type resourceMonitorSource struct {
|
||||||
|
config resourceCheckConfig
|
||||||
|
server, proxy, fingerprint string
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceStaticConfig(lookup func(string) (string, bool)) (string, string, error) {
|
||||||
|
dns, err := parseManagedDNSConfig(lookup)
|
||||||
|
raw, _ := lookup("ZAPRET_AUTOTUNE_SOCKS_PROXY")
|
||||||
|
proxy, proxyErr := managedProbeProxy(raw)
|
||||||
|
if err != nil || proxyErr != nil || dns.Mode == "native" && dns.NativeServer == "" {
|
||||||
|
return "", "", errResourceConfig
|
||||||
|
}
|
||||||
|
server := dns.serverAddress()
|
||||||
|
if dns.Mode == "native" {
|
||||||
|
server = net.JoinHostPort(server, "53")
|
||||||
|
}
|
||||||
|
return server, proxy, nil
|
||||||
|
}
|
||||||
|
func readResourceMonitorSource(path string, lookup func(string) (string, bool), key []byte) (resourceMonitorSource, error) {
|
||||||
|
// Resolve each referenced environment value once in this immutable snapshot.
|
||||||
|
values := map[string]struct {
|
||||||
|
value string
|
||||||
|
ok bool
|
||||||
|
}{}
|
||||||
|
snapshot := func(name string) (string, bool) {
|
||||||
|
if v, ok := values[name]; ok {
|
||||||
|
return v.value, v.ok
|
||||||
|
}
|
||||||
|
v, ok := lookup(name)
|
||||||
|
if len(v) > 16384 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
values[name] = struct {
|
||||||
|
value string
|
||||||
|
ok bool
|
||||||
|
}{v, ok}
|
||||||
|
return v, ok
|
||||||
|
}
|
||||||
|
cfg, err := readResourceChecksMode(path, snapshot, true)
|
||||||
|
server, proxy, staticErr := resourceStaticConfig(snapshot)
|
||||||
|
if err != nil || staticErr != nil {
|
||||||
|
return resourceMonitorSource{}, errResourceConfig
|
||||||
|
}
|
||||||
|
source := resourceMonitorSource{config: cfg, server: server, proxy: proxy}
|
||||||
|
b, _ := json.Marshal(cfg)
|
||||||
|
// Marshal omits the private curl headers; bind them explicitly as well.
|
||||||
|
parts := []string{string(b), server, proxy}
|
||||||
|
for _, s := range cfg.Services {
|
||||||
|
for _, r := range s.Resources {
|
||||||
|
parts = append(parts, r.privateConfig)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
b, _ = json.Marshal(parts)
|
||||||
|
source.fingerprint = resourceMAC(key, b)
|
||||||
|
return source, nil
|
||||||
|
}
|
||||||
|
func resourceRowFingerprint(key []byte, source resourceMonitorSource, r resourceCheckSpec) string {
|
||||||
|
b, _ := json.Marshal(r)
|
||||||
|
b, _ = json.Marshal([]string{string(b), r.privateConfig, source.server, source.proxy})
|
||||||
|
return resourceMAC(key, b)
|
||||||
|
}
|
||||||
|
func reconcileResourceRows(key []byte, source resourceMonitorSource, prior []resourceMonitorRow) []resourceMonitorRow {
|
||||||
|
old := map[string]resourceMonitorRow{}
|
||||||
|
for _, r := range prior {
|
||||||
|
old[r.Service+"/"+r.Resource] = r
|
||||||
|
}
|
||||||
|
rows := []resourceMonitorRow{}
|
||||||
|
for _, s := range source.config.Services {
|
||||||
|
for _, r := range s.Resources {
|
||||||
|
fp := resourceRowFingerprint(key, source, r)
|
||||||
|
row, ok := old[s.ID+"/"+r.ID]
|
||||||
|
if !ok || row.Fingerprint != fp {
|
||||||
|
row = resourceMonitorRow{Service: s.ID, Resource: r.ID, Fingerprint: fp, Health: "UNKNOWN", Outcome: "unknown"}
|
||||||
|
}
|
||||||
|
rows = append(rows, row)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return rows
|
||||||
|
}
|
||||||
|
|
||||||
|
type resourceMonitorDeps struct {
|
||||||
|
load func() (resourceMonitorState, error)
|
||||||
|
save func(resourceMonitorState) error
|
||||||
|
source func() (resourceMonitorSource, error)
|
||||||
|
guard func() error
|
||||||
|
probe func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error)
|
||||||
|
key []byte
|
||||||
|
now func() time.Time
|
||||||
|
wait func(context.Context, time.Duration) error
|
||||||
|
report func(bool)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sequential round robin with independent due times; reload at least every
|
||||||
|
// second while idle/blocked. No recovery leases or strategy mutations.
|
||||||
|
func runResourceMonitor(ctx context.Context, d resourceMonitorDeps) error {
|
||||||
|
cursor := 0
|
||||||
|
persistenceBlocked, failedMAC, attemptedMAC := false, "", ""
|
||||||
|
var shutdownError error
|
||||||
|
for ctx.Err() == nil {
|
||||||
|
state, stateErr := d.load()
|
||||||
|
if persistenceBlocked && stateErr == nil && state.MAC != failedMAC && state.MAC != attemptedMAC {
|
||||||
|
persistenceBlocked = false
|
||||||
|
}
|
||||||
|
source, sourceErr := d.source()
|
||||||
|
blocked := persistenceBlocked || stateErr != nil || sourceErr != nil || d.guard() != nil
|
||||||
|
dirty := false
|
||||||
|
if !blocked {
|
||||||
|
rows := reconcileResourceRows(d.key, source, state.Rows)
|
||||||
|
dirty = !reflect.DeepEqual(rows, state.Rows)
|
||||||
|
state.Rows = rows
|
||||||
|
if len(state.Rows) > 0 {
|
||||||
|
cursor %= len(state.Rows)
|
||||||
|
for n := 0; n < len(state.Rows); n++ {
|
||||||
|
i := (cursor + n) % len(state.Rows)
|
||||||
|
row := &state.Rows[i]
|
||||||
|
if row.NextDue > d.now().UnixMilli() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
var spec resourceCheckSpec
|
||||||
|
for _, s := range source.config.Services {
|
||||||
|
if s.ID == row.Service {
|
||||||
|
for _, r := range s.Resources {
|
||||||
|
if r.ID == row.Resource {
|
||||||
|
spec = r
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
guard := func() error {
|
||||||
|
if d.guard() != nil {
|
||||||
|
return errResourceConfig
|
||||||
|
}
|
||||||
|
fresh, err := d.source()
|
||||||
|
if err != nil || fresh.fingerprint != source.fingerprint {
|
||||||
|
return errResourceConfig
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
outcome, err := d.probe(ctx, row.Service, spec, source, guard)
|
||||||
|
cursor = (i + 1) % len(state.Rows)
|
||||||
|
if err != nil {
|
||||||
|
if ctx.Err() == nil || !managedCancellationOnly(err, ctx.Err()) {
|
||||||
|
blocked = true
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
shutdownError = err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
// Cancellation joins the probe but is not completed evidence.
|
||||||
|
// Keep the previous health, failure count and due time intact.
|
||||||
|
if ctx.Err() != nil || outcome == "cancelled" {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
dirty = true
|
||||||
|
row.Outcome, row.LastCompleted = outcome, d.now().UnixMilli()
|
||||||
|
delay := *spec.CooldownMS
|
||||||
|
if outcome == "pass" {
|
||||||
|
row.Health, row.Failures, delay = "HEALTHY", 0, *spec.IntervalMS
|
||||||
|
} else {
|
||||||
|
row.Health = "DEGRADED"
|
||||||
|
if row.Failures < ^uint64(0) {
|
||||||
|
row.Failures++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
row.NextDue = row.LastCompleted + int64(delay)
|
||||||
|
break // persist each completed (and joined) check before another probe
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
dirty = dirty || state.Blocked != blocked
|
||||||
|
state.Blocked = blocked
|
||||||
|
if dirty {
|
||||||
|
state.Updated = d.now().UnixMilli()
|
||||||
|
}
|
||||||
|
if dirty && stateErr == nil && !persistenceBlocked {
|
||||||
|
if d.save(state) != nil {
|
||||||
|
blocked, persistenceBlocked, failedMAC, attemptedMAC = true, true, state.MAC, resourceStateMAC(d.key, state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
d.report(blocked)
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
if persistenceBlocked {
|
||||||
|
return errors.Join(shutdownError, errResourceState)
|
||||||
|
}
|
||||||
|
return shutdownError
|
||||||
|
}
|
||||||
|
// Bound reloads, but don't delay an already due sibling by a cooldown.
|
||||||
|
delay := time.Second
|
||||||
|
if !blocked {
|
||||||
|
for _, r := range state.Rows {
|
||||||
|
milliseconds := r.NextDue - d.now().UnixMilli()
|
||||||
|
if milliseconds > 1000 {
|
||||||
|
milliseconds = 1000
|
||||||
|
}
|
||||||
|
remaining := time.Duration(milliseconds) * time.Millisecond
|
||||||
|
if remaining < 0 {
|
||||||
|
remaining = 0
|
||||||
|
}
|
||||||
|
if remaining < delay {
|
||||||
|
delay = remaining
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := d.wait(ctx, delay); err != nil {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
if persistenceBlocked {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if persistenceBlocked {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func probeResourceMonitor(ctx context.Context, service string, spec resourceCheckSpec, source resourceMonitorSource, guard func() error, check resourceCheckDeps) (string, error) {
|
||||||
|
check.guard = guard
|
||||||
|
protocols := []string{spec.TLS}
|
||||||
|
if spec.TLS == "both" {
|
||||||
|
protocols = []string{"tls12", "tls13"}
|
||||||
|
}
|
||||||
|
outcome := "pass"
|
||||||
|
for _, protocol := range protocols {
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return "cancelled", nil
|
||||||
|
}
|
||||||
|
result, err := runResourceProtocol(ctx, service, spec, protocol, source.server, source.proxy, check)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if result.Outcome != "pass" {
|
||||||
|
outcome = result.Outcome
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return outcome, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceMonitorCLI(ctx context.Context, path string, stdout, stderr io.Writer, target managedDeps) (code int) {
|
||||||
|
return resourceMonitorCLIWithCheck(ctx, path, stdout, stderr, target, productionResourceCheckDeps())
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceMonitorCLIWithCheck(ctx context.Context, path string, stdout, stderr io.Writer, target managedDeps, check resourceCheckDeps) (code int) {
|
||||||
|
port, settingsErr := managedTargetSettings(target.lookup)
|
||||||
|
if settingsErr != nil || port != 1080 {
|
||||||
|
fmt.Fprintln(stderr, "configerror: invalid generic target settings")
|
||||||
|
return 64
|
||||||
|
}
|
||||||
|
if _, _, err := resourceStaticConfig(target.lookup); err != nil {
|
||||||
|
fmt.Fprintln(stderr, "configerror: invalid generic DNS/proxy configuration")
|
||||||
|
return 64
|
||||||
|
}
|
||||||
|
var lock *managedMonitorSingleton
|
||||||
|
var store *resourceMonitorStore
|
||||||
|
defer func() {
|
||||||
|
if store != nil && store.close() != nil {
|
||||||
|
code = 78
|
||||||
|
}
|
||||||
|
if lock != nil && lock.close() != nil {
|
||||||
|
code = 78
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
lastBlocked := false
|
||||||
|
report := func(blocked bool) {
|
||||||
|
if blocked && !lastBlocked {
|
||||||
|
fmt.Fprintln(stderr, "BLOCKED: generic resource monitor guard/state unavailable")
|
||||||
|
}
|
||||||
|
lastBlocked = blocked
|
||||||
|
}
|
||||||
|
fmt.Fprintln(stdout, "generic resource monitor: configured checks only; no recovery/apply; HEALTHY is not production READY")
|
||||||
|
// Guard failures before store/lock acquisition remain cancellable and alive.
|
||||||
|
for ctx.Err() == nil {
|
||||||
|
if lock == nil {
|
||||||
|
var err error
|
||||||
|
lock, err = acquireManagedMonitorSingleton(target)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, syscall.EWOULDBLOCK) {
|
||||||
|
fmt.Fprintln(stderr, "BLOCKED: monitor singleton already held")
|
||||||
|
return 78
|
||||||
|
}
|
||||||
|
report(true)
|
||||||
|
if waitManagedMonitor(ctx, time.Second) != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
var err error
|
||||||
|
store, err = openResourceMonitorStore(target.directory, true)
|
||||||
|
if err != nil {
|
||||||
|
report(true)
|
||||||
|
if waitManagedMonitor(ctx, time.Second) != nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
err := runResourceMonitor(ctx, resourceMonitorDeps{
|
||||||
|
load: store.load, save: store.save, key: store.key,
|
||||||
|
source: func() (resourceMonitorSource, error) {
|
||||||
|
return readResourceMonitorSource(path, target.lookup, store.key)
|
||||||
|
},
|
||||||
|
guard: func() error {
|
||||||
|
if resourceSingletonGuard(lock, target.directory) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
// Probe guards validate the held snapshot, never reload/adopt a new
|
||||||
|
// authenticated operator edit while an older check is in flight.
|
||||||
|
if store.snapshotGuard() != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if resourceManagedIdentity(target.directory) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return check.guard()
|
||||||
|
},
|
||||||
|
probe: func(ctx context.Context, service string, spec resourceCheckSpec, source resourceMonitorSource, guard func() error) (string, error) {
|
||||||
|
return probeResourceMonitor(ctx, service, spec, source, guard, check)
|
||||||
|
},
|
||||||
|
now: time.Now, wait: waitManagedMonitor, report: report,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(stderr, "BLOCKED: generic monitor safety/finalization failure")
|
||||||
|
return 78
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceSingletonGuard(lock *managedMonitorSingleton, dir string) error {
|
||||||
|
path := filepath.Join(dir, managedMonitorLockName)
|
||||||
|
if managedPrivateFile(path) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
current, err := os.Lstat(path)
|
||||||
|
held, heldErr := lock.file.Stat()
|
||||||
|
if err != nil || heldErr != nil || !os.SameFile(current, held) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceStatusCLI(stdout io.Writer, target managedDeps) int {
|
||||||
|
blocked := func() int { fmt.Fprintln(stdout, `{"health":"BLOCKED","stale":true}`); return 1 }
|
||||||
|
store, err := openResourceMonitorStore(target.directory, false)
|
||||||
|
if err != nil {
|
||||||
|
return blocked()
|
||||||
|
}
|
||||||
|
defer store.close()
|
||||||
|
state, err := store.load()
|
||||||
|
if err != nil {
|
||||||
|
return blocked()
|
||||||
|
}
|
||||||
|
path, _ := target.lookup("ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
|
||||||
|
source, err := readResourceMonitorSource(path, target.lookup, store.key)
|
||||||
|
check := productionResourceCheckDeps()
|
||||||
|
if err != nil || check.guard() != nil || resourceManagedIdentity(target.directory) != nil {
|
||||||
|
return blocked()
|
||||||
|
}
|
||||||
|
rows := reconcileResourceRows(store.key, source, state.Rows)
|
||||||
|
b1, _ := json.Marshal(rows)
|
||||||
|
b2, _ := json.Marshal(state.Rows)
|
||||||
|
stale := resourceSnapshotStale(state, time.Now().UnixMilli()) || string(b1) != string(b2)
|
||||||
|
health := "CHECKS_ONLY"
|
||||||
|
if stale {
|
||||||
|
health = "STALE"
|
||||||
|
}
|
||||||
|
if state.Blocked {
|
||||||
|
health = "BLOCKED"
|
||||||
|
}
|
||||||
|
if json.NewEncoder(stdout).Encode(struct {
|
||||||
|
Health string `json:"health"`
|
||||||
|
Stale bool `json:"stale"`
|
||||||
|
State resourceMonitorState `json:"state"`
|
||||||
|
}{health, stale, state}) != nil {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
if stale || state.Blocked {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// Updated is a semantic-change timestamp, not a disk heartbeat. Evidence is
|
||||||
|
// stale only once its next scheduled check plus the maximum run budget is past.
|
||||||
|
func resourceSnapshotStale(state resourceMonitorState, now int64) bool {
|
||||||
|
if state.Updated > now {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if len(state.Rows) == 0 {
|
||||||
|
return now-state.Updated > 65000
|
||||||
|
}
|
||||||
|
for _, row := range state.Rows {
|
||||||
|
if row.NextDue == 0 {
|
||||||
|
if now-state.Updated > 65000 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
} else if now > row.NextDue && now-row.NextDue > 65000 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceManagedIdentity(dir string) error {
|
||||||
|
m := &managedTransaction{Dir: dir, Name: "config"}
|
||||||
|
if managedCheckIdentity(m) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
j, err := m.load()
|
||||||
|
if err != nil || j.State != "committed" {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
b, err := m.read("config")
|
||||||
|
if err != nil || managedHash(b) != j.Hash {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,326 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func resourceCompletedFixture(t *testing.T) (resourceMonitorSource, []byte, resourceMonitorState) {
|
||||||
|
t.Helper()
|
||||||
|
source, key := resourceMonitorFixture(t)
|
||||||
|
state := resourceMonitorState{Version: 1, Updated: 5000, Rows: reconcileResourceRows(key, source, nil)}
|
||||||
|
for i := range state.Rows {
|
||||||
|
state.Rows[i].Health, state.Rows[i].Outcome, state.Rows[i].LastCompleted = "HEALTHY", "pass", 5000
|
||||||
|
state.Rows[i].NextDue = 9000
|
||||||
|
}
|
||||||
|
return source, key, state
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorCancelledRowsUnchangedAndJoined(t *testing.T) {
|
||||||
|
for _, mixed := range []bool{false, true} {
|
||||||
|
t.Run(map[bool]string{false: "curl-cancel", true: "cancel-and-guard"}[mixed], func(t *testing.T) {
|
||||||
|
source, key, state := resourceCompletedFixture(t)
|
||||||
|
prior := append([]resourceMonitorRow(nil), state.Rows...)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
joined, writes := false, 0
|
||||||
|
check := resourceCheckDeps{resolve: func(context.Context, string, string) ([]net.IP, error) {
|
||||||
|
return []net.IP{net.ParseIP("93.184.216.34")}, nil
|
||||||
|
}, runner: func(ctx context.Context, _ []string, _ io.Reader, _ io.Writer, _ io.Writer) error {
|
||||||
|
cancel()
|
||||||
|
<-ctx.Done()
|
||||||
|
joined = true
|
||||||
|
return ctx.Err()
|
||||||
|
}}
|
||||||
|
d := resourceMonitorDeps{key: key, now: func() time.Time { return time.UnixMilli(10000) }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(s resourceMonitorState) error {
|
||||||
|
if !joined {
|
||||||
|
t.Error("persisted before joining curl")
|
||||||
|
}
|
||||||
|
writes++
|
||||||
|
state = s
|
||||||
|
return nil
|
||||||
|
}, source: func() (resourceMonitorSource, error) { return source, nil }, guard: func() error { return nil }, report: func(bool) {}, wait: func(context.Context, time.Duration) error { t.Error("wait after cancellation"); return nil }}
|
||||||
|
d.probe = func(ctx context.Context, service string, spec resourceCheckSpec, source resourceMonitorSource, guard func() error) (string, error) {
|
||||||
|
outcome, err := probeResourceMonitor(ctx, service, spec, source, guard, check)
|
||||||
|
if mixed {
|
||||||
|
return outcome, errors.Join(ctx.Err(), errResourceConfig)
|
||||||
|
}
|
||||||
|
return outcome, err
|
||||||
|
}
|
||||||
|
err := runResourceMonitor(ctx, d)
|
||||||
|
if mixed {
|
||||||
|
if !errors.Is(err, errResourceConfig) || !errors.Is(err, context.Canceled) || !state.Blocked || writes != 1 {
|
||||||
|
t.Fatal("lost mixed shutdown guard error", state, writes, err)
|
||||||
|
}
|
||||||
|
} else if err != nil || writes != 0 {
|
||||||
|
t.Fatal("cancel published evidence", writes, err)
|
||||||
|
}
|
||||||
|
if !joined || !reflect.DeepEqual(prior, state.Rows) {
|
||||||
|
t.Fatal("cancel changed completed rows or failed to join", state.Rows)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorIdleNoPersistenceHeartbeat(t *testing.T) {
|
||||||
|
source, key, state := resourceCompletedFixture(t)
|
||||||
|
for i := range state.Rows {
|
||||||
|
state.Rows[i].NextDue = 305000
|
||||||
|
}
|
||||||
|
now := time.UnixMilli(10000)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
writes, waits := 0, 0
|
||||||
|
d := resourceMonitorDeps{key: key, now: func() time.Time { return now }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(s resourceMonitorState) error { writes++; state = s; return nil }, source: func() (resourceMonitorSource, error) { return source, nil }, guard: func() error { return nil }, report: func(bool) {}, probe: func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error) {
|
||||||
|
t.Error("probed resource before due")
|
||||||
|
return "pass", nil
|
||||||
|
}, wait: func(_ context.Context, d time.Duration) error {
|
||||||
|
now = now.Add(d)
|
||||||
|
waits++
|
||||||
|
if waits == 5 {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}}
|
||||||
|
if err := runResourceMonitor(ctx, d); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if writes != 0 || state.Updated != 5000 {
|
||||||
|
t.Fatal("idle disk heartbeat", writes, state.Updated)
|
||||||
|
}
|
||||||
|
if resourceSnapshotStale(state, 100000) || !resourceSnapshotStale(state, 370001) {
|
||||||
|
t.Fatal("staleness still depends on idle heartbeat")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorBlockedTransitionsOnly(t *testing.T) {
|
||||||
|
source, key, state := resourceCompletedFixture(t)
|
||||||
|
for i := range state.Rows {
|
||||||
|
state.Rows[i].NextDue = 305000
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
tick, writes := 0, 0
|
||||||
|
d := resourceMonitorDeps{key: key, now: func() time.Time { return time.UnixMilli(10000 + int64(tick)*1000) }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(s resourceMonitorState) error { writes++; state = s; return nil }, source: func() (resourceMonitorSource, error) {
|
||||||
|
if tick < 3 {
|
||||||
|
return resourceMonitorSource{}, errResourceConfig
|
||||||
|
}
|
||||||
|
return source, nil
|
||||||
|
}, guard: func() error { return nil }, report: func(bool) {}, probe: func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error) {
|
||||||
|
t.Error("unexpected probe")
|
||||||
|
return "pass", nil
|
||||||
|
}, wait: func(context.Context, time.Duration) error {
|
||||||
|
tick++
|
||||||
|
if tick == 5 {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}}
|
||||||
|
if err := runResourceMonitor(ctx, d); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if writes != 2 || state.Blocked {
|
||||||
|
t.Fatal("blocked transition heartbeat or missing resume", writes, state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceBoundaryStore(t *testing.T, createState bool) (*resourceMonitorStore, resourceMonitorState) {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
if err := os.Chmod(dir, 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s, err := openResourceMonitorStore(dir, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { s.close() })
|
||||||
|
v, err := s.load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if createState {
|
||||||
|
if err := s.save(v); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return s, v
|
||||||
|
}
|
||||||
|
|
||||||
|
func replaceResourceEntry(t *testing.T, path string, data []byte) {
|
||||||
|
t.Helper()
|
||||||
|
if err := os.WriteFile(path+"-replacement", data, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(path+"-replacement", path); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorHeldEntriesRejectReplacement(t *testing.T) {
|
||||||
|
for _, name := range []string{resourceStateName, resourceKeyName} {
|
||||||
|
for _, point := range []string{"before-load", "loaded", "staged", "publish"} {
|
||||||
|
t.Run(name+"/"+point, func(t *testing.T) {
|
||||||
|
s, v := resourceBoundaryStore(t, true)
|
||||||
|
path := filepath.Join(s.path, name)
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var replacement os.FileInfo
|
||||||
|
mutate := func() {
|
||||||
|
replaceResourceEntry(t, path, data)
|
||||||
|
replacement, err = os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if point == "before-load" {
|
||||||
|
mutate()
|
||||||
|
} else {
|
||||||
|
s.fault = func(p, _ string) {
|
||||||
|
if p == point {
|
||||||
|
mutate()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if point == "before-load" || point == "loaded" {
|
||||||
|
if _, err := s.load(); err == nil {
|
||||||
|
t.Fatal("load adopted identical replaced inode")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.save(v); err == nil {
|
||||||
|
t.Fatal("save clobbered identical replacement")
|
||||||
|
}
|
||||||
|
current, err := os.Stat(path)
|
||||||
|
if err != nil || replacement == nil || !os.SameFile(current, replacement) {
|
||||||
|
t.Fatal("foreign inode changed", err)
|
||||||
|
}
|
||||||
|
got, err := os.ReadFile(path)
|
||||||
|
if err != nil || !bytes.Equal(got, data) {
|
||||||
|
t.Fatal("foreign bytes changed", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorMidStageCorruptionPreserved(t *testing.T) {
|
||||||
|
for _, name := range []string{resourceStateName, resourceKeyName, "stage", "stage-in-place"} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
s, v := resourceBoundaryStore(t, true)
|
||||||
|
bad := []byte("untrusted corrupt operator bytes")
|
||||||
|
path := ""
|
||||||
|
s.fault = func(point, stage string) {
|
||||||
|
if point != "staged" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
path = filepath.Join(s.path, name)
|
||||||
|
if name == "stage" || name == "stage-in-place" {
|
||||||
|
path = stage
|
||||||
|
}
|
||||||
|
if name == "stage-in-place" {
|
||||||
|
if err := os.WriteFile(path, bad, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
replaceResourceEntry(t, path, bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.save(v); err == nil {
|
||||||
|
t.Fatal("corruption overwritten")
|
||||||
|
}
|
||||||
|
got, err := os.ReadFile(path)
|
||||||
|
if err != nil || !bytes.Equal(got, bad) {
|
||||||
|
t.Fatal("unknown stage/file removed or clobbered", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorInitialPublicationExclusive(t *testing.T) {
|
||||||
|
s, v := resourceBoundaryStore(t, false)
|
||||||
|
path := filepath.Join(s.path, resourceStateName)
|
||||||
|
bad := []byte("new foreign entry created just before initial publication")
|
||||||
|
s.fault = func(point, _ string) {
|
||||||
|
if point == "publish" {
|
||||||
|
if err := os.WriteFile(path, bad, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.save(v); err == nil {
|
||||||
|
t.Fatal("exclusive initial publish overwrote raced entry")
|
||||||
|
}
|
||||||
|
got, err := os.ReadFile(path)
|
||||||
|
if err != nil || !bytes.Equal(got, bad) {
|
||||||
|
t.Fatal("new entry clobbered", err)
|
||||||
|
}
|
||||||
|
if _, err := s.load(); err == nil {
|
||||||
|
t.Fatal("new entry silently adopted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorPublicationRetainsNewAnchor(t *testing.T) {
|
||||||
|
s, v := resourceBoundaryStore(t, true)
|
||||||
|
old := s.stateFile
|
||||||
|
v.Updated = 10000
|
||||||
|
if err := s.save(v); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if old == s.stateFile {
|
||||||
|
t.Fatal("old inode retained instead of published inode")
|
||||||
|
}
|
||||||
|
if _, err := old.Stat(); err == nil {
|
||||||
|
t.Fatal("old anchor not closed after verified publication")
|
||||||
|
}
|
||||||
|
if s.entryMatches(resourceStateName, s.stateFile) != nil {
|
||||||
|
t.Fatal("published state not anchored")
|
||||||
|
}
|
||||||
|
if loaded, err := s.load(); err != nil || loaded.Updated != 10000 {
|
||||||
|
t.Fatal(loaded, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorOriginalIdentityRepair(t *testing.T) {
|
||||||
|
s, v := resourceBoundaryStore(t, true)
|
||||||
|
path := filepath.Join(s.path, resourceStateName)
|
||||||
|
valid, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
original, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(`{"version":99}`), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.load(); err == nil {
|
||||||
|
t.Fatal("corrupt original trusted")
|
||||||
|
}
|
||||||
|
if err := s.save(v); err == nil {
|
||||||
|
t.Fatal("corrupt original overwritten")
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, valid, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.load(); err != nil {
|
||||||
|
t.Fatal("valid original-identity repair blocked", err)
|
||||||
|
}
|
||||||
|
current, err := os.Stat(path)
|
||||||
|
if err != nil || !os.SameFile(current, original) {
|
||||||
|
t.Fatal("repair changed inode", err)
|
||||||
|
}
|
||||||
|
if err := s.save(v); err != nil {
|
||||||
|
t.Fatal("repaired original failed publish", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,159 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestResourceMonitorFinalPublicationStageGuard(t *testing.T) {
|
||||||
|
for _, existing := range []bool{false, true} {
|
||||||
|
for _, mutation := range []string{"byte", "truncate", "append", "growth", "path-replace", "key-replace", "key-in-place", "state-replace"} {
|
||||||
|
name := "initial/"
|
||||||
|
if existing {
|
||||||
|
name = "existing/"
|
||||||
|
}
|
||||||
|
t.Run(name+mutation, func(t *testing.T) {
|
||||||
|
s, v := resourceBoundaryStore(t, existing)
|
||||||
|
path := filepath.Join(s.path, resourceStateName)
|
||||||
|
oldBytes, oldErr := os.ReadFile(path)
|
||||||
|
oldInfo, _ := os.Stat(path)
|
||||||
|
if existing && oldErr != nil {
|
||||||
|
t.Fatal(oldErr)
|
||||||
|
}
|
||||||
|
v.Updated = 10000
|
||||||
|
var changedPath string
|
||||||
|
var changedBytes []byte
|
||||||
|
var changedInfo os.FileInfo
|
||||||
|
s.fault = func(point, stage string) {
|
||||||
|
if point != "publish" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(stage)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
changedPath = stage
|
||||||
|
switch mutation {
|
||||||
|
case "byte":
|
||||||
|
data[0] = '['
|
||||||
|
case "truncate":
|
||||||
|
data = data[:len(data)-1]
|
||||||
|
case "append":
|
||||||
|
data = append(data, ' ')
|
||||||
|
case "growth":
|
||||||
|
data = append(data, bytes.Repeat([]byte{'x'}, 65537)...)
|
||||||
|
case "key-replace", "key-in-place":
|
||||||
|
changedPath = filepath.Join(s.path, resourceKeyName)
|
||||||
|
data, err = os.ReadFile(changedPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if mutation == "key-in-place" {
|
||||||
|
data[0] ^= 1
|
||||||
|
}
|
||||||
|
case "state-replace":
|
||||||
|
changedPath = path
|
||||||
|
if existing {
|
||||||
|
data = append([]byte(nil), oldBytes...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
changedBytes = append([]byte(nil), data...)
|
||||||
|
if mutation == "path-replace" || mutation == "key-replace" || mutation == "state-replace" {
|
||||||
|
replaceResourceEntry(t, changedPath, data)
|
||||||
|
} else if err := os.WriteFile(changedPath, data, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
changedInfo, err = os.Stat(changedPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.save(v); !errors.Is(err, errResourceState) {
|
||||||
|
t.Fatal("unsafe stage published", err)
|
||||||
|
}
|
||||||
|
current, err := os.ReadFile(changedPath)
|
||||||
|
info, infoErr := os.Stat(changedPath)
|
||||||
|
if err != nil || infoErr != nil || !bytes.Equal(current, changedBytes) || !os.SameFile(info, changedInfo) {
|
||||||
|
t.Fatal("unknown artifact changed/deleted", err, infoErr)
|
||||||
|
}
|
||||||
|
if mutation != "state-replace" {
|
||||||
|
if existing {
|
||||||
|
current, err := os.ReadFile(path)
|
||||||
|
info, infoErr := os.Stat(path)
|
||||||
|
if err != nil || infoErr != nil || !bytes.Equal(current, oldBytes) || !os.SameFile(info, oldInfo) {
|
||||||
|
t.Fatal("trusted old state lost before final guard", err, infoErr)
|
||||||
|
}
|
||||||
|
} else if _, err := os.Lstat(path); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatal("initial corrupt state published", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if existing && s.entryMatches(resourceStateName, s.stateFile) != nil && mutation != "state-replace" {
|
||||||
|
t.Fatal("old trusted anchor lost")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorPublicationFailureBlocksFurtherProbes(t *testing.T) {
|
||||||
|
source, _, _ := resourceCompletedFixture(t)
|
||||||
|
s, _ := resourceBoundaryStore(t, true)
|
||||||
|
state, err := s.load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
state.Updated = 5000
|
||||||
|
state.Rows = reconcileResourceRows(s.key, source, nil)
|
||||||
|
for i := range state.Rows {
|
||||||
|
state.Rows[i].Health, state.Rows[i].Outcome = "HEALTHY", "pass"
|
||||||
|
state.Rows[i].LastCompleted, state.Rows[i].NextDue = 5000, 9000
|
||||||
|
}
|
||||||
|
if err := s.save(state); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
path := filepath.Join(s.path, resourceStateName)
|
||||||
|
old, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
oldInfo, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s.fault = func(point, stage string) {
|
||||||
|
if point == "publish" {
|
||||||
|
if err := os.WriteFile(stage, []byte("unknown stage corruption"), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
probes, waits, blocked := 0, 0, false
|
||||||
|
d := resourceMonitorDeps{key: s.key, now: func() time.Time { return time.UnixMilli(10000) }, load: s.load, save: s.save, source: func() (resourceMonitorSource, error) { return source, nil }, guard: s.snapshotGuard, report: func(v bool) { blocked = v }, probe: func(context.Context, string, resourceCheckSpec, resourceMonitorSource, func() error) (string, error) {
|
||||||
|
probes++
|
||||||
|
return "pass", nil
|
||||||
|
}, wait: func(context.Context, time.Duration) error {
|
||||||
|
waits++
|
||||||
|
if waits == 3 {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}}
|
||||||
|
if err := runResourceMonitor(ctx, d); !errors.Is(err, errResourceState) {
|
||||||
|
t.Fatal("lost unsafe publication failure", err)
|
||||||
|
}
|
||||||
|
if probes != 1 || !blocked {
|
||||||
|
t.Fatal("continued probing after failed publication", probes, blocked)
|
||||||
|
}
|
||||||
|
current, err := os.ReadFile(path)
|
||||||
|
currentInfo, infoErr := os.Stat(path)
|
||||||
|
if err != nil || infoErr != nil || !bytes.Equal(current, old) || !os.SameFile(currentInfo, oldInfo) {
|
||||||
|
t.Fatal("old completed snapshot lost", err, infoErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,402 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/rand"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"syscall"
|
||||||
|
)
|
||||||
|
|
||||||
|
const resourceStateName = ".resource-monitor-state.json"
|
||||||
|
const resourceKeyName = ".resource-monitor-key"
|
||||||
|
|
||||||
|
var errResourceState = errors.New("resource state blocked")
|
||||||
|
|
||||||
|
type resourceMonitorRow struct {
|
||||||
|
Service string `json:"service"`
|
||||||
|
Resource string `json:"resource"`
|
||||||
|
Fingerprint string `json:"fingerprint"`
|
||||||
|
Health string `json:"health"`
|
||||||
|
Outcome string `json:"outcome"`
|
||||||
|
Failures uint64 `json:"failures"`
|
||||||
|
LastCompleted int64 `json:"last_completed_ms"`
|
||||||
|
NextDue int64 `json:"next_due_ms"`
|
||||||
|
}
|
||||||
|
type resourceMonitorState struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
Updated int64 `json:"updated_ms"`
|
||||||
|
Blocked bool `json:"blocked"`
|
||||||
|
Rows []resourceMonitorRow `json:"rows"`
|
||||||
|
MAC string `json:"mac"`
|
||||||
|
}
|
||||||
|
type resourceMonitorStore struct {
|
||||||
|
dir *os.File
|
||||||
|
path string
|
||||||
|
key []byte
|
||||||
|
previous []byte
|
||||||
|
existed bool
|
||||||
|
keyFile, stateFile *os.File
|
||||||
|
// Deterministic filesystem interleavings in tests; nil in production.
|
||||||
|
fault func(point, stage string)
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourceMAC(key, data []byte) string {
|
||||||
|
m := hmac.New(sha256.New, key)
|
||||||
|
m.Write(data)
|
||||||
|
return hex.EncodeToString(m.Sum(nil))
|
||||||
|
}
|
||||||
|
func resourceStateMAC(key []byte, s resourceMonitorState) string {
|
||||||
|
s.MAC = ""
|
||||||
|
b, _ := json.Marshal(s)
|
||||||
|
return resourceMAC(key, b)
|
||||||
|
}
|
||||||
|
func (s *resourceMonitorStore) guard() error {
|
||||||
|
if managedDirectory(s.path) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
current, err := os.Stat(s.path)
|
||||||
|
held, heldErr := s.dir.Stat()
|
||||||
|
if err != nil || heldErr != nil || !os.SameFile(current, held) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func (s *resourceMonitorStore) entryPath(name string) string {
|
||||||
|
return filepath.Join("/proc/self/fd", strconv.FormatUint(uint64(s.dir.Fd()), 10), name)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *resourceMonitorStore) close() error {
|
||||||
|
var errs []error
|
||||||
|
for _, f := range []*os.File{s.stateFile, s.keyFile, s.dir} {
|
||||||
|
if f != nil {
|
||||||
|
errs = append(errs, f.Close())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
s.stateFile, s.keyFile, s.dir = nil, nil, nil
|
||||||
|
return errors.Join(errs...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func resourcePrivateEntry(f *os.File) error {
|
||||||
|
info, err := f.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
st, ok := info.Sys().(*syscall.Stat_t)
|
||||||
|
if !ok || !info.Mode().IsRegular() || info.Mode().Perm() != 0600 || st.Uid != uint32(os.Geteuid()) || st.Nlink != 1 || info.Size() > 65536 {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *resourceMonitorStore) entryMatches(name string, held *os.File) error {
|
||||||
|
current, err := os.Lstat(s.entryPath(name))
|
||||||
|
anchored, heldErr := held.Stat()
|
||||||
|
if err != nil || heldErr != nil || !os.SameFile(current, anchored) || resourcePrivateEntry(held) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *resourceMonitorStore) openEntry(name string) (*os.File, []byte, error) {
|
||||||
|
if s.guard() != nil {
|
||||||
|
return nil, nil, errResourceState
|
||||||
|
}
|
||||||
|
fd, err := syscall.Openat(int(s.dir.Fd()), name, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK|syscall.O_CLOEXEC, 0)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
f := os.NewFile(uintptr(fd), name)
|
||||||
|
if resourcePrivateEntry(f) != nil {
|
||||||
|
f.Close()
|
||||||
|
return nil, nil, errResourceState
|
||||||
|
}
|
||||||
|
b, err := io.ReadAll(io.LimitReader(f, 65537))
|
||||||
|
if err != nil || len(b) > 65536 || s.entryMatches(name, f) != nil {
|
||||||
|
f.Close()
|
||||||
|
return nil, nil, errResourceState
|
||||||
|
}
|
||||||
|
return f, b, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *resourceMonitorStore) trustedRead(name string, held *os.File) ([]byte, error) {
|
||||||
|
f, b, err := s.openEntry(name)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
if held == nil || s.entryMatches(name, held) != nil {
|
||||||
|
return nil, errResourceState
|
||||||
|
}
|
||||||
|
readInfo, err := f.Stat()
|
||||||
|
heldInfo, heldErr := held.Stat()
|
||||||
|
if err != nil || heldErr != nil || !os.SameFile(readInfo, heldInfo) {
|
||||||
|
return nil, errResourceState
|
||||||
|
}
|
||||||
|
return b, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Both content and held inode identities must still match the trusted snapshot.
|
||||||
|
// In particular, an identical signed replacement is not silently adopted.
|
||||||
|
func (s *resourceMonitorStore) snapshotGuard() error {
|
||||||
|
key, err := s.trustedRead(resourceKeyName, s.keyFile)
|
||||||
|
if err != nil || !hmac.Equal(key, s.key) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if s.existed {
|
||||||
|
b, err := s.trustedRead(resourceStateName, s.stateFile)
|
||||||
|
if err != nil || !bytes.Equal(b, s.previous) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if _, err := s.decode(b); err != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
f, _, err := s.openEntry(resourceStateName)
|
||||||
|
if f != nil {
|
||||||
|
f.Close()
|
||||||
|
}
|
||||||
|
if !errors.Is(err, os.ErrNotExist) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return s.guard()
|
||||||
|
}
|
||||||
|
func openResourceMonitorStore(path string, create bool) (*resourceMonitorStore, error) {
|
||||||
|
if managedDirectory(path) != nil {
|
||||||
|
return nil, errResourceState
|
||||||
|
}
|
||||||
|
fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errResourceState
|
||||||
|
}
|
||||||
|
s := &resourceMonitorStore{path: path, dir: os.NewFile(uintptr(fd), "resource state directory")}
|
||||||
|
fail := func() (*resourceMonitorStore, error) { s.close(); return nil, errResourceState }
|
||||||
|
stateFile, state, stateErr := s.openEntry(resourceStateName)
|
||||||
|
s.stateFile = stateFile
|
||||||
|
if stateErr != nil && !errors.Is(stateErr, os.ErrNotExist) {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
keyFile, key, err := s.openEntry(resourceKeyName)
|
||||||
|
s.keyFile = keyFile
|
||||||
|
if errors.Is(err, os.ErrNotExist) && errors.Is(stateErr, os.ErrNotExist) && create {
|
||||||
|
key = make([]byte, 32)
|
||||||
|
if _, err = rand.Read(key); err != nil {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
keyFD, err := syscall.Openat(fd, resourceKeyName, syscall.O_RDWR|syscall.O_CREAT|syscall.O_EXCL|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0600)
|
||||||
|
if err != nil {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
f := os.NewFile(uintptr(keyFD), "resource key")
|
||||||
|
s.keyFile = f
|
||||||
|
_, writeErr := f.Write(key)
|
||||||
|
syncErr := f.Sync()
|
||||||
|
if errors.Join(writeErr, syncErr, s.dir.Sync()) != nil {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
} else if err != nil {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
if len(key) != 32 {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
s.key = append([]byte(nil), key...)
|
||||||
|
s.previous, s.existed = state, stateErr == nil
|
||||||
|
if s.existed {
|
||||||
|
if _, err := s.decode(state); err != nil {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.snapshotGuard() != nil {
|
||||||
|
return fail()
|
||||||
|
}
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
func (s *resourceMonitorStore) decode(b []byte) (resourceMonitorState, error) {
|
||||||
|
var v resourceMonitorState
|
||||||
|
d := json.NewDecoder(bytes.NewReader(b))
|
||||||
|
if uniqueResourceJSON(d, 0) != nil {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
if _, err := d.Token(); err != io.EOF {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
d = json.NewDecoder(bytes.NewReader(b))
|
||||||
|
d.DisallowUnknownFields()
|
||||||
|
if d.Decode(&v) != nil || v.Version != 1 || v.Updated < 0 || len(v.Rows) > 64 || !hmac.Equal([]byte(v.MAC), []byte(resourceStateMAC(s.key, v))) {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
services := map[string]bool{}
|
||||||
|
for _, r := range v.Rows {
|
||||||
|
id := r.Service + "/" + r.Resource
|
||||||
|
fp, err := hex.DecodeString(r.Fingerprint)
|
||||||
|
if !resourceID.MatchString(r.Service) || !resourceID.MatchString(r.Resource) || seen[id] || err != nil || len(fp) != 32 || r.LastCompleted < 0 || r.NextDue < 0 {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
seen[id] = true
|
||||||
|
services[r.Service] = true
|
||||||
|
if len(services) > 16 || r.LastCompleted > v.Updated || r.LastCompleted > 253402300799999 || (r.Health == "UNKNOWN" && r.NextDue != 0) || (r.Health != "UNKNOWN" && (r.NextDue < r.LastCompleted || r.NextDue-r.LastCompleted > 86400000)) {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
if r.Health != "UNKNOWN" && r.Health != "HEALTHY" && r.Health != "DEGRADED" {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
switch r.Outcome {
|
||||||
|
case "unknown", "pass", "dns_failed", "destination_denied", "transfer_failed", "expectation_failed", "timeout", "cancelled":
|
||||||
|
default:
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
if (r.Health == "UNKNOWN" && (r.Outcome != "unknown" || r.LastCompleted != 0 || r.Failures != 0)) || (r.Health == "HEALTHY" && (r.Outcome != "pass" || r.Failures != 0)) || (r.Health == "DEGRADED" && (r.Outcome == "unknown" || r.Outcome == "pass" || r.Failures == 0)) {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return v, nil
|
||||||
|
}
|
||||||
|
func (s *resourceMonitorStore) load() (resourceMonitorState, error) {
|
||||||
|
if s.guard() != nil {
|
||||||
|
return resourceMonitorState{}, errResourceState
|
||||||
|
}
|
||||||
|
key, err := s.trustedRead(resourceKeyName, s.keyFile)
|
||||||
|
if err != nil || !hmac.Equal(key, s.key) {
|
||||||
|
return resourceMonitorState{}, errResourceState
|
||||||
|
}
|
||||||
|
b, err := s.trustedRead(resourceStateName, s.stateFile)
|
||||||
|
if errors.Is(err, os.ErrNotExist) && !s.existed {
|
||||||
|
return resourceMonitorState{Version: 1, Rows: []resourceMonitorRow{}}, nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return resourceMonitorState{}, errResourceState
|
||||||
|
}
|
||||||
|
v, err := s.decode(b)
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
if s.fault != nil {
|
||||||
|
s.fault("loaded", "")
|
||||||
|
}
|
||||||
|
// Authenticated operator repair on the original held inode can be reloaded;
|
||||||
|
// replaced files remain preserved/BLOCKED until deliberately repaired.
|
||||||
|
key, err = s.trustedRead(resourceKeyName, s.keyFile)
|
||||||
|
if err != nil || !hmac.Equal(key, s.key) || s.entryMatches(resourceStateName, s.stateFile) != nil {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
current, err := s.trustedRead(resourceStateName, s.stateFile)
|
||||||
|
if err != nil || !bytes.Equal(current, b) {
|
||||||
|
return v, errResourceState
|
||||||
|
}
|
||||||
|
s.previous = b
|
||||||
|
return v, nil
|
||||||
|
}
|
||||||
|
func (s *resourceMonitorStore) save(v resourceMonitorState) error {
|
||||||
|
if s.snapshotGuard() != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
v.MAC = resourceStateMAC(s.key, v)
|
||||||
|
b, err := json.Marshal(v)
|
||||||
|
if err != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if _, err := s.decode(b); err != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
// Keep the staged inode open from creation through publication; reopening a
|
||||||
|
// path after rename must never adopt a foreign replacement as our own inode.
|
||||||
|
f, err := os.CreateTemp(s.entryPath(""), ".resource-monitor-stage-")
|
||||||
|
if err != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
stage := f.Name()
|
||||||
|
stageGuard := func() bool {
|
||||||
|
if s.entryMatches(filepath.Base(stage), f) != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
contents, err := io.ReadAll(io.NewSectionReader(f, 0, 65537))
|
||||||
|
// Exact bytes bind the entire validated update, including its HMAC;
|
||||||
|
// truncation, append and in-place edits must fail before publication.
|
||||||
|
return err == nil && bytes.Equal(contents, b) && s.entryMatches(filepath.Base(stage), f) == nil
|
||||||
|
}
|
||||||
|
adopted := false
|
||||||
|
defer func() {
|
||||||
|
// Delete only the known staged inode, never an operator replacement.
|
||||||
|
info, err := os.Lstat(stage)
|
||||||
|
held, heldErr := f.Stat()
|
||||||
|
if err == nil && heldErr == nil && os.SameFile(info, held) && resourcePrivateEntry(f) == nil {
|
||||||
|
contents, readErr := io.ReadAll(io.NewSectionReader(f, 0, 65537))
|
||||||
|
if readErr == nil && bytes.Equal(contents, b) {
|
||||||
|
os.Remove(stage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !adopted {
|
||||||
|
f.Close()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
if _, err := f.Write(b); err != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if f.Sync() != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if s.fault != nil {
|
||||||
|
s.fault("staged", stage)
|
||||||
|
}
|
||||||
|
if s.snapshotGuard() != nil || !stageGuard() {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if s.fault != nil {
|
||||||
|
s.fault("publish", stage)
|
||||||
|
}
|
||||||
|
// Detect benign/operator replacements immediately before publication. An
|
||||||
|
// actively hostile same-UID actor racing between this final check and rename
|
||||||
|
// is outside the guarantee: POSIX rename has no inode compare-and-swap.
|
||||||
|
if s.existed {
|
||||||
|
if s.snapshotGuard() != nil || !stageGuard() {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if os.Rename(stage, s.entryPath(resourceStateName)) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Exclusive initial publication cannot clobber a newly appeared entry.
|
||||||
|
key, err := s.trustedRead(resourceKeyName, s.keyFile)
|
||||||
|
if err != nil || !hmac.Equal(key, s.key) || s.guard() != nil || !stageGuard() {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if os.Link(stage, s.entryPath(resourceStateName)) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
stageInfo, stageErr := os.Lstat(stage)
|
||||||
|
heldInfo, heldErr := f.Stat()
|
||||||
|
if stageErr != nil || heldErr != nil || !os.SameFile(stageInfo, heldInfo) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
if os.Remove(stage) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.entryMatches(resourceStateName, f) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
published, err := s.trustedRead(resourceStateName, f)
|
||||||
|
if err != nil || !bytes.Equal(published, b) {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
old := s.stateFile
|
||||||
|
s.stateFile, s.previous, s.existed, adopted = f, b, true, true
|
||||||
|
var closeErr error
|
||||||
|
if old != nil {
|
||||||
|
closeErr = old.Close()
|
||||||
|
}
|
||||||
|
if errors.Join(closeErr, s.dir.Sync()) != nil {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,541 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func resourceMonitorFixture(t *testing.T) (resourceMonitorSource, []byte) {
|
||||||
|
t.Helper()
|
||||||
|
key := bytes.Repeat([]byte{7}, 32)
|
||||||
|
path := resourceCheckFile(t, resourceCheckExample)
|
||||||
|
s, err := readResourceMonitorSource(path, resourceCheckLookup, key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return s, key
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorSchedule(t *testing.T) {
|
||||||
|
source, key := resourceMonitorFixture(t)
|
||||||
|
a, b, cooldown := 1000, 4000, 200
|
||||||
|
source.config.Services[0].Resources[0].IntervalMS = &a
|
||||||
|
source.config.Services[0].Resources[0].CooldownMS = &cooldown
|
||||||
|
source.config.Services[1].Resources[0].IntervalMS = &b
|
||||||
|
state := resourceMonitorState{Version: 1}
|
||||||
|
now := time.UnixMilli(10000)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
var visits []string
|
||||||
|
var elapsed []int64
|
||||||
|
d := resourceMonitorDeps{key: key, now: func() time.Time { return now }, source: func() (resourceMonitorSource, error) { return source, nil }, guard: func() error { return nil }, load: func() (resourceMonitorState, error) { return state, nil }, save: func(v resourceMonitorState) error { state = v; return nil }, report: func(bool) {}, wait: func(_ context.Context, d time.Duration) error { now = now.Add(d); return nil }}
|
||||||
|
d.probe = func(_ context.Context, service string, _ resourceCheckSpec, _ resourceMonitorSource, guard func() error) (string, error) {
|
||||||
|
if err := guard(); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
visits = append(visits, service)
|
||||||
|
elapsed = append(elapsed, now.UnixMilli()-10000)
|
||||||
|
if len(visits) == 8 {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
if len(visits) == 1 {
|
||||||
|
return "timeout", nil
|
||||||
|
}
|
||||||
|
return "pass", nil
|
||||||
|
}
|
||||||
|
if err := runResourceMonitor(ctx, d); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(visits, []string{"first", "second", "first", "first", "first", "first", "second", "first"}) || !reflect.DeepEqual(elapsed, []int64{0, 0, 200, 1200, 2200, 3200, 4000, 4200}) {
|
||||||
|
t.Fatalf("unfair/due schedule: %v %v", visits, elapsed)
|
||||||
|
}
|
||||||
|
if state.Rows[0].Health != "HEALTHY" || state.Rows[0].Failures != 0 || state.Rows[1].NextDue != 18000 {
|
||||||
|
t.Fatal(state)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorBlockedRepairAndPersistenceFailure(t *testing.T) {
|
||||||
|
for _, kind := range []string{"source", "ownership", "storage", "save"} {
|
||||||
|
t.Run(kind, func(t *testing.T) {
|
||||||
|
source, key := resourceMonitorFixture(t)
|
||||||
|
state := resourceMonitorState{Version: 1, MAC: "initial"}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
tick, probes := 0, 0
|
||||||
|
d := resourceMonitorDeps{key: key, now: func() time.Time { return time.UnixMilli(10000 + int64(tick)*1000) }, report: func(bool) {}, wait: func(_ context.Context, _ time.Duration) error {
|
||||||
|
tick++
|
||||||
|
if tick == 4 {
|
||||||
|
cancel()
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}, source: func() (resourceMonitorSource, error) {
|
||||||
|
if kind == "source" && tick < 2 {
|
||||||
|
return resourceMonitorSource{}, errResourceConfig
|
||||||
|
}
|
||||||
|
return source, nil
|
||||||
|
}, guard: func() error {
|
||||||
|
if kind == "ownership" && tick < 2 {
|
||||||
|
return errResourceConfig
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}, load: func() (resourceMonitorState, error) {
|
||||||
|
if kind == "storage" && tick < 2 {
|
||||||
|
return state, errResourceState
|
||||||
|
}
|
||||||
|
return state, nil
|
||||||
|
}, save: func(v resourceMonitorState) error {
|
||||||
|
if kind == "save" {
|
||||||
|
return errResourceState
|
||||||
|
}
|
||||||
|
state = v
|
||||||
|
return nil
|
||||||
|
}}
|
||||||
|
d.probe = func(_ context.Context, _ string, _ resourceCheckSpec, _ resourceMonitorSource, g func() error) (string, error) {
|
||||||
|
if err := g(); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
probes++
|
||||||
|
return "pass", nil
|
||||||
|
}
|
||||||
|
if err := runResourceMonitor(ctx, d); (err != nil) != (kind == "save") {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if kind == "save" {
|
||||||
|
if probes != 1 {
|
||||||
|
t.Fatalf("probed after persistence failure: %d", probes)
|
||||||
|
}
|
||||||
|
} else if probes != 2 {
|
||||||
|
t.Fatalf("did not resume safely: %d", probes)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorConfigReconcilePrivacy(t *testing.T) {
|
||||||
|
source, key := resourceMonitorFixture(t)
|
||||||
|
rows := reconcileResourceRows(key, source, nil)
|
||||||
|
rows[0].Health, rows[0].Outcome, rows[0].LastCompleted = "HEALTHY", "pass", 10000
|
||||||
|
rows[1].Health, rows[1].Outcome, rows[1].LastCompleted = "HEALTHY", "pass", 10000
|
||||||
|
old := rows[1].Fingerprint
|
||||||
|
source.config.Services[0].Resources[0].URL = "https://example.com/changed-secret"
|
||||||
|
changed := reconcileResourceRows(key, source, rows)
|
||||||
|
if changed[0].Health != "UNKNOWN" || changed[1].Health != "HEALTHY" || changed[1].Fingerprint != old {
|
||||||
|
t.Fatal(changed)
|
||||||
|
}
|
||||||
|
source.config.Services = source.config.Services[1:]
|
||||||
|
if got := reconcileResourceRows(key, source, changed); len(got) != 1 || got[0].Service != "second" {
|
||||||
|
t.Fatal(got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorStore(t *testing.T) {
|
||||||
|
source, key := resourceMonitorFixture(t)
|
||||||
|
dir := t.TempDir()
|
||||||
|
if err := os.Chmod(dir, 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s, err := openResourceMonitorStore(dir, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
key = s.key
|
||||||
|
state, err := s.load()
|
||||||
|
if err != nil || len(state.Rows) != 0 {
|
||||||
|
t.Fatal(state, err)
|
||||||
|
}
|
||||||
|
state.Updated = 10000
|
||||||
|
state.Rows = reconcileResourceRows(key, source, nil)
|
||||||
|
if err := s.save(state); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s.close()
|
||||||
|
s, err = openResourceMonitorStore(dir, false)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer s.close()
|
||||||
|
loaded, err := s.load()
|
||||||
|
if err != nil || !reflect.DeepEqual(loaded.Rows, state.Rows) {
|
||||||
|
t.Fatal(loaded, err)
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(filepath.Join(dir, resourceStateName))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, secret := range []string{"example.com", "private-token", "secret=value", "example.org", "Authorization"} {
|
||||||
|
if bytes.Contains(data, []byte(secret)) {
|
||||||
|
t.Fatal("secret serialized")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, name := range []string{resourceStateName, resourceKeyName} {
|
||||||
|
info, err := os.Stat(filepath.Join(dir, name))
|
||||||
|
if err != nil || info.Mode().Perm() != 0600 {
|
||||||
|
t.Fatal(info, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Corruption is preserved; restoring a valid signed version resumes loading.
|
||||||
|
bad := []byte(`{"version":99}`)
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, resourceStateName), bad, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.load(); err == nil {
|
||||||
|
t.Fatal("corrupt state trusted")
|
||||||
|
}
|
||||||
|
if err := s.save(state); err == nil {
|
||||||
|
t.Fatal("corrupt state overwritten")
|
||||||
|
}
|
||||||
|
got, _ := os.ReadFile(filepath.Join(dir, resourceStateName))
|
||||||
|
if !bytes.Equal(got, bad) {
|
||||||
|
t.Fatal("evidence changed")
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(dir, resourceStateName), data, 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.load(); err != nil {
|
||||||
|
t.Fatal("valid repair refused", err)
|
||||||
|
}
|
||||||
|
if err := os.Remove(filepath.Join(dir, resourceKeyName)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := openResourceMonitorStore(dir, true); err == nil {
|
||||||
|
t.Fatal("missing key silently regenerated")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorUnsafeStorePreserved(t *testing.T) {
|
||||||
|
for _, kind := range []string{"symlink", "hardlink", "permissions", "wrong-owner", "directory-replaced"} {
|
||||||
|
t.Run(kind, func(t *testing.T) {
|
||||||
|
dir := filepath.Join(t.TempDir(), "state")
|
||||||
|
if err := os.Mkdir(dir, 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s, err := openResourceMonitorStore(dir, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer s.close()
|
||||||
|
path := filepath.Join(dir, resourceKeyName)
|
||||||
|
original, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
switch kind {
|
||||||
|
case "symlink":
|
||||||
|
if err := os.Rename(path, path+"-original"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Symlink(path+"-original", path); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "hardlink":
|
||||||
|
if err := os.Link(path, path+"-alias"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "permissions":
|
||||||
|
if err := os.Chmod(path, 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "wrong-owner":
|
||||||
|
if os.Geteuid() != 0 {
|
||||||
|
t.Skip("requires chown capability")
|
||||||
|
}
|
||||||
|
if err := os.Chown(path, 12345, -1); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "directory-replaced":
|
||||||
|
if err := os.Rename(dir, dir+"-old"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Mkdir(dir, 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
path = filepath.Join(dir+"-old", resourceKeyName)
|
||||||
|
}
|
||||||
|
if _, err := s.load(); err == nil {
|
||||||
|
t.Fatal("unsafe storage accepted")
|
||||||
|
}
|
||||||
|
if err := s.save(resourceMonitorState{Version: 1}); err == nil {
|
||||||
|
t.Fatal("unsafe storage overwritten")
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil || !bytes.Equal(data, original) {
|
||||||
|
t.Fatal("key changed", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorReloadBeforeDNSAndTransfer(t *testing.T) {
|
||||||
|
for _, phase := range []string{"before-dns", "after-dns", "replace-after-dns", "env-after-dns", "owner-after-dns"} {
|
||||||
|
t.Run(phase, func(t *testing.T) {
|
||||||
|
key := bytes.Repeat([]byte{1}, 32)
|
||||||
|
path := resourceCheckFile(t, resourceCheckExample)
|
||||||
|
envChanged, ownerLost := false, false
|
||||||
|
lookup := func(k string) (string, bool) {
|
||||||
|
if envChanged && k == "CHECK_TOKEN" {
|
||||||
|
return "changed secret", true
|
||||||
|
}
|
||||||
|
return resourceCheckLookup(k)
|
||||||
|
}
|
||||||
|
source, err := readResourceMonitorSource(path, lookup, key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
transfers, dnsCalls := 0, 0
|
||||||
|
guard := func() error {
|
||||||
|
if ownerLost {
|
||||||
|
return errResourceConfig
|
||||||
|
}
|
||||||
|
fresh, err := readResourceMonitorSource(path, lookup, key)
|
||||||
|
if err != nil || fresh.fingerprint != source.fingerprint {
|
||||||
|
return errResourceConfig
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
mutate := func() {
|
||||||
|
switch phase {
|
||||||
|
case "env-after-dns":
|
||||||
|
envChanged = true
|
||||||
|
case "owner-after-dns":
|
||||||
|
ownerLost = true
|
||||||
|
case "replace-after-dns":
|
||||||
|
data := strings.Replace(resourceCheckExample, "secret=value", "secret=rotated", 1)
|
||||||
|
if err := os.WriteFile(path+"-next", []byte(data), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(path+"-next", path); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
if err := os.WriteFile(path, []byte(`{"version":1}`), 0600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if phase == "before-dns" {
|
||||||
|
mutate()
|
||||||
|
}
|
||||||
|
check := resourceCheckDeps{resolve: func(context.Context, string, string) ([]net.IP, error) {
|
||||||
|
dnsCalls++
|
||||||
|
mutate()
|
||||||
|
return []net.IP{net.ParseIP("93.184.216.34")}, nil
|
||||||
|
}, runner: func(context.Context, []string, io.Reader, io.Writer, io.Writer) error { transfers++; return nil }}
|
||||||
|
_, err = probeResourceMonitor(context.Background(), "first", source.config.Services[0].Resources[0], source, guard, check)
|
||||||
|
if err == nil || transfers != 0 || (phase == "before-dns" && dnsCalls != 0) {
|
||||||
|
t.Fatal("unsafe probe", transfers, dnsCalls, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorBothTLSAndCancellationJoin(t *testing.T) {
|
||||||
|
source, _ := resourceMonitorFixture(t)
|
||||||
|
spec := source.config.Services[1].Resources[0]
|
||||||
|
check := resourceCheckDeps{resolve: func(context.Context, string, string) ([]net.IP, error) {
|
||||||
|
return []net.IP{net.ParseIP("93.184.216.34")}, nil
|
||||||
|
}}
|
||||||
|
calls := 0
|
||||||
|
check.runner = func(_ context.Context, args []string, _ io.Reader, out, diag io.Writer) error {
|
||||||
|
calls++
|
||||||
|
protocol := "tls12"
|
||||||
|
mode := ""
|
||||||
|
if strings.Contains(strings.Join(args, " "), "--tlsv1.3") {
|
||||||
|
protocol = "tls13"
|
||||||
|
mode = "verify"
|
||||||
|
}
|
||||||
|
configuredWire(spec.URL, spec.Method, protocol, mode, out, diag)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
outcome, err := probeResourceMonitor(context.Background(), "second", spec, source, func() error { return nil }, check)
|
||||||
|
if err != nil || outcome == "pass" || calls != 2 {
|
||||||
|
t.Fatal(outcome, calls, err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
joined := false
|
||||||
|
check.runner = func(ctx context.Context, _ []string, _ io.Reader, _ io.Writer, _ io.Writer) error {
|
||||||
|
cancel()
|
||||||
|
<-ctx.Done()
|
||||||
|
joined = true
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
outcome, err = probeResourceMonitor(ctx, "second", spec, source, func() error { return nil }, check)
|
||||||
|
if err != nil || outcome != "cancelled" || !joined {
|
||||||
|
t.Fatal("did not join cancellation", outcome, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorSchemaScheduling(t *testing.T) {
|
||||||
|
for _, field := range []string{"interval_ms", "cooldown_ms"} {
|
||||||
|
for _, value := range []string{"0", "-1", "99", "86400001", "null", "\"1000\""} {
|
||||||
|
data := strings.Replace(resourceCheckExample, `"timeout_ms":1000`, `"timeout_ms":1000,"`+field+`":`+value, 1)
|
||||||
|
if _, err := parseResourceChecks([]byte(data), resourceCheckLookup); err == nil {
|
||||||
|
t.Fatal(field, value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorSingletonAndGenericDispatch(t *testing.T) {
|
||||||
|
d, env, _ := managedCLIFixture(t)
|
||||||
|
lock, err := acquireManagedMonitorSingleton(d)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer lock.close()
|
||||||
|
if _, err := acquireManagedMonitorSingleton(d); !errors.Is(err, syscall.EWOULDBLOCK) {
|
||||||
|
t.Fatal("singleton conflict ignored", err)
|
||||||
|
}
|
||||||
|
// Generic dispatch precedes invalid YouTube/adaptive settings. A cancelled
|
||||||
|
// generic startup needs no runtime/network, database or valid source JSON.
|
||||||
|
env["SOCKS_PORT"] = "1080"
|
||||||
|
env["ZAPRET_AUTOTUNE_SOCKS_PROXY"] = "socks5h://127.0.0.1:1080"
|
||||||
|
env["ZAPRET_AUTOTUNE_RESOURCE_CONFIG"] = "/missing-invalid-config"
|
||||||
|
env["ZAPRET_AUTOTUNE_VIDEO_ID"] = "bad"
|
||||||
|
env["ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE"] = "invalid"
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
var out, diag bytes.Buffer
|
||||||
|
code := managedMonitorCLIContext(ctx, nil, &out, &diag, d, func(context.Context, managedMonitorDeps) error { t.Fatal("YouTube supervisor invoked"); return nil })
|
||||||
|
if code != 0 || !strings.Contains(out.String(), "generic resource monitor") {
|
||||||
|
t.Fatal(code, out.String(), diag.String())
|
||||||
|
}
|
||||||
|
delete(env, "ZAPRET_AUTOTUNE_RESOURCE_CONFIG")
|
||||||
|
if code := managedMonitorCLIContext(ctx, nil, &out, &diag, d, nil); code != 64 {
|
||||||
|
t.Fatal("legacy invalid config no longer rejected", code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorInvalidInitialConfigAliveAndTERM(t *testing.T) {
|
||||||
|
d, env, _ := managedCLIFixture(t)
|
||||||
|
env["SOCKS_PORT"] = "1080"
|
||||||
|
env["ZAPRET_AUTOTUNE_SOCKS_PROXY"] = "socks5h://127.0.0.1:1080"
|
||||||
|
d.verifySOCKS = func(int) error { return nil }
|
||||||
|
initManagedCLI(t, d)
|
||||||
|
path := resourceCheckFile(t, `{"invalid":true}`)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
started := make(chan struct{}, 1)
|
||||||
|
check := resourceCheckDeps{guard: func() error {
|
||||||
|
select {
|
||||||
|
case started <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}, resolve: func(context.Context, string, string) ([]net.IP, error) {
|
||||||
|
t.Error("DNS with invalid source")
|
||||||
|
return nil, nil
|
||||||
|
}, runner: func(context.Context, []string, io.Reader, io.Writer, io.Writer) error {
|
||||||
|
t.Error("transfer with invalid source")
|
||||||
|
return nil
|
||||||
|
}}
|
||||||
|
var out, diag bytes.Buffer
|
||||||
|
done := make(chan int, 1)
|
||||||
|
go func() { done <- resourceMonitorCLIWithCheck(ctx, path, &out, &diag, d, check) }()
|
||||||
|
// Source error short-circuits the runtime guard, so wait for authenticated
|
||||||
|
// BLOCKED persistence rather than relying on a diagnostic log race.
|
||||||
|
deadline := time.Now().Add(2 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
s, err := openResourceMonitorStore(d.directory, false)
|
||||||
|
if err == nil {
|
||||||
|
state, err := s.load()
|
||||||
|
s.close()
|
||||||
|
if err == nil && state.Blocked {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case code := <-done:
|
||||||
|
t.Fatalf("invalid source terminated controller: %d", code)
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
select {
|
||||||
|
case code := <-done:
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatal(code, diag.String())
|
||||||
|
}
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("TERM did not join controller")
|
||||||
|
}
|
||||||
|
if !strings.Contains(diag.String(), "BLOCKED") {
|
||||||
|
t.Fatal("no blocked status", diag.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorConcurrentEnvironmentReload(t *testing.T) {
|
||||||
|
path := resourceCheckFile(t, resourceCheckExample)
|
||||||
|
t.Setenv("CHECK_URL", "https://example.org/private")
|
||||||
|
t.Setenv("CHECK_TOKEN", "initial-secret")
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://127.0.0.1:1080")
|
||||||
|
key := bytes.Repeat([]byte{3}, 32)
|
||||||
|
baseline, err := readResourceMonitorSource(path, os.LookupEnv, key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
defer close(done)
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
os.Setenv("CHECK_TOKEN", "rotated-secret")
|
||||||
|
os.Setenv("CHECK_TOKEN", "initial-secret")
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
s, err := readResourceMonitorSource(path, os.LookupEnv, key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if s.fingerprint == "" {
|
||||||
|
t.Fatal("empty fingerprint")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
<-done
|
||||||
|
if err := os.Setenv("CHECK_TOKEN", "rotated-secret"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
fresh, err := readResourceMonitorSource(path, os.LookupEnv, key)
|
||||||
|
if err != nil || fresh.fingerprint == baseline.fingerprint {
|
||||||
|
t.Fatal("secret rotation not bound", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResourceMonitorTargetConfigBoundary(t *testing.T) {
|
||||||
|
_, env, _ := managedCLIFixture(t)
|
||||||
|
for k, v := range env {
|
||||||
|
t.Setenv(k, v)
|
||||||
|
}
|
||||||
|
t.Setenv("SOCKS_PORT", "1080")
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://127.0.0.1:1080")
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_RESOURCE_CONFIG", resourceCheckFile(t, `{"invalid":true}`))
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_VIDEO_ID", "invalid")
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE", "invalid")
|
||||||
|
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 0 {
|
||||||
|
t.Fatal("bad resource JSON/YouTube blocked safe generic startup", code)
|
||||||
|
}
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://foreign.example:1080")
|
||||||
|
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 64 {
|
||||||
|
t.Fatal("bad proxy allowed", code)
|
||||||
|
}
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_SOCKS_PROXY", "socks5h://127.0.0.1:1080")
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_DNS_MODE", "invalid")
|
||||||
|
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 64 {
|
||||||
|
t.Fatal("bad DNS allowed", code)
|
||||||
|
}
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_DNS_MODE", "doh")
|
||||||
|
t.Setenv("ZAPRET_AUTOTUNE_TARGET_ROLE", "foreign")
|
||||||
|
if code := run([]string{"target-config-check"}, io.Discard, io.Discard); code != 64 {
|
||||||
|
t.Fatal("bad target role allowed", code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -59,21 +59,24 @@ assert " stop_grace_period: 240s\n" in compose
|
|||||||
assert 'bounded_wait "$monitor_pid" 45' in entrypoint
|
assert 'bounded_wait "$monitor_pid" 45' in entrypoint
|
||||||
assert '/bin/sh /opt/managed-target-supervision.sh --watchdog' in entrypoint
|
assert '/bin/sh /opt/managed-target-supervision.sh --watchdog' in entrypoint
|
||||||
assert 'setsid /bin/sh /opt/managed-target-supervision.sh --monitor' in entrypoint
|
assert 'setsid /bin/sh /opt/managed-target-supervision.sh --monitor' in entrypoint
|
||||||
assert entrypoint.index('if ! cleanup_session; then') < entrypoint.index('if timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile; then')
|
assert entrypoint.index('if ! cleanup_session; then') < entrypoint.index('if cleanup_command 45 /usr/local/bin/zapret-autotune managed reconcile; then')
|
||||||
assert 'COPY managed-target-supervision.sh /opt/managed-target-supervision.sh' in (directory.parents[1] / 'Dockerfile.target').read_text()
|
assert 'COPY managed-target-supervision.sh /opt/managed-target-supervision.sh' in (directory.parents[1] / 'Dockerfile.target').read_text()
|
||||||
assert '!managed-target-supervision.sh' in (directory.parents[1] / '.dockerignore').read_text()
|
assert '!managed-target-supervision.sh' in (directory.parents[1] / '.dockerignore').read_text()
|
||||||
assert '"$count" -lt 3' in entrypoint
|
assert '"$count" -ge 3' in entrypoint
|
||||||
|
assert 'BLOCKED: generic target restart budget; checks paused' in entrypoint
|
||||||
assert '"$((now - started))" -lt 21600' in entrypoint
|
assert '"$((now - started))" -lt 21600' in entrypoint
|
||||||
assert "backoff=15" in entrypoint
|
assert "backoff=15" in entrypoint
|
||||||
assert "systemctl" not in entrypoint and "docker " not in entrypoint
|
assert "systemctl" not in entrypoint and "docker " not in entrypoint
|
||||||
assert "timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 stop" in entrypoint
|
assert "cleanup_command 30 /opt/zapret2/init.d/sysv/zapret2 stop" in entrypoint
|
||||||
|
assert 'generic_cleanup_deadline=$(($(date +%s) + 80))' in entrypoint
|
||||||
|
assert 'timeout -k 1 "$cleanup_command_cap" "$@"' in entrypoint
|
||||||
for command, seconds in (("/usr/local/bin/zapret-autotune managed init", 45),
|
for command, seconds in (("/usr/local/bin/zapret-autotune managed init", 45),
|
||||||
("/usr/local/bin/zapret-autotune managed reconcile", 45),
|
("/usr/local/bin/zapret-autotune managed reconcile", 45),
|
||||||
("/opt/zapret2/init.d/sysv/zapret2 start", 30)):
|
("/opt/zapret2/init.d/sysv/zapret2 start", 30)):
|
||||||
assert f"timeout -k 5 {seconds} {command} || exit 78" in entrypoint
|
assert f"timeout -k 5 {seconds} {command} || return 78" in entrypoint
|
||||||
assert entrypoint.index("managed reconcile || exit 78") < entrypoint.index("zapret2 start || exit 78")
|
assert entrypoint.index("managed reconcile || return 78") < entrypoint.index("zapret2 start || return 78")
|
||||||
experimental = 'managed apply --googlevideo-experiment native-circular --googlevideo-scope experimental_googlevideo_suffix --googlevideo-request startup || exit 78'
|
experimental = 'managed apply --googlevideo-experiment native-circular --googlevideo-scope experimental_googlevideo_suffix --googlevideo-request startup || exit 78'
|
||||||
assert entrypoint.index('zapret2 start || exit 78') < entrypoint.index(experimental)
|
assert entrypoint.index('zapret2 start || return 78') < entrypoint.index(experimental)
|
||||||
assert entrypoint.index(experimental) < entrypoint.index('setsid /bin/sh /opt/managed-target-supervision.sh --monitor')
|
assert entrypoint.index(experimental) < entrypoint.index('setsid /bin/sh /opt/managed-target-supervision.sh --monitor')
|
||||||
assert entrypoint.index('managed experimental-request-check || exit 78') < entrypoint.index('managed init || exit 78')
|
assert entrypoint.index('managed experimental-request-check || return 78') < entrypoint.index('managed init || return 78')
|
||||||
assert 'rm "${ZAPRET_CONFIG%/*}/.autotune-experimental-suffix-request"' not in entrypoint
|
assert 'rm "${ZAPRET_CONFIG%/*}/.autotune-experimental-suffix-request"' not in entrypoint
|
||||||
@@ -67,10 +67,19 @@ if name == 'awk':
|
|||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
if name == 'date':
|
if name == 'date':
|
||||||
n = increment('clock')
|
n = increment('clock')
|
||||||
print(100000 + (n * 21601 if scenario == 'window' else n))
|
print(100000 + (n * 21601 if scenario == 'window' else n + (21601 if (root / 'budget-pause').exists() else 0)))
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
if name == 'sleep':
|
if name == 'sleep':
|
||||||
seconds = int(sys.argv[1])
|
seconds = int(sys.argv[1])
|
||||||
|
if scenario.startswith('generic-budget-') and seconds != 60:
|
||||||
|
os.execv(os.environ['REAL_SLEEP'], [os.environ['REAL_SLEEP'], *sys.argv[1:]])
|
||||||
|
if seconds == 60 and scenario in ('generic-ss', 'generic-unsafe', 'generic-budget-monitor', 'generic-budget-reconcile', 'generic-budget-ack'):
|
||||||
|
log('generic-pause')
|
||||||
|
(root / 'budget-pause').touch()
|
||||||
|
if scenario != 'generic-ss':
|
||||||
|
os.kill(os.getppid(), signal.SIGTERM)
|
||||||
|
time.sleep(.01)
|
||||||
|
sys.exit(0)
|
||||||
if seconds in (15, 30):
|
if seconds in (15, 30):
|
||||||
log('cooldown ' + str(seconds))
|
log('cooldown ' + str(seconds))
|
||||||
if scenario == 'cancel':
|
if scenario == 'cancel':
|
||||||
@@ -96,9 +105,13 @@ if name in ('ss-server', 'ss-local'):
|
|||||||
os.kill(int((root / 'dns-pid').read_text()), 0)
|
os.kill(int((root / 'dns-pid').read_text()), 0)
|
||||||
assert (pathlib.Path(sys.argv[2]).stat().st_mode & 0o777) == 0o600
|
assert (pathlib.Path(sys.argv[2]).stat().st_mode & 0o777) == 0o600
|
||||||
log(name)
|
log(name)
|
||||||
if name == 'ss-local' and scenario in ('ss', 'mixed', 'joinfail') and increment('ss_runs') == 1:
|
if scenario.startswith('generic-budget-'):
|
||||||
|
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||||
|
while True:
|
||||||
|
time.sleep(.01)
|
||||||
|
if name == 'ss-local' and scenario in ('ss', 'mixed', 'joinfail', 'generic-ss') and increment('ss_runs') <= (3 if scenario == 'generic-ss' else 1):
|
||||||
# Let startup/monitor begin before simulating a child crash.
|
# Let startup/monitor begin before simulating a child crash.
|
||||||
while not (root / 'monitor_runs').exists():
|
while not (root / 'monitor_runs').exists() or (scenario == 'generic-ss' and int((root / 'monitor_runs').read_text()) < int((root / 'ss_runs').read_text())):
|
||||||
time.sleep(.01)
|
time.sleep(.01)
|
||||||
time.sleep(.03)
|
time.sleep(.03)
|
||||||
sys.exit(137)
|
sys.exit(137)
|
||||||
@@ -161,6 +174,8 @@ if sys.argv[1] == 'dns-forwarder':
|
|||||||
ready.unlink()
|
ready.unlink()
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
signal.signal(signal.SIGTERM, stop_dns)
|
signal.signal(signal.SIGTERM, stop_dns)
|
||||||
|
if scenario.startswith('generic-budget-'):
|
||||||
|
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||||
if scenario in ('dns-term', 'dns-int'):
|
if scenario in ('dns-term', 'dns-int'):
|
||||||
os.kill(os.getppid(), signal.SIGTERM if scenario == 'dns-term' else signal.SIGINT)
|
os.kill(os.getppid(), signal.SIGTERM if scenario == 'dns-term' else signal.SIGINT)
|
||||||
if scenario in ('dns-death', 'dns-killed', 'dns-replace', 'dns-replace-symlink') and increment('dns_runs') == 1:
|
if scenario in ('dns-death', 'dns-killed', 'dns-replace', 'dns-replace-symlink') and increment('dns_runs') == 1:
|
||||||
@@ -186,9 +201,15 @@ if command == 'init':
|
|||||||
elif command == 'reconcile':
|
elif command == 'reconcile':
|
||||||
n = increment('reconciles')
|
n = increment('reconciles')
|
||||||
log('reconcile')
|
log('reconcile')
|
||||||
|
if scenario.startswith('generic-budget-') and n == 2:
|
||||||
|
with (root / 'children').open('a') as f:
|
||||||
|
f.write(str(os.getpid()) + '\n')
|
||||||
|
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||||
|
while True:
|
||||||
|
time.sleep(.01)
|
||||||
if scenario == 'orphans':
|
if scenario == 'orphans':
|
||||||
assert_workers_gone()
|
assert_workers_gone()
|
||||||
if scenario == 'prestartfail':
|
if scenario in ('prestartfail', 'generic-unsafe'):
|
||||||
sys.exit(78)
|
sys.exit(78)
|
||||||
if scenario == 'dns-prestart':
|
if scenario == 'dns-prestart':
|
||||||
os.kill(int((root / 'dns-pid').read_text()), signal.SIGKILL)
|
os.kill(int((root / 'dns-pid').read_text()), signal.SIGKILL)
|
||||||
@@ -197,11 +218,23 @@ elif command == 'reconcile':
|
|||||||
sys.exit(78)
|
sys.exit(78)
|
||||||
(root / 'prepared').unlink(missing_ok=True)
|
(root / 'prepared').unlink(missing_ok=True)
|
||||||
elif command == 'monitor':
|
elif command == 'monitor':
|
||||||
|
if scenario.startswith('generic-'):
|
||||||
|
assert os.environ['ZAPRET_AUTOTUNE_RESOURCE_CONFIG'] == '/operator/resources.json'
|
||||||
with (root / 'children').open('a') as f:
|
with (root / 'children').open('a') as f:
|
||||||
f.write(str(os.getpid()) + '\n')
|
f.write(str(os.getpid()) + '\n')
|
||||||
n = increment('monitor_runs')
|
n = increment('monitor_runs')
|
||||||
log('monitor ' + str(n))
|
log('monitor ' + str(n))
|
||||||
assert os.getsid(0) == os.getpid(), 'monitor not in its own verified session'
|
assert os.getsid(0) == os.getpid(), 'monitor not in its own verified session'
|
||||||
|
if scenario.startswith('generic-budget-'):
|
||||||
|
time.sleep(.03)
|
||||||
|
os.kill(os.getppid(), signal.SIGTERM)
|
||||||
|
if scenario == 'generic-budget-monitor':
|
||||||
|
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||||
|
while True:
|
||||||
|
time.sleep(.01)
|
||||||
|
linger()
|
||||||
|
if scenario == 'generic-ss' and n <= 3:
|
||||||
|
linger()
|
||||||
if scenario in ('dns-death', 'dns-killed', 'dns-replace', 'dns-replace-symlink') and n == 1:
|
if scenario in ('dns-death', 'dns-killed', 'dns-replace', 'dns-replace-symlink') and n == 1:
|
||||||
linger()
|
linger()
|
||||||
if scenario == 'dns-outage':
|
if scenario == 'dns-outage':
|
||||||
@@ -245,6 +278,10 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
fake.chmod(0o755)
|
fake.chmod(0o755)
|
||||||
for name in ('awk', 'date', 'sleep', 'timeout', 'ss-server', 'ss-local', 'autotune', 'zapret2', 'blockcheck', 'nfq'):
|
for name in ('awk', 'date', 'sleep', 'timeout', 'ss-server', 'ss-local', 'autotune', 'zapret2', 'blockcheck', 'nfq'):
|
||||||
(binary / name).symlink_to(fake)
|
(binary / name).symlink_to(fake)
|
||||||
|
realtime = scenario.startswith('generic-budget-')
|
||||||
|
if realtime:
|
||||||
|
for name in ('date', 'timeout'):
|
||||||
|
(binary / name).unlink()
|
||||||
configs = root / 'private'
|
configs = root / 'private'
|
||||||
configs.mkdir()
|
configs.mkdir()
|
||||||
if scenario.startswith('dns-stale'):
|
if scenario.startswith('dns-stale'):
|
||||||
@@ -258,15 +295,28 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
broken_helper = root / 'broken-helper.sh'
|
broken_helper = root / 'broken-helper.sh'
|
||||||
broken_helper.write_text(helper.read_text().replace('sleep "$seconds" & sleeper=$!', 'exit 78'))
|
broken_helper.write_text(helper.read_text().replace('sleep "$seconds" & sleeper=$!', 'exit 78'))
|
||||||
helper = broken_helper
|
helper = broken_helper
|
||||||
|
if scenario == 'generic-budget-ack':
|
||||||
|
delayed_helper = root / 'delayed-helper.sh'
|
||||||
|
delayed_helper.write_text(helper.read_text().replace(': > "$config_dir/join-ready"',
|
||||||
|
f'{shutil.which("sleep")} 300\n : > "$config_dir/join-ready"'))
|
||||||
|
helper = delayed_helper
|
||||||
text = text.replace('/opt/managed-target-supervision.sh', str(helper))
|
text = text.replace('/opt/managed-target-supervision.sh', str(helper))
|
||||||
text = text.replace('config_dir=/run/zapret-autotune-ss', f'config_dir={configs}')
|
text = text.replace('config_dir=/run/zapret-autotune-ss', f'config_dir={configs}')
|
||||||
text = text.replace('/usr/local/bin/zapret-autotune', str(binary / 'autotune'))
|
text = text.replace('/usr/local/bin/zapret-autotune', str(binary / 'autotune'))
|
||||||
text = text.replace('/opt/zapret2/init.d/sysv/zapret2', str(binary / 'zapret2'))
|
text = text.replace('/opt/zapret2/init.d/sysv/zapret2', str(binary / 'zapret2'))
|
||||||
|
if realtime:
|
||||||
|
# Private fixture scaling only: production has no configurable grace.
|
||||||
|
text = text.replace('$(date +%s) + 80', '$(date +%s) + 12')
|
||||||
entry = root / 'entry.sh'
|
entry = root / 'entry.sh'
|
||||||
entry.write_text(text)
|
entry.write_text(text)
|
||||||
env = dict(os.environ, PATH=f'{binary}:{os.environ["PATH"]}', FIXTURE=tmp,
|
env = dict(os.environ, PATH=f'{binary}:{os.environ["PATH"]}', FIXTURE=tmp,
|
||||||
SCENARIO=scenario, SS_PASSWORD='ab' * 32, SS_DNS_SERVER=dns,
|
SCENARIO=scenario, SS_PASSWORD='ab' * 32, SS_DNS_SERVER=dns,
|
||||||
ZAPRET_CONFIG=str(root / 'config'), DNS_CLI=str(DNS_CLI))
|
ZAPRET_CONFIG=str(root / 'config'), DNS_CLI=str(DNS_CLI),
|
||||||
|
REAL_SLEEP=shutil.which('sleep'))
|
||||||
|
if scenario.startswith('generic-'):
|
||||||
|
env['ZAPRET_AUTOTUNE_RESOURCE_CONFIG'] = '/operator/resources.json'
|
||||||
|
else:
|
||||||
|
env.pop('ZAPRET_AUTOTUNE_RESOURCE_CONFIG', None)
|
||||||
if mode is None:
|
if mode is None:
|
||||||
env.pop('ZAPRET_AUTOTUNE_DNS_MODE', None)
|
env.pop('ZAPRET_AUTOTUNE_DNS_MODE', None)
|
||||||
else:
|
else:
|
||||||
@@ -279,7 +329,10 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
peer = subprocess.Popen([shutil.which('sleep'), '300'], start_new_session=True)
|
peer = subprocess.Popen([shutil.which('sleep'), '300'], start_new_session=True)
|
||||||
(root / 'peer').write_text(str(peer.pid))
|
(root / 'peer').write_text(str(peer.pid))
|
||||||
try:
|
try:
|
||||||
result = subprocess.run(['sh', str(entry)], env=env, capture_output=True, timeout=15)
|
started = time.monotonic()
|
||||||
|
result = subprocess.run(['sh', str(entry)], env=env, capture_output=True, timeout=18 if realtime else 15)
|
||||||
|
if realtime:
|
||||||
|
assert time.monotonic() - started < 16, (scenario, 'aggregate cleanup exceeded grace')
|
||||||
if peer:
|
if peer:
|
||||||
assert peer.poll() is None, 'unowned helper was killed'
|
assert peer.poll() is None, 'unowned helper was killed'
|
||||||
finally:
|
finally:
|
||||||
@@ -311,6 +364,9 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
assert ready.exists() or ready.is_symlink(), (scenario, 'foreign readiness deleted')
|
assert ready.exists() or ready.is_symlink(), (scenario, 'foreign readiness deleted')
|
||||||
if scenario != 'dns-stale-symlink' and scenario != 'dns-symlink':
|
if scenario != 'dns-stale-symlink' and scenario != 'dns-symlink':
|
||||||
assert ready.read_text() == 'foreign evidence\n', (scenario, 'foreign bytes changed')
|
assert ready.read_text() == 'foreign evidence\n', (scenario, 'foreign bytes changed')
|
||||||
|
elif realtime and (configs / 'dns-ready').exists():
|
||||||
|
# Exhaustion cannot authorize unlinking without the owned Go helper.
|
||||||
|
assert (configs / 'dns-ready').read_text().strip() == (root / 'dns-pid').read_text()
|
||||||
else:
|
else:
|
||||||
assert not (configs / 'dns-ready').exists(), (scenario, 'stale DNS readiness')
|
assert not (configs / 'dns-ready').exists(), (scenario, 'stale DNS readiness')
|
||||||
if mode == 'native':
|
if mode == 'native':
|
||||||
@@ -339,6 +395,15 @@ events = run('cancel', 0)
|
|||||||
assert events.count('start') == 1
|
assert events.count('start') == 1
|
||||||
events = run('prestartfail', 78)
|
events = run('prestartfail', 78)
|
||||||
assert 'start' not in events and 'stop' not in events
|
assert 'start' not in events and 'stop' not in events
|
||||||
|
events = run('generic-ss', 0, mode='doh')
|
||||||
|
assert events.count('start') == 4 and events.count('stop') == 4, events
|
||||||
|
assert events.count('generic-pause') == 1, events
|
||||||
|
assert events.count('dns-start') == events.count('dns-stop') == 4, events
|
||||||
|
events = run('generic-unsafe', 0)
|
||||||
|
assert 'start' not in events and events.count('generic-pause') == 1, events
|
||||||
|
for scenario in ('generic-budget-monitor', 'generic-budget-reconcile', 'generic-budget-ack'):
|
||||||
|
events = run(scenario, 78, mode='doh')
|
||||||
|
assert events.count('start') == 1 and 'stop' not in events, (scenario, events)
|
||||||
for dns in ('8.8.8.8', '2001:4860:4860::8888'):
|
for dns in ('8.8.8.8', '2001:4860:4860::8888'):
|
||||||
events = run('fatal64', 64, dns)
|
events = run('fatal64', 64, dns)
|
||||||
assert events.count('ss-server') == 1 and events.count('ss-local') == 1
|
assert events.count('ss-server') == 1 and events.count('ss-local') == 1
|
||||||
@@ -384,6 +449,7 @@ def check_watchdog_traps(shell):
|
|||||||
mode=$1
|
mode=$1
|
||||||
shift
|
shift
|
||||||
config_dir={directory}
|
config_dir={directory}
|
||||||
|
generic_mode=0
|
||||||
. {ROOT / 'managed-target-supervision.sh'}
|
. {ROOT / 'managed-target-supervision.sh'}
|
||||||
{bounded_wait}
|
{bounded_wait}
|
||||||
cleanup() {{
|
cleanup() {{
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ services:
|
|||||||
ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS: ${ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS-}
|
ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS: ${ZAPRET_AUTOTUNE_DOH_BOOTSTRAP_IPS-}
|
||||||
ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC: ${ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC-5}
|
ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC: ${ZAPRET_AUTOTUNE_DOH_TIMEOUT_SEC-5}
|
||||||
ZAPRET_AUTOTUNE_YOUTUBE_SERVICE: ${ZAPRET_AUTOTUNE_YOUTUBE_SERVICE-0}
|
ZAPRET_AUTOTUNE_YOUTUBE_SERVICE: ${ZAPRET_AUTOTUNE_YOUTUBE_SERVICE-0}
|
||||||
|
ZAPRET_AUTOTUNE_RESOURCE_CONFIG: ${ZAPRET_AUTOTUNE_RESOURCE_CONFIG-}
|
||||||
ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE: ${ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE-cold}
|
ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE: ${ZAPRET_AUTOTUNE_GOOGLEVIDEO_MODE-cold}
|
||||||
ZAPRET_AUTOTUNE_VIDEO_ID: ${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}
|
ZAPRET_AUTOTUNE_VIDEO_ID: ${ZAPRET_AUTOTUNE_VIDEO_ID-dQw4w9WgXcQ}
|
||||||
ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS:-}'
|
ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS: '${ZAPRET_AUTOTUNE_GOOGLEVIDEO_VIDEO_IDS:-}'
|
||||||
|
|||||||
+134
-15
@@ -16,6 +16,8 @@ esac
|
|||||||
/usr/local/bin/zapret-autotune target-config-check || exit 64
|
/usr/local/bin/zapret-autotune target-config-check || exit 64
|
||||||
dns_mode=$(/usr/local/bin/zapret-autotune dns-forwarder mode) || exit 64
|
dns_mode=$(/usr/local/bin/zapret-autotune dns-forwarder mode) || exit 64
|
||||||
dns_address=$(/usr/local/bin/zapret-autotune dns-forwarder address) || exit 64
|
dns_address=$(/usr/local/bin/zapret-autotune dns-forwarder address) || exit 64
|
||||||
|
generic_mode=0
|
||||||
|
[ -z "${ZAPRET_AUTOTUNE_RESOURCE_CONFIG-}" ] || generic_mode=1
|
||||||
case "$dns_mode" in doh|native) ;; *) exit 64 ;; esac
|
case "$dns_mode" in doh|native) ;; *) exit 64 ;; esac
|
||||||
config_dir=/run/zapret-autotune-ss
|
config_dir=/run/zapret-autotune-ss
|
||||||
awk '$2 == "/run/zapret-autotune-ss" && $3 == "tmpfs" { found=1 } END { exit !found }' /proc/mounts || {
|
awk '$2 == "/run/zapret-autotune-ss" && $3 == "tmpfs" { found=1 } END { exit !found }' /proc/mounts || {
|
||||||
@@ -40,10 +42,45 @@ monitor_pid=
|
|||||||
monitor_sid=
|
monitor_sid=
|
||||||
zapret_started=0
|
zapret_started=0
|
||||||
cooldown_pid=
|
cooldown_pid=
|
||||||
|
generic_cleanup_deadline=
|
||||||
|
generic_cleanup_failed=0
|
||||||
# Signal only descendants of an owned child, never the container/host namespace.
|
# Signal only descendants of an owned child, never the container/host namespace.
|
||||||
# The monitor gets TERM itself first so it can join scans and restore bytes;
|
# The monitor gets TERM itself first so it can join scans and restore bytes;
|
||||||
# descendant signalling is reserved for SS shutdown and forced watchdog cleanup.
|
# descendant signalling is reserved for SS shutdown and forced watchdog cleanup.
|
||||||
. /opt/managed-target-supervision.sh
|
. /opt/managed-target-supervision.sh
|
||||||
|
# One budget per cleanup cycle, including ACKs, kill escalation and helper joins.
|
||||||
|
# Ten seconds of Docker's 90s grace remain for coarse clocks and final reaping.
|
||||||
|
cleanup_remaining() {
|
||||||
|
cleanup_left=$((generic_cleanup_deadline - $(date +%s)))
|
||||||
|
[ "$cleanup_left" -ge 0 ] || cleanup_left=0
|
||||||
|
}
|
||||||
|
cleanup_command() {
|
||||||
|
if [ "$generic_mode" -eq 0 ]; then
|
||||||
|
timeout -k 5 "$@"
|
||||||
|
return $?
|
||||||
|
fi
|
||||||
|
cleanup_command_cap=$1
|
||||||
|
shift
|
||||||
|
cleanup_remaining
|
||||||
|
# timeout's KILL grace is part of, not additional to, the shared budget.
|
||||||
|
[ "$cleanup_left" -gt 1 ] || return 78
|
||||||
|
cleanup_command_left=$((cleanup_left - 1))
|
||||||
|
[ "$cleanup_command_cap" -le "$cleanup_command_left" ] || cleanup_command_cap=$cleanup_command_left
|
||||||
|
timeout -k 1 "$cleanup_command_cap" "$@"
|
||||||
|
}
|
||||||
|
cleanup_reap() {
|
||||||
|
reap_pid=$1
|
||||||
|
reap_deadline=$2
|
||||||
|
[ "$reap_deadline" -le "$((generic_cleanup_deadline + 2))" ] || reap_deadline=$((generic_cleanup_deadline + 2))
|
||||||
|
child_joined=0
|
||||||
|
while proc_identity "$reap_pid" && [ "$proc_state" != Z ]; do
|
||||||
|
[ "$(date +%s)" -lt "$reap_deadline" ] || return 0
|
||||||
|
pause 1
|
||||||
|
done
|
||||||
|
waited_code=0
|
||||||
|
wait "$reap_pid" 2>/dev/null || waited_code=$?
|
||||||
|
child_joined=1
|
||||||
|
}
|
||||||
pause() {
|
pause() {
|
||||||
sleep "$1" 8<&- 9<&- </dev/null >/dev/null 2>&1 & cooldown_pid=$!
|
sleep "$1" 8<&- 9<&- </dev/null >/dev/null 2>&1 & cooldown_pid=$!
|
||||||
wait "$cooldown_pid"
|
wait "$cooldown_pid"
|
||||||
@@ -57,6 +94,9 @@ cleanup_session() {
|
|||||||
for session_signal in TERM KILL; do
|
for session_signal in TERM KILL; do
|
||||||
signal_session "$monitor_sid" "$session_signal" || return 78
|
signal_session "$monitor_sid" "$session_signal" || return 78
|
||||||
session_deadline=$(($(date +%s) + 3))
|
session_deadline=$(($(date +%s) + 3))
|
||||||
|
if [ "$generic_mode" -eq 1 ] && [ "$session_deadline" -gt "$generic_cleanup_deadline" ]; then
|
||||||
|
session_deadline=$generic_cleanup_deadline
|
||||||
|
fi
|
||||||
while :; do
|
while :; do
|
||||||
signal_session "$monitor_sid" CHECK || return 78
|
signal_session "$monitor_sid" CHECK || return 78
|
||||||
if [ "$session_live" -eq 0 ]; then
|
if [ "$session_live" -eq 0 ]; then
|
||||||
@@ -85,6 +125,19 @@ capture_monitor_session() {
|
|||||||
bounded_wait() {
|
bounded_wait() {
|
||||||
child=$1
|
child=$1
|
||||||
seconds=$2
|
seconds=$2
|
||||||
|
if [ "$generic_mode" -eq 1 ]; then
|
||||||
|
cleanup_remaining
|
||||||
|
# Reserve ACK setup (5s) and post-KILL reaping (2s) before granting wait.
|
||||||
|
if [ "$cleanup_left" -le 7 ]; then
|
||||||
|
signal_tree "$child" KILL
|
||||||
|
cleanup_reap "$child" "$(($(date +%s) + 2))"
|
||||||
|
waited_code=${waited_code-137}
|
||||||
|
join_timed_out=1
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
cleanup_wait_left=$((cleanup_left - 7))
|
||||||
|
[ "$seconds" -le "$cleanup_wait_left" ] || seconds=$cleanup_wait_left
|
||||||
|
fi
|
||||||
rm -f "$config_dir/join-timeout" "$config_dir/join-ready"
|
rm -f "$config_dir/join-timeout" "$config_dir/join-ready"
|
||||||
# Fresh interpreter: a forked shell can inherit a deferred EXIT/TERM trap
|
# Fresh interpreter: a forked shell can inherit a deferred EXIT/TERM trap
|
||||||
# context and never run its cancellation handler during PID 1 shutdown.
|
# context and never run its cancellation handler during PID 1 shutdown.
|
||||||
@@ -93,6 +146,9 @@ bounded_wait() {
|
|||||||
# A fast-exiting child must not cancel the watchdog before its trap exists.
|
# A fast-exiting child must not cancel the watchdog before its trap exists.
|
||||||
# Fail closed if exec/setup fails before acknowledgement; never wait forever.
|
# Fail closed if exec/setup fails before acknowledgement; never wait forever.
|
||||||
ack_deadline=$(($(date +%s) + 5))
|
ack_deadline=$(($(date +%s) + 5))
|
||||||
|
if [ "$generic_mode" -eq 1 ] && [ "$ack_deadline" -gt "$generic_cleanup_deadline" ]; then
|
||||||
|
ack_deadline=$generic_cleanup_deadline
|
||||||
|
fi
|
||||||
while [ ! -e "$config_dir/join-ready" ]; do
|
while [ ! -e "$config_dir/join-ready" ]; do
|
||||||
if ! kill -0 "$watchdog" 2>/dev/null || [ "$(date +%s)" -ge "$ack_deadline" ]; then
|
if ! kill -0 "$watchdog" 2>/dev/null || [ "$(date +%s)" -ge "$ack_deadline" ]; then
|
||||||
: > "$config_dir/join-timeout"
|
: > "$config_dir/join-timeout"
|
||||||
@@ -102,13 +158,39 @@ bounded_wait() {
|
|||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
waited_code=0
|
waited_code=0
|
||||||
|
if [ "$generic_mode" -eq 1 ]; then
|
||||||
|
cleanup_join_deadline=$(($(date +%s) + seconds))
|
||||||
|
[ "$cleanup_join_deadline" -le "$generic_cleanup_deadline" ] || cleanup_join_deadline=$generic_cleanup_deadline
|
||||||
|
cleanup_reap "$child" "$cleanup_join_deadline"
|
||||||
|
if [ "$child_joined" -eq 0 ]; then
|
||||||
|
: > "$config_dir/join-timeout"
|
||||||
|
signal_tree "$child" KILL
|
||||||
|
cleanup_reap "$child" "$(($(date +%s) + 2))"
|
||||||
|
fi
|
||||||
|
joined_service=$child_joined
|
||||||
|
else
|
||||||
wait "$child" 2>/dev/null || waited_code=$?
|
wait "$child" 2>/dev/null || waited_code=$?
|
||||||
|
fi
|
||||||
kill -TERM "$watchdog" 2>/dev/null || true
|
kill -TERM "$watchdog" 2>/dev/null || true
|
||||||
|
if [ "$generic_mode" -eq 1 ]; then
|
||||||
|
service_waited_code=$waited_code
|
||||||
|
cleanup_reap "$watchdog" "$(($(date +%s) + 2))"
|
||||||
|
if [ "$child_joined" -eq 0 ]; then
|
||||||
|
signal_tree "$watchdog" KILL
|
||||||
|
cleanup_reap "$watchdog" "$(($(date +%s) + 2))"
|
||||||
|
fi
|
||||||
|
child_joined=$joined_service
|
||||||
|
waited_code=$service_waited_code
|
||||||
|
else
|
||||||
wait "$watchdog" 2>/dev/null || true
|
wait "$watchdog" 2>/dev/null || true
|
||||||
|
fi
|
||||||
join_timed_out=0
|
join_timed_out=0
|
||||||
[ ! -e "$config_dir/join-timeout" ] || join_timed_out=1
|
[ ! -e "$config_dir/join-timeout" ] || join_timed_out=1
|
||||||
}
|
}
|
||||||
cleanup_cycle() {
|
cleanup_cycle() {
|
||||||
|
if [ "$generic_mode" -eq 1 ] && [ -z "$generic_cleanup_deadline" ]; then
|
||||||
|
generic_cleanup_deadline=$(($(date +%s) + 80))
|
||||||
|
fi
|
||||||
cleanup_code=0
|
cleanup_code=0
|
||||||
# Let the monitor cancel/join scans and restore committed bytes BEFORE stop.
|
# Let the monitor cancel/join scans and restore committed bytes BEFORE stop.
|
||||||
if [ -n "$monitor_pid" ]; then
|
if [ -n "$monitor_pid" ]; then
|
||||||
@@ -133,8 +215,8 @@ cleanup_cycle() {
|
|||||||
# Stop also sources config: clear trust before reconciliation, never
|
# Stop also sources config: clear trust before reconciliation, never
|
||||||
# source unknown bytes left by a crashed monitor, even for shutdown.
|
# source unknown bytes left by a crashed monitor, even for shutdown.
|
||||||
zapret_started=0
|
zapret_started=0
|
||||||
if timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile; then
|
if cleanup_command 45 /usr/local/bin/zapret-autotune managed reconcile; then
|
||||||
timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 stop || cleanup_code=78
|
cleanup_command 30 /opt/zapret2/init.d/sysv/zapret2 stop || cleanup_code=78
|
||||||
else
|
else
|
||||||
cleanup_code=78
|
cleanup_code=78
|
||||||
fi
|
fi
|
||||||
@@ -142,9 +224,11 @@ cleanup_cycle() {
|
|||||||
for pid in "$local_pid" "$server_pid"; do
|
for pid in "$local_pid" "$server_pid"; do
|
||||||
[ -z "$pid" ] || signal_tree "$pid" TERM
|
[ -z "$pid" ] || signal_tree "$pid" TERM
|
||||||
done
|
done
|
||||||
|
ss_joined=1
|
||||||
for pid in "$local_pid" "$server_pid"; do
|
for pid in "$local_pid" "$server_pid"; do
|
||||||
if [ -n "$pid" ]; then
|
if [ -n "$pid" ]; then
|
||||||
bounded_wait "$pid" 5
|
bounded_wait "$pid" 5
|
||||||
|
if [ "$generic_mode" -eq 1 ] && [ "$child_joined" -eq 0 ]; then ss_joined=0; fi
|
||||||
# Its original failure triggered supervision; only forced shutdown
|
# Its original failure triggered supervision; only forced shutdown
|
||||||
# is a cleanup failure, not the already-dead Shadowsocks status.
|
# is a cleanup failure, not the already-dead Shadowsocks status.
|
||||||
[ "$join_timed_out" -eq 0 ] || { echo 'error: Shadowsocks join timeout' >&2; cleanup_code=78; }
|
[ "$join_timed_out" -eq 0 ] || { echo 'error: Shadowsocks join timeout' >&2; cleanup_code=78; }
|
||||||
@@ -153,15 +237,27 @@ cleanup_cycle() {
|
|||||||
server_pid=
|
server_pid=
|
||||||
local_pid=
|
local_pid=
|
||||||
# DNS remains available through monitor rollback, zapret stop and SS joins.
|
# DNS remains available through monitor rollback, zapret stop and SS joins.
|
||||||
if [ -n "$dns_pid" ]; then
|
if [ -n "$dns_pid" ] && [ "$ss_joined" -eq 1 ]; then
|
||||||
kill -TERM "$dns_pid" 2>/dev/null || true
|
kill -TERM "$dns_pid" 2>/dev/null || true
|
||||||
bounded_wait "$dns_pid" 5
|
bounded_wait "$dns_pid" 5
|
||||||
[ "$join_timed_out" -eq 0 ] || { echo 'error: DNS join timeout' >&2; cleanup_code=78; }
|
[ "$join_timed_out" -eq 0 ] || { echo 'error: DNS join timeout' >&2; cleanup_code=78; }
|
||||||
|
if [ "$generic_mode" -eq 1 ] && [ "$child_joined" -eq 0 ]; then
|
||||||
|
cleanup_code=78
|
||||||
|
elif [ "$generic_mode" -eq 1 ]; then
|
||||||
|
cleanup_command 5 /usr/local/bin/zapret-autotune dns-forwarder readiness-cleanup "$dns_pid" || cleanup_code=78
|
||||||
|
else
|
||||||
/usr/local/bin/zapret-autotune dns-forwarder readiness-cleanup "$dns_pid" || cleanup_code=78
|
/usr/local/bin/zapret-autotune dns-forwarder readiness-cleanup "$dns_pid" || cleanup_code=78
|
||||||
dns_pid=
|
|
||||||
fi
|
fi
|
||||||
|
if [ "$generic_mode" -eq 0 ] || [ "$child_joined" -eq 1 ]; then dns_pid=; fi
|
||||||
|
fi
|
||||||
|
if [ -z "$dns_pid" ]; then
|
||||||
if [ "$dns_anchor_file" -eq 1 ]; then exec 9<&-; dns_anchor_file=0; fi
|
if [ "$dns_anchor_file" -eq 1 ]; then exec 9<&-; dns_anchor_file=0; fi
|
||||||
if [ "$dns_anchor_dir" -eq 1 ]; then exec 8<&-; dns_anchor_dir=0; fi
|
if [ "$dns_anchor_dir" -eq 1 ]; then exec 8<&-; dns_anchor_dir=0; fi
|
||||||
|
fi
|
||||||
|
if [ "$generic_mode" -eq 1 ]; then
|
||||||
|
[ "$cleanup_code" -eq 0 ] || generic_cleanup_failed=1
|
||||||
|
generic_cleanup_deadline=
|
||||||
|
fi
|
||||||
return "$cleanup_code"
|
return "$cleanup_code"
|
||||||
}
|
}
|
||||||
cleanup() {
|
cleanup() {
|
||||||
@@ -174,6 +270,7 @@ cleanup() {
|
|||||||
wait "$cooldown_pid" 2>/dev/null || true
|
wait "$cooldown_pid" 2>/dev/null || true
|
||||||
fi
|
fi
|
||||||
cleanup_cycle || final_code=78
|
cleanup_cycle || final_code=78
|
||||||
|
[ "$generic_cleanup_failed" -eq 0 ] || final_code=78
|
||||||
exit "$final_code"
|
exit "$final_code"
|
||||||
}
|
}
|
||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
@@ -232,19 +329,21 @@ until awk '$2 == "00000000:0438" && $4 == "0A" { found=1 } END { exit !found }'
|
|||||||
done
|
done
|
||||||
dns_alive || return 75
|
dns_alive || return 75
|
||||||
# Reject unsafe operator requests before init/reconcile can mutate owned state.
|
# Reject unsafe operator requests before init/reconcile can mutate owned state.
|
||||||
timeout -k 5 45 /usr/local/bin/zapret-autotune managed experimental-request-check || exit 78
|
if [ "$generic_mode" -eq 0 ]; then
|
||||||
|
timeout -k 5 45 /usr/local/bin/zapret-autotune managed experimental-request-check || return 78
|
||||||
|
fi
|
||||||
if [ ! -e "$ZAPRET_CONFIG" ]; then
|
if [ ! -e "$ZAPRET_CONFIG" ]; then
|
||||||
timeout -k 5 45 /usr/local/bin/zapret-autotune managed init || exit 78
|
timeout -k 5 45 /usr/local/bin/zapret-autotune managed init || return 78
|
||||||
fi
|
fi
|
||||||
# Never activate pending or unknown bytes. Any reconciliation failure is fatal,
|
# Never activate pending or unknown bytes. Any reconciliation failure is fatal,
|
||||||
# not a transient (75) restart request. Reload restores known bytes first.
|
# not a transient (75) restart request. Reload restores known bytes first.
|
||||||
timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile || exit 78
|
timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile || return 78
|
||||||
zapret_started=1
|
zapret_started=1
|
||||||
timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 start || exit 78
|
timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 start || return 78
|
||||||
|
|
||||||
# Private operator request, consumed only after a successful guarded transaction.
|
# Private operator request, consumed only after a successful guarded transaction.
|
||||||
# Never suppress qualification gates globally or race the owned monitor.
|
# Never suppress qualification gates globally or race the owned monitor.
|
||||||
if [ -e "${ZAPRET_CONFIG%/*}/.autotune-experimental-suffix-request" ] || [ -L "${ZAPRET_CONFIG%/*}/.autotune-experimental-suffix-request" ]; then
|
if [ "$generic_mode" -eq 0 ] && { [ -e "${ZAPRET_CONFIG%/*}/.autotune-experimental-suffix-request" ] || [ -L "${ZAPRET_CONFIG%/*}/.autotune-experimental-suffix-request" ]; }; then
|
||||||
timeout -k 5 120 /usr/local/bin/zapret-autotune managed apply --googlevideo-experiment native-circular --googlevideo-scope experimental_googlevideo_suffix --googlevideo-request startup || exit 78
|
timeout -k 5 120 /usr/local/bin/zapret-autotune managed apply --googlevideo-experiment native-circular --googlevideo-scope experimental_googlevideo_suffix --googlevideo-request startup || exit 78
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -259,13 +358,13 @@ monitor_ack_deadline=$(($(date +%s) + 5))
|
|||||||
while [ ! -e "$config_dir/monitor-ready" ]; do
|
while [ ! -e "$config_dir/monitor-ready" ]; do
|
||||||
if ! kill -0 "$monitor_pid" 2>/dev/null || [ "$(date +%s)" -ge "$monitor_ack_deadline" ]; then
|
if ! kill -0 "$monitor_pid" 2>/dev/null || [ "$(date +%s)" -ge "$monitor_ack_deadline" ]; then
|
||||||
signal_tree "$monitor_pid" KILL
|
signal_tree "$monitor_pid" KILL
|
||||||
exit 78
|
return 78
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
capture_monitor_session || exit 78
|
capture_monitor_session || return 78
|
||||||
if proc_identity "$monitor_pid" && [ "$proc_sid" != "$monitor_sid" ]; then
|
if proc_identity "$monitor_pid" && [ "$proc_sid" != "$monitor_sid" ]; then
|
||||||
monitor_sid=
|
monitor_sid=
|
||||||
exit 78
|
return 78
|
||||||
fi
|
fi
|
||||||
while dns_alive && kill -0 "$server_pid" && kill -0 "$local_pid" && kill -0 "$monitor_pid"; do
|
while dns_alive && kill -0 "$server_pid" && kill -0 "$local_pid" && kill -0 "$monitor_pid"; do
|
||||||
pause 1
|
pause 1
|
||||||
@@ -300,11 +399,29 @@ while :; do
|
|||||||
count=$((count + 1))
|
count=$((count + 1))
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
[ "$count" -lt 3 ] || { echo 'error: target restart budget exhausted' >&2; exit 78; }
|
if [ "$count" -ge 3 ]; then
|
||||||
|
if [ "$generic_mode" -eq 1 ]; then
|
||||||
|
echo 'BLOCKED: generic target restart budget; checks paused' >&2
|
||||||
|
pause 60
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
echo 'error: target restart budget exhausted' >&2; exit 78
|
||||||
|
fi
|
||||||
starts="$recent $now"
|
starts="$recent $now"
|
||||||
code=0
|
code=0
|
||||||
run_cycle || code=$?
|
run_cycle || code=$?
|
||||||
cleanup_cycle || code=78
|
if ! cleanup_cycle; then
|
||||||
|
if [ "$generic_mode" -eq 1 ]; then
|
||||||
|
echo 'BLOCKED: generic target unsafe cleanup; operator repair/restart required' >&2
|
||||||
|
while :; do pause 60; done
|
||||||
|
fi
|
||||||
|
code=78
|
||||||
|
fi
|
||||||
|
if [ "$generic_mode" -eq 1 ] && [ "$code" -eq 78 ]; then
|
||||||
|
echo 'BLOCKED: generic target trust/startup failure; checks paused' >&2
|
||||||
|
pause 60
|
||||||
|
continue
|
||||||
|
fi
|
||||||
[ "$code" -eq 75 ] || exit "$code"
|
[ "$code" -eq 75 ] || exit "$code"
|
||||||
# A long-running monitor may outlive the window; recheck at failure time.
|
# A long-running monitor may outlive the window; recheck at failure time.
|
||||||
now=$(date +%s)
|
now=$(date +%s)
|
||||||
@@ -312,7 +429,9 @@ while :; do
|
|||||||
for started in $starts; do
|
for started in $starts; do
|
||||||
[ "$((now - started))" -ge 21600 ] || retry_count=$((retry_count + 1))
|
[ "$((now - started))" -ge 21600 ] || retry_count=$((retry_count + 1))
|
||||||
done
|
done
|
||||||
[ "$retry_count" -lt 3 ] || { echo 'error: target restart budget exhausted' >&2; exit 78; }
|
if [ "$retry_count" -ge 3 ] && [ "$generic_mode" -eq 0 ]; then
|
||||||
|
echo 'error: target restart budget exhausted' >&2; exit 78
|
||||||
|
fi
|
||||||
pause "$backoff"
|
pause "$backoff"
|
||||||
[ "$backoff" -ge 60 ] || backoff=$((backoff * 2))
|
[ "$backoff" -ge 60 ] || backoff=$((backoff * 2))
|
||||||
done
|
done
|
||||||
@@ -12,6 +12,8 @@
|
|||||||
"method": "GET",
|
"method": "GET",
|
||||||
"tls": "both",
|
"tls": "both",
|
||||||
"timeout_ms": 10000,
|
"timeout_ms": 10000,
|
||||||
|
"interval_ms": 300000,
|
||||||
|
"cooldown_ms": 30000,
|
||||||
"expect": {
|
"expect": {
|
||||||
"statuses": [200],
|
"statuses": [200],
|
||||||
"content_type": "text/html",
|
"content_type": "text/html",
|
||||||
|
|||||||
Reference in new issue
Block a user