fix(startup): reconcile profiles before bounded activation
Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
b1e458a0a5
commit
e2cfccedf1
9 files changed
+338
-25
No files matched your search
+15
-3
@@ -1,6 +1,6 @@
|
|||||||
# Fresh managed-test target contract (not deployed)
|
# Fresh managed-test target contract (not deployed)
|
||||||
|
|
||||||
`managed init|status|active-ids|apply` is exclusively for a **new dedicated container**
|
`managed init|reconcile|status|active-ids|apply` is exclusively for a **new dedicated container**
|
||||||
which owns its entire config. The existing/dev/production container must refuse
|
which owns its entire config. The existing/dev/production container must refuse
|
||||||
these commands. A role env or CLI-created identity is not enough. Provisioning
|
these commands. A role env or CLI-created identity is not enough. Provisioning
|
||||||
must supply all of the following before any managed command can run:
|
must supply all of the following before any managed command can run:
|
||||||
@@ -160,7 +160,7 @@ DB/safety/journal/runner failures stop nonzero; joined cancellation plus a clean
|
|||||||
failure is also fatal. A qualified frontend at startup is probed immediately,
|
failure is also fatal. A qualified frontend at startup is probed immediately,
|
||||||
not rescanned merely because the process restarted.
|
not rescanned merely because the process restarted.
|
||||||
|
|
||||||
Entrypoint ordering is SS children → listener → managed init/status → zapret start
|
Entrypoint ordering is SS children → listener → fresh managed init if needed → managed reconcile → zapret start
|
||||||
→ one installed `/usr/local/bin/zapret-autotune managed monitor` child. It supervises
|
→ one installed `/usr/local/bin/zapret-autotune managed monitor` child. It supervises
|
||||||
all three children; monitor exit is fatal even if it exits zero unexpectedly.
|
all three children; monitor exit is fatal even if it exits zero unexpectedly.
|
||||||
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
|
SIGTERM/SIGINT cancels and waits the monitor before target-local zapret stop and
|
||||||
@@ -197,6 +197,12 @@ Pins constrain build inputs; they do not establish live acceptance or guarantee
|
|||||||
bit-for-bit builds on different platforms. Record the source commit, platform,
|
bit-for-bit builds on different platforms. Record the source commit, platform,
|
||||||
builder/runtime digests and resulting image digest for every release.
|
builder/runtime digests and resulting image digest for every release.
|
||||||
|
|
||||||
|
Both Dockerfiles require BuildKit for the compilation-cache mount at
|
||||||
|
`/root/.cache/go-build` (enable with `DOCKER_BUILDKIT=1` if not already default).
|
||||||
|
This cache only speeds rebuilds; it is not copied into the final runtime image
|
||||||
|
and is not a secret store. No deployment credentials are needed or passed to
|
||||||
|
the build; keep secrets out of build arguments and cache contents.
|
||||||
|
|
||||||
Example (replace registry and release; never reuse a release tag):
|
Example (replace registry and release; never reuse a release tag):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
@@ -317,7 +323,13 @@ be reported as complete playback health.
|
|||||||
A restored legacy/live/unverifiable recovery owner can fail closed; never
|
A restored legacy/live/unverifiable recovery owner can fail closed; never
|
||||||
clear its lease or unlink the lock merely to force startup.
|
clear its lease or unlink the lock merely to force startup.
|
||||||
|
|
||||||
On drift, pending transaction, rollback failure, or exhausted systemd start
|
On startup, `managed reconcile` accepts no flags and uses only the fixed private
|
||||||
|
`/var/lib/zapret-autotune/target/candidates.db`. Known interrupted prepared state
|
||||||
|
is rolled back to qualified committed bytes before any reload/start; fresh
|
||||||
|
baseline and known committed state are idempotent. Unknown/legacy state, drift,
|
||||||
|
missing qualification or failed rollback blocks startup with a non-retry exit.
|
||||||
|
|
||||||
|
On drift, unreconcilable pending transaction, rollback failure, or exhausted systemd start
|
||||||
limit, leave state intact and service stopped. Diagnose using sanitized status
|
limit, leave state intact and service stopped. Diagnose using sanitized status
|
||||||
and private artifacts; repair the actual provisioning/configuration issue or
|
and private artifacts; repair the actual provisioning/configuration issue or
|
||||||
restore a known consistent backup. Only then manually start (and reset the
|
restore a known consistent backup. Only then manually start (and reset the
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ systemd stop timeout 180s.
|
|||||||
|
|
||||||
## Manual managed-test activation (not deployed)
|
## Manual managed-test activation (not deployed)
|
||||||
|
|
||||||
`managed init|status|active-ids|apply` requires exact role `managed-test`, target ID
|
`managed init|reconcile|status|active-ids|apply` requires exact role `managed-test`, target ID
|
||||||
`zapret-autotune-managed-v1`, a provisioned runtime marker and zapret executables,
|
`zapret-autotune-managed-v1`, a provisioned runtime marker and zapret executables,
|
||||||
and a private dedicated mount at `/var/lib/zapret-autotune/target`.
|
and a private dedicated mount at `/var/lib/zapret-autotune/target`.
|
||||||
`ZAPRET_CONFIG` must equal `/var/lib/zapret-autotune/target/config`: the standalone
|
`ZAPRET_CONFIG` must equal `/var/lib/zapret-autotune/target/config`: the standalone
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ Commands:
|
|||||||
discover youtube --video-id ID --proxy socks5://HOST:PORT
|
discover youtube --video-id ID --proxy socks5://HOST:PORT
|
||||||
Discover cache hostname only (no media-transfer check)
|
Discover cache hostname only (no media-transfer check)
|
||||||
health youtube One-shot Player API health through configured loopback SOCKS
|
health youtube One-shot Player API health through configured loopback SOCKS
|
||||||
managed init|status|apply|monitor
|
managed init|reconcile|status|apply|monitor
|
||||||
Guarded activation/monitoring in the dedicated managed-test target only
|
Guarded activation/monitoring in the dedicated managed-test target only
|
||||||
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
|
check youtube Refuse checks until probe URLs/video IDs are configured and agreed
|
||||||
tune youtube Test youtube.com frontend only; print options without applying
|
tune youtube Test youtube.com frontend only; print options without applying
|
||||||
|
|||||||
@@ -55,8 +55,8 @@ func managedDependencies(player playerRunner) managedDeps {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func managedArguments(args []string) (string, []int64, error) {
|
func managedArguments(args []string) (string, []int64, error) {
|
||||||
invalid := fmt.Errorf("managed accepts init, status, active-ids, or apply with one or both candidate flags (full desired set); no path flags")
|
invalid := fmt.Errorf("managed accepts init, reconcile, status, active-ids, or apply with one or both candidate flags (full desired set); no path flags")
|
||||||
if len(args) == 1 && (args[0] == "init" || args[0] == "status" || args[0] == "active-ids") {
|
if len(args) == 1 && (args[0] == "init" || args[0] == "reconcile" || args[0] == "status" || args[0] == "active-ids") {
|
||||||
return args[0], nil, nil
|
return args[0], nil, nil
|
||||||
}
|
}
|
||||||
if (len(args) != 3 && len(args) != 5) || args[0] != "apply" {
|
if (len(args) != 3 && len(args) != 5) || args[0] != "apply" {
|
||||||
@@ -326,7 +326,9 @@ func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
|||||||
if command == "init" {
|
if command == "init" {
|
||||||
err = managedInitialize(m)
|
err = managedInitialize(m)
|
||||||
} else if err = managedCheckIdentity(m); err == nil {
|
} else if err = managedCheckIdentity(m); err == nil {
|
||||||
if command == "status" || command == "active-ids" {
|
if command == "reconcile" {
|
||||||
|
err = managedPrestartReconcile(m, d)
|
||||||
|
} else if command == "status" || command == "active-ids" {
|
||||||
err = managedReport(m, stdout, d.openDB, command == "active-ids")
|
err = managedReport(m, stdout, d.openDB, command == "active-ids")
|
||||||
} else {
|
} else {
|
||||||
expected := map[int64]string{}
|
expected := map[int64]string{}
|
||||||
@@ -341,7 +343,7 @@ func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
|||||||
fmt.Fprintln(stderr, "error:", err)
|
fmt.Fprintln(stderr, "error:", err)
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
if command == "init" || command == "apply" {
|
if command == "init" || command == "reconcile" || command == "apply" {
|
||||||
if _, err := fmt.Fprintln(stdout, "managed-test "+command+": OK"); err != nil {
|
if _, err := fmt.Fprintln(stdout, "managed-test "+command+": OK"); err != nil {
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
@@ -354,6 +356,45 @@ func managedCLI(args []string, stdout, stderr io.Writer, d managedDeps) int {
|
|||||||
return 0
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Startup may activate only a known baseline or DB-qualified rendered config.
|
||||||
|
// Reconcile restores bytes before its fixed target-local restart hook runs.
|
||||||
|
func managedPrestartReconcile(m *managedTransaction, d managedDeps) (err error) {
|
||||||
|
path, err := managedMonitorDatabase(d)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
db, err := openStore(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("candidate database unavailable")
|
||||||
|
}
|
||||||
|
defer func() {
|
||||||
|
if closeErr := db.Close(); closeErr != nil && err == nil {
|
||||||
|
err = closeErr
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
// The transaction engine supports legacy hash-only state for other callers;
|
||||||
|
// initial activation must never adopt it, even with a matching hash.
|
||||||
|
err = m.locked(func() error {
|
||||||
|
j, err := m.load()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
baselineHash := managedHash([]byte(managedBaseline))
|
||||||
|
if j.Version != 2 || (len(j.Candidates) == 0 && j.Hash != baselineHash) || (j.State == "prepared" && len(j.PendingCandidates) == 0 && j.Next != baselineHash) {
|
||||||
|
return fmt.Errorf("startup candidate state unresolved; refusing activation")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := m.Reconcile(db); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_, err = m.ActiveCandidates(db)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
func managedActivate(m *managedTransaction, d managedDeps, ids []int64, expected map[int64]string) error {
|
func managedActivate(m *managedTransaction, d managedDeps, ids []int64, expected map[int64]string) error {
|
||||||
info, err := os.Stat(d.script)
|
info, err := os.Stat(d.script)
|
||||||
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0111 == 0 {
|
if err != nil || !info.Mode().IsRegular() || info.Mode().Perm()&0111 == 0 {
|
||||||
|
|||||||
@@ -340,7 +340,7 @@ func TestManagedCLIIdentityAndStatusRedaction(t *testing.T) {
|
|||||||
|
|
||||||
func TestManagedCLIFreshTargetContract(t *testing.T) {
|
func TestManagedCLIFreshTargetContract(t *testing.T) {
|
||||||
for _, mode := range []string{"wrong-id", "missing-id", "old-env", "other-directory", "old-config", "other-config", "non-mountpoint", "missing-base-runtime", "wrong-port", "other-loopback", "wrong-process", "missing-proxy", "missing-port"} {
|
for _, mode := range []string{"wrong-id", "missing-id", "old-env", "other-directory", "old-config", "other-config", "non-mountpoint", "missing-base-runtime", "wrong-port", "other-loopback", "wrong-process", "missing-proxy", "missing-port"} {
|
||||||
for _, command := range []string{"init", "status", "apply"} {
|
for _, command := range []string{"init", "reconcile", "status", "apply"} {
|
||||||
t.Run(mode+"/"+command, func(t *testing.T) {
|
t.Run(mode+"/"+command, func(t *testing.T) {
|
||||||
d, env, calls := managedCLIFixture(t)
|
d, env, calls := managedCLIFixture(t)
|
||||||
if command != "init" {
|
if command != "init" {
|
||||||
@@ -382,6 +382,11 @@ func TestManagedCLIFreshTargetContract(t *testing.T) {
|
|||||||
if code, _, _ := callManaged(t, d, args...); code == 0 || *calls != 0 {
|
if code, _, _ := callManaged(t, d, args...); code == 0 || *calls != 0 {
|
||||||
t.Fatal("unsafe target contract accepted or reloaded")
|
t.Fatal("unsafe target contract accepted or reloaded")
|
||||||
}
|
}
|
||||||
|
if command == "reconcile" {
|
||||||
|
if _, err := os.Lstat(filepath.Join(d.directory, "candidates.db")); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatal("runtime guard opened database", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
if command == "init" {
|
if command == "init" {
|
||||||
entries, err := os.ReadDir(d.directory)
|
entries, err := os.ReadDir(d.directory)
|
||||||
if err != nil || len(entries) != 0 {
|
if err != nil || len(entries) != 0 {
|
||||||
|
|||||||
@@ -14,6 +14,213 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Subprocess adapter exercises the real CLI/transaction/DB behind the shell
|
||||||
|
// entrypoint. Only runtime/socket validation and fixed fixture paths are fake.
|
||||||
|
func TestManagedPrestartHelper(t *testing.T) {
|
||||||
|
if os.Getenv("MANAGED_PRESTART_HELPER") != "1" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
args := os.Args
|
||||||
|
for len(args) > 0 && args[0] != "--" {
|
||||||
|
args = args[1:]
|
||||||
|
}
|
||||||
|
if len(args) < 3 {
|
||||||
|
os.Exit(1)
|
||||||
|
}
|
||||||
|
args = args[2:] // -- managed COMMAND
|
||||||
|
if args[0] == "monitor" {
|
||||||
|
os.Exit(78) // Stop after initial activation; no scans or network.
|
||||||
|
}
|
||||||
|
d := managedDependencies(nil)
|
||||||
|
d.directory, d.script = os.Getenv("FIXTURE_TARGET"), os.Getenv("FIXTURE_SCRIPT")
|
||||||
|
d.verifyRuntime = func(string) error { return nil }
|
||||||
|
d.verifySOCKS = func(int) error { return nil }
|
||||||
|
os.Exit(managedCLI(args, os.Stdout, os.Stderr, d))
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedEntrypointPrestartReconciliation(t *testing.T) {
|
||||||
|
for _, mode := range []string{"fresh", "baseline", "committed", "prepared", "drift", "unknown", "legacy", "corrupt-backup", "unqualified", "reload-failure"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, env, _ := managedCLIFixture(t)
|
||||||
|
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
|
||||||
|
m := &managedTransaction{Dir: d.directory, Name: "config", Reload: func() error { return nil }, Health: func() (string, error) { return "OK", nil }}
|
||||||
|
known := []byte(managedBaseline)
|
||||||
|
if mode != "fresh" {
|
||||||
|
initManagedCLI(t, d)
|
||||||
|
}
|
||||||
|
if mode == "committed" || mode == "prepared" || mode == "corrupt-backup" || mode == "unqualified" || mode == "reload-failure" {
|
||||||
|
db, err := openStore(filepath.Join(d.directory, "candidates.db"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
id := managedCandidateFixture(t, db, "frontend", "youtube.com", "--lua-desync=fake", true)
|
||||||
|
if mode != "committed" {
|
||||||
|
m.fault = func(point string) error {
|
||||||
|
if point == "replaced" {
|
||||||
|
return errors.New("simulated process death")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
err = m.Apply(db, []int64{id})
|
||||||
|
if (mode == "committed") != (err == nil) {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if mode == "committed" {
|
||||||
|
known, err = renderManagedConfig(db, []int64{id})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
j, err := m.load()
|
||||||
|
b, readErr := m.read("config")
|
||||||
|
if err != nil || readErr != nil || j.State != "prepared" || managedHash(b) != j.Next || j.Next == j.Hash {
|
||||||
|
t.Fatal("not a true interrupted replacement fixture", j, err, readErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode == "unqualified" {
|
||||||
|
if _, err := db.Exec("DELETE FROM candidate_evidence WHERE candidate_id = ?", id); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := db.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode == "drift" || mode == "unknown" || mode == "corrupt-backup" {
|
||||||
|
name := "config"
|
||||||
|
if mode == "corrupt-backup" {
|
||||||
|
name = managedBackupName
|
||||||
|
}
|
||||||
|
if err := m.replace(name, []byte("UNRECOGNIZED=1\n")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode == "unknown" || mode == "legacy" {
|
||||||
|
j, err := m.load()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if mode == "legacy" {
|
||||||
|
j.Version = 1
|
||||||
|
} else {
|
||||||
|
j.Hash = managedHash([]byte("UNRECOGNIZED=1\n"))
|
||||||
|
}
|
||||||
|
if err := m.journal(j); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
root := t.TempDir()
|
||||||
|
write := func(name, text string) string {
|
||||||
|
path := filepath.Join(root, name)
|
||||||
|
if err := os.WriteFile(path, []byte(text), 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
ss := filepath.Join(root, "ss")
|
||||||
|
if err := os.Mkdir(ss, 0700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
write("awk", "#!/bin/sh\nexit 0\n")
|
||||||
|
child := "#!/bin/sh\ntrap 'exit 0' TERM INT\nwhile :; do sleep 0.1 & wait $!; done\n"
|
||||||
|
write("ss-server", child)
|
||||||
|
write("ss-local", child)
|
||||||
|
log := filepath.Join(root, "log")
|
||||||
|
zapret := write("zapret", "#!/bin/sh\nprintf '%s ' \"$1\" >> \"$LOG\"\nsha256sum \"$FIXTURE_CONFIG\" >> \"$LOG\"\n[ \"$MODE:$1\" != reload-failure:restart ]\n")
|
||||||
|
binary, err := os.Executable()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
autotune := write("autotune", "#!/bin/sh\nexec \"$FIXTURE_BINARY\" -test.run='^TestManagedPrestartHelper$' -- \"$@\"\n")
|
||||||
|
data, err := os.ReadFile("../../managed-target-entrypoint.sh")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
entry := write("entrypoint", strings.NewReplacer("/run/zapret-autotune-ss", ss, "/usr/local/bin/zapret-autotune", autotune, managedZapretScript, zapret, `"$ZAPRET_CONFIG"`, `"$FIXTURE_CONFIG"`).Replace(string(data)))
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
cmd := exec.CommandContext(ctx, "/bin/sh", entry)
|
||||||
|
cmd.Env = append(os.Environ(), "PATH="+root+":"+os.Getenv("PATH"), "LOG="+log, "MODE="+mode, "SS_PASSWORD="+strings.Repeat("a", 32), "MANAGED_PRESTART_HELPER=1", "FIXTURE_BINARY="+binary, "FIXTURE_TARGET="+d.directory, "FIXTURE_SCRIPT="+zapret)
|
||||||
|
for key, value := range env {
|
||||||
|
cmd.Env = append(cmd.Env, key+"="+value)
|
||||||
|
}
|
||||||
|
// The CLI guard sees fixed production bindings; only fake daemon reads
|
||||||
|
// use the fixture config, via its own script rather than CLI path flags.
|
||||||
|
cmd.Env = append(cmd.Env, "FIXTURE_CONFIG="+filepath.Join(d.directory, "config"))
|
||||||
|
output, err := cmd.CombinedOutput()
|
||||||
|
var exit *exec.ExitError
|
||||||
|
if ctx.Err() != nil || !errors.As(err, &exit) || exit.ExitCode() != 78 {
|
||||||
|
t.Fatal(err, ctx.Err(), string(output))
|
||||||
|
}
|
||||||
|
b, _ := os.ReadFile(log)
|
||||||
|
allowed := mode == "fresh" || mode == "baseline" || mode == "committed" || mode == "prepared"
|
||||||
|
if !allowed && mode != "reload-failure" && len(b) != 0 {
|
||||||
|
t.Fatal("unsafe startup invoked daemon script", string(b))
|
||||||
|
}
|
||||||
|
if allowed && !strings.Contains("\n"+string(b), "\nstart "+managedHash(known)) {
|
||||||
|
t.Fatal("no committed initial activation", string(b), string(output))
|
||||||
|
}
|
||||||
|
for _, line := range strings.Split(strings.TrimSpace(string(b)), "\n") {
|
||||||
|
if line != "" && !strings.Contains(line, managedHash(known)) {
|
||||||
|
t.Fatal("daemon saw uncommitted bytes", line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if mode == "reload-failure" && (strings.HasPrefix(string(b), "start ") || strings.Contains(string(b), "\nstart ")) {
|
||||||
|
t.Fatal("failed rollback continued initial activation", string(b))
|
||||||
|
}
|
||||||
|
if err := m.locked(func() error { return nil }); err != nil {
|
||||||
|
t.Fatal("reconciliation leaked transaction lock", err)
|
||||||
|
}
|
||||||
|
if allowed {
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
if code, _, diagnostic := callManaged(t, d, "reconcile"); code != 0 {
|
||||||
|
t.Fatal("non-idempotent reconciliation/lock leak", code, diagnostic)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
j, err := m.load()
|
||||||
|
if err != nil || j.State != "committed" || j.Hash != managedHash(known) {
|
||||||
|
t.Fatal(j, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManagedReconcileDatabaseGuards(t *testing.T) {
|
||||||
|
for _, mode := range []string{"path", "symlink", "public", "flags"} {
|
||||||
|
t.Run(mode, func(t *testing.T) {
|
||||||
|
d, env, calls := managedCLIFixture(t)
|
||||||
|
initManagedCLI(t, d)
|
||||||
|
env["ZAPRET_AUTOTUNE_DB"] = managedTargetDirectory + "/candidates.db"
|
||||||
|
path := filepath.Join(d.directory, "candidates.db")
|
||||||
|
args := []string{"reconcile"}
|
||||||
|
switch mode {
|
||||||
|
case "path":
|
||||||
|
env["ZAPRET_AUTOTUNE_DB"] = "/other/db"
|
||||||
|
case "symlink":
|
||||||
|
if err := os.Symlink(filepath.Join(t.TempDir(), "missing"), path); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "public":
|
||||||
|
if err := os.WriteFile(path, []byte("unchanged"), 0644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
case "flags":
|
||||||
|
args = append(args, "--db", path)
|
||||||
|
}
|
||||||
|
if code, _, _ := callManaged(t, d, args...); code == 0 || code == 75 || *calls != 0 {
|
||||||
|
t.Fatal("unsafe guard accepted", code)
|
||||||
|
}
|
||||||
|
if mode == "path" || mode == "flags" {
|
||||||
|
if _, err := os.Lstat(path); !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatal("guard opened database", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestManagedMonitorArguments(t *testing.T) {
|
func TestManagedMonitorArguments(t *testing.T) {
|
||||||
lookup := func(string) (string, bool) { return "", false }
|
lookup := func(string) (string, bool) { return "", false }
|
||||||
cfg, err := managedMonitorConfig(nil, lookup)
|
cfg, err := managedMonitorConfig(nil, lookup)
|
||||||
@@ -261,7 +468,13 @@ func TestManagedMonitorBootstrapExistingFrontend(t *testing.T) {
|
|||||||
// Execute only rewritten target-local contracts and fake programs; no network,
|
// Execute only rewritten target-local contracts and fake programs; no network,
|
||||||
// Docker, real credentials, zapret or host firewall. Exercise the actual traps.
|
// Docker, real credentials, zapret or host firewall. Exercise the actual traps.
|
||||||
func TestManagedEntrypointSupervision(t *testing.T) {
|
func TestManagedEntrypointSupervision(t *testing.T) {
|
||||||
for _, mode := range []string{"term", "fatal", "technical", "panic", "killed", "segv", "zero", "unknown", "ss", "ss-unsafe", "ss-unknown", "ss-technical", "term-unsafe", "term-unknown", "term-hang", "stop-failure", "stop-hang"} {
|
modes := []string{"term", "fatal", "technical", "panic", "killed", "segv", "zero", "unknown", "ss", "ss-unsafe", "ss-unknown", "ss-technical", "term-unsafe", "term-unknown", "term-hang", "stop-failure", "stop-hang"}
|
||||||
|
for _, stage := range []string{"init", "reconcile", "start"} {
|
||||||
|
for _, failure := range []string{"hang", "75", "124", "137", "7"} {
|
||||||
|
modes = append(modes, "startup-"+stage+"-"+failure)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, mode := range modes {
|
||||||
t.Run(mode, func(t *testing.T) {
|
t.Run(mode, func(t *testing.T) {
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
config := filepath.Join(root, "ss")
|
config := filepath.Join(root, "ss")
|
||||||
@@ -276,12 +489,23 @@ func TestManagedEntrypointSupervision(t *testing.T) {
|
|||||||
return path
|
return path
|
||||||
}
|
}
|
||||||
log := filepath.Join(root, "log")
|
log := filepath.Join(root, "log")
|
||||||
|
// Shorten only startup watchdogs in this fixture. Execute a real
|
||||||
|
// timeout, including its forced-kill path, without a 45-second test.
|
||||||
|
if strings.HasPrefix(mode, "startup-") {
|
||||||
|
timeout, err := exec.LookPath("timeout")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
write("timeout", "#!/bin/sh\nif [ \"$3\" = 45 ] || { [ \"$3\" = 30 ] && [ \"$5\" = start ]; }; then\nshift 3\nexec \"$REAL_TIMEOUT\" -k 1 1 \"$@\"\nfi\nexec \"$REAL_TIMEOUT\" \"$@\"\n")
|
||||||
|
t.Setenv("REAL_TIMEOUT", timeout)
|
||||||
|
}
|
||||||
|
startupFailure := "case \"$MODE\" in startup-$stage-hang) trap '' TERM INT; while :; do sleep 0.1; done;; startup-$stage-*) exit \"${MODE##*-}\";; esac\n"
|
||||||
write("awk", "#!/bin/sh\ncase \"$*\" in */proc/net/tcp*) until grep -q ready-ss-server \"$LOG\" && grep -q ready-ss-local \"$LOG\"; do sleep 0.02; done;; esac\nexit 0\n")
|
write("awk", "#!/bin/sh\ncase \"$*\" in */proc/net/tcp*) until grep -q ready-ss-server \"$LOG\" && grep -q ready-ss-local \"$LOG\"; do sleep 0.02; done;; esac\nexit 0\n")
|
||||||
child := "#!/bin/sh\n[ -z \"${SS_PASSWORD+x}\" ] || exit 88\ntrap 'echo stopped-$0 >> \"$LOG\"; exit 0' TERM INT\necho ready-$(basename \"$0\") >> \"$LOG\"\nwhile :; do\ncase \"$MODE:$0\" in ss*:*/ss-local) if grep -q managed-monitor \"$LOG\"; then echo stopped-$0 >> \"$LOG\"; exit 7; fi;; esac\nsleep 0.1 & wait $!; done\n"
|
child := "#!/bin/sh\n[ -z \"${SS_PASSWORD+x}\" ] || exit 88\ntrap 'echo stopped-$0 >> \"$LOG\"; exit 0' TERM INT\necho ready-$(basename \"$0\") >> \"$LOG\"\nwhile :; do\ncase \"$MODE:$0\" in ss*:*/ss-local) if grep -q managed-monitor \"$LOG\"; then echo stopped-$0 >> \"$LOG\"; exit 7; fi;; esac\nsleep 0.1 & wait $!; done\n"
|
||||||
write("ss-server", child)
|
write("ss-server", child)
|
||||||
write("ss-local", child)
|
write("ss-local", child)
|
||||||
zapret := write("zapret", "#!/bin/sh\necho zapret-$1 >> \"$LOG\"\nif [ \"$MODE:$1\" = stop-failure:stop ]; then exit 1; fi\nif [ \"$MODE:$1\" = stop-hang:stop ]; then trap '' TERM INT; while :; do sleep 0.1; done; fi\n")
|
zapret := write("zapret", "#!/bin/sh\necho zapret-$1 >> \"$LOG\"\nstage=$1\n"+startupFailure+"if [ \"$MODE:$1\" = stop-failure:stop ]; then exit 1; fi\nif [ \"$MODE:$1\" = stop-hang:stop ]; then trap '' TERM INT; while :; do sleep 0.1; done; fi\n")
|
||||||
monitor := write("autotune", "#!/bin/sh\nif [ \"$2\" != monitor ]; then echo managed-$2 >> \"$LOG\"; exit 0; fi\ntrap 'echo monitor-stopped >> \"$LOG\"; case \"$MODE\" in *unsafe) exit 78;; *unknown) exit 7;; ss-technical) exit 75;; esac; exit 0' TERM INT\nif [ \"$MODE\" = term-hang ]; then trap '' TERM INT; fi\necho managed-monitor >> \"$LOG\"\ncase \"$MODE\" in fatal) exit 78;; technical|stop-failure|stop-hang) exit 75;; panic) exit 2;; killed) kill -KILL $$;; segv) exit 139;; zero) exit 0;; unknown) exit 7;; esac\nwhile :; do sleep 0.1 & wait $!; done\n")
|
monitor := write("autotune", "#!/bin/sh\nif [ \"$2\" != monitor ]; then\necho managed-$2 >> \"$LOG\"\nstage=$2\n"+startupFailure+"exit 0\nfi\ntrap 'echo monitor-stopped >> \"$LOG\"; case \"$MODE\" in *unsafe) exit 78;; *unknown) exit 7;; ss-technical) exit 75;; esac; exit 0' TERM INT\nif [ \"$MODE\" = term-hang ]; then trap '' TERM INT; fi\necho managed-monitor >> \"$LOG\"\ncase \"$MODE\" in fatal) exit 78;; technical|stop-failure|stop-hang) exit 75;; panic) exit 2;; killed) kill -KILL $$;; segv) exit 139;; zero) exit 0;; unknown) exit 7;; esac\nwhile :; do sleep 0.1 & wait $!; done\n")
|
||||||
data, err := os.ReadFile("../../managed-target-entrypoint.sh")
|
data, err := os.ReadFile("../../managed-target-entrypoint.sh")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -332,6 +556,9 @@ func TestManagedEntrypointSupervision(t *testing.T) {
|
|||||||
t.Fatal("entrypoint did not join children", ctx.Err())
|
t.Fatal("entrypoint did not join children", ctx.Err())
|
||||||
}
|
}
|
||||||
want := 75
|
want := 75
|
||||||
|
if strings.HasPrefix(mode, "startup-") {
|
||||||
|
want = 78
|
||||||
|
}
|
||||||
switch mode {
|
switch mode {
|
||||||
case "term":
|
case "term":
|
||||||
want = 0
|
want = 0
|
||||||
@@ -352,7 +579,20 @@ func TestManagedEntrypointSupervision(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
text = string(b)
|
text = string(b)
|
||||||
if !strings.Contains(text, "managed-init\nzapret-start\nmanaged-monitor\n") || !strings.Contains(text, "zapret-stop") || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-server")) || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-local")) {
|
if strings.HasPrefix(mode, "startup-") {
|
||||||
|
if strings.Contains(text, "managed-monitor") || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-server")) || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-local")) {
|
||||||
|
t.Fatal("startup failure escaped cleanup", text)
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(mode, "startup-start-") {
|
||||||
|
if !strings.Contains(text, "managed-init\nmanaged-reconcile\nzapret-start\n") || !strings.Contains(text, "zapret-stop") {
|
||||||
|
t.Fatal("startup ordering/daemon cleanup", text)
|
||||||
|
}
|
||||||
|
} else if strings.Contains(text, "zapret-") {
|
||||||
|
t.Fatal("daemon invoked after failed prestart guard", text)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !strings.Contains(text, "managed-init\nmanaged-reconcile\nzapret-start\nmanaged-monitor\n") || !strings.Contains(text, "zapret-stop") || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-server")) || !strings.Contains(text, "stopped-"+filepath.Join(root, "ss-local")) {
|
||||||
t.Fatal(text)
|
t.Fatal(text)
|
||||||
}
|
}
|
||||||
if mode == "term" && (!strings.Contains(text, "monitor-stopped") || strings.Index(text, "monitor-stopped") > strings.Index(text, "zapret-stop")) {
|
if mode == "term" && (!strings.Contains(text, "monitor-stopped") || strings.Index(text, "monitor-stopped") > strings.Index(text, "zapret-stop")) {
|
||||||
|
|||||||
@@ -84,10 +84,20 @@ capabilities are dropped except `NET_ADMIN`, `NET_RAW`, `SETUID`, `SETGID`, `KIL
|
|||||||
`KILL` allows bounded cleanup of workers that have dropped their UID.
|
`KILL` allows bounded cleanup of workers that have dropped their UID.
|
||||||
|
|
||||||
Startup order: ss-server/ss-local → listener 1080 → guarded managed init on
|
Startup order: ss-server/ss-local → listener 1080 → guarded managed init on
|
||||||
fresh state (status on recreation) → target-local `zapret2 start` → supervised
|
fresh state → `managed reconcile` on every startup → target-local `zapret2 start`
|
||||||
managed monitor. Drift/pending state refuses startup; unknown state is never
|
→ supervised managed monitor. Reconciliation uses the fixed private target DB
|
||||||
adopted. SIGTERM/SIGINT cancels and waits for the monitor before stopping
|
and restores an interrupted prepared transaction's qualified last-good bytes
|
||||||
target-local zapret and Shadowsocks children. Monitor fatal exits stop the
|
before any daemon reload/start. Drift, unknown state or failed rollback exits
|
||||||
|
78 (no automatic retry); nothing is adopted. Restore a consistent whole-state
|
||||||
|
backup if needed; do not wipe/re-init persisted state.
|
||||||
|
|
||||||
|
Startup init and reconcile each have a 45-second deadline; initial daemon start
|
||||||
|
has 30 seconds, with forced kill after five more seconds for each command.
|
||||||
|
Any startup command failure/timeout exits 78, never the retry code 75. These
|
||||||
|
bounds do not shorten the monitor's 90-minute bootstrap/recovery scan budget.
|
||||||
|
|
||||||
|
SIGTERM/SIGINT cancels and waits for the monitor before stopping target-local
|
||||||
|
zapret and Shadowsocks children. Monitor fatal exits stop the
|
||||||
entrypoint. Monitor health is YouTube Player API status through the owned SOCKS,
|
entrypoint. Monitor health is YouTube Player API status through the owned SOCKS,
|
||||||
not video playback/SABR; GGC candidates are scoped to the exact discovered
|
not video playback/SABR; GGC candidates are scoped to the exact discovered
|
||||||
hostname.
|
hostname.
|
||||||
|
|||||||
@@ -43,3 +43,8 @@ compose_command = "/usr/bin/docker compose --env-file deploy/managed-target/.env
|
|||||||
assert directives["ExecStart"] == compose_command + " up --no-build --abort-on-container-exit --exit-code-from target target"
|
assert directives["ExecStart"] == compose_command + " up --no-build --abort-on-container-exit --exit-code-from target target"
|
||||||
assert directives["ExecStop"] == compose_command + " stop target"
|
assert directives["ExecStop"] == compose_command + " stop target"
|
||||||
assert "timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 stop" in entrypoint
|
assert "timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 stop" in entrypoint
|
||||||
|
for command, seconds in (("/usr/local/bin/zapret-autotune managed init", 45),
|
||||||
|
("/usr/local/bin/zapret-autotune managed reconcile", 45),
|
||||||
|
("/opt/zapret2/init.d/sysv/zapret2 start", 30)):
|
||||||
|
assert f"timeout -k 5 {seconds} {command} || exit 78" in entrypoint
|
||||||
|
assert entrypoint.index("managed reconcile || exit 78") < entrypoint.index("zapret2 start || exit 78")
|
||||||
@@ -77,13 +77,13 @@ cleanup() {
|
|||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
trap 'exit 0' INT TERM
|
trap 'exit 0' INT TERM
|
||||||
|
|
||||||
# Never invoke /opt/entrypoint.sh: zapret must start AFTER managed init.
|
# Never invoke /opt/entrypoint.sh: zapret must start AFTER reconciliation.
|
||||||
ss-server -c "$config_dir/server.json" &
|
ss-server -c "$config_dir/server.json" &
|
||||||
server_pid=$!
|
server_pid=$!
|
||||||
ss-local -c "$config_dir/local.json" &
|
ss-local -c "$config_dir/local.json" &
|
||||||
local_pid=$!
|
local_pid=$!
|
||||||
|
|
||||||
# Bounded readiness, no network probe. Init/status verifies socket ownership.
|
# Bounded readiness, no network probe. Managed commands verify socket ownership.
|
||||||
attempt=0
|
attempt=0
|
||||||
until awk '$2 == "00000000:0438" && $4 == "0A" { found=1 } END { exit !found }' /proc/net/tcp; do
|
until awk '$2 == "00000000:0438" && $4 == "0A" { found=1 } END { exit !found }' /proc/net/tcp; do
|
||||||
{ kill -0 "$server_pid" && kill -0 "$local_pid"; } || exit 75
|
{ kill -0 "$server_pid" && kill -0 "$local_pid"; } || exit 75
|
||||||
@@ -92,13 +92,13 @@ until awk '$2 == "00000000:0438" && $4 == "0A" { found=1 } END { exit !found }'
|
|||||||
sleep 1
|
sleep 1
|
||||||
done
|
done
|
||||||
if [ ! -e "$ZAPRET_CONFIG" ]; then
|
if [ ! -e "$ZAPRET_CONFIG" ]; then
|
||||||
/usr/local/bin/zapret-autotune managed init
|
timeout -k 5 45 /usr/local/bin/zapret-autotune managed init || exit 78
|
||||||
else
|
|
||||||
# Never reinitialize/adopt persisted state; drift/pending state fails closed.
|
|
||||||
/usr/local/bin/zapret-autotune managed status
|
|
||||||
fi
|
fi
|
||||||
|
# Never activate pending or unknown bytes. Any reconciliation failure is fatal,
|
||||||
|
# not a transient (75) restart request. Reload restores known bytes first.
|
||||||
|
timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile || exit 78
|
||||||
zapret_started=1
|
zapret_started=1
|
||||||
/opt/zapret2/init.d/sysv/zapret2 start
|
timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 start || exit 78
|
||||||
|
|
||||||
/usr/local/bin/zapret-autotune managed monitor &
|
/usr/local/bin/zapret-autotune managed monitor &
|
||||||
monitor_pid=$!
|
monitor_pid=$!
|
||||||
|
|||||||
Reference in new issue
Block a user