fix: prevent automatic retries of failed startup apply
Latch explicit resource apply once per PID 1 lifetime. Fail closed before launch and permanently pause after authoritative cleanup on failed or incomplete apply; retain reconciliation on successful monitor and service restarts. Cover deterministic 180-second retry windows, launch/ACK/child/session/service failures, cancellation, successful restarts and fresh-PID1 flags. Document RAM-only reset limits and repair-only startup; no remote health acceptance is claimed. Co-authored-by: GPT-6 <gpt-6@openai.com>
This commit is contained in:
1 parent
36aace02da
commit
643d0c6220
3 files changed
+111
-13
No files matched your search
+21
-3
@@ -3,6 +3,7 @@
|
|||||||
`plan` remains **proposal-only** and never changes runtime or config.
|
`plan` remains **proposal-only** and never changes runtime or config.
|
||||||
Explicit generic startup apply is implemented and independently reviewed locally;
|
Explicit generic startup apply is implemented and independently reviewed locally;
|
||||||
remote acceptance is pending. This is not automatic recovery or production READY.
|
remote acceptance is pending. This is not automatic recovery or production READY.
|
||||||
|
The per-PID-1 one-shot latch correction passed independent local review only.
|
||||||
Explicit isolated qualification temporarily installs an owned NFQ worker and
|
Explicit isolated qualification temporarily installs an owned NFQ worker and
|
||||||
queue rules, but never changes legacy SQL groups, renderer or apply journals.
|
queue rules, but never changes legacy SQL groups, renderer or apply journals.
|
||||||
Monitor HEALTHY, old SQL measurements and public check JSON are not proof.
|
Monitor HEALTHY, old SQL measurements and public check JSON are not proof.
|
||||||
@@ -174,9 +175,22 @@ manager, new proof importer, SQL frontend surrogate or parallel journal exists.
|
|||||||
PID 1 starts owned SS/DoH, reconciles known bytes, launches the acknowledged
|
PID 1 starts owned SS/DoH, reconciles known bytes, launches the acknowledged
|
||||||
`managed resource-apply --resource-config FILE` child before the checks-only
|
`managed resource-apply --resource-config FILE` child before the checks-only
|
||||||
monitor, and starts the native daemon only with verified committed config.
|
monitor, and starts the native daemon only with verified committed config.
|
||||||
This is one finite apply invocation per startup cycle, not exactly once per
|
This is at most one apply launch per PID 1 lifetime. A RAM latch is marked failed
|
||||||
container: PID 1 can retry cycles under its existing restart budget. Identical
|
before launch and succeeds only after child wait=0 and owned-session cleanup.
|
||||||
committed requests resume by authenticated refs without reapplying the transaction.
|
Launch/ACK failure, nonzero/signal exit, SS/DoH loss, or incomplete cleanup leaves
|
||||||
|
it failed: after authoritative cleanup PID 1 remains cancellably BLOCKED, without
|
||||||
|
restarting SS/DoH, native zapret, apply or monitoring, even across former 60s retry
|
||||||
|
windows. Cleanup still stops SS/DoH (DNS last); BLOCKED does not keep them alive.
|
||||||
|
Successful apply is skipped on later monitor/SS restarts, but every runtime cycle
|
||||||
|
still reconciles committed refs before native start. Qualification and legacy
|
||||||
|
retry behavior are unchanged.
|
||||||
|
|
||||||
|
**Ceiling:** this is not a durable manual-intent latch. A fresh PID 1 (including
|
||||||
|
an external Docker restart policy) resets authorization and, with flag=1, permits
|
||||||
|
one new attempt. Operators must disable unattended external restart/recreation
|
||||||
|
for this workflow. Set `ZAPRET_AUTOTUNE_RESOURCE_APPLY=0` for explicit repair-only
|
||||||
|
startup: prepared state is reconciled, never promoted or reapplied. Identical
|
||||||
|
committed requests still resume by authenticated refs without a new transaction.
|
||||||
The command refuses standalone/docker-exec callers: actual PID 1 environment,
|
The command refuses standalone/docker-exec callers: actual PID 1 environment,
|
||||||
entrypoint, namespace, direct-child executable/argv and held launch ACK are
|
entrypoint, namespace, direct-child executable/argv and held launch ACK are
|
||||||
required, plus the existing controller singleton and no pre-existing NFQ worker.
|
required, plus the existing controller singleton and no pre-existing NFQ worker.
|
||||||
@@ -228,6 +242,10 @@ managed/resource status validates generic refs without inventing legacy groups.
|
|||||||
Local root fixture tests cover the production snapshot factory and shared engine;
|
Local root fixture tests cover the production snapshot factory and shared engine;
|
||||||
independent local review passed. Actual native/network/PID 1 apply acceptance is
|
independent local review passed. Actual native/network/PID 1 apply acceptance is
|
||||||
**not run**. The next gate is isolated bounded remote acceptance, not live rollout.
|
**not run**. The next gate is isolated bounded remote acceptance, not live rollout.
|
||||||
|
The per-PID-1 latch correction passed independent local review and deterministic
|
||||||
|
180-second simulated retry-window fixtures; no remote PASS is claimed for it.
|
||||||
|
Roadmap: authorized isolated acceptance; no remote deployment is authorized by
|
||||||
|
local fixtures.
|
||||||
|
|
||||||
## Bounded remote acceptance — source `5cdc0cd`
|
## Bounded remote acceptance — source `5cdc0cd`
|
||||||
|
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ def increment(key):
|
|||||||
return n
|
return n
|
||||||
def linger(code=0):
|
def linger(code=0):
|
||||||
def stop(*_):
|
def stop(*_):
|
||||||
if os.environ.get('ZAPRET_AUTOTUNE_DNS_MODE', 'doh') == 'doh' and scenario not in ('dns-death', 'dns-killed', 'dns-replace', 'dns-replace-symlink', 'dns-prestart'):
|
if os.environ.get('ZAPRET_AUTOTUNE_DNS_MODE', 'doh') == 'doh' and scenario not in ('dns-death', 'dns-killed', 'dns-replace', 'dns-replace-symlink', 'dns-prestart', 'generic-apply-dns-loss', 'generic-apply-dns-restart'):
|
||||||
os.kill(int((root / 'dns-pid').read_text()), 0)
|
os.kill(int((root / 'dns-pid').read_text()), 0)
|
||||||
sys.exit(code)
|
sys.exit(code)
|
||||||
signal.signal(signal.SIGTERM, stop)
|
signal.signal(signal.SIGTERM, stop)
|
||||||
@@ -67,10 +67,21 @@ if name == 'awk':
|
|||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
if name == 'date':
|
if name == 'date':
|
||||||
n = increment('clock')
|
n = increment('clock')
|
||||||
print(100000 + (n * 21601 if scenario == 'window' else n + (21601 if (root / 'budget-pause').exists() else 0)))
|
elapsed = int((root / 'elapsed').read_text()) if (root / 'elapsed').exists() else 0
|
||||||
|
print(100000 + elapsed + (n * 21601 if scenario == 'window' else n + (21601 if (root / 'budget-pause').exists() else 0)))
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
if name == 'sleep':
|
if name == 'sleep':
|
||||||
seconds = int(sys.argv[1])
|
seconds = int(sys.argv[1])
|
||||||
|
if seconds == 60 and scenario.startswith('generic-apply-'):
|
||||||
|
# Three former 60s retry windows; the failed producer becomes healthy
|
||||||
|
# after its first call, so any unintended retry would reach native READY.
|
||||||
|
n = increment('blocked_windows')
|
||||||
|
(root / 'elapsed').write_text(str(n * 60))
|
||||||
|
log('blocked-window ' + str(n))
|
||||||
|
if n == 3:
|
||||||
|
os.kill(os.getppid(), signal.SIGTERM)
|
||||||
|
time.sleep(.01)
|
||||||
|
sys.exit(0)
|
||||||
if scenario.startswith('qualify-') and (root / 'qualified').exists():
|
if scenario.startswith('qualify-') and (root / 'qualified').exists():
|
||||||
log('safe-idle')
|
log('safe-idle')
|
||||||
os.kill(os.getppid(), signal.SIGTERM)
|
os.kill(os.getppid(), signal.SIGTERM)
|
||||||
@@ -78,7 +89,7 @@ if name == 'sleep':
|
|||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
if scenario.startswith('generic-budget-') and seconds != 60:
|
if scenario.startswith('generic-budget-') and seconds != 60:
|
||||||
os.execv(os.environ['REAL_SLEEP'], [os.environ['REAL_SLEEP'], *sys.argv[1:]])
|
os.execv(os.environ['REAL_SLEEP'], [os.environ['REAL_SLEEP'], *sys.argv[1:]])
|
||||||
if seconds == 60 and scenario in ('generic-ss', 'generic-unsafe', 'generic-budget-monitor', 'generic-budget-reconcile', 'generic-budget-ack', 'generic-apply-blocked', 'generic-apply-abrupt', 'generic-apply-cleanup'):
|
if seconds == 60 and scenario in ('generic-ss', 'generic-unsafe', 'generic-budget-monitor', 'generic-budget-reconcile', 'generic-budget-ack'):
|
||||||
log('generic-pause')
|
log('generic-pause')
|
||||||
(root / 'budget-pause').touch()
|
(root / 'budget-pause').touch()
|
||||||
if scenario != 'generic-ss':
|
if scenario != 'generic-ss':
|
||||||
@@ -110,6 +121,8 @@ if name in ('ss-server', 'ss-local'):
|
|||||||
os.kill(int((root / 'dns-pid').read_text()), 0)
|
os.kill(int((root / 'dns-pid').read_text()), 0)
|
||||||
assert (pathlib.Path(sys.argv[2]).stat().st_mode & 0o777) == 0o600
|
assert (pathlib.Path(sys.argv[2]).stat().st_mode & 0o777) == 0o600
|
||||||
log(name)
|
log(name)
|
||||||
|
if name == 'ss-local':
|
||||||
|
(root / 'ss-pid').write_text(str(os.getpid()))
|
||||||
if scenario.startswith('generic-budget-'):
|
if scenario.startswith('generic-budget-'):
|
||||||
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
||||||
while True:
|
while True:
|
||||||
@@ -153,6 +166,9 @@ if sys.argv[1] == 'qualification-completion-check':
|
|||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
if sys.argv[1:] == ['target-config-check']:
|
if sys.argv[1:] == ['target-config-check']:
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
|
if sys.argv[1:] == ['apply-launch']:
|
||||||
|
log('apply-launch')
|
||||||
|
sys.exit(0)
|
||||||
if sys.argv[1] == 'dns-forwarder':
|
if sys.argv[1] == 'dns-forwarder':
|
||||||
if sys.argv[2] in ('readiness-check', 'readiness-cleanup'):
|
if sys.argv[2] in ('readiness-check', 'readiness-cleanup'):
|
||||||
# exec preserves PID1's actual inherited anchors; no simulated unlink.
|
# exec preserves PID1's actual inherited anchors; no simulated unlink.
|
||||||
@@ -284,6 +300,14 @@ elif command == 'monitor':
|
|||||||
f.write(str(os.getpid()) + '\n')
|
f.write(str(os.getpid()) + '\n')
|
||||||
n = increment('monitor_runs')
|
n = increment('monitor_runs')
|
||||||
log('monitor ' + str(n))
|
log('monitor ' + str(n))
|
||||||
|
if n == 1 and scenario == 'generic-apply-monitor-restart':
|
||||||
|
sys.exit(75)
|
||||||
|
if n == 1 and scenario == 'generic-apply-ss-restart':
|
||||||
|
os.kill(int((root / 'ss-pid').read_text()), signal.SIGKILL)
|
||||||
|
linger()
|
||||||
|
if n == 1 and scenario == 'generic-apply-dns-restart':
|
||||||
|
os.kill(int((root / 'dns-pid').read_text()), signal.SIGKILL)
|
||||||
|
linger()
|
||||||
assert os.getsid(0) == os.getpid(), 'monitor not in its own verified session'
|
assert os.getsid(0) == os.getpid(), 'monitor not in its own verified session'
|
||||||
if scenario.startswith('generic-budget-'):
|
if scenario.startswith('generic-budget-'):
|
||||||
time.sleep(.03)
|
time.sleep(.03)
|
||||||
@@ -331,12 +355,23 @@ elif command == 'resource-apply':
|
|||||||
assert os.getsid(0) == os.getpid(), 'apply lacks owned session'
|
assert os.getsid(0) == os.getpid(), 'apply lacks owned session'
|
||||||
assert (root / 'private' / 'monitor-ready').read_text() == str(os.getpid()) + '\n'
|
assert (root / 'private' / 'monitor-ready').read_text() == str(os.getpid()) + '\n'
|
||||||
events = (root / 'events').read_text().splitlines()
|
events = (root / 'events').read_text().splitlines()
|
||||||
assert events[-1] == 'reconcile' and 'start' not in events
|
assert events[-2:] == ['reconcile', 'apply-launch'] and 'start' not in events
|
||||||
log('resource-apply')
|
log('resource-apply')
|
||||||
if scenario == 'generic-apply-blocked':
|
n = increment('apply_runs')
|
||||||
|
if scenario == 'generic-apply-blocked' and n == 1:
|
||||||
sys.exit(78)
|
sys.exit(78)
|
||||||
if scenario in ('generic-apply-abrupt', 'generic-apply-cleanup'):
|
if scenario == 'generic-apply-abrupt' and n == 1:
|
||||||
sys.exit(23)
|
sys.exit(23)
|
||||||
|
if scenario == 'generic-apply-cleanup' and n == 1:
|
||||||
|
log('resource-returned-zero')
|
||||||
|
sys.exit(0)
|
||||||
|
if scenario == 'generic-apply-signal' and n == 1:
|
||||||
|
os.kill(os.getpid(), signal.SIGKILL)
|
||||||
|
if scenario in ('generic-apply-dns-loss', 'generic-apply-ss-loss') and n == 1:
|
||||||
|
os.kill(int((root / ('dns-pid' if scenario.endswith('dns-loss') else 'ss-pid')).read_text()), signal.SIGKILL)
|
||||||
|
signal.signal(signal.SIGTERM, lambda *_: sys.exit(0))
|
||||||
|
while True:
|
||||||
|
time.sleep(.01)
|
||||||
if scenario == 'generic-apply-cancel':
|
if scenario == 'generic-apply-cancel':
|
||||||
def restored(*_):
|
def restored(*_):
|
||||||
log('resource-restored')
|
log('resource-restored')
|
||||||
@@ -366,6 +401,8 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
(binary / name).unlink()
|
(binary / name).unlink()
|
||||||
configs = root / 'private'
|
configs = root / 'private'
|
||||||
configs.mkdir()
|
configs.mkdir()
|
||||||
|
if scenario == 'generic-apply-disabled':
|
||||||
|
(root / 'prepared').touch()
|
||||||
if scenario.startswith('dns-stale'):
|
if scenario.startswith('dns-stale'):
|
||||||
if scenario == 'dns-stale-symlink':
|
if scenario == 'dns-stale-symlink':
|
||||||
(configs / 'dns-ready').symlink_to(configs / 'missing')
|
(configs / 'dns-ready').symlink_to(configs / 'missing')
|
||||||
@@ -386,6 +423,16 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
text = text.replace('config_dir=/run/zapret-autotune-ss', f'config_dir={configs}')
|
text = text.replace('config_dir=/run/zapret-autotune-ss', f'config_dir={configs}')
|
||||||
text = text.replace('/usr/local/bin/zapret-autotune', str(binary / 'autotune'))
|
text = text.replace('/usr/local/bin/zapret-autotune', str(binary / 'autotune'))
|
||||||
text = text.replace('/opt/zapret2/init.d/sysv/zapret2', str(binary / 'zapret2'))
|
text = text.replace('/opt/zapret2/init.d/sysv/zapret2', str(binary / 'zapret2'))
|
||||||
|
if scenario.startswith('generic-apply-'):
|
||||||
|
launch = ' launch_controller managed resource-apply'
|
||||||
|
assert launch in text
|
||||||
|
text = text.replace(launch, f' {binary / "autotune"} apply-launch\n' + launch)
|
||||||
|
if scenario == 'generic-apply-launch-fail':
|
||||||
|
text = text.replace(' launch_controller managed resource-apply', ' false || return 78\n launch_controller managed resource-apply')
|
||||||
|
if scenario == 'generic-apply-ack-fail':
|
||||||
|
broken_helper = root / 'ack-helper.sh'
|
||||||
|
broken_helper.write_text(helper.read_text().replace(' ack_publish ||', ' false && ack_publish ||'))
|
||||||
|
text = text.replace(str(helper), str(broken_helper))
|
||||||
if scenario == 'generic-apply-cleanup':
|
if scenario == 'generic-apply-cleanup':
|
||||||
text = text.replace('cleanup_session || { echo', "cleanup_session() { return 78; }\n cleanup_session || { echo", 1)
|
text = text.replace('cleanup_session || { echo', "cleanup_session() { return 78; }\n cleanup_session || { echo", 1)
|
||||||
if scenario.endswith('-race'):
|
if scenario.endswith('-race'):
|
||||||
@@ -411,7 +458,7 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
env.pop('ZAPRET_AUTOTUNE_QUALIFY_CONFIG', None)
|
env.pop('ZAPRET_AUTOTUNE_QUALIFY_CONFIG', None)
|
||||||
env.pop('ZAPRET_AUTOTUNE_STRATEGY_CONFIG', None)
|
env.pop('ZAPRET_AUTOTUNE_STRATEGY_CONFIG', None)
|
||||||
env.pop('ZAPRET_AUTOTUNE_RESOURCE_APPLY', None)
|
env.pop('ZAPRET_AUTOTUNE_RESOURCE_APPLY', None)
|
||||||
if scenario.startswith('generic-apply-'):
|
if scenario.startswith('generic-apply-') and scenario != 'generic-apply-disabled':
|
||||||
env['ZAPRET_AUTOTUNE_RESOURCE_APPLY'] = '1'
|
env['ZAPRET_AUTOTUNE_RESOURCE_APPLY'] = '1'
|
||||||
if scenario.startswith('qualify-'):
|
if scenario.startswith('qualify-'):
|
||||||
env['ZAPRET_AUTOTUNE_QUALIFY_CONFIG'] = '/operator/resources.json'
|
env['ZAPRET_AUTOTUNE_QUALIFY_CONFIG'] = '/operator/resources.json'
|
||||||
@@ -444,7 +491,7 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
assert b'Illegal number' not in result.stderr, (scenario, result.stderr)
|
assert b'Illegal number' not in result.stderr, (scenario, result.stderr)
|
||||||
assert b'ab' * 32 not in result.stdout + result.stderr
|
assert b'ab' * 32 not in result.stdout + result.stderr
|
||||||
if scenario in ('generic-apply-blocked', 'generic-apply-abrupt', 'generic-apply-cleanup'):
|
if scenario in ('generic-apply-blocked', 'generic-apply-abrupt', 'generic-apply-cleanup'):
|
||||||
code = 78 if scenario == 'generic-apply-blocked' else 23
|
code = {'generic-apply-blocked': 78, 'generic-apply-abrupt': 23, 'generic-apply-cleanup': 0}[scenario]
|
||||||
assert f'controller-startup: apply_wait_exit={code}\n'.encode() in result.stderr, result.stderr
|
assert f'controller-startup: apply_wait_exit={code}\n'.encode() in result.stderr, result.stderr
|
||||||
assert (b'controller-startup: apply_cleanup=failed\n' in result.stderr) == (scenario == 'generic-apply-cleanup'), result.stderr
|
assert (b'controller-startup: apply_cleanup=failed\n' in result.stderr) == (scenario == 'generic-apply-cleanup'), result.stderr
|
||||||
for pid in ((root / 'children').read_text().splitlines() if (root / 'children').exists() else []):
|
for pid in ((root / 'children').read_text().splitlines() if (root / 'children').exists() else []):
|
||||||
@@ -476,6 +523,15 @@ def run(scenario, expected, dns='', mode='native'):
|
|||||||
assert not (configs / 'dns-ready').exists(), (scenario, 'stale DNS readiness')
|
assert not (configs / 'dns-ready').exists(), (scenario, 'stale DNS readiness')
|
||||||
if mode == 'native':
|
if mode == 'native':
|
||||||
assert 'dns-start' not in events
|
assert 'dns-start' not in events
|
||||||
|
if scenario.startswith('generic-apply-') and 'blocked-window 1' in events:
|
||||||
|
assert events.count('apply-launch') == 1, (scenario, events)
|
||||||
|
assert int((root / 'elapsed').read_text()) == 180, (scenario, events)
|
||||||
|
assert events.count('dns-start') == events.count('ss-server') == events.count('ss-local') == 1, (scenario, events)
|
||||||
|
assert 'start' not in events and not any(e.startswith('monitor ') for e in events), (scenario, events)
|
||||||
|
assert 'cooldown 15' not in events and 'cooldown 30' not in events, (scenario, events)
|
||||||
|
assert events[events.index('blocked-window 1'):] == ['blocked-window 1', 'blocked-window 2', 'blocked-window 3'], (scenario, events)
|
||||||
|
if scenario != 'generic-apply-dns-loss':
|
||||||
|
assert events.index('dns-stop') < events.index('blocked-window 1'), (scenario, events)
|
||||||
return events
|
return events
|
||||||
|
|
||||||
|
|
||||||
@@ -513,10 +569,25 @@ events = run('generic-apply-cancel', 0, mode='doh')
|
|||||||
assert 'resource-restored' in events and 'start' not in events and 'monitor 1' not in events, events
|
assert 'resource-restored' in events and 'start' not in events and 'monitor 1' not in events, events
|
||||||
assert events.index('resource-restored') < events.index('stop') < events.index('dns-stop'), events
|
assert events.index('resource-restored') < events.index('stop') < events.index('dns-stop'), events
|
||||||
events = run('generic-apply-blocked', 0, mode='doh')
|
events = run('generic-apply-blocked', 0, mode='doh')
|
||||||
assert 'start' not in events and 'monitor 1' not in events and 'generic-pause' in events, events
|
assert events.count('resource-apply') == 1 and 'blocked-window 3' in events, events
|
||||||
for scenario, code in (('generic-apply-abrupt', 0), ('generic-apply-cleanup', 78)):
|
for scenario, code in (('generic-apply-abrupt', 0), ('generic-apply-cleanup', 78)):
|
||||||
events = run(scenario, code, mode='doh')
|
events = run(scenario, code, mode='doh')
|
||||||
assert 'start' not in events and 'monitor 1' not in events and 'resource-applied' not in events, events
|
assert 'start' not in events and 'monitor 1' not in events and 'resource-applied' not in events, events
|
||||||
|
for scenario in ('signal', 'dns-loss', 'ss-loss', 'launch-fail', 'ack-fail'):
|
||||||
|
events = run('generic-apply-' + scenario, 78 if scenario == 'ack-fail' else 0, mode='doh')
|
||||||
|
assert 'blocked-window 3' in events and 'resource-applied' not in events, events
|
||||||
|
assert events.count('resource-apply') == (0 if scenario in ('launch-fail', 'ack-fail') else 1), events
|
||||||
|
for scenario in ('monitor-restart', 'ss-restart', 'dns-restart'):
|
||||||
|
events = run('generic-apply-' + scenario, 0, mode='doh')
|
||||||
|
assert events.count('resource-apply') == events.count('apply-launch') == 1, events
|
||||||
|
assert events.count('start') == events.count('stop') == 2 and 'monitor 2' in events, events
|
||||||
|
assert events.count('dns-start') == 2 and events.count('dns-stop') == (1 if scenario == 'dns-restart' else 2), events
|
||||||
|
# Independent fresh PID 1 invocations reset the RAM latch; flag=0 still repairs
|
||||||
|
# prepared bytes by reconcile, without launching apply.
|
||||||
|
events = run('generic-apply-pass', 0, mode='doh')
|
||||||
|
assert events.count('resource-apply') == 1, events
|
||||||
|
events = run('generic-apply-disabled', 0, mode='doh')
|
||||||
|
assert 'resource-apply' not in events and 'start' in events, events
|
||||||
for scenario in ('generic-budget-monitor', 'generic-budget-reconcile', 'generic-budget-ack'):
|
for scenario in ('generic-budget-monitor', 'generic-budget-reconcile', 'generic-budget-ack'):
|
||||||
events = run(scenario, 78, mode='doh')
|
events = run(scenario, 78, mode='doh')
|
||||||
assert events.count('start') == 1 and 'stop' not in events, (scenario, events)
|
assert events.count('start') == 1 and 'stop' not in events, (scenario, events)
|
||||||
|
|||||||
@@ -20,6 +20,8 @@ generic_mode=0
|
|||||||
[ -z "${ZAPRET_AUTOTUNE_RESOURCE_CONFIG-}" ] || generic_mode=1
|
[ -z "${ZAPRET_AUTOTUNE_RESOURCE_CONFIG-}" ] || generic_mode=1
|
||||||
qualification_mode=0
|
qualification_mode=0
|
||||||
qualification_attempted=0
|
qualification_attempted=0
|
||||||
|
# RAM-only authorization: a new PID 1 may attempt again, never a runtime retry.
|
||||||
|
resource_apply_state=unattempted
|
||||||
if [ -n "${ZAPRET_AUTOTUNE_QUALIFY_CONFIG-}${ZAPRET_AUTOTUNE_STRATEGY_CONFIG-}" ]; then
|
if [ -n "${ZAPRET_AUTOTUNE_QUALIFY_CONFIG-}${ZAPRET_AUTOTUNE_STRATEGY_CONFIG-}" ]; then
|
||||||
[ -n "${ZAPRET_AUTOTUNE_QUALIFY_CONFIG-}" ] && [ -n "${ZAPRET_AUTOTUNE_STRATEGY_CONFIG-}" ] || {
|
[ -n "${ZAPRET_AUTOTUNE_QUALIFY_CONFIG-}" ] && [ -n "${ZAPRET_AUTOTUNE_STRATEGY_CONFIG-}" ] || {
|
||||||
echo 'error: isolated qualification requires both config and strategy paths' >&2; exit 64;
|
echo 'error: isolated qualification requires both config and strategy paths' >&2; exit 64;
|
||||||
@@ -375,6 +377,7 @@ launch_controller() {
|
|||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
run_cycle() {
|
run_cycle() {
|
||||||
|
[ "$resource_apply_state" != failed ] || return 78
|
||||||
[ "$qualification_attempted" -eq 0 ] || return 78
|
[ "$qualification_attempted" -eq 0 ] || return 78
|
||||||
start_dns || return $?
|
start_dns || return $?
|
||||||
set -- ss-server -c "$config_dir/server.json"
|
set -- ss-server -c "$config_dir/server.json"
|
||||||
@@ -406,7 +409,8 @@ fi
|
|||||||
timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile || return 78
|
timeout -k 5 45 /usr/local/bin/zapret-autotune managed reconcile || return 78
|
||||||
if [ "$qualification_mode" -eq 0 ]; then
|
if [ "$qualification_mode" -eq 0 ]; then
|
||||||
zapret_started=1
|
zapret_started=1
|
||||||
if [ "${ZAPRET_AUTOTUNE_RESOURCE_APPLY-0}" = 1 ]; then
|
if [ "${ZAPRET_AUTOTUNE_RESOURCE_APPLY-0}" = 1 ] && [ "$resource_apply_state" = unattempted ]; then
|
||||||
|
resource_apply_state=failed
|
||||||
launch_controller managed resource-apply --resource-config "$ZAPRET_AUTOTUNE_RESOURCE_CONFIG" || return 78
|
launch_controller managed resource-apply --resource-config "$ZAPRET_AUTOTUNE_RESOURCE_CONFIG" || return 78
|
||||||
while dns_alive && kill -0 "$server_pid" && kill -0 "$local_pid" && kill -0 "$monitor_pid"; do pause 1; done
|
while dns_alive && kill -0 "$server_pid" && kill -0 "$local_pid" && kill -0 "$monitor_pid"; do pause 1; done
|
||||||
{ dns_alive && kill -0 "$server_pid" && kill -0 "$local_pid"; } || return 75
|
{ dns_alive && kill -0 "$server_pid" && kill -0 "$local_pid"; } || return 75
|
||||||
@@ -416,6 +420,7 @@ if [ "${ZAPRET_AUTOTUNE_RESOURCE_APPLY-0}" = 1 ]; then
|
|||||||
monitor_pid=
|
monitor_pid=
|
||||||
cleanup_session || { echo 'controller-startup: apply_cleanup=failed' >&2; return 78; }
|
cleanup_session || { echo 'controller-startup: apply_cleanup=failed' >&2; return 78; }
|
||||||
[ "$apply_code" -eq 0 ] || return 78
|
[ "$apply_code" -eq 0 ] || return 78
|
||||||
|
resource_apply_state=succeeded
|
||||||
fi
|
fi
|
||||||
# Idempotent apply may have only verified the already committed config.
|
# Idempotent apply may have only verified the already committed config.
|
||||||
timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 start || return 78
|
timeout -k 5 30 /opt/zapret2/init.d/sysv/zapret2 start || return 78
|
||||||
@@ -501,6 +506,10 @@ while :; do
|
|||||||
fi
|
fi
|
||||||
code=78
|
code=78
|
||||||
fi
|
fi
|
||||||
|
if [ "$resource_apply_state" = failed ]; then
|
||||||
|
echo 'BLOCKED: resource apply failed/incomplete; operator repair/restart required' >&2
|
||||||
|
while :; do pause 60; done
|
||||||
|
fi
|
||||||
if [ "$generic_mode" -eq 1 ] && [ "$code" -eq 78 ]; then
|
if [ "$generic_mode" -eq 1 ] && [ "$code" -eq 78 ]; then
|
||||||
echo 'BLOCKED: generic target trust/startup failure; checks paused' >&2
|
echo 'BLOCKED: generic target trust/startup failure; checks paused' >&2
|
||||||
pause 60
|
pause 60
|
||||||
|
|||||||
Reference in new issue
Block a user