fix(forgejo-runner): clean up cancelled startup containers #20

Merged

john merged 1 commit from feat/forgejo-runner-cancel-cleanup into main

2026-08-03 13:38:13 +00:00

john commented

2026-08-03 13:22:52 +00:00

Owner

Summary

add runner patch 0003 for startup cancellation cleanup
move the existing stop/close finalizers around the complete startup-plus-execution pipeline
add a regression that cancels inside startContainer(), requires a live cleanup context, and checks stop/close ordering
document the evidence boundary and production rollout checkpoint

Evidence

During the patched tiki canary, Forgejo terminalized task 77755 as cancelled while
the runner still reported a local unspecified result, but its idle job container
remained running. This is consistent with cancellation after container creation and
before startContainer() returned, but a canary with patch 0003 is still required to
tie that exact production event to the reproduced path.

The new local regression failed before the fix with only startContainer executed;
stopContainer and closeContainer were never called. It passes after the cleanup
finalizers wrap the complete lifecycle.

Validation

all patches apply cleanly to immutable runner v12.7.3 commit f89f3e3cdb1c89b51c925c78028752d78c2478c1
go test -race ./act/runner -run '^TestJobExecutorCleansUpWhenCancelledDuringContainerStart$' -count=10
go test ./act/runner -short -count=1
go test -race ./internal/pkg/client -count=1
go test ./internal/pkg/report ./internal/app/poll ./internal/app/run -count=1
go vet ./...
static Linux/amd64 build succeeded; verification artifact SHA-256:
db9f24c277a02b04827dfe40d72f182578444ed32ebe2be514355c09c6f6ac3b

The broad go test ./... -short run reached all packages but failed in two unchanged
act/container macOS environment cases: Docker credential lookup returned a Keychain
error, and one option test rejected Docker's reported darwin server OS. The changed
act/runner package and all directly relevant packages passed.

Deploy Runbook

This PR does not deploy anything.

After merge, rebuild from immutable runner commit f89f3e3 with patches 0001–0003.
Re-run the documented race, vet, static-build, metadata, and patch-application gates.
At a fresh production checkpoint, require tiki to have zero active task containers.
Back up the current tiki binary, upload the reviewed checksum, install atomically,
and restart only forgejo-runner on tiki.
Soak on natural traffic without dispatching Superclean refreshes. Verify terminal
cancellation leaves no task container, plus zero panic, reporter error, zombie
candidate, and service restart.
Promotion to forge requires a separate production checkpoint.

Rollback is the existing exact binary backup restoration plus a runner restart on the
single canary host.

Spec Drift Callouts

None.

## Summary - add runner patch 0003 for startup cancellation cleanup - move the existing stop/close finalizers around the complete startup-plus-execution pipeline - add a regression that cancels inside `startContainer()`, requires a live cleanup context, and checks stop/close ordering - document the evidence boundary and production rollout checkpoint ## Evidence During the patched `tiki` canary, Forgejo terminalized task 77755 as cancelled while the runner still reported a local unspecified result, but its idle job container remained running. This is consistent with cancellation after container creation and before `startContainer()` returned, but a canary with patch 0003 is still required to tie that exact production event to the reproduced path. The new local regression failed before the fix with only `startContainer` executed; `stopContainer` and `closeContainer` were never called. It passes after the cleanup finalizers wrap the complete lifecycle. ## Validation - all patches apply cleanly to immutable runner v12.7.3 commit `f89f3e3cdb1c89b51c925c78028752d78c2478c1` - `go test -race ./act/runner -run '^TestJobExecutorCleansUpWhenCancelledDuringContainerStart$' -count=10` - `go test ./act/runner -short -count=1` - `go test -race ./internal/pkg/client -count=1` - `go test ./internal/pkg/report ./internal/app/poll ./internal/app/run -count=1` - `go vet ./...` - static Linux/amd64 build succeeded; verification artifact SHA-256: `db9f24c277a02b04827dfe40d72f182578444ed32ebe2be514355c09c6f6ac3b` The broad `go test ./... -short` run reached all packages but failed in two unchanged `act/container` macOS environment cases: Docker credential lookup returned a Keychain error, and one option test rejected Docker's reported `darwin` server OS. The changed `act/runner` package and all directly relevant packages passed. ## Deploy Runbook This PR does not deploy anything. 1. After merge, rebuild from immutable runner commit `f89f3e3` with patches 0001–0003. 2. Re-run the documented race, vet, static-build, metadata, and patch-application gates. 3. At a fresh production checkpoint, require `tiki` to have zero active task containers. 4. Back up the current `tiki` binary, upload the reviewed checksum, install atomically, and restart only `forgejo-runner` on `tiki`. 5. Soak on natural traffic without dispatching Superclean refreshes. Verify terminal cancellation leaves no task container, plus zero panic, reporter error, zombie candidate, and service restart. 6. Promotion to `forge` requires a separate production checkpoint. Rollback is the existing exact binary backup restoration plus a runner restart on the single canary host. ## Spec Drift Callouts None.

john added 1 commit

2026-08-03 13:22:52 +00:00