<# .SYNOPSIS Make the Gitea Actions runner on this machine survive restarts, and give it a toolchain the SYSTEM service account can actually see. .DESCRIPTION Why this file exists (2026-10-11 CI audit, the "many jobs never ran" bug): Six consecutive runs (16..21, i.e. 12 jobs) produced ZERO log lines. The runner `win-stm32-ci` was simply offline, and two independent causes made that permanent: 1. NO SUPERVISION. gitea-runner v4.0.1 `daemon` has no service support (v4 removed `daemon install`), and it EXITS when the first Declare call fails. The runner log shows exactly that after a Gitea restart: level=error msg="fail to invoke Declare" error="unavailable: dial tcp 127.0.0.1:3001: connectex: ... actively refused it." Error: unavailable: dial tcp 127.0.0.1:3001 ... ...and the process is gone. So one Gitea restart (or one reboot) silently kills CI until a human notices days later. Queued runs then get cancelled by the NEXT push, because ci.yml sets `concurrency.cancel-in-progress: true` -- which is why the symptom looks like "tasks vanish" rather than "tasks fail". 2. NO MACHINE-LEVEL TOOLCHAIN. The runner service runs as SYSTEM. pre-commit / cppcheck were only reachable through the interactive user's PATH, and no machine-level DTU_* variable was ever set, so even a live runner died in the static-gate environment check. This script installs the missing supervision (a watchdog launcher + a scheduled task) and, with -SetEnv, writes the machine-level DTU_* variables by discovering the tools the same way CI does. Run elevated, from the repo: powershell -ExecutionPolicy Bypass -File scripts\ci_runner_bootstrap.ps1 -SetEnv -StartNow .PARAMETER RunnerDir Directory holding gitea-runner.exe / config.yaml / .runner. .PARAMETER GiteaUrl Instance address the runner registered against (must match .runner's "address"; the runner talks to it directly, not through the public URL). .PARAMETER TaskName Scheduled-task name for the watchdog. .PARAMETER SetEnv Discover and write MACHINE-level DTU_PRECOMMIT_PYTHON / DTU_CPPCHECK / DTU_MSVC_CL. Existing values are preserved unless -Force. .PARAMETER StartNow Start the task after registering it. .PARAMETER Status Only report: task state, daemon process, Gitea reachability. .PARAMETER DryRun Build and validate the action / trigger / settings objects and print what WOULD be done, writing and registering nothing. This is also the only way to verify this script without elevation. .PARAMETER Force With -SetEnv, overwrite existing machine-level values. .OUTPUTS Progress lines. Exit codes: 0 = ok, 1 = precondition failed, 2 = not elevated. .NOTES Keep this file pure ASCII (PowerShell 5.1 decodes BOM-less files as CP936). #> [CmdletBinding()] param( [string]$RunnerDir = 'D:\Docker\act_runner', [string]$GiteaUrl = 'http://127.0.0.1:3001', [string]$TaskName = 'GiteaActionsRunner', [switch]$SetEnv, [switch]$StartNow, [switch]$Status, [switch]$DryRun, [switch]$Force ) $ErrorActionPreference = 'Stop' function Info($m) { Write-Host "[INFO] $m" -ForegroundColor Cyan } function Ok($m) { Write-Host "[ OK ] $m" -ForegroundColor Green } function Warn($m) { Write-Host "[WARN] $m" -ForegroundColor Yellow } function Fail($m) { Write-Host "[FAIL] $m" -ForegroundColor Red } function Test-Admin { $id = [Security.Principal.WindowsIdentity]::GetCurrent() (New-Object Security.Principal.WindowsPrincipal($id) ).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator) } function Test-Gitea([string]$url) { try { Invoke-WebRequest -Uri "$url/api/v1/version" -UseBasicParsing -TimeoutSec 10 | Out-Null return $true } catch { return $false } } function Get-MachineEnv([string]$name) { $v = [Environment]::GetEnvironmentVariable($name, 'Machine') if ([string]::IsNullOrWhiteSpace($v)) { return $null } return $v } function Set-MachineEnv([string]$name, [string]$value, [string]$label) { $cur = Get-MachineEnv $name if ($cur) { if ($cur -ieq $value) { Ok "$label already set: $cur"; return } if (-not $Force) { Warn "$label already set to a different value: $cur (use -Force to overwrite)" return } Warn "$label overwriting $cur -> $value" } [Environment]::SetEnvironmentVariable($name, $value, 'Machine') Ok "$label set (MACHINE scope): $value" } function Find-Cppcheck { $pf = $env:ProgramFiles $pf86 = ${env:ProgramFiles(x86)} if ([string]::IsNullOrWhiteSpace($pf)) { $pf = 'C:\Program Files' } if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' } foreach ($c in @((Join-Path $pf 'Cppcheck\cppcheck.exe'), (Join-Path $pf86 'Cppcheck\cppcheck.exe'))) { if (Test-Path -LiteralPath $c) { return $c } } $cmd = Get-Command cppcheck -ErrorAction SilentlyContinue if ($cmd) { return $cmd.Source } return $null } function Find-MsvcCl { $pf = $env:ProgramFiles $pf86 = ${env:ProgramFiles(x86)} if ([string]::IsNullOrWhiteSpace($pf)) { $pf = 'C:\Program Files' } if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' } $roots = @() foreach ($year in @('2019', '2022', '2026', '18')) { foreach ($edition in @('BuildTools', 'Community', 'Professional', 'Enterprise')) { $roots += (Join-Path $pf86 "Microsoft Visual Studio\$year\$edition") $roots += (Join-Path $pf "Microsoft Visual Studio\$year\$edition") } } foreach ($r in $roots) { if (-not (Test-Path -LiteralPath $r)) { continue } $cl = Get-ChildItem -LiteralPath $r -Recurse -Filter 'cl.exe' -ErrorAction SilentlyContinue | Where-Object { $_.FullName -like '*Hostx64\x64*' } | Select-Object -First 1 if ($cl) { return $cl.FullName } } return $null } # Builds the task objects in ONE place so -DryRun validates exactly what the # real run registers. # # MEASURED TRAP (2026-10-11, hit on the user's machine): `StartWhenAvailable` # is a TASK SETTINGS property (MSFT_TaskSettings, inherited by # MSFT_TaskSettings3) -- it does NOT exist on triggers. Assigning it to the # trigger aborts the script with a SetValueInvocationException along the lines # of "setting StartWhenAvailable threw an exception: property # 'StartWhenAvailable' is not found on this object" (message localized), # and, worse, that happened AFTER the old task had already been unregistered, # leaving CI with no task at all. Hence: build everything first, only then # unregister the old task. function New-WatchdogTaskPlan { param([string]$Launcher, [string]$WorkDir) $action = New-ScheduledTaskAction ` -Execute 'powershell.exe' ` -Argument "-NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File `"$Launcher`"" ` -WorkingDirectory $WorkDir # Boot trigger + a 1 minute grace period: Gitea may still be starting when # the session-0 boot fires (and the watchdog waits for it anyway). $trigger = New-ScheduledTaskTrigger -AtStartup $trigger.Delay = 'PT1M' $settings = New-ScheduledTaskSettingsSet ` -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries ` -ExecutionTimeLimit ([TimeSpan]::Zero) ` -RestartCount 999 -RestartInterval (New-TimeSpan -Minutes 1) ` -MultipleInstances IgnoreNew # StartWhenAvailable also covers "the machine was off when this was missed". $settings.StartWhenAvailable = $true return [pscustomobject]@{ Action = $action; Trigger = $trigger; Settings = $settings } } # --------------------------------------------------------------------------- # -Status: read-only report # --------------------------------------------------------------------------- if ($Status) { Info "task $TaskName" $t = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue if ($t) { Ok ("task state: {0}" -f $t.State) $li = Get-ScheduledTaskInfo -TaskName $TaskName -ErrorAction SilentlyContinue if ($li) { Info ("last run: {0} | last result: {1} | next run: {2}" -f ` $li.LastRunTime, ('0x{0:X}' -f $li.LastTaskResult), $li.NextRunTime) } } else { Warn "task $TaskName not found" if (-not (Test-Admin)) { Warn ' this session is NOT elevated: a non-admin session cannot always see' Warn ' every task, so re-run -Status elevated before concluding it is missing' } } $proc = @(Get-Process | Where-Object { $_.ProcessName -match 'gitea-runner|act_runner' }) if ($proc.Count -eq 1) { Ok ("daemon running: pid {0}" -f $proc[0].Id) } elseif ($proc.Count -gt 1) { Warn (" {0} daemon processes -- only ONE can hold .runner.lock, the rest are in a restart loop" -f $proc.Count) Warn ' fix: re-run this script elevated with -StartNow (it clears stale instances first)' } else { Warn 'daemon not running' } # A watchdog that cannot get the lock fails every 10s while the runner still # reports ONLINE, so the loop is invisible from the web UI. Count watchdogs and # grep the daemon log for the lock error (measured 2026-10-12). # MEASURED TRAP #3 (2026-10-12): a NON-ELEVATED session cannot read SYSTEM # process command lines -- CommandLine comes back $null, so the filter below # silently matches NOTHING and reporting "no watchdog" would be a lie (this # bit us twice while diagnosing). Say what we cannot see instead. if (-not (Test-Admin)) { Info ' watchdog count skipped: needs elevation (SYSTEM command lines are hidden)' } else { $wd = @(Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" -ErrorAction SilentlyContinue | Where-Object { $_.CommandLine -match 'run_runner_watchdog\.ps1' }) if ($wd.Count -gt 1) { Warn (" {0} watchdog processes (pids {1}) -- an old one survived a task re-registration" -f ` $wd.Count, (($wd | ForEach-Object { $_.ProcessId }) -join ',')) Warn ' and is racing the new one for .runner.lock; re-run elevated with -StartNow' } elseif ($wd.Count -eq 1) { Ok ("watchdog running: pid {0}" -f $wd[0].ProcessId) } else { Warn 'no watchdog process found (task not running?)' } } # Restart-loop detection. Do NOT just grep the log tail: the file is never # rotated, so old failures stay in the last lines forever and a HEALTHY runner # keeps "reporting" a stale error (measured false positive 2026-10-12). The # reliable signal is a daemon that keeps dying young -- a process alive longer # than the 10s restart interval cannot be inside that loop. CIM CreationDate is # readable without elevation; Process.StartTime is NOT for SYSTEM processes. $dmon = Get-CimInstance Win32_Process -Filter "Name='gitea-runner.exe'" -ErrorAction SilentlyContinue | Select-Object -First 1 $upSec = if ($dmon) { [int]((Get-Date) - $dmon.CreationDate).TotalSeconds } else { -1 } if ($upSec -ge 90) { Ok ("daemon up for {0} min -- no restart loop" -f [int]($upSec / 60)) } else { $dlog = Join-Path $RunnerDir 'log\runner-daemon.log' if (Test-Path -LiteralPath $dlog) { $tail = Get-Content -LiteralPath $dlog -Tail 40 -Encoding UTF8 -ErrorAction SilentlyContinue if ($tail -match 'already locked by another process') { Warn ' daemon keeps dying young and the log shows ".runner is already locked' Warn ' by another process" -- a stale instance holds the lock; re-run with -StartNow' } } } if (Test-Gitea $GiteaUrl) { Ok "Gitea reachable at $GiteaUrl" } else { Fail "Gitea NOT reachable at $GiteaUrl" } foreach ($n in @('DTU_PRECOMMIT_PYTHON', 'DTU_CPPCHECK', 'DTU_MSVC_CL')) { $v = Get-MachineEnv $n if ($v) { Ok ("MACHINE {0} = {1}" -f $n, $v) } else { Warn "MACHINE $n not set (CI auto-discovery still applies; see scripts/ci_find_python.ps1)" } } exit 0 } # --------------------------------------------------------------------------- # Preconditions # --------------------------------------------------------------------------- if (-not $DryRun -and -not (Test-Admin)) { Fail 'not elevated -- rerun from an elevated PowerShell (this writes a scheduled task and machine env vars)' Fail 'use -DryRun to validate the whole plan (action/trigger/settings) without elevation' exit 2 } $exe = Join-Path $RunnerDir 'gitea-runner.exe' $cfg = Join-Path $RunnerDir 'config.yaml' $regFile = Join-Path $RunnerDir '.runner' $launcher = Join-Path $RunnerDir 'run_runner_watchdog.ps1' foreach ($f in @($exe, $cfg)) { if (-not (Test-Path -LiteralPath $f)) { Fail "missing file: $f"; exit 1 } } if (-not (Test-Path -LiteralPath $regFile)) { Fail "not registered: $regFile is missing" Info 'register first, e.g.:' Write-Host " & '$exe' register --no-interactive --instance '$GiteaUrl' --token --name win-stm32-ci --labels 'windows:host'" exit 1 } # The daemon must talk to the address baked into .runner, not the public URL. $registeredAddress = $null try { $registeredAddress = (Get-Content -LiteralPath $regFile -Raw | ConvertFrom-Json).address } catch { Warn "cannot parse $regFile : $_" } if ($registeredAddress -and $registeredAddress.TrimEnd('/') -ne $GiteaUrl.TrimEnd('/')) { Warn ".runner address is $registeredAddress but -GiteaUrl is $GiteaUrl -- using the .runner value for the readiness probe" $GiteaUrl = $registeredAddress } # --------------------------------------------------------------------------- # 0. build the task plan up front; -DryRun stops here without writing anything # --------------------------------------------------------------------------- $plan = New-WatchdogTaskPlan -Launcher $launcher -WorkDir $RunnerDir if ($DryRun) { Info 'dry run -- writing and registering NOTHING' Info ("would write launcher : {0}" -f $launcher) Info ("would run : {0} {1}" -f $plan.Action.Execute, $plan.Action.Arguments) Info ("would trigger : {0} delay={1}" -f $plan.Trigger.CimClass.CimClassName, $plan.Trigger.Delay) Info ("would settings : StartWhenAvailable={0} RestartCount={1} RestartInterval={2} ExecTimeLimit={3} MultipleInstances={4}" -f ` $plan.Settings.StartWhenAvailable, $plan.Settings.RestartCount, $plan.Settings.RestartInterval, $plan.Settings.ExecutionTimeLimit, $plan.Settings.MultipleInstances) Info 'would register as : user SYSTEM, RunLevel Highest' if (Test-Admin) { Ok 'elevation: OK (admin)' } else { Warn 'elevation: NOT admin -- the real run must be started from an elevated shell' } if (Test-Gitea $GiteaUrl) { Ok "Gitea reachable at $GiteaUrl" } else { Warn "Gitea not reachable at $GiteaUrl -- the watchdog would keep waiting" } exit 0 } # --------------------------------------------------------------------------- # 1. machine-level toolchain (SYSTEM cannot see the interactive user's PATH) # --------------------------------------------------------------------------- if ($SetEnv) { Info 'discovering toolchain for MACHINE scope ...' $findPy = Join-Path $PSScriptRoot 'ci_find_python.ps1' if (Test-Path -LiteralPath $findPy) { $py = & $findPy -NeedPreCommit if ($LASTEXITCODE -eq 0 -and $py) { Set-MachineEnv 'DTU_PRECOMMIT_PYTHON' ("$py".Trim()) 'DTU_PRECOMMIT_PYTHON' } else { Warn 'no interpreter with pre-commit found -- install pre-commit into a machine-wide interpreter first' } } else { Warn "not found: $findPy (skipping DTU_PRECOMMIT_PYTHON)" } $cpp = Find-Cppcheck if ($cpp) { Set-MachineEnv 'DTU_CPPCHECK' $cpp 'DTU_CPPCHECK' } else { Warn 'cppcheck not found (install Cppcheck.Cppcheck, or set DTU_CPPCHECK manually)' } $cl = Find-MsvcCl if ($cl) { Set-MachineEnv 'DTU_MSVC_CL' $cl 'DTU_MSVC_CL' } else { Warn 'MSVC cl.exe (Hostx64\x64) not found -- host-side unit tests will degrade to "skipped" without it' } Info 'machine env vars are read when a PROCESS STARTS: the daemon must be restarted below' } # --------------------------------------------------------------------------- # 2. watchdog launcher (the actual fix for "daemon exits and nobody notices") # --------------------------------------------------------------------------- Info "writing watchdog launcher: $launcher" $template = @' # AUTO-GENERATED by scripts/ci_runner_bootstrap.ps1 -- do not edit by hand. # # gitea-runner `daemon` exits when its first Declare call fails (no retry, no # service support in v4), so a plain scheduled task is NOT enough: a Gitea # restart would take CI down until a human noticed. This loop # 1) waits until Gitea answers, then # 2) runs the daemon, and # 3) restarts it after a short pause whenever it exits. # Keep this file pure ASCII (PowerShell 5.1 decodes BOM-less files as CP936). $ErrorActionPreference = 'Continue' $runnerDir = '__RUNNER_DIR__' $runnerExe = Join-Path $runnerDir 'gitea-runner.exe' $runnerCfg = Join-Path $runnerDir 'config.yaml' $logDir = Join-Path $runnerDir 'log' $log = Join-Path $logDir 'runner-watchdog.log' $daemonLog = Join-Path $logDir 'runner-daemon.log' $probeUrl = '__GITEA_URL__/api/v1/version' $maxWaitSec = 1800 if (-not (Test-Path -LiteralPath $logDir)) { New-Item -ItemType Directory -Force -Path $logDir | Out-Null } function Log($m) { $line = '{0} [watchdog] {1}' -f (Get-Date -Format 'o'), $m Add-Content -LiteralPath $log -Value $line -Encoding ascii } while ($true) { $waited = 0 while ($true) { $ok = $false try { Invoke-WebRequest -Uri $probeUrl -UseBasicParsing -TimeoutSec 10 | Out-Null $ok = $true } catch { $ok = $false } if ($ok) { break } if ($waited -eq 0) { Log "waiting for Gitea at $probeUrl" } Start-Sleep -Seconds 15 $waited += 15 if ($waited -ge $maxWaitSec) { Log 'Gitea still not answering after 30m -- starting anyway so the real error lands in the log' break } } Log 'starting daemon' # MEASURED TRAP (2026-10-12, hit by this very script): in PowerShell 5.1 the # *>> redirection is implemented with Out-File, whose DEFAULT ENCODING IS # UTF-16LE. Appending that into a file whose other lines come from # Add-Content -Encoding ascii leaves ONE FILE WITH TWO ENCODINGS, and the # daemon lines read back as "t i m e = 2 0 2 6 ..." (a NUL after every byte). # Never rely on a redirection's default encoding -- state it explicitly. # utf8 (not ascii) because job/commit text can legitimately be non-ASCII. # MEASURED TRAP #2 (2026-10-12): the daemon writes its log to STDERR, and in # PS 5.1 a native `2>&1` merge wraps every stderr line into an ErrorRecord. # Out-File then renders "gitea-runner.exe : " plus a 4-line CategoryInfo # block -- measured: a 2-line message became 8 lines, so 3 of every 4 lines in # the daemon log were PowerShell noise, in the very file the docs tell people # to read first. Stringifying each pipeline item keeps the text and drops the # decoration (ErrorRecord.ToString() IS the original stderr line) and leaves # $LASTEXITCODE intact. Verified against a cmd.exe wrapper; same result, and # this form avoids cmd's quote-stripping rules for spaced paths. & $runnerExe daemon --config $runnerCfg 2>&1 | ForEach-Object { "$_" } | Out-File -FilePath $daemonLog -Append -Encoding utf8 Log ("daemon exited, exit code {0}; restarting in 10s" -f $LASTEXITCODE) Start-Sleep -Seconds 10 } '@ $content = $template.Replace('__RUNNER_DIR__', $RunnerDir).Replace('__GITEA_URL__', $GiteaUrl.TrimEnd('/')) $utf8NoBom = New-Object System.Text.UTF8Encoding($false) [System.IO.File]::WriteAllText($launcher, $content, $utf8NoBom) Ok 'launcher written' # --------------------------------------------------------------------------- # 3. scheduled task: start at boot, run as SYSTEM, restart if it dies # --------------------------------------------------------------------------- $existing = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue if ($existing) { Info "task $TaskName exists -- unregistering to re-register with the current settings" Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false } try { Register-ScheduledTask -TaskName $TaskName -Action $plan.Action -Trigger $plan.Trigger ` -Settings $plan.Settings -User 'SYSTEM' -RunLevel Highest ` -Description 'Gitea Actions runner watchdog (STM32F4-Base CI). Supervises the gitea-runner daemon: waits for Gitea, restarts the daemon when it exits. Generated by scripts/ci_runner_bootstrap.ps1.' | Out-Null } catch { Fail "Register-ScheduledTask failed: $($_.Exception.Message)" if ($existing) { Fail "the previous task was ALREADY removed -- fix the cause and re-run this script immediately" } Fail 'until this succeeds the runner has no auto-start and CI stays dead' exit 1 } Ok "scheduled task registered: $TaskName (at boot +1m, as SYSTEM, unlimited runtime, restart x999/1m)" # --------------------------------------------------------------------------- # 4. start it -- but clear stale instances first, or the new watchdog never gets # the .runner lock. # # MEASURED TRAP (2026-10-12): Unregister-ScheduledTask does NOT stop the process # the old instance had started. That orphaned watchdog keeps its daemon alive and # the daemon holds .runner.lock, so the freshly started watchdog fails every 10s: # level=error msg="another gitea-runner process is already using \".runner\"..." # Error: runner file is already locked by another process # The runner still shows ONLINE (the stale daemon serves fine), so CI looks healthy # while the log fills with a restart loop and the task supervises nothing at all. # --------------------------------------------------------------------------- function Stop-StaleRunnerInstances { # Supervisors first: killing the daemon alone lets a live watchdog respawn it # within 10s and the race starts over. $killed = 0 Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" -ErrorAction SilentlyContinue | Where-Object { $_.CommandLine -match 'run_runner_watchdog\.ps1' } | ForEach-Object { Info (" stopping stale watchdog pid {0}" -f $_.ProcessId) Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue $killed++ } Get-Process -Name 'gitea-runner' -ErrorAction SilentlyContinue | ForEach-Object { Info (" stopping stale daemon pid {0} (releases .runner.lock)" -f $_.Id) Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue $killed++ } return $killed } if ($StartNow) { if (-not (Test-Gitea $GiteaUrl)) { Warn "Gitea is not answering at $GiteaUrl right now -- the watchdog will keep waiting" } $stale = Stop-StaleRunnerInstances if ($stale -gt 0) { Warn "$stale stale process(es) stopped; the runner is offline for a few seconds" Start-Sleep -Seconds 2 } Start-ScheduledTask -TaskName $TaskName Start-Sleep -Seconds 12 # the watchdog probes Gitea, then execs the daemon $d = @(Get-Process -Name 'gitea-runner' -ErrorAction SilentlyContinue) if ($d.Count -eq 1) { Ok ("daemon up: pid {0} (single instance, holds .runner.lock)" -f $d[0].Id) } elseif ($d.Count -eq 0) { Warn 'no daemon yet -- the watchdog may still be waiting for Gitea; check the watchdog log' } else { Warn ("{0} daemons running -- only one can hold the lock; inspect and kill the extras" -f $d.Count) } Ok "daemon log: Get-Content '$(Join-Path $RunnerDir 'log\runner-daemon.log')' -Tail 20" Ok "watchdog log: Get-Content '$(Join-Path $RunnerDir 'log\runner-watchdog.log')' -Tail 20" Info 'verify on the web UI: repo -> Settings -> Actions -> Runners -> win-stm32-ci should turn Idle' } Ok 'done' exit 0