wandering1
2d18193d38
两个 job 的第 1 步原先是 `uses: actions/checkout@v4`。`uses:` 的解析发生在 Prepare all required actions 阶段——先于任何步骤、无条件联网（默认 github.com）， 且每个 job 各自解析进自己的 .cache/act/<hash>、跨 job 不复用。这条链在本机实测 不稳：run 36（10-21）/ 41（10-25）/ 42（10-26）三次都是 ~45 s 后整 job 死亡、 零钩子输出，而同一分钟手工 curl 同一地址返回 200——手工可达 ≠ CI 可用（解析器 跑在 SYSTEM 下）。Gitea 1.25.4 又去掉了自定义 DEFAULT_ACTIONS_URL，服务端没有 可拧的键，唯一归本仓库管的修法就是不再依赖它：两个 job 第 1 步换成原生 git 全量检出。 关键选择（依据与实测表见 docs/CI/远程CI完整指南.md 同文 §5.3，此处只记结论）： - 完整 refspec + 全量，不是 --depth 1：提交信息门禁要解 base..head 的 range； 且目标 sha 不在历史里就 ::error:: 硬失败，绝不默默构建一棵错的树； - clean -ffd 而非 -ffdx：-x 会删被忽略文件，而 runner 工作区跨运行复用它们； - GIT_TERMINAL_PROMPT=0 + 空 credential.helper：否则认证挑战会等人类输入， 把 30 分钟 job 超时白烧掉，表现得像 CI 挂了； - 令牌只走 per-invocation -c http.extraheader（Basic + x-access-token，与 checkout@v4 同拼法），不落 .git/config、不回显；仓库公开时匿名即可； - 不抓 tag：本仓库 tag 数 0、无脚本用 describe --tags（实测）。 另修 scripts/ci_runner_bootstrap.ps1 的 Find-MsvcCl。旧版只扫 %ProgramFiles% 派生根且取首个命中：本机 C 盘 VS2019 已于今日 01:00 卸载（C 盘 x64 cl.exe 实测 0 个），旧版因此是「找不到 → 只 Warn → 不写」，机器级变量一直指着那条失效路径， 而 harness 对「显式设置但无效」是硬失败、不做自动回退。现在改为 vswhere 取实例 （能看见非默认盘）+ 工具集版本降序 + -MsvcToolset 前缀钉住世代（默认 14.51.， 与覆盖率基线同版本；钉不上连 WARN 再回落最新，绝不静默）+ -MsvcLatest； -SetEnv 与 -Status 同步纳入 DTU_MS_COVERAGE。 验证（内层一律 powershell.exe 5.1，忠实复现 runner 执行方式）：临时工作区六项 A/B/C/D/E/F——E 是执行**从 ci.yml 里抽出的正文**，测的就是文件那份字节。 A rc=0、commits=665、refspec 完整、分离 HEAD；B 复用工作区后 dirty_files=0、 被忽略文件保留、17ece86..HEAD 解出 18 个提交；C rc=1 且 HEAD 不动；D 坏令牌 fatal: unable to get password 秒失败；令牌改造后再跑 ANON/BOGUS/ANON2=0/1/0。 ci_local 全量：L0.5 = 40 runner、3188/3188 断言、fail=0（MSVC 14.51）； test-coverage 与 ui-golden 均 Passed。.ps1 非 ASCII 字节 0、语法错误 0、无 BOM、 LF；ci.yml 无 BOM 无 CR、13 步、0 个 run 块含非 ASCII、uses: 计数 0。
655 lines
31 KiB
PowerShell
655 lines
31 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Make the Gitea Actions runner on this machine survive restarts, and give it
|
|
a toolchain the SYSTEM service account can actually see.
|
|
|
|
.DESCRIPTION
|
|
Why this file exists (2026-10-11 CI audit, the "many jobs never ran" bug):
|
|
|
|
Six consecutive runs (16..21, i.e. 12 jobs) produced ZERO log lines. The
|
|
runner `win-stm32-ci` was simply offline, and two independent causes made
|
|
that permanent:
|
|
|
|
1. NO SUPERVISION. gitea-runner v4.0.1 `daemon` has no service support
|
|
(v4 removed `daemon install`), and it EXITS when the first Declare
|
|
call fails. The runner log shows exactly that after a Gitea restart:
|
|
|
|
level=error msg="fail to invoke Declare" error="unavailable:
|
|
dial tcp 127.0.0.1:3001: connectex: ... actively refused it."
|
|
Error: unavailable: dial tcp 127.0.0.1:3001 ...
|
|
|
|
...and the process is gone. So one Gitea restart (or one reboot)
|
|
silently kills CI until a human notices days later. Queued runs then
|
|
get cancelled by the NEXT push, because ci.yml sets
|
|
`concurrency.cancel-in-progress: true` -- which is why the symptom
|
|
looks like "tasks vanish" rather than "tasks fail".
|
|
|
|
2. NO MACHINE-LEVEL TOOLCHAIN. The runner service runs as SYSTEM.
|
|
pre-commit / cppcheck were only reachable through the interactive
|
|
user's PATH, and no machine-level DTU_* variable was ever set, so
|
|
even a live runner died in the L0 environment check.
|
|
|
|
This script installs the missing supervision (a watchdog launcher + a
|
|
scheduled task) and, with -SetEnv, writes the machine-level DTU_* variables
|
|
by discovering the tools the same way CI does.
|
|
|
|
Run elevated, from the repo:
|
|
|
|
powershell -ExecutionPolicy Bypass -File scripts\ci_runner_bootstrap.ps1 -SetEnv -StartNow
|
|
|
|
.PARAMETER RunnerDir
|
|
Directory holding gitea-runner.exe / config.yaml / .runner.
|
|
|
|
.PARAMETER GiteaUrl
|
|
Instance address the runner registered against (must match .runner's
|
|
"address"; the runner talks to it directly, not through the public URL).
|
|
|
|
.PARAMETER TaskName
|
|
Scheduled-task name for the watchdog.
|
|
|
|
.PARAMETER SetEnv
|
|
Discover and write MACHINE-level DTU_PRECOMMIT_PYTHON / DTU_CPPCHECK /
|
|
DTU_MSVC_CL / DTU_MS_COVERAGE. Existing values are preserved unless
|
|
-Force. MSVC discovery goes through vswhere (so VS on a non-default
|
|
drive is found) and pins the toolset with -MsvcToolset, because the
|
|
coverage baseline was recorded with one specific MSVC version -- see
|
|
the remote-CI guide (docs/CI/) section 4.4.
|
|
|
|
.PARAMETER StartNow
|
|
Start the task after registering it.
|
|
|
|
.PARAMETER Status
|
|
Only report: task state, daemon process, Gitea reachability.
|
|
|
|
.PARAMETER DryRun
|
|
Build and validate the action / trigger / settings objects and print what
|
|
WOULD be done, writing and registering nothing. This is also the only way
|
|
to verify this script without elevation.
|
|
|
|
.PARAMETER Force
|
|
With -SetEnv, overwrite existing machine-level values.
|
|
|
|
.PARAMETER MsvcToolset
|
|
Toolset version PREFIX to prefer when writing DTU_MSVC_CL (default
|
|
'14.51.'). 'newest available' is deliberately NOT the rule: the host
|
|
unit-test coverage baseline was measured with MSVC 14.51 and the
|
|
denominator changes with the compiler generation. When no toolset
|
|
matches the prefix the script warns and falls back to the newest.
|
|
|
|
.PARAMETER MsvcLatest
|
|
Ignore -MsvcToolset and take the newest MSVC toolset found.
|
|
|
|
.OUTPUTS
|
|
Progress lines. Exit codes: 0 = ok, 1 = precondition failed, 2 = not elevated.
|
|
|
|
.NOTES
|
|
Keep this file pure ASCII (PowerShell 5.1 decodes BOM-less files as CP936).
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[string]$RunnerDir = 'D:\Docker\act_runner',
|
|
[string]$GiteaUrl = 'http://127.0.0.1:3001',
|
|
[string]$TaskName = 'GiteaActionsRunner',
|
|
[switch]$SetEnv,
|
|
[switch]$StartNow,
|
|
[switch]$Status,
|
|
[switch]$DryRun,
|
|
[switch]$Force,
|
|
[string]$MsvcToolset = '14.51.',
|
|
[switch]$MsvcLatest
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
function Info($m) { Write-Host "[INFO] $m" -ForegroundColor Cyan }
|
|
function Ok($m) { Write-Host "[ OK ] $m" -ForegroundColor Green }
|
|
function Warn($m) { Write-Host "[WARN] $m" -ForegroundColor Yellow }
|
|
function Fail($m) { Write-Host "[FAIL] $m" -ForegroundColor Red }
|
|
|
|
function Test-Admin {
|
|
$id = [Security.Principal.WindowsIdentity]::GetCurrent()
|
|
(New-Object Security.Principal.WindowsPrincipal($id)
|
|
).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
|
|
}
|
|
|
|
function Test-Gitea([string]$url) {
|
|
try {
|
|
Invoke-WebRequest -Uri "$url/api/v1/version" -UseBasicParsing -TimeoutSec 10 | Out-Null
|
|
return $true
|
|
}
|
|
catch { return $false }
|
|
}
|
|
|
|
function Get-MachineEnv([string]$name) {
|
|
$v = [Environment]::GetEnvironmentVariable($name, 'Machine')
|
|
if ([string]::IsNullOrWhiteSpace($v)) { return $null }
|
|
return $v
|
|
}
|
|
|
|
function Set-MachineEnv([string]$name, [string]$value, [string]$label) {
|
|
$cur = Get-MachineEnv $name
|
|
if ($cur) {
|
|
if ($cur -ieq $value) { Ok "$label already set: $cur"; return }
|
|
if (-not $Force) {
|
|
Warn "$label already set to a different value: $cur (use -Force to overwrite)"
|
|
return
|
|
}
|
|
Warn "$label overwriting $cur -> $value"
|
|
}
|
|
[Environment]::SetEnvironmentVariable($name, $value, 'Machine')
|
|
Ok "$label set (MACHINE scope): $value"
|
|
}
|
|
|
|
function Find-Cppcheck {
|
|
$pf = $env:ProgramFiles
|
|
$pf86 = ${env:ProgramFiles(x86)}
|
|
if ([string]::IsNullOrWhiteSpace($pf)) { $pf = 'C:\Program Files' }
|
|
if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' }
|
|
foreach ($c in @((Join-Path $pf 'Cppcheck\cppcheck.exe'),
|
|
(Join-Path $pf86 'Cppcheck\cppcheck.exe'))) {
|
|
if (Test-Path -LiteralPath $c) { return $c }
|
|
}
|
|
$cmd = Get-Command cppcheck -ErrorAction SilentlyContinue
|
|
if ($cmd) { return $cmd.Source }
|
|
return $null
|
|
}
|
|
|
|
function Find-VsInstances {
|
|
# VS install roots, newest first, via vswhere. This matters because
|
|
# vswhere knows about NON-DEFAULT INSTALL DRIVES: this machine keeps
|
|
# VS 18 BuildTools under E:, which a %ProgramFiles%-derived probe can
|
|
# never see. Returns an empty array when vswhere is unusable so the
|
|
# caller can fall back to the classic (incomplete) root scan.
|
|
$vswhere = $env:DTU_VSWHERE
|
|
if ([string]::IsNullOrWhiteSpace($vswhere)) {
|
|
$pf86 = [Environment]::GetFolderPath('ProgramFilesX86')
|
|
if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' }
|
|
$vswhere = Join-Path $pf86 'Microsoft Visual Studio\Installer\vswhere.exe'
|
|
}
|
|
if (-not (Test-Path -LiteralPath $vswhere)) { return @() }
|
|
$raw = & $vswhere -all -products * -format json 2>$null
|
|
if ($LASTEXITCODE -ne 0 -or -not $raw) { return @() }
|
|
$list = @()
|
|
foreach ($i in @($raw | ConvertFrom-Json)) {
|
|
if (-not $i -or -not $i.installationPath) { continue }
|
|
if (-not (Test-Path -LiteralPath $i.installationPath)) { continue }
|
|
$v = [version]'0.0'
|
|
if ($i.installationVersion) { [void][version]::TryParse([string]$i.installationVersion, [ref]$v) }
|
|
$list += [pscustomobject]@{ Root = [string]$i.installationPath; Version = $v }
|
|
}
|
|
return @($list | Sort-Object -Property Version -Descending)
|
|
}
|
|
|
|
function Find-MsvcClCandidates {
|
|
# Every x64-host cl.exe on this machine, tagged with its toolset version.
|
|
# Comparison operators in PowerShell are case-insensitive by default, so
|
|
# -match / -notmatch cover HostX64 / Hostx64 spelling differences.
|
|
$cands = @()
|
|
$roots = @(Find-VsInstances | ForEach-Object { $_.Root })
|
|
if ($roots.Count -eq 0) {
|
|
Warn 'vswhere unusable -- falling back to %ProgramFiles% roots; VS installs on other drives are INVISIBLE to that probe'
|
|
$pf = [Environment]::GetFolderPath('ProgramFiles')
|
|
$pf86 = [Environment]::GetFolderPath('ProgramFilesX86')
|
|
if ([string]::IsNullOrWhiteSpace($pf)) { $pf = 'C:\Program Files' }
|
|
if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' }
|
|
foreach ($year in @('18', '2026', '2022', '2019')) {
|
|
foreach ($edition in @('BuildTools', 'Community', 'Professional', 'Enterprise')) {
|
|
foreach ($base in @($pf, $pf86)) {
|
|
$r = Join-Path $base ('Microsoft Visual Studio\' + $year + '\' + $edition)
|
|
if (Test-Path -LiteralPath $r) { $roots += $r }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
foreach ($r in $roots) {
|
|
$msvc = Join-Path $r 'VC\Tools\MSVC'
|
|
if (-not (Test-Path -LiteralPath $msvc)) { continue }
|
|
foreach ($f in (Get-ChildItem -LiteralPath $msvc -Recurse -Filter 'cl.exe' -ErrorAction SilentlyContinue)) {
|
|
$full = $f.FullName
|
|
if ($full -notmatch '\\bin\\Hostx64\\x64\\cl\.exe$') { continue }
|
|
$toolset = ''
|
|
if ($full -match '\\VC\\Tools\\MSVC\\([^\\]+)\\bin\\') { $toolset = $Matches[1] }
|
|
$ver = [version]'0.0'
|
|
[void][version]::TryParse($toolset, [ref]$ver)
|
|
$cands += [pscustomobject]@{ Cl = $full; Toolset = $toolset; Version = $ver; VsRoot = $r }
|
|
}
|
|
}
|
|
return @($cands)
|
|
}
|
|
|
|
function Find-MsvcCl {
|
|
# Rewritten 2026-10-27. The previous version walked a hard-coded root list
|
|
# derived from %ProgramFiles% and returned the FIRST hit: with VS2019
|
|
# (MSVC 14.29) and a newer VS both present it handed back 14.29, and it
|
|
# could not see the E: install at all. "First hit" is never right for a
|
|
# compiler: the coverage baseline (scripts/test_coverage_baseline.json)
|
|
# was recorded with MSVC 14.51 and the instrumented denominator moves when
|
|
# the compiler generation moves (measured: App/util/sha256.c is 105 lines
|
|
# under gcc vs 114 under MSVC -- see the host coverage policy doc, 2.1).
|
|
# Hence the default is a PINNED toolset prefix, not "newest".
|
|
param([string]$ToolsetPrefix = '', [switch]$Latest)
|
|
$cands = @(Find-MsvcClCandidates)
|
|
if ($cands.Count -eq 0) { return $null }
|
|
$sorted = @($cands | Sort-Object -Property Version -Descending)
|
|
$seen = (@($sorted | ForEach-Object { $_.Toolset } | Select-Object -Unique)) -join ', '
|
|
if (-not $Latest -and $ToolsetPrefix) {
|
|
$hit = @($sorted | Where-Object { $_.Toolset -like ($ToolsetPrefix + '*') })
|
|
if ($hit.Count -gt 0) {
|
|
Info ('MSVC pinned to toolset prefix ' + $ToolsetPrefix + '* -> ' + $hit[0].Toolset + ' (' + $hit.Count + ' of ' + $cands.Count + ' candidate(s); toolsets found: ' + $seen + ')')
|
|
return $hit[0].Cl
|
|
}
|
|
Warn ('no MSVC toolset matching prefix ' + $ToolsetPrefix + ' on this machine (found: ' + $seen + ')')
|
|
Warn 'falling back to the NEWEST toolset -- that changes the coverage denominator; read the remote-CI guide (docs/CI/) section 4.4 before accepting it'
|
|
}
|
|
Info ('using newest MSVC toolset ' + $sorted[0].Toolset + ' of ' + $cands.Count + ' candidate(s); toolsets found: ' + $seen)
|
|
return $sorted[0].Cl
|
|
}
|
|
|
|
function Find-CoverageConsole {
|
|
# Microsoft.CodeCoverage.Console.exe ships ONLY inside Visual Studio
|
|
# (measured 2026-10-27: NuGet candidates contain no such .exe, or only the
|
|
# legacy Team Tools CodeCoverage.exe whose CLI differs). Its location
|
|
# inside a VS install is a fixed relative path:
|
|
param([string]$ClPath = '')
|
|
$rel = 'Common7\IDE\Extensions\Microsoft\CodeCoverage.Console\Microsoft.CodeCoverage.Console.exe'
|
|
$roots = @()
|
|
if ($ClPath) {
|
|
$vsRoot = ''
|
|
# cl.exe = <root>\VC\Tools\MSVC\<ver>\bin\Hostx64\x64\cl.exe
|
|
if ($ClPath -match '^(.*)\\VC\\Tools\\MSVC\\') { $vsRoot = $Matches[1] }
|
|
if ($vsRoot) { $roots += $vsRoot }
|
|
}
|
|
foreach ($i in @(Find-VsInstances)) { $roots += $i.Root }
|
|
foreach ($r in ($roots | Where-Object { $_ } | Select-Object -Unique)) {
|
|
$cand = Join-Path $r $rel
|
|
if (Test-Path -LiteralPath $cand) { return $cand }
|
|
}
|
|
return $null
|
|
}
|
|
|
|
# Builds the task objects in ONE place so -DryRun validates exactly what the
|
|
# real run registers.
|
|
#
|
|
# MEASURED TRAP (2026-10-11, hit on the user's machine): `StartWhenAvailable`
|
|
# is a TASK SETTINGS property (MSFT_TaskSettings, inherited by
|
|
# MSFT_TaskSettings3) -- it does NOT exist on triggers. Assigning it to the
|
|
# trigger aborts the script with a SetValueInvocationException along the lines
|
|
# of "setting StartWhenAvailable threw an exception: property
|
|
# 'StartWhenAvailable' is not found on this object" (message localized),
|
|
# and, worse, that happened AFTER the old task had already been unregistered,
|
|
# leaving CI with no task at all. Hence: build everything first, only then
|
|
# unregister the old task.
|
|
function New-WatchdogTaskPlan {
|
|
param([string]$Launcher, [string]$WorkDir)
|
|
|
|
$action = New-ScheduledTaskAction `
|
|
-Execute 'powershell.exe' `
|
|
-Argument "-NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File `"$Launcher`"" `
|
|
-WorkingDirectory $WorkDir
|
|
|
|
# Boot trigger + a 1 minute grace period: Gitea may still be starting when
|
|
# the session-0 boot fires (and the watchdog waits for it anyway).
|
|
$trigger = New-ScheduledTaskTrigger -AtStartup
|
|
$trigger.Delay = 'PT1M'
|
|
|
|
$settings = New-ScheduledTaskSettingsSet `
|
|
-AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
|
|
-ExecutionTimeLimit ([TimeSpan]::Zero) `
|
|
-RestartCount 999 -RestartInterval (New-TimeSpan -Minutes 1) `
|
|
-MultipleInstances IgnoreNew
|
|
# StartWhenAvailable also covers "the machine was off when this was missed".
|
|
$settings.StartWhenAvailable = $true
|
|
|
|
return [pscustomobject]@{ Action = $action; Trigger = $trigger; Settings = $settings }
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# -Status: read-only report
|
|
# ---------------------------------------------------------------------------
|
|
if ($Status) {
|
|
Info "task $TaskName"
|
|
$t = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
|
if ($t) {
|
|
Ok ("task state: {0}" -f $t.State)
|
|
$li = Get-ScheduledTaskInfo -TaskName $TaskName -ErrorAction SilentlyContinue
|
|
if ($li) {
|
|
Info ("last run: {0} | last result: {1} | next run: {2}" -f `
|
|
$li.LastRunTime, ('0x{0:X}' -f $li.LastTaskResult), $li.NextRunTime)
|
|
}
|
|
}
|
|
else {
|
|
Warn "task $TaskName not found"
|
|
if (-not (Test-Admin)) {
|
|
Warn ' this session is NOT elevated: a non-admin session cannot always see'
|
|
Warn ' every task, so re-run -Status elevated before concluding it is missing'
|
|
}
|
|
}
|
|
|
|
$proc = @(Get-Process | Where-Object { $_.ProcessName -match 'gitea-runner|act_runner' })
|
|
if ($proc.Count -eq 1) {
|
|
Ok ("daemon running: pid {0}" -f $proc[0].Id)
|
|
} elseif ($proc.Count -gt 1) {
|
|
Warn (" {0} daemon processes -- only ONE can hold .runner.lock, the rest are in a restart loop" -f $proc.Count)
|
|
Warn ' fix: re-run this script elevated with -StartNow (it clears stale instances first)'
|
|
} else { Warn 'daemon not running' }
|
|
|
|
# A watchdog that cannot get the lock fails every 10s while the runner still
|
|
# reports ONLINE, so the loop is invisible from the web UI. Count watchdogs and
|
|
# grep the daemon log for the lock error (measured 2026-10-12).
|
|
# MEASURED TRAP #3 (2026-10-12): a NON-ELEVATED session cannot read SYSTEM
|
|
# process command lines -- CommandLine comes back $null, so the filter below
|
|
# silently matches NOTHING and reporting "no watchdog" would be a lie (this
|
|
# bit us twice while diagnosing). Say what we cannot see instead.
|
|
if (-not (Test-Admin)) {
|
|
Info ' watchdog count skipped: needs elevation (SYSTEM command lines are hidden)'
|
|
} else {
|
|
$wd = @(Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" -ErrorAction SilentlyContinue |
|
|
Where-Object { $_.CommandLine -match 'run_runner_watchdog\.ps1' })
|
|
if ($wd.Count -gt 1) {
|
|
Warn (" {0} watchdog processes (pids {1}) -- an old one survived a task re-registration" -f `
|
|
$wd.Count, (($wd | ForEach-Object { $_.ProcessId }) -join ','))
|
|
Warn ' and is racing the new one for .runner.lock; re-run elevated with -StartNow'
|
|
} elseif ($wd.Count -eq 1) {
|
|
Ok ("watchdog running: pid {0}" -f $wd[0].ProcessId)
|
|
} else {
|
|
Warn 'no watchdog process found (task not running?)'
|
|
}
|
|
}
|
|
# Restart-loop detection. Do NOT just grep the log tail: the file is never
|
|
# rotated, so old failures stay in the last lines forever and a HEALTHY runner
|
|
# keeps "reporting" a stale error (measured false positive 2026-10-12). The
|
|
# reliable signal is a daemon that keeps dying young -- a process alive longer
|
|
# than the 10s restart interval cannot be inside that loop. CIM CreationDate is
|
|
# readable without elevation; Process.StartTime is NOT for SYSTEM processes.
|
|
$dmon = Get-CimInstance Win32_Process -Filter "Name='gitea-runner.exe'" -ErrorAction SilentlyContinue |
|
|
Select-Object -First 1
|
|
$upSec = if ($dmon) { [int]((Get-Date) - $dmon.CreationDate).TotalSeconds } else { -1 }
|
|
if ($upSec -ge 90) {
|
|
Ok ("daemon up for {0} min -- no restart loop" -f [int]($upSec / 60))
|
|
} else {
|
|
$dlog = Join-Path $RunnerDir 'log\runner-daemon.log'
|
|
if (Test-Path -LiteralPath $dlog) {
|
|
$tail = Get-Content -LiteralPath $dlog -Tail 40 -Encoding UTF8 -ErrorAction SilentlyContinue
|
|
if ($tail -match 'already locked by another process') {
|
|
Warn ' daemon keeps dying young and the log shows ".runner is already locked'
|
|
Warn ' by another process" -- a stale instance holds the lock; re-run with -StartNow'
|
|
}
|
|
}
|
|
}
|
|
|
|
if (Test-Gitea $GiteaUrl) { Ok "Gitea reachable at $GiteaUrl" }
|
|
else { Fail "Gitea NOT reachable at $GiteaUrl" }
|
|
|
|
foreach ($n in @('DTU_PRECOMMIT_PYTHON', 'DTU_CPPCHECK', 'DTU_MSVC_CL', 'DTU_MS_COVERAGE')) {
|
|
$v = Get-MachineEnv $n
|
|
if ($v) { Ok ("MACHINE {0} = {1}" -f $n, $v) }
|
|
else { Warn "MACHINE $n not set (CI auto-discovery still applies; see scripts/ci_find_python.ps1)" }
|
|
}
|
|
exit 0
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Preconditions
|
|
# ---------------------------------------------------------------------------
|
|
if (-not $DryRun -and -not (Test-Admin)) {
|
|
Fail 'not elevated -- rerun from an elevated PowerShell (this writes a scheduled task and machine env vars)'
|
|
Fail 'use -DryRun to validate the whole plan (action/trigger/settings) without elevation'
|
|
exit 2
|
|
}
|
|
|
|
$exe = Join-Path $RunnerDir 'gitea-runner.exe'
|
|
$cfg = Join-Path $RunnerDir 'config.yaml'
|
|
$regFile = Join-Path $RunnerDir '.runner'
|
|
$launcher = Join-Path $RunnerDir 'run_runner_watchdog.ps1'
|
|
|
|
foreach ($f in @($exe, $cfg)) {
|
|
if (-not (Test-Path -LiteralPath $f)) { Fail "missing file: $f"; exit 1 }
|
|
}
|
|
if (-not (Test-Path -LiteralPath $regFile)) {
|
|
Fail "not registered: $regFile is missing"
|
|
Info 'register first, e.g.:'
|
|
Write-Host " & '$exe' register --no-interactive --instance '$GiteaUrl' --token <RegistrationToken> --name win-stm32-ci --labels 'windows:host'"
|
|
exit 1
|
|
}
|
|
|
|
# The daemon must talk to the address baked into .runner, not the public URL.
|
|
$registeredAddress = $null
|
|
try {
|
|
$registeredAddress = (Get-Content -LiteralPath $regFile -Raw | ConvertFrom-Json).address
|
|
}
|
|
catch { Warn "cannot parse $regFile : $_" }
|
|
if ($registeredAddress -and $registeredAddress.TrimEnd('/') -ne $GiteaUrl.TrimEnd('/')) {
|
|
Warn ".runner address is $registeredAddress but -GiteaUrl is $GiteaUrl -- using the .runner value for the readiness probe"
|
|
$GiteaUrl = $registeredAddress
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 0. build the task plan up front; -DryRun stops here without writing anything
|
|
# ---------------------------------------------------------------------------
|
|
$plan = New-WatchdogTaskPlan -Launcher $launcher -WorkDir $RunnerDir
|
|
|
|
if ($DryRun) {
|
|
Info 'dry run -- writing and registering NOTHING'
|
|
Info ("would write launcher : {0}" -f $launcher)
|
|
Info ("would run : {0} {1}" -f $plan.Action.Execute, $plan.Action.Arguments)
|
|
Info ("would trigger : {0} delay={1}" -f $plan.Trigger.CimClass.CimClassName, $plan.Trigger.Delay)
|
|
Info ("would settings : StartWhenAvailable={0} RestartCount={1} RestartInterval={2} ExecTimeLimit={3} MultipleInstances={4}" -f `
|
|
$plan.Settings.StartWhenAvailable, $plan.Settings.RestartCount, $plan.Settings.RestartInterval,
|
|
$plan.Settings.ExecutionTimeLimit, $plan.Settings.MultipleInstances)
|
|
Info 'would register as : user SYSTEM, RunLevel Highest'
|
|
if (Test-Admin) { Ok 'elevation: OK (admin)' }
|
|
else { Warn 'elevation: NOT admin -- the real run must be started from an elevated shell' }
|
|
if (Test-Gitea $GiteaUrl) { Ok "Gitea reachable at $GiteaUrl" }
|
|
else { Warn "Gitea not reachable at $GiteaUrl -- the watchdog would keep waiting" }
|
|
exit 0
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. machine-level toolchain (SYSTEM cannot see the interactive user's PATH)
|
|
# ---------------------------------------------------------------------------
|
|
if ($SetEnv) {
|
|
Info 'discovering toolchain for MACHINE scope ...'
|
|
|
|
$findPy = Join-Path $PSScriptRoot 'ci_find_python.ps1'
|
|
if (Test-Path -LiteralPath $findPy) {
|
|
$py = & $findPy -NeedPreCommit
|
|
if ($LASTEXITCODE -eq 0 -and $py) {
|
|
Set-MachineEnv 'DTU_PRECOMMIT_PYTHON' ("$py".Trim()) 'DTU_PRECOMMIT_PYTHON'
|
|
}
|
|
else {
|
|
Warn 'no interpreter with pre-commit found -- install pre-commit into a machine-wide interpreter first'
|
|
}
|
|
}
|
|
else { Warn "not found: $findPy (skipping DTU_PRECOMMIT_PYTHON)" }
|
|
|
|
$cpp = Find-Cppcheck
|
|
if ($cpp) { Set-MachineEnv 'DTU_CPPCHECK' $cpp 'DTU_CPPCHECK' }
|
|
else { Warn 'cppcheck not found (install Cppcheck.Cppcheck, or set DTU_CPPCHECK manually)' }
|
|
|
|
$cl = Find-MsvcCl -ToolsetPrefix $MsvcToolset -Latest:$MsvcLatest
|
|
if ($cl) {
|
|
Set-MachineEnv 'DTU_MSVC_CL' $cl 'DTU_MSVC_CL'
|
|
# Pin the coverage collector too, and pin it INSIDE the same VS as the
|
|
# chosen cl.exe: the gate derives it from the compiler ancestor anyway,
|
|
# but an explicit machine value also covers the case where somebody
|
|
# later points DTU_MSVC_CL at another install. Both vars must be
|
|
# changed together (docs/CI/ guide, section 4.4).
|
|
$cov = Find-CoverageConsole -ClPath $cl
|
|
if ($cov) { Set-MachineEnv 'DTU_MS_COVERAGE' $cov 'DTU_MS_COVERAGE' }
|
|
else {
|
|
Warn 'Microsoft.CodeCoverage.Console.exe NOT found -- the test-coverage gate FAILS with [ENV] exit 2 (by the 2026-10-24 ruling it does not skip).'
|
|
Warn 'install VS BuildTools 17.9+/18.x including Microsoft.VisualStudio.PackageGroup.TestTools.CodeCoverage, then re-run -SetEnv'
|
|
}
|
|
}
|
|
else { Warn 'MSVC cl.exe (Hostx64\x64) not found -- L0.5 will degrade to "skipped" without it' }
|
|
|
|
Info 'machine env vars are read when a PROCESS STARTS: the daemon must be restarted below'
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. watchdog launcher (the actual fix for "daemon exits and nobody notices")
|
|
# ---------------------------------------------------------------------------
|
|
Info "writing watchdog launcher: $launcher"
|
|
|
|
$template = @'
|
|
# AUTO-GENERATED by scripts/ci_runner_bootstrap.ps1 -- do not edit by hand.
|
|
#
|
|
# gitea-runner `daemon` exits when its first Declare call fails (no retry, no
|
|
# service support in v4), so a plain scheduled task is NOT enough: a Gitea
|
|
# restart would take CI down until a human noticed. This loop
|
|
# 1) waits until Gitea answers, then
|
|
# 2) runs the daemon, and
|
|
# 3) restarts it after a short pause whenever it exits.
|
|
# Keep this file pure ASCII (PowerShell 5.1 decodes BOM-less files as CP936).
|
|
$ErrorActionPreference = 'Continue'
|
|
|
|
$runnerDir = '__RUNNER_DIR__'
|
|
$runnerExe = Join-Path $runnerDir 'gitea-runner.exe'
|
|
$runnerCfg = Join-Path $runnerDir 'config.yaml'
|
|
$logDir = Join-Path $runnerDir 'log'
|
|
$log = Join-Path $logDir 'runner-watchdog.log'
|
|
$daemonLog = Join-Path $logDir 'runner-daemon.log'
|
|
$probeUrl = '__GITEA_URL__/api/v1/version'
|
|
$maxWaitSec = 1800
|
|
|
|
if (-not (Test-Path -LiteralPath $logDir)) {
|
|
New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
|
}
|
|
|
|
function Log($m) {
|
|
$line = '{0} [watchdog] {1}' -f (Get-Date -Format 'o'), $m
|
|
Add-Content -LiteralPath $log -Value $line -Encoding ascii
|
|
}
|
|
|
|
while ($true) {
|
|
$waited = 0
|
|
while ($true) {
|
|
$ok = $false
|
|
try {
|
|
Invoke-WebRequest -Uri $probeUrl -UseBasicParsing -TimeoutSec 10 | Out-Null
|
|
$ok = $true
|
|
}
|
|
catch { $ok = $false }
|
|
if ($ok) { break }
|
|
if ($waited -eq 0) { Log "waiting for Gitea at $probeUrl" }
|
|
Start-Sleep -Seconds 15
|
|
$waited += 15
|
|
if ($waited -ge $maxWaitSec) {
|
|
Log 'Gitea still not answering after 30m -- starting anyway so the real error lands in the log'
|
|
break
|
|
}
|
|
}
|
|
|
|
Log 'starting daemon'
|
|
# MEASURED TRAP (2026-10-12, hit by this very script): in PowerShell 5.1 the
|
|
# *>> redirection is implemented with Out-File, whose DEFAULT ENCODING IS
|
|
# UTF-16LE. Appending that into a file whose other lines come from
|
|
# Add-Content -Encoding ascii leaves ONE FILE WITH TWO ENCODINGS, and the
|
|
# daemon lines read back as "t i m e = 2 0 2 6 ..." (a NUL after every byte).
|
|
# Never rely on a redirection's default encoding -- state it explicitly.
|
|
# utf8 (not ascii) because job/commit text can legitimately be non-ASCII.
|
|
# MEASURED TRAP #2 (2026-10-12): the daemon writes its log to STDERR, and in
|
|
# PS 5.1 a native `2>&1` merge wraps every stderr line into an ErrorRecord.
|
|
# Out-File then renders "gitea-runner.exe : <line>" plus a 4-line CategoryInfo
|
|
# block -- measured: a 2-line message became 8 lines, so 3 of every 4 lines in
|
|
# the daemon log were PowerShell noise, in the very file the docs tell people
|
|
# to read first. Stringifying each pipeline item keeps the text and drops the
|
|
# decoration (ErrorRecord.ToString() IS the original stderr line) and leaves
|
|
# $LASTEXITCODE intact. Verified against a cmd.exe wrapper; same result, and
|
|
# this form avoids cmd's quote-stripping rules for spaced paths.
|
|
& $runnerExe daemon --config $runnerCfg 2>&1 |
|
|
ForEach-Object { "$_" } |
|
|
Out-File -FilePath $daemonLog -Append -Encoding utf8
|
|
Log ("daemon exited, exit code {0}; restarting in 10s" -f $LASTEXITCODE)
|
|
Start-Sleep -Seconds 10
|
|
}
|
|
'@
|
|
|
|
$content = $template.Replace('__RUNNER_DIR__', $RunnerDir).Replace('__GITEA_URL__', $GiteaUrl.TrimEnd('/'))
|
|
$utf8NoBom = New-Object System.Text.UTF8Encoding($false)
|
|
[System.IO.File]::WriteAllText($launcher, $content, $utf8NoBom)
|
|
Ok 'launcher written'
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. scheduled task: start at boot, run as SYSTEM, restart if it dies
|
|
# ---------------------------------------------------------------------------
|
|
$existing = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
|
if ($existing) {
|
|
Info "task $TaskName exists -- unregistering to re-register with the current settings"
|
|
Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false
|
|
}
|
|
|
|
try {
|
|
Register-ScheduledTask -TaskName $TaskName -Action $plan.Action -Trigger $plan.Trigger `
|
|
-Settings $plan.Settings -User 'SYSTEM' -RunLevel Highest `
|
|
-Description 'Gitea Actions runner watchdog (STM32F4-Base CI). Supervises the gitea-runner daemon: waits for Gitea, restarts the daemon when it exits. Generated by scripts/ci_runner_bootstrap.ps1.' |
|
|
Out-Null
|
|
}
|
|
catch {
|
|
Fail "Register-ScheduledTask failed: $($_.Exception.Message)"
|
|
if ($existing) {
|
|
Fail "the previous task was ALREADY removed -- fix the cause and re-run this script immediately"
|
|
}
|
|
Fail 'until this succeeds the runner has no auto-start and CI stays dead'
|
|
exit 1
|
|
}
|
|
Ok "scheduled task registered: $TaskName (at boot +1m, as SYSTEM, unlimited runtime, restart x999/1m)"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. start it -- but clear stale instances first, or the new watchdog never gets
|
|
# the .runner lock.
|
|
#
|
|
# MEASURED TRAP (2026-10-12): Unregister-ScheduledTask does NOT stop the process
|
|
# the old instance had started. That orphaned watchdog keeps its daemon alive and
|
|
# the daemon holds .runner.lock, so the freshly started watchdog fails every 10s:
|
|
# level=error msg="another gitea-runner process is already using \".runner\"..."
|
|
# Error: runner file is already locked by another process
|
|
# The runner still shows ONLINE (the stale daemon serves fine), so CI looks healthy
|
|
# while the log fills with a restart loop and the task supervises nothing at all.
|
|
# ---------------------------------------------------------------------------
|
|
function Stop-StaleRunnerInstances {
|
|
# Supervisors first: killing the daemon alone lets a live watchdog respawn it
|
|
# within 10s and the race starts over.
|
|
$killed = 0
|
|
Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" -ErrorAction SilentlyContinue |
|
|
Where-Object { $_.CommandLine -match 'run_runner_watchdog\.ps1' } |
|
|
ForEach-Object {
|
|
Info (" stopping stale watchdog pid {0}" -f $_.ProcessId)
|
|
Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue
|
|
$killed++
|
|
}
|
|
Get-Process -Name 'gitea-runner' -ErrorAction SilentlyContinue | ForEach-Object {
|
|
Info (" stopping stale daemon pid {0} (releases .runner.lock)" -f $_.Id)
|
|
Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue
|
|
$killed++
|
|
}
|
|
return $killed
|
|
}
|
|
|
|
if ($StartNow) {
|
|
if (-not (Test-Gitea $GiteaUrl)) {
|
|
Warn "Gitea is not answering at $GiteaUrl right now -- the watchdog will keep waiting"
|
|
}
|
|
$stale = Stop-StaleRunnerInstances
|
|
if ($stale -gt 0) {
|
|
Warn "$stale stale process(es) stopped; the runner is offline for a few seconds"
|
|
Start-Sleep -Seconds 2
|
|
}
|
|
Start-ScheduledTask -TaskName $TaskName
|
|
Start-Sleep -Seconds 12 # the watchdog probes Gitea, then execs the daemon
|
|
$d = @(Get-Process -Name 'gitea-runner' -ErrorAction SilentlyContinue)
|
|
if ($d.Count -eq 1) {
|
|
Ok ("daemon up: pid {0} (single instance, holds .runner.lock)" -f $d[0].Id)
|
|
} elseif ($d.Count -eq 0) {
|
|
Warn 'no daemon yet -- the watchdog may still be waiting for Gitea; check the watchdog log'
|
|
} else {
|
|
Warn ("{0} daemons running -- only one can hold the lock; inspect and kill the extras" -f $d.Count)
|
|
}
|
|
Ok "daemon log: Get-Content '$(Join-Path $RunnerDir 'log\runner-daemon.log')' -Tail 20"
|
|
Ok "watchdog log: Get-Content '$(Join-Path $RunnerDir 'log\runner-watchdog.log')' -Tail 20"
|
|
Info 'verify on the web UI: repo -> Settings -> Actions -> Runners -> win-stm32-ci should turn Idle'
|
|
}
|
|
|
|
Ok 'done'
|
|
exit 0
|