wandering1
20ee35b76a
1) runner 掉线：run 16~21 共 12 个 job 零日志。gitea-runner v4 的
daemon 无服务支持、首次 Declare 失败即退出进程，Gitea/机器重启
一次 CI 就静默停摆；叠加 cancel-in-progress 后症状是"任务凭空
消失"。新增 scripts/ci_runner_bootstrap.ps1：等 Gitea 就绪 +
崩溃自拉起看门狗 + 开机计划任务 + -SetEnv 写 MACHINE 级 DTU_*。
2) L0 自 33a4aa4 起在 runner 上必挂：Python 候选被收窄成 PATH python
(msys64，无 pre-commit) + 仓库 .venv(gitignore)，cppcheck 目录兜底
亦被删，而机器级 DTU_* 从未设置。探测收进 ci_find_python.ps1
（两 job 共用、可离线自测、恢复机器级安装兜底）。
3) gitlint 声明 stages:[commit-msg]，而 --all-files 只跑 pre-commit
stage → 17 个钩子定义 CI 只报 16 个且全绿。新增
ci_gitlint_range.ps1 逐条校验本次范围，ci_local.py 增 L0b 段。
另：GITHUB_ENV/PATH/OUTPUT 写入显式 -Encoding ascii（PS 5.1 的
Out-File -Append 默认 UTF-16LE，会把带 NUL 的行追加进 UTF-8 文件）
353 lines
14 KiB
PowerShell
353 lines
14 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Make the Gitea Actions runner on this machine survive restarts, and give it
|
|
a toolchain the SYSTEM service account can actually see.
|
|
|
|
.DESCRIPTION
|
|
Why this file exists (2026-10-11 CI audit, the "many jobs never ran" bug):
|
|
|
|
Six consecutive runs (16..21, i.e. 12 jobs) produced ZERO log lines. The
|
|
runner `win-stm32-ci` was simply offline, and two independent causes made
|
|
that permanent:
|
|
|
|
1. NO SUPERVISION. gitea-runner v4.0.1 `daemon` has no service support
|
|
(v4 removed `daemon install`), and it EXITS when the first Declare
|
|
call fails. The runner log shows exactly that after a Gitea restart:
|
|
|
|
level=error msg="fail to invoke Declare" error="unavailable:
|
|
dial tcp 127.0.0.1:3001: connectex: ... actively refused it."
|
|
Error: unavailable: dial tcp 127.0.0.1:3001 ...
|
|
|
|
...and the process is gone. So one Gitea restart (or one reboot)
|
|
silently kills CI until a human notices days later. Queued runs then
|
|
get cancelled by the NEXT push, because ci.yml sets
|
|
`concurrency.cancel-in-progress: true` -- which is why the symptom
|
|
looks like "tasks vanish" rather than "tasks fail".
|
|
|
|
2. NO MACHINE-LEVEL TOOLCHAIN. The runner service runs as SYSTEM.
|
|
pre-commit / cppcheck were only reachable through the interactive
|
|
user's PATH, and no machine-level DTU_* variable was ever set, so
|
|
even a live runner died in the L0 environment check.
|
|
|
|
This script installs the missing supervision (a watchdog launcher + a
|
|
scheduled task) and, with -SetEnv, writes the machine-level DTU_* variables
|
|
by discovering the tools the same way CI does.
|
|
|
|
Run elevated, from the repo:
|
|
|
|
powershell -ExecutionPolicy Bypass -File scripts\ci_runner_bootstrap.ps1 -SetEnv -StartNow
|
|
|
|
.PARAMETER RunnerDir
|
|
Directory holding gitea-runner.exe / config.yaml / .runner.
|
|
|
|
.PARAMETER GiteaUrl
|
|
Instance address the runner registered against (must match .runner's
|
|
"address"; the runner talks to it directly, not through the public URL).
|
|
|
|
.PARAMETER TaskName
|
|
Scheduled-task name for the watchdog.
|
|
|
|
.PARAMETER SetEnv
|
|
Discover and write MACHINE-level DTU_PRECOMMIT_PYTHON / DTU_CPPCHECK /
|
|
DTU_MSVC_CL. Existing values are preserved unless -Force.
|
|
|
|
.PARAMETER StartNow
|
|
Start the task after registering it.
|
|
|
|
.PARAMETER Status
|
|
Only report: task state, daemon process, Gitea reachability.
|
|
|
|
.PARAMETER Force
|
|
With -SetEnv, overwrite existing machine-level values.
|
|
|
|
.OUTPUTS
|
|
Progress lines. Exit codes: 0 = ok, 1 = precondition failed, 2 = not elevated.
|
|
|
|
.NOTES
|
|
Keep this file pure ASCII (PowerShell 5.1 decodes BOM-less files as CP936).
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[string]$RunnerDir = 'D:\Docker\act_runner',
|
|
[string]$GiteaUrl = 'http://127.0.0.1:3001',
|
|
[string]$TaskName = 'GiteaActionsRunner',
|
|
[switch]$SetEnv,
|
|
[switch]$StartNow,
|
|
[switch]$Status,
|
|
[switch]$Force
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
function Info($m) { Write-Host "[INFO] $m" -ForegroundColor Cyan }
|
|
function Ok($m) { Write-Host "[ OK ] $m" -ForegroundColor Green }
|
|
function Warn($m) { Write-Host "[WARN] $m" -ForegroundColor Yellow }
|
|
function Fail($m) { Write-Host "[FAIL] $m" -ForegroundColor Red }
|
|
|
|
function Test-Admin {
|
|
$id = [Security.Principal.WindowsIdentity]::GetCurrent()
|
|
(New-Object Security.Principal.WindowsPrincipal($id)
|
|
).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
|
|
}
|
|
|
|
function Test-Gitea([string]$url) {
|
|
try {
|
|
Invoke-WebRequest -Uri "$url/api/v1/version" -UseBasicParsing -TimeoutSec 10 | Out-Null
|
|
return $true
|
|
}
|
|
catch { return $false }
|
|
}
|
|
|
|
function Get-MachineEnv([string]$name) {
|
|
$v = [Environment]::GetEnvironmentVariable($name, 'Machine')
|
|
if ([string]::IsNullOrWhiteSpace($v)) { return $null }
|
|
return $v
|
|
}
|
|
|
|
function Set-MachineEnv([string]$name, [string]$value, [string]$label) {
|
|
$cur = Get-MachineEnv $name
|
|
if ($cur) {
|
|
if ($cur -ieq $value) { Ok "$label already set: $cur"; return }
|
|
if (-not $Force) {
|
|
Warn "$label already set to a different value: $cur (use -Force to overwrite)"
|
|
return
|
|
}
|
|
Warn "$label overwriting $cur -> $value"
|
|
}
|
|
[Environment]::SetEnvironmentVariable($name, $value, 'Machine')
|
|
Ok "$label set (MACHINE scope): $value"
|
|
}
|
|
|
|
function Find-Cppcheck {
|
|
$pf = $env:ProgramFiles
|
|
$pf86 = ${env:ProgramFiles(x86)}
|
|
if ([string]::IsNullOrWhiteSpace($pf)) { $pf = 'C:\Program Files' }
|
|
if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' }
|
|
foreach ($c in @((Join-Path $pf 'Cppcheck\cppcheck.exe'),
|
|
(Join-Path $pf86 'Cppcheck\cppcheck.exe'))) {
|
|
if (Test-Path -LiteralPath $c) { return $c }
|
|
}
|
|
$cmd = Get-Command cppcheck -ErrorAction SilentlyContinue
|
|
if ($cmd) { return $cmd.Source }
|
|
return $null
|
|
}
|
|
|
|
function Find-MsvcCl {
|
|
$pf = $env:ProgramFiles
|
|
$pf86 = ${env:ProgramFiles(x86)}
|
|
if ([string]::IsNullOrWhiteSpace($pf)) { $pf = 'C:\Program Files' }
|
|
if ([string]::IsNullOrWhiteSpace($pf86)) { $pf86 = 'C:\Program Files (x86)' }
|
|
$roots = @()
|
|
foreach ($year in @('2019', '2022', '2026', '18')) {
|
|
foreach ($edition in @('BuildTools', 'Community', 'Professional', 'Enterprise')) {
|
|
$roots += (Join-Path $pf86 "Microsoft Visual Studio\$year\$edition")
|
|
$roots += (Join-Path $pf "Microsoft Visual Studio\$year\$edition")
|
|
}
|
|
}
|
|
foreach ($r in $roots) {
|
|
if (-not (Test-Path -LiteralPath $r)) { continue }
|
|
$cl = Get-ChildItem -LiteralPath $r -Recurse -Filter 'cl.exe' -ErrorAction SilentlyContinue |
|
|
Where-Object { $_.FullName -like '*Hostx64\x64*' } |
|
|
Select-Object -First 1
|
|
if ($cl) { return $cl.FullName }
|
|
}
|
|
return $null
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# -Status: read-only report
|
|
# ---------------------------------------------------------------------------
|
|
if ($Status) {
|
|
Info "task $TaskName"
|
|
$t = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
|
if ($t) { Ok ("task state: {0}" -f $t.State) } else { Warn 'task not registered' }
|
|
|
|
$proc = Get-Process | Where-Object { $_.ProcessName -match 'gitea-runner|act_runner' }
|
|
if ($proc) { Ok ("daemon running: pid(s) " + ($proc.Id -join ',')) } else { Warn 'daemon not running' }
|
|
|
|
if (Test-Gitea $GiteaUrl) { Ok "Gitea reachable at $GiteaUrl" }
|
|
else { Fail "Gitea NOT reachable at $GiteaUrl" }
|
|
|
|
foreach ($n in @('DTU_PRECOMMIT_PYTHON', 'DTU_CPPCHECK', 'DTU_MSVC_CL')) {
|
|
$v = Get-MachineEnv $n
|
|
if ($v) { Ok ("MACHINE {0} = {1}" -f $n, $v) }
|
|
else { Warn "MACHINE $n not set (CI auto-discovery still applies; see scripts/ci_find_python.ps1)" }
|
|
}
|
|
exit 0
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Preconditions
|
|
# ---------------------------------------------------------------------------
|
|
if (-not (Test-Admin)) {
|
|
Fail 'not elevated -- rerun from an elevated PowerShell (this writes a scheduled task and machine env vars)'
|
|
exit 2
|
|
}
|
|
|
|
$exe = Join-Path $RunnerDir 'gitea-runner.exe'
|
|
$cfg = Join-Path $RunnerDir 'config.yaml'
|
|
$regFile = Join-Path $RunnerDir '.runner'
|
|
$launcher = Join-Path $RunnerDir 'run_runner_watchdog.ps1'
|
|
|
|
foreach ($f in @($exe, $cfg)) {
|
|
if (-not (Test-Path -LiteralPath $f)) { Fail "missing file: $f"; exit 1 }
|
|
}
|
|
if (-not (Test-Path -LiteralPath $regFile)) {
|
|
Fail "not registered: $regFile is missing"
|
|
Info 'register first, e.g.:'
|
|
Write-Host " & '$exe' register --no-interactive --instance '$GiteaUrl' --token <RegistrationToken> --name win-stm32-ci --labels 'windows:host'"
|
|
exit 1
|
|
}
|
|
|
|
# The daemon must talk to the address baked into .runner, not the public URL.
|
|
$registeredAddress = $null
|
|
try {
|
|
$registeredAddress = (Get-Content -LiteralPath $regFile -Raw | ConvertFrom-Json).address
|
|
}
|
|
catch { Warn "cannot parse $regFile : $_" }
|
|
if ($registeredAddress -and $registeredAddress.TrimEnd('/') -ne $GiteaUrl.TrimEnd('/')) {
|
|
Warn ".runner address is $registeredAddress but -GiteaUrl is $GiteaUrl -- using the .runner value for the readiness probe"
|
|
$GiteaUrl = $registeredAddress
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. machine-level toolchain (SYSTEM cannot see the interactive user's PATH)
|
|
# ---------------------------------------------------------------------------
|
|
if ($SetEnv) {
|
|
Info 'discovering toolchain for MACHINE scope ...'
|
|
|
|
$findPy = Join-Path $PSScriptRoot 'ci_find_python.ps1'
|
|
if (Test-Path -LiteralPath $findPy) {
|
|
$py = & $findPy -NeedPreCommit
|
|
if ($LASTEXITCODE -eq 0 -and $py) {
|
|
Set-MachineEnv 'DTU_PRECOMMIT_PYTHON' ("$py".Trim()) 'DTU_PRECOMMIT_PYTHON'
|
|
}
|
|
else {
|
|
Warn 'no interpreter with pre-commit found -- install pre-commit into a machine-wide interpreter first'
|
|
}
|
|
}
|
|
else { Warn "not found: $findPy (skipping DTU_PRECOMMIT_PYTHON)" }
|
|
|
|
$cpp = Find-Cppcheck
|
|
if ($cpp) { Set-MachineEnv 'DTU_CPPCHECK' $cpp 'DTU_CPPCHECK' }
|
|
else { Warn 'cppcheck not found (install Cppcheck.Cppcheck, or set DTU_CPPCHECK manually)' }
|
|
|
|
$cl = Find-MsvcCl
|
|
if ($cl) { Set-MachineEnv 'DTU_MSVC_CL' $cl 'DTU_MSVC_CL' }
|
|
else { Warn 'MSVC cl.exe (Hostx64\x64) not found -- L0.5 will degrade to "skipped" without it' }
|
|
|
|
Info 'machine env vars are read when a PROCESS STARTS: the daemon must be restarted below'
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. watchdog launcher (the actual fix for "daemon exits and nobody notices")
|
|
# ---------------------------------------------------------------------------
|
|
Info "writing watchdog launcher: $launcher"
|
|
|
|
$template = @'
|
|
# AUTO-GENERATED by scripts/ci_runner_bootstrap.ps1 -- do not edit by hand.
|
|
#
|
|
# gitea-runner `daemon` exits when its first Declare call fails (no retry, no
|
|
# service support in v4), so a plain scheduled task is NOT enough: a Gitea
|
|
# restart would take CI down until a human noticed. This loop
|
|
# 1) waits until Gitea answers, then
|
|
# 2) runs the daemon, and
|
|
# 3) restarts it after a short pause whenever it exits.
|
|
# Keep this file pure ASCII (PowerShell 5.1 decodes BOM-less files as CP936).
|
|
$ErrorActionPreference = 'Continue'
|
|
|
|
$runnerDir = '__RUNNER_DIR__'
|
|
$runnerExe = Join-Path $runnerDir 'gitea-runner.exe'
|
|
$runnerCfg = Join-Path $runnerDir 'config.yaml'
|
|
$logDir = Join-Path $runnerDir 'log'
|
|
$log = Join-Path $logDir 'runner-stdout.log'
|
|
$probeUrl = '__GITEA_URL__/api/v1/version'
|
|
$maxWaitSec = 1800
|
|
|
|
if (-not (Test-Path -LiteralPath $logDir)) {
|
|
New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
|
}
|
|
|
|
function Log($m) {
|
|
$line = '{0} [watchdog] {1}' -f (Get-Date -Format 'o'), $m
|
|
Add-Content -LiteralPath $log -Value $line -Encoding ascii
|
|
}
|
|
|
|
while ($true) {
|
|
$waited = 0
|
|
while ($true) {
|
|
$ok = $false
|
|
try {
|
|
Invoke-WebRequest -Uri $probeUrl -UseBasicParsing -TimeoutSec 10 | Out-Null
|
|
$ok = $true
|
|
}
|
|
catch { $ok = $false }
|
|
if ($ok) { break }
|
|
if ($waited -eq 0) { Log "waiting for Gitea at $probeUrl" }
|
|
Start-Sleep -Seconds 15
|
|
$waited += 15
|
|
if ($waited -ge $maxWaitSec) {
|
|
Log 'Gitea still not answering after 30m -- starting anyway so the real error lands in the log'
|
|
break
|
|
}
|
|
}
|
|
|
|
Log 'starting daemon'
|
|
& $runnerExe daemon --config $runnerCfg *>> $log
|
|
Log ("daemon exited, exit code {0}; restarting in 10s" -f $LASTEXITCODE)
|
|
Start-Sleep -Seconds 10
|
|
}
|
|
'@
|
|
|
|
$content = $template.Replace('__RUNNER_DIR__', $RunnerDir).Replace('__GITEA_URL__', $GiteaUrl.TrimEnd('/'))
|
|
$utf8NoBom = New-Object System.Text.UTF8Encoding($false)
|
|
[System.IO.File]::WriteAllText($launcher, $content, $utf8NoBom)
|
|
Ok 'launcher written'
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. scheduled task: start at boot, run as SYSTEM, restart if it dies
|
|
# ---------------------------------------------------------------------------
|
|
$existing = Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue
|
|
if ($existing) {
|
|
Info "task $TaskName exists -- unregistering to re-register with the current settings"
|
|
Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false
|
|
}
|
|
|
|
$action = New-ScheduledTaskAction `
|
|
-Execute 'powershell.exe' `
|
|
-Argument "-NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File `"$launcher`"" `
|
|
-WorkingDirectory $RunnerDir
|
|
|
|
# -AtStartup needs the "System" account; a boot trigger plus StartWhenAvailable
|
|
# also covers "machine was off when the task was created".
|
|
$trigger = New-ScheduledTaskTrigger -AtStartup
|
|
$trigger.Delay = 'PT1M'
|
|
$trigger.StartWhenAvailable = $true
|
|
|
|
$settings = New-ScheduledTaskSettingsSet `
|
|
-AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
|
|
-ExecutionTimeLimit ([TimeSpan]::Zero) `
|
|
-RestartCount 999 -RestartInterval (New-TimeSpan -Minutes 1) `
|
|
-MultipleInstances IgnoreNew
|
|
|
|
Register-ScheduledTask -TaskName $TaskName -Action $action -Trigger $trigger `
|
|
-Settings $settings -User 'SYSTEM' -RunLevel Highest `
|
|
-Description 'Gitea Actions runner watchdog (STM32F4-Base CI). Supervises gitea-runner daemon: waits for Gitea, restarts the daemon when it exits. Generated by scripts/ci_runner_bootstrap.ps1.' |
|
|
Out-Null
|
|
Ok "scheduled task registered: $TaskName (at boot +1m, as SYSTEM, unlimited runtime, restart x999/1m)"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. start it
|
|
# ---------------------------------------------------------------------------
|
|
if ($StartNow) {
|
|
if (-not (Test-Gitea $GiteaUrl)) {
|
|
Warn "Gitea is not answering at $GiteaUrl right now -- the watchdog will keep waiting"
|
|
}
|
|
Start-ScheduledTask -TaskName $TaskName
|
|
Ok "task started; watch: Get-Content '$(Join-Path $RunnerDir 'log\runner-stdout.log')' -Tail 20"
|
|
Info 'verify on the web UI: repo -> Settings -> Actions -> Runners -> win-stm32-ci should turn Idle'
|
|
}
|
|
|
|
Ok 'done'
|
|
exit 0
|