CI runner 修"双看门狗抢 .runner.lock"：-StartNow 先清残留进程再启动
实测发现（比文档问题更要紧）：runner 网页 online、-Status 说 daemon 在跑， 但 runner-watchdog.log 每 10 秒刷 "starting daemon → daemon exited, exit code 1"， daemon 日志写 "another gitea-runner process is already using .runner" / "runner file is already locked by another process"。 根因：Unregister-ScheduledTask 不会带走旧实例启动的进程。13:22 重跑 bootstrap 时， 12:53 起的旧看门狗成为孤儿并继续养着旧 daemon，旧 daemon 占住 .runner.lock， 新注册任务启动的看门狗永远抢不到锁。online 是旧 daemon 在硬撑——任务实际谁也没管， 日志还在以约 20 MB/天 增长。 - ci_runner_bootstrap.ps1：-StartNow 新增 Stop-StaleRunnerInstances，先停孤儿看门狗 再停残留 daemon（顺序不能反：只杀 daemon 会被活着的看门狗 10 秒内重拉），启动后 等 12 秒回显单实例确认；-Status 新增重复实例计数 + daemon 日志锁冲突检测 - 指南 §3.4 补注销不停进程的说明与"纯注释改动不需重启任务"；§7 排障表加 "online 却每 10 秒重启"一行（含手工应急命令） - §3.2 / README：deploy_runner_task.ps1 与 run_runner.cmd 状态由"仍在磁盘、建议删除" 改为"已删除"；README 补重跑部署前的锁冲突警示 机器侧同步（不在版本库）：config.yaml 头部"常驻改用 sc.exe / New-Service"是 §3.0 已证伪的错误指引，改为看门狗口径并指向本文——改后 YAML 可解析且所有配置值逐项与 改前一致（只动注释）；删除两个废弃脚本前核实计划任务动作目标是 run_runner_watchdog.ps1，删除不影响在跑 runner；check_env.ps1 指向废弃脚本的提示 改指 bootstrap（带 BOM 文件，改后 BOM 完好、语法 0 错）。 验证：pre-commit run --all-files 17 项全 Passed；bootstrap parse 0 错、纯 ASCII、 -DryRun 正常。当前 runner 仍 online（旧 daemon 硬撑），彻底归一需管理员跑一次 -StartNow 清残留。
This commit is contained in:
@@ -218,8 +218,31 @@ if ($Status) {
|
||||
}
|
||||
}
|
||||
|
||||
$proc = Get-Process | Where-Object { $_.ProcessName -match 'gitea-runner|act_runner' }
|
||||
if ($proc) { Ok ("daemon running: pid(s) " + ($proc.Id -join ',')) } else { Warn 'daemon not running' }
|
||||
$proc = @(Get-Process | Where-Object { $_.ProcessName -match 'gitea-runner|act_runner' })
|
||||
if ($proc.Count -eq 1) {
|
||||
Ok ("daemon running: pid {0}" -f $proc[0].Id)
|
||||
} elseif ($proc.Count -gt 1) {
|
||||
Warn (" {0} daemon processes -- only ONE can hold .runner.lock, the rest are in a restart loop" -f $proc.Count)
|
||||
Warn ' fix: re-run this script elevated with -StartNow (it clears stale instances first)'
|
||||
} else { Warn 'daemon not running' }
|
||||
|
||||
# A watchdog that cannot get the lock fails every 10s while the runner still
|
||||
# reports ONLINE, so the loop is invisible from the web UI. Count watchdogs and
|
||||
# grep the daemon log for the lock error (measured 2026-10-12).
|
||||
$wd = @(Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.CommandLine -match 'run_runner_watchdog\.ps1' })
|
||||
if ($wd.Count -gt 1) {
|
||||
Warn (" {0} watchdog processes (pids {1}) -- an old one survived a task re-registration" -f `
|
||||
$wd.Count, (($wd | ForEach-Object { $_.ProcessId }) -join ','))
|
||||
Warn ' and is racing the new one for .runner.lock; re-run elevated with -StartNow'
|
||||
}
|
||||
$dlog = Join-Path $RunnerDir 'log\runner-daemon.log'
|
||||
if (Test-Path -LiteralPath $dlog) {
|
||||
$tail = Get-Content -LiteralPath $dlog -Tail 40 -Encoding UTF8 -ErrorAction SilentlyContinue
|
||||
if ($tail -match 'already locked by another process') {
|
||||
Warn ' daemon log says ".runner is already locked by another process" -- stale instance present'
|
||||
}
|
||||
}
|
||||
|
||||
if (Test-Gitea $GiteaUrl) { Ok "Gitea reachable at $GiteaUrl" }
|
||||
else { Fail "Gitea NOT reachable at $GiteaUrl" }
|
||||
@@ -417,14 +440,56 @@ catch {
|
||||
Ok "scheduled task registered: $TaskName (at boot +1m, as SYSTEM, unlimited runtime, restart x999/1m)"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. start it
|
||||
# 4. start it -- but clear stale instances first, or the new watchdog never gets
|
||||
# the .runner lock.
|
||||
#
|
||||
# MEASURED TRAP (2026-10-12): Unregister-ScheduledTask does NOT stop the process
|
||||
# the old instance had started. That orphaned watchdog keeps its daemon alive and
|
||||
# the daemon holds .runner.lock, so the freshly started watchdog fails every 10s:
|
||||
# level=error msg="another gitea-runner process is already using \".runner\"..."
|
||||
# Error: runner file is already locked by another process
|
||||
# The runner still shows ONLINE (the stale daemon serves fine), so CI looks healthy
|
||||
# while the log fills with a restart loop and the task supervises nothing at all.
|
||||
# ---------------------------------------------------------------------------
|
||||
function Stop-StaleRunnerInstances {
|
||||
# Supervisors first: killing the daemon alone lets a live watchdog respawn it
|
||||
# within 10s and the race starts over.
|
||||
$killed = 0
|
||||
Get-CimInstance Win32_Process -Filter "Name='powershell.exe'" -ErrorAction SilentlyContinue |
|
||||
Where-Object { $_.CommandLine -match 'run_runner_watchdog\.ps1' } |
|
||||
ForEach-Object {
|
||||
Info (" stopping stale watchdog pid {0}" -f $_.ProcessId)
|
||||
Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue
|
||||
$killed++
|
||||
}
|
||||
Get-Process -Name 'gitea-runner' -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
Info (" stopping stale daemon pid {0} (releases .runner.lock)" -f $_.Id)
|
||||
Stop-Process -Id $_.Id -Force -ErrorAction SilentlyContinue
|
||||
$killed++
|
||||
}
|
||||
return $killed
|
||||
}
|
||||
|
||||
if ($StartNow) {
|
||||
if (-not (Test-Gitea $GiteaUrl)) {
|
||||
Warn "Gitea is not answering at $GiteaUrl right now -- the watchdog will keep waiting"
|
||||
}
|
||||
$stale = Stop-StaleRunnerInstances
|
||||
if ($stale -gt 0) {
|
||||
Warn "$stale stale process(es) stopped; the runner is offline for a few seconds"
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
Start-ScheduledTask -TaskName $TaskName
|
||||
Ok "task started; daemon log: Get-Content '$(Join-Path $RunnerDir 'log\runner-daemon.log')' -Tail 20"
|
||||
Start-Sleep -Seconds 12 # the watchdog probes Gitea, then execs the daemon
|
||||
$d = @(Get-Process -Name 'gitea-runner' -ErrorAction SilentlyContinue)
|
||||
if ($d.Count -eq 1) {
|
||||
Ok ("daemon up: pid {0} (single instance, holds .runner.lock)" -f $d[0].Id)
|
||||
} elseif ($d.Count -eq 0) {
|
||||
Warn 'no daemon yet -- the watchdog may still be waiting for Gitea; check the watchdog log'
|
||||
} else {
|
||||
Warn ("{0} daemons running -- only one can hold the lock; inspect and kill the extras" -f $d.Count)
|
||||
}
|
||||
Ok "daemon log: Get-Content '$(Join-Path $RunnerDir 'log\runner-daemon.log')' -Tail 20"
|
||||
Ok "watchdog log: Get-Content '$(Join-Path $RunnerDir 'log\runner-watchdog.log')' -Tail 20"
|
||||
Info 'verify on the web UI: repo -> Settings -> Actions -> Runners -> win-stm32-ci should turn Idle'
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user