software-factory/RUNBOOK.md

RUNBOOK — software factory operations

Cookbook for operating the Phyllome OS factory on git.phyllo.me.

Prereqs & inventory

• See server-phyllome-cloudron.md for the box reference and architecture.md for the layout.
• MCP servers (from the automation repo) used by an operator: gitea-phyllome (read), gitea-phyllome-write (write), cloudron-phyllome (read), cloudron-phyllome-write (write).

1. Register a runner (one-time)

The runner connects out to git.phyllo.me, so it never needs an inbound rule. Registration needs a one-time registration token.

Scope gotcha (2026-09-19). Registration tokens are scope-specific and one-time-use. Where you click to create the token determines which repos the runner will serve:

Where you click	Scope	Serves
Settings → Actions → Runners	user	only that user's repos
Org Settings → Actions → Runners	org	that org's repos
Site Administration → Actions → Runners	site-wide	every repo

A user-scope runner will happily register and show online but will never pick up jobs from org repos — jobs sit queued with runner_id: 0 (symptom: Gitea logs show only Declare/Register, no FetchTask). For the factory (org + repo-wide jobs) you must use a site-wide token. Tokens are one-time and admin/owner only — cannot be minted with the API token.

1. Get a registration token: Log in to git.phyllo.me → Site Administration → Actions → Runners → New runner → copy the token. Leave the window open; the token is one-time-use.
2. Deploy the runner VM on the phyllome Cloudron host. Two supported routes:

   • Ansible (existing playbook): devops/ansible-gitea-runner — set registration_token in roles/runner_setup.yml, point inventory.ini at the VM, then ansible-playbook main.yml.

   • Manual: install gitea-runner v3.5.0 on a Fedora 44 VM, then:

     $ sudo -u act_runner gitea-runner register --no-interactive \
         --instance https://git.phyllo.me --token <TOKEN> \
         --name fedora-0 --labels fedora:host
     

     Then run gitea-runner daemon under systemd (see devops/ansible-gitea-runner/roles/runner_setup.yml for the unit).

3. Verify: Gitea UI → Site Administration → Actions → Runners shows the runner online, label fedora.

Runner label syntax is name:executor (e.g. fedora:host). The label name is fedora — jobs must runs-on: fedora. The Fedora version (44) is carried in the container image tag (e.g. fedora-runner-image:44), not in the label (a bare fedora:44 would be parsed as an invalid executor).

The old runner labels fedora-cloud-42 were renamed to fedora (2026-09-15) — all workflows must use runs-on: fedora.

2. Add factory CI to a repo (pattern)

1. Push a workflow to .gitea/workflows/<name>.yml:

   name: ci
   on:
     push:
     pull_request:
   jobs:
     checks:
       runs-on: fedora
       defaults:
         run:
           shell: bash
       container:
         image: git.phyllo.me/devops/fedora-runner-image:latest
       steps:
         - uses: https://git.phyllo.me/devops/checkout@v5
           with:
             fetch-depth: 0
         - run: make test
   

2. Use host jobs (no container: block) only when the job needs mock, livemedia-creator, or the host build cache — see the create-iso workflow for the canonical example.

3. Put a product pipeline under CI

Repo	Workflow	Trigger
roots/phyllomeos	ci.yml (lint/test/validate)	push + PR
roots/xml-definition-for-domains	ci.yml (xmllint)	push + PR
roots/rpm-sources	smoke.yml (mock build)	push + PR (host job)
devops/create-iso	existing build-iso.yaml	tag v*.*.*

4. Build & release an ISO (create-iso)

Triggered by tagging v*.*.* in devops/create-iso. The build-iso.yaml workflow: mock --init → install lorax-lmc-novirt + pykickstart + livecd-tools → livemedia-creator --make-iso → release the ISO via devops/action-gh-release@v2. Requires a host runner (mock needs real chroots).

5. Troubleshooting

• Runner never comes online: re-check the registration token (one-time use) and that gitea-runner daemon is running (systemctl status act_runner).
• Runner online but jobs never start (stuck queued / waiting for runner, runner_id: 0): registration scope mismatch — a user/org-scope runner can't serve jobs from repos outside that scope. Re-register with a site-wide token (Site Administration → Actions → Runners). See §1.
• Job stuck in queued/waiting for runner, runner_id set: label mismatch — the job's runs-on must exactly match a label the runner registers (fedora).
• Container job can't pull the image: runner needs network to git.phyllo.me package registry; check docker pull git.phyllo.me/devops/fedora-runner-image:latest on the VM; verify the token used for registry login has package write/read scope.
• Push to git.lukasgreve.ch (perso) times out: home Cloudron reachability is flaky from this workstation; retry — it is not a factory failure path (the factory only talks to git.phyllo.me).

6. Admin actions that stay in the Cloudron UI

Restore/clear backups, reboot, domain/user management, runner registration tokens — the MCP write surface deliberately excludes these (see the automation README).