docs: software factory README, server reference, architecture, RUNBOOK

RUNBOOK.md

@@ -0,0 +1,107 @@
+# RUNBOOK — software factory operations
+
+Cookbook for operating the Phyllome OS factory on `git.phyllo.me`.
+
+## Prereqs & inventory
+
+- See [server-phyllome-cloudron.md](server-phyllome-cloudron.md) for the box
+  reference and [architecture.md](architecture.md) for the layout.
+- MCP servers (from the `automation` repo) used by an operator:
+  `gitea-phyllome` (read), `gitea-phyllome-write` (write), `cloudron-phyllome`
+  (read), `cloudron-phyllome-write` (write).
+
+## 1. Register a runner (one-time)
+
+The runner connects **out** to `git.phyllo.me`, so it never needs an inbound
+rule. Registration needs a token from the Gitea UI:
+
+1. **Get a registration token** (admin/owner action — cannot be done with the
+   API token):
+   Log in to `git.phyllo.me` → **Settings → Actions → Runners** →
+   *New runner* → copy the token. Leave the window open; the token is
+   one-time-use.
+2. **Deploy the runner VM** on the phyllome Cloudron host. Two supported
+   routes:
+   - *Ansible* (existing playbook): `devops/ansible-gitea-runner` — set
+     `registration_token` in `roles/runner_setup.yml`, point
+     `inventory.ini` at the VM, then `ansible-playbook main.yml`.
+   - *Manual*: install `act_runner` v0.2.13 on a Fedora VM, then:
+
+     ```console
+     $ sudo -u act_runner act_runner register --no-interactive \
+         --instance https://git.phyllo.me --token <TOKEN> \
+         --name fedora-0 --labels fedora
+     ```
+
+     Then run `act_runner daemon` under systemd (see
+     `devops/ansible-gitea-runner/roles/runner_setup.yml` for the unit).
+3. **Verify**: Gitea UI → **Settings → Actions → Runners** shows the runner
+   **online**, label `fedora`.
+
+> The old runner labels `fedora-cloud-42` were renamed to `fedora`
+> (2026-09-15) — all workflows must use `runs-on: fedora`.
+
+## 2. Add factory CI to a repo (pattern)
+
+1. Push a workflow to `.gitea/workflows/<name>.yml`:
+
+   ```yaml
+   name: ci
+   on:
+     push:
+     pull_request:
+   jobs:
+     checks:
+       runs-on: fedora
+       defaults:
+         run:
+           shell: bash
+       container:
+         image: git.phyllo.me/devops/fedora-runner-image:latest
+       steps:
+         - uses: https://git.phyllo.me/devops/checkout@v5
+           with:
+             fetch-depth: 0
+         - run: make test
+   ```
+
+2. Use **host** jobs (no `container:` block) only when the job needs `mock`,
+   `livemedia-creator`, or the host build cache — see the `create-iso`
+   workflow for the canonical example.
+
+## 3. Put a product pipeline under CI
+
+| Repo | Workflow | Trigger |
+|---|---|---|
+| `roots/phyllomeos` | `ci.yml` (lint/test/validate) | push + PR |
+| `roots/xml-definition-for-domains` | `ci.yml` (xmllint) | push + PR |
+| `roots/rpm-sources` | `smoke.yml` (mock build) | push + PR (host job) |
+| `devops/create-iso` | existing `build-iso.yaml` | tag `v*.*.*` |
+
+## 4. Build & release an ISO (create-iso)
+
+Triggered by tagging `v*.*.*` in `devops/create-iso`. The `build-iso.yaml`
+workflow: `mock --init` → install `lorax-lmc-novirt` + `pykickstart` +
+`livecd-tools` → `livemedia-creator --make-iso` → release the ISO via
+`devops/action-gh-release@v2`. Requires a **host** runner (mock needs real
+chroots).
+
+## 5. Troubleshooting
+
+- **Runner never comes online**: re-check the registration token (one-time use)
+  and that `act_runner daemon` is running (`systemctl status act_runner`).
+- **Job stuck in `queued`/`waiting for runner`**: label mismatch — the job's
+  `runs-on` must exactly match a label the runner registers (`fedora`).
+- **Container job can't pull the image**: runner needs network to
+  `git.phyllo.me` package registry; check `docker pull
+  git.phyllo.me/devops/fedora-runner-image:latest` on the VM; verify the token
+  used for registry login has package write/read scope.
+- **Push to `git.lukasgreve.ch` (perso) times out**: home Cloudron
+  reachability is flaky from this workstation; retry — it is *not* a factory
+  failure path (the factory only talks to `git.phyllo.me`).
+
+## 6. Admin actions that stay in the Cloudron UI
+
+Restore/clear backups, reboot, domain/user management, runner registration
+tokens — the MCP write surface deliberately excludes these (see the
+`automation` README).

docs/architecture.md

@@ -0,0 +1,82 @@
+# Software factory architecture
+
+GitHub-style CI for Phyllome OS, self-hosted on the project's own forge and
+iron.
+
+## Goal
+
+Runs CI/CD for Phyllome OS repos on `git.phyllo.me` (Gitea + Gitea Actions)
+with a self-hosted runner, without depending on any third-party CI service.
+
+## Components
+
+```mermaid
+flowchart LR
+    subgraph forge["git.phyllo.me (Cloudron app, Gitea 1.40.2)"]
+        act[Gitea Actions]
+        pkg[(Package Registry)]
+        rel[Releases]
+    end
+
+    subgraph runbox["runner VM (on the phyllome Cloudron host)"]
+        r1[act_runner daemon]
+        host[HOST labels: run-on-host jobs<br>mock / livemedia-creator / pykickstart]
+        img[CONTAINER labels:<br>docker://git.phyllo.me/devops/fedora-runner-image]
+    end
+
+    repo[Product repos: roots/* , devops/*]<br>.gitea/workflows/*.yml → runs-on: fedora
+    dev[devcontainer git.phyllo.me/roots/fedora-iso-creation:42]
+
+    repo --> act
+    act -->|picks runner| r1
+    r1 --> host
+    r1 --> img
+    act --> pkg
+    act --> rel
+    dev --> act
+```
+
+## How a run works
+
+1. A push/PR/tag to a Product repo triggers Gitea Actions.
+2. `runs-on: fedora` matches the self-hosted runner labeled `fedora`.
+3. Two execution modes, chosen per job:
+   - **container** — steps run inside `devops/fedora-runner-image` (clean,
+     hermetically sealed; good for pure build/lint/test jobs).
+   - **host** — steps run directly on the runner VM (needed for jobs that rely
+     on `mock`/QEMU nesting, kernel-level features, or the local build cache).
+4. Steps use the mirrored actions (`devops/checkout@v5`,
+   `devops/action-gh-release@v2`) so nothing reaches out to github.com at
+   runtime.
+
+## Products and their pipelines
+
+| Repo | Pipeline | Job type | Notes |
+|---|---|---|---|
+| `roots/phyllomeos` | `make lint` → `make test` → `make validate` | container | pykickstart data pipeline; generated recipes/dishes are untracked build products |
+| `roots/xml-definition-for-domains` | xmllint check | container | libvirt XML definitions |
+| `roots/rpm-sources` | mock smoke build | host | needs mock chroots (host-only cache) |
+| `devops/create-iso` | mock + livemedia-creator ISO build → release | host | already has a workflow (label `fedora`) |
+| `devops/test-actions` | checkout smoke tests | both | CI playground, already label `fedora` |
+
+## Image inventory (Package Registry)
+
+| Image | Used by |
+|---|---|
+| `devops/fedora-runner-image:latest` | default job container on the runner |
+| `devops/fedora-oci` | base Fedora image |
+| `devops/qemu-libvirt-oci-image` | QEMU/libvirt tooling (VM-in-CI experiments) |
+| `roots/fedora-iso-creation:42` | devcontainer referenced by test workflows |
+
+## Security posture
+
+- **Self-contained actions**: all `uses:` point at `git.phyllo.me/...` mirrors —
+  no third-party action code, no github.com calls.
+- **Runner host**: dedicated VM on the phyllome Cloudron host; `act_runner`
+  runs as unprivileged `act_runner` system user; Docker access for container
+  jobs only.
+- **Secrets**: CI secrets live in Gitea (repo/org) — never in the runner config.
+  Registration token is UI-generated and consumed once.
+- **Read-only infra automation**: the `automation` repo's MCP servers scope
+  Cloudron/Gitea reads and writes by credential pair (read token / write token),
+  and opencode gates writes behind a dedicated `ops` agent.

docs/server-phyllome-cloudron.md

@@ -0,0 +1,63 @@
+# Phyllome Cloudron server (`my.phyllo.me`)
+
+The Phyllome OS project's Cloudron platform. This is the *project* infrastructure
+server — distinct from the home-LAN Cloudron (`my.lesloutres.ch`, see the
+`inventory-of-devices` repo, which is personal-resource scoped and intentionally
+does **not** document this box).
+
+## At a glance
+
+| Field | Value |
+|---|---|
+| Platform | Cloudron **10.0.5** |
+| Admin URL | `https://my.phyllo.me` |
+| Host OS | Ubuntu (managed by Cloudron installer) |
+| Location | Not on the home LAN — out-of-LAN host; public DNS via IPv6 (`2001:41d0:1008:44d::1`) + the box behind the project's hosting |
+| DNS provider | Gandi (PAT, `phyllo.me` + `phyllome.org`) |
+| TLS | Let's Encrypt, wildcard (production `letsencrypt-prod`) |
+| Domains | `phyllo.me`, `phyllome.org` |
+| Backup | Cloudron box + app backups nightly; cleanup job keeps a rolling window (see event log `backup.cleanup.finish`) |
+
+Remote-management interfaces:
+
+- **Cloudron API / UI** — `cloudron-phyllome` (read) and `cloudron-phyllome-write`
+  (write) MCP servers from the `automation` repo.
+- **Gitea forge** — `git.phyllo.me`, Gitea **1.40.2** app (id `io.gitea.cloudronapp`).
+  Read via `gitea-phyllome`, write via `gitea-phyllome-write`. Git-over-SSH on
+  port **29418** (e.g. `ssh://git@git.phyllo.me:29418/owner/repo.git`).
+
+## Apps
+
+Cloudron app inventory on this box (subset relevant to the factory):
+
+| FQDN | App | State | Role in the factory |
+|---|---|---|---|
+| `git.phyllo.me` | Gitea | running | Code host + Gitea Actions bus |
+| `minio.phyllo.me` (+ `minio-api.phyllo.me`) | MinIO | running | S3 object storage (factory artifact staging candidate) |
+| `kanboard.phyllo.me` / `kanboard.phyllome.org` | Wekan | running / stopped (preprod dup) | Project kanban |
+| `wiki.phyllo.me` / `wiki.phyllome.org` | Wiki.js | running / stopped | Docs (live = wiki.phyllome.org) |
+| `llamas.phyllo.me` | Open WebUI | stopped | AI chat (on demand) |
+| `ks.phyllome.org` | Kutt | stopped | URL shortener (preprod) |
+| `phyllo.me`, `www.phyllo.me` | Grav | running | Project website |
+
+Full inventory is one `cloudron-phyllome_list_apps` call away; this file only
+documents what the factory depends on.
+
+## Factory-relevant services
+
+- **Package registry** (Gitea Packages) holds the container images:
+  - `devops/fedora-runner-image` (with `latest`) — runner job container
+  - `devops/fedora-oci`, `devops/qemu-libvirt-oci-image` — builder images
+  - `roots/fedora-iso-creation:42` — devcontainer referenced by test-actions
+- **Gitea Actions** is enabled on the product repos.
+- **Runners**: none registered yet → see `RUNBOOK.md`. Expected label: `fedora`.
+
+## Access model
+
+- `git.phyllo.me` user `lukas` (id 4, **not** Gitea admin) — used by the read and
+  write API tokens.
+- Runner registration requires a **runner registration token** from the Gitea UI
+  (Settings → Actions → Runners) — admin/owner action, see RUNBOOK.
+- Cloudron ownership is via an OIDC/owner account `superuser@phyllo.me`; API
+  tokens (`tower-read`, `tower`, …) are created/deleted via the Cloudron UI
+  (visible in the event log).