diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md new file mode 100644 index 0000000..aa5ce02 --- /dev/null +++ b/DEVELOPMENT.md @@ -0,0 +1,855 @@ +# Phyllome OS Development Guide + +This guide covers development workflows for contributors to Phyllome OS. + +## Table of Contents +- [Architecture Overview](#architecture-overview) +- [Development Environment Setup](#development-environment-setup) +- [Fragment Development](#fragment-development) +- [Recipe Generation](#recipe-generation) +- [Testing](#testing) +- [CI/CD](#cicd) +- [Common Workflows](#common-workflows) +- [Migration Guide: Fragment-Based Architecture](#migration-guide-fragment-based-architecture) + +--- + +## Architecture Overview + +Phyllome OS uses a **fragment-driven** kickstart generation system: + +``` +fragments/ (54 .ks files) + ↓ (modular snippets) +scripts/generate_recipe.py + ↓ (YAML templates + manifest) +recipes/ (16 auto-generated .cfg) + ↓ (ksflatten) +dishes/ (28 flattened .cfg) + ↓ (virt-install) +VMs and ISO images +``` + +### Directory Structure + +| Path | Purpose | Contents | +|------|---------|----------| +| `fragments/` | Modular kickstart snippets | 54 `.ks` files | +| `fragments/platform/` | Version-specific configs | `generic-43/`, `generic-rawhide/` | +| `fragments/shared/` | Common components | `core/`, `packages/`, `desktop/`, `hypervisor/`, `live/`, `initial-setup/` | +| `recipes/` | Generated recipes | Manifest-driven compositions | +| `dishes/` | Flattened kickstarts | Ready-to-deploy artifacts | +| `ingredients/` | Legacy building blocks | 35 `.cfg` files (legacy) | +| `scripts/` | Automation tools | `generate_recipe.py`, Makefile | + +### Data Flow + +1. **Fragments** (`fragments/**/*.ks`) - Small, reusable kickstart snippets +2. **Templates** (`scripts/recipe_templates.yaml`) - Define recipe structures +3. **Manifest** (`scripts/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.) +4. **Generator** (`scripts/generate_recipe.py`) - Composes fragments via `%ksappend` directives +5. **Recipes** (`recipes/*.cfg`) - Generated kickstart files with fragment references +6. **Flattening** (`ksflatten`) - Resolves `%ksappend` into single dish file +7. **Deployment** (`virt-install`) - Creates VMs from dish files + +--- + +## Development Environment Setup + +### Prerequisites + +**System dependencies:** +```bash +# Fedora/RHEL-based systems +sudo dnf install qemu libvirt virt-install pykickstart + +# Start libvirt +sudo systemctl start libvirtd +sudo systemctl enable libvirtd +``` + +**Python dependencies:** +```bash +cd scripts +pip install -r requirements.txt +# Required: PyYAML>=6.0, pytest>=7.0 +# Optional: pykickstart (for validation) +``` + +### Verify Setup + +```bash +# Generate all recipes from manifest +cd scripts +make generate-recipes + +# Run all tests +make test + +# Check recipe validation +make validate-recipes +``` + +--- + +## Fragment Development + +Fragments are modular kickstart snippets stored in `fragments/`. Each `.ks` file contains a single feature section. + +### Creating a New Fragment + +**Step 1: Choose location** +- `fragments/shared/core/` - Base settings (security, services, networking) +- `fragments/shared/storage/` - Partition layouts +- `fragments/shared/packages/` - Package groups +- `fragments/shared/desktop/` - Desktop environment configs +- `fragments/shared/hypervisor/` - Virtualization hardware configs +- `fragments/shared/live/` - Live system components +- `fragments/shared/initial-setup/` - First-boot configuration +- `fragments/platform/generic-43/` or `generic-rawhide/` - Version-specific + +**Step 2: Create the fragment file** + +```bash +# Example: Add Luanti game engine +cat > fragments/shared/packages/luanti.ks << 'EOF' +%packages +luanti +%end +EOF +``` + +**Step 3: Validate with pykickstart** + +```bash +python3 -c " +from pykickstart.parser import KickstartParser +from pykickstart.version import makeVersion, DEVEL + +parser = KickstartParser(makeVersion(DEVEL)) +try: + with open('fragments/shared/packages/luanti.ks') as f: + parser.readKickstart(f.read()) + print('✓ Validation passed') +except Exception as e: + print(f'✗ Validation failed: {e}') +" +``` + +### Fragment Naming Conventions + +- **Pattern:** `/.ks` +- **Examples:** + - `fragments/shared/core/security/enabled.ks` + - `fragments/shared/desktop/gnome/packages.ks` + - `fragments/platform/generic-43/repo/fedora-mirrors.ks` + +### Common Fragment Types + +**Packages:** +```bash +%packages +@base-graphical +package-name +%end +``` + +**Storage:** +```bash +part / --fstype="ext4" --grow +part /boot/efi --fstype="efi" --size=512 +``` + +**Services:** +```bash +%services +--enabled=sshd,chronyd +EOF +``` + +### Fragment Validation Script + +Create `scripts/validate-fragment.sh`: + +```bash +#!/bin/bash +set -e + +fragment="$1" + +if [ -z "$fragment" ]; then + echo "Usage: $0 " + exit 1 +fi + +python3 -c " +from pykickstart.parser import KickstartParser +from pykickstart.version import makeVersion, DEVEL + +parser = KickstartParser(makeVersion(DEVEL)) +try: + with open('$fragment') as f: + parser.readKickstart(f.read()) + print('✓ Valid: $fragment') +except Exception as e: + print(f'✗ Invalid: $fragment - {e}') + exit(1) +" +``` + +--- + +## Recipe Generation + +Recipes are generated from templates and the manifest file. This section covers modifying existing recipes and creating new ones. + +### Manifest Editing + +**File:** `scripts/recipes_manifest.yaml` + +The manifest defines all recipe variants using modifiers from templates. + +**Structure:** +```yaml +recipes: + - name: virtual-desktop # Template name + variants: # Modifier combinations + - version: 43 + desktop: gnome + storage: standard + security: secure + extras: true + - version: rawhide + desktop: labwc + storage: encrypted + security: devel +``` + +**Adding a new variant:** +```yaml + - name: virtual-server + variants: + - version: 43 + security: secure + extras: true + post: true + # Add new variant below + - version: 43 + security: secure + extras: false + post: false +``` + +**Valid modifiers** (per template): +- `version`: `43` or `rawhide` +- `desktop`: `gnome` or `labwc` (desktop templates) +- `storage`: `standard` or `encrypted` +- `security`: `secure` or `devel` +- `cpu`: `amdcpu`, `intelcpu` (hypervisor) +- `gpu`: `intelgpu` (hypervisor) +- `extras`: `true` or `false` +- `post`: `true` or `false` + +### Template Editing + +**File:** `scripts/recipe_templates.yaml` + +Templates define the structure and fragment composition for each recipe type. + +**Structure:** +```yaml +templates: + virtual-desktop: + description: "A recipe for a virtual desktop" + base: core + required: # Always included fragments + - core: fragments/shared/core/base.ks + - storage: fragments/shared/storage/standard.ks + optional: # Conditional fragments + security: + secure: fragments/shared/core/security/enabled.ks + devel: fragments/shared/core/security/disabled.ks + modifiers: # Storage/bootloader alternatives + storage: + standard: fragments/shared/storage/standard.ks + encrypted: fragments/shared/storage/encrypted.ks +``` + +**Adding a new required fragment:** +```yaml + required: + - core: fragments/shared/core/base.ks + - storage: fragments/shared/storage/standard.ks + # New fragment + - packages: fragments/shared/packages/hand-picked.ks +``` + +**Adding an optional modifier:** +```yaml + optional: + security: + secure: fragments/shared/core/security/enabled.ks + devel: fragments/shared/core/security/disabled.ks + # New optional - post-install scripts + post: fragments/shared/section-data/post/base.ks +``` + +**Adding a modifier alternative:** +```yaml + modifiers: + storage: + standard: fragments/shared/storage/standard.ks + encrypted: fragments/shared/storage/encrypted.ks + # New storage option + btrfs: fragments/shared/storage/btrfs.ks +``` + +### Generation Workflow + +```bash +cd scripts + +# Generate all recipes from manifest +make generate-recipes + +# Validate generated recipes +make validate-recipes + +# Validate with strict mode (checks fragment existence) +python3 generate_recipe.py --validate ../recipes/*.cfg --strict +``` + +**Output:** +``` +Generating: virtual-desktop_43.cfg +Generating: virtual-desktop_rawhide_encrypted.cfg +Generating: virtual-server_43.cfg +... +✓ Generated 16 recipes +``` + +### Manual Recipe Creation + +**Option 1: Copy existing recipe** + +```bash +# Copy template +cp recipes/_list-of-ingredients.cfg recipes/my-custom.cfg + +# Edit with your favorite editor +nano recipes/my-custom.cfg + +# Add custom ingredient +echo "%include ../ingredients/extra-luanti.cfg" >> recipes/my-custom.cfg + +# Flatten to dish +ksflatten -c recipes/my-custom.cfg -o dishes/my-custom.cfg +``` + +**Option 2: Create from scratch** + +```bash +cat > recipes/my-distro.cfg << 'EOF' +# My custom Phyllome OS variant + +# Installation method +%include ../ingredients/core.cfg + +# Storage configuration +%include ../ingredients/core-storage.cfg + +# Bootloader +%include ../fragments/platform/generic-43/bootloader/grub.ks + +# Network configuration +%include ../fragments/shared/core/network.ks + +# Desktop environment +%include ../fragments/shared/desktop/gnome/packages.ks + +# Additional packages +%packages +my-custom-package +@base-graphical +%end +EOF +``` + +### Flattening Recipes + +Convert recipe (with `%ksappend`) to dish (flattened): + +```bash +# Single recipe +ksflatten -c recipes/virtual-desktop_43.cfg -o dishes/virtual-desktop_43.cfg + +# All recipes +cd recipes +for filename in *.cfg; do + ksflatten -c "$filename" -o "../dishes/$filename" +done +``` + +--- + +## Testing + +Phyllome OS uses a comprehensive test suite with 36+ tests covering unit, integration, and regression scenarios. + +### Test Suite Structure + +| Test File | Tests | Coverage | +|-----------|-------|----------| +| `tests/test_recipe_generator.py` | 36 | Unit tests for RecipeGenerator | +| `tests/integration/test_integration.py` | 5+ | End-to-end workflow tests | +| `tests/integration/test_fragments.py` | ~15 | Fragment validation | +| `tests/integration/test_recipe_composition.py` | ~10 | Recipe generation | +| `tests/integration/test_semantic_validation.py` | ~10 | pykickstart validation | +| `tests/integration/test_golden_masters.py` | ~5 | Regression tests | + +### Running Tests + +```bash +cd scripts + +# All tests (unit + integration) +make test + +# Unit tests only +python3 -m pytest tests/test_recipe_generator.py -v + +# Integration tests only +make test-integration + +# Containerized tests +make test-container +# or: podman run --rm -v .:/phyllomeos:ro phyllo/test-runner +``` + +**Expected output:** +``` +============================= test session starts ============================= +collected 41 items + +tests/test_recipe_generator.py ............. [ 29%] +tests/integration/test_integration.py ..... [ 43%] +... + +============================== 41 passed in 2.34s ============================== +``` + +### Fragment Validation Test + +```bash +# Test all fragments with pykickstart +for fragment in $(find fragments -name "*.ks"); do + python3 -c " +from pykickstart.parser import KickstartParser +from pykickstart.version import makeVersion, DEVEL +parser = KickstartParser(makeVersion(DEVEL)) +parser.readKickstart(open('$fragment').read()) +" && echo "✓ $fragment" || echo "✗ $fragment" +done +``` + +### Adding New Tests + +**Unit test example:** + +```python +# tests/test_recipe_generator.py + +def test_generate_recipe_with_new_modifier(): + """Test recipe generation with custom modifier.""" + content = self.generator.generate_recipe( + 'virtual-desktop', '43', + desktop='gnome', + storage='standard', + security='secure' + ) + assert '# A recipe for a virtual desktop' in content + assert '%ksappend fragments/shared/desktop/gnome/packages.ks' in content +``` + +**Integration test example:** + +```python +# tests/integration/test_recipe_composition.py + +def test_manifest_generates_correct_count(): + """Test manifest generates expected number of recipes.""" + result = subprocess.run( + ['python3', 'generate_recipe.py', + '--manifest', 'recipes_manifest.yaml', + '--output-dir', '../recipes/'], + capture_output=True, text=True + ) + assert result.returncode == 0 + recipe_count = len(list(Path('../recipes').glob('*.cfg'))) + assert recipe_count == 16 +``` + +### Golden Master Tests + +Golden masters (expected outputs) are stored in `tests/fixtures/expected_recipes/`: + +```bash +# View expected output for virtual-desktop +cat tests/fixtures/expected_recipes/virtual-desktop_43.cfg +``` + +**To update golden masters** (after intentional changes): + +```bash +# Generate and compare +python3 -m pytest tests/integration/test_golden_masters.py -v +``` + +--- + +## CI/CD + +The project uses Gitea Actions for automated testing and building. + +### Workflow Files + +| File | Trigger | Purpose | +|------|---------|---------| +| `.gitea/workflows/validate-fragments.yaml` | Push/PR fragments | Validate all 54 .ks files | +| `.gitea/workflows/test-generation.yaml` | Push/PR scripts | Generate 16 recipes | +| `.gitea/workflows/validate-recipes.yaml` | Push PR main | Full validation suite | +| `.gitea/workflows/build-iso.yaml` | Push main, release | Build live ISO | + +### Workflow: Validate Fragments + +**File:** `.gitea/workflows/validate-fragments.yaml` + +**Triggers:** +- Push to any `fragments/**/*.ks` file +- Pull request with fragment changes + +**Steps:** +1. Checkout code +2. Install pykickstart +3. Validate each `.ks` file +4. Report errors + +**Local equivalent:** +```bash +for fragment in $(find fragments -name "*.ks"); do + python3 -c " +from pykickstart.parser import KickstartParser +from pykickstart.version import makeVersion, DEVEL +parser = KickstartParser(makeVersion(DEVEL)) +parser.readKickstart(open('$fragment').read()) +" || exit 1 +done +``` + +### Workflow: Test Generation + +**File:** `.gitea/workflows/test-generation.yaml` + +**Triggers:** +- Push to `scripts/**/*.py` or `scripts/**/*.yaml` +- Pull request with script changes + +**Steps:** +1. Pull request with manifest changes + +**Steps:** +1. Checkout code +2. Install dependencies (PyYAML, pykickstart) +3. Generate all recipes +4. Verify 16 recipes created +5. Upload as artifact + +### Workflow: Build ISO + +**File:** `.gitea/workflows/build-iso.yaml` + +**Triggers:** +- Push to main branch +- Release published + +**Steps:** +1. Generate and validate recipes +2. Initialize mock build environment +3. Install build tools (lorax-lmc-novirt, livemedia-creator) +4. Copy recipe to mock +5. Build ISO with livemedia-creator +6. Upload ISO as artifact + +--- + +## Common Workflows + +### Quick Reference + +| Task | Command | File | +|------|---------|------| +| Generate all recipes | `cd scripts && make generate-recipes` | - | +| Validate all fragments | `for f in $(find fragments -name "*.ks"); do python3 -c "from pykickstart.parser import KickstartParser; from pykickstart.version import makeVersion, DEVEL; parser = KickstartParser(makeVersion(DEVEL)); parser.readKickstart(open('$f').read())" && echo "✓ $f"; done` | - | +| Run all tests | `cd scripts && make test` | - | +| Flatten recipe to dish | `ksflatten -c recipes/X.cfg -o dishes/X.cfg` | - | +| Deploy VM from dish | `./deploy-vm.sh` | `deploy.sh`, `deploy-distro.sh` | +| Build ISO | See `.gitea/workflows/build-iso.yaml` | - | +| Update golden masters | `python3 -m pytest tests/integration/test_golden_masters.py --snapshot-update` | - | + +### Workflow: Add New Package + +**Example: Add Luanti game engine** + +```bash +# Step 1: Create fragment +cat > fragments/shared/packages/luanti.ks << 'EOF' +%packages +luanti +%end +EOF + +# Step 2: Add to recipe template +# Edit scripts/recipe_templates.yaml +# Add to 'required' section: +# - luanti: fragments/shared/packages/luanti.ks + +# Step 3: Regenerate recipes +cd scripts +make generate-recipes + +# Step 4: Validate +make validate-recipes + +# Step 5: Flatten and test +ksflatten -c ../recipes/virtual-desktop_43.cfg -o ../dishes/virtual-desktop_43.cfg +``` + +### Workflow: Add New Desktop Environment + +**Example: Add KDE Plasma** + +```bash +# Step 1: Create desktop fragment +cat > fragments/shared/desktop/kde/packages.ks << 'EOF' +%packages +@kde-desktop +plasma-workspace +EOF + +# Step 2: Add to template +# Edit scripts/recipe_templates.yaml +# Add to optional/desktop section: +# kde: fragments/shared/desktop/kde/packages.ks + +# Step 3: Add variant to manifest +# Edit scripts/recipes_manifest.yaml +# Add variant: +# - version: 43 +# desktop: kde +# storage: standard +# security: secure + +# Step 4: Generate and test +make generate-recipes +make test +``` + +### Workflow: Create New Recipe Type + +**Example: Create minimal-server recipe** + +```bash +# Step 1: Add template to recipe_templates.yaml +cat >> scripts/recipe_templates.yaml << 'EOF' + + minimal-server: + description: "A minimal server recipe" + base: core + required: + - core: fragments/shared/core/base.ks + - storage: fragments/shared/storage/standard.ks + - bootloader: fragments/platform/generic-43/bootloader/grub.ks + - packages: fragments/shared/packages/core-group.ks + - fedora-remix: fragments/shared/packages/fedora-remix.ks + optional: + security: + secure: fragments/shared/core/security/enabled.ks + devel: fragments/shared/core/security/disabled.ks + version: + "43": fragments/platform/generic-43/repo/fedora-mirrors.ks + "rawhide": fragments/platform/generic-rawhide/repo/rawhide-mirrors.ks +EOF + +# Step 2: Add variant to recipes_manifest.yaml +cat >> scripts/recipes_manifest.yaml << 'EOF' + + - name: minimal-server + variants: + - version: 43 + security: secure +EOF + +# Step 3: Generate and validate +make generate-recipes +make validate-recipes +``` + +--- + +## Migration Guide: Fragment-Based Architecture + +Phyllome OS migrated from a monolithic ingredient-based system to a modular fragment-based architecture (Phase 2). + +### Before: Ingredient-Based + +**File:** `ingredients/extra-luanti.cfg` +```bash +%packages +luanti +%end +``` + +**Recipe:** `recipes/virtual-desktop-luanti.cfg` +```bash +%include ../ingredients/core.cfg +%include ../ingredients/extra-luanti.cfg +``` + +**Issues:** +- Duplicate `%include` directives across recipes +- Hard to maintain common patterns +- No automatic version matrix + +### After: Fragment-Based + +**Fragment:** `fragments/shared/packages/luanti.ks` +```bash +%packages +luanti +%end +``` + +**Template:** `scripts/recipe_templates.yaml` +```yaml +templates: + virtual-desktop: + required: + - luanti: fragments/shared/packages/luanti.ks +``` + +**Recipe:** `recipes/virtual-desktop_43.cfg` +```bash +# Generated automatically +%ksappend fragments/shared/core/base.ks +%ksappend fragments/shared/packages/luanti.ks +``` + +### Migration Benefits + +| Aspect | Before | After | +|--------|--------|-------| +| Maintenance | Repetitive includes | Single source in template | +| Variants | Manual recipe creation | Manifest-driven generation | +| Testing | Per-recipe validation | Fragment-level validation | +| Code duplication | High | None (DRY principle) | + +### Why `%ksappend` over `%include`? + +- **`%include`** - Simple file inclusion (copied into result) +- **`%ksappend`** - References external file (keeps file path in flattened output) + +**Benefits of `%ksappend`:** +1. Smaller dish files (no duplicate content) +2. Clear dependency tracking +3. Better validation (fragment existence checks) +4. Easier debugging (visible fragment References) + +### Migration Checklist + +- [ ] Review all `ingredients/*.cfg` files +- [ ] Identify reusable patterns +- [ ] Create `fragments/shared/` for common components +- [ ] Update `recipe_templates.yaml` with new structure +- [ ] Update `recipes_manifest.yaml` for variants +- [ ] Regenerate recipes with `make generate-recipes` +- [ ] Validate with `make validate-recipes` +- [ ] Flatten to dishes with `ksflatten` +- [ ] Run tests with `make test` + +--- + +## Troubleshooting + +### Common Issues + +**1. Duplicate %ksappend entries** + +```bash +# Detect duplicates +grep "^%ksappend" recipes/*.cfg | sort | uniq -d +``` + +**Fix:** Check template for duplicate fragment references + +**2. Missing fragment error** + +``` +ERROR: Missing fragment: fragments/shared/unknown/missing.ks +``` + +**Fix:** Verify fragment exists in `fragments/shared/` + +**3. Deprecated command warning** + +``` +keyboard command is deprecated. Use keyboard --vckeymap instead. +``` + +**Fix:** Update fragment to use modern syntax (see `DEPRECATED_COMMANDS` in `generate_recipe.py`) + +**4. pykickstart validation fails** + +```bash +# Check fragment with pykickstart +python3 -c " +from pykickstart.parser import KickstartParser +from pykickstart.version import makeVersion, DEVEL +parser = KickstartParser(makeVersion(DEVEL)) +try: + with open('fragment.ks') as f: + parser.readKickstart(f.read()) + print('Valid') +except Exception as e: + print(f'Error: {e}') +" +``` + +### Debugging Generator + +```bash +# Verbose mode +python3 generate_recipe.py --manifest recipes_manifest.yaml --output-dir ../recipes/ --verbose + +# Check template loading +python3 -c " +from pathlib import Path +import sys +sys.path.insert(0, '.') +from generate_recipe import RecipeGenerator +gen = RecipeGenerator(Path('../ingredients'), Path('recipe_templates.yaml')) +print('Templates:', list(gen.templates.keys())) +" +``` + +--- + +## Additional Resources + +- **Kickstart Documentation:** https://pykickstart.readthedocs.io/ +- **Fedora Kickstarts:** https://pagure.io/fedora-kickstarts +- **virt-install:** `man virt-install` +- **livemedia-creator:** `man livemedia-creator` + +--- + +*Last updated: March 2026* diff --git a/DEVELOPMENT_QUICK.md b/DEVELOPMENT_QUICK.md new file mode 100644 index 0000000..d856610 --- /dev/null +++ b/DEVELOPMENT_QUICK.md @@ -0,0 +1,66 @@ +# Phyllome OS Development Workflow + +This is a quick-reference guide for developers. For comprehensive coverage, see [DEVELOPMENT.md](./DEVELOPMENT.md). + +## Quick Start + +```bash +# Prerequisites +sudo dnf install qemu libvirt virt-install pykickstart +pip install PyYAML pytest + +# Verify setup +cd scripts && make generate-recipes && make test +``` + +## Core Workflows + +### Add Fragment + +```bash +# Create new kickstart snippet +cat > fragments/shared/packages/new-package.ks << 'EOF' +%packages +new-package +%end +``` + +### Add to Recipe + +```bash +# Edit scripts/recipe_templates.yaml to include fragment +# Edit scripts/recipes_manifest.yaml to add variant + +# Regenerate +cd scripts && make generate-recipes && make validate-recipes +``` + +### Run Tests + +```bash +cd scripts +make test # All tests +make test-integration # Integration only +make test-container # Containerized +``` + +### Validate Fragments + +```bash +for f in $(find fragments -name "*.ks"); do + python3 -c " +from pykickstart.parser import KickstartParser +from pykickstart.version import makeVersion, DEVEL +parser = KickstartParser(makeVersion(DEVEL)) +parser.readKickstart(open('$f').read()) +" || echo "✗ $f" +done +``` + +## Architecture + +``` +fragments/ (54 .ks) → generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg) +``` + +See `DEVELOPMENT.md` Section 1 for detailed architecture overview.