refactor: functional separation with deploy/, recipe-generator/, and bin/

- deploy/: deployment bash scripts (4 files)
- recipe-generator/: recipe generation tool (Python + YAML configs)
- bin/: executable wrappers (single generate-recipe entry point)
- Updated deploy.sh, README.md, DEVELOPMENT.md, DEVELOPMENT_QUICK.md
- Updated tests/test_recipe_generator.py and tests/integration/test_integration.py
- Updated CI workflow and Makefile paths
- All 41 tests pass
This commit is contained in:
Lukas Greve
2026-03-25 09:34:00 +01:00
parent 00c4356ff3
commit 3782777058
25 changed files with 55 additions and 51 deletions
+5 -5
View File
@@ -3,13 +3,13 @@ name: test-generation
on:
push:
paths:
- 'scripts/**/*.py'
- 'scripts/**/*.yaml'
- 'recipe-generator/**/*.py'
- 'recipe-generator/**/*.yaml'
- 'fragments/**/*.ks'
pull_request:
paths:
- 'scripts/**/*.py'
- 'scripts/**/*.yaml'
- 'recipe-generator/**/*.py'
- 'recipe-generator/**/*.yaml'
jobs:
generate:
@@ -28,7 +28,7 @@ jobs:
- name: Generate all variants
run: |
cd scripts
cd recipe-generator
python3 generate_recipe.py \
--manifest recipes_manifest.yaml \
--output-dir ../recipes/
+18 -16
View File
@@ -21,7 +21,7 @@ Phyllome OS uses a **fragment-driven** kickstart generation system:
```
fragments/ (54 .ks files)
↓ (modular snippets)
scripts/generate_recipe.py
recipe-generator/generate_recipe.py
↓ (YAML templates + manifest)
recipes/ (16 auto-generated .cfg)
↓ (ksflatten)
@@ -40,14 +40,16 @@ VMs and ISO images
| `recipes/` | Generated recipes | Manifest-driven compositions |
| `dishes/` | Flattened kickstarts | Ready-to-deploy artifacts |
| `ingredients/` | Legacy building blocks | 35 `.cfg` files (legacy) |
| `scripts/` | Automation tools | `generate_recipe.py`, Makefile |
| `recipe-generator/` | Recipe generation | `generate_recipe.py`, YAML configs, Makefile |
| `deploy/` | Deployment scripts | Bash automation tools |
| `bin/` | Executables | Wrapper scripts (e.g., `generate-recipe`) |
### Data Flow
1. **Fragments** (`fragments/**/*.ks`) - Small, reusable kickstart snippets
2. **Templates** (`scripts/recipe_templates.yaml`) - Define recipe structures
3. **Manifest** (`scripts/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.)
4. **Generator** (`scripts/generate_recipe.py`) - Composes fragments via `%ksappend` directives
2. **Templates** (`recipe-generator/recipe_templates.yaml`) - Define recipe structures
3. **Manifest** (`recipe-generator/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.)
4. **Generator** (`recipe-generator/generate_recipe.py`) - Composes fragments via `%ksappend` directives
5. **Recipes** (`recipes/*.cfg`) - Generated kickstart files with fragment references
6. **Flattening** (`ksflatten`) - Resolves `%ksappend` into single dish file
7. **Deployment** (`virt-install`) - Creates VMs from dish files
@@ -169,7 +171,7 @@ EOF
### Fragment Validation Script
Create `scripts/validate-fragment.sh`:
Create `deploy/validate-fragment.sh`:
```bash
#!/bin/bash
@@ -205,7 +207,7 @@ Recipes are generated from templates and the manifest file. This section covers
### Manifest Editing
**File:** `scripts/recipes_manifest.yaml`
**File:** `recipe-generator/recipes_manifest.yaml`
The manifest defines all recipe variants using modifiers from templates.
@@ -252,7 +254,7 @@ recipes:
### Template Editing
**File:** `scripts/recipe_templates.yaml`
**File:** `recipe-generator/recipe_templates.yaml`
Templates define the structure and fragment composition for each recipe type.
@@ -551,7 +553,7 @@ done
**File:** `.gitea/workflows/test-generation.yaml`
**Triggers:**
- Push to `scripts/**/*.py` or `scripts/**/*.yaml`
- Push to `recipe-generator/**/*.py` or `recipe-generator/**/*.yaml`
- Pull request with script changes
**Steps:**
@@ -609,12 +611,12 @@ luanti
EOF
# Step 2: Add to recipe template
# Edit scripts/recipe_templates.yaml
# Edit recipe-generator/recipe_templates.yaml
# Add to 'required' section:
# - luanti: fragments/shared/packages/luanti.ks
# Step 3: Regenerate recipes
cd scripts
cd recipe-generator
make generate-recipes
# Step 4: Validate
@@ -637,12 +639,12 @@ plasma-workspace
EOF
# Step 2: Add to template
# Edit scripts/recipe_templates.yaml
# Edit recipe-generator/recipe_templates.yaml
# Add to optional/desktop section:
# kde: fragments/shared/desktop/kde/packages.ks
# Step 3: Add variant to manifest
# Edit scripts/recipes_manifest.yaml
# Edit recipe-generator/recipes_manifest.yaml
# Add variant:
# - version: 43
# desktop: kde
@@ -660,7 +662,7 @@ make test
```bash
# Step 1: Add template to recipe_templates.yaml
cat >> scripts/recipe_templates.yaml << 'EOF'
cat >> recipe-generator/recipe_templates.yaml << 'EOF'
minimal-server:
description: "A minimal server recipe"
@@ -681,7 +683,7 @@ cat >> scripts/recipe_templates.yaml << 'EOF'
EOF
# Step 2: Add variant to recipes_manifest.yaml
cat >> scripts/recipes_manifest.yaml << 'EOF'
cat >> recipe-generator/recipes_manifest.yaml << 'EOF'
- name: minimal-server
variants:
@@ -729,7 +731,7 @@ luanti
%end
```
**Template:** `scripts/recipe_templates.yaml`
**Template:** `recipe-generator/recipe_templates.yaml`
```yaml
templates:
virtual-desktop:
+6 -6
View File
@@ -10,7 +10,7 @@ sudo dnf install qemu libvirt virt-install pykickstart
pip install PyYAML pytest
# Verify setup
cd scripts && make generate-recipes && make test
cd recipe-generator && make generate-recipes && make test
```
## Core Workflows
@@ -28,17 +28,17 @@ new-package
### Add to Recipe
```bash
# Edit scripts/recipe_templates.yaml to include fragment
# Edit scripts/recipes_manifest.yaml to add variant
# Edit recipe-generator/recipe_templates.yaml to include fragment
# Edit recipe-generator/recipes_manifest.yaml to add variant
# Regenerate
cd scripts && make generate-recipes && make validate-recipes
cd recipe-generator && make generate-recipes && make validate-recipes
```
### Run Tests
```bash
cd scripts
cd recipe-generator
make test # All tests
make test-integration # Integration only
make test-container # Containerized
@@ -60,7 +60,7 @@ done
## Architecture
```
fragments/ (54 .ks) → generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg)
fragments/ (54 .ks) → recipe-generator/generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg)
```
See `DEVELOPMENT.md` Section 1 for detailed architecture overview.
+1 -1
View File
@@ -15,7 +15,7 @@ chmod +x deploy-vm.sh
```
./deploy-vm.sh
Executing: ./scripts/core-count.sh
Executing: ./deploy/core-count.sh
System has more than 2 core (nproc --all: 6).
[...]
10. virtual-desktop-hypervisor
+2
View File
@@ -0,0 +1,2 @@
#!/bin/bash
exec "$(dirname "$0")/../recipe-generator/generate_recipe.py" "$@"
+4 -4
View File
@@ -13,10 +13,10 @@ execute_script() {
# Array of scripts
scripts=(
"./scripts/install-prerequisites-on-linux.sh"
"./scripts/core-count.sh"
"./scripts/system-memory.sh"
"./scripts/deploy-distro.sh"
"./deploy/install-prerequisites-on-linux.sh"
"./deploy/core-count.sh"
"./deploy/system-memory.sh"
"./deploy/deploy-distro.sh"
)
# Iterate through the scripts and execute them
@@ -25,13 +25,13 @@ validate-recipes:
--validate ../recipes/*.cfg
test:
python3 -m pytest tests/ -v --tb=short
python3 -m pytest ../tests/ -v --tb=short
test-integration:
python3 -m pytest tests/integration/ -v --tb=short
python3 -m pytest ../tests/integration/ -v --tb=short
test-container:
podman build -t phyllo/test-runner tests/container/
podman build -t phyllo/test-runner ../tests/container/
podman run --rm -v .:/phyllomeos:ro phyllo/test-runner
clean:
+4 -6
View File
@@ -69,7 +69,6 @@ class RecipeGenerator:
"""Generate kickstart recipes from templates and modifiers."""
def __init__(self, ingredients_dir: Path, templates_file: Path):
# Resolve ingredients_dir relative to the project root (parent of scripts/)
self.project_root = Path(__file__).parent.parent
self.ingredients_dir = self.project_root / ingredients_dir
self.templates = self.load_templates(templates_file)
@@ -90,7 +89,7 @@ class RecipeGenerator:
if path.is_absolute():
template_path = path
else:
template_path = self.project_root / 'scripts' / path
template_path = self.project_root / path
with open(template_path) as f:
data = yaml.safe_load(f)
return data['templates']
@@ -419,7 +418,6 @@ def main():
)
# Global options
# Use __file__ to find the scripts directory, then go up to project root
SCRIPTS_DIR = Path(__file__).resolve().parent
PROJECT_ROOT = SCRIPTS_DIR.parent
@@ -428,14 +426,14 @@ def main():
help='Ingredients directory (default: parent/ingredients)')
parser.add_argument('--templates', '-t',
type=Path, default=SCRIPTS_DIR / 'recipe_templates.yaml',
help='Templates YAML file (default: parent/recipe_templates.yaml)')
help='Templates YAML file (default: ./recipe_templates.yaml)')
# Batch mode
parser.add_argument('--manifest', '-m',
type=Path, help='Manifest YAML for batch generation')
parser.add_argument('--output-dir', '-d',
type=Path, default=Path(__file__).parent / 'recipes',
help='Output directory (batch generation, default: parent/recipes)')
type=Path, default=SCRIPTS_DIR / 'recipes',
help='Output directory (batch generation, default: ./recipes)')
parser.add_argument('--dry-run', '-n',
action='store_true',
help='Show what would be generated without writing files')
+3 -3
View File
@@ -11,7 +11,7 @@ import os
# Use actual project root
PROJECT_ROOT = Path('/home/lukas/Code/virt/phyllomeos')
SCRIPTS_DIR = PROJECT_ROOT / 'scripts'
RECIPE_GENERATOR_DIR = PROJECT_ROOT / 'recipe-generator'
RECIPE_DIR = PROJECT_ROOT / 'recipes'
FRAGMENTS_DIR = PROJECT_ROOT / 'fragments'
CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
@@ -19,7 +19,7 @@ CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
def test_generate_recipes_from_manifest():
"""Test generating all recipes from manifest."""
os.chdir(SCRIPTS_DIR)
os.chdir(RECIPE_GENERATOR_DIR)
result = subprocess.run(
['python3', 'generate_recipe.py',
@@ -51,7 +51,7 @@ def test_make_targets():
# Test generate-recipes
result = subprocess.run(
['make', 'generate-recipes'],
cwd=SCRIPTS_DIR,
cwd=RECIPE_GENERATOR_DIR,
capture_output=True,
text=True
)
+7 -5
View File
@@ -3,11 +3,9 @@
import pytest
from pathlib import Path
import sys
import os
# Add scripts directory to path
SCRIPTS_DIR = Path(__file__).parent.parent / 'scripts'
sys.path.insert(0, str(SCRIPTS_DIR))
RECIPE_GENERATOR_DIR = Path(__file__).parent.parent / 'recipe-generator'
sys.path.insert(0, str(RECIPE_GENERATOR_DIR))
from generate_recipe import RecipeGenerator
@@ -19,7 +17,11 @@ class TestRecipeGenerator:
def setup_method(self):
"""Set up test fixtures."""
self.generator = RecipeGenerator(Path('ingredients'), Path('recipe_templates.yaml'))
project_root = Path(__file__).parent.parent
self.generator = RecipeGenerator(
project_root / 'ingredients',
project_root / 'recipe-generator' / 'recipe_templates.yaml'
)
def test_template_loading(self):
"""Test that templates are loaded correctly."""