refactor: functional separation with deploy/, recipe-generator/, and bin/
- deploy/: deployment bash scripts (4 files) - recipe-generator/: recipe generation tool (Python + YAML configs) - bin/: executable wrappers (single generate-recipe entry point) - Updated deploy.sh, README.md, DEVELOPMENT.md, DEVELOPMENT_QUICK.md - Updated tests/test_recipe_generator.py and tests/integration/test_integration.py - Updated CI workflow and Makefile paths - All 41 tests pass
This commit is contained in:
@@ -3,13 +3,13 @@ name: test-generation
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- 'scripts/**/*.py'
|
||||
- 'scripts/**/*.yaml'
|
||||
- 'recipe-generator/**/*.py'
|
||||
- 'recipe-generator/**/*.yaml'
|
||||
- 'fragments/**/*.ks'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'scripts/**/*.py'
|
||||
- 'scripts/**/*.yaml'
|
||||
- 'recipe-generator/**/*.py'
|
||||
- 'recipe-generator/**/*.yaml'
|
||||
|
||||
jobs:
|
||||
generate:
|
||||
@@ -28,7 +28,7 @@ jobs:
|
||||
|
||||
- name: Generate all variants
|
||||
run: |
|
||||
cd scripts
|
||||
cd recipe-generator
|
||||
python3 generate_recipe.py \
|
||||
--manifest recipes_manifest.yaml \
|
||||
--output-dir ../recipes/
|
||||
|
||||
+18
-16
@@ -21,7 +21,7 @@ Phyllome OS uses a **fragment-driven** kickstart generation system:
|
||||
```
|
||||
fragments/ (54 .ks files)
|
||||
↓ (modular snippets)
|
||||
scripts/generate_recipe.py
|
||||
recipe-generator/generate_recipe.py
|
||||
↓ (YAML templates + manifest)
|
||||
recipes/ (16 auto-generated .cfg)
|
||||
↓ (ksflatten)
|
||||
@@ -40,14 +40,16 @@ VMs and ISO images
|
||||
| `recipes/` | Generated recipes | Manifest-driven compositions |
|
||||
| `dishes/` | Flattened kickstarts | Ready-to-deploy artifacts |
|
||||
| `ingredients/` | Legacy building blocks | 35 `.cfg` files (legacy) |
|
||||
| `scripts/` | Automation tools | `generate_recipe.py`, Makefile |
|
||||
| `recipe-generator/` | Recipe generation | `generate_recipe.py`, YAML configs, Makefile |
|
||||
| `deploy/` | Deployment scripts | Bash automation tools |
|
||||
| `bin/` | Executables | Wrapper scripts (e.g., `generate-recipe`) |
|
||||
|
||||
### Data Flow
|
||||
|
||||
1. **Fragments** (`fragments/**/*.ks`) - Small, reusable kickstart snippets
|
||||
2. **Templates** (`scripts/recipe_templates.yaml`) - Define recipe structures
|
||||
3. **Manifest** (`scripts/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.)
|
||||
4. **Generator** (`scripts/generate_recipe.py`) - Composes fragments via `%ksappend` directives
|
||||
2. **Templates** (`recipe-generator/recipe_templates.yaml`) - Define recipe structures
|
||||
3. **Manifest** (`recipe-generator/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.)
|
||||
4. **Generator** (`recipe-generator/generate_recipe.py`) - Composes fragments via `%ksappend` directives
|
||||
5. **Recipes** (`recipes/*.cfg`) - Generated kickstart files with fragment references
|
||||
6. **Flattening** (`ksflatten`) - Resolves `%ksappend` into single dish file
|
||||
7. **Deployment** (`virt-install`) - Creates VMs from dish files
|
||||
@@ -169,7 +171,7 @@ EOF
|
||||
|
||||
### Fragment Validation Script
|
||||
|
||||
Create `scripts/validate-fragment.sh`:
|
||||
Create `deploy/validate-fragment.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
@@ -205,7 +207,7 @@ Recipes are generated from templates and the manifest file. This section covers
|
||||
|
||||
### Manifest Editing
|
||||
|
||||
**File:** `scripts/recipes_manifest.yaml`
|
||||
**File:** `recipe-generator/recipes_manifest.yaml`
|
||||
|
||||
The manifest defines all recipe variants using modifiers from templates.
|
||||
|
||||
@@ -252,7 +254,7 @@ recipes:
|
||||
|
||||
### Template Editing
|
||||
|
||||
**File:** `scripts/recipe_templates.yaml`
|
||||
**File:** `recipe-generator/recipe_templates.yaml`
|
||||
|
||||
Templates define the structure and fragment composition for each recipe type.
|
||||
|
||||
@@ -551,7 +553,7 @@ done
|
||||
**File:** `.gitea/workflows/test-generation.yaml`
|
||||
|
||||
**Triggers:**
|
||||
- Push to `scripts/**/*.py` or `scripts/**/*.yaml`
|
||||
- Push to `recipe-generator/**/*.py` or `recipe-generator/**/*.yaml`
|
||||
- Pull request with script changes
|
||||
|
||||
**Steps:**
|
||||
@@ -609,12 +611,12 @@ luanti
|
||||
EOF
|
||||
|
||||
# Step 2: Add to recipe template
|
||||
# Edit scripts/recipe_templates.yaml
|
||||
# Edit recipe-generator/recipe_templates.yaml
|
||||
# Add to 'required' section:
|
||||
# - luanti: fragments/shared/packages/luanti.ks
|
||||
|
||||
# Step 3: Regenerate recipes
|
||||
cd scripts
|
||||
cd recipe-generator
|
||||
make generate-recipes
|
||||
|
||||
# Step 4: Validate
|
||||
@@ -637,12 +639,12 @@ plasma-workspace
|
||||
EOF
|
||||
|
||||
# Step 2: Add to template
|
||||
# Edit scripts/recipe_templates.yaml
|
||||
# Edit recipe-generator/recipe_templates.yaml
|
||||
# Add to optional/desktop section:
|
||||
# kde: fragments/shared/desktop/kde/packages.ks
|
||||
|
||||
# Step 3: Add variant to manifest
|
||||
# Edit scripts/recipes_manifest.yaml
|
||||
# Edit recipe-generator/recipes_manifest.yaml
|
||||
# Add variant:
|
||||
# - version: 43
|
||||
# desktop: kde
|
||||
@@ -660,7 +662,7 @@ make test
|
||||
|
||||
```bash
|
||||
# Step 1: Add template to recipe_templates.yaml
|
||||
cat >> scripts/recipe_templates.yaml << 'EOF'
|
||||
cat >> recipe-generator/recipe_templates.yaml << 'EOF'
|
||||
|
||||
minimal-server:
|
||||
description: "A minimal server recipe"
|
||||
@@ -681,7 +683,7 @@ cat >> scripts/recipe_templates.yaml << 'EOF'
|
||||
EOF
|
||||
|
||||
# Step 2: Add variant to recipes_manifest.yaml
|
||||
cat >> scripts/recipes_manifest.yaml << 'EOF'
|
||||
cat >> recipe-generator/recipes_manifest.yaml << 'EOF'
|
||||
|
||||
- name: minimal-server
|
||||
variants:
|
||||
@@ -729,7 +731,7 @@ luanti
|
||||
%end
|
||||
```
|
||||
|
||||
**Template:** `scripts/recipe_templates.yaml`
|
||||
**Template:** `recipe-generator/recipe_templates.yaml`
|
||||
```yaml
|
||||
templates:
|
||||
virtual-desktop:
|
||||
|
||||
@@ -10,7 +10,7 @@ sudo dnf install qemu libvirt virt-install pykickstart
|
||||
pip install PyYAML pytest
|
||||
|
||||
# Verify setup
|
||||
cd scripts && make generate-recipes && make test
|
||||
cd recipe-generator && make generate-recipes && make test
|
||||
```
|
||||
|
||||
## Core Workflows
|
||||
@@ -28,17 +28,17 @@ new-package
|
||||
### Add to Recipe
|
||||
|
||||
```bash
|
||||
# Edit scripts/recipe_templates.yaml to include fragment
|
||||
# Edit scripts/recipes_manifest.yaml to add variant
|
||||
# Edit recipe-generator/recipe_templates.yaml to include fragment
|
||||
# Edit recipe-generator/recipes_manifest.yaml to add variant
|
||||
|
||||
# Regenerate
|
||||
cd scripts && make generate-recipes && make validate-recipes
|
||||
cd recipe-generator && make generate-recipes && make validate-recipes
|
||||
```
|
||||
|
||||
### Run Tests
|
||||
|
||||
```bash
|
||||
cd scripts
|
||||
cd recipe-generator
|
||||
make test # All tests
|
||||
make test-integration # Integration only
|
||||
make test-container # Containerized
|
||||
@@ -60,7 +60,7 @@ done
|
||||
## Architecture
|
||||
|
||||
```
|
||||
fragments/ (54 .ks) → generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg)
|
||||
fragments/ (54 .ks) → recipe-generator/generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg)
|
||||
```
|
||||
|
||||
See `DEVELOPMENT.md` Section 1 for detailed architecture overview.
|
||||
|
||||
@@ -15,7 +15,7 @@ chmod +x deploy-vm.sh
|
||||
```
|
||||
./deploy-vm.sh
|
||||
|
||||
Executing: ./scripts/core-count.sh
|
||||
Executing: ./deploy/core-count.sh
|
||||
System has more than 2 core (nproc --all: 6).
|
||||
[...]
|
||||
10. virtual-desktop-hypervisor
|
||||
|
||||
Executable
+2
@@ -0,0 +1,2 @@
|
||||
#!/bin/bash
|
||||
exec "$(dirname "$0")/../recipe-generator/generate_recipe.py" "$@"
|
||||
@@ -13,10 +13,10 @@ execute_script() {
|
||||
|
||||
# Array of scripts
|
||||
scripts=(
|
||||
"./scripts/install-prerequisites-on-linux.sh"
|
||||
"./scripts/core-count.sh"
|
||||
"./scripts/system-memory.sh"
|
||||
"./scripts/deploy-distro.sh"
|
||||
"./deploy/install-prerequisites-on-linux.sh"
|
||||
"./deploy/core-count.sh"
|
||||
"./deploy/system-memory.sh"
|
||||
"./deploy/deploy-distro.sh"
|
||||
)
|
||||
|
||||
# Iterate through the scripts and execute them
|
||||
|
||||
@@ -25,13 +25,13 @@ validate-recipes:
|
||||
--validate ../recipes/*.cfg
|
||||
|
||||
test:
|
||||
python3 -m pytest tests/ -v --tb=short
|
||||
python3 -m pytest ../tests/ -v --tb=short
|
||||
|
||||
test-integration:
|
||||
python3 -m pytest tests/integration/ -v --tb=short
|
||||
python3 -m pytest ../tests/integration/ -v --tb=short
|
||||
|
||||
test-container:
|
||||
podman build -t phyllo/test-runner tests/container/
|
||||
podman build -t phyllo/test-runner ../tests/container/
|
||||
podman run --rm -v .:/phyllomeos:ro phyllo/test-runner
|
||||
|
||||
clean:
|
||||
BIN
Binary file not shown.
Regular → Executable
+4
-6
@@ -69,7 +69,6 @@ class RecipeGenerator:
|
||||
"""Generate kickstart recipes from templates and modifiers."""
|
||||
|
||||
def __init__(self, ingredients_dir: Path, templates_file: Path):
|
||||
# Resolve ingredients_dir relative to the project root (parent of scripts/)
|
||||
self.project_root = Path(__file__).parent.parent
|
||||
self.ingredients_dir = self.project_root / ingredients_dir
|
||||
self.templates = self.load_templates(templates_file)
|
||||
@@ -90,7 +89,7 @@ class RecipeGenerator:
|
||||
if path.is_absolute():
|
||||
template_path = path
|
||||
else:
|
||||
template_path = self.project_root / 'scripts' / path
|
||||
template_path = self.project_root / path
|
||||
with open(template_path) as f:
|
||||
data = yaml.safe_load(f)
|
||||
return data['templates']
|
||||
@@ -419,7 +418,6 @@ def main():
|
||||
)
|
||||
|
||||
# Global options
|
||||
# Use __file__ to find the scripts directory, then go up to project root
|
||||
SCRIPTS_DIR = Path(__file__).resolve().parent
|
||||
PROJECT_ROOT = SCRIPTS_DIR.parent
|
||||
|
||||
@@ -428,14 +426,14 @@ def main():
|
||||
help='Ingredients directory (default: parent/ingredients)')
|
||||
parser.add_argument('--templates', '-t',
|
||||
type=Path, default=SCRIPTS_DIR / 'recipe_templates.yaml',
|
||||
help='Templates YAML file (default: parent/recipe_templates.yaml)')
|
||||
help='Templates YAML file (default: ./recipe_templates.yaml)')
|
||||
|
||||
# Batch mode
|
||||
parser.add_argument('--manifest', '-m',
|
||||
type=Path, help='Manifest YAML for batch generation')
|
||||
parser.add_argument('--output-dir', '-d',
|
||||
type=Path, default=Path(__file__).parent / 'recipes',
|
||||
help='Output directory (batch generation, default: parent/recipes)')
|
||||
type=Path, default=SCRIPTS_DIR / 'recipes',
|
||||
help='Output directory (batch generation, default: ./recipes)')
|
||||
parser.add_argument('--dry-run', '-n',
|
||||
action='store_true',
|
||||
help='Show what would be generated without writing files')
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -11,7 +11,7 @@ import os
|
||||
|
||||
# Use actual project root
|
||||
PROJECT_ROOT = Path('/home/lukas/Code/virt/phyllomeos')
|
||||
SCRIPTS_DIR = PROJECT_ROOT / 'scripts'
|
||||
RECIPE_GENERATOR_DIR = PROJECT_ROOT / 'recipe-generator'
|
||||
RECIPE_DIR = PROJECT_ROOT / 'recipes'
|
||||
FRAGMENTS_DIR = PROJECT_ROOT / 'fragments'
|
||||
CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
|
||||
@@ -19,8 +19,8 @@ CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
|
||||
|
||||
def test_generate_recipes_from_manifest():
|
||||
"""Test generating all recipes from manifest."""
|
||||
os.chdir(SCRIPTS_DIR)
|
||||
|
||||
os.chdir(RECIPE_GENERATOR_DIR)
|
||||
|
||||
result = subprocess.run(
|
||||
['python3', 'generate_recipe.py',
|
||||
'--manifest', 'recipes_manifest.yaml',
|
||||
@@ -28,7 +28,7 @@ def test_generate_recipes_from_manifest():
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
|
||||
assert result.returncode == 0, f"Recipe generation failed: {result.stderr}"
|
||||
assert 'Generating:' in result.stdout
|
||||
|
||||
@@ -51,7 +51,7 @@ def test_make_targets():
|
||||
# Test generate-recipes
|
||||
result = subprocess.run(
|
||||
['make', 'generate-recipes'],
|
||||
cwd=SCRIPTS_DIR,
|
||||
cwd=RECIPE_GENERATOR_DIR,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
@@ -3,11 +3,9 @@
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add scripts directory to path
|
||||
SCRIPTS_DIR = Path(__file__).parent.parent / 'scripts'
|
||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
||||
RECIPE_GENERATOR_DIR = Path(__file__).parent.parent / 'recipe-generator'
|
||||
sys.path.insert(0, str(RECIPE_GENERATOR_DIR))
|
||||
|
||||
from generate_recipe import RecipeGenerator
|
||||
|
||||
@@ -19,7 +17,11 @@ class TestRecipeGenerator:
|
||||
|
||||
def setup_method(self):
|
||||
"""Set up test fixtures."""
|
||||
self.generator = RecipeGenerator(Path('ingredients'), Path('recipe_templates.yaml'))
|
||||
project_root = Path(__file__).parent.parent
|
||||
self.generator = RecipeGenerator(
|
||||
project_root / 'ingredients',
|
||||
project_root / 'recipe-generator' / 'recipe_templates.yaml'
|
||||
)
|
||||
|
||||
def test_template_loading(self):
|
||||
"""Test that templates are loaded correctly."""
|
||||
|
||||
Reference in New Issue
Block a user