refactor: functional separation with deploy/, recipe-generator/, and bin/
- deploy/: deployment bash scripts (4 files) - recipe-generator/: recipe generation tool (Python + YAML configs) - bin/: executable wrappers (single generate-recipe entry point) - Updated deploy.sh, README.md, DEVELOPMENT.md, DEVELOPMENT_QUICK.md - Updated tests/test_recipe_generator.py and tests/integration/test_integration.py - Updated CI workflow and Makefile paths - All 41 tests pass
This commit is contained in:
@@ -3,13 +3,13 @@ name: test-generation
|
|||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
paths:
|
paths:
|
||||||
- 'scripts/**/*.py'
|
- 'recipe-generator/**/*.py'
|
||||||
- 'scripts/**/*.yaml'
|
- 'recipe-generator/**/*.yaml'
|
||||||
- 'fragments/**/*.ks'
|
- 'fragments/**/*.ks'
|
||||||
pull_request:
|
pull_request:
|
||||||
paths:
|
paths:
|
||||||
- 'scripts/**/*.py'
|
- 'recipe-generator/**/*.py'
|
||||||
- 'scripts/**/*.yaml'
|
- 'recipe-generator/**/*.yaml'
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
generate:
|
generate:
|
||||||
@@ -28,7 +28,7 @@ jobs:
|
|||||||
|
|
||||||
- name: Generate all variants
|
- name: Generate all variants
|
||||||
run: |
|
run: |
|
||||||
cd scripts
|
cd recipe-generator
|
||||||
python3 generate_recipe.py \
|
python3 generate_recipe.py \
|
||||||
--manifest recipes_manifest.yaml \
|
--manifest recipes_manifest.yaml \
|
||||||
--output-dir ../recipes/
|
--output-dir ../recipes/
|
||||||
|
|||||||
+18
-16
@@ -21,7 +21,7 @@ Phyllome OS uses a **fragment-driven** kickstart generation system:
|
|||||||
```
|
```
|
||||||
fragments/ (54 .ks files)
|
fragments/ (54 .ks files)
|
||||||
↓ (modular snippets)
|
↓ (modular snippets)
|
||||||
scripts/generate_recipe.py
|
recipe-generator/generate_recipe.py
|
||||||
↓ (YAML templates + manifest)
|
↓ (YAML templates + manifest)
|
||||||
recipes/ (16 auto-generated .cfg)
|
recipes/ (16 auto-generated .cfg)
|
||||||
↓ (ksflatten)
|
↓ (ksflatten)
|
||||||
@@ -40,14 +40,16 @@ VMs and ISO images
|
|||||||
| `recipes/` | Generated recipes | Manifest-driven compositions |
|
| `recipes/` | Generated recipes | Manifest-driven compositions |
|
||||||
| `dishes/` | Flattened kickstarts | Ready-to-deploy artifacts |
|
| `dishes/` | Flattened kickstarts | Ready-to-deploy artifacts |
|
||||||
| `ingredients/` | Legacy building blocks | 35 `.cfg` files (legacy) |
|
| `ingredients/` | Legacy building blocks | 35 `.cfg` files (legacy) |
|
||||||
| `scripts/` | Automation tools | `generate_recipe.py`, Makefile |
|
| `recipe-generator/` | Recipe generation | `generate_recipe.py`, YAML configs, Makefile |
|
||||||
|
| `deploy/` | Deployment scripts | Bash automation tools |
|
||||||
|
| `bin/` | Executables | Wrapper scripts (e.g., `generate-recipe`) |
|
||||||
|
|
||||||
### Data Flow
|
### Data Flow
|
||||||
|
|
||||||
1. **Fragments** (`fragments/**/*.ks`) - Small, reusable kickstart snippets
|
1. **Fragments** (`fragments/**/*.ks`) - Small, reusable kickstart snippets
|
||||||
2. **Templates** (`scripts/recipe_templates.yaml`) - Define recipe structures
|
2. **Templates** (`recipe-generator/recipe_templates.yaml`) - Define recipe structures
|
||||||
3. **Manifest** (`scripts/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.)
|
3. **Manifest** (`recipe-generator/recipes_manifest.yaml`) - Specify variants (version, desktop, storage, etc.)
|
||||||
4. **Generator** (`scripts/generate_recipe.py`) - Composes fragments via `%ksappend` directives
|
4. **Generator** (`recipe-generator/generate_recipe.py`) - Composes fragments via `%ksappend` directives
|
||||||
5. **Recipes** (`recipes/*.cfg`) - Generated kickstart files with fragment references
|
5. **Recipes** (`recipes/*.cfg`) - Generated kickstart files with fragment references
|
||||||
6. **Flattening** (`ksflatten`) - Resolves `%ksappend` into single dish file
|
6. **Flattening** (`ksflatten`) - Resolves `%ksappend` into single dish file
|
||||||
7. **Deployment** (`virt-install`) - Creates VMs from dish files
|
7. **Deployment** (`virt-install`) - Creates VMs from dish files
|
||||||
@@ -169,7 +171,7 @@ EOF
|
|||||||
|
|
||||||
### Fragment Validation Script
|
### Fragment Validation Script
|
||||||
|
|
||||||
Create `scripts/validate-fragment.sh`:
|
Create `deploy/validate-fragment.sh`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
@@ -205,7 +207,7 @@ Recipes are generated from templates and the manifest file. This section covers
|
|||||||
|
|
||||||
### Manifest Editing
|
### Manifest Editing
|
||||||
|
|
||||||
**File:** `scripts/recipes_manifest.yaml`
|
**File:** `recipe-generator/recipes_manifest.yaml`
|
||||||
|
|
||||||
The manifest defines all recipe variants using modifiers from templates.
|
The manifest defines all recipe variants using modifiers from templates.
|
||||||
|
|
||||||
@@ -252,7 +254,7 @@ recipes:
|
|||||||
|
|
||||||
### Template Editing
|
### Template Editing
|
||||||
|
|
||||||
**File:** `scripts/recipe_templates.yaml`
|
**File:** `recipe-generator/recipe_templates.yaml`
|
||||||
|
|
||||||
Templates define the structure and fragment composition for each recipe type.
|
Templates define the structure and fragment composition for each recipe type.
|
||||||
|
|
||||||
@@ -551,7 +553,7 @@ done
|
|||||||
**File:** `.gitea/workflows/test-generation.yaml`
|
**File:** `.gitea/workflows/test-generation.yaml`
|
||||||
|
|
||||||
**Triggers:**
|
**Triggers:**
|
||||||
- Push to `scripts/**/*.py` or `scripts/**/*.yaml`
|
- Push to `recipe-generator/**/*.py` or `recipe-generator/**/*.yaml`
|
||||||
- Pull request with script changes
|
- Pull request with script changes
|
||||||
|
|
||||||
**Steps:**
|
**Steps:**
|
||||||
@@ -609,12 +611,12 @@ luanti
|
|||||||
EOF
|
EOF
|
||||||
|
|
||||||
# Step 2: Add to recipe template
|
# Step 2: Add to recipe template
|
||||||
# Edit scripts/recipe_templates.yaml
|
# Edit recipe-generator/recipe_templates.yaml
|
||||||
# Add to 'required' section:
|
# Add to 'required' section:
|
||||||
# - luanti: fragments/shared/packages/luanti.ks
|
# - luanti: fragments/shared/packages/luanti.ks
|
||||||
|
|
||||||
# Step 3: Regenerate recipes
|
# Step 3: Regenerate recipes
|
||||||
cd scripts
|
cd recipe-generator
|
||||||
make generate-recipes
|
make generate-recipes
|
||||||
|
|
||||||
# Step 4: Validate
|
# Step 4: Validate
|
||||||
@@ -637,12 +639,12 @@ plasma-workspace
|
|||||||
EOF
|
EOF
|
||||||
|
|
||||||
# Step 2: Add to template
|
# Step 2: Add to template
|
||||||
# Edit scripts/recipe_templates.yaml
|
# Edit recipe-generator/recipe_templates.yaml
|
||||||
# Add to optional/desktop section:
|
# Add to optional/desktop section:
|
||||||
# kde: fragments/shared/desktop/kde/packages.ks
|
# kde: fragments/shared/desktop/kde/packages.ks
|
||||||
|
|
||||||
# Step 3: Add variant to manifest
|
# Step 3: Add variant to manifest
|
||||||
# Edit scripts/recipes_manifest.yaml
|
# Edit recipe-generator/recipes_manifest.yaml
|
||||||
# Add variant:
|
# Add variant:
|
||||||
# - version: 43
|
# - version: 43
|
||||||
# desktop: kde
|
# desktop: kde
|
||||||
@@ -660,7 +662,7 @@ make test
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Step 1: Add template to recipe_templates.yaml
|
# Step 1: Add template to recipe_templates.yaml
|
||||||
cat >> scripts/recipe_templates.yaml << 'EOF'
|
cat >> recipe-generator/recipe_templates.yaml << 'EOF'
|
||||||
|
|
||||||
minimal-server:
|
minimal-server:
|
||||||
description: "A minimal server recipe"
|
description: "A minimal server recipe"
|
||||||
@@ -681,7 +683,7 @@ cat >> scripts/recipe_templates.yaml << 'EOF'
|
|||||||
EOF
|
EOF
|
||||||
|
|
||||||
# Step 2: Add variant to recipes_manifest.yaml
|
# Step 2: Add variant to recipes_manifest.yaml
|
||||||
cat >> scripts/recipes_manifest.yaml << 'EOF'
|
cat >> recipe-generator/recipes_manifest.yaml << 'EOF'
|
||||||
|
|
||||||
- name: minimal-server
|
- name: minimal-server
|
||||||
variants:
|
variants:
|
||||||
@@ -729,7 +731,7 @@ luanti
|
|||||||
%end
|
%end
|
||||||
```
|
```
|
||||||
|
|
||||||
**Template:** `scripts/recipe_templates.yaml`
|
**Template:** `recipe-generator/recipe_templates.yaml`
|
||||||
```yaml
|
```yaml
|
||||||
templates:
|
templates:
|
||||||
virtual-desktop:
|
virtual-desktop:
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ sudo dnf install qemu libvirt virt-install pykickstart
|
|||||||
pip install PyYAML pytest
|
pip install PyYAML pytest
|
||||||
|
|
||||||
# Verify setup
|
# Verify setup
|
||||||
cd scripts && make generate-recipes && make test
|
cd recipe-generator && make generate-recipes && make test
|
||||||
```
|
```
|
||||||
|
|
||||||
## Core Workflows
|
## Core Workflows
|
||||||
@@ -28,17 +28,17 @@ new-package
|
|||||||
### Add to Recipe
|
### Add to Recipe
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Edit scripts/recipe_templates.yaml to include fragment
|
# Edit recipe-generator/recipe_templates.yaml to include fragment
|
||||||
# Edit scripts/recipes_manifest.yaml to add variant
|
# Edit recipe-generator/recipes_manifest.yaml to add variant
|
||||||
|
|
||||||
# Regenerate
|
# Regenerate
|
||||||
cd scripts && make generate-recipes && make validate-recipes
|
cd recipe-generator && make generate-recipes && make validate-recipes
|
||||||
```
|
```
|
||||||
|
|
||||||
### Run Tests
|
### Run Tests
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd scripts
|
cd recipe-generator
|
||||||
make test # All tests
|
make test # All tests
|
||||||
make test-integration # Integration only
|
make test-integration # Integration only
|
||||||
make test-container # Containerized
|
make test-container # Containerized
|
||||||
@@ -60,7 +60,7 @@ done
|
|||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
fragments/ (54 .ks) → generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg)
|
fragments/ (54 .ks) → recipe-generator/generate_recipe.py → recipes/ (16 .cfg) → ksflatten → dishes/ (28 .cfg)
|
||||||
```
|
```
|
||||||
|
|
||||||
See `DEVELOPMENT.md` Section 1 for detailed architecture overview.
|
See `DEVELOPMENT.md` Section 1 for detailed architecture overview.
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ chmod +x deploy-vm.sh
|
|||||||
```
|
```
|
||||||
./deploy-vm.sh
|
./deploy-vm.sh
|
||||||
|
|
||||||
Executing: ./scripts/core-count.sh
|
Executing: ./deploy/core-count.sh
|
||||||
System has more than 2 core (nproc --all: 6).
|
System has more than 2 core (nproc --all: 6).
|
||||||
[...]
|
[...]
|
||||||
10. virtual-desktop-hypervisor
|
10. virtual-desktop-hypervisor
|
||||||
|
|||||||
Executable
+2
@@ -0,0 +1,2 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
exec "$(dirname "$0")/../recipe-generator/generate_recipe.py" "$@"
|
||||||
@@ -13,10 +13,10 @@ execute_script() {
|
|||||||
|
|
||||||
# Array of scripts
|
# Array of scripts
|
||||||
scripts=(
|
scripts=(
|
||||||
"./scripts/install-prerequisites-on-linux.sh"
|
"./deploy/install-prerequisites-on-linux.sh"
|
||||||
"./scripts/core-count.sh"
|
"./deploy/core-count.sh"
|
||||||
"./scripts/system-memory.sh"
|
"./deploy/system-memory.sh"
|
||||||
"./scripts/deploy-distro.sh"
|
"./deploy/deploy-distro.sh"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Iterate through the scripts and execute them
|
# Iterate through the scripts and execute them
|
||||||
|
|||||||
@@ -25,13 +25,13 @@ validate-recipes:
|
|||||||
--validate ../recipes/*.cfg
|
--validate ../recipes/*.cfg
|
||||||
|
|
||||||
test:
|
test:
|
||||||
python3 -m pytest tests/ -v --tb=short
|
python3 -m pytest ../tests/ -v --tb=short
|
||||||
|
|
||||||
test-integration:
|
test-integration:
|
||||||
python3 -m pytest tests/integration/ -v --tb=short
|
python3 -m pytest ../tests/integration/ -v --tb=short
|
||||||
|
|
||||||
test-container:
|
test-container:
|
||||||
podman build -t phyllo/test-runner tests/container/
|
podman build -t phyllo/test-runner ../tests/container/
|
||||||
podman run --rm -v .:/phyllomeos:ro phyllo/test-runner
|
podman run --rm -v .:/phyllomeos:ro phyllo/test-runner
|
||||||
|
|
||||||
clean:
|
clean:
|
||||||
BIN
Binary file not shown.
Regular → Executable
+4
-6
@@ -69,7 +69,6 @@ class RecipeGenerator:
|
|||||||
"""Generate kickstart recipes from templates and modifiers."""
|
"""Generate kickstart recipes from templates and modifiers."""
|
||||||
|
|
||||||
def __init__(self, ingredients_dir: Path, templates_file: Path):
|
def __init__(self, ingredients_dir: Path, templates_file: Path):
|
||||||
# Resolve ingredients_dir relative to the project root (parent of scripts/)
|
|
||||||
self.project_root = Path(__file__).parent.parent
|
self.project_root = Path(__file__).parent.parent
|
||||||
self.ingredients_dir = self.project_root / ingredients_dir
|
self.ingredients_dir = self.project_root / ingredients_dir
|
||||||
self.templates = self.load_templates(templates_file)
|
self.templates = self.load_templates(templates_file)
|
||||||
@@ -90,7 +89,7 @@ class RecipeGenerator:
|
|||||||
if path.is_absolute():
|
if path.is_absolute():
|
||||||
template_path = path
|
template_path = path
|
||||||
else:
|
else:
|
||||||
template_path = self.project_root / 'scripts' / path
|
template_path = self.project_root / path
|
||||||
with open(template_path) as f:
|
with open(template_path) as f:
|
||||||
data = yaml.safe_load(f)
|
data = yaml.safe_load(f)
|
||||||
return data['templates']
|
return data['templates']
|
||||||
@@ -419,7 +418,6 @@ def main():
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Global options
|
# Global options
|
||||||
# Use __file__ to find the scripts directory, then go up to project root
|
|
||||||
SCRIPTS_DIR = Path(__file__).resolve().parent
|
SCRIPTS_DIR = Path(__file__).resolve().parent
|
||||||
PROJECT_ROOT = SCRIPTS_DIR.parent
|
PROJECT_ROOT = SCRIPTS_DIR.parent
|
||||||
|
|
||||||
@@ -428,14 +426,14 @@ def main():
|
|||||||
help='Ingredients directory (default: parent/ingredients)')
|
help='Ingredients directory (default: parent/ingredients)')
|
||||||
parser.add_argument('--templates', '-t',
|
parser.add_argument('--templates', '-t',
|
||||||
type=Path, default=SCRIPTS_DIR / 'recipe_templates.yaml',
|
type=Path, default=SCRIPTS_DIR / 'recipe_templates.yaml',
|
||||||
help='Templates YAML file (default: parent/recipe_templates.yaml)')
|
help='Templates YAML file (default: ./recipe_templates.yaml)')
|
||||||
|
|
||||||
# Batch mode
|
# Batch mode
|
||||||
parser.add_argument('--manifest', '-m',
|
parser.add_argument('--manifest', '-m',
|
||||||
type=Path, help='Manifest YAML for batch generation')
|
type=Path, help='Manifest YAML for batch generation')
|
||||||
parser.add_argument('--output-dir', '-d',
|
parser.add_argument('--output-dir', '-d',
|
||||||
type=Path, default=Path(__file__).parent / 'recipes',
|
type=Path, default=SCRIPTS_DIR / 'recipes',
|
||||||
help='Output directory (batch generation, default: parent/recipes)')
|
help='Output directory (batch generation, default: ./recipes)')
|
||||||
parser.add_argument('--dry-run', '-n',
|
parser.add_argument('--dry-run', '-n',
|
||||||
action='store_true',
|
action='store_true',
|
||||||
help='Show what would be generated without writing files')
|
help='Show what would be generated without writing files')
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -11,7 +11,7 @@ import os
|
|||||||
|
|
||||||
# Use actual project root
|
# Use actual project root
|
||||||
PROJECT_ROOT = Path('/home/lukas/Code/virt/phyllomeos')
|
PROJECT_ROOT = Path('/home/lukas/Code/virt/phyllomeos')
|
||||||
SCRIPTS_DIR = PROJECT_ROOT / 'scripts'
|
RECIPE_GENERATOR_DIR = PROJECT_ROOT / 'recipe-generator'
|
||||||
RECIPE_DIR = PROJECT_ROOT / 'recipes'
|
RECIPE_DIR = PROJECT_ROOT / 'recipes'
|
||||||
FRAGMENTS_DIR = PROJECT_ROOT / 'fragments'
|
FRAGMENTS_DIR = PROJECT_ROOT / 'fragments'
|
||||||
CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
|
CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
|
||||||
@@ -19,7 +19,7 @@ CONTAINER_DIR = PROJECT_ROOT / 'tests' / 'container'
|
|||||||
|
|
||||||
def test_generate_recipes_from_manifest():
|
def test_generate_recipes_from_manifest():
|
||||||
"""Test generating all recipes from manifest."""
|
"""Test generating all recipes from manifest."""
|
||||||
os.chdir(SCRIPTS_DIR)
|
os.chdir(RECIPE_GENERATOR_DIR)
|
||||||
|
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
['python3', 'generate_recipe.py',
|
['python3', 'generate_recipe.py',
|
||||||
@@ -51,7 +51,7 @@ def test_make_targets():
|
|||||||
# Test generate-recipes
|
# Test generate-recipes
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
['make', 'generate-recipes'],
|
['make', 'generate-recipes'],
|
||||||
cwd=SCRIPTS_DIR,
|
cwd=RECIPE_GENERATOR_DIR,
|
||||||
capture_output=True,
|
capture_output=True,
|
||||||
text=True
|
text=True
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -3,11 +3,9 @@
|
|||||||
import pytest
|
import pytest
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
import sys
|
import sys
|
||||||
import os
|
|
||||||
|
|
||||||
# Add scripts directory to path
|
RECIPE_GENERATOR_DIR = Path(__file__).parent.parent / 'recipe-generator'
|
||||||
SCRIPTS_DIR = Path(__file__).parent.parent / 'scripts'
|
sys.path.insert(0, str(RECIPE_GENERATOR_DIR))
|
||||||
sys.path.insert(0, str(SCRIPTS_DIR))
|
|
||||||
|
|
||||||
from generate_recipe import RecipeGenerator
|
from generate_recipe import RecipeGenerator
|
||||||
|
|
||||||
@@ -19,7 +17,11 @@ class TestRecipeGenerator:
|
|||||||
|
|
||||||
def setup_method(self):
|
def setup_method(self):
|
||||||
"""Set up test fixtures."""
|
"""Set up test fixtures."""
|
||||||
self.generator = RecipeGenerator(Path('ingredients'), Path('recipe_templates.yaml'))
|
project_root = Path(__file__).parent.parent
|
||||||
|
self.generator = RecipeGenerator(
|
||||||
|
project_root / 'ingredients',
|
||||||
|
project_root / 'recipe-generator' / 'recipe_templates.yaml'
|
||||||
|
)
|
||||||
|
|
||||||
def test_template_loading(self):
|
def test_template_loading(self):
|
||||||
"""Test that templates are loaded correctly."""
|
"""Test that templates are loaded correctly."""
|
||||||
|
|||||||
Reference in New Issue
Block a user