Commit Graph

24 Commits

Author SHA1 Message Date
Peter Krempa
7a1285216b ci: Regenerate with latest 'lcitool' and fix manual job definitions
The latest lcitool merged the 'prebuilt-env' and 'local-env' jobs into
one which use variables to pick up the right environment and steps
rather than duplicating everything.

Regenerate the generated job definitions, fix the helper definitions
and also fix the manually defined jobs (website-job).

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2024-01-19 14:48:21 +01:00
Andrea Bolognani
10042f0253 ci: Do more as part of .qemu-build-template
Entering $SCRATCH_DIR, going back to the original directory and
setting SELinux labels for the newly-installed QEMU binaries
are all steps that logically belong to this template rather
than its callers.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2024-01-12 14:28:43 +01:00
Andrea Bolognani
9cf0faac48 ci: Fix .integration_tests_upstream_qemu
We enter $SCRATCH_DIR before going through the process of
cloning QEMU's upstream repo and building it, but once we're
done we don't get back to libvirt's sources, so the very next
step fails with

  /tmp/script.: line 188: ci/jobs.sh: No such file or directory

Use pushd/popd to ensure that we're back to the correct place
once QEMU has been built and installed.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2024-01-12 14:28:38 +01:00
Erik Skultety
b0061e77c8 ci: integration: Drop the 'install-deps' hidden job and reference
Since the section now only consists of a single command, we can happily
move the command to the main integration template job body.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-19 13:35:58 +02:00
Erik Skultety
6e9bd600d7 ci: integration: Extract the integration CI main recipe to jobs.sh
Follow what's been done to other jobs in .gitlab-ci.yml and extract the
shell logic from YAML to a function in ci/jobs.sh

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-19 13:35:58 +02:00
Erik Skultety
6d46808a0e ci: integration: Drop the Fedora 35 check
We're already past Fedora 35 and so all new fedora's default to
modular daemon setup.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-08-11 14:55:56 +02:00
Erik Skultety
917749314c ci: Test with latest Avocado again
Test with the following fix:
https://github.com/avocado-framework/avocado/pull/5567/commits

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-02-08 16:36:10 +01:00
Erik Skultety
ad00beffa6 ci: integration: Set an expiration on logs artifacts
The default expiry time is 30 days. Since the RPM artifacts coming from
the previous pipeline stages are set to expire in 1 day we can set the
failed integration job log artifacts to the same value. The sentiment
here is that if an integration job legitimately failed (i.e. not with
an infrastructure failure) unless it was fixed in the meantime it will
fail the next day with the scheduled pipeline again, meaning, that even
if the older log artifacts are removed, they'll be immediately
replaced with fresh ones.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-01-12 09:57:52 +01:00
Erik Skultety
c9a65eb8a6 ci: integration: Lock the Avocado version to 98.0 for now
Avocado 99.0 causes the TCK test suite to fail with the nwfilter tests
(which is another Bash framework underneath). Until the culprit is
identified and fixed in Avocado, let's lock the version to 98.0 which
worked with the test suite just fine.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-11-16 10:14:20 +01:00
Erik Skultety
5597ae9609 ci: integration: Bundle only failed Avocado test results
Latest versions of Avocado create 'by-status' symlink shortcuts to test
results, IOW:

 # this is the main test results directory containing all data
 $ ls <path>/avocado/job-results/latest/test-results/
 01-scripts_networks_050-transient-lifecycle.t
 02-scripts_networks_051-transient-autostart.t
 ...
 22-scripts_networks_400-guest-bandwidth.t
 by_status/

 # list only the failed tests
 $ ls -l <path>/avocado/job-results/latest/test-results/by-status/FAIL
 19-scripts_networks_360-guest-network-vepa.t ->
    <path>/avocado/job-results/latest/test-results/19-scripts_networks_360-guest-network-vepa.t

Therefore, let's bundle only the failed ones, it's going to make the
log artifacts more obvious when looking for libvirt errors.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
2b26127397 ci: integration: Create the avocado logs artifact dir conditionally
Don't create an avocado directory in the resulting log artifacts
if Avocado didn't even run (e.g. libvirt errored out on service
restart).

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
5243dcea65 ci: integration: Use 'set +e' in the log collecting job
All 'script' blocks are defined as 'set -e' and so a single failed
return value means we won't collect some of the logs. Because of
the nature of the original job's failure some of the log sources
might not be available, but that's fine, however, the gitlab
after_script job cannot finish prematurely.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
124eb56342 ci: integration: Only create a coredump log when a core exists
It could be quite confusing looking at the job log artifacts and having
an empty coredump log in there, IOW it doesn't really give much
confidence that the reporting mechanism actually works.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
1df36ecbd0 ci: integration: Fix the test on avocado directory existence
It's a directory, so -d should be used with 'test'.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:43 +02:00
Erik Skultety
ddc4898080 ci: integration: Add an extra level of quoting for augeas set options
Both log filters and log outputs expect string values, however, augeas
apparently requires an extra level of quotes apart from the ones we
pass via shell (see comment [1]) to work properly, otherwise augeas
ignores the value and returns 0.
Without this fix we don't set libvirt's log level to debug, we don't
set logging to a file and hence we don't include the logs in CI
artifacts in case the test suite fails.

[1] https://github.com/hercules-team/augeas/issues/301#issuecomment-143699880

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:43 +02:00
Erik Skultety
045e642d29 ci: integration: Add virtnetworkd to the list of restarted services
It was missing from the set. While at it, order the daemon set
alphabetically.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:43 +02:00
Peter Krempa
bb0bec4827 ci: Install python RPMs from libvirt/libvirt-python CI artifacts
After addition of the new libvirt-client-qemu sub-package which is using
python bindings (thus creating a circular dependency between the libvirt
and libvirt-python projects) the integration jobs fail with:

  Error:
   Problem: conflicting requests
    - nothing provides python3-libvirt >= 8.9.0-1.el9 needed by libvirt-client-qemu-8.9.0-1.el9.x86_64

The libvirt-python project now provides the RPMs in artifacts:
 https://gitlab.com/libvirt/libvirt-python/-/merge_requests/96

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-10-13 12:23:01 +02:00
Daniel P. Berrangé
491d918502 ci: refresh with latest lcitool manifest
This refresh switches the CI for contributors to be triggered by merge
requests. Pushing to a branch in a fork will no longer run CI pipelines,
in order to avoid consuming CI minutes. To regain the original behaviour
contributors can opt-in to a pipeline on push

   git push <remote> -o ci.variable=RUN_PIPELINE=1

This variable can also be set globally on the repository, through the
web UI options Settings -> CI/CD -> Variables, though this is not
recommended. Upstream repo pushes to branches will run CI.

The use of containers has changed in this update, with only the upstream
repo creating containers, in order to avoid consuming contributors'
limited storage quotas. A fork with existing container images may delete
them. Containers will be rebuilt upstream when pushing commits with CI
changes to the default branch. Any other scenario with CI changes will
simply install build pre-requisite packages in a throaway environment,
using the ci/buildenv/ scripts. These scripts may also be used on a
contributor's local machines.

With pipelines triggered by merge requests, it is also now possible to
workaround the inability of contributors to run pipelines if they have
run out of CI quota. A project member can trigger a pipeline from the
merge request, which will run in context of upstream, however, note
this should only be done after reviewing the code for any malicious
CI changes.

Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-10-06 05:15:54 -04:00
Erik Skultety
d1e3247675 ci: integration: Set 'safe.directory' when installing QEMU from git
Since a fix for CVE-2022-24765 was released every git command is now
checked against the context repo in which it's supposed to run
resulting in a fatal error if the repo is owned by other user than the
one running the git command.
This means that in order to be able to do 'sudo make install', we have
to set the 'safe.directory' for the root user. This is because QEMU
runs 'git submodule update' automatically on 'make install'.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-06-09 13:38:17 +02:00
Erik Skultety
3b943f0c43 ci: integration: SELinux relabel the QEMU we installed from git
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-06-09 13:38:13 +02:00
Erik Skultety
80ffd571a8 ci: Add a new integration job template for the upstream QEMU scenario
YAML anchors don't work with Shell condition structures, so we cannot
simply reference the QEMU build template YAML anchor conditionally and
hence have everything as part of a single job template.
Instead, we have to "subclass" the .integration_tests template and
inject the QEMU building bits explicitly.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
de98075faf ci: Introduce a template for upstream QEMU build
This was heavily inspired by QEMU's upstream CI buildtest-template.yml.
Rather than referencing QEMU's template directly (which GitLab can do),
this patch resorts to hard-coding the build steps ourselves, solely
because there's no guarantee QEMU will keep either the template file
name or the template name from which the build steps were mostly copied
from.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
72887b8f59 ci: Break off the integration_tests template into more templates
There's quite a lot happening in the .integration_tests template
already even without adding upstream QEMU build into the mix.
Let's break the template into more pieces which can then reference
in the .integration_tests template when putting all the pieces back
together using YAML anchors.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
387932771a ci: Separate the integration job template to a separate file
Future patches will do more code extraction from the existing template
using YAML anchors so it'd be better that the templates would live
separately from job definitions.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00