Commit Graph

208 Commits

Author SHA1 Message Date
Erik Skultety
e2332fe72a ci: lcitool: Add libvirt-tck+runtime deps list
This change was supposed to be part of commit 120a674f , but was
proposed against the libvirt TCK project instead. Since we're running
the TCK test suite as part of this project, this is the right place for
the TCK runtime deps list config.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-01 13:30:57 +02:00
Andrea Bolognani
d40c6cad64 ci: Fix quoting and option name
Multiple values passed to --meson-args need to be quoted so that
the shell will interpret them correctly. The option's name was
also reported incorrectly, so fix that as well.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2023-08-28 14:21:54 +02:00
Andrea Bolognani
b68d253c46 ci: Fix precedence between arguments passed to meson
Commit 9c9848f955 merged $MESON_OPTS into $MESON_ARGS, and
while doing so changed their behavior: while until then the
contents of $MESON_ARGS had precedence over those of $MESON_OPTS,
now the opposite is true. Restore the original behavior and
document it.

The argument for merging the two variables in the first place
was that having both present on the meson command line could be
confusing; however, that should no longer be the case now that
we have reasonably extensive comments explaining the role of
each of the variables and how they interact with each other, so
return the meson command line to its original form.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2023-08-28 14:21:44 +02:00
Erik Skultety
120a674f25 ci: lcitool: Maintain project package deps lists here
Each respective project that lcitool knows about and currently
maintains its list of package dependencies knows best what packages
they actually depend on. If a new dependency is currently needed, first
a change in lcitool is necessary before GitLab jobs and containers can
be updated. Provided a mapping already exists in lcitool (which can
quickly be added as an override via mappings.yml temporarily) we speed
up the whole CI update process by one step.
This patch adds all libvirt deps lists lcitool currently maintains for
libvirt.

Note that as with any overrides (since commit f199dd50) lcitool must be
invoked as '$ lcitool -d/--data-dir ci/lcitool ...'

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-22 12:10:01 +02:00
Erik Skultety
bcc04f39ff ci: Udate FreeBSD-13 image with lcitool manifest
Now that we have a local OS target override for lcitool in place, we
can bump the cirrus FreeBSD image version in GitLab CI.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:06:32 +02:00
Erik Skultety
f199dd50df ci: Introduce a new 'lcitool' data directory
We've reached a point in lcitool where we can't steer its development
based solely on libvirt's needs IOW there will be times where a local
override of value (e.g. package mapping) will be necessary - an example
of this would be QEMU.
In case of this particular patch we need to add an override for the
cirrus FreeBSD 13 image we request in our CI to fix:

    /usr/local/lib/libtasn1.so.6: Undefined symbol "strverscmp@FBSD_1.7"

The reason why we can't/should not make the fix in upstream lcitool
just yet is that we store a libosinfo ID in lcitool's OS target YAML
configs and at the time of writing this patch libosinfo does not have
a corresponding entry/ID for FreeBSD 13.2 so we have to stick with 13.1
in lcitool until they do so.

For the time being, the fix can easily be done on libvirt side as does
this patch.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:06:32 +02:00
Erik Skultety
29ede88aa1 ci: Move Debian-11 workloads to Debian-12
Let's move our Debian CI workloads to Debian-12 since it's the latest
release and mark Debian-11 jobs as optional.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:05:18 +02:00
Erik Skultety
61360d91a7 ci: Add Debian-12 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:05:18 +02:00
Erik Skultety
9c9848f955 ci: build.sh: Join MESON_ARGS and MESON_OPTS
It is quite confusing seeing these two in a call like this one:
    $ meson build $MESON_OPTS $MESON_ARGS

One has to ask 'how are they different' and 'shouldn't these be
merged'. In fact, these variables hold very different things and we
should make it more obvious. The problem is that renaming MESON_OPTS to
something more meaningful, like 'MESON_CROSS_OPTS' which is what
MESON_OPTS really does would require changes to lcitool and would
impact Dockerfile generation which in turn might have an impact on
other projects which rely on this lcitool functionality which is risky.

Instead, provide a docstring for the former to supplement the latter
and join the two variables in a single one MESON_ARGS which is then
passed to meson's command line so it's a little less confusing.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:11:50 +02:00
Erik Skultety
6788b2d3cd ci: build.sh: Drop the CI prefix from the CI_{MESON,NINJA}_ARGS vars
Although it is currently consistent with the other variables we define
when running ci in a local container environment, it isn't consistent
with the variable naming we use in GitLab recipes. Since the idea is
to unite the two, we're likely going to drop a few other variables from
the local env configuration anyway, hence this renaming.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
ed1a0a63ce ci: build.sh: Always assume -Dsystem=true
There's no harm in always building in system mode, i.e. setting the
right paths.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
b845c2fd3b ci: build.sh: Use 'meson setup' explicitly
Even though 'setup' is assumed when no other command is given, we're
being explicit in our GitLab recipes, so do the same for the local
build.sh script too.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
d95b6e993f ci: build.sh: Drop the commentary about CI_BUILD_SCRIPT
build.sh is not the place where this should be mentioned as the
official entrypoint for this script locally is ci/helper which can
download the right image from our upstream CI registry. Since the idea
is to ultimately drop the usage of a Makefile for the local executions,
this patch doesn't provide an alternative place for the comment in
question as the functionality is going to be altered substantially in
the future.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
6d46808a0e ci: integration: Drop the Fedora 35 check
We're already past Fedora 35 and so all new fedora's default to
modular daemon setup.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-08-11 14:55:56 +02:00
Michal Privoznik
51ae80d6a7 ci: Switch to OpenSUSE Leap 15.5
OpenSUSE Leap was released recently (2023-06-07). Refresh our CI
with latest lcitool which brings this minor update.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-07-25 11:47:56 +02:00
Han Han
28141adfdc scripts: Fix the flake8 syntax-check failures
Fix the syntax-check failures (which can be seen after
python3-flake8-import-order package is installed) with the help
of isort[1]:

289/316 libvirt:syntax-check / flake8   FAIL   5.24s   exit status 2

[1]: https://pycqa.github.io/isort/

Signed-off-by: Han Han <hhan@redhat.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-06-29 11:51:27 +02:00
Daniel P. Berrangé
25420111ba ci: fix Cirrus CI jobs run from merge requests
Preferentially fetch from $CI_MERGE_REQUEST_REF_PATH if it is
defined, otherwise use $CI_COMMIT_REF_NAME

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2023-06-05 11:39:12 +01:00
Erik Skultety
adf67ca987 ci: refresh with latest lcitool manifest
Main lcitool changes:
- added Alpine 3.17 and 3.18 targets
- dropped Alpine 3.15 and 3.16

Note that we're not actively testing all Alpine targets due to CI
quota, so only 3.17 is used as a replacement for 3.15 in this patch.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-06-05 10:51:18 +02:00
Erik Skultety
dcb9ee99f3 ci: integration: Flip QEMU upstream integration tests to Fedora 38
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Erik Skultety
2bb0efa367 ci: Flip mingw jobs from Fedora 37 to Fedora 38
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Erik Skultety
ffd2a97070 ci: Drop Fedora 36 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Erik Skultety
96ffb7318e ci: Add Fedora 38 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Michal Privoznik
a3cc0e9ceb ci: Regenerate files
This removes minor version number from OpenSUSE LEAP target names
and on CentOS Stream 9 installs flake8 from repositories, instead
of pip.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2023-04-27 15:02:42 +02:00
Andrea Bolognani
5be133b39e ci: Regenerate files
This unbreaks the various $CROSS-$NAME-local-env jobs.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-04-04 10:51:53 +02:00
Martin Kletzander
e9adb4dcb1 Revert "ci: Disable optimization on macos-12"
This reverts commit 1f76b5365e.

There were two issues with this commit.  First is the missing propagation
of CFLAGS into the build environment and second is the fact that this is
not enough to disable the check for -fsemantic-interposition.  The
proper fix would require setting MESON_OPTS or similar and also add the
propagation of such variable into the cirrus builds etc., but at this
point I burned so much time on this trivial piece of rubbish that I
think it's easier to just wait for macos to gain a newer clang =D

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-03-24 17:15:15 +01:00
Martin Kletzander
1f76b5365e ci: Disable optimization on macos-12
This enables our CI to run tests on on macos-12 after commit
7944700b40.

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
2023-03-24 13:15:32 +01:00
Martin Kletzander
1b0c07db75 ci: Refresh CI files with current lcitool
This updates to FreeBSD 12.4 which has clang that supports
-fsemantic-interposition, plus of course updates the system.

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-03-24 12:59:51 +01:00
Peter Krempa
7334972c9a ci: Drop EOL OpenSUSE 15.3
The '15.3' version is EOL now:

  https://get.opensuse.org/leap/15.3/

Also switch the 'codestyle' job to the appropriate container image.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2023-03-23 16:24:56 +01:00
Peter Krempa
0cbd00525d ci: Add OpenSUSE leap 15.4 job
As a precursor to dropping the EOL OpenSUSE 15.3 job add first the
definitions for the replacement version.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-03-23 16:13:01 +01:00
Peter Krempa
20e063c2fd ci: Regenerate gitlab CI config with latest lcitool
The latest 'lcitool' now generates the CI config in a way which
allows users to kick off pipelines with the upstream projects container
environment rather than building a throwaway updated environment each
time and enables a gitlab feature to time individual script lines.
Pull it into libvirt.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2023-03-01 14:42:19 +01:00
Peter Krempa
5d59142122 ci: Regenerate gitlab config with latest lcitool
The 'cirrus-run' and 'check-dco' containers are now exported as
':latest' instead of ':master'.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-03-01 11:57:37 +01:00
Erik Skultety
917749314c ci: Test with latest Avocado again
Test with the following fix:
https://github.com/avocado-framework/avocado/pull/5567/commits

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-02-08 16:36:10 +01:00
Erik Skultety
ad00beffa6 ci: integration: Set an expiration on logs artifacts
The default expiry time is 30 days. Since the RPM artifacts coming from
the previous pipeline stages are set to expire in 1 day we can set the
failed integration job log artifacts to the same value. The sentiment
here is that if an integration job legitimately failed (i.e. not with
an infrastructure failure) unless it was fixed in the meantime it will
fail the next day with the scheduled pipeline again, meaning, that even
if the older log artifacts are removed, they'll be immediately
replaced with fresh ones.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-01-12 09:57:52 +01:00
Erik Skultety
3ebfeaa206 ci: Refresh and add Fedora 37 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2023-01-09 14:53:34 +01:00
Erik Skultety
c9a65eb8a6 ci: integration: Lock the Avocado version to 98.0 for now
Avocado 99.0 causes the TCK test suite to fail with the nwfilter tests
(which is another Bash framework underneath). Until the culprit is
identified and fixed in Avocado, let's lock the version to 98.0 which
worked with the test suite just fine.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-11-16 10:14:20 +01:00
Daniel P. Berrangé
539e851c5d ci: regenerate with lcitool manifest
Two notable changes:

 * the macOS platform has switched from x86_64 to aarch64
 * if a new pipeline starts before a previous one finishes,
   jobs marked 'interruptible: true' will be auto-cancelled

Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-11-14 17:22:31 +00:00
Erik Skultety
5597ae9609 ci: integration: Bundle only failed Avocado test results
Latest versions of Avocado create 'by-status' symlink shortcuts to test
results, IOW:

 # this is the main test results directory containing all data
 $ ls <path>/avocado/job-results/latest/test-results/
 01-scripts_networks_050-transient-lifecycle.t
 02-scripts_networks_051-transient-autostart.t
 ...
 22-scripts_networks_400-guest-bandwidth.t
 by_status/

 # list only the failed tests
 $ ls -l <path>/avocado/job-results/latest/test-results/by-status/FAIL
 19-scripts_networks_360-guest-network-vepa.t ->
    <path>/avocado/job-results/latest/test-results/19-scripts_networks_360-guest-network-vepa.t

Therefore, let's bundle only the failed ones, it's going to make the
log artifacts more obvious when looking for libvirt errors.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
2b26127397 ci: integration: Create the avocado logs artifact dir conditionally
Don't create an avocado directory in the resulting log artifacts
if Avocado didn't even run (e.g. libvirt errored out on service
restart).

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
5243dcea65 ci: integration: Use 'set +e' in the log collecting job
All 'script' blocks are defined as 'set -e' and so a single failed
return value means we won't collect some of the logs. Because of
the nature of the original job's failure some of the log sources
might not be available, but that's fine, however, the gitlab
after_script job cannot finish prematurely.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
124eb56342 ci: integration: Only create a coredump log when a core exists
It could be quite confusing looking at the job log artifacts and having
an empty coredump log in there, IOW it doesn't really give much
confidence that the reporting mechanism actually works.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:44 +02:00
Erik Skultety
1df36ecbd0 ci: integration: Fix the test on avocado directory existence
It's a directory, so -d should be used with 'test'.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:43 +02:00
Erik Skultety
ddc4898080 ci: integration: Add an extra level of quoting for augeas set options
Both log filters and log outputs expect string values, however, augeas
apparently requires an extra level of quotes apart from the ones we
pass via shell (see comment [1]) to work properly, otherwise augeas
ignores the value and returns 0.
Without this fix we don't set libvirt's log level to debug, we don't
set logging to a file and hence we don't include the logs in CI
artifacts in case the test suite fails.

[1] https://github.com/hercules-team/augeas/issues/301#issuecomment-143699880

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:43 +02:00
Erik Skultety
045e642d29 ci: integration: Add virtnetworkd to the list of restarted services
It was missing from the set. While at it, order the daemon set
alphabetically.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-25 09:40:43 +02:00
Peter Krempa
bb0bec4827 ci: Install python RPMs from libvirt/libvirt-python CI artifacts
After addition of the new libvirt-client-qemu sub-package which is using
python bindings (thus creating a circular dependency between the libvirt
and libvirt-python projects) the integration jobs fail with:

  Error:
   Problem: conflicting requests
    - nothing provides python3-libvirt >= 8.9.0-1.el9 needed by libvirt-client-qemu-8.9.0-1.el9.x86_64

The libvirt-python project now provides the RPMs in artifacts:
 https://gitlab.com/libvirt/libvirt-python/-/merge_requests/96

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-10-13 12:23:01 +02:00
Daniel P. Berrangé
491d918502 ci: refresh with latest lcitool manifest
This refresh switches the CI for contributors to be triggered by merge
requests. Pushing to a branch in a fork will no longer run CI pipelines,
in order to avoid consuming CI minutes. To regain the original behaviour
contributors can opt-in to a pipeline on push

   git push <remote> -o ci.variable=RUN_PIPELINE=1

This variable can also be set globally on the repository, through the
web UI options Settings -> CI/CD -> Variables, though this is not
recommended. Upstream repo pushes to branches will run CI.

The use of containers has changed in this update, with only the upstream
repo creating containers, in order to avoid consuming contributors'
limited storage quotas. A fork with existing container images may delete
them. Containers will be rebuilt upstream when pushing commits with CI
changes to the default branch. Any other scenario with CI changes will
simply install build pre-requisite packages in a throaway environment,
using the ci/buildenv/ scripts. These scripts may also be used on a
contributor's local machines.

With pipelines triggered by merge requests, it is also now possible to
workaround the inability of contributors to run pipelines if they have
run out of CI quota. A project member can trigger a pipeline from the
merge request, which will run in context of upstream, however, note
this should only be done after reviewing the code for any malicious
CI changes.

Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-10-06 05:15:54 -04:00
Erik Skultety
0327bbb767 ci: integration: Update the perl CI job names
libvirt-derived repos recently changed the way how and when CI
containers are built and for that a different naming scheme was adopted
to differentiate between the 2. Update the integration pipeline config
to reflect this change.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-10-06 10:47:56 +02:00
Pavel Hrdina
84d9279aef ci: refresh generated files
Signed-off-by: Pavel Hrdina <phrdina@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2022-10-03 12:05:20 +02:00
Daniel P. Berrangé
c83c6e4e7a ci: refresh with latest lcitool manifest
This updates the FreeBSD 13 image to 13.1 which should fix the
symbol lookup errors seen in CI recently.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-09-14 10:54:15 -04:00
Peter Krempa
5598ce16d5 ci: Regenerate dockerfiles after 'sheepdog' removal
After support for the sheepdog storage driver backend was removed we
don't need to install it any longer in the containers.

Regenerate the dockerfiles after:
https://gitlab.com/libvirt/libvirt-ci/-/merge_requests/314

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Jiri Denemark <jdenemar@redhat.com>
2022-09-01 16:29:15 +02:00
Andrea Bolognani
915d566f42 ci: Switch from macOS 11 to macOS 12
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-08-24 11:07:19 +02:00