Commit Graph

227 Commits

Author SHA1 Message Date
Erik Skultety
a35b4e4e97 ci: helper: Add --lcitool-path CLI option
We'll soon be relying solely on lcitool so we need to be able to run it
from a user-provided location if it's not installed in a known
location.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
f6f930b435 ci: helper: Don't make ':' literal a static part of the image tag
':' is just a connecting character, we can add it to the appropriate
place later in the Python script later, but it doesn't make sense to be
part of the image 'tag' string.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
5f78628fb6 ci: helper: Drop _lcitool_get_targets method
This method unused anywhere, so drop it.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
ed7aa04de4 ci: Rename build.sh -> jobs.sh
After the recent changes, this script no longer executes any logic
anymore, it merely defines the jobs run in the GitLab environment. In
order to use it, one has to source the file in the environment and then
run one of the job "functions". For that, the 'build.sh' name is no
longer descriptive enough and 'jobs.sh' feels more suitable and less
misleading.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
5a2f2eb221 ci: build.sh: Drop MESON_ARGS definition from global level
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
1460572b66 ci: build.sh: Drop direct invocation of meson/ninja commands
We've moved all invocations to the respective helper function which
we'll execute both from gitlab CI jobs and local environments so we
don't need to have them on the global level as it would also not work
with "sourcing" this file to populate the environment with function
definitions.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
56b37fe381 ci: build.sh: Drop changing working directory to CI_CONT_DIR
Firstly, this would mangle with "sourcing" this file in either
execution environment later down the road. Secondly, we won't need this
as future ci/helper patches will generate a throwaway script that will
take care of a correct execution of a build job in a similar fashion as
if the job ran in a GitLab environment.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
fe9f788812 ci: build.sh: Add a wrapper function over the 'website' job
This helper is a shell function transcript of its original GitLab CI
counterpart.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
da0a354476 ci: build.sh: Add a wrapper function over the 'rpmbuild' job
This helper is a shell function transcript of its original GitLab CI
counterpart.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
07a8556076 ci: build.sh: Add a wrapper function over the 'potfile' job
This helper is a shell function transcript of its original GitLab CI
counterpart. There's one notable difference such that we pass '-j1' to
the meson compile command otherwise we'd have to execute the 'run_build'
function twice, passing 'libvirt-pot-dep' and 'libvirt-pot' targets
in a serial manner.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
1b5f2d6b09 ci: build.sh: Add a wrapper function over the 'codestyle' job
This helper is a shell function transcript of its original GitLab CI
counterpart.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
8e660c5286 ci: build.sh: Add a wrapper function over the 'test' job
This helper is a shell function transcript of its original GitLab CI
counterpart.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
387553db48 ci: build.sh: Add a helper function to create the dist tarball
This helper function does not correspond to a particular GitLab job, it
just logically separates the necessary step of creating a dist tarball
from the RPM build job that takes over.
One notable change here is the need to update git's file index which
causes issues in local container executions which rely on a shallow
copy of the libvirt repo created as:

    $ git clone --local

Even if all changes have been committed, git often complained
otherwise. Updating the index in a GitLab environment is a NOP.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
5452ca9b1c ci: build.sh: Add a wrapper function over the 'build' job
This helper is a shell function transcript of its original GitLab CI
counterpart.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
5ee2851481 ci: build.sh: Add a wrapper function over meson's setup
The reason for this wrapper is that all job functions introduced in
future patches will refer to this one instead of open-coding the same
'meson setup' invocation N times. It also prevents 'setup' to be called
multiple times as some future job functions might actually do just that
in a transitive manner.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:36:03 +02:00
Erik Skultety
6214ae55f6 ci: build.sh: Add a wrapper function executing 'shell' commands
This would normally be not needed at all, but the problem here is the
Shell-in-YAML which GitLab interprets. It outputs every command that
appears as a line in the 'script' segment in a color-coded fashion for
easy identification of problems. Well, that useful feature is lost when
there's indirection and one script calls into another in which case it
would only output the respective script name which would make failure
investigation harder. This simple helper tackles that by echoing the
command to be run by any script/function with a color escape sequence
so that we don't lose track of the *actual* shell commands being run as
part of the GitLab job pipelines. An example of what the output then
might look like:
    [RUN COMMAND]: 'meson compile -C build install-web'

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 11:35:58 +02:00
Erik Skultety
d3c6fb6a28 ci: build.sh: Don't mention that MESON_ARGS are available via CLI
Previous patches have removed the code that allowed injecting arbitrary
meson arguments, same for ninja args.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 10:36:13 +02:00
Erik Skultety
e61241d7f4 ci: build.sh: Add GIT_ROOT env helper variable
We'll use this one in many of the job functions future patches will
introduce, it's a neat shortcut to avoid using relative paths.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 10:36:13 +02:00
Erik Skultety
470596b850 ci: build.sh: Add variables from .gitlab-ci.yml
These are common variables we wish to use in containerized environments
both in GitLab and locally. Having these defined in a single place
rather than twice is highly preferable.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-12 10:36:04 +02:00
Erik Skultety
e2332fe72a ci: lcitool: Add libvirt-tck+runtime deps list
This change was supposed to be part of commit 120a674f , but was
proposed against the libvirt TCK project instead. Since we're running
the TCK test suite as part of this project, this is the right place for
the TCK runtime deps list config.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-09-01 13:30:57 +02:00
Andrea Bolognani
d40c6cad64 ci: Fix quoting and option name
Multiple values passed to --meson-args need to be quoted so that
the shell will interpret them correctly. The option's name was
also reported incorrectly, so fix that as well.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2023-08-28 14:21:54 +02:00
Andrea Bolognani
b68d253c46 ci: Fix precedence between arguments passed to meson
Commit 9c9848f955 merged $MESON_OPTS into $MESON_ARGS, and
while doing so changed their behavior: while until then the
contents of $MESON_ARGS had precedence over those of $MESON_OPTS,
now the opposite is true. Restore the original behavior and
document it.

The argument for merging the two variables in the first place
was that having both present on the meson command line could be
confusing; however, that should no longer be the case now that
we have reasonably extensive comments explaining the role of
each of the variables and how they interact with each other, so
return the meson command line to its original form.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2023-08-28 14:21:44 +02:00
Erik Skultety
120a674f25 ci: lcitool: Maintain project package deps lists here
Each respective project that lcitool knows about and currently
maintains its list of package dependencies knows best what packages
they actually depend on. If a new dependency is currently needed, first
a change in lcitool is necessary before GitLab jobs and containers can
be updated. Provided a mapping already exists in lcitool (which can
quickly be added as an override via mappings.yml temporarily) we speed
up the whole CI update process by one step.
This patch adds all libvirt deps lists lcitool currently maintains for
libvirt.

Note that as with any overrides (since commit f199dd50) lcitool must be
invoked as '$ lcitool -d/--data-dir ci/lcitool ...'

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-22 12:10:01 +02:00
Erik Skultety
bcc04f39ff ci: Udate FreeBSD-13 image with lcitool manifest
Now that we have a local OS target override for lcitool in place, we
can bump the cirrus FreeBSD image version in GitLab CI.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:06:32 +02:00
Erik Skultety
f199dd50df ci: Introduce a new 'lcitool' data directory
We've reached a point in lcitool where we can't steer its development
based solely on libvirt's needs IOW there will be times where a local
override of value (e.g. package mapping) will be necessary - an example
of this would be QEMU.
In case of this particular patch we need to add an override for the
cirrus FreeBSD 13 image we request in our CI to fix:

    /usr/local/lib/libtasn1.so.6: Undefined symbol "strverscmp@FBSD_1.7"

The reason why we can't/should not make the fix in upstream lcitool
just yet is that we store a libosinfo ID in lcitool's OS target YAML
configs and at the time of writing this patch libosinfo does not have
a corresponding entry/ID for FreeBSD 13.2 so we have to stick with 13.1
in lcitool until they do so.

For the time being, the fix can easily be done on libvirt side as does
this patch.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:06:32 +02:00
Erik Skultety
29ede88aa1 ci: Move Debian-11 workloads to Debian-12
Let's move our Debian CI workloads to Debian-12 since it's the latest
release and mark Debian-11 jobs as optional.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:05:18 +02:00
Erik Skultety
61360d91a7 ci: Add Debian-12 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-08-16 14:05:18 +02:00
Erik Skultety
9c9848f955 ci: build.sh: Join MESON_ARGS and MESON_OPTS
It is quite confusing seeing these two in a call like this one:
    $ meson build $MESON_OPTS $MESON_ARGS

One has to ask 'how are they different' and 'shouldn't these be
merged'. In fact, these variables hold very different things and we
should make it more obvious. The problem is that renaming MESON_OPTS to
something more meaningful, like 'MESON_CROSS_OPTS' which is what
MESON_OPTS really does would require changes to lcitool and would
impact Dockerfile generation which in turn might have an impact on
other projects which rely on this lcitool functionality which is risky.

Instead, provide a docstring for the former to supplement the latter
and join the two variables in a single one MESON_ARGS which is then
passed to meson's command line so it's a little less confusing.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:11:50 +02:00
Erik Skultety
6788b2d3cd ci: build.sh: Drop the CI prefix from the CI_{MESON,NINJA}_ARGS vars
Although it is currently consistent with the other variables we define
when running ci in a local container environment, it isn't consistent
with the variable naming we use in GitLab recipes. Since the idea is
to unite the two, we're likely going to drop a few other variables from
the local env configuration anyway, hence this renaming.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
ed1a0a63ce ci: build.sh: Always assume -Dsystem=true
There's no harm in always building in system mode, i.e. setting the
right paths.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
b845c2fd3b ci: build.sh: Use 'meson setup' explicitly
Even though 'setup' is assumed when no other command is given, we're
being explicit in our GitLab recipes, so do the same for the local
build.sh script too.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
d95b6e993f ci: build.sh: Drop the commentary about CI_BUILD_SCRIPT
build.sh is not the place where this should be mentioned as the
official entrypoint for this script locally is ci/helper which can
download the right image from our upstream CI registry. Since the idea
is to ultimately drop the usage of a Makefile for the local executions,
this patch doesn't provide an alternative place for the comment in
question as the functionality is going to be altered substantially in
the future.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-08-11 15:09:04 +02:00
Erik Skultety
6d46808a0e ci: integration: Drop the Fedora 35 check
We're already past Fedora 35 and so all new fedora's default to
modular daemon setup.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-08-11 14:55:56 +02:00
Michal Privoznik
51ae80d6a7 ci: Switch to OpenSUSE Leap 15.5
OpenSUSE Leap was released recently (2023-06-07). Refresh our CI
with latest lcitool which brings this minor update.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-07-25 11:47:56 +02:00
Han Han
28141adfdc scripts: Fix the flake8 syntax-check failures
Fix the syntax-check failures (which can be seen after
python3-flake8-import-order package is installed) with the help
of isort[1]:

289/316 libvirt:syntax-check / flake8   FAIL   5.24s   exit status 2

[1]: https://pycqa.github.io/isort/

Signed-off-by: Han Han <hhan@redhat.com>
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-06-29 11:51:27 +02:00
Daniel P. Berrangé
25420111ba ci: fix Cirrus CI jobs run from merge requests
Preferentially fetch from $CI_MERGE_REQUEST_REF_PATH if it is
defined, otherwise use $CI_COMMIT_REF_NAME

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2023-06-05 11:39:12 +01:00
Erik Skultety
adf67ca987 ci: refresh with latest lcitool manifest
Main lcitool changes:
- added Alpine 3.17 and 3.18 targets
- dropped Alpine 3.15 and 3.16

Note that we're not actively testing all Alpine targets due to CI
quota, so only 3.17 is used as a replacement for 3.15 in this patch.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-06-05 10:51:18 +02:00
Erik Skultety
dcb9ee99f3 ci: integration: Flip QEMU upstream integration tests to Fedora 38
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Erik Skultety
2bb0efa367 ci: Flip mingw jobs from Fedora 37 to Fedora 38
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Erik Skultety
ffd2a97070 ci: Drop Fedora 36 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Erik Skultety
96ffb7318e ci: Add Fedora 38 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2023-05-10 16:41:21 +02:00
Michal Privoznik
a3cc0e9ceb ci: Regenerate files
This removes minor version number from OpenSUSE LEAP target names
and on CentOS Stream 9 installs flake8 from repositories, instead
of pip.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2023-04-27 15:02:42 +02:00
Andrea Bolognani
5be133b39e ci: Regenerate files
This unbreaks the various $CROSS-$NAME-local-env jobs.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-04-04 10:51:53 +02:00
Martin Kletzander
e9adb4dcb1 Revert "ci: Disable optimization on macos-12"
This reverts commit 1f76b5365e.

There were two issues with this commit.  First is the missing propagation
of CFLAGS into the build environment and second is the fact that this is
not enough to disable the check for -fsemantic-interposition.  The
proper fix would require setting MESON_OPTS or similar and also add the
propagation of such variable into the cirrus builds etc., but at this
point I burned so much time on this trivial piece of rubbish that I
think it's easier to just wait for macos to gain a newer clang =D

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-03-24 17:15:15 +01:00
Martin Kletzander
1f76b5365e ci: Disable optimization on macos-12
This enables our CI to run tests on on macos-12 after commit
7944700b40.

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
2023-03-24 13:15:32 +01:00
Martin Kletzander
1b0c07db75 ci: Refresh CI files with current lcitool
This updates to FreeBSD 12.4 which has clang that supports
-fsemantic-interposition, plus of course updates the system.

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-03-24 12:59:51 +01:00
Peter Krempa
7334972c9a ci: Drop EOL OpenSUSE 15.3
The '15.3' version is EOL now:

  https://get.opensuse.org/leap/15.3/

Also switch the 'codestyle' job to the appropriate container image.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2023-03-23 16:24:56 +01:00
Peter Krempa
0cbd00525d ci: Add OpenSUSE leap 15.4 job
As a precursor to dropping the EOL OpenSUSE 15.3 job add first the
definitions for the replacement version.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2023-03-23 16:13:01 +01:00
Peter Krempa
20e063c2fd ci: Regenerate gitlab CI config with latest lcitool
The latest 'lcitool' now generates the CI config in a way which
allows users to kick off pipelines with the upstream projects container
environment rather than building a throwaway updated environment each
time and enables a gitlab feature to time individual script lines.
Pull it into libvirt.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2023-03-01 14:42:19 +01:00
Peter Krempa
5d59142122 ci: Regenerate gitlab config with latest lcitool
The 'cirrus-run' and 'check-dco' containers are now exported as
':latest' instead of ':master'.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2023-03-01 11:57:37 +01:00