One advantage that GitLab's YAML has with Shell commands is that every
single line is printed out as is, including control structures. In
order to see whether the logic did the same thing and the tests are
going to operate on the right set of daemons (monolithic vs modular),
lets print the DAEMONS variable that we set depending on the distro we
execute the tests on.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Unfortunately, once we go down the line of running our own scripts as
part of GitLab CI jobs rather than open coding Shell in YAML, we lose
the benefit of seeing each line the script executes. The downside of
the default YAML however is that we have to maintain the same piece of
code on 2 places in that case. Let's adopt what we use with other
container jobs and prefix each shell command with 'run_cmd' or
'run_cmd_quiet' which will dump it in the logs before executing.
Flow control expressions and structures are a problem though in this
regard, so let's just print some important values for debugging
purposes.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We've started using the run_cmd helper function to log what kind of
command is being executed as well as actually executing the command.
The problem however is doing I/O redirections for commands which we
don't wish to see any output for whatever reason. Now, if the
redirection is applied at parameter passing to run_cmd it's going to be
applied to the debug print as well. Let's introduce another helper,
run_cmd_quiet which takes care of the I/O redirection and executes the
command completely silently.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Neither '&>' nor 'source' are defined in POSIX.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Because of the nature of writing inline shell commands to YAML, most of
the commentaries where inlined with the command not to hinder YAML
readability any further. Since we moved the logic to a standalone
script, we can now do whatever formatting & readability adjustments we
want.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We've not been interested in any extra output from the command at all
since we always redirected both stdout and stderr to /dev/null. Future
patch will change that slightly, so --quiet will start making sense.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We needed v98.0 in commit c9a65eb8 due to a bug in Avocado in the past
and have been installing the latest Avocado for a while since commit
91774931, yet we kept the comment by a mistake.
Besides, looks like v98.0 ignores the avocado.config file in the TCK
repo instructing it to run the test suite sequentially leading to test
stability issues, so abandoning the v98.0 in commit 91774931 was a good
thing in the end.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Since the section now only consists of a single command, we can happily
move the command to the main integration template job body.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
All supported versions of Fedora and CentOS Stream 9 default to modular
setup, it's probably better if we cosmetically adjust the CentOS Stream
version check to make it explicit that monolithic daemon services ought
to be started only on Stream 8.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Follow what's been done to other jobs in .gitlab-ci.yml and extract the
shell logic from YAML to a function in ci/jobs.sh
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Commit f688a53a converted .gitlab-ci.yml to the usage of ci/jobs.sh
functions, but in doing that our test options
'--no-suite syntax-check --print-errorlogs'
got lost in the process and since commit 8e660c52 didn't introduce them
in the first place, it caused a behavioral regression. This patch adds
them back.
Fixes: 8e660c5286
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
All the functionality this script provided has been incorporated either
in the Python ci/helper tool or lcitool directly.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We've successfully migrated over to lcitool to take care of the
container workload execution, so dropping this 'make' prep code is a
prerequisite of finally getting rid of the ci/Makefile script.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
These originally allowed customizing the ci/Makefile script which was
the core of the local container executions. The problem was that
however flexible this may have been, it never mirrored what was being
done as part of the GitLab jobs. Motivated by the effort of mirroring
GitLab jobs locally, these would only ever make sense to be set/used in
interactive shell container sessions where the developer is perfectly
capable of using the right meson/ninja CLI options directly without
going through another shell variable indirection as it was the case
with these ci/helper options.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Previous patches added a single 'run' command parametrized with GitLab
job specs via '--job' that cover all of these original actions, adding
some more in the process. Drop the original actions as we don't need
them anymore.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
The idea behind this subcommand is to follow whatever build job we have
defined in the GitLab CI pipeline, so that we only have a single source
of truth for the recipes. Adds 'shell' as an extra option for
interactive container build debugging.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Just like with the other CLI sub-commands, add an action to run a
GitLab spec job in a local container via lcitool.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This method wasn't even utilized before this patch. This patch adds all
the necessary logic to successfully execute a container workload via
lcitool (which will later allow us to ditch ci/Makefile). Because
container executions via lcitool creates the following inside the
container:
$ ls
script datadir
where 'datadir' is the workload directory (in this case a local git
repo clone) and 'script' is the code that runs whatever the workload is
over 'datadir'.
In order to satisfy the ^above, our helper generates a trivial
temporary 'script' that will source ci/build.sh and run whatever was
specified as --job essentially to simulate the exact steps a GitLab
pipeline job would go through.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
A proper Python equivalent of 'git clone --local'.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper will be utilized by a future patch which will add the
lcitool container execution logic. The reason why the required_deps
decorator isn't being used here is because this is a property.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Since we'll depend on GitPython for repo cloning, we need to make sure
to emit a user friendly error if the module is not installed. This
patch introduces a helper which future patches will use as a decorator.
Inspiration for this helper has been taken out of lcitool where we use
an identical helper for this purpose.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We'll soon be relying solely on lcitool so we need to be able to run it
from a user-provided location if it's not installed in a known
location.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
':' is just a connecting character, we can add it to the appropriate
place later in the Python script later, but it doesn't make sense to be
part of the image 'tag' string.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
After the recent changes, this script no longer executes any logic
anymore, it merely defines the jobs run in the GitLab environment. In
order to use it, one has to source the file in the environment and then
run one of the job "functions". For that, the 'build.sh' name is no
longer descriptive enough and 'jobs.sh' feels more suitable and less
misleading.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We've moved all invocations to the respective helper function which
we'll execute both from gitlab CI jobs and local environments so we
don't need to have them on the global level as it would also not work
with "sourcing" this file to populate the environment with function
definitions.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Firstly, this would mangle with "sourcing" this file in either
execution environment later down the road. Secondly, we won't need this
as future ci/helper patches will generate a throwaway script that will
take care of a correct execution of a build job in a similar fashion as
if the job ran in a GitLab environment.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper is a shell function transcript of its original GitLab CI
counterpart.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper is a shell function transcript of its original GitLab CI
counterpart.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper is a shell function transcript of its original GitLab CI
counterpart. There's one notable difference such that we pass '-j1' to
the meson compile command otherwise we'd have to execute the 'run_build'
function twice, passing 'libvirt-pot-dep' and 'libvirt-pot' targets
in a serial manner.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper is a shell function transcript of its original GitLab CI
counterpart.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper is a shell function transcript of its original GitLab CI
counterpart.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper function does not correspond to a particular GitLab job, it
just logically separates the necessary step of creating a dist tarball
from the RPM build job that takes over.
One notable change here is the need to update git's file index which
causes issues in local container executions which rely on a shallow
copy of the libvirt repo created as:
$ git clone --local
Even if all changes have been committed, git often complained
otherwise. Updating the index in a GitLab environment is a NOP.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This helper is a shell function transcript of its original GitLab CI
counterpart.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
The reason for this wrapper is that all job functions introduced in
future patches will refer to this one instead of open-coding the same
'meson setup' invocation N times. It also prevents 'setup' to be called
multiple times as some future job functions might actually do just that
in a transitive manner.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This would normally be not needed at all, but the problem here is the
Shell-in-YAML which GitLab interprets. It outputs every command that
appears as a line in the 'script' segment in a color-coded fashion for
easy identification of problems. Well, that useful feature is lost when
there's indirection and one script calls into another in which case it
would only output the respective script name which would make failure
investigation harder. This simple helper tackles that by echoing the
command to be run by any script/function with a color escape sequence
so that we don't lose track of the *actual* shell commands being run as
part of the GitLab job pipelines. An example of what the output then
might look like:
[RUN COMMAND]: 'meson compile -C build install-web'
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Previous patches have removed the code that allowed injecting arbitrary
meson arguments, same for ninja args.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We'll use this one in many of the job functions future patches will
introduce, it's a neat shortcut to avoid using relative paths.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
These are common variables we wish to use in containerized environments
both in GitLab and locally. Having these defined in a single place
rather than twice is highly preferable.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
This change was supposed to be part of commit 120a674f , but was
proposed against the libvirt TCK project instead. Since we're running
the TCK test suite as part of this project, this is the right place for
the TCK runtime deps list config.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Multiple values passed to --meson-args need to be quoted so that
the shell will interpret them correctly. The option's name was
also reported incorrectly, so fix that as well.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Commit 9c9848f955 merged $MESON_OPTS into $MESON_ARGS, and
while doing so changed their behavior: while until then the
contents of $MESON_ARGS had precedence over those of $MESON_OPTS,
now the opposite is true. Restore the original behavior and
document it.
The argument for merging the two variables in the first place
was that having both present on the meson command line could be
confusing; however, that should no longer be the case now that
we have reasonably extensive comments explaining the role of
each of the variables and how they interact with each other, so
return the meson command line to its original form.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Each respective project that lcitool knows about and currently
maintains its list of package dependencies knows best what packages
they actually depend on. If a new dependency is currently needed, first
a change in lcitool is necessary before GitLab jobs and containers can
be updated. Provided a mapping already exists in lcitool (which can
quickly be added as an override via mappings.yml temporarily) we speed
up the whole CI update process by one step.
This patch adds all libvirt deps lists lcitool currently maintains for
libvirt.
Note that as with any overrides (since commit f199dd50) lcitool must be
invoked as '$ lcitool -d/--data-dir ci/lcitool ...'
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Now that we have a local OS target override for lcitool in place, we
can bump the cirrus FreeBSD image version in GitLab CI.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
We've reached a point in lcitool where we can't steer its development
based solely on libvirt's needs IOW there will be times where a local
override of value (e.g. package mapping) will be necessary - an example
of this would be QEMU.
In case of this particular patch we need to add an override for the
cirrus FreeBSD 13 image we request in our CI to fix:
/usr/local/lib/libtasn1.so.6: Undefined symbol "strverscmp@FBSD_1.7"
The reason why we can't/should not make the fix in upstream lcitool
just yet is that we store a libosinfo ID in lcitool's OS target YAML
configs and at the time of writing this patch libosinfo does not have
a corresponding entry/ID for FreeBSD 13.2 so we have to stick with 13.1
in lcitool until they do so.
For the time being, the fix can easily be done on libvirt side as does
this patch.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Let's move our Debian CI workloads to Debian-12 since it's the latest
release and mark Debian-11 jobs as optional.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
It is quite confusing seeing these two in a call like this one:
$ meson build $MESON_OPTS $MESON_ARGS
One has to ask 'how are they different' and 'shouldn't these be
merged'. In fact, these variables hold very different things and we
should make it more obvious. The problem is that renaming MESON_OPTS to
something more meaningful, like 'MESON_CROSS_OPTS' which is what
MESON_OPTS really does would require changes to lcitool and would
impact Dockerfile generation which in turn might have an impact on
other projects which rely on this lcitool functionality which is risky.
Instead, provide a docstring for the former to supplement the latter
and join the two variables in a single one MESON_ARGS which is then
passed to meson's command line so it's a little less confusing.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Although it is currently consistent with the other variables we define
when running ci in a local container environment, it isn't consistent
with the variable naming we use in GitLab recipes. Since the idea is
to unite the two, we're likely going to drop a few other variables from
the local env configuration anyway, hence this renaming.
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
