Commit Graph

153 Commits

Author SHA1 Message Date
Andrea Bolognani
ae2dabe5ed ci: Move upstream QEMU integration test to Fedora 36
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-06-09 14:41:44 +02:00
Andrea Bolognani
4a4b7f2854 ci: Add Fedora 36 to integration tests
This requires publishing the RPMs as artifacts from the regular
build job.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-06-09 14:41:40 +02:00
Erik Skultety
d1e3247675 ci: integration: Set 'safe.directory' when installing QEMU from git
Since a fix for CVE-2022-24765 was released every git command is now
checked against the context repo in which it's supposed to run
resulting in a fatal error if the repo is owned by other user than the
one running the git command.
This means that in order to be able to do 'sudo make install', we have
to set the 'safe.directory' for the root user. This is because QEMU
runs 'git submodule update' automatically on 'make install'.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-06-09 13:38:17 +02:00
Erik Skultety
3b943f0c43 ci: integration: SELinux relabel the QEMU we installed from git
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-06-09 13:38:13 +02:00
Daniel P. Berrangé
8b8fd1bc67 ci: eliminate many cross arch CI builds
We currently build cross-arch containers for all three Debian
releases (10, 11, Sid), and do libvirt builds covering each
arch. This is overkill in terms of the number of problems it
identifies. The most important aspect of cross arch builds is
to find problems with 32-bit builds and problems with big
endian builds.

With this in mind the cross arch jobs are altered as follows

 - Debian 10
     - build the containers by default
     - build armv7 (32-bit) & s390x (big endian)
     - other arch builds manual

 - Debian 11 / Sid
     - container builds all optional
     - arch builds all optional

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-31 17:23:04 +02:00
Daniel P. Berrangé
bb9abaed26 ci: move Ubuntu GCC santizers build to 20.04
We currently build on Ubuntu 22.04 twice, for GCC and CLang
with santizers turned on. Moving the GCC santizers build
to 20.04 lets us cull one of the 22.04 jobs

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-31 17:23:04 +02:00
Daniel P. Berrangé
b7565a2145 ci: disable native builds on certain distros
Currently we do native builds on all distros that are covered
by the support matrix. This reduces that such that we mostly
only run builds on the newest (ie bleeding edge non-released)
version and the oldest version. The effect is that cut out
builds on the newest release version. This is acceptable,
because that version is sandwiched between two versions we
do still test, so unlikely to have failures not already
identified by other jobs.

This has the effect of disabling:

  - AlmaLinux 8 GCC - still has a CLang build
    and CentOS 8 Stream also gives coverage

  - Debian 11 - still has a Debian 10 and Sid
    build

  - Alpine 3.15 - still has a Alpine 3.14 and Edge
    build

Ideally Fedora 35 would be disabled too, but we rely on that
for the integration tests.

The Ubuntu jobs will be handled in the next patch.

The containers are still built since this is cheap-ish.

The build jobs can also be triggered manually if desired.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-31 17:23:04 +02:00
Daniel P. Berrangé
1c5bc460d3 ci: refresh with lcitool manifest
This refreshes the containers bringing in new behaviour when
builds/containers are disabled.

Instead of deleting the job entirely, the job still exists
but is set to be a manual job. It won't affect the pipeline
result, but can be triggered by the developer if they wish
to test a specific scenario.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-31 17:23:04 +02:00
Martin Kletzander
76802e5dc6 ci: Refresh generated files
Notable changes:

* 'lcitool manifest' now generates absolute include paths

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
2022-05-27 13:35:33 +02:00
Andrea Bolognani
be1d39f6bd ci: Don't mark any Fedora 36 job as optional
It's a stable distro, so we expect all jobs to succeed.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-26 16:41:23 +02:00
Andrea Bolognani
1234ea1d38 ci: Move MinGW jobs to Fedora 36
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-26 16:41:20 +02:00
Andrea Bolognani
a6f7ed6e72 ci: Add Fedora 36
The target is intentionally not added to the integration tests
at this time, because the corresponding VM template is not yet
available on the runner. A later patch will take care of that.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-26 16:41:19 +02:00
Andrea Bolognani
c000499c4f ci: Refresh generated files
Notable changes:

  * 'lcitool manifest' now generates GitLab CI rules spread
    across a bunch of files;

  * container images are built less frequently for the main
    repository.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-26 16:41:17 +02:00
Andrea Bolognani
af89e4b8f8 ci: Drop Fedora 34
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-05-26 16:41:14 +02:00
Andrea Bolognani
40d78e47e5 docs: Update URL for MinGW
The MinGW-w64 project has effectively replaced the original
MinGW project, and distributions such as Fedora have been shipping
packages based on the former for years now.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-24 12:19:15 +02:00
Andrea Bolognani
7f403bc251 syntax-check: Enforce sc_prohibit_backslash_alignment everywhere
Basically all files in the repository are already passing the
check, except for syntax-check.mk itself. Fix that, and stop
limiting the files on which the test is performed.

These changes have been generated by running

  $ sed -Ei 's/[ '$'\t'']+\\$/ \\/g' $(git grep -El '[ '$'\t'']+\\$')

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-24 11:09:51 +02:00
Erik Skultety
b7fa48081d ci: Add a Fedora integration test job utilizing upstream QEMU
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
80ffd571a8 ci: Add a new integration job template for the upstream QEMU scenario
YAML anchors don't work with Shell condition structures, so we cannot
simply reference the QEMU build template YAML anchor conditionally and
hence have everything as part of a single job template.
Instead, we have to "subclass" the .integration_tests template and
inject the QEMU building bits explicitly.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
de98075faf ci: Introduce a template for upstream QEMU build
This was heavily inspired by QEMU's upstream CI buildtest-template.yml.
Rather than referencing QEMU's template directly (which GitLab can do),
this patch resorts to hard-coding the build steps ourselves, solely
because there's no guarantee QEMU will keep either the template file
name or the template name from which the build steps were mostly copied
from.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
72887b8f59 ci: Break off the integration_tests template into more templates
There's quite a lot happening in the .integration_tests template
already even without adding upstream QEMU build into the mix.
Let's break the template into more pieces which can then reference
in the .integration_tests template when putting all the pieces back
together using YAML anchors.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
387932771a ci: Separate the integration job template to a separate file
Future patches will do more code extraction from the existing template
using YAML anchors so it'd be better that the templates would live
separately from job definitions.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Andrea Bolognani
0495f841ee ci: Move clang/asan/ubsan jobs to Ubuntu 22.04
Using an updated toolchain means we're likely to catch more
issues.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-06 15:25:18 +02:00
Andrea Bolognani
29ee8c1d2e ci: Add Ubuntu 22.04
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-06 11:13:12 +02:00
Andrea Bolognani
1e25a480f5 ci: Refresh configuration
Notable changes:

  * use the most recent point release for FreeBSD 12.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-06 11:13:05 +02:00
Daniel P. Berrangé
d4d5cebdf1 ci: print stack traces on macOS if any tests fail
Mostly the tests should fail cleanly, but sometimes the test might crash
or abort. In this case we'll need to know the stack traces in order to
debug the problem. Fortunately on the Cirrus CI macOS instances, the
crash reporting service is active and saving crashes to the directory
~/Library/Logs/DiagnosticReports/

Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-04-28 13:36:34 +01:00
Daniel P. Berrangé
a45e4b42fe ci: print test suite logs on failure for Cirrus jobs
We don't have access to the 'testlog.txt' file, so we need meson to
print the failures for any broken tests directly.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-04-26 13:17:52 +01:00
Andrea Bolognani
08e097f605 ci: Drop openSUSE Leap 15.2
It has reached end of life in January 2022.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:34 +02:00
Andrea Bolognani
e33366fc19 ci: Add openSUSE Leap 15.3
It was released in June 2021.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:31 +02:00
Andrea Bolognani
67e3589120 ci: Add Alpine Linux 3.15
It was released in November 2021.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:30 +02:00
Andrea Bolognani
de390af9d7 ci: Simplify Alpine Linux 3.14 definition
Using the extended definition format is only necessary when
the defaults are not good enough, but that's not the case here.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:28 +02:00
Andrea Bolognani
766a2d2e52 ci: Drop FreeBSD -CURRENT
All build jobs for the target are explicitly disabled, so
there's no point in keeping the variables file around and we
can simply not mention it in the manifest at all.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:11 +02:00
Erik Skultety
cc05c4e5e4 ci: Refresh Dockerfiles and vars files
In this batch:
- dnsmasq is dropped as build dependency
- Alpine Edge rpcgen package collision fix

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-04-07 16:19:42 +02:00
Peter Krempa
e3ab7900da ci: Drop Ubuntu 1804
As of April 23 2022, Ubuntu 20.04 will be out for two years, which per
our platform support policy means we no longer have to support
Ubuntu 18.04.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2022-04-04 12:26:58 +02:00
Tim Wiederhake
49156f8070 Fix spelling
Signed-off-by: Tim Wiederhake <twiederh@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-03-31 11:31:08 +02:00
Erik Skultety
69bcfa5e3b ci: integration: Rename all Avocado standard stream log files to *.log
By default, stdout/stderr Avocado test log files do not have any file
extension which confuses GitLab's web UI to mangle the MIME type for
these and so the browser will never offer the option to open such file
from in a text editor rather than dowloading it.
Since GitLab sets a proper MIME for .txt and .log file extensions,
rename all Avocado log files without an extension to *.log . This pairs
nicely with the coredumpctl info file which we already name as
'coredumpctl.txt' because of this.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-03-22 17:11:51 +01:00
Erik Skultety
908cb26b5d ci: integration: Collect stack traces with coredumpctl
Some Red Hat-like distros have cores limited with a soft limit of 0
which means that neither a stack trace nor a core file will be
available. Since we want the stack trace we need to set the core limit
with systemd globally to unlimited/infinity.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-22 12:07:37 +01:00
Erik Skultety
e23353756e ci: Define the integration job tag dynamically via a variable
Custom runners are private to a project, so naturally forks cannot run
any workloads on these. The integration test suite which requires
access to our custom runner is naturally disabled on forks and can be
enabled by setting LIBVIRT_CI_INTEGRATION=1.
The problem is that the current integration jobs definitions have tags
statically defined as 'redhat-vm-host'. If users are going to supply
their own private runners for their forks, they can define whatever
tags they want with it and so unless they add 'redhat-vm-host' to their
own runner's tags, the pipeline won't run.
To solve this, define the integration job tag using a variable. The
repo config will use the value defined in the job for the variable
while users can override the value easily on a project/pipeline level
thanks to GitLab's CI variable precedence [1].

[1] https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-22 12:07:37 +01:00
Daniel P. Berrangé
e4327557a4 ci: only run integration tests if $LIBVIRT_CI_INTEGRATION=1 is set
Right now the jobs have no rules so they will always be created in
a pipeline. If the user's fork has no runner configured, then the
jobs will never be able to execute and the pipeline will not finish.

Even on upstream, there might be times the runner has to be taken
offline for maint work, or unexpectedly fail. We need a quick way
to disable the integration tests if we decide we don't want to
have pipelines queued until the runner comes back online.

Both these problems can be addressed by requiring a environment
variable to be set

   LIBVIRT_CI_INTEGRATION=1

This can be done in the GitLab repo CI settings for permanent
enablement. Alternatively it can be set for individual
scheduled jobs, or using a push option

  git push -o ci.variable=LIBVIRT_CI_INTEGRATION=1

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-03-18 16:45:57 +01:00
Daniel P. Berrangé
b1c7cca6c9 ci: rename integration test template
Although we split out jobs across many files, the template / job
namespace is global, so we should use something more specific
than '.tests' as the template name.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-03-18 16:45:35 +01:00
Erik Skultety
1dfd308843 gitlab-ci: Introduce new 'integration_tests' pipeline stage
This stage will download build artifacts from both the libvirt and
libvirt-perl (multi-project CI) builds, install all them on the custom
runners and configures libvirt debug logging on the runners prior to
executing the actual test suite. In case of a failure, libvirt and
Avocado logs will be saved and published as pipeline artifacts.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-17 16:09:53 +01:00
Erik Skultety
4ee1c4cc6d ci: manifest: Publish RPMs as artifacts on CentOS Stream and Fedoras
We're already building libvirt in the containers already, if we publish
the build in form of, say, RPMs, later stages of the pipeline can
consume the RPMs instead of re-building libvirt from scratch.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-17 16:09:53 +01:00
Erik Skultety
68141bd148 ci: gitlab: Refresh gitlab.yml
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-17 16:09:53 +01:00
Martin Kletzander
92e00c7afc Add Alpine builds to CI
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-03-17 11:28:40 +01:00
Peter Krempa
79c4e4e5c4 ci: Regenerate with new lcitool
Regenerate with lcitool as of:

  commit f83b916d5efa4bd33fbf4b7ea41bf6d535cc63fb
  Author: Alex Bennée <alex.bennee@linaro.org>
  Date:   Fri Feb 11 09:39:30 2022 +0000

      mappings: skip multipath-tools for cross Debian

      This package is both a mix of library files, headers and native
      binaries so cannot be installed in a cross environment. For now skip
      it for cross targets.

      See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005323

      Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-15 09:33:03 +01:00
Peter Krempa
9a21d6848c ci: Move from 'centos-8' to 'almalinux-8'
Upstream lcitool suggests that as a solution to 'centos-8' being
removed.

Move also the website and other jobs to depend on
'x86_64-almalinux-8-container'.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-15 09:33:03 +01:00
Peter Krempa
eca3daaf5b ci: Regenerate with new lcitool
Re-generate our CI infra with 'lcitool' as of:

  commit b346752e98bd12395233ebba8c9312e08212b639 (HEAD)
  Author: Erik Skultety <eskultet@redhat.com>
  Date:   Tue Feb 1 10:48:53 2022 +0100

      tests: Replace CentOS 8 with AlmaLinux 8 in test scenarios

      Switch the test target before actually dropping CentOS 8.

      Signed-off-by: Erik Skultety <eskultet@redhat.com>

This is prior to upstream lcitool dropping 'centos-8' support to
minimize the differences.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-15 09:33:03 +01:00
Erik Skultety
d955cbd3a1 ci: containers: Add CentOS Stream 9 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-01 17:33:17 +01:00
Erik Skultety
7485f47082 ci: manifest: Allow RPM builds on CentOS Stream 8
The meson version provided by the package managing system satisfies our
minimum requirement.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-01 17:33:16 +01:00
Andrea Bolognani
4c7316b2f0 ci: Refresh configuration
Notable changes:

  * drop parted and XFS headers.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2022-01-04 18:08:28 +01:00
Andrea Bolognani
d004171806 ci: Refresh configuration
Notable changes:

  * drop libdbus and radvd;
  * add codespell.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2022-01-03 16:52:25 +01:00