Commit Graph

184 Commits

Author SHA1 Message Date
Erik Skultety
72887b8f59 ci: Break off the integration_tests template into more templates
There's quite a lot happening in the .integration_tests template
already even without adding upstream QEMU build into the mix.
Let's break the template into more pieces which can then reference
in the .integration_tests template when putting all the pieces back
together using YAML anchors.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Erik Skultety
387932771a ci: Separate the integration job template to a separate file
Future patches will do more code extraction from the existing template
using YAML anchors so it'd be better that the templates would live
separately from job definitions.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-05-12 11:12:26 +02:00
Andrea Bolognani
0495f841ee ci: Move clang/asan/ubsan jobs to Ubuntu 22.04
Using an updated toolchain means we're likely to catch more
issues.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-06 15:25:18 +02:00
Andrea Bolognani
29ee8c1d2e ci: Add Ubuntu 22.04
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-06 11:13:12 +02:00
Andrea Bolognani
1e25a480f5 ci: Refresh configuration
Notable changes:

  * use the most recent point release for FreeBSD 12.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-05-06 11:13:05 +02:00
Daniel P. Berrangé
d4d5cebdf1 ci: print stack traces on macOS if any tests fail
Mostly the tests should fail cleanly, but sometimes the test might crash
or abort. In this case we'll need to know the stack traces in order to
debug the problem. Fortunately on the Cirrus CI macOS instances, the
crash reporting service is active and saving crashes to the directory
~/Library/Logs/DiagnosticReports/

Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-04-28 13:36:34 +01:00
Daniel P. Berrangé
a45e4b42fe ci: print test suite logs on failure for Cirrus jobs
We don't have access to the 'testlog.txt' file, so we need meson to
print the failures for any broken tests directly.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2022-04-26 13:17:52 +01:00
Andrea Bolognani
08e097f605 ci: Drop openSUSE Leap 15.2
It has reached end of life in January 2022.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:34 +02:00
Andrea Bolognani
e33366fc19 ci: Add openSUSE Leap 15.3
It was released in June 2021.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:31 +02:00
Andrea Bolognani
67e3589120 ci: Add Alpine Linux 3.15
It was released in November 2021.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:30 +02:00
Andrea Bolognani
de390af9d7 ci: Simplify Alpine Linux 3.14 definition
Using the extended definition format is only necessary when
the defaults are not good enough, but that's not the case here.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:28 +02:00
Andrea Bolognani
766a2d2e52 ci: Drop FreeBSD -CURRENT
All build jobs for the target are explicitly disabled, so
there's no point in keeping the variables file around and we
can simply not mention it in the manifest at all.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-04-14 17:59:11 +02:00
Erik Skultety
cc05c4e5e4 ci: Refresh Dockerfiles and vars files
In this batch:
- dnsmasq is dropped as build dependency
- Alpine Edge rpcgen package collision fix

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-04-07 16:19:42 +02:00
Peter Krempa
e3ab7900da ci: Drop Ubuntu 1804
As of April 23 2022, Ubuntu 20.04 will be out for two years, which per
our platform support policy means we no longer have to support
Ubuntu 18.04.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2022-04-04 12:26:58 +02:00
Tim Wiederhake
49156f8070 Fix spelling
Signed-off-by: Tim Wiederhake <twiederh@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-03-31 11:31:08 +02:00
Erik Skultety
69bcfa5e3b ci: integration: Rename all Avocado standard stream log files to *.log
By default, stdout/stderr Avocado test log files do not have any file
extension which confuses GitLab's web UI to mangle the MIME type for
these and so the browser will never offer the option to open such file
from in a text editor rather than dowloading it.
Since GitLab sets a proper MIME for .txt and .log file extensions,
rename all Avocado log files without an extension to *.log . This pairs
nicely with the coredumpctl info file which we already name as
'coredumpctl.txt' because of this.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-03-22 17:11:51 +01:00
Erik Skultety
908cb26b5d ci: integration: Collect stack traces with coredumpctl
Some Red Hat-like distros have cores limited with a soft limit of 0
which means that neither a stack trace nor a core file will be
available. Since we want the stack trace we need to set the core limit
with systemd globally to unlimited/infinity.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-22 12:07:37 +01:00
Erik Skultety
e23353756e ci: Define the integration job tag dynamically via a variable
Custom runners are private to a project, so naturally forks cannot run
any workloads on these. The integration test suite which requires
access to our custom runner is naturally disabled on forks and can be
enabled by setting LIBVIRT_CI_INTEGRATION=1.
The problem is that the current integration jobs definitions have tags
statically defined as 'redhat-vm-host'. If users are going to supply
their own private runners for their forks, they can define whatever
tags they want with it and so unless they add 'redhat-vm-host' to their
own runner's tags, the pipeline won't run.
To solve this, define the integration job tag using a variable. The
repo config will use the value defined in the job for the variable
while users can override the value easily on a project/pipeline level
thanks to GitLab's CI variable precedence [1].

[1] https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-22 12:07:37 +01:00
Daniel P. Berrangé
e4327557a4 ci: only run integration tests if $LIBVIRT_CI_INTEGRATION=1 is set
Right now the jobs have no rules so they will always be created in
a pipeline. If the user's fork has no runner configured, then the
jobs will never be able to execute and the pipeline will not finish.

Even on upstream, there might be times the runner has to be taken
offline for maint work, or unexpectedly fail. We need a quick way
to disable the integration tests if we decide we don't want to
have pipelines queued until the runner comes back online.

Both these problems can be addressed by requiring a environment
variable to be set

   LIBVIRT_CI_INTEGRATION=1

This can be done in the GitLab repo CI settings for permanent
enablement. Alternatively it can be set for individual
scheduled jobs, or using a push option

  git push -o ci.variable=LIBVIRT_CI_INTEGRATION=1

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-03-18 16:45:57 +01:00
Daniel P. Berrangé
b1c7cca6c9 ci: rename integration test template
Although we split out jobs across many files, the template / job
namespace is global, so we should use something more specific
than '.tests' as the template name.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-03-18 16:45:35 +01:00
Erik Skultety
1dfd308843 gitlab-ci: Introduce new 'integration_tests' pipeline stage
This stage will download build artifacts from both the libvirt and
libvirt-perl (multi-project CI) builds, install all them on the custom
runners and configures libvirt debug logging on the runners prior to
executing the actual test suite. In case of a failure, libvirt and
Avocado logs will be saved and published as pipeline artifacts.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-17 16:09:53 +01:00
Erik Skultety
4ee1c4cc6d ci: manifest: Publish RPMs as artifacts on CentOS Stream and Fedoras
We're already building libvirt in the containers already, if we publish
the build in form of, say, RPMs, later stages of the pipeline can
consume the RPMs instead of re-building libvirt from scratch.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-17 16:09:53 +01:00
Erik Skultety
68141bd148 ci: gitlab: Refresh gitlab.yml
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2022-03-17 16:09:53 +01:00
Martin Kletzander
92e00c7afc Add Alpine builds to CI
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-03-17 11:28:40 +01:00
Peter Krempa
79c4e4e5c4 ci: Regenerate with new lcitool
Regenerate with lcitool as of:

  commit f83b916d5efa4bd33fbf4b7ea41bf6d535cc63fb
  Author: Alex Bennée <alex.bennee@linaro.org>
  Date:   Fri Feb 11 09:39:30 2022 +0000

      mappings: skip multipath-tools for cross Debian

      This package is both a mix of library files, headers and native
      binaries so cannot be installed in a cross environment. For now skip
      it for cross targets.

      See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005323

      Signed-off-by: Alex Bennée <alex.bennee@linaro.org>

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-15 09:33:03 +01:00
Peter Krempa
9a21d6848c ci: Move from 'centos-8' to 'almalinux-8'
Upstream lcitool suggests that as a solution to 'centos-8' being
removed.

Move also the website and other jobs to depend on
'x86_64-almalinux-8-container'.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-15 09:33:03 +01:00
Peter Krempa
eca3daaf5b ci: Regenerate with new lcitool
Re-generate our CI infra with 'lcitool' as of:

  commit b346752e98bd12395233ebba8c9312e08212b639 (HEAD)
  Author: Erik Skultety <eskultet@redhat.com>
  Date:   Tue Feb 1 10:48:53 2022 +0100

      tests: Replace CentOS 8 with AlmaLinux 8 in test scenarios

      Switch the test target before actually dropping CentOS 8.

      Signed-off-by: Erik Skultety <eskultet@redhat.com>

This is prior to upstream lcitool dropping 'centos-8' support to
minimize the differences.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-15 09:33:03 +01:00
Erik Skultety
d955cbd3a1 ci: containers: Add CentOS Stream 9 target
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-01 17:33:17 +01:00
Erik Skultety
7485f47082 ci: manifest: Allow RPM builds on CentOS Stream 8
The meson version provided by the package managing system satisfies our
minimum requirement.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2022-02-01 17:33:16 +01:00
Andrea Bolognani
4c7316b2f0 ci: Refresh configuration
Notable changes:

  * drop parted and XFS headers.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2022-01-04 18:08:28 +01:00
Andrea Bolognani
d004171806 ci: Refresh configuration
Notable changes:

  * drop libdbus and radvd;
  * add codespell.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2022-01-03 16:52:25 +01:00
Peter Krempa
019eab7240 ci: Refresh dockerfiles
Apart from some churn, the important is the removal of 'netcf-devel'
from the fedora rawhide container.

Update to state as of 174fe4999204afcae (libvirt-ci).

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
2021-12-22 17:24:43 +01:00
Daniel P. Berrangé
c7edcb320b ci: run a mingw64 job on stable Fedora
Both of the current mingw jobs are marked as 'allow_failure' because
they are running against Fedora rawhide which is an unstable distro.

We need at least one mingw job to be gating to more reliably detect
problems.

This introduces dockerfiles for both mingw variants on Fedora 35
and sets the mingw64 build to run on Fedora 34, and mingw32 on
Fedora rawhide.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-11-23 16:44:18 +00:00
Daniel P. Berrangé
3ca8611455 ci: refresh variables/dockerfiles with latest content
- The Cirrus CI variables are now sorted
  - The dockerfiles update commands changed for some distros
  - Meson in CentOS is now new enough to use

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-11-23 16:44:15 +00:00
Daniel P. Berrangé
4e2dcfcb82 ci: replace Fedora 33 with Fedora 35
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-11-23 16:43:57 +00:00
Daniel P. Berrangé
9ff311f105 ci: regenerate container from manifest
This removes the libnetcf-dev package from Debian Sid, as it is no
longer available in that distro stream.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-10-25 12:45:31 +01:00
Daniel P. Berrangé
28a1059ddd ci: switch all non-released distros to be non-gating
The non-released distros have reasonably frequent package installation
failures that can last for days at a time. This makes them unsuitable
for use as gating CI jobs.

This ensures all of the jobs in Debian Sid, Fedora Rawhide, openSUSE
Tumbleweed and FreeBSD Current are marked "allow-failure: true".

This means the jobs still run, but any failure will not be considered
fatal to the pipeline.

Reviewed-by: Ján Tomko <jtomko@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-10-20 12:57:56 +01:00
Daniel P. Berrangé
13eab52429 ci: refresh configuration from manifest
The layering of the cross containers is fixed to move arch specific
ccache setup out of the common base layer.

A missing Cirrus CI variable substitution is added, though this is
irrelevant given libvirt's package list.

Reviewed-by: Ján Tomko <jtomko@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-10-20 12:57:16 +01:00
Andrea Bolognani
235e7c9509 ci: Regenerate configuration from manifest
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-14 18:58:49 +02:00
Andrea Bolognani
8c5eced101 ci: Add Debian 11 builds
The armv7l and ppc64le cross-builds as well as the Clang build
are adopted from Debian 10, while the mips64el build is adopted
from Debian sid. As always, the way jobs are distributed across
Debian versions is fairly arbitrary.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-14 18:58:45 +02:00
Andrea Bolognani
4ba191f7e5 ci: Avoid use of magic constants
The value 3 is the length of the "ci-" prefix, which is present
in the items returned by get_registry_images() but not in those
returned by get_dockerfiles().

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2021-09-14 18:56:49 +02:00
Andrea Bolognani
a4b00fd2b1 ci: Restore information about Coverity integration
These were removed along with the outdated information on how
to regenerate the Dockerfiles contained in the repository, but
this part is still relevant.

Reverts: 30856d2865 (partially)
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
2021-09-14 18:56:35 +02:00
Daniel P. Berrangé
9810633253 ci: remove obsolete logic for refreshing containers/cirrus
We now use lcitool's manifest feature to generate files. The logic
for checking for stale containers in the registry, however, is still
relevant so that is propagated to a standalone command.

Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-10 13:36:12 +01:00
Daniel P. Berrangé
30856d2865 ci: remove obsolete refresh documentation
We now use lcitool's manifest feature to generate files.

Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-10 13:36:11 +01:00
Daniel P. Berrangé
ccc7a44adb ci: re-generate containers/gitlab config from manifest
This uses the command "lcitool manifest ci/manifest.yml" to re-generate
all existing dockerfiles and gitlab CI config.

Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-10 13:36:09 +01:00
Daniel P. Berrangé
712407ed1e gitlab: define a manifest file to express the CI config
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-10 13:36:07 +01:00
Daniel P. Berrangé
c14d958737 gitlab: refresh containers / cirrus files with latest content
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-09-10 13:36:03 +01:00
Andrea Bolognani
d55547ec37 ci: Adapt to lcitool command line changes
lcitool now uses the term "target" instead of "host" to refer to
the various operating systems it supports, and we need to adapt
our helper script so that it works with the new command line
interface.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-08-04 09:52:28 +02:00
Daniel P. Berrangé
b9533b5ecf ci: refresh cirrus variables for FreeBSD python rename
All the python packages got renamed from py37- to py38-

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-07-15 11:10:52 +01:00
Martin Kletzander
5f96ea10d4 ci: Also perform package upgrades on macOS and FreeBSD
The base OS image might include outdated contents, and we don't
want to get spurious failures caused by bugs that have already been
fixed in the respective packages.

This is particularly important on macOS, because 'brew install foo'
will fail if 'foo' is already installed but outdated: upgrading all
packages first ensures we never run into this scenario.

Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-06-24 09:14:08 +02:00
Daniel P. Berrangé
156315cff4 gitlab: avoid building libvirt twice and running syntax-check twice
In the distros using RPMs, we build libvirt once as a side effect
of running "ninja dist", and once via rpmbuild.

In addition "ninja dist" will run all tests including the "syntax-check"
suite, despite use having a separate "codestyle" job for for that.

There is no way to pass "--no-suite" when creating the dist, but if we
switch to invoking "meson dist", we can skip the build+test part
entirely  using "--no-tests".

When doing this we then run explicit "meson compile" and "meson test"
commands for the distros that don't build the RPMs, and in the latter
case we can now skip the "syntax-check" suite.

The RPM builds already skipped the "syntax-check" suite.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-12 14:18:15 +01:00
Daniel P. Berrangé
b56e2be68e gitlab: run "meson" directly instead of "ninja" indirection
The "dist" and "test" targets in ninja end up calling back into
the equivalent meson commands. The meson commands support various
arguments that are not accessible when invoked via ninja, so it
is preferrable to use meson directly.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-12 14:18:08 +01:00
Daniel P. Berrangé
68e8fbe6b1 gitlab: enable debug output for test suite
When CI jobs fail on the test suite, we don't see much helpful
information by default:

stderr:
TEST: bhyvexml2argvtest
      ......!!.............!......!........... 40
      ........................!......          71  FAIL
Some tests failed. Run them using:
VIR_TEST_DEBUG=1 VIR_TEST_RANGE=7-8,22,29,65
/tmp/cirrus-ci-build/build/meson-private/dist-build/tests/bhyvexml2argvtest

Following the instructions to re-run the test with VIR_TEST_DEBUG=1 is
quite unfriendly when we could have had that set for CI already.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 14:45:34 +01:00
Andrea Bolognani
d35003aee7 ci: Drop CentOS 7
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:21 +02:00
Andrea Bolognani
eabfb73356 ci: Drop Fedora 32
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:20 +02:00
Andrea Bolognani
b488200a92 ci: Add Fedora 34
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:19 +02:00
Andrea Bolognani
7f2f7ef23b ci: Add FreeBSD 13
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:17 +02:00
Andrea Bolognani
103d019a4f ci: Add openSUSE Tumbleweed
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:16 +02:00
Andrea Bolognani
a3fef27fb5 ci: Refresh contents
Notable changes:

  * the CentOS Stream 8 container is now using a proper base
    image instead of starting from a CentOS 8 image and then
    adding the CentOS Stream 8 repositories on top;

  * distributions that have a perl-base package are now using
    that one instead of the regular perl package, which
    contains a bunch of features we don't need, resulting in
    smaller containers.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:14 +02:00
Andrea Bolognani
043adb8b50 ci: Rename CentOS Stream 8 and openSUSE Leap 15.2 containers
The names have been recently changed in libvirt-ci to be more
accurate, so we should follow along.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-05-05 13:25:12 +02:00
Andrea Bolognani
a0491637e1 ci: Refresh contents
Notable changes:

  * cross-building container images are smaller because they
    no longer include the native compilers;

  * ccache is enabled for clang builds.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2021-04-15 19:07:16 +02:00
Peter Krempa
0fa141376c ci: README: Mention necessary step for cirrus to pick up the github project
Unless you create such an commit, cirrus-ci.com will not pick up the
github project and cirrus-run will fail.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2021-04-12 16:38:11 +02:00
Andrea Bolognani
3ad73d876e ci: Don't use --prefix with meson for Cirrus CI builds
It's no longer used.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-04-01 14:09:43 +02:00
Andrea Bolognani
228a87d307 ci: Refresh information
Notable changes:

  * HAL is no longer installed on FreeBSD;

  * the native version of libwsman is no longer installed in
    containers intended for cross-compilation;

  * Meson 0.55 rather than 0.54 is requested when installing
    it from PyPI;

  * GNU sed and GNU grep are installed explicitly everywhere.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-26 09:17:22 +01:00
Andrea Bolognani
552a501d55 ci: Drop FreeBSD 11 build
FreeBSD 12 was released in December 2018, so according to our
platform support policy we can now drop support for the previous
major release. It would be going EOL in September anyway.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2021-03-23 13:41:11 +01:00
Andrea Bolognani
792843c05e ci: Drop prefix from Dockerfiles
Since the string "ci" is already contained in the path, it
seems unnecessary to include it into the filename too: in fact,
we only do that for Dockerfiles and not for files in ci/cirrus,
even though those are generated the very same way.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-22 11:40:57 +01:00
Erik Skultety
fc47ba38aa ci: helper: Apply Python naming practice to private methods/attributes
As documented at [1], the common practice with respect to private
attributes/methods naming is to prefix them with an underscore.

[1] https://docs.python.org/3/tutorial/classes.html#private-variables

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-03-19 11:50:07 +01:00
Erik Skultety
efa8ca47b9 ci: util: Add a registry checker for stale images
This function checks whether there are any stale Docker images in the
registry that can be purged. Since we're pulling available container
images from our GitLab registry with the 'list-images' action, it
could happen that we'd list old (already unsupported) images and make
them available for the user to consume and run a build in them.
Naturally, the build will most likely fail leaving the user confused.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-03-19 11:50:07 +01:00
Erik Skultety
c5f9617b1c ci: helper: Rewrite image listing to Python
The corresponding Bash script is dropped.
After this patch's rewrite, the Makefile's original image listing
target remains intact only to notify the user to use the Python helper
instead.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-03-19 11:47:26 +01:00
Erik Skultety
5a0e4d79cc ci: Introduce a util module
With the gradual rewrite of the Makefile to the 'helper' script will
require helper functions that would better live in a separate util
module.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-03-19 11:47:06 +01:00
Erik Skultety
013b5a085a ci: helper: Introduce --quiet for the refresh command
Offer an option to silence all output to stdout coming out of the
dockerfiles/varsfiles generating code.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-03-19 11:46:33 +01:00
Erik Skultety
cbd7124f49 ci: helper: Use the ArgumentDefaultsHelpFormatter help formatter
This help formatter class reports the defaults we use for options
taking an argument.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-03-19 10:58:51 +01:00
Andrea Bolognani
c747e51e92 ci: Discourage users from using the Makefile directly
We now wrap all its important functionality with the much more
user-friendly ci/helper script, and the long term plan is for
the Makefile to disappear completely.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
b79e48db7e ci: Delete refresh scripts
The functionality is now available in the ci/helper script.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
22ed68d0a9 ci: Implement 'test' helper action
This simply calls the underlying Makefile target, but allows
additional arguments to be specified in a more convenient and
discoverable way.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
f0fd72381d ci: Implement 'build' helper action
This simply calls the underlying Makefile target, but allows
additional arguments to be specified in a more convenient and
discoverable way.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
897974c0b3 ci: Implement 'shell' helper action
This simply calls the underlying Makefile target, but allows
additional arguments to be specified in a more convenient and
discoverable way.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
2481ad1125 ci: Implement 'list-images' helper action
This simply calls the underlying Makefile target.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
06cb54f36b ci: Implement 'refresh' helper action
This provides the same functionality as the two refresh scripts
that are currently in the repository, with the following
advantages:

  * all files are refreshed with a single command;

  * if lcitool is present in the user's $PATH, it will be
    discovered and used automatically;

  * some output is produced, so the user can follow along with
    the progress of the operation.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
b204cdab69 ci: Add helper script
This is intended to be perform a number of CI-related operations
that are currently implemented in various different scripts
written in various different programming languages.

Eventually, all existing functionality will be reimplemented in
Python and made available through this single entry point; for
now, let's start with a very basic skeleton.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Andrea Bolognani
7f601d1f10 ci: Fix name for ci-test target in help output
The target was renamed when moving to Meson, but the help text
was not updated accordingly.

Fixes: 1a0af38ae7
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-03-15 18:49:03 +01:00
Erik Skultety
f1157ee530 ci: Makefile: Expose CI_IMAGE_PREFIX and CI_IMAGE_TAG in 'ci-help'
Using locally built images is a useful feature; our commentaries even
mention overriding them may be useful in some scenarios. Expose the
variables in the help to let users know they can use the feature.

Formatting would definitely break, so this patch adds more spacing for
proper alignment.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-02-17 11:05:42 +01:00
Erik Skultety
07103a0cae ci: Drop the CI_PREPARE_SCRIPT variable
In commit 321293e2 I dropped the prepare.sh script, but forgot to
remove the corresponding variable from the Makefile.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-02-17 11:05:42 +01:00
Andrea Bolognani
cf325da1f8 ci: Remove Debian 9 special case
We no longer target this platform.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2021-02-17 09:53:32 +01:00
Andrea Bolognani
3ab9b399bc ci: Build on macOS 11 instead of macOS 10.15
macOS builder capacity on Cirrus CI is quite limited, and so we
can't afford to keep the old build job around after adding the
new one like we do for FreeBSD.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2021-02-16 11:35:44 +01:00
Andrea Bolognani
cb5defccb1 ci: Update package list on Cirrus CI
While pkgng on FreeBSD updates the package list automatically
when it's run, homebrew on macOS doesn't do the same thing, which
can result in stale packages being installed. Explicitly call
'brew update' before 'brew install' to avoid that scenario.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2021-02-16 11:35:38 +01:00
Andrea Bolognani
2d92970d8f ci: Refresh Dockerfiles
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2021-02-15 12:11:23 +01:00
Erik Skultety
f28a652a32 ci: Makefile: Expose the CI_USER_LOGIN variable for users to use
More often than not I find myself debugging in the containers which
means that I need to have root inside, but without manually tweaking
the Makefile each time the execution would simply fail thanks to the
uid/gid mapping we do. What if we expose the CI_USER_LOGIN variable, so
that when needed, the root can be simply passed with this variable and
voila - you have a root shell inside the container with CWD=~root.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-02-12 17:01:41 +01:00
Erik Skultety
321293e2a3 ci: Drop the prepare.sh script
The purpose of this script was to prepare a customized environment in
the container, but was actually never used and it required the usage of
sudo to switch the environment from root's context to a regular user's
one.
The thing is that once someone needs a custom script they would very
likely to debug something and would also benefit from root privileges
in general, so the usage of 'sudo' in such case was a bit cumbersome.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-02-12 17:01:41 +01:00
Erik Skultety
ee07bffacc ci: Run podman command directly without wrapping it with prepare.sh
The prepare.sh script isn't currently used and forces us to make use
of sudo to switch the user inside the container from root to $USER
which created a problem on our Debian Slim-based containers which don't
have the 'sudo' package installed.
This patch removes the sudo invocation and instead runs the CMD
directly with podman.

Summary of the changes:
- move the corresponding env variables which we need to be set in the
  environment from the sudo invocation to the podman invocation
- pass --workdir to podman to retain the original behaviour we had with
  sudo spawning a login shell.
- MESON_OPTS env variable doesn't need to propagated to the execution
  environment anymore (like we had to do with sudo), because it's
  defined in the Dockerfile

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-02-12 17:01:41 +01:00
Erik Skultety
3ca7299a00 ci: Specify the shebang sequence for build.sh
This is necessary for the follow up patch, because the default
entrypoint for a Dockerfile is exec.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2021-02-12 17:01:41 +01:00
Andrea Bolognani
0947902d87 ci: Refresh dockerfiles
dtrace is now also installed when cross-building.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-26 11:01:55 +01:00
Erik Skultety
a79ac43f04 ci: Drop openSUSE Leap 15.1
Leap 15.1 will reach EOL on January 31st 2021, so we should not test on
it during the current development cycle ending on March 1st.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-20 11:20:26 +01:00
Erik Skultety
3114cb4bfd ci: Add openSUSE Leap 15.2
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-20 11:20:26 +01:00
Erik Skultety
6315d923c2 ci: Refresh Dockerfiles
In this refresh CentOS 7 now uses docker.io registry and the PowerTools
repo name regression was fixed for CentOS Stream this time.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-20 11:20:26 +01:00
Daniel P. Berrangé
d855f12945 ci: refresh cirrus vars files
In this refresh the PIP variable is renamed to PIP3 and the
PYPI_PKGS variable disappears since we (currently) don't have
any need for it.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-11 16:14:12 +00:00
Daniel P. Berrangé
855399e382 ci: rename the cirrus vars file to remove the "libvirt-" prefix
The "libvirt-" prefix was removed from hostnames in libvirt-ci.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-11 16:14:06 +00:00
Erik Skultety
555b880aae ci: containers: Refresh the Dockerfiles
Contains changes utilizing "nosync" and "eatmydata" for speedup as well
as fixes for CentOS-8 repoid regression.
ci-commit: b098ec6631a85880f818f2dd25c437d509e53680

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Erik Skultety <eskultet@redhat.com>
2020-12-15 09:52:14 +00:00
Daniel P. Berrangé
4e750e932a gitlab: replace "libvirt-" prefix with "ci-" in dockerfiles
This makes the dockerfile name match the output container name

Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-12-08 16:04:03 +00:00
Daniel P. Berrangé
82a7f760df gitlab: refresh containers with lcitool for fully minimized base
Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-12-08 16:03:59 +00:00