Commit Graph

90 Commits

Author SHA1 Message Date
Ján Tomko
09464e6929 ci: run Cirrus-based builds sooner
Jobs using cirrus-run don't need to wait for any jobs from
the container stage. Run them as soon as possible.

Signed-off-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Neal Gompa <ngompa13@gmail.com>
2020-08-03 08:35:22 +02:00
Ján Tomko
14ef032e20 ci: add clang jobs
Run the build with clang on Fedora Rawhide, Debian 10 and CentOS 8.

Signed-off-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Neal Gompa <ngompa13@gmail.com>
2020-08-03 08:35:22 +02:00
Ján Tomko
31c69e188c ci: move sanity_checks after other stages
Be less hostile to developers and run the build jobs even
if the dco-check failed. That way they can test their own
"private" branches without a sign-off.

Also specify empty `needs` for it, since it does not depend
on any of the jobs in the container phase and can be run
right away.

Signed-off-by: Ján Tomko <jtomko@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-08-03 07:23:45 +02:00
Andrea Bolognani
8a4f807536 ci: Use correct name for armv7l architecture
Turns out 'ldebian' was not a typo, but rather the result of
mistakenly shifting the dash in 'armv7l-debian' one spot to the
left.

Fixes: 821100e812
Reported-by: Pavel Hrdina <phrdina@redhat.com>
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2020-07-29 10:55:01 +02:00
Andrea Bolognani
821100e812 ci: Fix typo 'ldebian'
Fixes: 95abbdc432
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2020-07-29 10:21:24 +02:00
Andrea Bolognani
fc832369d0 ci: Drop Debian 9 jobs
The existing cross-compilation jobs are carefully redistributed
among Debian 10 and Debian sid to ensure we don't use the latter
for aarch64, mipsel or mips64el, since those architectures are
currently broken.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-07-15 12:06:26 +02:00
Andrea Bolognani
fb91290131 cirrus: Generate jobs dynamically
Instead of having static job definitions for FreeBSD and macOS,
use a generic template for both and fill in the details that are
actually different, such as the list of packages to install, in
the GitLab CI job, right before calling cirrus-run.

The target-specific information are provided by lcitool, so that
keeping them up to date is just a matter of running the refresh
script when necessary.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2020-07-03 11:00:22 +02:00
Andrea Bolognani
4268e18753 ci: Run all jobs, for all branches, all the time
After recent changes (increasing the parallelism of the pipeline
by reducing the number of stages, introducing FreeBSD builds that
take longer than any other job), the difference between running
the full pipeline or a reduced one has basically disappeared: in
both cases, the completion time is around 25-35 minutes depending
on whether containers need to be rebuilt and how many shared
runners are available.

Reduce the complexity of our .gitlab-ci.yml and make things
simpler for contributors by simply always running all jobs.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-06-29 17:43:28 +02:00
Jonathon Jongsma
d66181c84e ci: Use openSUSE image for code style checking
CentOS does not have the cppi package, so some code style checks are
skipped. Switch to a openSUSE image to do the code style checks.

Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2020-06-26 09:25:45 +02:00
Andrea Bolognani
60a7c78348 ci: Drop mips container for Debian sid
The build job for this container has been failing every single
time, and as it turns out the explanation for that is very simple:
Debian is just not going to support the mips architecture going
forward.

Reported-by: Pino Toscano <ptoscano@redhat.com>
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-06-23 12:14:00 +02:00
Andrea Bolognani
63d08bec0b ci: Swap mipsel and ppc64le builds
Debian sid is currently broken on mipsel, so use Debian 10 for
that architecture; at the same time, move the ppc64le build from
Debian 10 to Debian sid to keep things balanced.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
2020-06-11 20:08:32 +02:00
Andrea Bolognani
95abbdc432 ci: Use GitLab container registry
Instead of using pre-built containers hosted on Quay, build
containers as part of the GitLab CI pipeline and upload them to the
GitLab container registry for later use.

This will not significantly slow down builds, because containers are
only rebuilt when the corresponding Dockerfile has been modified.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-06-11 19:06:51 +02:00
Andrea Bolognani
7ef1324284 ci: Use variables to build image names
This removes a lot of repetition and makes the configuration much
easier to read.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-06-11 19:06:49 +02:00
Andrea Bolognani
0d009ca646 ci: Reduce number of stages
Right now we're dividing the jobs into three stages: prebuild, which
includes DCO checking as well as building artifacts such as the
website, and native_build/cross_build, which do exactly what you'd
expect based on their names.

This organization is nice from the logical point of view, but results
in poor utilization of the available CI resources: in particular, the
fact that cross_build jobs can only start after all native_build jobs
have finished means that if even a single one of the latter takes a
bit longer the pipeline will stall, and with native builds taking
anywhere from less than 10 minutes to more than 20, this happens all
the time.

Building artifacts in a separate pipeline stage also doesn't have any
advantages, and only delays further stages by a couple of minutes.
The only job that really makes sense in its own stage is the DCO
check, because it's extremely fast (less than 1 minute) and, if that
fails, we can avoid kicking off all other jobs.

Reducing the number of stages results in significant speedups:
specifically, going from three stages to two stages reduces the
overall completion time for a full CI pipeline from ~45 minutes[1]
to ~30 minutes[2].

[1] https://gitlab.com/abologna/libvirt/-/pipelines/154751893
[2] https://gitlab.com/abologna/libvirt/-/pipelines/154771173

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-06-10 19:33:09 +02:00
Andrea Bolognani
6190c14151 ci: Enable Cirrus CI integration
We use cirrus-run to trigger Cirrus CI jobs from GitLab CI jobs,
making it possible to extend our platform coverage to include
FreeBSD without having to maintain our own runners; additionally,
we'll be able to ditch Travis CI and, since results for Cirrus CI
jobs are reflected back to the GitLab CI jobs that triggered them,
we will be able to get all information from a single dashboard.

The FreeBSD and macOS job definitions can be improved further: for
example, we will want to enable caching to speed up builds, and
ultimately we should figure out a way to generate at least part of
them, notably the list of packages to be installed, using lcitool.
All of that will happen in later patches: for now, this is good
enough to start using Cirrus CI.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2020-06-10 10:30:56 +02:00
Daniel P. Berrangé
3cdda5d5e8 gitlab: fix potfile job for new location
Since we're storing the libvirt.pot file in git now, we should pull from
the source dir, not the build dir.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-06-04 18:53:21 +01:00
Daniel P. Berrangé
019b71dead gitlab: move some jobs onto CentOS 8
So that we don't have to chase frequent Fedora releases, move the
non-build related jobs onto the long life CentOS 8 distro.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-05-06 13:16:58 +01:00
Daniel P. Berrangé
57e9f3a7e0 gitlab: update list of distros to use latest versions
Replace Fedora 30 with Fedora 32
Replace Ubuntu 16.04 with Ubuntu 20.04

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-05-04 17:20:08 +01:00
Daniel P. Berrangé
333c7e0784 gitlab: convert to shared container for DCO check
This removes the locally maintained DCO checking script in favour of the
shared container image provided by libvirt-ci.git.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-05-04 15:43:55 +01:00
Andrea Bolognani
a158eb5c8e gitlab: Enable improved ccache usage
Setting CC="ccache cc" works in most cases, but sometimes it will
break the build: in particular, we have experienced issues in the
past with that approach when using cgo to build our Go bindings.

A more robust approach is to have a directory containing symlinks
from the compiler name to the ccache binary: in that case, ccache
itself will invoke the compiler, and the build system will be none
the wiser.

Since libvirt-ci commit 2563aebb6c5c, container images contain a
suitable symlink directory, so all that's needed to enable the new
approach is to add this directory to $PATH.

Since we're touching this anyway, we make a few more changes:
$CCACHE_DIR is no longer created manually, because ccache will
take care of creating it for us if it doesn't already exist; the
ccache setup is moved out of the job template and into
script_variables, removing unnecessary duplication; a limit is
set on the size of the cache (500 MB, which is twice the amount
used by a fresh build on my Fedora 31 machine).

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-04-15 17:50:49 +02:00
Andrea Bolognani
29e09694bc gitlab: Don't define $MAKE
Since libvirt-ci commit 27cfddee8835, paths to build tools such as
ninja and make are exported in the container's environment and can
be used directly.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-04-15 17:50:27 +02:00
Andrea Bolognani
cab35ae380 gitlab: Add script_variables and define $MAKEFLAGS there
This new template provides a standardized place where environment
variables which are not static, but rather depend on the output of
some shell command, can be defined for later use. This pattern is
already used in libosinfo's GitLab CI integration.

Defining $MAKEFLAGS there means we don't need to call getconf over
and over, and the actual build steps don't end up drowned in the
noise.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 19:08:50 +01:00
Daniel P. Berrangé
2e78ba6a36 gitlab: don't run the DCO check on git master
The "only: branches" does not in fact exclude "master", so it must be
excluded explicitly.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 14:36:16 +00:00
Daniel P. Berrangé
2c33dfb66f gitlab: add explicit early job for syntax-check
Running the code style syntax-check as part of the build jobs leads to
all jobs failing in the same way. Have a prebuild job for validating
syntax-check to catch code style problems upfront and thus avoid needing
to run all the build jobs.

Reviewed-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>a
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
c7f11d0a6c gitlab: introduce a check for validate DCO sign-off
This introduces a CI job for validating DCO sign-off in every commit
message. The CI jobs are not provided any information on what the
baseline commit for the branch was. We can't compare against the forked
repo's master branch, as there's no guarantee the user is keeping master
up2date in their fork. Thus we add the master upstream repo as a git
remote and identify the common ancestor.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
1e16d4f483 gitlab: introduce use of ccache for speeding up rebuilds
For any given job there is a high likelihood that ccache will be able to
reuse previously built object files. This will result in faster build
pipelines in later updates.

Reviewed-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
6cdfbef1a7 gitlab: add job for building latest potfile
Whenever there is a change to the translatable strings we need to push
a new libvirt.pot to weblate. This only needs to be done when code
merges into git master, so the job is restricted to that branch.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
74aa4671cc gitlab: add x86_64 native CI jobs
This patch adds x86_64 native CI jobs for all distros that we currently
build container images for. This is a superset of the Linux jobs run on
current Jenkins and Travis platforms.

The remaining missing platforms are FreeBSD and macOS, neither of which
can use the shared runner container based infrastructure.

We may add further native jobs in the future which are not x86_64 based,
if we get access to suitable hardware, thus the jobs all have an arch
prefix in their name, just like the cross-built jobs do.

As with the cross-arch builds, the native jobs are split into two
groups. One group is run in all situations, while the other group is
only run on the master branch, or branches with a name prefix
'ci-full-'. This avoids the build time getting too long when
developers are testing their code prior to submission, while keeping
full coverage of code that is merged.

Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
75b88557c4 gitlab: add mingw cross build CI jobs
This pulls in the mingw cross build jobs using Fedora 30 as a base,
matching what is done on Jenkins and Travis.

Reviewed-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
562ada4ee7 gitlab: rename the cross build jobs
The pipeline UI will truncate the names of jobs after about 15
characters. As a result with the cross-builds, we truncate the
most important part of the job name. Putting the most important
part first is robust against truncation, and we can drop the
redundant "-cross" stub.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <skultety.erik@gmail.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
bbbd4e48be gitlab: reduce number of cross build jobs run by default
Currently we have nine different cross build jobs, but as we introduce
more native jobs this is going to result in a very long CI execution
time. For developers testing their personal branches under development
it is generally sufficient to just look at a couple of interesting
scenarios, namely 32-bit and big endian.

This splits the cross build jobs so that by default only the armv7
and s390x archs are built. The remainining archs are setup so that they
are only built for code on the master branch, which will have the effect
of doing post-merge testing. Developers can opt-in to full testing of
their pre-merge code by pushing it to a branch with a name prefix of
"ci-full-".

Reviewed-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
336f93ac55 gitlab: use CI for building website contents
Run the bare minimum build that is possible to create the docs, avoiding
compiling code which other jobs will deal with.

The generated website is published as an artifact and thus is browsable
by developers on build completion and can be downloaded as a zip file.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
98674c8bb0 gitlab: create an explicit stage for cross build jobs
As we introduce more build jobs, it will be useful to have a grouping of
jobs to more easily visualize the results and potentially control build
ordering.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
396480d067 gitlab: restrict git history to 100 commits
We don't need the full git history when running CI jobs. From a code POV
we only need the most recent commit, but we want to be able to run
checks on the commits too. In particular to validate the DCO signoff for
each commit.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Daniel P. Berrangé
9e4f7cecc7 gitlab: add variable for make command name
To facilitate future jobs that will use FreeBSD

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2020-03-27 12:32:48 +00:00
Andrea Bolognani
2ecb855b83 gitlab: Move mips job to Debian 9
Support for the mips architecture has been dropped from Debian
sid. Move the mipsel job from Debian 9 to Debian sid at the same
time to keep things balanced.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-12-13 13:12:13 +01:00
Andrea Bolognani
fae7e8504c gitlab: Adapt to container name changes
GitLab CI unfortunately doesn't use the standard Makefile.ci
machinery, so its configuration needs to be updated separately.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Fabiano Fidêncio <fidencio@redhat.com>
2019-08-20 13:08:42 +02:00
Andrea Bolognani
5a23d22b6c gitlab: Perform some builds on Debian 10
The split is mostly arbitrary, but we purposefully moved
the mips64el builds off sid since currently that container
is failing to build and is thus unavailable.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-07-11 15:03:39 +02:00
Andrea Bolognani
319cc0045b ci: Use default image tag "latest"
Up until now, our images have been tagged as "master" instead
of the default "latest" due to the way the build process
worked, but we're using the default now.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
2019-07-11 15:03:32 +02:00
Daniel P. Berrangé
75c5ebc8ef tests: perform cross compiler builds on GitLab CI
GitLab CI provides some shared build runners that use Docker containers.
This resource can usefully run cross-compiled builds since all other CI
build testing is currently x86 only, and Travis CI is already very busy
testing native builds.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2019-05-09 11:34:53 +01:00