Commit Graph

46140 Commits

Author SHA1 Message Date
Michal Privoznik
c4c57cef33 conf: Move iothread formatter into a separate function
Formatting iothreads is currently open coded inside of
virDomainDefFormatInternalSetRootName(). While this works, it
makes the function needlessly long, especially if the formatting
code will expand in near future. Therefore, move it into a
separate function. At the same time, make
virDomainDefIothreadShouldFormat() accept const domain definition
so that the new function can also accept const domain definition.
Formatters shouldn't need to change definition.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-10 13:53:52 +02:00
Michal Privoznik
13a8c0aa61 virDomainIOThreadIDDefArrayInit: Decrease scope of @iothrid
In virDomainIOThreadIDDefArrayInit() the variable @iothrid is
used only inside a loop but is declared for whole function. Bring
the variable into the loop so that it's obvious that the variable
is not used elsewhere.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-10 13:53:52 +02:00
Michal Privoznik
d10b6b4c89 virDomainDefParseIOThreads: Use g_autoptr() for @iothrid
Using g_autoptr() for @iothrid variable inside
virDomainDefParseIOThreads() allows us to drop explicit call to
virDomainIOThreadIDDefFree() in one case.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-10 13:53:52 +02:00
Michal Privoznik
95c95f1b75 virml: Introduce VIR_XML_PROP_NONNEGATIVE flag
For easier attribute parsing we have virXMLProp*() family of
functions. These accept flags through which a caller can pose
some conditions onto the attribute value, for instance:
VIR_XML_PROP_NONZERO when the attribute may not be zero, etc.

What we are missing is VIR_XML_PROP_NONNEGATIVE when the
attribute value may be non-negative. Obviously, this flag makes
sense only for some members of the virXMLProp*() family.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-10 13:53:52 +02:00
Andrea Bolognani
ae2dabe5ed ci: Move upstream QEMU integration test to Fedora 36
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-06-09 14:41:44 +02:00
Andrea Bolognani
4a4b7f2854 ci: Add Fedora 36 to integration tests
This requires publishing the RPMs as artifacts from the regular
build job.

Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Reviewed-by: Erik Skultety <eskultet@redhat.com>
2022-06-09 14:41:40 +02:00
Erik Skultety
d1e3247675 ci: integration: Set 'safe.directory' when installing QEMU from git
Since a fix for CVE-2022-24765 was released every git command is now
checked against the context repo in which it's supposed to run
resulting in a fatal error if the repo is owned by other user than the
one running the git command.
This means that in order to be able to do 'sudo make install', we have
to set the 'safe.directory' for the root user. This is because QEMU
runs 'git submodule update' automatically on 'make install'.

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-06-09 13:38:17 +02:00
Erik Skultety
3b943f0c43 ci: integration: SELinux relabel the QEMU we installed from git
Signed-off-by: Erik Skultety <eskultet@redhat.com>
Reviewed-by: Andrea Bolognani <abologna@redhat.com>
2022-06-09 13:38:13 +02:00
Michal Privoznik
2177de7b6e virsh: Check whether enough arguments was passed to iothreadset
Virsh has iothreadset command which allows setting various
attributes of IOThreads. However, when the command is called
without any arguments (besides domain and IOThread IDs), then
@params stays NULL and is passed to virDomainSetIOThreadParams()
which produces rather user unfriendly error message:

  error: params in virDomainSetIOThreadParams must not be NULL

Introduce a check and produce better error message.

Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Claudio Fontana <cfontana@suse.de>
2022-06-08 15:37:35 +02:00
Jiri Denemark
4582267782 qemu: Improve error messages using qemuMigrationJobName
They were constructed from two separate strings using "%s: %s", which
is ugly and does not work well with translations.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
2022-06-08 11:00:43 +02:00
Jiri Denemark
87257c76b9 qemu: Fix VSERPORT_CHANGE event in post-copy migration
When a domain has a guest agent channel enabled and the agent is running
in the guest, we will get VSERPORT_CHANGE event on a destination host as
soon as we start vCPUs there. This is not an issue for normal migration,
but post-copy migration will remain running after we started vCPUs on
the destination. If it runs for more than 30s, the VSERPORT_CHANGE event
handler will fail to get a job and log the following error message:

    Timed out during operation: cannot acquire state change lock (held
    by monitor=remoteDispatchDomainMigrateFinish3Params)

and of course we will think the guest agent is not connected and thus
all APIs talking to it will fail. Until the agent or libvirt daemon is
restarted.

Luckily we only need to update the channel state (to mark it as
connected) and connect to the agent neither of which conflicts with
migration. Thus we can safely enable processing this event during
migration.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
b01426a238 Introduce VIR_JOB_MIGRATION_SAFE job type
This is a special job for operations that need to modify domain state
during an active migration. The modification must not affect any state
that could conflict with the migration code. This is useful mainly for
event handlers that need to be processed during migration and which
could otherwise time out on acquiring a normal MODIFY job.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
5915f93ed0 NEWS: Add support for post-copy recovery
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
ce34977c1f virsh: Add --postcopy option for domjobabort command
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
01d65a1520 qemu: Implement VIR_DOMAIN_ABORT_JOB_POSTCOPY flag
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
479d8e9f09 Add VIR_DOMAIN_ABORT_JOB_POSTCOPY flag for virDomainAbortJobFlags
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
fb50e56569 qemu: Implement virDomainAbortJobFlags
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
57762f8527 Add virDomainAbortJobFlags public API
The original virDomainAbortJob did not support flags.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:21 +02:00
Jiri Denemark
cf3842ef08 qemu: Enable support for VIR_MIGRATE_POSTCOPY_RESUME
Since all parts of post-copy recovery have been implemented now, it's
time to enable the corresponding flag.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
9189301fe5 qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for peer-to-peer migration
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
56348173fa qemu: Call qemuDomainCleanupAdd from qemuMigrationJobContinue
Every single call to qemuMigrationJobContinue needs to register a
cleanup callback in case the migrating domain dies between phases or
when migration is paused due to a failure in postcopy mode.

Let's integrate registering the callback in qemuMigrationJobContinue to
make sure the current thread does not release a migration job without
setting a cleanup callback.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
21469f6076 qemu: Register qemuProcessCleanupMigrationJob after Begin phase
The callback will properly cleanup non-p2p migration job in case the
migrating domain dies between Begin and Perform while the client which
controls the migration is not cooperating (normally the API for the next
migration phase would handle this).

The same situation can happen even after Prepare and Perform phases, but
they both already register a suitable callback, so no fix is needed
there.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
776311df23 qemu: Create completed jobData in qemuMigrationSrcComplete
Normally the structure is created once the source reports completed
migration, but with post-copy migration we can get here even after
libvirt daemon was restarted. It doesn't make sense to preserve the
structure in our status XML as we're going to rewrite almost all of it
while refreshing the stats anyway. So we just create the structure here
if it doesn't exist to make sure we can properly report statistics of a
completed migration.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
c1d40d2c40 qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Finish phase
Everything was already done in the normal Finish phase and vCPUs are
running. We just need to wait for all remaining data to be transferred.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
55e49b1b72 qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase
The QEMU process is already running, all we need to do is to call
migrate-recover QMP command. Except for some checks and cookie handling,
of course.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
c04c93ea57 qemu: Start a migration phase in qemuMigrationAnyConnectionClosed
Non-postcopy case talks to QEMU monitor and thus needs to create a
nested job. Since qemuMigrationAnyConnectionClosed is called in case
there's no thread processing a migration API, we need to make the
current thread a temporary owner of the migration job to avoid "This
thread doesn't seem to be the async job owner: 0". This is done by
starting a migration phase.

While no monitor interaction happens in postcopy case and just setting
the phase (to indicate a broken postcopy migration) would be enough,
being consistent and setting the owner does not hurt anything.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
4f22d2f883 qemu: Handle incoming migration in qemuMigrationAnyConnectionClosed
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
5eec0b4b2d qemu: Refactor qemuMigrationAnyConnectionClosed
To prepare the code for handling incoming migration too.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
14aae98212 qemu: Rename qemuMigrationSrcCleanup
The function is now called qemuMigrationAnyConnectionClosed to make it
clear it is supposed to handle broken connection during migration. It
will soon be used on both sides of migration so the "Src" part was changed
to "Any" to avoid renaming the function twice in a row.

The original *Cleanup name could easily be confused with cleanup
callbacks called when a domain is destroyed.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
8c4da941c0 qemu: Add support for migrate-recover QMP command
This command tells QEMU to start listening for an incoming post-copy
recovery connection. Just like migrate-incoming is used for starting
fresh migration on the destination host.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
8e0d86fb6a qemu: Use autoptr for mig in qemuMigrationDstPrepareFresh
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
62181e8af6 qemu: Refactor qemuMigrationDstPrepareFresh
Offline migration jumps over a big part of qemuMigrationDstPrepareFresh.
Let's move that part into a new qemuMigrationDstPrepareActive function
to make the code easier to follow.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
c7727296f8 qemu: Introduce qemuMigrationDstPrepareFresh
Moves most of the code from qemuMigrationDstPrepareAny to a new
qemuMigrationDstPrepareFresh so that qemuMigrationDstPrepareAny can be
shared with post-copy resume.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
491c9eff6c qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Confirm phase
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
f2a19b4cd0 qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Perform phase
It just calls migrate QMP command with resume=true without having to
worry about migration capabilities or parameters, storage migration,
etc. since everything has already been done in the normal Perform phase.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
07a2a6e511 qemu: Add support for 'resume' parameter of migrate QMP command
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
691ce4ea8f qemu: Separate starting migration from qemuMigrationSrcRun
qemuMigrationSrcRun does a lot of thing before and after telling QEMU to
start the migration. Let's make the core reusable by moving it to a new
qemuMigrationSrcStart function.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
d1b6bce02c qemu: Refactor qemuMigrationSrcPerformPhase
To make the code flow a bit more sensible.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
37a67122dd qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Begin phase
Mostly we just need to check whether the domain is in a failed post-copy
migration that can be resumed.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
a5e603c808 qemu: Don't set VIR_MIGRATE_PAUSED for post-copy resume
For historical reasons we automatically enabled VIR_MIGRATE_PAUSED flag
when a migration was started for a paused domain. However, when resuming
failed post-copy migration the domain on the source host will always be
paused (as it is already running on the destination host). We must avoid
enabling VIR_MIGRATE_PAUSED in this case.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
6280246971 virsh: Add --postcopy-resume option for migrate command
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
f55aedc845 Introduce VIR_MIGRATE_POSTCOPY_RESUME flag
This flag can be used to restart post-copy migration once it failed
because of a broken connection.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
b3110bb8e4 qemu: Use QEMU_MIGRATION_PHASE_POSTCOPY_FAILED
This phase marks a migration protocol as broken in a post-copy phase.
Libvirt is no longer actively watching the migration in this phase as
the migration API that started the migration failed.

This may either happen when post-copy migration really fails (QEMU
enters postcopy-paused migration state) or when the migration still
progresses between both QEMU processes, but libvirt lost control of it
because the connection between libvirt daemons (in p2p migration) or a
daemon and client (non-p2p migration) was closed. For example, when one
of the daemons was restarted.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
f1da6e9076 qemu: Do not set job owner in qemuMigrationJobSetPhase
Both qemuMigrationJobSetPhase and qemuMigrationJobStartPhase were doing
the same thing, which made little sense. Let's follow the difference
between qemuDomainObjSetJobPhase and qemuDomainObjStartJobPhase and
change job owner only in qemuMigrationJobStartPhase.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
2b163ca301 qemu: Refactor qemuDomainObjSetJobPhase
We will want to update migration phase without affecting job ownership.
Either in the thread that already owns the job or from an event handler
which only changes the phase (of a job no-one owns) without assuming it.

Let's move the ownership change to a new qemuDomainObjStartJobPhase
helper and let qemuDomainObjSetJobPhase set the phase without touching
ownership.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
69d9f54c66 qemu: Make qemuMigrationCheckPhase failure fatal
The check can reveal a serious bug in our migration code and we should
not silently ignore it.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
a5dc9bf96b qemu: Separate protocol checks from qemuMigrationJobSetPhase
Into a new qemuMigrationCheckPhase helper, which can be reused in other
places.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
6efd671a11 qemu: Add new migration phases for post-copy recovery
When recovering from a failed post-copy migration, we need to go through
all migration phases again, but don't need to repeat all the steps in
each phase. Let's create a new set of migration phases dedicated to
post-copy recovery so that we can easily distinguish between normal and
recovery code.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
ab3fd13932 qemu: Introduce qemuMigrationSrcBeginXML helper
Turn the final part of Begin phase formatting a domain XML for migration
into a reusable helper.

Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00
Jiri Denemark
3c1f37848d qemu: Check flags incompatible with offline migration earlier
Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
2022-06-07 17:40:20 +02:00