Commit Graph

163 Commits

Author SHA1 Message Date
Sebastien Boeuf
e155e3690c vm-virtio: Simplify virtio-fs configuration
This commit introduces a clear definition of the virtio-fs
configuration structure, allowing vhost-user-fs device to
rely on it.

This makes the code more readable for developers.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-28 10:28:14 +00:00
Sebastien Boeuf
8e48fc445f vm-virtio: Simplify virtio-blk configuration
This commit reuses the clear definition of the virtio-blk
configuration structure, allowing both vhost-user-blk and
virtio-blk devices to rely on it.

This makes the code more readable for developers.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-28 10:28:14 +00:00
Sebastien Boeuf
8946a09afd vm-virtio: Simplify virtio-net configuration
This commit introduces a clear definition of the virtio-net
configuration structure, allowing both vhost-user-net and
virtio-net devices to rely on it.

This makes the code more readable for developers.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-28 10:28:14 +00:00
Sebastien Boeuf
f5b53ae4be vm-virtio: Implement multiqueue/multithread support for virtio-blk
This commit improves the existing virtio-blk implementation, allowing
for better I/O performance. The cost for the end user is to accept
allocating more vCPUs to the virtual machine, so that multiple I/O
threads can run in parallel.

One thing to notice, the amount of vCPUs must be egal or superior to the
amount of queues dedicated to the virtio-blk device.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-28 09:26:53 +01:00
Samuel Ortiz
c4b3ed7223 vm-virtio: Further factorization
The trait bound and non trait bound virtio devices can use the same
inner implementation.
Also, the virtio pausable trait definiton can also be factorized.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-28 07:51:13 +01:00
Samuel Ortiz
bce76271c5 vm-virtio: Define a separate macro alias for ctrl queue devices
Now that we have factorized the common virtio pausable implementation,
it's cleaner to have a dedicated macro for control queue devices rather
than overload the macro prototype.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-28 07:51:13 +01:00
Samuel Ortiz
2e2b1e4230 vm-virtio: Remove the multiqueue argument from the pausable macro
We only need the ctrl queue one.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-28 07:51:13 +01:00
Samuel Ortiz
2cb7ec04a4 vm-virtio: Pausable macro factorization improvements
By adding an internal layer of abstraction (the hidden VirtioPausable
trait), we can factorize the virtio common code.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-28 07:51:13 +01:00
Samuel Ortiz
c06a827cbb vm-virtio: Rename epoll_thread to epoll_threads
Now that we unified epoll_thread to potentially be a vector of threads,
it makes sense to make it a plural field.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-28 07:51:13 +01:00
Samuel Ortiz
f648f2856d vm-virtio: Make all virtio devices potentially multi-threaded
Although only the block and net virtio devices can actually be multi
threaded (for now), handling them as special cases makes the code more
complex.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-28 07:51:13 +01:00
Sebastien Boeuf
0a7bcc9a7d vm-virtio: Fix map_err losing the inner error
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-24 12:42:09 +01:00
Sebastien Boeuf
9ac06bf613 ci: Run clippy for each specific feature
The build is run against "--all-features", "pci,acpi", "pci" and "mmio"
separately. The clippy validation must be run against the same set of
features in order to validate the code is correct.

Because of these new checks, this commit includes multiple fixes
related to the errors generated when manually running the checks.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-21 11:44:40 +01:00
Sebastien Boeuf
99f39291fd pci: Simplify PciDevice trait
There's no need for assign_irq() or assign_msix() functions from the
PciDevice trait, as we can see it's never used anywhere in the codebase.
That's why it's better to remove these methods from the trait, and
slightly adapt the existing code.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-21 10:44:48 +01:00
Sebastien Boeuf
8d7c4ea334 vmm: Use LegacyUserspaceInterruptGroup for mmio devices
This commit replaces the way legacy interrupts were handled with the
brand new implementation of the legacy InterruptSourceGroup for KVM.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-21 10:44:48 +01:00
Sebastien Boeuf
8049666eff vm-virtio: Cleanup from kvm_iotcls and kvm_bindings dependencies
Now that KVM specific interrupts are handled through InterruptManager
trait implementation, the vm-virtio crate does not need to rely on
kvm_ioctls and kvm_bindings crates.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
4bb12a2d8d interrupt: Reorganize all interrupt management with InterruptManager
Based on all the previous changes, we can at this point replace the
entire interrupt management with the implementation of InterruptManager
and InterruptSourceGroup traits.

By using KvmInterruptManager from the DeviceManager, we can provide both
VirtioPciDevice and VfioPciDevice a way to pick the kind of
InterruptSourceGroup they want to create. Because they choose the type
of interrupt to be MSI/MSI-X, they will be given a MsiInterruptGroup.

Both MsixConfig and MsiConfig are responsible for the update of the GSI
routes, which is why, by passing the MsiInterruptGroup to them, they can
still perform the GSI route management without knowing implementation
details. That's where the InterruptSourceGroup is powerful, as it
provides a generic way to manage interrupt, no matter the type of
interrupt and no matter which hypervisor might be in use.

Once the full replacement has been achieved, both SystemAllocator and
KVM specific dependencies can be removed.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
be421dccea vm-virtio: Optimize vhost-user interrupt notification
Thanks to the recently introduced function notifier() in the
VirtioInterrupt trait, all vhost-user devices can now bypass
listening onto an intermediate event fd as they can provide the
actual fd responsible for triggering the interrupt directly to
the vhost-user backend.

In case the notifier does not provide the event fd, the code falls
back onto the creation of an intermediate event fd it needs to listen
to, so that it can trigger the interrupt on behalf of the backend.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
1f029dd2dc vm-virtio: Add notifier to VirtioInterrupt trait
The point is to be able to retrieve directly the event fd related to
the interrupt, as this might optimize the way VirtioDevice devices are
implemented.

For instance, this can be used by vhost-user devices to provide
vhost-user backends directly with the event fd triggering the
interrupt related to a virtqueue.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
c396baca46 vm-virtio: Modify VirtioInterrupt callback into a trait
Callbacks are not the most Rust idiomatic way of programming. The right
way is to use a Trait to provide multiple implementation of the same
interface.

Additionally, a Trait will allow for multiple functions to be defined
while using callbacks means that a new callback must be introduced for
each new function we want to add.

For these two reasons, the current commit modifies the existing
VirtioInterrupt callback into a Trait of the same name.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
19aeac40c9 msix: Remove the need for interrupt callback
Now that MsixConfig has access to the irq_fd descriptors associated with
each vector, it can directly write to it anytime it needs to trigger an
interrupt.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
2381f32ae0 msix: Add gsi_msi_routes to MsixConfig
Because MsixConfig will be responsible for updating KVM GSI routes at
some point, it is necessary that it can access the list of routes
contained by gsi_msi_routes.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
9b60fcdc39 msix: Add VmFd to MsixConfig
Because MsixConfig will be responsible for updating the KVM GSI routes
at some point, it must have access to the VmFd to invoke the KVM ioctl
KVM_SET_GSI_ROUTING.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sebastien Boeuf
86c760a0d9 msix: Add SystemAllocator to MsixConfig
The point here is to let MsixConfig take care of the GSI allocation,
which means the SystemAllocator must be passed from the vmm crate all
the way down to the pci crate.

Once this is done, the GSI allocation and irq_fd creation is performed
by MsixConfig directly.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-17 23:43:45 +01:00
Sergio Lopez
c5a656c9dc vm-virtio: block: Add support for alignment restrictions
Doing I/O on an image opened with O_DIRECT requires to adhere to
certain restrictions, requiring the following elements to be aligned:

 - Address of the source/destination memory buffer.
 - File offset.
 - Length of the data to be read/written.

The actual alignment value depends on various elements, and according
to open(2) "(...) there is currently no filesystem-independent
interface for an application to discover these restrictions (...)".

To discover such value, we iterate through a list of alignments
(currently, 512 and 4096) calling pread() with each one and checking
if the operation succeeded.

We also extend RawFile so it can be used as a backend for QcowFile,
so the later can be easily adapted to support O_DIRECT too.

Signed-off-by: Sergio Lopez <slp@redhat.com>
2020-01-17 17:28:44 +00:00
Cathy Zhang
652e7b9b8a vm-virtio: Implement multiple queue support for net devices
Update the common part in net_util.rs under vm-virtio to add mq
support, meanwhile enable mq for virtio-net device, vhost-user-net
device and vhost-user-net backend. Multiple threads will be created,
one thread will be responsible to handle one queue pair separately.
To gain the better performance, it requires to have the same amount
of vcpus as queue pair numbers defined for the net device, due to
the cpu affinity.

Multiple thread support is not added for vhost-user-net backend
currently, it will be added in future.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
4ab88a8173 net_util: Add multiple queue support for tap
Add support to allow VMMs to open the same tap device many times, it will
create multiple file descriptors meanwhile.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
cf7e0cdf95 vm-virtio: Add multiple queue handling with control queue
Current guest kernel will check the oneline cpu count, in principle,
if the online cpu count is not smaller than the number of queue pairs
VMM reported, the net packets could be put/get to all the virtqueues,
otherwise, only the number of queue pairs that match the oneline cpu
count will have packets work with. guest kernel will send command
through control queue to tell VMMs the actual queue pair numbers which
it could currently play with. Add mq process in control queue handling
to get the queue pair number, VMM will verify if it is in a valid range,
nothing else but this.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
709f7fe607 vm-virtio: Implement control queue support for net devices
While feature VIRTIO_NET_F_CTRL_VQ is negotiated, control queue
will exits besides the Tx/Rx virtqueues, an epoll handler should
be started to monitor and handle the control queue event.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
d38787c578 vm-virtio: Add control queue support in net_util.rs
As virtio spec 1.1 said, the driver uses the control queue
to send commands to manipulate various features of the devices,
such as VIRTIO_NET_F_MQ which is required by multiple queue
support. Here add the control queue handling process.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
1ae7deb393 vm-virtio: Implement refactor for net devices and backend
Since the common parts are put into net_util.rs under vm-virtio,
refactoring code for virtio-net device, vhost-user-net device
and backend to shrink the code size and improve readability
meanwhile.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
6ae2597d19 vm-virtio: Create new module to abstract common parts for net devices
There are some common logic shared among virtio-net device, vhost-user-net
device and vhost-user-net backend, abstract those parts into net_util.rs
to improve code maintainability and readability.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Cathy Zhang
3485e89080 vm-virtio: Stop delivering interrupt while NO_VECTOR
According to virtio spec, for used buffer notifications, if
MSI-X capability is enabled, and queue msix vector is
VIRTIO_MSI_NO_VECTOR 0xffff, the device must not deliver an
interrupt for that virtqueue.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
2020-01-17 12:06:19 +01:00
Samuel Ortiz
c2f6dfce88 vm-virtio: Fix VirtioDeviceType traits
The From and Display traits were not handling some of the enum
definitions. We no longer have a default case for Display so any future
misses will fail at build time.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2020-01-09 15:29:09 +01:00
Sebastien Boeuf
38468d3d9b vm-virtio: Improve virtio-console input processing
The way the code is currently implemented, only by writing to STDIN a
user can trigger some input to reach the VM through virtio-console. But
in case, there were not enough virtio descriptors to process what was
retrieved from STDIN, the remaining bits would be transferred only if
STDIN was triggered again. The missing part is that when some
descriptors are made available from the guest, the virtio-console device
should try to send any possible remaining bits.

By triggering the function process_input_queue() whenever the guest
notifies the host that some new descriptors are ready for the receive
queue, this patch allows to fill the implementation void that was left.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-08 15:37:02 +01:00
Sebastien Boeuf
e4c3401a33 vm-virtio: Don't waste a descriptor if not filled
In case the virtio descriptor is pulled out of the Queue iterator, it
is important to fill it and tag it as used. This is already done from
the successful code path, but in case there's an error during the
filling, we should make sure to put the descriptor back in the list of
available descriptors. This way, when the error occurs, we don't loose
a descriptor, and it could be used later.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-08 15:37:02 +01:00
Sebastien Boeuf
7a3e6caca4 vm-virtio: Simplify virtio-console input processing
The existing code was a bit too complex and it was introducing a bug
when trying to paste long lines directly to the console. By simplifying
the code, and by doing proper usage of the drain() function, the bug is
fixed by this commit.

Here is the similar output one could have gotten from time to time, when
pasting important amounts of bytes:

ERROR:vm-virtio/src/console.rs:104 -- Failed to write slice:
InvalidGuestAddress(GuestAddress(1040617472))

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-08 15:37:02 +01:00
Sebastien Boeuf
84445aae93 vm-virtio: Implement multi-mapping for virtio-fs
The virtio-fs messages coming from the slave can contain multiple
mappings (up to 8) through one single request. By implementing such
feature, the virtio-fs implementation of cloud-hypervisor is optimal and
fully functional as it resolves a bug that was seen when running fio
testing without this patch.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-08 09:27:07 +01:00
Sebastien Boeuf
e1822cfdad vm-virtio: Implement VIRTIO_IOMMU_F_PROBE feature
By implementing this virtio feature, we let the virtio-iommu driver call
the device backend so that it can probe each device that gets attached.

Through this probing, the device provides a range of reserved memory
related to MSI. This is mandatory for x86 architecture as we want to
avoid the default MSI range assigned by the virtio-iommu driver if no
range is provided at all. The default range is 0x8000000-0x80FFFFF but
it only makes sense for ARM architectures.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2020-01-08 09:27:07 +01:00
Rob Bradford
32a39f9b95 vm-virtio: Fix broken write_base_regs() unit test
The following commit broke this unit test:

"""
vmm: Convert virtio devices to Arc<Mutex<T>>

Migratable devices can be virtio or legacy devices.
In any case, they can potentially be tracked through one of the IO bus
as an Arc<Mutex<dyn BusDevice>>. In order for the DeviceManager to also
keep track of such devices as Migratable trait objects, they must be
shared as mutable atomic references, i.e. Arc<Mutex<T>>. That forces all
Migratable objects to be tracked as Arc<Mutex<dyn Migratable>>.

Virtio devices are typically migratable, and thus for them to be
referenced by the DeviceManager, they now should be built as
Arc<Mutex<VirtioDevice>>.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
"""

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-01-03 10:20:53 +01:00
Rob Bradford
b2589d4f3f vm-virtio, vmm, vfio: Store GuestMemoryMmap in an Arc<ArcSwap<T>>
This allows us to change the memory map that is being used by the
devices via an atomic swap (by replacing the map with another one). The
ArcSwap provides the mechanism for atomically swapping from to another
whilst still giving good read performace. It is inside an Arc so that we
can use a single ArcSwap for all users.

Not covered by this change is replacing the GuestMemoryMmap itself.

This change also removes some vertical whitespace from use blocks in the
files that this commit also changed. Vertical whitespace was being used
inconsistently and broke rustfmt's behaviour of ordering the imports as
it would only do it within the block.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2020-01-02 13:20:11 +00:00
Rob Bradford
9fb1c46cd1 vm-virtio: Remove unnecessary cloning
Found by updated clippy:

error: redundant clone
   --> vm-virtio/src/block.rs:182:5
    |
182 |     .to_owned();
    |     ^^^^^^^^^^^ help: remove this
    |
    = note: `-D clippy::redundant-clone` implied by `-D warnings`
note: this value is dropped without further use
   --> vm-virtio/src/block.rs:176:21
    |
176 |       let device_id = format!(
    |  _____________________^
177 | |         "{}{}{}",
178 | |         blk_metadata.st_dev(),
179 | |         blk_metadata.st_rdev(),
180 | |         blk_metadata.st_ino()
181 | |     )
182 | |     .to_owned();
    | |____^
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#redundant_clone

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2019-12-20 00:52:03 +01:00
Sebastien Boeuf
9701fde209 vm-virtio: Add connection handshake to vsock
This patch has been cherry-picked from the Firecracker tree. The
reference commit is 1db04ccc69862f30b7814f30024d112d1b86b80e.

Changed the host-initiated vsock connection protocol to include a
trivial handshake.

The new protocol looks like this:
- [host] CONNECT <port><LF>
- [guest/success] OK <assigned_host_port><LF>

On connection failure, the host host connection is reset without any
accompanying message, as before.

This allows host software to more easily detect connection failures, for
instance when attempting to connect to a guest server that may have not
yet started listening for client connections.

Signed-off-by: Dan Horobeanu <dhr@amazon.com>
Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2019-12-19 09:48:17 +01:00
Samuel Ortiz
664431ff14 vsock: vhost_user: vfio: Fix potential host memory overflow
The vsock packets that we're building are resolving guest addresses to
host ones and use the latter as raw pointers.
If the corresponding guest mapped buffer spans across several regions in
the guest, they will do so in the host as well. Since we have no
guarantees that host regions are contiguous, it may lead the VMM into
trying to access memory outside of its memory space.

For now we fix that by ensuring that the guest buffers do not span
across several regions. If they do, we error out.
Ideally, we should enhance the rust-vmm memory model to support safe
acces across host regions.

Fixes CVE-2019-18960

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2019-12-12 22:15:50 +01:00
Samuel Ortiz
a122da4bef vm-virtio: vhost: Implement the Pausable trait for all vhost-user devices
Due to the amount of code currently duplicated across vhost-user devices,
the stats for this commit is on the large side but it's mostly more
duplicated code, unfortunately.

Migratable and Snapshotable placeholder implementations are provided as
well, making all vhost-user devices Migratable.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2019-12-12 08:50:36 +01:00
Samuel Ortiz
dae0b2ef72 vm-virtio: Implement the Pausable trait for all virtio devices
Due to the amount of code currently duplicated across virtio devices,
the stats for this commit is on the large side but it's mostly more
duplicated code, unfortunately.

Migratable and Snapshotable placeholder implementations are provided as
well, making all virtio devices Migratable.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2019-12-12 08:50:36 +01:00
Samuel Ortiz
35d7721683 vmm: Convert virtio devices to Arc<Mutex<T>>
Migratable devices can be virtio or legacy devices.
In any case, they can potentially be tracked through one of the IO bus
as an Arc<Mutex<dyn BusDevice>>. In order for the DeviceManager to also
keep track of such devices as Migratable trait objects, they must be
shared as mutable atomic references, i.e. Arc<Mutex<T>>. That forces all
Migratable objects to be tracked as Arc<Mutex<dyn Migratable>>.

Virtio devices are typically migratable, and thus for them to be
referenced by the DeviceManager, they now should be built as
Arc<Mutex<VirtioDevice>>.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2019-12-12 08:50:36 +01:00
Rob Bradford
c61104df47 vmm: Port to latest vmm-sys-util
The signal handling for vCPU signals has changed in the latest release
so switch to the new API.

Signed-off-by: Rob Bradford <robert.bradford@intel.com>
2019-12-11 14:11:11 +00:00
Samuel Ortiz
0f21781fbe cargo: Bump the kvm and vmm-sys-util crates
Since the kvm crates now depend on vmm-sys-util, the bump must be
atomic.
The kvm-bindings and ioctls 0.2.0 and 0.4.0 crates come with a few API
changes, one of them being the use of a kvm_ioctls specific error type.
Porting our code to that type makes for a fairly large diff stat.

Signed-off-by: Samuel Ortiz <sameo@linux.intel.com>
2019-11-29 17:48:02 +00:00
Sebastien Boeuf
8845326aa2 vm-virtio: Introduce DescriptorChain iterator
In order to iterate over a chain of descriptor chains, this code has
been ported over from crosvm, based on the commit
961461350c0b6824e5f20655031bf6c6bf6b7c30.

The main modification compared to the original code is the way the
sorting between readable and writable descriptors happens.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
2019-11-22 22:17:47 +01:00
Jose Carlos Venegas Munoz
7498647e3f cargo: Update micro_http
Update micro_http create to allow set content type.

Suggested-by:  Samuel Ortiz <sameo@linux.intel.com>
Tested-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
2019-11-08 22:49:08 +01:00