From de88bef429588b363ef76bc735232d7c923d4b04 Mon Sep 17 00:00:00 2001
From: Sebastien Boeuf <sebastien.boeuf@intel.com>
Date: Thu, 24 Sep 2020 18:05:52 +0200
Subject: [PATCH] pci: msix: Fix masking/enabling semantics

By looking at Linux kernel boot time, we identified that a lot of time
was spent registering and unregistering IRQ fds to KVM. This is not
efficient and certainly not a wrong behavior from the Linux kernel,
but rather a problem with the Cloud-Hypervisor's implementation of
MSI-X.

The way to fix this issue is by ensuring the initial conditions are
correct, which means the entire MSI-X vector table must be disabled
and masked. Additionally, each vector must be individually masked.

With these correct conditions, Linux won't start masking interrupt
vectors, and later unmask them since they will be seen as masked from
the beginning. This means the OS will simply have to unmask them when
needed, avoiding the extra operation.

Another aspect of this patch is to prevent Cloud-Hypervisor from
enabling (by registering IRQ fd) all vectors when either the global
'mask' or 'enable' bits are set. Instead, we can simply let the mask()
and unmask() operations take care of it if needed.

Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>
---
 pci/src/msix.rs | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/pci/src/msix.rs b/pci/src/msix.rs
index 4cb566a4c..5055e98b4 100644
--- a/pci/src/msix.rs
+++ b/pci/src/msix.rs
@@ -56,7 +56,7 @@ impl Default for MsixTableEntry {
             msg_addr_lo: 0,
             msg_addr_hi: 0,
             msg_data: 0,
-            vector_ctl: 0,
+            vector_ctl: 0x1,
         }
     }
 }
@@ -97,7 +97,7 @@ impl MsixConfig {
             pba_entries,
             devid,
             interrupt_source_group,
-            masked: false,
+            masked: true,
             enabled: false,
         }
     }
@@ -185,12 +185,6 @@ impl MsixConfig {
                         error!("Failed unmasking vector: {:?}", e);
                     }
                 }
-
-                if !old_enabled || old_masked {
-                    if let Err(e) = self.interrupt_source_group.enable() {
-                        error!("Failed enabling irq_fd: {:?}", e);
-                    }
-                }
             } else if old_enabled || !old_masked {
                 if let Err(e) = self.interrupt_source_group.disable() {
                     error!("Failed disabling irq_fd: {:?}", e);