virtio-net: net: Wait for threads to exit on Drop

It is required to close all file descriptors pointing to an opened TAP
device prior to reopening the TAP device; otherwise it will return
-EBUSY as the device can only be opened once (excluding MQ use cases.)

When rebooting the VM the virtio-net threads would still be running and
so the TAP file descriptor may not have been closed. To ensure that the
TAP FD is closed wait for all the epoll threads to exit after receiving the
KILL_EVENT.

Fixes: #4868

Signed-off-by: Rob Bradford <robert.bradford@intel.com>

virtio-devices/src/device.rs

@@ -282,7 +282,6 @@ impl VirtioCommon {
     }
 
     // Wait for the worker thread to finish and return
-    #[cfg(fuzzing)]
     pub fn wait_for_epoll_threads(&mut self) {
         if let Some(mut threads) = self.epoll_threads.take() {
             for t in threads.drain(..) {

virtio-devices/src/net.rs

@@ -631,6 +631,8 @@ impl Drop for Net {
             // Ignore the result because there is nothing we can do about it.
             let _ = kill_evt.write(1);
         }
+        // Needed to ensure all references to tap FDs are dropped (#4868)
+        self.common.wait_for_epoll_threads();
     }
 }