<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml"> <body> <h1>Bhyve driver</h1> <ul id="toc"></ul> <p> Bhyve is a FreeBSD hypervisor. It first appeared in FreeBSD 10.0. However, it's recommended to keep tracking FreeBSD 10-STABLE to make sure all new features of bhyve are supported. In order to enable bhyve on your FreeBSD host, you'll need to load the <code>vmm</code> kernel module. Additionally, <code>if_tap</code> and <code>if_bridge</code> modules should be loaded for networking support. Also, <span class="since">since 3.2.0</span> the <code>virt-host-validate(1)</code> supports the bhyve host validation and could be used like this: </p> <pre> $ virt-host-validate bhyve BHYVE: Checking for vmm module : PASS BHYVE: Checking for if_tap module : PASS BHYVE: Checking for if_bridge module : PASS BHYVE: Checking for nmdm module : PASS $ </pre> <p> Additional information on bhyve could be obtained on <a href="http://bhyve.org/">bhyve.org</a>. </p> <h2><a id="uri">Connections to the Bhyve driver</a></h2> <p> The libvirt bhyve driver is a single-instance privileged driver. Some sample connection URIs are: </p> <pre> bhyve:///system (local access) bhyve+unix:///system (local access) bhyve+ssh://root@example.com/system (remote access, SSH tunnelled) </pre> <h2><a id="exconfig">Example guest domain XML configurations</a></h2> <h3>Example config</h3> <p> The bhyve driver in libvirt is in its early stage and under active development. So it supports only limited number of features bhyve provides. </p> <p> Note: in older libvirt versions, only a single network device and a single disk device were supported per-domain. However, <span class="since">since 1.2.6</span> the libvirt bhyve driver supports up to 31 PCI devices. </p> <p> Note: the Bhyve driver in libvirt will boot whichever device is first. If you want to install from CD, put the CD device first. If not, put the root HDD first. </p> <p> Note: Only the SATA bus is supported. Only <code>cdrom</code>- and <code>disk</code>-type disks are supported. </p> <pre> <domain type='bhyve'> <name>bhyve</name> <uuid>df3be7e7-a104-11e3-aeb0-50e5492bd3dc</uuid> <memory>219136</memory> <currentMemory>219136</currentMemory> <vcpu>1</vcpu> <os> <type>hvm</type> </os> <features> <apic/> <acpi/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <disk type='file'> <driver name='file' type='raw'/> <source file='/path/to/bhyve_freebsd.img'/> <target dev='hda' bus='sata'/> </disk> <disk type='file' device='cdrom'> <driver name='file' type='raw'/> <source file='/path/to/cdrom.iso'/> <target dev='hdc' bus='sata'/> <readonly/> </disk> <interface type='bridge'> <model type='virtio'/> <source bridge="virbr0"/> </interface> </devices> </domain> </pre> <p>(The <disk> sections may be swapped in order to install from <em>cdrom.iso</em>.)</p> <h3>Example config (Linux guest)</h3> <p> Note the addition of <bootloader>. </p> <pre> <domain type='bhyve'> <name>linux_guest</name> <uuid>df3be7e7-a104-11e3-aeb0-50e5492bd3dc</uuid> <memory>131072</memory> <currentMemory>131072</currentMemory> <vcpu>1</vcpu> <bootloader>/usr/local/sbin/grub-bhyve</bootloader> <os> <type>hvm</type> </os> <features> <apic/> <acpi/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <disk type='file' device='disk'> <driver name='file' type='raw'/> <source file='/path/to/guest_hdd.img'/> <target dev='hda' bus='sata'/> </disk> <disk type='file' device='cdrom'> <driver name='file' type='raw'/> <source file='/path/to/cdrom.iso'/> <target dev='hdc' bus='sata'/> <readonly/> </disk> <interface type='bridge'> <model type='virtio'/> <source bridge="virbr0"/> </interface> </devices> </domain> </pre> <h3>Example config (Linux UEFI guest, VNC, tablet)</h3> <p>This is an example to boot into Fedora 25 installation:</p> <pre> <domain type='bhyve'> <name>fedora_uefi_vnc_tablet</name> <memory unit='G'>4</memory> <vcpu>2</vcpu> <os> <type>hvm</type> <b><loader readonly="yes" type="pflash">/usr/local/share/uefi-firmware/BHYVE_UEFI.fd</loader></b> </os> <features> <apic/> <acpi/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <disk type='file' device='cdrom'> <driver name='file' type='raw'/> <source file='/path/to/Fedora-Workstation-Live-x86_64-25-1.3.iso'/> <target dev='hdc' bus='sata'/> <readonly/> </disk> <disk type='file' device='disk'> <driver name='file' type='raw'/> <source file='/path/to/linux_uefi.img'/> <target dev='hda' bus='sata'/> </disk> <interface type='bridge'> <model type='virtio'/> <source bridge="virbr0"/> </interface> <serial type="nmdm"> <source master="/dev/nmdm0A" slave="/dev/nmdm0B"/> </serial> <b><graphics type='vnc' port='5904'> <listen type='address' address='127.0.0.1'/> </graphics> <controller type='usb' model='nec-xhci'/> <input type='tablet' bus='usb'/></b> </devices> </domain> </pre> <p>Please refer to the <a href="#uefi">UEFI</a> section for a more detailed explanation.</p> <h2><a id="usage">Guest usage / management</a></h2> <h3><a id="console">Connecting to a guest console</a></h3> <p> Guest console connection is supported through the <code>nmdm</code> device. It could be enabled by adding the following to the domain XML (<span class="since">Since 1.2.4</span>): </p> <pre> ... <devices> <serial type="nmdm"> <source master="/dev/nmdm0A" slave="/dev/nmdm0B"/> </serial> </devices> ...</pre> <p>Make sure to load the <code>nmdm</code> kernel module if you plan to use that.</p> <p> Then <code>virsh console</code> command can be used to connect to the text console of a guest.</p> <p><b>NB:</b> Some versions of bhyve have a bug that prevents guests from booting until the console is opened by a client. This bug was fixed in FreeBSD <a href="http://svnweb.freebsd.org/changeset/base/262884">r262884</a>. If an older version is used, one either has to open a console manually with <code>virsh console</code> to let a guest boot or start a guest using:</p> <pre>start --console domname</pre> <p><b>NB:</b> A bootloader configured to require user interaction will prevent the domain from starting (and thus <code>virsh console</code> or <code>start --console</code> from functioning) until the user interacts with it manually on the VM host. Because users typically do not have access to the VM host, interactive bootloaders are unsupported by libvirt. <em>However,</em> if you happen to run into this scenario and also happen to have access to the Bhyve host machine, you may select a boot option and allow the domain to finish starting by using an alternative terminal client on the VM host to connect to the domain-configured null modem device. One example (assuming <code>/dev/nmdm0B</code> is configured as the slave end of the domain serial device) is:</p> <pre>cu -l /dev/nmdm0B</pre> <h3><a id="xmltonative">Converting from domain XML to Bhyve args</a></h3> <p> The <code>virsh domxml-to-native</code> command can preview the actual <code>bhyve</code> commands that will be executed for a given domain. It outputs two lines, the first line is a <code>bhyveload</code> command and the second is a <code>bhyve</code> command. </p> <p>Please note that the <code>virsh domxml-to-native</code> doesn't do any real actions other than printing the command, for example, it doesn't try to find a proper TAP interface and create it, like what is done when starting a domain; and always returns <code>tap0</code> for the network interface. So if you're going to run these commands manually, most likely you might want to tweak them.</p> <pre> # virsh -c "bhyve:///system" domxml-to-native --format bhyve-argv --xml /path/to/bhyve.xml /usr/sbin/bhyveload -m 214 -d /home/user/vm1.img vm1 /usr/sbin/bhyve -c 2 -m 214 -A -I -H -P -s 0:0,hostbridge -s 3:0,virtio-net,tap0,mac=52:54:00:5d:74:e3 -s 2:0,virtio-blk,/home/user/vm1.img -s 1,lpc -l com1,/dev/nmdm0A vm1 </pre> <h3><a id="zfsvolume">Using ZFS volumes</a></h3> <p>It's possible to use ZFS volumes as disk devices <span class="since">since 1.2.8</span>. An example of domain XML device entry for that will look like:</p> <pre> ... <disk type='volume' device='disk'> <source pool='zfspool' volume='vol1'/> <target dev='vdb' bus='virtio'/> </disk> ...</pre> <p>Please refer to the <a href="storage.html">Storage documentation</a> for more details on storage management.</p> <h3><a id="grubbhyve">Using grub2-bhyve or Alternative Bootloaders</a></h3> <p>It's possible to boot non-FreeBSD guests by specifying an explicit bootloader, e.g. <code>grub-bhyve(1)</code>. Arguments to the bootloader may be specified as well. If the bootloader is <code>grub-bhyve</code> and arguments are omitted, libvirt will try and infer boot ordering from user-supplied <boot order='N'> configuration in the domain. Failing that, it will boot the first disk in the domain (either <code>cdrom</code>- or <code>disk</code>-type devices). If the disk type is <code>disk</code>, it will attempt to boot from the first partition in the disk image.</p> <pre> ... <bootloader>/usr/local/sbin/grub-bhyve</bootloader> <bootloader_args>...</bootloader_args> ... </pre> <p>Caveat: <code>bootloader_args</code> does not support any quoting. Filenames, etc, must not have spaces or they will be tokenized incorrectly.</p> <h3><a id="uefi">Using UEFI bootrom, VNC, and USB tablet</a></h3> <p><span class="since">Since 3.2.0</span>, in addition to <a href="#grubbhyve">grub-bhyve</a>, non-FreeBSD guests could be also booted using an UEFI boot ROM, provided both guest OS and installed <code>bhyve(1)</code> version support UEFI. To use that, <code>loader</code> should be specified in the <code>os</code> section:</p> <pre> <domain type='bhyve'> ... <os> <type>hvm</type> <loader readonly="yes" type="pflash">/usr/local/share/uefi-firmware/BHYVE_UEFI.fd</loader> </os> ... </pre> <p>This uses the UEFI firmware provided by the <a href="https://www.freshports.org/sysutils/bhyve-firmware/">sysutils/bhyve-firmware</a> FreeBSD port.</p> <p>VNC and the tablet input device could be configured this way:</p> <pre> <domain type='bhyve'> <devices> ... <graphics type='vnc' port='5904'> <listen type='address' address='127.0.0.1'/> </graphics> <controller type='usb' model='nec-xhci'/> <input type='tablet' bus='usb'/> </devices> ... </domain> </pre> <p>This way, VNC will be accessible on <code>127.0.0.1:5904</code>.</p> <p>Please note that the tablet device requires to have a USB controller of the <code>nec-xhci</code> model. Currently, only a single controller of this type and a single tablet are supported per domain.</p> <p><span class="since">Since 3.5.0</span>, it's possible to configure how the video device is exposed to the guest using the <code>vgaconf</code> attribute:</p> <pre> <domain type='bhyve'> <devices> ... <graphics type='vnc' port='5904'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <driver vgaconf='on'/> <model type='gop' heads='1' primary='yes'/> </video> ... </devices> ... </domain> </pre> <p>If not specified, bhyve's default mode for <code>vgaconf</code> will be used. Please refer to the <a href="https://www.freebsd.org/cgi/man.cgi?query=bhyve&sektion=8&manpath=FreeBSD+12-current">bhyve(8)</a> manual page and the <a href="https://wiki.freebsd.org/bhyve">bhyve wiki</a> for more details on using the <code>vgaconf</code> option.</p> <p><span class="since">Since 3.7.0</span>, it's possible to use <code>autoport</code> to let libvirt allocate VNC port automatically (instead of explicitly specifying it with the <code>port</code> attribute):</p> <pre> <graphics type='vnc' autoport='yes'> </pre> <h3><a id="clockconfig">Clock configuration</a></h3> <p>Originally bhyve supported only localtime for RTC. Support for UTC time was introduced in <a href="http://svnweb.freebsd.org/changeset/base/284894">r284894</a> for <i>10-STABLE</i> and in <a href="http://svnweb.freebsd.org/changeset/base/279225">r279225</a> for <i>-CURRENT</i>. It's possible to use this in libvirt <span class="since">since 1.2.18</span>, just place the following to domain XML:</p> <pre> <domain type="bhyve"> ... <clock offset='utc'/> ... </domain> </pre> <p>Please note that if you run the older bhyve version that doesn't support UTC time, you'll fail to start a domain. As UTC is used as a default when you do not specify clock settings, you'll need to explicitly specify 'localtime' in this case:</p> <pre> <domain type="bhyve"> ... <clock offset='localtime'/> ... </domain> </pre> <h3><a id="e1000">e1000 NIC</a></h3> <p>As of <a href="https://svnweb.freebsd.org/changeset/base/302504">r302504</a> bhyve supports Intel e1000 network adapter emulation. It's supported in libvirt <span class="since">since 3.1.0</span> and could be used as follows:</p> <pre> ... <interface type='bridge'> <source bridge='virbr0'/> <model type='<b>e1000</b>'/> </interface> ... </pre> <h3><a id="wired">Wiring guest memory</a></h3> <p><span class="since">Since 4.4.0</span>, it's possible to specify that guest memory should be wired and cannot be swapped out as follows:</p> <pre> <domain type="bhyve"> ... <memoryBacking> <locked/> </memoryBacking> ... </domain> </pre> <h3><a id="cputopology">CPU topology</a></h3> <p><span class="since">Since 4.5.0</span>, it's possible to specify guest CPU topology, if bhyve supports that. Support for specifying guest CPU topology was added to bhyve in <a href="http://svnweb.freebsd.org/changeset/base/332298">r332298</a> for <i>-CURRENT</i>. Example:</p> <pre> <domain type="bhyve"> ... <cpu> <topology sockets='1' cores='2' threads='1'/> </cpu> ... </domain> </pre> </body> </html>