VMware vSphere USB support

USB standards

Universal Serial Bus (USB) is an industry-standard that establishes specifications for cables and connectors and protocols for connection, communication, also and power supply between computers, peripheral devices and other computers.

Basically, it is used to connect external devices to a PC and has become very popular for keyboard, mice, external disks, USB keys, printers, …. but is becoming widely used also for video and other types of devices.

USB Logo

It’s one of the oldest PC standard (the first release was in 1996), still actively used today. The USB standard is currently maintained by the USB Implementers Forum (USB-IF).

There are four main generations of USB standards:

USB 1.x with 1.5 Mbit/s (Low Speed) or 12 Mbit/s (Full Speed)
USB 2.x with 480 Mbit/s (High Speed)
USB 3.x with 5 Gbit/s (SuperSpeed) or 10/20 Gbit/s (SuperSpeed+ in USB 3.1 and 3.2)
USB 4.x (not yet available) with 40 Gbit/s (SuperSpeed+ and Thunderbolt 3)

Each standard has different connectors with several “mini” or “micro” connector, but finally, the “type-C” (introduced with USB 3.0) has become a common standard (for example it’s used also on the iPad device and most of the Android’s phone).

USB type-C connector

Like often happens not necessary wins the best standard (think about VHS for home video, in the past), and USB was a good example: there were other interesting alternatives like Firewire (at the beginning) or Thunderbolt (pushed by Intel first and then Apple)… but USB has become the most used standards (just look at the connector for main of the smartphones, or almost all the tablets…. or also the standard connectors in all the PC and laptops).

And with latest standards USB has become fast enough for most use cases!

USB on servers

USB has become also popular on servers: not only the keyboard and the mouse are all USB (replacing the very old and formally not pluggable PS/2 connectors), but also for other use cases.

For example, the dual boot SD card on most of the servers, is just an internal SD reader connected to an internal USB port (or other port, connected to the USB bus)!

And USB is standing to replace also the serial connector, maybe not yet on servers, but more and more on switches or enterprise storage (in those cases, the micro-USB port provide, with a proper driver, an emulation of a serial connection).

For entry-level servers, the USB bus is widely used also for storage, connecting external disk devices to expand the space or to implement simply backup solutions.

Of course, does not replace, yet, all the storage part and never will be the scope.

To access block-based devices (like virtual disks, CD/DVD-ROM), but also SATA/SAS devices, the SATA and SAS buses will be still there… but with the increase of NVMe devices, those buses can become quickly old or, at least, less used.

USB support in ESXi

VMware ESXi supports USB for different use cases:

USB boot devices to boot from some kind of USB block device (usually an SD card reader)
USB pass-through to map USB devices directly to the proper VM
USB disks directly from the ESXi, but with several limitations

Let’s discuss each different case by providing more details.

For a list of supported USB devices, see the VMware Compatibility Guide at http://www.vmware.com/resources/compatibility, but basically, almost all of the USB chipset on server’s motherboards are supported.

ESXi USB boot

Installing ESXi on a USB flash drive or SD flash card (connected to the USB bus) has been supported since ESXi 3.5 and has the same minimum requirements as installing to typical SCSI or SATA hard disk.

You can use a USB disk, a USB key, or an SD card reader connected to a USB port.

The device will recognize still in a standard way, from ESXi point of view: like a disk connected to a (USB) storage controller.

About the sizing, potentially you can install ESXi (also the 6.7 version) on a USB device with just 1 GB of space, but VMware recommends using a retail purchased USB flash drive of 16 GB or larger so that the “extra” flash cells can prolong the life of the boot media but high-quality parts of 4 GB or larger can be sufficient to hold the extended coredump partition or potentially other type of partitions used for write oeprations.

Using a USB device as a boot device in ESXi has become a very common scenario, thanks to the redundant SD card options available in several servers. It’s not only an affordable solution and reliable, but it’s usually very cheap. In lab environments also the USB key option is widely used, but can also be an interesting choice to test a specific version of ESXi on a server, without touching the local installation.

But, of course, if you choose a USB device as a boot option for ESXi, there are also drawbacks that you have to consider.

For example, some partitions are not created at all on a USB device: the scratch partition and the VMFS datastore. For more information on ESXi partitions see this blog post: VMware ESXi 6.0 partitions.

The reason is to increase the durability of the USB devices, that maybe not designed for enterprise storage usage (like most of the SD cards or USB keys). For all the other partitions there are no issues, because they are loaded on a ramdisk and used from there.

But for the scratch and the VMFS partitions, the I/O operations are synchronous and directly on the storage partition.

So, due to the I/O sensitivity of USB and SD devices the installer does not create a scratch partition on these devices.

When installing on USB or SD devices, the installer attempts to allocate a scratch region on an available local disk or datastore. If no local disk or datastore is found, /scratch is placed on the ramdisk. After the installation, you should reconfigure /scratch to use a persistent datastore. For more information, see the KB 1033696 (Creating a persistent scratch location for ESXi 4.x and 5.x).

For the VMFS datastore, the reason is the same as the scratch partition, and VMware does not officially support VMFS datastore on USB device. This case will be discussed later.

But what about hyper-converged systems where you need a guest VM for the local storage controller, or any way you just need local space for logs and other “scratch” related parts, but you still want a “disk-less” server? One interesting solution is using internal cards like the Dell Boot Optimized Storage Solution (BOSS) card (other vendors, like Lenovo, have similar solutions).

Note that in most cases, the SD cards (or also the NVMe devices, like in BOSS card) are not hot-swappable, because are mounted internally, but also because the RAID-1 mechanism to provide storage redundancy, usually can be managed only from the server BIOS.

For more information on how to install ESXi on a USB device, see also VMware KB 2004784 (Installing ESXi on a supported USB flash drive or SD flash card).

There is also a special case for USB boot in ESXi and is when you want to start the installation process from a bootable USB device!

This is a little different compared to the case of booting from a USB device, because you have also to manually prepare your device to make it bootable! Tools like Rufus are your best friend in this case.

USB pass-through connected to a VM

Usually, when you attach a USB device to a physical host, the device is available only to the ESXi host and not necessary is supported (for example a USB scanner or printer).

But with the USB pass-through feature (introduced in vSphere 4.1) you can attach a USB device to a virtual machine, that must reside on that host. Normally, the device cannot connect to virtual machines that reside on another host in the datacenter. But you can migrate a virtual machine to another ESXi host in the same datacenter and maintain the USB passthrough device connections to the original host (until the original host remain alive). For more information see this manual page.

Also, the USB device is available to only one virtual machine at a time due to the exclusive access that you can have to a specific device. When a device is connected to a powered-on virtual machine, it is not available to connect to other virtual machines that run on the same host. When you remove the active connection of a USB device from a virtual machine, it becomes available to connect to other virtual machines that run on the same host.

The concept of USB pass-through is quite similar to the concept of PCI pass-through (or VMDirectPath), but with fewer restrictions, limitations and requirements. The vSphere features available for USB pass-through is bigger:

For more information see the manual page.

Connecting a USB pass-through device to a virtual machine that runs on the ESXi host to which the device is physically attached requires:

virtual hardware version 7 or higher (higher is required for USB 3.x support)
an arbitrator service running on the ESXi host (a user-space service)
a USB controller
and a physical USB device or device hub

The virtual USB Controller is the software virtualization of the USB host controller function in the virtual machine.

You can add a USB Controller to a VM (also if it is powered on) using the UI, in the VM settings:

There are also now two types of USB controller available for virtual machines to utilize:

EHCI+UHCI controller: This is the classic USB controller allowing the usage of USB 2.0 and USB 1.1 devices.
xHCI controller: This is the newer controller type allowing the usage of faster USB 3.0 devices. Virtual hardware version 8 is required to use this controller.

If you set the USB 3.0 controller, it will be renamed in “USB xHCI controller”.

The USB Controller has these limitations:

A minimum of virtual hardware version 7 is required.
You must add a USB controller to a virtual machine before you can add a USB device.
You must remove all USB devices from a virtual machine before you can remove the controller.
Only one USB controller of each type can be added to a virtual machine.
The USB arbitrator can monitor a maximum of 15 USB controllers. If your system includes more than 15 controllers and you connect USB devices to these controllers, the devices are not available to be passed through to a virtual machine.

Also is better to install VMware Tools to provide better support, although the device drivers are usually native guest OS drivers.

Adding the USB device is very easy, again you can do from the UI in the VM Settings:

You can select the device and choose if you need the vMotion support as described before.

You can add up to 20 USB devices to a virtual machine. This is the maximum number of devices supported for simultaneous connection to one virtual machine. The maximum number of USB devices supported on a single ESXi host for simultaneous connection to one or more virtual machines is also 20.

USB devices have these limitations:

A virtual machine may have up to 20 USB devices attached to it, however each unique USB device can only be attached to one virtual machine at a time.
Before you can hot-add memory, CPU, or PCI devices, you must remove any USB devices. Hot adding these resources disconnects USB devices, which may result in data loss.
When you suspend and resume a virtual machine, USB devices behave as if they have been disconnected, then reconnected.
The virtual machine boot from the attached USB device is not supported (although it might work has written in this blog post).
You cannot map HID devices, like mouse or keyboard, because those devices are directly connected to the ESXi host.
Unsupported USB devices may not interact as expected with other ESXi/ESX features.

Note that the USB arbitrator server is an ESXi service that is not visible from the UI and you cannot stop or disable from there, but only from the command line.

For a list of supported USB devices, see the VMware KB 1021345 (Supported USB device models for passthrough from an ESX or ESXi host to a virtual machine).

For more information on USB pass-through support see also VMware KB 1022290 (USB support in ESXi/ESX 4.1, ESXi 5.x, and ESXi 6.x).

USB disks connected to ESXi

Also if ESXi can boot from USB disks, usually it cannot access any USB connected disks. The reason is the USB arbitrator service used for the USB pass-through feature described previously.

But what’s happen if you disable this service?

To stop the USB arbitrator service you can use this command (from SSH or directly on the ESXi Shell):

~ # /etc/init.d/usbarbitrator stop

1	~ # /etc/init.d/usbarbitrator stop

Note that if you disable the USB arbitrator service, you can no longer passthrough USB devices to VMs.

Now if you add a USB key or disk formatted with Microsoft FAT32 or exFAT (or also Linux ext3) you can see that this disk is automatically mounted:

The used label is the one on the filesystem, in this case, there is no label and the mount point used is just “NO NAME”.

Now you can access the disk with the usual command, but you cannot run a VM from there because is not a VMFS filesystem.

VMware only supports the vfat filesystem and could be useful to pass a network device driver, for example.

But you can thread this disk as another disk and just partition and format as a VMFS datastore! Of course, it’s not supported and you have to do from the command line.

There is a good guide available on https://www.virten.net/2016/11/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-5/

Note that there is a new ESXi kernel driver for USB introduced in ESXi 6.5 that has changed some of the behavior and some USB devices are no more recognized. For more information see also the VMware KB 2147650 (Important information about the new ESXi 6.5 USB driver vmkusb, and the legacy USB drivers).

USB support in VMware vSphere