Storage Controllers in VMware vSphere

Storage Controllers

To access block-based devices (like virtual disks, CD/DVD-ROM), but also generic SCSI devices, a virtual machine uses storage controllers, which at least one is added by default when you create the virtual machine and the type of it depends by the type of guest OS that you have chosen. You can make several change changes while you are in the creation wizard.

Then, after virtual machine creation, you can add additional controllers (in some cases you can also hot-add) or change the controller type (only cold operation).

How Storage Controller Technology Works

Storage controllers appear to a virtual machine as different types of block-based controllers.

Most of them are SCSI (or SAS) based including BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS, and VMware Paravirtual SCSI.

But there are available also AHCI, SATA, and NVMe based controllers with different purposes and different usage.

Use of more storage controllers has the advantages to handle different queue for the I/O request and could be useful for specific virtual disks or virtual machines that require more performance (maybe using PVSCI or NVMe controllers).

Each virtual machine can have a maximum of two IDE controllers, four SCSI controllers, four SATA controllers, and four NVMe controllers. The default controller is number 0, for each type of controller.

When you create a virtual machine, the first hard disk is assigned to the default controller 0 at bus node (0:0).

When you add storage controllers, they are numbered sequentially 1, 2, and 3. If you add a hard disk, SCSI, or CD/DVD-ROM device to a virtual machine after virtual machine creation, the device is assigned to the first available virtual device node on the default controller, for example (0:1) in the case that you only have one OS virtual disk.

Storage controller compatibility

The type of controllers does not only depend by the virtual hardware and the guest OS, but also on how the virtual machine boot mode is configured.

Note that adding different types of storage controllers to a virtual machine that uses BIOS boot mode can cause operating system boot issues.

In the following cases, the virtual machine might fail to boot correctly and you might have to enter the BIOS setup and select the correct boot device:

• If the virtual machine boots from LSI Logic SAS or VMware Paravirtual SCSI, and you add a disk that uses BusLogic, LSI Logic, or AHCI SATA controllers.
• If the virtual machine boots from AHCI SATA, and you add BusLogic Parallel or LSI Logic controllers.

For more information see the following table:

Note that virtual machines with EFI boot mode are not affected.

Change the controller configuration

You can specify the SCSI controller type and you can set the type of SCSI bus sharing for a virtual machine.

Also, you can connect a virtual disk to another controller or another controller type.

The choice of a SCSI controller type does not affect whether your virtual disk is an IDE or SCSI disk. The IDE adapter is always ATAPI. The default for your guest operating system is already selected.

But changing the controller type after the guest operating system is installed will make the disk and any other devices connected to the adapter inaccessible.

Before you change the controller type or add a new controller, make sure that the guest operating system installation media contains the necessary drivers. On Windows guest operating systems, the driver must be installed and configured as the boot drive.

Storage Controller limitations

Storage controllers have the following requirements and limitations:

• LSI Logic SAS and VMware Paravirtual SCSI are available for virtual machines with ESXi 4.x and later compatibility.
• AHCI SATA is available only for virtual machines with ESXi 5.5 and later compatibility.
• NVMe is available only for virtual machines with ESXi 6.5 and later compatibility.
• Disks on VMware Paravirtual SCSI controllers might not experience optimal performance gains if they have snapshots or if the host’s memory is overcommitted.
• IDE does not provide disk hot add or disk space hot-add.
• BusLogic Parallel controllers are not supported with virtual disks larger than 2 TB.

The maximum number of disks depends by the controller type. Actually, we can have:

• Max 4 virtual disks on IDE controllers
• Max 60 virtual disks on SCSI/SAS controllers (up to 4 controllers)
• Max 256 virtual disks on PVSCI controllers (up to 4 controllers) new in vSphere 6.7
• Max 60 virtual disks on NVMe controllers (up to 4 controllers)
• Max 120 virtual disks and/or CDROM devices on SATA controllers (up to 4 controllers)
• Usually, a single virtual disk can have up to 62 TB of space with some limitations (see https://vinfrastructure.it/2017/02/jumbo-disk-vmware-esxi/).

SCSI emulated controller

Many virtual machines have a SCSI controller by default, depending on the guest operating system. If you have a heavily loaded virtual machine with multiple hard disks, you can add up to three additional SCSI controllers to assign the disks to. When you spread the disks among several controllers, you can improve performance and avoid data traffic congestion. You can also add additional controllers if you exceed the 15-device limit for a single controller.

If you add a SCSI controller, you can reassign an existing or new hard disk or device to that controller. For example, you can assign the device to (1:z ), where 1 is SCSI controller 1 and z is a virtual device node from 0 to 15. For SCSI controllers, z cannot be 7. By default, the virtual SCSI controller is assigned to virtual device node (z:7), so that device node is unavailable for hard disks or other devices.

SCSI paravitualized controller

VMware Paravirtual SCSI controllers are quite similar with emulated SCSI controllers (they are just a type of SCSI controllers), but they are best suited for environments, especially SAN environments, running I/O-intensive applications.

You can add a VMware Paravirtual SCSI high-performance storage controller to a virtual machine to provide greater throughput and lower CPU use, which boosts performance as compared to the other SCSI controller options.

For platform support for VMware Paravirtual SCSI controllers, see the VMware Compatibility Guide at http://www.vmware.com/resources/compatibility.

Compared with other controllers there are much more prerequisites:

• Verify that the virtual machine compatibility is ESXi 4.x and later.
• Verify that the virtual machine has a guest operating system with VMware Tools installed.
• Ensure that you are familiar with the VMware Paravirtual SCSI controller type.
• To access boot disk devices attached to a VMware Paravirtual SCSI controller, verify that the virtual machine has a Windows 2003 or Windows 2008 guest operating system.
• In some operating systems, before you change the controller type, create a virtual machine with an LSI Logic controller, install VMware Tools, and then change to paravirtual mode.

For more information on PVSCI see also KB 1010398: Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters (https://kb.vmware.com/kb/1010398).

SATA Controller

If you add a SATA controller, you can reassign an existing or new hard disk or device to that controller. For example, you can assign the device to (1:z ), where 1 is SATA controller 1 and z is a virtual device node from 0 to 29. For SATA controllers, you can use device nodes 0 through 29, including 0:7.

You can boot virtual machines from SATA controllers and use them for large-capacity virtual hard disks.

Not all guest OSes support AHCI SATA controllers, but most of them, including Windows Vista and later version of Windows have a default SATA controller for CD/DVD-ROM devices. To verify the support, see the VMware Compatibility Guides at http://www.vmware.com/resources/compatibility.

NVMe controller

Starting with vSphere 6.5, alternatively, each virtual machine can have a maximum of four NVMe controllers.

NVM Express (NVMe) or Non-Volatile Memory Host Controller Interface Specification (NVMHCI) is a new logical device interface specification for accessing non-volatile memory (NVE) attached, in a physical system, via a PCI Express (PCIe) bus instead of using SAS bus. Supported form factors include add-in PCIe cards, M.2 and U.2 interfaces. NVE is still flash memory, but now with a stack designed to reach the best performance.

NVM Express reduces I/O overhead and brings various performance improvements in comparison to previous logical-device interfaces, including multiple, long command queues, and reduced latency.

Using an NVMe virtual controller significantly reduces the software overhead for processing guest OS I/O, as compared to AHCI SATA or SCSI controllers.

You can reassign an existing or new hard disk or device to that controller.

If a virtual machine has multiple hard disks, you can add up to four virtual NVMe controllers to which to assign the disks.

For example, you can assign the hard disk to (x:z ), where x is NVMe controller and z is a virtual device node. x has values from 0 to 3, and z has values from 0 to 14.

NVMe controllers perform best with virtual disks on an all-flash disk array, local NVMe SSD, and PMem storage.

Compared with other controllers there are much more prerequisites:

• Verify that the virtual machine compatibility is ESXi 6.5 or later.
• Verify that the virtual machine has a guest operating system that supports NVMe.
• Verify that you are familiar with storage controller behavior and limitations. See SCSI and SATA Storage Controller Conditions, Limitations, and Compatibility.

For more information on NVMe see also KB 2147714: Using Virtual NVMe With ESXi 6.5 and virtual machine Hardware Version 13 (https://kb.vmware.com/kb/2147714).

Which controller?

When you create a virtual machine, usually the first controller (Controller 0) is optimized for best performance for the OS boot disk.

The controller type depends on the guest operating system, the device type, and sometimes, the virtual machine’s compatibility. The reason is almost historical and related with the default drivers included in the guest OS. For example, for Windows Server 2003 the default controller is SCSI LSI Logic Parallel (with SCSI2 commands), but starting with Windows Server 2008 the new default controller become the LSI Logic SAS (capable also to some SCSI3 commands). A SATA controller is usually the default for the CD/DVD drive.

There are different blog posts or documents that compare the different type of controllers for the performance improvements, for example, see:

• https://www.virtualizationhowto.com/2017/07/vmware-lsi-sas-vs-pvscsi-vs-nvme-controller-performance/
• https://blogs.vmware.com/vsphere/2014/02/vscsi-controller-choose-performance.html
• https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/1M-iops-perf-vsphere5.pdf

For the different virtual storage controllers this table recap the different types and possible use case and the required virtual hardware:

Note that using guest clustering may require specific controller depending by the OS and the vSphere version (for example for Microsoft Fail-Over cluster with old vSphere version the PVSCI controller was not supported).

But for very high throughput there can be also another option, starting with vSphere 6.7: the Persistent Memory (PM).

Some applications, like huge database are moving their data from disk to RAM using the in-memory paradigm. But, of course, traditional RAM is not persistent, so at hardware level you need new type of DIMMs based on flash chip that works like RAM DIMMs, but provide data persistency.

Related materials:

• Why upgrade to VMware vSphere 6.5 (or why not)
• VMware vSphere: Handy tips on working with Content Libraries