VMware vSphere has long established itself as one of the leading virtualization platforms, providing powerful tools for managing virtual machines (VMs) and data center infrastructure (DCs). In VMware vSphere 8 Update 3, an important new feature was introduced – Live Patch. This innovation allows administrators to make critical fixes and security updates to the ESXi hypervisor kernel without having to reboot the system or turn off virtual machines. In this article, we will consider what Live Patching is, its benefits, technical aspects of operation and potential limitations.
By the way, administrators will remember that the VMware vSphere platform previously had features that accelerated update processes – Quick Boot for rebooting the hypervisor without restarting the entire ESXi host, as well as Fast Upgrades in Lifecycle Manager, where the Suspend-to-Memory function allowed you to pause a VM while maintaining its state (Suspend) in RAM. Now, Live Patching expands the range of these techniques.
What is Live Patching?
Live Patch is a technology that allows you to make changes to a running system without interrupting its operation. In the context of VMware vSphere, this technology allows you to apply security updates and bug fixes directly to the ESXi hypervisor, which is critical to maintaining the continuous operation of services and applications. The principle of Live Patch is that changes are made at the level of some components of the system kernel, which avoids the need to reboot servers and minimizes the risk of downtime.
Live Patch is powered by vSphere Lifecycle Manager. It simplifies cluster image management, provides full lifecycle management of drivers and firmware, and accelerates full cluster recovery. Certificate management is seamless, and vCenter updates can be performed with significantly less downtime than previous platform releases.
vSphere Live Patch enables vSphere clusters to be updated and patched without migrating workloads from target hosts or requiring hosts to go into full maintenance mode. The patch is applied in real time while workloads continue to run.
Benefits of Using Live Patching
- Minimizing Downtime
One of the most obvious benefits of using Live Patch is the ability to minimize downtime for VMware ESXi servers. In the traditional infrastructure management model, updates and patches often require host reboots, which can be problematic for services that require high availability. With Live Patch, such updates can be applied without the need to stop the system, ensuring business continuity and reducing the risk of financial losses due to downtime. - Improved Security
Security updates are a critical aspect of IT infrastructure management. Live Patch allows administrators to immediately apply patches for vulnerabilities, which significantly reduces the system’s vulnerability window. This is especially important in the context of ever-growing cybersecurity threats, where any delay can lead to serious consequences. Quickly applying patches reduces the risk of system compromise and data leakage. - Saving resources and simplifying processes
The process of updating ESXi servers traditionally requires significant time and effort from IT staff associated with putting hosts into maintenance mode. Live Patch simplifies this process by using partial maintenance mode, allowing administrators to centrally manage updates and minimize intervention in the system. This reduces the workload on IT staff and increases the efficiency of infrastructure management.
Technical aspects of Live Patch in vSphere 8 Update 3
1. Downloading and applying patches
Live Patch in vSphere 8 Update 3 allows you to download and apply patches to a running system without rebooting it. This is achieved by using specialized code that is embedded in the working kernel of the ESXi hypervisor. This code can dynamically change the operation of the system, eliminating vulnerabilities and fixing errors. While applying the patch, the system continues to function as usual, ensuring the continuity of services.
To use the vSphere Live Patch feature, a number of conditions must be met:
- Software versions: vCenter and ESXi hosts must be version 8.0 Update 3 or higher. This ensures compatibility and support for all new features.
- Live Patch configuration: The Enforce Live Patch option must be enabled in the global vSphere Lifecycle Manager recovery settings or in the cluster recovery settings.
- DRS requirements: Distributed Resource Scheduler (DRS) must be enabled for the cluster and running in fully automatic mode, which allows for optimized resource allocation during patching.
- Support for virtual machines with vGPU: Passthrough VM DRS Automation must be enabled for virtual machines using vGPU, which ensures compatibility with DRS during patching.
- Cluster readiness: The current vSphere cluster build must be suitable for live patching. This includes compatibility with installed driver and firmware versions, as well as the absence of critical configuration errors.
- Note that vSphere Live Patch is not compatible with systems running TPM or DPU devices using vSphere Distributed Services Engine.
Before updating Live Patch, cluster hosts will be automatically checked for compatibility with this technology, and information about support for the Quick Boot and Suspend To Memory techniques, which we discussed at the beginning of the article, will also be displayed:
Here is how the Live Patch technique works:
Let’s describe this process step by step:
- The ESXi host goes into partial maintenance mode. Partial maintenance mode is a special state that every cluster host goes into. In this special state, existing virtual machines continue to run, but creating new VMs on the host or migrating VMs to or from this host is prohibited. It is important to understand that you cannot manually put a host into this state, but you can manually take it out of it.
- The new version of the target patch components is mounted in parallel with the current version.
- Files and processes are updated using the mounted version of the patch.
- Virtual machines undergo a fast-suspend-resume to use the updated version of the components.
2. Fast-suspend-resume
The ESXi virtual machine execution component patches are the first implementation of the vSphere Live Patch mechanism. This means that while VMs do not need to be evacuated from the host, they do need to perform what is called a fast suspend-resume (FSR) to use the updated VM runtime.
A VM FSR is a non-disruptive action that is already used in VM operations when adding or removing virtual hardware devices (such as Hot Add) to running VMs.
Some VMs are not compatible with FSR. VMs configured with Fault Tolerance enabled, machines using Direct Path I/O, and vSphere Pods cannot use FSR and require administrator intervention. This can be accomplished either by manually migrating the VM or rebooting the VM – so schedule time for this in the update window.
3. Management and Monitoring
VMware provides tools to manage and monitor the patching process. This includes version control, checking patch compatibility with the current system configuration, and tracking the status of updates. Administrators can use vCenter Server to centrally manage the Live Patching process, making it easy to coordinate and monitor all stages of patching.
4. Integration and compatibility with existing VMware products
Live Patching integrates with existing VMware products such as VMware vSphere Lifecycle Manager (vLCM). This allows you to leverage existing patch management processes and procedures, minimizing the need for infrastructure changes and staff training. Integration with vLCM also provides automation of the patching process, which further simplifies infrastructure management.
vSphere Live Patch is currently only available for a specific type of patches. Only patches for the ESXi virtual machine execution component are currently supported by vSphere Live Patch. Patches that may change other ESXi components, such as the VMkernel, are not natively supported for vSphere Live Patch and currently require following the existing patching process, which involves putting hosts into maintenance mode and evacuating VMs.
vSphere Live Patches can only be installed on supported, compatible versions of ESXi. Each Live Patch will indicate which previous build it is compatible with. vSphere Lifecycle Manager will indicate the appropriate versions when defining a cluster image. You can also see the appropriate version in the vSphere Lifecycle Manager image repository:
You can also run a compliance scan in vSphere Lifecycle Manager, which will show you which VMs are incompatible with FSR and the reason for the incompatibility:
After a successful cluster upgrade, any hosts running VMs that do not support FSR will continue to report non-compliant. In this case, the administrator must manually move the VMs using vMotion or reboot them. Only then will the cluster be fully compliant and enter the compliant state.
Limitations and Potential Risks
Despite its many advantages, Live Patching also has its limitations that must be considered when using it.
- Patch Compatibility
Not all patches can be applied using Live Patch technology. Some updates may require significant changes to the system that cannot be implemented without a reboot. In such cases, administrators will still have to plan time for server maintenance and reboots. - Difficulties in testing and verification
Applying patches in real time requires thorough testing and verification to avoid potential problems in the system’s operation. Let’s recall how insufficiently thorough testing of updates led to a large-scale failure of Windows systems with CrowdStrike software. In large and complex infrastructures, this can be a complex task that requires additional resources and time. It is necessary to consider the possible risks and carefully plan the update application process. - Potential performance issues
In some cases, applying patches without rebooting may cause temporary performance issues. Although such situations are rare, they cannot be completely ruled out. Administrators should closely monitor the system state after applying patches and be prepared for possible adjustments.
Conclusion
Live Patch in VMware vSphere 8 Update 3 represents a significant improvement in the management and security of virtual infrastructures. This technology allows organizations to maintain a high level of availability of their systems while minimizing the risks and costs associated with patch management. Despite the existing limitations and risks, the correct use of Live Patch can significantly improve the efficiency of IT infrastructure management and ensure data protection in the face of ever-increasing cyber threats.
