Introduction
Many years ago, I wrote a white paper on how to configure a VEEAM Off-host backup proxy server for backing up a Windows Server 2012 R2 Hyper-V cluster that uses a hardware VSS provider with VEEAM Backup & Replication 7.0. It has aged well and you can still use it as a guide to set it all up. But in this article, I revisit the use of a hardware VSS provider dedicated specifically to some changes in Windows Server 2016 and its use by Veeam Backup & Replication v9.5 or later. The information here is valid for any good hardware VSS provider like the one VSAN from StarWind provides (see Do I need StarWind Hardware VSS provider?)
Major Changes in Windows Server 2016 Hyper-V Backup
This is actually a big subject. To fully appreciate the steps forward made by Windows Server 2016 Hyper-V Backup I encourage you to read two articles on the StarWind Software blog: Hyper-V backup challenges Windows Server 2016 needs to address and Windows Server 2016 Hyper-V Backup Rises to the challenges. I’m not going to repeat all that info here.
Hyper-V backup has seen major improvements in Windows Server 2016, especially around performance, scalability and reliability and scalability. One of the significant changes that helped in achieving that was the removal of the hard dependence on the Hyper-V host VSS provider. To be clear, it is no longer needed to backup Hyper-V guests at all.
This doesn’t mean however that it’s impossible now to leverage a hardware VSS provider with Windows Server 2016. If you have a storage array that has a supported hardware VSS provider you can use that to integrate the SAN snapshots into the backup software (or vice versa depending on your point of view) for various use cases.
So why would I still use an off-host proxy?
As we stated above that you don’t need to use the host VSS provider anymore and this has helped improve Hyper-V backups this leads to the question why you would still need or want to do this in this day and age? Especially in environments where you have 10Gbps or better RDMA capable networks and you can configure backups to be copied from the source to the target leveraging SMB Multichannel and SMB Direct and multichannel, this seems old-fashioned or in many cases with certain Hyper-V converged or hyper-converged solutions even impossible. Think Storage Spaces (Direct) for example.
Some (hyper-)converged solutions do have a hardware VSS provider. VSAN from StarWind is an excellent example of this. While it was primarily developed for internal use such as for LSFS based HA synchronization which leverages snapshots. You can however also leverage this hardware VSS provider with Veeam Backup & Replication if you wish to do so.
On the other hand, even today there is still a huge percentage of big and small SAN storage around or being actively deployed leveraging iSCSI and Fibre Channel. Sometimes this is because it’s known familiar technology, it works and serves the needs. Sometimes it’s because the use case and/or environment are not a good fit for hyper-converged. So, no, it’s not always because people are caveman era type engineers that will let you have their SAN when you pry it from their cold dead hands. But no matter what, in essence, you do not need to leverage a host (Windows native or hardware) VSS provider anymore even with those traditional SANs. So, let’s take a look why you might still do this.
An off-host backup proxy has following benefits:
- It offloads the backup IO way from the production Hyper-V host. This relieves the IO on the CSV and the FC or iSCSI HBAs.
- This might still help to avoid potential IO-related issues like timeout issues under heavy backup IO with CSVs or some of the bugs related to software snapshots of Hyper-V CSVs on Windows Failover Cluster nodes. Even if that’s is probably less so than in previous versions.
- You offload the backup process from the Hyper-V nodes to a separate server. This preserves CPU cycles and network bandwidth associated with the backup process for virtualization workloads.
- Depending on your particular setup you avoid copying data over the network if the backup repository storage is attached directly to the Off-Host Backup Proxy.
Does this mean you sacrifice the progress made in Windows Server 2016 Hyper-V backups? Do you lose the improvements in performance, scalability, and reliability? The answer to that is no. You can use hardware VSS providers and still enjoy many of the benefits we gained.
If you’ve read the above-mentioned articles Hyper-V backup challenges Windows Server 2016 needs to address and Windows Server 2016 Hyper-V Backup Rises to the challenges it will help to understand why.
Even with a hardware VSS provider the Hyper-V guest backup is now leveraging the Hyper-V recovery checkpoint and relies solely on the in guest VSS to create an application consistent state. This means even when leveraging the hardware VSS provider no interaction is needed with it to create that application consistent recovery checkpoint and the host VSS snapshot process doesn’t need to wait for all VMs to finish their part of the backup process. This translates into significant time savings and reduction in complexity. The time needed to make the hardware VSS snapshot is strongly reduced as it no longer starts with the first and last guest VM finishing it’s the vhdx creation and having to mount the vhdx into each guest for further consistency processing against that guest VSS snapshot. Actually, even a generation 1 VM with only IDE controllers and no ISCSI controller will backup fine now. That iSCSI controller had to be there on Windows Server 2012 R2 Hyper-V guests to be able to mount and dismount the VHDX in line. That’s no longer the case (*).
(*) Note: The fact that you will not see a disk mount event with a duplicate disk ID warning and a disk unmount during backups of VMs on Windows Server 2016 Hyper-V host is valid when you use Veeam with the Replay Manager 7.8 hardware VSS provider. When back-ups are made of the VMs with the Replay Manager 7.8 Server software without Veeam you’ll still see those events. This seems to be a legacy artifact on Windows Server 2016 Hyper-V host for their backup software.
Typically, the system event log of a VM on a Windows Server 2016 host that’s being backed up with Veeam shows only informational events about the stopping and starting of the Hyper-V Volume Shadow Copy Requestor Service (i.e. the backup software), the Veeam service-related events around the VeeamVSSSupport service being installed and started (VeeamGuestHelpe.exe) some VSS snapshot related events.
That’s a big difference with how it used to look like this inside a VM on a Windows Server 2012 R2 host. There you would get a warning about duplicated disk ID’s and about the surprise removal of that disk during the mounting/unmounting of the VHDX to the guest on a vSCSI controller.
Now it can just create a “volume snapshot” using recovery checkpoints without all that mount/dismount overhead. On top of that, the reliability improvements carry over to the off-host backup proxy process of taking backups making the process more robust and allow for fewer moments for potential conflicts or problems when multiple VSS snapshots are being taken simultaneously or very close to each other. In this example, we even had the extra help from an updated Hardware VSS provider that has fixed a Veeam specific issue. Take look what all these improvements mean to the duration of creating a host VSS snapshot. It’s impressive.
In the above picture, you can see that the hardware VSS snapshot time is 21 seconds. That at least 10 times faster than what it used to be (3.5 to 4.5 minutes depending on what the SAN is under at that moment). I have seen it drop a low of 11 seconds and have never seen it go over 25. Actually, the import phase can now take longer than the snapshot phase.
Do note that this doesn’t mean you need or must use off-host proxy backups. There are still some VSS / LUN related drawbacks to this. But if your environment today can still benefit from them to reduce CPU cycles and reduce bandwidth on the Hyper-Host iSCSI or FC HBA’s and avoid possible network congestion you certainly can. You might end up as a very happy camper and have a better experience with faster, more reliable and more scalable solution when you do leverage Veeam off-host backup proxies. I know I am.
Conclusion
Leveraging off host proxy backups for Windows Server 2016 Hyper-V environments does still make sense in certain environments. We have shown that even as it relies on a host-based hardware VSS provider it actually still benefits from the improvements Hyper-V backups got in Windows Server 2016. That gives us better performance and reliability. Make no mistake, decoupling Hyper-V backups from host VSS snapshots is a very good thing. This does not, however, mean that a Windows Server 2016 Hyper-V aware (hardware) VSS provider cannot benefit from this as well. It will work faster, scale better and more reliable too. Examples of this a when leveraging transportable snapshots with off-host backup proxies and when it with certain types of backups that integrate with SAN storage snapshots. So, if you can benefit from using a hardware VSS providers, do so and don’t worry but be happy!
