In part III, we dive deeper under the hood to take a better look and understand a bit better how it works. Here, we explore scenarios bad actors might try and what would be needed for them to succeed. If you’ve done the setup right, they won’t be able to delete your immutable backup files.
A look under the hood
Let’s look around a bit in the folders and at the files our Veeam backups jobs create. Next to that, we’ll look at the Veeam binaries and other files we can find. That should help us wrap our heads around how Veeam implemented immutability and how it works.
Navigate to the folder of a backup job. With lsattr we can see the” i” attribute that has been set on the backup files in the folder of a backup job.
Figure 1: The “i” attribute set on the backup files
Use lsattr -l for a human friendly output of the attributes that are set.
Figure 2: Immutabe is the only attribute set on the backup files
Note that the .vbm file cannot be set to immutable as that one Veeam needs to update.
With ps aux | grep -i Veeam, we list the Veeam processes. Pay attention to veeamimmureposvc. Note that it has root. When Veeam creates the backup files, this dedicated local service sets the “i” attribute. After the configured period of immutability has passed, this same dedicated local Veeam service removes that “i” attribute, allowing deletion of the file (chain). While the Veeam service veeamimmureposvc has root access to set the “i” attribute, it is 100% local, has no network access, doesn’t listen on any port, and cannot be compromised remotely.
Figure 3: Note the veeamimmureposvc service that has root access but does not listen on any port
By the way, run the same command while you run some backup jobs. There is interesting stuff to discover!
Figure 4: The Veeamagent is busy during a backup
The next question is, where are the date and timestamp stored for that immutability period with that “i” attribute set? Let’s investigate this inside the folders of the backup jobs.
With ls -a -l, we see a single lock file per backup job folder with root as its owner. The veeamimmureposvc service is in control of that file.
Figure 5: An exciting hidden lock file to look at!
When we run cat .veeam.3.lock, we get the output and see it is an XML file. For every backup file, it holds the date and timestamp until it will be immutable! That is cool. The file number increases with every run of the backup job, by the way.
Figure 6: Behold the immutability dates and timestamp per file in the backup chain!
Is there anything more? Yes, actually. When we look for extended file attributes with getfattr *, we see pseach backup file has a “user.immutable.until” extended attribute
Figure 7: Let’s look at that attribute, shall we?
We now run getfattr * -n user.immutable.unil and what do we see? The timestamp until when that backup file is immutable!
Figure 8: Aha! The date and timestamp until when a file is immutable!
So Veeam has some dual immutability persistence setting mechanism. May that is for redundancy or may to do a check against each other to make sure it is correct. Maybe it also has to do with the fact Veeam backup files are transportable and self-contained so that information is stored as an extended file attribute. Exciting stuff going on here!
One last thing. If you dig around, you will find /opt/Veeam/Transport and see a file named VeeamTransportConfig. Open it with sudo, and you will be able to see the service account for the transport service and even change it to another user. Under normal circumstances, you should not have to do this, but I tried, and it worked after restarting the service with sudo service veeamtransport restart.
Figure 9: Seeing the service account for the transport service and changing it to another user
Mind that if you change it, you need to change ownership on the repository mount and files, which is a bit tricky with the immutability set. You can’t change ownership on those files. It would be best to start a new backup chain with an active full backup, although it seems to make a full one by itself in testing. I also note that this appears to have no effect on restores, which is good. But do not play around with this in production without advice from Veeam support! I just added this for some added insights.
Testing the protection
Yes! Let’s play Monday morning backup admin, rogue admin, or hacker who has access to the VBR server. In any of those roles, we will try to delete the backups in several ways.
Figure 10: Nice try, but no cigar!
No, that doesn’t work. You are protected. There is no way that you can delete the backup files from a compromised VBR host or console.
We pretend to be a bad actor who has gotten admin access to the VBR host via lateral movement and compromised credentials.
What if we turn of immutability on the repository in VBR? Can we delete the backup data? No! The “i ” is set on the existing files on the repository, so these are protected. Only new backups are unprotected.
Remember that we said NTP was important? Well, what if we can manipulate the date and time? Let’s try. First, we set the date well into the future beyond our defined immutability period. I try to delete the backups from the disk, but this fails. The check is also happening at the repository server, and that is not having any of this.
Figure 11: Manipulating date/ time on a compromised VBR server alone will not allow deletion of the backups from the VBR console
Assume we have access to the repository. We could format the disks, remove the -I attribute to delete the backup data, or manipulate the date/time setting to delete them. But let’s play around a bit.
I did “succeed” in deleting the files via the VBR server by manipulating the time on both the VBR server and on the hardened repository. It took me a few attempts, but I got there. That is only possible because I have root access to the repository, and it is game over at that point.
Figure 12: Only with root access to both the repository and the VBR host to manipulate the dates can the attack succeed
Changing the date on the repository to beyond the immutability period will clear the “i” attribute. Maybe not immediately, but I did notice the “i” attribute was gone pretty quickly. So that local veeamimmureposvc is keeping watch.
Figure 13: The “i” attribute is removed after setting the date far in the future
That saved me the trouble of removing the “i” attribute with chattr -i. Myself.
An interesting scenario is that removing the “i” alone, so not by manipulating the date, will not allow you to remove the data via the VBR host. That is somehow still protected.
Figure 14: Removing the immutability attribute. Did you see me missing sudo first?
Figure 15: Manually removing the “i” attribute – I have MFA enabled, as you can see
What could it be? Well, maybe because the hidden lock file is still there and is also still immutable!
Figure 16: The lock file still has the immutability attribute set
So, we remove the immutability attribute of that file as well. To do so run chattr -i .* instead of chattr -i *.
Figure 17: The lock file no longer has the immutability attribute set
We delete the lock file as well. But even then, we fail to delete the backup files via VBR. Well, the extended file attribute per backup file that holds the data/timestamp is also no longer there after removing the immutability attribute. So I’m not sure how this works. Veeam knows it is an immutable repository, and it seems to be fighting back.
That’s a nice try by Veeam. But, when I have root access to the repository and clear the “i” attribute, all I need to do is run rm -rf on the files, and it is all gone. I can even shut down the Veeam services to clear the folders and anything it might be locking still. Or I could go for the RAID controller and clear the physical disk setup, initialize the disks, etc. I don’t need to use VBR to delete the backup files. That can balk at me all it wants, and I delete the files on the backup repository directly when I am root.
The above is the reason why we implemented multi-factor authentication on our Veeam backup fabric! Or you must disable SSH. Next to that, you should take all the other measures discussed above to protect your repository hosts.
Migrating to the Linux repositories without worries
Migrating to the Linux repositories without worries is relatively easy by leveraging one or more scale-out backup repositories (SOBR). You either already have it, which is perfect. If you don’t have a SOBR yet, there is a quick fix. In the latter case, create a scale-out backup repository and add the Windows-based repositories to it as extents. No worries, you won’t lose your backups. Don’t forget to point your backup jobs to the SOBR now. Using your existing or newly created scale-out backup repository, you add the new Linux repositories to it as extents. Now seal the Windows-based SOBR extents. By doing this, you keep your existing backups around to do restores if needed.
Meanwhile, the retention policies will gradually expire the old backups until the Windows SOBR extents are empty. At the same time, Veeam writes the new backups to the SOBR Linux extents. Those backups then already enjoy the added protection of the hardened and immutable repository feature. The process I described here is just one more example of the power of Veeam SOBR in action!
Conclusion
Veeam is on to something here. You take a Veeam supported Linux distro, grab any spare or purposely procured hardware with sufficient storage that meets the requirements, and build immutable repositories yourself. Just like that.
There is even better news. You can make an existing Linux repository immutable after upgrading to V11 by changing the owner and permissions for the current backup files. I have not been able to test this as the Beta 2 software does not allow for upgrades.
I love it! I do. Still, I have some outstanding questions. Can we leverage NFS or SMB to use file shares backed by XFS and enjoy Reflink/Fastclone? Do we need to use a clustered XFS solution to make it highly available, and if so, is it genuinely transparent failover, or is it done via retries? Is there a Linux solution for local and shared storage to build hybrid (flash/HDD) storage that tiers/destages data for me? All questions that I still need to answer. But that is for another time!
