Veeam Hardened Linux Repository – Part I

In part I, we introduce you to the hardened repository and why we need that. We also show you how to set up an Ubuntu Linux server for this purpose. Finally, we discuss the security measure you should take to harden the repository host.

Introduction

Veeam Backup & Replication V11 keeps improving and moving ahead. As the IT landscape changes, the needs and requirements of their customers also evolve with it. That is not only about new capabilities or enhancing existing features. It is also about ever-changing threat landscapes. It is a world where existing threats evolve to perfect the vectors to deliver their malicious payloads with devastating results.

In response to the above, Veeam keeps improving its defensive capabilities to deal with the ransomware threat. But as always, you have to face the hard possibility that bad actors will succeed in wiping or encrypting your environment’s data. What you need is an air-gapped capability to ensure your ability to restore data.

Veeam had such capabilities via 3^rd parties with tape, WORM disk storage, S3 compatible storage with object-lock and compliance mode in AWS, other clouds, or on-premises solutions. In Veeam Backup & Replication V11, they introduced the ability to build your own immutable, hardened backup repositories. You can do this using any server with storage that meets the requirements combined with several supported Linux distros and the XFS file system.

Veeam hardened Linux repository in v11 provides immutability

The hardened repository in Veeam Backup & Replication V11 provides the air-gap or immutability we need. It merely means that the backup files cannot be changed or deleted from the Linux server’s storage without having root access. Veeam does not require that kind of access to function, which means the Veeam backup fabric users and accounts cannot delete backups.

Requirements

We already know we need Veeam Backup & Replication V11. That is the version where Veeam introduced hardened repositories. So what else do we need? A supported Linux distro to provide the storage. For now, that is Ubuntu 20.4 LTS, which uses kernel v5.4. That is the version that delivers the best “Reflink” (Fast Clone in Veeam speak) functionality with the XFS file system, according to Veeam.

XFS with Reflink clone is the way to go. My readers know I am a big fan of ReFS for Veeam repositories. XFS Reflink achieves the same benefits as ReFS in terms of speed and space consumption. Hence Veeam calls it Fast Clone. Let’s dive into that why Fast Clone makes the difference here. Because the Veeam backup chains must be compatible with immutable files, Veeam cannot change or delete any existing backup files during the specified period of immutability. The backup chain only can create new files without changing any of the existing ones. All this means that only forward incremental with periodic synthetic or active full backups are supported. For backup copy jobs, this means that GFS settings are a hard requirement. Fast Clone makes synthetic full backups incredibly fast, and they don’t consume any disk space. That is because it is all done via metadata operations. Hence the file system of choice is XFS, which delivers the same benefits as ReFS.

Last but not least, to achieve immutability, Veeam leverages the immutability “i” flag. XFS also supports this flag, so with that, all the requirements are met, and the plan comes together.

Use Cases

The use cases for the hardened repository are the same as for other repositories. You use it as a standard repository or, as an extent in a scale-out backup repository. I always recommend the latter if you can. Note that certain backup chain types are not compatible with immutable repositories (more on that later). For those needs, you will not use immutability.

Any use case where immutability is vital is one where you must seriously consider using the hardened repository. You can use the hardened repositories for both primary backups and backup copies

Install an Ubuntu server

First, we need one or more Linux repositories setup that use the XFS file system. So, grab Ubuntu 20.04.1 and spin up a server.

That part is described plentiful over the internet, so I will no repeat it here.

Once the server installation is complete, update it. Then, log in to set up the storage. I added separate disks to hold the backup data. Let’s list them.

As you can see, I have two disks of 127GB. Those are the ones I will use for the Veeam backup data. We’ll walk through setting up one of them.

I create a partition with an XFS volume using the disk’s full size by running the following command.

That 4K in size is the maximum supported block size to have fast clone support.

Now we create a mount point for the partition/volume. As this is a disk that I want to be mounted permanently for any user, I do this under /mnt.

Now I need to find out the UUID because I will use it to automount the drive. For this, I run blkid.

Use vi or nano as you see fit to edit /etc/fstab. We are going to add the below entry to automount /dev/sdc.

If you don’t want to mess with an editor just do this

When we reboot, the disks will automount. We can also mount them right now as we want to set the permissions correctly.

We need to set the permissions, so the veeamtransport service user has the correct rights to the Veeam mount(s).

So we create a user.

Make it sudo user (we will take it away later).

We change the ownership recursively for the drive mount to that user.

We set the permission for the user on the drive

Reboot the host to test everything once more, but you should be good to go if all is well. That’s it. We can now add the repository to VBR v11. But before we do so, let’s look at some extra security considerations to make sure your immutable backup files is as safe as possible.

Additional security considerations

Introduction

Next to the immutability capability in the hardened repository, there are some additional security considerations. Remember, the primary goal is to protect the repository from ransomware attacks. These are remote attacks. While mitigation is essential, any military commander knows that you cannot prevent attacks. They will occur, they can do damage, and they will succeed. Defense aims to make attacking hard and minimize the damage an attack can cause. But, you have to assume breach and have a doctrine that allows surviving this. That is where the hardened repository comes into play. Even if attackers have gained access to your VBR server with admin rights, they are out of luck. They might hack the credentials inside the Veeam database, but there is nothing they can do to get the hardened Linux repository credentials. The VBR server doesn’t need them and doesn’t store them. Authentication happens via certificates after installation. The single-use account doesn’t require sudo access either. It can only write and read to the backup storage, where the immutability attribute on the files protects the data. After we add the repository, you can change the password and remove the users’ sudo rights. You might need it for maintenance tasks or upgrades with Veeam, but I am not sure you do.

Now the question is, what other attack vectors exist that we must mitigate? How do we prevent deletion or destruction of the backup data via other means?

Multi-factor authentication

We protect our repository user accounts with multi-factor authentication (MFA). Note that Veeam Backup & Replication (VBR)cannot pass along MFA requests to SSH … enabling MFA will ensure that no one can even use that account from VBR until someone disables MFA for that user.

Disabling or even removing SSH offers the best security against remote access. Now, if you want to keep it enabled, I won’t judge you. I understand, but please add multi-factor authentication (MFA) with SSH in this case. Duo has you covered if you need an MFA solution for SSH like Duo Unix – Two-Factor Authentication for SSH (login_duo).

There is an essential note on securing SSH with MFA. MFA will only work with Veeam Backup & Replication v11, where you only need to disable when adding the repository as a managed server. With the previous versions that require SSH over port 22, this would break the functionality. Also, be prepared to disable it when needed. Luckily that is quickly done with DUO on a per-user basis.

On the subject of MFA, you might also want to protect console access in this way. To do so, you’ll need to look into GUI and Virtual Terminal Logins with pam duo. With this, you can protect both SSH and console access.

I even enabled MFA for sudo actions, just in case someone gets access to my console.

That’s why I went with the Duo Unix – Two-Factor Authentication for SSH with PAM Support (pam_duo) on my Ubuntu 20.04.1 solution. If that is not an option, use the above mentioned Two-Factor Authentication for SSH (login_duo).

As always with MFA, do it right, don’t lock yourself out! Also, have multiple MFA options just in case. I tend to have a Yubikey or similar for backup access.

If you like, you can, and you probably should use certificate-based authentication in combination with a password. With DUO, you can also combine this with MFA. Ideally, in larger environments, you’ll have a CA set up to handle creating user certificates, but you can create self-signed certificates as well.

Do note that you cannot use certificate-based authentication as the single-use credentials to add a hardened Linux server under managed servers in Veeam. It needs the private cert/passphrase of the user, which is what they want to avoid. We do not want anything of value on the VBR server that can be used to gain access to the repository by adding the private key, even temporarily, which incurs the overhead and responsibility of keeping it secure. Also, it is a bit overkill to set it up for that one-time user.

Other SSH security measures (in /etc/ssh/sshd_config) to consider:

• Reject empty passwords (PermitEmptyPasswords no)
• Reject root login (PermitRootLogin no)
• Set a session time out limit for SSH sessions (ClientAliveINterval 300)
• Limit the number of authentication attempts (MaxAuthTries 3)
• Limit the maximum number of unauthenticated connections (MaxStartups 3:50:XX)
• Reduce the maximum amount of time allowed to successfully login (LoginGraceTime 60)
• Limit the users on the server and those allowed to login via SSH (AllowUsers User1 User2)
• Consider installing failban and whitelisting the know IP addresses that may log in.

Get your time sync on

Accurate time is vital with the immutability flag to check what files can be deleted or not. Configure the NTP servers for your repository with the ones you use in your environment for consistent and correct time settings. Use a reliable source local to your area. You can find excellent and redundant NTP servers at pool.ntp.org. What is good to know is that even if people gain access to your VBR host and mess with the date and time of the system (not the hardware clock), it does not matter. The date and time on the repository server are most important for this use case.

BMC, DRAC, ILO, XCC, etc

When you use DELL DRAC, HP ILO, Lenovo XCC, or any BMC tool, realize that this provides other ways of deleting data when that password is compromised, they might not be able to logon to the OS, but they can wipe your local storage that way. This kind of access I have not been able to protect via MFA to this day. The most secure option here is to disable it. If you need this kind of access, leave it offline (unplugged) and have hands and eyes in the server room to plug it in when needed. That is the “sneakernet” way of disabling connectivity. You could also do it at the switch port level. It raises the bar slightly while moving the risk to a switch whose credentials might be compromised. Whatever method you choose, you’ll need reliable processes to ensure that it is never left connected after use. In any case, protect your credentials from being compromised!

Leave root alone

By default, Ubuntu does not set up a root password. That means it can’t be used to login to the systems as an empty password is disabled. Leave it that way. Use a standard user with sudo capabilities for administration and use another, dedicated account for the Veeam single-use credentials for the hardened repository. Live by the minimum privilege required principle for user accounts.

The host firewall can be your friend but Veeam handles a lot for you

The short version is that you do not need to configure the firewall. It will work and you will be optimally protected as Veeam takes care if this behind the scenes. That’s it.

For those that want a bit more details, here it comes. When you enable SSH on Ubuntu to add the repository host you can disable it immediately after doing so.  That means the SSH port (22 normally) can be removed from the firewall or even blocked. It is important to know that Veeam will handle all firewall rules it needs behind the scenes for you.

That why you will find entries in the firewall like below.

6162/tcp                   ALLOW       Anywhere                   # Veeam transport rule

2501/tcp                   ALLOW       Anywhere                   # Veeam rule ad684de2-5a4f-49d5-81d9-0368b4281af9

Leave these alone. Veeam handles this. The Veeam Linux transport service (or more correctly, its child process – enviromentsvc) takes care of all that for you. After an agent has finished its work for a job, the transport service removes the firewall rule for that agents particular work. That’s what I call optimized security.

If a port has already been opened by the administrator of the server, the transport service won’t open or close it.

All this means that if and when you need to configure the firewall, you can and should not touch the rules for Veeam. That is handled for you. This is truly a case of “just because you can doesn’t mean you should”. Veeam has that covered. I  did not expect anything less. You get optimal security by leaving the repository host alone.

When enabling and configuring the firewall, don’t forget about the sources on the internet you might need to reach, such as your NTP Servers, DNS, S3 storage (Backblaze, …), MFA (Duo, …), Ubuntu for updates, etc. Lock it down as needed or required by security policies, but there is no need to touch the Veeam rules!

If you are new to Linux, let Veeam handle it all. If you need to configure the host firewall, for Windows people, like me, it is easier and probably best to use the Ubuntu UFW (Uncomplicated Firewall) and not dive into IP tables. UFW is disabled by default. Enabling and using it is pretty easy. You can read more about that in How To Set Up a Firewall with UFW on Ubuntu 20.04 | DigitalOcean

A proxy and an external firewall complete the security stance

Consider a proxy server/firewall that allows only outbound internet traffic to predefined FQDN’s (DUO, UBUNTU, Backblaze, NTP), and that blocks any unwanted inbound traffic. For remote access, use a jump host behind a secured gateway. Do not allow direct access to the repository server from the internet. If you are looking for free solutions, I tend to use SQUID for a proxy and OPNSense as a firewall.

Thou shalt not pass!

Then there is physical access to the servers, which will allow anyone to gain access to the server if they have the skills to do so. If not, they can resort to physically destroying the repositories or even the entire building. But let’s be realistic. Ransomware is about monetary gain, so they will not set the servers on fire or blow up the building. And in the case of wipers, physical assaults are in a different category altogether. That is out of scope. Here we deal with ransomware and wipers attacks. These are remotely executed and based on lateral movement and compromised credentials. So, what’s left? Insider threats that have console access, but at that point, the only thing that can help is auditing, vetting of people, and control measures.

If you are an important enough target, they might plant people at your company, bribe or threaten/blackmail an employee who holds the needed credentials. But suppose you are in that segment of security requirements. In that case, you will have auditing and shared responsibilities. No single person can even be in the server room alone, let alone work on a system without someone to assist, and who knows what they are there to do and observes those actions. Measures will be in place to trip alarms in such cases. Most of you are not in that target group, but 99.9% of you are a target for ransomware and wipers, if only as collateral damage.