Welcome to part II. In part I of this series, we configured DUO MFA for a Veeam hardened repository. We will add public key authentication and ensure this also works with MFA. On top of that, we require that you will also still need to enter the password for the sudo account. However, the Veeam service account should support MFA but must not use public-key authentication, but only password authentication.
Adding public-key authentication to the mix
Set up public-key authentication for SSH
There are several steps to making public-key authentication for SSH work.
Creating a private./public key pair
First of all, set up an SSH key pair for your sudo account. We do this by running ssh-keygen in a Windows Terminal Powershell session/ Specify a path and a name for the key pair, enter the passphrase twice, and that’s it. I do not use the default name as I have multiple keys for different servers. However, I do use the default path. Be careful not to overwrite any existing keys with the same name if you use the defaults.
Figure : Create a private/public key pair for your sudo account
You will find your keys under the .ssh folder in your home directory.
Figure : Note the custom names for the keys
As noted above, I have chosen a custom name instead of id_rsa, id_dsa, or the identity default names. That means I need to specify which one to use for what host in the config files under .your home directory .ssh folder, or the login will fail. That is handy, especially when you have multiple key pairs and don’t want to specify them explicitly when connecting to the server using ssh user@server -i /folder/subfolder/mysshkeypair.
Figure : The config file is your friend
Upload the public key to the server
Now you need to get the public key stored on your repository server to use it to log in.
In PowerShell, you can do this as follows. I use the IP address of the remote host. The FQDN works fine if you have name resolution in some shape or form.
get-content $env:USERPROFILE\.ssh\repo001-admin.pub | ssh repo001-admin@192.168.2.188 “cat >> .ssh/authorized_keys”
Why this way because the Linux option of “ssh-copy-id username@myremotehost” does not exist on Windows, that’s why.
Finally, note that if you already have MFA configured for the user account with a password as we did before, this action will prompt you with an MFA challenge.
Adding public-key authentication to the mix 
Figure : On a Windows 10 or higher machine, we use PowerShell to upload the public key to the server
So that is set. Public key auth will work now. But first, let’s look at how this all ties together. When a client connects to a server, there is an exchange of public keys. The first time you connect to a host, you need to accept its fingerprint and store it in the known_host files in your home folder. After that, you send yours to the server. First of all, the server checks if you are allowed to connect at all (which we configure in /etc/ssh/sshd_config). Secondly, it will try to find your public key in the /.ssh/authorized_keys file. Hence if you did not upload it, authentication would fail.
But here is a tip. I always explicitly define the AuthorizedKeysFile in /etc/ssh/sshd_config. If you have them in the default path, you don’t need to do this. But it’s there already when you want it stored somewhere else on the server.
Test your setup
You can test this now. Note that because we set a passphrase, we need to provide it. The key alone is not enough. What you can also do is disable password authentication altogether. But that is not what we want. We want the sudo user to leverage public key authentication (with a passphrase) and provide the password for the user account. Next to that, we want all this to work with Duo MFA. That means we still have some work to do!
FIDO2 security key
When it comes to SSH and public-key authentication, you can also leverage a FIDO2 security key. I prefer biometric ones like a FEITIAN BioPass K26/27 or K49/50. That way, you can leave the passphrase blank but require that you provide your fingerprint for MFA with SSH public key authentication.
Figure 6: That’s typically me, a notebook, a pen, and one of my trusted FIDO2 biometric security keys
You could go wild and put the M in MFA to work and require both the passphrase and the fingerprint touch on your FIDO security key, but don’t go overboard.
Setting SSH up with a FIDO key requires a recent SSH version that supports this. And note that this only works with SSH, not with console access or sudo actions. I mention this here for completeness’ sake. I hope to show you how to do this in a future article, but for now, I leave you with an animated gif of it all combined as a teaser.
Figure : Adding FIDO2 into the mix!
Making it work with DUO MFA
For public-key authentication to work, we need to configure several things.
Configure /etc/ssh/sshd_config
In the ./etc/ssh/sshd_config files, you need to add the following
PubkeyAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
PasswordAuthentication no
Cool! But wait a minute. Now my Veeam service account also wants to use public-key authentication. Not good as Veeam Backup & Replication does not support this. So how do we solve this? In the file ./etc/ssh/sshd_config we also define the following block.
Match User repo001-veeam
PasswordAuthentication yes
KbdInteractiveAuthentication yes
PubkeyAuthentication no
AuthenticationMethods keyboard-interactive
Match All
As the user name matches rep001-veeam, it gets a different configuration for authentication. Cool huh! It will only have to provide a password, not public-key authentication. So once you disable MFA, this will work with Veeam Backup & Replication for installation and maintenance.
The rest, for now, remains as it was for DUO MFA with passwords.
Configure SSH Public Key Authentication
Finally, we must ensure that the sudo account uses public-key authentication (with a passphrase) and the password. To make that happen need to make a change to the /etc/pam.d/sshd file.
#@include common-auth
auth required pam_env.so
auth requisite pam_unix.so
auth [success=1 default=ignore] /lib64/security/pam_duo.so
auth requisite pam_deny.so
auth required pam_permit.so
The text in Red is what you have to comment out, or you’ll have to meet two MFA challenges (one for the password and another for the public key authentication. The text in green is to make sure you have to specify not just the public key passphrase but also the password. Without those two lines, it would not ask for a password. Finally, the blue text is what you need to make MFA with public-key authentication work, but it also has to be there when you combine it with a password.
Testing is important
You must test all scenarios to prevent missing a configuration issue. For example, in Duo, you can (permanently) delete users and test the behavior of a non-en-rolled user logging in. Do this for both the sudo admin account and the Veeam service account. Then, test the behaviors with MFA enabled and disabled (bypass in DUO) when the user accounts have been enrolled. Make sure things work as expected.
The results and behavior before the accounts enroll with DUO
We will describe what happens when a user (our sudo user to manage the server) who is required to provide both public key and password authentication logs in for the first time and has not yet enrolled with DUO. After that, we will also look at the experience with a user exempted from public key authentication which has not yet enrolled with DUO. In our case, the Veeam service account. Finally, we will look at both the console and SSH scenario for completeness.
You cannot log in before an account has enrolled with DUO, so the sudo scenario does not exist here.
Console
The console does not use public-key authentication. That is only for SSH. So, you only need to provide the username and the password, after which you will be prompted with a link to enroll with DUO.
That also means that the experience for the Veeam service account at the console is the same.
SSH
With SSH and the sudo account, you must provide the passphrase for public-key authentication. Next, you will have to give the password. Only when both are correct will you see a link to enroll with DUO. Then, in any decent terminal (I am using Windows Terminal), you can CTRL-click on that link and run through the enrollment process for DUO.
For the Veeam service account, the experience differs as we must not provide the public key authentication passphrase. Instead, the user needs only to enter a valid password, after which they see a link to enroll with DUO.
The behavior with accounts after they enrolled with DUO
Console
You must only provide the password at the console, select an MFA method, and approve the MFA challenge. That’s normal as public-key authentication is only an option for SSH.
So, in this scenario, the experience for the sudo account and the Veeam service account is the same.
SSH
First, let’s look at our sudo account. In the screenshot below, you can see that I need to provide both the passphrase for my private key and enter the password for the account. Only then the prompt for MFA appears. That is only the case for a connection made via SSH.
For the Veeam service account, we do not get a prompt for the public key authentication passphrase because this account is exempt.
Sudo
When performing sudo operations, you are only required to provide the password, select an MFA method and approve the MFA challenge. But, again, that’s normal; public-key authentication is only an option with SSH.
Under normal circumstances, the Veeam service account is not a member of the sudo group. Only during the installation of the hardened repository and afterward, during maintenance for updates or upgrades of Veeam, is SSH enabled, but for that purpose, we bypass MFA. Hence we keep that maintenance window short.
Be careful with Linux updates
When installing updates, you might get into a situation where they ask to overwrite the existing configuration. Choose wisely and make sure you can restore/recover your original implementation.
If you do overwrite the file, there is normally a backup copy. For /etc/pam.d/common-auth that would be /etc/pam.d/common-auth.pam-old.
Impact on Veeam maintenance
The Veeam Backup & Replication console and MFA do not mix with SSH connections. So as you need SSH to update the components, that means that during installation or maintenance, you need to set MFA to “Bypass” in the DUO admin portal for the account you will use to perform the updates.
That is also why, while we do have MFA on the Veeam service account on the hardened repository, we do not configure this account to use public-key authentication. Only user name/password will work for Veeam Backup & Replication SSH connectivity.
Performing maintenance
We usually have a couple of options to upgrade the Veeam components of a hardened repository with single-use credentials. We will look at both.
Pre-maintenance
No matter what option you use, you will have to turn off MFA as Veeam Backup & Replication cannot handle this for SSH. Also, in any case, you will have to enable SSH on the repository and allow SSH in the host firewall.
Option one
Option one is the Veeam sanctioned way of doing updates. Add the Veeam service account to the sudo user group temporarily for maintenance, and use this account and password in Veeam to perform the updates or upgrades. Afterward, remove this account from the sudo group and set a new password. Alternatively, you might return it to the original password if you set a temporary one for the duration of the maintenance. You might have given the password to someone else to perform the maintenance, and you don’t want it to be known. The drawback of this is that the person doing the Veeam updates has to know this user’s password. If you have different groups/people handling this, divulging that information leads to additional steps to keep it secure.
Option two
Warning, this is not the Veeam sanctioned way of doing this, but it works. Use a sudo account from the repository to perform the update or upgrade. Again, if you are not the person doing the maintenance, you want to give this account a new password afterward or set it to a temporary one for the duration of the maintenance. You could also create a separate “maintenance sudo account” to perform updates or upgrades. However you approach this, Option two also has a drawback. You will need to edit the file on the hardened repository and set the account in that file back to the original single-use credential user name.
The benefit is that you never add the Veeam service account back to the sudo group and reduce the risk of forgetting to remove it. You also don’t have to provide the password for the Veeam service account during the upgrade. Finally, one more, option 2 is not the Veeam sanctioned way of performing hardened repository with single-use credentials updates.
IMPORTANT! The sudo accounts have to use public-key authentication with a passphrase and a password per the requirements in this article. As a result, option two will not work. Veeam Backup & Replication does not support public key authentication. So, unless you also create that 3rd maintenance sudo account and exclude that from public-key authentication, just like you did for the Veeam service account, you must use option one.
Post maintenance
After Veeam maintenance, you need to disable SSH again and deny SSH access in the local host firewall. Then, in the DUO admin portal, activate MFA again for the accounts you use during maintenance.
Securing the hardened repository
Finally, MFA is lovely, but remember that we have the SSH service disabled by default. We also block SSH (TCP/22) in the Ubuntu UFW. That means that we have to log in locally via the console, which requires that we have physical access to the server. The server resides in a secured cage in a secured data center. There are cameras, biometrics, and access gateways.
On top of that, you need a second person to accompany you. Likewise, we keep the DRAC card unplugged and the switch port disabled for security. When you want to use the DRAC connection, you need to call the data center, identify mutually, and get someone to plug the DRAC RJ45 cable into the server. Then, you need someone to enable the switch port for the DRAC connection to start working. You get the idea. There are some roadblocks for people who want to access the server. So, if you need SSH for some reason, do implement MFA. It is your best line of defense in that case, and it also helps secure local access and sudo use.
Separation of duties will also help mitigate insider threats.
Protect the credentials for all components such as the DUO administrative portal, your switches, the DRAC card, etc. Please put them in a password manager. Where possible, leverage MFA to help prevent abuse of these credentials.
While I do like smartphones for MFA, I also use FIDO2 security keys with biometric capabilities like those from FEITIAN. I usually make sure the personnel dealing with security systems have both. When it comes to FIDO2 keys, these are handy to keep as a spare for “break glass” scenarios. Just register the accounts and add the fingerprints of one or more responsible persons. Also, keep in mind that on servers, USB-A is the standard. So, if you have USB-C dongles, carry a converter or USB hub.
Paranoia?
Remember that the goal here is to keep your backup safe no matter what. That means you must protect against both what you know and what you do not know. The skills and determination of your adversaries may be superior to yours. The hardened repository with immutability will act as your failsafe. So, while all this sounds paranoid, your main goal here is safeguarding the backups, not comfort and ease of maintenance. That is the price to pay. Pay it.
Conclusion
There you have it. As you can see, we met the requirements, and this configuration works. This way, we provide the best possible protection for the credentials on a Veeam hardened repository that I can offer.
Combined with all the other protective measures and settings, it is no wonder Veeam meets the compliance requirements for the key U.S. financial industry regulations. For example, the Veeam hardened repository meets the requirements for non-rewritable, non-erasable storage as specified by SEC 17a-4(f), FINRA 4511(c), and CFTC 1.31(c)-(d) regulations. Download Protect against Ransomware with Immutable Backups (veeam.com) to learn more.
The MFA requirements took some lab work to get right. But in the end, with lots of testing, we achieved the criteria. So I hope it helps some of you out there who might come across this challenge.
There is a lot of information and possible permutations to consider. I also noted the drawback of these requirements regarding maintenance (updates and upgrades). Whether these pose a concern depends on how you handle those.
