Introduction
This article will protect all local users on an Ubuntu system with Multi-Factor Authentication (MFA). We will use a smartphone and a FEITIAN security key with biometrics (fingerprints). You might say, that’s easy enough. Correct! But it is a bit less straightforward when you can only use password authentication with MFA for one account. For the others, such as the sudo account, the requirement is to use both public key authentication and password authentication for SSH.
In part I of this series, which you are reading now, we configure DUO MFA to protect the credentials for a Veeam hardened repository. After that works, we will add public key authentication and make sure this works with MFA in part II. On top of that, we require that you still need to enter the password for the sudo account. The Veeam service account should support MFA but must not use public-key authentication, but only password authentication.
Technologies used
You must run Veeam Backup & Replication (VBR) v11a or later. Older versions of VBR do not have the hardened repository with single-use credentials capability. I run VBR on Windows Server 2022 at the time of writing. For the operating system of the Veeam hardened repositories, I run Ubuntu 20.04 LTS for the Veeam hardened repositories. The 22.04 LTS is available when writing this, but with OEM hardware support as a requirement, I stick to 20.04 for now. The file system is XFS with fact cloning enabled. I leverage RAID 60 on the Dell EMC R740-DX2 servers, which boot from mirrored BOSS SSDs and have a PERC H740P raid controller with 8GB of cache with 26 HDD attached. With disk sizes from 8TB to 16TB, this gives you real-world usable storage from 145TB to 290TB with dual parity on 24 disks and two global hot spare disks. Rebuild times are long with large HHDs. That’s why we have two global hot spares. With this number of disks, RAID 60 helps survive multiple disk failures within a short period and gets better IO than RAID 6. Rebuilds can take days or longer as backups jobs are still writing to the storage array. You can read more details about such a configuration in the following article DELL R740XD2 Configuration For Veeam Hardened Repositories.
Networking is 1Gbps for DRAC/Host connectivity and dual 10Gbps or 25Gbps for backup traffic. The switches are Dell EMC PowerSwitch S52XXF-ON series.
I use DUO as an MFA provider as they have outstanding capabilities and great support. I like the FEITIAN biometric models such as the K27 (USB-A )and the K26 (USB-C) for my security keys. They also have the newer K49 (USB-C) and K50 (USB-A) models.
The SSH/MFA requirements?
One use case for MFA on Linux servers is Veeam Backup & Replication hardened repositories with immutability. I will use this as an example use case for the configuration in this article. That is also the use case for which I came up with an answer to the challenge of combining public key authentication (SSH) and password authentication (SSH, console, and sudo use) for the administrative accounts (member of the sudo group) of the host. However, the Veeam service account on the hardened repository should only use a password as the single-use credential cannot handle SSH public key authentication for setup and maintenance (updates and upgrades).
Requirements
The administrative (sudo) account must leverage both public key authentication with a passphrase AND the password when logging in over SSH in combination with an MFA challenge. Next to that, console access and sudo privileges use a password combined with an MFA challenge.
The Veeam service account (the one you configure with single-use credentials) in Veeam backup and replication will only use a password combined with an MFA challenge for SSH, console, and sudo access. Note that this account in production use is not in the sudo group. That is only the case during installation and maintenance (updates and upgrades). That is also when you need to bypass MFA for that account on the DUO admin page.
How do I achieve the requirements?
To achieve the above requirements, we need to use the Duo Unix – 2FA for SSH with PAM Support (pam_duo) module. This module supports SSH, console, and sudo access. Please don’t confuse this with their module, which only covers SSH.
If you follow the instructions and configure this module, you will achieve MFA quickly enough for use with either password or public key authentication. But we want to use both for the sudo account and only passwords for the Veeam service account. That requires some creativity beyond the standard configuration examples.
So first, let’s set up MFA for passwords. Once we have that working, we can introduce public key authentication and ensure it works. Remember, both are required.
That means that for the sudo account used to manage the repository server, we need the SSH private key, which is protected with a passphrase, the password of this account, and approve the MFA challenge by DUO.
All in all, this is a pretty secure setup.
Preparations
During setup and maintenance, we need to enable SSH on the repository. So make sure SSH is installed and enabled. We must also allow SSH traffic in the firewall since we leverage the Ubuntu User-Friendly Firewall (UFW).
# Install and enable SSH
sudo apt-get install openssh-server
sudo systemctl enable ssh
sudo systemctl status ssh
#check UFW status and enable it, and allow SSH.
sudo ufw status
sudo ufw enable
sudo ufw allow ssh
sudo ufw status ssh
Once the hardened repository goes into production, you can disable SSH and block it in the firewall.
sudo systemctl disable ssh
sudo ufw disable ssh
You can allow it in the firewall when you need it again and enable SHH. While doing so, MFA is protecting credentials. You will only bypass MFA in the DUO admin console to maintain the Veeam components during updates or upgrades. After so, you deny SSH in the firewall and disable SSH again.
Configuring pam_duo for MFA
Get started with Duo
If you are new to Duo, sign up for a Duo account. It is free for up to 10 users! So you have no excuse not to try this out in the lab.
Log in to the Duo Admin Panel. Then, in the left panel, navigate to Applications.
Click Protect an Application and scroll down to find UNIX Application in the applications list. Next, click Protect to get your integration key, secret key, and API hostname.
Keep the information, especially the secret key, confidential. Note that, If needed, you can reset it. Also, give the application a better name. I suggest the name of your repo server.
Ubuntu 20.04 pam_duo installation
If you want to compile pam_duo yourself, you need to install the OpenSSL development headers and libraries and libpam. They explain this in https://duo.com/docs/duounix. However, for my comfort, I install. That means I do not need to install any prerequisites.
To install pam_duo is I create /etc/apt/sources.list.d/duosecurity.list with the following contents: deb [arch=amd64] https://pkg.duosecurity.com/Ubuntu focal main
You can do this via the command below.
When that is done, run
curl -s https://duo.com/DUO-GPG-PUBLIC-KEY.asc | sudo apt-key add –
That should return OK. If not, you might not be able to reach that site, and you need to fix this. If you get an OK run
Sudo apt-get update && sudo apt-get install duo-unix
Congratulations, you installed the modules you need. Now you need to configure them.
TIPS:
Make a backup copy of any file you edit. For reference and to restore just in case you run into issues. For example: sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup creates a backup copy for you. You get the idea. Do this for any file you edit.
Have an authenticated SSH session with a sudo/root shell open to avoid locking yourself out. Test changes by opening a new session, leaving the one you are working on open. If you mess up SSH, console access, where you can have a second root shell open, might save you. Have options!
In the worst case, you will need to boot into the recovery console and enter the root shell to correct your mistake. That is stress, safe yourself from the hassle!
Take note that this is an excellent reminder that you need to have physical security for your servers sorted out!
Duo configuration on Ubuntu 20.04
Edit the pam_duo.conf file in /etc/security with following command
Sudo nano /etc/security/pam_duo.conf
Enter the Duo integration key, secret key, and API hostname you can find under the Unix application you created.
Leave all other settings default, especially “autopush”. That way, you can select a push notification, a call, or a text to be used. With an automatic push, you lose getting a passcode via text. Not saying a text is the best option, but you might need that option.
Configuring your ubuntu system and SSH settings for Duo MFA
First, we will make sure MFA works with your password. I will add public key authentication to the mix when I confirm this is working.
Configure /etc/ssh/sshd_config
UsePAM yes
ChallengeResponseAuthentication yes
UseDNS no
Find the above lines in /etc/ssh/sshd_config and change or uncomment the line. Changes to this file require you to restart the sshd service
Sudo systemctl restart sshd
Configure system-wide authentication
Run the following command
Sudo nano /etc/pam.d/common-auth
Edit the file as below.
# auth [success=1 default=ignore] pam_unix.so nullok_secure
auth requisite pam_unix.so nullok_secure
auth [success=1 default=ignore] /lib64/security/pam_duo.so
auth requisite pam_deny.so
auth required pam_permit.so
The red line is the one you put in a comment. The dark green lines are the ones you add. The two white ones you should find lower in your default common-auth file.
WARNING: You must use the correct path to the pam_duo.so file. In Ubuntu 20.04 this has always been /lib64/security/pam_duo.so. Make sure you verify this to prevent locking yourself out.
Enroll for MFA with Duo
When you log in the first time at the console or via SSH, after configuring Duo MFA on your host, you see a prompt to enroll with Duo via the link provided.
That process goes as follows. First, click on the link or copy it to a browser. You will get to a start page where you click on Start setup.
Select your MFA device. Here I have chosen a Mobile phone. But I usually also register my FEITIAN biometric security key to have a second option at hand.
Fill out your country, and add your mobile number.
Choose a way to verify ownership. I usually use Text me. Please enter the code they sent you. They will verify that code and put a green checkbox when successful. Click Continue.
You will get an overview of what you have done. Leave the When I log in setting at the default so you can choose how to meet the MFA challenge.
Congrats! You have enrolled your user for MFA with DUO on this server.
Now test your handy work.
Now, you can log in using a password with an MFA challenge.
Success, that was fun, right? MFA works now with a password for SSH, console, and sudo actions.
There is still more to do! Please head over to part II of this article, where we will add public key authentication and ensure this also works with MFA. On top of that, we require that you will also still need to enter the password for the sudo account. The Veeam service account should support MFA but must not use public-key authentication, but only password authentication.
