Mutual TLS (mTLS) or client certificate authentication with an Azure Application Gateway and an App Service application

Introduction

This article will discuss mutual TLS (mTLS) or Client Certificate authentication with an Azure Application Gateway and Application servers/Web App. So what is it, and what does it do? It is about authentication, proving something is what it claims to be.

Figure 1: Example code by Microsoft – Certificates, TLS, and mTLS are essential to help secure your resources

With TLS, it is about the server proving to the client that it is genuinely the server with which it wants to communicate. With mTLS, the client also proves to the server that it is allowed to connect. TLS is also about encrypting information, but it is nice to know you are talking to the intended party. Otherwise, you have secure comms to a potential hostile actor, which is not very helpful.

Why and how to use mTLS

Why use mTLS

mTLS or client authentication comes into play when the server wants to allow only specific clients to connect. As with TLS, where the server proves its identity to the client via a server certificate, we can use a client certificate to verify that the client is who it claims to be.

How to use mTLS

The mTLS process happens inside of the well know TLS handshake. The client needs a certificate for mTLS, so how does the client get a certificate? Well, it is most common that a certificate authority (CA) on the server side signs a CSR from the client and hands the certificate over (in a secure manner). The client will use that certificate when the server requires mTLS, which the server then verifies. Note that it is OK to use an internal CA to do so.

Trusted devices or user

Let’s look at an example. When all users or clients in your organization can only access certain services with a valid client certificate, you can issue those certificates via an internal PKI via auto-enrolment configured via Active Directory group policy objects. Likewise, you can revoke certs when needed, etc. So you are in complete control of what certificates to trust.

Now at that scale, there are so many client certificates to check that we will not persist the details for all of them securely in a database or KeyVault, and have the service (application) verify every single one to the thumbprint. However, the clients and users are under our control, and our PKI signed their certificates. So a valid client certificate from our known and trusted CA is good enough.

So what if we need to communicate between multiple organizations? The principle stays the same. You use a trusted CA that all organizations trust. That could be a government CA or one of a trade association or institute like the European Central Bank, a federation for the notaries or insurance businesses of your country, etc. If they all have their own CA, they could decide to trust them all or choose one. The idea is to keep things manageable and easy to maintain

Specific clients that need to provide authentication

Bar highly secure environments and use cases that use mTLS with users and clients; most use cases revolve around specific client services. When not dealing with mTLS at a massive scale, you might want to check the individual properties of a client cert, such as the thumbprint. For example, a client service needs to provide a client cert to an application service, and you want to make sure no other service that might also have a valid cert from your trusted CA can connect. In that case, we will have to handle the details in our application code. That is also the case when you use a commercial CA. Therefore, you do not want every with a valid certificate from a commercial CA to connect to your sensitive application service.

Similar devices, different behaviors, and capabilities

There are many brands of reverse proxies, load balancers, or gateways. While similar, the capabilities of these regarding mTLS might differ. This article deals with Azure Application Gateway and Azure App Services and discusses the details I discovered while trying to implement them. I do not claim this is complete. However, it will help you configure it and get things working.

A short refresher on how certs work with TLS and mTLS

Before we dive into the Azure App Service and Azure Application Gateway part of this article, we will do a short refresher on how certs work with TLS and mTLS.

The TLS handshake confirms the validity of a TLS certificate. It also proves the server owns the private key for that certificate. Additionally, both sides agree on a cipher to use for encryption when the handshake completes. Finally, client authentication adds extra security.

A server typically has the certificate and the public/private key pair. The server stores the private key securely and is not viewable or visible. With mTLS, the client has the client certificate and the public/private key pair. The client stores the private key securely and is not viewable or visible.

It is paramount that your environment must protect the private key against loss or theft. A compromised private key means that your certificate is no longer trustworthy.

The extended version of the Client-authenticated TLS handshake

Warning TLS/mTLS is a highly complex subject. Hopefully, this is a correct simplification for your reference to have a usable overview of how the process works for TLS 1.2. While functional, do not expect this to be perfect. TLS 1.3 is different but slow to arrive in Azure services, so I don’t cover this yet.

1. Negotiation

• A client sends a “Client Hello” message specifying the highest TLS protocol version it supports, a random number, a list of suggested cipher suites, and compression methods.
• The server responds with a “Server Hello” message containing the chosen protocol version, a random number, cipher suite, and compression method from the choices offered by the client. The server may also send a session id as part of the message to perform a resumed handshake.
• The server sends its “Certificate” message. However, depending on the selected cipher suite, it may omit this.
• The TLS client verifies the server’s digital certificate.

– The client checks the digital signature

– The client verifies the certificate chain

– The client checks the expiration and activation dates and the validity period.

– The client checks the revocation status of the certificate

• The server sends its “Server Key Exchange” message. However, depending on the selected cipher suite, it may omit this.
• The server sends a “Certificate Request” message to request a certificate from the client.
• The server sends a “Server Hello Done” message, indicating it completed the handshake negotiation.
• The client responds with a “Certificate” message containing the client’s certificate but not its private key.
• The server verifies the client’s digital certificate.

– The server checks the digital signature

– The server verifies the certificate chain

– The server checks the expiration and activation dates and the validity period.

– The server checks the revocation status of the certificate

• The client sends a Client Key Exchange message, which may contain a PreMasterSecret, public key, or nothing. (Again, this depends on the selected cipher.) This PreMasterSecret is encrypted using the public key of the server certificate.
• The client sends a Certificate Verify message, a signature over the previous handshake messages using the client’s certificate’s private key. The server verifies the signature by using the client’s certificate’s public key. That is how it knows the client has access to the private key of the client certificate, which proves ownership of the certificate.

2. The client and server then use the random numbers and PreMasterSecret to compute a shared secret called the “master secret. Subsequent session keys are derived from this master secret and the client and server-generated random values.

3. The client now sends a “Change Cipher Spec” record. That tells the server that from now on, everything the client sends everything authenticated and encrypted.

• Finally, the client sends an “Encrypted Finished” message containing a hash and MAC over the previous handshake messages.
• The server will attempt to decrypt the client’s Finished message and verify the hash and MAC. If the decryption or verification fails, the handshake is no good, and the connection fails.

4. The server sends a “Change Cipher Spec.” That tells the client that, from now on, everything the server sends everything authenticated and encrypted.

• The server sends its own “Encrypted Finished” message.
• The client performs the identical decryption and verification procedure as the server did in step 2.

5. The TLS handshake is complete, and the HTTPS session starts. After that, data exchanged between the client and server use the agreed-on encryption.

Note that the colors green and blue return in my images to show (approximately) where the steps happen.

Now we had a refresher on how a certificate enables authentication; we can look at the challenges we face when trying to set this up with an Azure Application Gateway in front of an Azure App Service.

Mutual TLS Particularities

Before we dive into the exact details necessary to understanding and configuring mTLS with an App Service or an App Gateway, I want to draw attention to two particularities.

The TLS handshake, mTLS, and HTTP headers

Headers are an HTTP/HTTPS construct and are not part of TLS/mTLS. Remember that the TLS handshake happens before establishing an HTTPS session, so there are no headers TLS/mTLS. In an HTTP/HTTPS session, we can fill a request header with a base64 encoded string containing the client cert public information. As headers are no part of mTLS, that also means mTLS is all or nothing. Either you enable it on a lister for your website, or you don’t. That will lead to some clever workaround for excluding specific FQDN paths requiring it. More on that later.

Azure Application Gateway and mTLS

The Azure Application Gateway is like a layer-7 load balancer or a reverse proxy. However, it does not do layer-4, and it does not support passthrough mode. In passthrough mode, you could send the TLS traffic straight through the device without touching anything it cannot see. But the A Azure Application Gateway wants to leverage its capabilities. For this, it needs to terminate incoming HTTPS sessions. It then does whatever it needs to do, including the Web Application Firewall (WAF) role when enabled. It can then start a new TLS session to the backend and re-encrypt the data over an HTTPS session. It can also connect to the backend over HTTP.

With the latter, you have a theoretical benefit on the backend when using TLS offloading and avoiding the TLS overhead on your application service. Two things about that. First, I don’t see it a lot anymore because everyone requires end-to-end encryption. Two, the resource savings are not, per definition, colossal in modern systems. Now environments differ, so test this at scale when needed, especially with longer keys.

As mTLS is part of the original session which we terminated, and we create a new TLS session in which the AGW does not have the client cert to offer when the backend asks for it, there is no way the backend server can check the validity of the client cert. However, a workaround is using a request header, as mentioned before.

Armed with this information, we can look at configuring mTLS for an Azure App Service, an Azure Application Gateway, and a combination of both. I will focus on what tripped me up while learning about this.

mTLS with an Azure App Service

Microsoft has excellent documentation on configuring this in Configure TLS mutual authentication for Azure App Service. So in this article, I will focus on the little but important details on why mTLS in an Azure App Service works the way it does. First, if you want to enforce mTLS, you must set the client certificate mode to require. Then, the client must present it, or the connection will fail.

Also, note the following in Microsoft’s documentation. It is crucial!

In App Service, TLS termination of the request happens at the frontend load balancer. When forwarding the request to your app code with client certificates enabled, App Service injects an X-ARR-ClientCert request header with the client certificate. App Service does not do anything with this client certificate other than forwarding it to your app. Your app code is responsible for validating the client certificate.

For ASP.NET, the client certificate is available through the HttpRequest.ClientCertificate property.

For other application stacks (Node.js, PHP, etc.), the client cert is available in your app through a base64 encoded value in the X-ARR-ClientCert request header.

The App Service has a frontend load balancer where TLS termination happens. And that load balancer populates an X-ARR-ClientCert request header with the client certificate. App Service does NOTHING with this client certificate other than forwarding it to your application code, which is responsible for validating the client certificate. You can throw pretty much any client cert at your web application. As long as there is nothing in the code that says “no, you are not valid and allowed,” it will work. The code has to verify the cert based on a thumbprint, DN, and expiration date, to determine whether the cert is revoked and if the certificate chain is OK.

I created the below image to give you a simplified overview of the process.

Figure 2: mTLS with an Azure App Service

Remember, when we enable mTLS, it is for the entire domain FQDN of the site. So how do we exclude specific paths? That works because the App Service does not validate the client cert. It just passes the public information of the client cert through to your code in an X-ARR-ClientCert request header. That is not part of the TLS sessions, and as such, you can choose to exclude specific paths from verification in your code. For example, in the screenshot below, I excluded all paths on my site that start with /Public or /General. That is because these pages are not sensitive and do not need the validation of the client cert.

Figure 3: Set the client certificate mode and certificate exclusion paths

That same principle will come in handy later when we add a gateway. For example, we do not want a client cert check on our mtls.demo.com/HealthCheck.htlm file.

Nice right? OK, so now we have this figured out, let’s introduce an Azure Application Gateway into the mix.

mTLS and an Azure Application Gateway

What realistic options exist for mTLS when we put an Azure Application Gateway between the client and the App Service?

1. Let the Azure Application Gateway handle mTLS and be satisfied with that. While this works and is not hard to do on an Azure Gateway, it has a drawback. The Azure Application Gateway checks for a client cert with a valid certificate chain in its configuration. For this, the Azure Application Gateway uses a .cer or a .pem file containing root and intermediate certificates from the CA to sign the client certificate. It does not check a thumbprint, for example. So you have less control over what specific client cert you allow. Between the Azure Application Gateway and the App Service, you have TLS/HTPPS, but in regards to mTLS, nothing happens on the backend.
2. Do (1) and have the Azure Application Gateway send the public client certificate information to the backend. That is why they invented the X-ARR-ClientCert request header. That way, the backend has all the required information to do all possible checks. It behaves as if the client made the mTLS session directly to the AppService.

mTLS with for an Azure Application Gateway

There are two good documents that you should read on this subject. Overview of mutual authentication on Azure Application Gateway and Configure mutual authentication on Azure Application Gateway through the portal.

The significant steps are:

1. Create a certificate chain file with the root and intermediate certificates and their public info
2. Upload that file under the client authentication tab of an SSL Profile
3. Associate the SSL profile with the correct listener enabling mTLS for that service

Enabling mTLS on an Application Gateway is done by uploading a trusted client CA certificate chain as part of the client authentication portion of an SSL profile. You’ll find SSL Profiles under the SSL settings in the left navigation pane of your Azure Application Gateway.

Figure 4: Upload the entire trusted client CA certificate chain to the Application Gateway

Finally, don’t forget that you must associate that SSL profile with a listener, enabling mTLS for that service. Again, Microsoft’s documentation covers this well.

The trusted client CA certificate must contain the root CA certificate in the client certificate that you upload. If you have intermediate certificates, you can also upload those to get the complete certificate chain. Follow Microsoft Guidance and avoid multiple chains in one file. For hands-on guidance on creating the certificate chain for your client cert, follow the Export trusted client CA certificate chain for client authentication – Azure Application Gateway guidance. Note that a chain with the root and the intermediate cert(s) is enough. Also, understand that this only contains the certs’ public info. It is similar to having that info in your Windows cert store.

If you got the cert file chain wrong, I have noticed the upload will fail, so that’s a clear indicator there is a problem.

Figure 5: mTLS with for an Azure Application Gateway

The Azure Application Gateway uses mTLS verification to ensure that the client has the private key corresponding to the certificate & public key it provided. After that, it does not do much more. I am not sure if it checks the CRL for certificate revocation. It is interesting to note that the documentation states that client certificate revocation with OCSP (Online Certificate Status Protocol) is not supported, but it will be soon. Unfortunately, there is no date reference.

mTLS configured for an Azure Application Gateway and an Application Service

Like above, the Azure Application Gateway uses mTLS verification to verify that the client has the private key corresponding to the certificate & public key it provided. So far, things are like we saw before. However, in the TLS connection, without mTLS, to the App Service, we add the certificate to a header so that the application can verify and validate the certificate to the level of detail you require. That is precisely why they invented the X-ARR-ClientCert. In my opinion, this is the best solution. If you control or have a say in the code, I advise you to do this.

The confusion is that you might want to use X-ARR-ClientCert for the request header name. Sounds logical, right? Well, If you do that, the App Service drops it. The reason is, I think, that the App Service itself reserves the use of -ARR-ClientCert to pass the client certificate information into the application code when it requires mutual authentication itself.

That means there are two things we need to make this work.

1. In the rewrite rule, use a different name than X-ARR-ClientCert to pass the server variable. I chose X-ARR-ClientCert-AGW.
2. In the App Service configuration, under general, set the client certificate mode to “Allow” or “Optional,” not to “Required.” Remember, the Azure Application Gateway is not the actual mTLS client but sits between it and your app service. It does not have the client cert, so “Required” won’t work. With “Allow,” the app requests a client cert, but if it does not get one, it falls back to another form of authentication you must handle in your code. That is where you will use the client cert info you passed via the custom header. With “Optional,” the App Service does not even prompt for a client cert. The Azure App Service will block unauthenticated requests in both cases.

To achieve this, we create a Rewrite set with a rule to a custom header that contains the client cert information in PEM format.

Figure 6: The rewrite set associated with its routing rule

Figure 7: The rewrite rules with the action adding the server variable client_certificate to a custom request header

We associate that rewrite rule set with the correct route for our service to activate it.

Instead of sending the public cert info, we can also choose other server variables available for mutual authentication.

Still, you might or will run into an issue. The table above shows that the server variable client_certificate contains the client certificate in PEM format for an established SSL connection. What you find in the header you sent to App Service and the code can be an encoded URL and not Base64-encoded PEM. You must decode the URL to get a usable certificate in PEM format.

You can find an example in Configure TLS mutual authentication – Azure App Service. For our use case here, we need to change the X-ARR-ClientCert header to the chosen name of our rewrite rule, namely X-ARR-ClientCert-AGW.

Don’t forget to decode the request header with something like the HttpUtility.UrlDecode Method. What you will use for this and what your code looks like exactly depends on your programming language. The line of code here is just a sample to give you an idea.

I can refer you to this blog if you want a more elaborate code sample. That blog is by a fellow Azure Architect, Alexander Vanoosthuyse, but while I am more focused on infrastructure, he puts the dev into DevSecOps.

In a simplified image, the process looks as below.

Figure 8: mTLS configured for an Azure Application Gateway and an Application Service

It took a bit of research, trial, and error, to get everyone on the same level of understanding mTLS, how load balancers deal with mTLS and how App Services handles it. But, yes, it works!

Conclusion

I hope you will find this article helpful in your endeavors with mTLS, an Azure Application gateway, and an App Service app. I spent quite some time collaborating with some developers to determine the best solutions for various requirements. From my point of view, I am sure some of you can benefit from my learning process that led to this article.