Introduction
It is safe to say that Private Endpoints have spread like wildfire into most Azure networking designs. The adoption rate is significant and still growing. However, while very powerful and popular, I also noticed a lot of confusion around the subject as more and more people are starting to investigate using it. So in this article, we will look at what Private Endpoints do and their history.
The Private Endpoint / Private Link / Private Link Service solution came into existence mainly because of the need for private, more secure connectivity to Azure services. Since its launch, the number of Azure services supporting Private Link has grown, and the list today is extensive. You can check that out in Azure Private Link availability | Microsoft Docs.
When dinosaurs roamed the cloud world (about six years ago at the time of writing), the only way to connect to Azure services was via their public IP addresses. But unfortunately, you only had the built-in firewall feature available to filter what IP addresses were allowed to access the service.
Because of this, Microsoft went on a journey to address this and allow us to use private IP addresses to access PAAS services.
Service endpoints
First, Microsoft brought us Service Endpoints (February 2018). It provides visibility of and connectivity from Azure VNets into the PAAS service via the private address space of the VNets.
Service endpoints extend the VNet identity to the PAAS service. A service endpoint allows for more than just filtering what public IP address is allowed to access your Azure PAAS solution. As the PAAS service can now see the VNet identity, you can filter on those VNets in the built-in firewall of the service and, if required, block Internet access altogether. So the traffic is leaving your VNet, and the PaaS service is still separate from your VNet. But the PaaS service can identify traffic from your virtual network (private IP addresses) and allow that without you needing to configure public IP addresses in your resource vNet to provide filtering and going across the internet. That provides for added security, but there is even more. It also means that VNet private address spaces can overlap. Usually, when filtering on VNet IP ranges, you cannot have overlapping address spaces, or you won’t be able to identify traffic that originates from your VNet uniquely. But as service endpoints extend the VNet identity to the PAAS service, you can use its identity instead of its IP address space, which works perfectly.
On top of that, it offers route optimization for Azure traffic as you don’t need to go out via the internet and come back in via the PAAS public IP. That means no egress of Azure, and all traffic goes over the might Azure backbone network. That allows for monitoring and auditing, which is essential. Remember that the default system route of 0.0.0.0/0 routes everything to the internet. One giant exception is when the destination address is for Azure services. That is always routed directly to the service over Azure’s backbone network instead of the internet. So how does a Service Endpoint help here?
Service Endpoints optimize routing even when a UDR is present
A User Defined Routing (UDR) table can override the above behavior of the system route. But with service endpoints, even if you have a UDR defined on your subnet, which routes internet traffic back on-premises or through a firewall device, the traffic is still sent directly to the Azure Resource. Without a service endpoint, you would get sub-optimal routing.
Service Endpoints also have some risks involved. A Service Endpoint, by default, allows uncontrolled data egress from any storage account it has access to into another storage account in the same or another tenant via a compromised virtual machine, for example. Therefore, they introduced Service Endpoint policies (Azure virtual network service endpoint policies) to filter what can be egressed from and to storage accounts.
Service Endpoint Policies (image courtesy of Microsoft)
Service endpoint policies are “allow” policies associated with a subnet and block access to anything that is not explicitly allowed. See Restrict data exfiltration to Azure Storage – Azure PowerShell for a step-by-set exercise to wrap your head around this concept.
However, the most significant limitation is that using the service endpoints for traffic from your premises or third-party clouds to Azure services is impossible. That is because a Service Endpoint does not have a private IP assigned. Instead, it “only” extends the identity of the VNet into the PAAS service. So that requires Microsoft peering (Express Route) or NAT IP addresses filtered on the PAAS service firewall to come in via the internet.
So, it is easy to set up and maintain, cost-effective, and still valuable today. But for the best privacy and solution, Microsoft brought us the Private Links.
Private Link, Private Link Service, and Private Endpoints
Firstly, the difference and relation between Private Link, Private Link Service, and Private Endpoints confuse people sometimes. Microsoft has extensive documentation on the subject. But I will try to explain it as straightforwardly as possible.
Private Link
To keep things easy, think of the Azure Private Link as the construct that provides connectivity from a private IP to an Azure or 3rd party resource. Microsoft defines this as follows: “Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer-owned/partner services over a private endpoint in your virtual network.” A Private Endpoint (basically a network interface) is what lets your access a resource over the Microsoft network backbone and which you create in the VNet/SUBNET where you will consume it. Azure provides the magic under the hood for you to do this with its Azure services. What magic? Private Link!
My playing field reference design for various AppGateWay – Private Endpoint patterns & design discussions
But what about your homegrown services or 3rd party services you might want to connect to via a Private end Point? Well, that is where a Private Link Service comes into play. Microsoft defines it as follows: “Azure Private Link service is the reference to your own service that is powered by Azure Private Link.” You can create a Private Link Service with an Azure load balancer you put in front of your service. That’s how you can enable your custom-written service for Private Link and hence, lets you create a Private Endpoint to connect to your homegrown service.
Private Link Service allows you to create a Private Endpoint to your self-created service
In the case of a 3rd party, they will do that for you. That Private Link Service allows you to create a Private Endpoint to access your private service.
Snowflake is an example of a 3rd party supporting Private Endpoints via a Private Link Service (image courtesy of Microsoft)
DNS
Secondly, people struggle with DNS. Nothing new there, you might say. But Private Endpoints add more options and patterns to discuss regarding DNS name resolution. That goes from super simple to more complex permutations of Private DNS zones, custom DNS, forwarding, conditional forwarding, forward lookup zones, reverse lookup zones, and the Azure Private DNS Resolver, which went into public preview. No matter what, Private Endpoints require DNS, and depending on your current configuration, that can get a bit involved.
You can approach DNS name resolution in a decentralized, a centralized manner, or a combination of both (hybrid). I usually use the centralized approach by default. However, decentralized has its place and benefits; in some environments, that’s what I use. And I allow both centralized and decentralized solutions to become hybrid solutions when and where needed. We will leave this topic for future articles.
Benefits of Private Endpoints
You can think of a Private Endpoint as an extension from the service you want to reach into your VNet, where it gets a vNIC with a private IP. Doing so allows private access to Azure services with a public IP address via the Private Endpoints virtual network interface within your own VNet, which has an IP address from your subnet range. Traffic to the resource is like accessing a resource in your own VNet Private Endpoints.
Private endpoints can protect against data exfiltration by the endpoint providing access to only a specific instance of a PaaS service.
It helps protect your data as you can block any access to your resources except via the private endpoints you expose. As a result, it has become a key component of defense-in-depth in highly regulated and security-aware environments.
You can expose resources in Azure to clients that run in peered networks, other regions, or Azure AD tenants, and even to on-premises or other clouds as long as there is connectivity in some form (VPN, Express Route, …)
It allows to limit egress to only your specific PaaS services and helps mitigate the risk of data leakage
Exposing only mapped resources reduces the threat of data exfiltration (image courtesy of Microsoft)
Do note that different Azure services can behave differently with a Private Endpoint. For example, with an Application Service, you will see that using a Private Endpoint will block all internet access to the application. However, with a storage account, you can still allow one or more VNets or specific NAT IP addresses from on-premises, 3rd parties, etc.
Drawbacks & solutions
On August 17th, 2022, Microsoft mitigated most of the drawbacks with the general availability of User-defined routes support for private endpoints, and general availability of network security groups support for private endpoints. One of the drawbacks is/was the routing complexity with UDRs when sending private endpoint data via a firewall’s private IP for logging and monitoring. Now, not all drawbacks are gone, but things have gotten better. It is a bit funky as this must be the first time a shorter prefix in an Azure routing table invalidates a longer prefix, which is the system /32 entries for the private endpoints. Do note that it only works when “PrivateEndpointNetworkPolicies” is enabled. See Manage network policies for private endpoints. Another is/was the lack of support for Network Security Groups (NSG). That also only works as you would expect as long as “PrivateEndpointNetworkPolicies” is enabled.
Some cloud purists are upset
Apart from confusion, there has also been some pushback from the cloud camp that thinks everything in the cloud should be accessible over public IP addresses. Even more, only over public IP addresses. I am not throwing stones here, but more developers are in this camp than infrastructure people. They feel this way because it is easy and keeps them from dealing with uncollaborative network admins. For others, “private networking” is a return to the evils of on-premises solutions that bank on “isolation” to provide security instead of turning every endpoint into a self-sufficient fortress to protect itself from malicious use. The truth is seldom this clear cut. And while purists love the idea that black and white is how it should be, endless shades of grey are the colors I see. I like Private Endpoints. You know, I need to solve real work problems as well as I can, and I will use the best tools to do so within the opportunities and limitations of the environment. I do not make a living evangelizing some grand principle over another. Unless they don’t make sense, all of them have their time and place. DevSecOps addresses most concerns and issues anyone has around the subject. But that takes effort and time, which is less popular than dogmatic stances on specific concepts and patterns.
Conclusion
Private endpoints are very powerful to access resources in Azure more securely. They serve a real-world need, which is why it has become so popular. The fact is, you cannot ignore them anymore. That alone shows how rapidly the adoption went. Eighteen months ago, I had people shrugging them off. Many of them were developers in the cloud purist camp. But today, with ever stricter rules regarding privacy, security, and the move towards zero trust, they play an essential part in most designs. In future articles, we will look at how we can resolve the IP addresses of the resources private endpoints expose. That’s the DNS side of things, which you should never ignore as you cannot consume what you cannot find. But that is for future articles.
