Recently whilst deploying a new Horizon Cloud on Azure Next-Gen Edge (using an AKS cluster as opposed to a single Edge VM) the deployment failed after about 20 minutes with a pretty useless "deployment failed" error. The activity logs with Horizon Cloud didn't tell anything more and a retrying the deployment did not work either. As far as we knew, all the documented pre-requisites were in place.
What was somewhat useful however where the Activity Logs from Azure, which told us a lot more about what was going on under the bonnet within the Azure AKS deployment. The "Create or Update Managed Cluster" task was failing an in the JSON view of the logs was reporting "Unable to establish outbound connection from agents" and providing a link to Troubleshoot the OutboundConnFailVMExtensionError error code (50) - Azure | Microsoft Learn
The above link listed some possible causes of this error, amongst them DNS, but we verified that was working by deploying a test VM into the same subnet. What was pertinent however was the "Traffic is blocked by firewall rules......" possible cause, as shown below.
In the case of this particular customer deployment , a NAT Gateway was not being used for internet access on the Horizon subnets, but rather a UDR [route table] was employed to route all networks to a customer managed Azure Firewall. This is typical in a hub and spoke deployment where DMZ hubs are required to filter through a hub firewall.
The assumption here was that the Azure Firewall was blocking something, rather annoyingly, because all the URL's specified in the pre-requisites list had been whitelisted, as per this link: https://docs.omnissa.com/bundle/HorizonCloudServicesUsingNextGenGuide/page/PortandProtocolRequirementsforYourHorizonCloudDeploymentinMicrosoftAzure.html
Analysis of the Azure firewall logs however showed a number of Microsoft FQDN's being blocked however. For example, mcr.microsoft.com as shown below.
As it turns out, the AKS documentation shows a long list of various Microsoft addresses that should be permitted, slightly less for private AKS clusters as is that case for Horizon Cloud on Azure Next-Gen.
Link here: Outbound network and FQDN rules for Azure Kubernetes Service (AKS) clusters
I have collapsed the above document in what's needed for Horizon Next-Gen AKS Clusters based on my understanding of Horizon Cloud, and the various URL's seen in the Azure Firewall logs, and sure enough, once whitelisted, the deployment is then successful.
See table below for a summary of the addresses to be permitted, though it's a shame Omnissa don't just list this in their own KB
| Destination FQDN | Port | Description |
|---|---|---|
| mcr.microsoft.com | HTTPS:443 | Required to access images in Microsoft Container Registry (MCR). This registry contains first-party images/charts (for example, coreDNS, etc.). These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations. |
| *.data.mcr.microsoft.com, mcr-0001.mcr-msedge.net | HTTPS:443 | Required for MCR storage backed by the Azure content delivery network (CDN) |
| management.azure.com | HTTPS:443 | Required for Kubernetes operations against the Azure API. |
| login.microsoftonline.com | HTTPS:443 | Required for Microsoft Entra authentication. |
| packages.microsoft.com | HTTPS:443 | This address is the Microsoft packages repository used for cached apt-get operations. Example packages include Moby, PowerShell, and Azure CLI. |
| acs-mirror.azureedge.net | HTTPS:443 | This address is for the repository required to download and install required binaries like kubenet and Azure CNI. |
| packages.aks.azure.com | HTTPS:443 | This address will be replacing acs-mirror.azureedge.net in the future and will be used to download and install required Kubernetes and Azure CNI binaries. |
| security.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com | HTTP:80 | This address lets the Linux cluster nodes download the required security patches and updates. |
| snapshot.ubuntu.com | HTTPS:443 | This address lets the Linux cluster nodes download the required security patches and updates from ubuntu snapshot service. |
Add comment
Comments