Azure tutorials unclear to this newcomer (part 1)

[ian-barlow]

Looking for some advice on getting started with ExternalDNS on AKS (rbac enabled, Azure CNI network) that doesn't assume too much prior knowledge of helm or nginx. I'm failing to get things running so looking for hints, or straight up help, on where things might be going wrong along the way.

Install NGINX Ingress Controller

The helm chart mentioned... stable/nginx-ingress has been depreciated. Pulling together from multiple sources I try:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
kubectl create namespace ingress-nginx

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.replicaCount=2 \
  --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
  --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
  --set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux \
  --set rbac.create=true \
  --set controller.publishService.enabled=true

Results look plausible with these services being created in a ingress-nginx namespace, where 10.33.0.0/16 is the service network:

PS> kubectl get service --namespace ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.33.104.25    52.138.119.34   80:30290/TCP,443:32067/TCP   21m
ingress-nginx-controller-admission   ClusterIP      10.33.155.149   <none>          443/TCP                      21m

But this isn't quite what we're looking for, we want to publish a load balancer with an address from a Vnet associated with the AKS cluster - so our private services are accessible only from our Azure network environment. To do that we need to grant the AKS cluster (an MI in our case but could be an SP) rights on the Vnet as described here.

PS> $id=az network vnet show -g <Vnet RG> -n <Vnet> --query id
PS> az role assignment create --assignee-object-id <MI (or SP)> --role "Network Contributor" --scope $id

And finally we pass an additional flag to the helm installer, which we do on the command line:

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --set controller.replicaCount=2 \
  --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
  --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
  --set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux \
  --set rbac.create=true \
  --set controller.publishService.enabled=true \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"="true"

This again looks plausible and the "External IP" is from the desired Vnet range (10.36.0.0/16).

PS /home/ian> kubectl get service --namespace ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.33.229.239   10.36.252.115   80:32227/TCP,443:31484/TCP   2m9s
ingress-nginx-controller-admission   ClusterIP      10.33.172.74    <none>          443/TCP                      2m9s

Provision Azure Private DNS and Configure service principal for managing the zone

We create the SP for ExternalDNS

PS> az ad sp create-for-rbac --skip-assignment -n http://sp-iced-externaldns
{
  "appId": "11111111-1111-1111-1111-111111111111",
  "displayName": "sp-iced-externaldns",
  "name": "http://sp-iced-externaldns",
  "password": "<The SP's secret>",
  "tenant": "22222222-2222-2222-2222-222222222222"
}

Then we create an RG and populate it with a Private DNS Zone. The instructions to create a 'random' RFC1918 don't seem to fit either a internal or an external load balancer situation and the Vnet we're deploying the internal-ingress too is already in existence and has a load balancer deployed to it. So here we veer away from the tutorial a bit by using our own Vnet.

PS> az group create -n externaldns -l eastus2
PS> az network private-dns zone create -g externaldns -n example.com

We linked the private DNS zone to the AKS Vnet via the portal, our AKS Vnet is located in another RG.

PS> $rgid = az group show --name externaldns --query id -o tsv
PS> $zoneid = az network private-dns zone show --name example.com -g externaldns --query id -o tsv
PS> az role assignment create --role "Reader" --assignee <appId GUID> --scope $rgid  
PS> az role assignment create --role "Private DNS Zone Contributor" --assignee <appId GUID> --scope $zoneid 

There aren't any particular test we can do at this point to check we have everything right but we've checked and double checked and for the purposes of testing granted both the AKS MI and ExternalDNS SP the Owner role in the subscription being used for testing. Then we continue and deploy ExternalDNS...

Deploy ExternalDNS

First we construct the json file named azure.json containing details of the RG of the Private DNS Zone resource, tenant ID, subscription ID and the SP ID and secret for External-DNS.

{
        "tenantId": "<AureAD tenant ID>",                    # e.g. "22222222-2222-2222-2222-222222222222"
        "subscriptionId": "<Azure Subscription ID>",         # e.g. "33333333-3333-3333-3333-333333333333"
        "resourceGroup": "<RG for Private DNS Zone>",        # e.g. "externaldns"
        "aadClientId": "<The SP appId>",                     # e.g. "11111111-1111-1111-1111-111111111111"
        "aadClientSecret": "<The SP's secret>"
}

Then we convert this to a generic secret in AKS named "azure-config-file"

PS> kubectl create secret generic azure-config-file --from-file=azure.json

As our AKS cluster has RBAC enabled and we're standing up ExternalDNS for the cluster we choose the "Manifest (for clusters with RBAC enabled, cluster access)" and copy this to a file externaldns.yaml, then modify the final 'args' section, inserting the Azure subscription ID and the debug flags:

        args:
        - --source=service
        - --source=ingress
        - --domain-filter=example.com
        - --provider=azure-private-dns
        - --azure-resource-group=externaldns
        - --azure-subscription-id=<Azure Subscription ID>
        - --log-level=debug

Then run ExternalDNS:

PS> kubectl create -f externaldns.yaml
serviceaccount/externaldns created
clusterrole.rbac.authorization.k8s.io/externaldns created
clusterrolebinding.rbac.authorization.k8s.io/externaldns-viewer created
deployment.apps/externaldns created

Problem is this is a great long homerun deployment and at least for us it's not working out. ExternalDNS starts and runs but reports a 403 in the logs for the pod at startup and approximately every 60s thereafter. The object id referenced in the error is not something I can find in my tenant right now - it's not the tenant ID, Subs ID, not the SP ID, not the MI's object or app ID, can't find it in AzureAD at all.

PS> kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
externaldns-56b8b6956c-jqrgn   1/1     Running   0          2m36s

PS> kubectl logs -f externaldns-56b8b6956c-jqrgn
time="2021-02-11T15:30:49Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s 
---snip--
File: DigitalOceanAPIPageSize:50}"
time="2021-02-11T15:30:49Z" level=info msg="Instantiating new Kubernetes client"
time="2021-02-11T15:30:49Z" level=debug msg="apiServerURL: "
time="2021-02-11T15:30:49Z" level=debug msg="kubeConfig: "
time="2021-02-11T15:30:49Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-02-11T15:30:49Z" level=info msg="Created Kubernetes client https://10.33.0.1:443"
time="2021-02-11T15:30:56Z" level=debug msg="Retrieving Azure Private DNS zones for Resource Group 'externaldns'"
time="2021-02-11T15:30:57Z" level=error msg="privatedns.PrivateZonesClient#ListByResourceGroup: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '318ad65a-aa6d-4e9e-9a94-baf2e518fb7d' with object id '318ad65a-aa6d-4e9e-9a94-baf2e518fb7d' does not have authorization to perform action 'Microsoft.Network/privateDnsZones/read' over scope '/subscriptions/33333333-3333-3333-3333-333333333333/resourceGroups/externaldns/providers/Microsoft.Network' or the scope is invalid. If access was recently granted, please refresh your credentials.\""

In debugging a permission issue we've granted the AKS MI and the ExternalDNS SP owner rights on the subscription - no difference. We don't speak 'go' but even took a pass at the source code but that's a little over my head.

Would really appreciate any hints folks might have as to getting this working?

[ian-barlow]

Tried bumping the version to 0.7.6 by amending the externaldns.yaml file. No change.
Tried creating a DNS Zone (i.e. not Private DNS Zone), switching to the provider=azure and bumping version to 0.7.6. That specific combo finds the DNS zone file that's been created - doesn't do anything with it (yet due to filtering) debug below. Wishing we knew enough to spot the differences between the two providers but I see it as hopeful.

PS /home/ian/externaldns> kubectl logs -f externaldns-b94dbd56b-r6vlw
time="2021-02-11T20:46:23Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s 
---snip---
DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME]}"
time="2021-02-11T20:46:23Z" level=info msg="Instantiating new Kubernetes client"
time="2021-02-11T20:46:23Z" level=debug msg="apiServerURL: "
time="2021-02-11T20:46:23Z" level=debug msg="kubeConfig: "
time="2021-02-11T20:46:23Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-02-11T20:46:23Z" level=info msg="Created Kubernetes client https://10.33.0.1:443"
time="2021-02-11T20:46:25Z" level=info msg="Using client_id+client_secret to retrieve access token for Azure API."
time="2021-02-11T20:46:30Z" level=debug msg="Retrieving Azure DNS zones for resource group: externaldns."
time="2021-02-11T20:46:31Z" level=debug msg="Found 1 Azure DNS zone(s)."
time="2021-02-11T20:46:31Z" level=debug msg="Retrieving Azure DNS records for zone 'dcflex.xyz'."
time="2021-02-11T20:46:31Z" level=debug msg="Skipping return of record dcflex.xyz because it was filtered out by the specified --domain-filter"
time="2021-02-11T20:46:31Z" level=debug msg="No endpoints could be generated from service ingress-nginx/ingress-nginx-controller-admission"
time="2021-02-11T20:46:31Z" level=debug msg="No endpoints could be generated from service ingress-nginx/ingress-nginx-controller"
time="2021-02-11T20:46:31Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"
time="2021-02-11T20:46:31Z" level=debug msg="No endpoints could be generated from service kube-system/metrics-server"
time="2021-02-11T20:46:31Z" level=debug msg="No endpoints could be generated from service kube-system/npm-metrics-cluster-service"
time="2021-02-11T20:46:31Z" level=debug msg="No endpoints could be generated from service default/kubernetes"
time="2021-02-11T20:46:31Z" level=debug msg="Retrieving Azure DNS zones for resource group: externaldns."
time="2021-02-11T20:46:31Z" level=debug msg="Found 1 Azure DNS zone(s)."