When using "tls-alpn-01" challenge for let's encrypt certs in kubernetes using traefik, I'm getting "acme: error: 400 Timeout during connect" - ssl

Question

I'm following the tutorial to use traefik as the ingress and ingress controller for Azure Kubernetes Service (AKS) cluster. I'm using terraform to deploy the traefik (version 1.7.24) helm chart.
resource "helm_release" "traefik" {
name = "traefik"
namespace = "traefik"
repository = "https://charts.helm.sh/stable"
chart = "traefik"
version = "1.87.2"
values = [<<EOF
loadBalancerIP: "50.100.200.300"
service:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: "aks-rg"
kubernetes:
ingressClass: traefik
ingressEndpoint:
useDefaultPublishedService: true
dashboard:
enabled: true
domain: traefik.mydomain.tld
ingress:
annotations:
kubernetes.io/ingress.class: traefik
metrics:
serviceMonitor:
enabled: true
rbac:
enabled: true
ssl:
enabled: true
enforced: true
acme:
enabled: true
email: admin#mydomain.tld
staging: true
tlsChallenge: true
entrypoint: https
ports: "443:443"
challengeType: tls-alpn-01
onHostRule: true
domains:
enabled: true
domainsList:
- main: "mydomain.tld"
- sans:
- "traefik.mydomain.tld"
EOF
]
}
The DNS records are correctly pointed to the AKS Load Balancer IP.
When I check the traefik logs, I could see that "tls-alpn-01" challenge fails with the following error:
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"mydomain.tld,traefik.mydomain.tld\" : unable to generate a certificatefor the domains [mydomain.tld traefik.mydomain.tld]: acme: Error -\u003e One or more domains had a problem:\n[mydomain.tld] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n[traefik.mydomain.tld] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","time":"2021-02-26T02:32:05Z"}
Full log is given below:
{"level":"info","msg":"Using TOML configuration file /config/traefik.toml","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"No tls.defaultCertificate given for https: using the first item in tls.certificates as a fallback.","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Traefik version v1.7.24 built on 2020-03-25_04:34:11PM","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/v1.7/basics/#collected-data\n","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Preparing server traefik \u0026{Address::8080 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:false ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc000851700} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Preparing server http \u0026{Address::80 TLS:\u003cnil\u003e Redirect:0xc0000b1b80 Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0008516a0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Starting server on :8080","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Preparing server https \u0026{Address::443 TLS:0xc000431e60 Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0008516c0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Starting provider configuration.ProviderAggregator {}","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Starting server on :80","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Starting server on :443","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"traefik\",\"IngressEndpoint\":{\"IP\":\"\",\"Hostname\":\"\",\"PublishedService\":\"traefik/traefik\"},\"ThrottleDuration\":0}","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"ingress label selector is: \"\"","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Creating in-cluster Provider client","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Starting provider *acme.Provider {\"Email\":\"admin#mydomain.tld\",\"ACMELogging\":false,\"CAServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"Storage\":\"/acme/acme.json\",\"EntryPoint\":\"https\",\"KeyType\":\"RSA4096\",\"OnHostRule\":true,\"OnDemand\":false,\"DNSChallenge\":null,\"HTTPChallenge\":null,\"TLSChallenge\":{},\"Domains\":[{\"Main\":\"mydomain.tld\",\"SANs\":[\"traefik.mydomain.tld\"]}],\"Store\":{}}","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Testing certificate renew...","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2021-02-26T02:31:38Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2021-02-26T02:31:39Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2021-02-26T02:31:39Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2021-02-26T02:31:39Z"}
{"level":"info","msg":"Register...","time":"2021-02-26T02:31:42Z"}
{"level":"info","msg":"Updated status on ingress traefik/traefik-dashboard","time":"2021-02-26T02:31:42Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2021-02-26T02:31:55Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2021-02-26T02:31:55Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2021-02-26T02:31:55Z"}
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"mydomain.tld,traefik.mydomain.tld\" : unable to generate a certificatefor the domains [mydomain.tld traefik.mydomain.tld]: acme: Error -\u003e One or more domains had a problem:\n[mydomain.tld] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n[traefik.mydomain.tld] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url: \n","time":"2021-02-26T02:32:05Z"}
I have even added a "AllowAll" rule in AKS LoadBalancer NSG (firewall). But still tls-alpn-01 validation is facing the timeout error. The ssl certificate generation doesn't happen and my website is using the default example.com expired ssl certificate.
I can confirm that telnet to port 443 of mydomain.tld also works fine.
PS: I don't want to use "dns-01" challenge for ssl certificate as the dns provider doesn't have the APIs for let's encrypt. I can't use "http-01" as these are backend servers which do not have any web server.
Any help is much appreciated. I would also like to know how the tls-alpn-01 challenge works.