Traefik Error forwarding EOF / Bad Gateway (Let's Encrypt Related) - ssl

I'm attempting to set up Cryptpad via Docker, reachable through Traefik, on a public server. I have Traefik set up as well as Cryptpad but so far, upon navigating to http://cryptpad.myserver.com (which redirects to https, as specifically configured) I get a Bad Gateway error in the browser and the following error in the treafik container's logs:
level=warning msg="Error forwarding to https://172.19.0.2:3000, err: EOF"
Further, another issue which may be related is that it does not seem that Let's Encrypt is able to issue certs for neither cryptpad.myserver.com nor monitor.myserver.com (which I configured as per the instructions here https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-16-04).
EDIT:
The Bad Gateway issue does seem to be stemming from a failure to create a valid cert, as I can reach Cryptpad through Traefik fine over plain HTTP (after turning off the related HTTPS configurations of course). The title of this question has been edited accordingly to reflect this relation.
For example, when attempting to reach https://monitor.myserver.com, I get the following errors in the traefik container's logs:
time="2018-01-10T13:53:37Z" level=info msg="Server configuration reloaded on :9080"
time="2018-01-10T13:53:37Z" level=info msg="Server configuration reloaded on :9443"
time="2018-01-10T13:53:37Z" level=debug msg="LoadCertificateForDomains [monitor.myserver.com]..."
time="2018-01-10T13:53:37Z" level=debug msg="Look for provided certificate to validate [monitor.myserver.com]..."
time="2018-01-10T13:53:37Z" level=debug msg="No provided certificate found for domains [monitor.myserver.com], get ACME certificate."
time="2018-01-10T13:53:37Z" level=debug msg="Loading ACME certificates [monitor.myserver.com]..."
time="2018-01-10T13:53:37Z" level=warning msg="A new release has been found: 1.4.6. Please consider updating."
time="2018-01-10T13:53:37Z" level=error msg="map[monitor.myserver.com:[monitor.myserver.com] acme: Could not determine solvers]"
time="2018-01-10T13:53:37Z" level=error msg="Error getting ACME certificates [monitor.myserver.com] : Cannot obtain certificates map[monitor.myserver.com:[monitor.myserver.com] acme: Could not determine solvers]+v"
Similarly, when attempting to reach http://cryptpad.myserver.com, the following ssl errors are logged (ending with the EOF / Bad Gatewway error noted above):
time="2018-01-10T11:59:18Z" level=info msg="Server configuration reloaded on :9443"
time="2018-01-10T11:59:18Z" level=info msg="Server configuration reloaded on :9080"
time="2018-01-10T11:59:18Z" level=debug msg="LoadCertificateForDomains [cryptpad.myserver.com]..."
time="2018-01-10T11:59:18Z" level=debug msg="Look for provided certificate to validate [cryptpad.myserver.com]..."
time="2018-01-10T11:59:18Z" level=debug msg="No provided certificate found for domains [cryptpad.myserver.com], get ACME certificate."
time="2018-01-10T11:59:18Z" level=debug msg="Loading ACME certificates [cryptpad.myserver.com]..."
time="2018-01-10T11:59:18Z" level=error msg="map[cryptpad.myserver.com:[cryptpad.myserver.com] acme: Could not determine solvers]"
time="2018-01-10T11:59:18Z" level=error msg="Error getting ACME certificates [cryptpad.myserver.com] : Cannot obtain certificates map[cryptpad.myserver.com:[cryptpad.myserver.com] acme: Could not determine solvers]+v"
time="2018-01-10T11:59:52Z" level=debug msg="Look for provided certificate to validate [cryptpad.myserver.com]..."
time="2018-01-10T11:59:52Z" level=debug msg="No provided certificate found for domains [cryptpad.myserver.com], get ACME certificate."
time="2018-01-10T11:59:52Z" level=debug msg="Challenge GetCertificate cryptpad.myserver.com"
time="2018-01-10T11:59:52Z" level=debug msg="ACME got nothing cryptpad.myserver.com"
time="2018-01-10T11:59:52Z" level=debug msg="Look for provided certificate to validate [cryptpad.myserver.com]..."
time="2018-01-10T11:59:52Z" level=debug msg="No provided certificate found for domains [cryptpad.myserver.com], get ACME certificate."
time="2018-01-10T11:59:52Z" level=debug msg="Challenge GetCertificate cryptpad.myserver.com"
time="2018-01-10T11:59:52Z" level=debug msg="ACME got nothing cryptpad.myserver.com"
time="2018-01-10T11:59:52Z" level=warning msg="Error forwarding to https://172.19.0.2:3000, err: EOF"
The following is the docker-compose.yml file for Traefik and its traefik.toml file (both configured by consulting the guide already mentioned above [via Digital Ocean] and Traefik's own here https://docs.traefik.io/user-guide/docker-and-lets-encrypt/):
version: '2'
services:
traefik:
image: traefik
networks:
- proxy
ports:
- "9080:9080"
- "9443:9443"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /opt/traefik/traefik.toml:/traefik.toml
- /opt/traefik/acme.json:/acme.json
labels:
- "traefik.frontend.rule=Host:monitor.myserver.com"
- "traefik.port=8080"
container_name: traefik
networks:
proxy:
external: true
traefik.toml:
checkNewVersion = true
logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]
[entryPoints]
[entryPoints.http]
address = ":9080"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":9443"
[entryPoints.https.tls]
[retry]
[acme]
email = "example#myserver.com"
storage = "acme.json"
entryPoint = "https"
onHostRule = true
onDemand = false
[web]
address = ":8080"
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "myserver.com"
watch = true
exposedbydefault = false
And here are the .env and docker-compose.yml files for Cryptpad, which I received and edited according to https://github.com/xwiki-labs/cryptpad/blob/master/docs/cryptpad-docker.md and the guides previously mentioned:
VERSION=latest
USE_SSL=true
STORAGE='./storage/file'
LOG_TO_STDOUT=true
docker-compose.yml
version: '2'
services:
cryptpad:
build:
context: .
args:
- VERSION=${VERSION}
image: "xwiki/cryptpad:${VERSION}"
hostname: cryptpad
labels:
- "traefik.backend=cryptpad"
- "traefik.docker.network=proxy"
- "traefik.frontend.rule=Host:cryptpad.myserver.com"
- "traefik.enable=true"
- "traefik.port=3000"
- "traefik.frontend.passHostHeader=true"
- "traefik.default.protocol=https"
environment:
- USE_SSL=${USE_SSL}
- STORAGE=${STORAGE}
- LOG_TO_STDOUT=${LOG_TO_STDOUT}
restart: always
volumes:
- ./data/files:/cryptpad/datastore:rw
- ./data/customize:/cryptpad/customize:rw
networks:
- proxy
- default
expose:
- "3000"
networks:
proxy:
external: true
Any help would be greatly appreciated. & of course I can provide much more details if necessary.

I think you are running into this issue:
https://community.letsencrypt.org/t/solution-client-with-the-currently-selected-authenticator-does-not-support-any-combination-of-challenges-that-will-satisfy-the-ca/49983
Apparently letsencrypt has disabled TLS-SNI-01 because of security issues. Here is the link to the issue: https://community.letsencrypt.org/t/2018-01-09-issue-with-tls-sni-01-and-shared-hosting-infrastructure/49996
It looks like that letsencrypt needs a few days, before they can enable it again.

Related

Traefik and certbot cannot generate certificate on OVH

On Docker I installed Portainer and then create new Stack with Traefik (certbot).
I logged into my OVH and generate API keys for certbot.
My docker-compose file looks like bellow.
I use this manual for OVH https://doc.traefik.io/traefik/user-guides/docker-compose/acme-dns/
I want to certbot generate wildcard certificate for my domain.
version: "3.8"
volumes:
trafeik_crt:
networks:
frontend_proxy:
driver: bridge
enable_ipv6: false
ipam:
driver: default
config:
- subnet: 172.20.5.0/26
services:
traefik:
container_name: Traefik
image: traefik:latest
environment:
TRAEFIK_LOG_LEVEL: 'DEBUG'
TRAEFIK_GLOBAL_CHECKNEWVERSION: 'true'
TRAEFIK_PROVIDERS_DOCKER: 'true'
TRAEFIK_PROVIDERS_DOCKER_EXPOSEDBYDEFAULT: 'true'
TRAEFIK_API: 'true'
TRAEFIK_API_DASHBOARD: 'true'
TRAEFIK_API_INSECURE: 'true'
OVH_ENDPOINT: 'ovh-eu'
OVH_APPLICATION_KEY: 'my_app_key'
OVH_APPLICATION_SECRET: 'my_secret'
OVH_CONSUMER_KEY: 'my_cons_key'
TRAEFIK_CERTIFICATESRESOLVERS_certbot: 'true'
TRAEFIK_CERTIFICATESRESOLVERS_certbot_ACME_DNSCHALLENGE: 'true'
TRAEFIK_CERTIFICATESRESOLVERS_certbot_ACME_DNSCHALLENGE_PROVIDER: 'ovh'
# TRAEFIK_CERTIFICATESRESOLVERS_certbot_ACME_CASERVER: 'ttps://acme-v02.api.letsencrypt.org/directory'
TRAEFIK_CERTIFICATESRESOLVERS_certbot_ACME_EMAIL: 'info#mydomain.com'
TRAEFIK_CERTIFICATESRESOLVERS_certbot_ACME_STORAGE: '/letsencrypt/acme.json'
TRAEFIK_ENTRYPOINTS_web: 'true'
TRAEFIK_ENTRYPOINTS_webs: 'true'
TRAEFIK_ENTRYPOINTS_web_ADDRESS: ':80'
TRAEFIK_ENTRYPOINTS_webs_ADDRESS: ':443'
TRAEFIK_ENTRYPOINTS_web_HTTP_REDIRECTIONS_ENTRYPOINT_TO: 'webs'
TRAEFIK_ENTRYPOINTS_web_HTTP_REDIRECTIONS_ENTRYPOINT_PERMANENT: 'true'
TRAEFIK_ENTRYPOINTS_web_HTTP_REDIRECTIONS_ENTRYPOINT_SCHEME: 'https'
hostname:
srv_traefik1
ports:
- 8051:80/tcp # Trafeik HTTP
- 8052:8080/tcp # Trafeik WebUI
- 4351:443/tcp # Trafeik HTTPS
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- trafeik_crt:/letsencrypt
networks:
frontend_proxy:
ipv4_address: 172.20.5.2
dns:
- 172.16.25.1;
When i enable DEBUG logs I got below error
level=debug msg="Creating load-balancer" entryPointName=web serviceName=httpd-httpd-main routerName=httpd_main#docker
level=debug msg="Creating server 0 http://172.20.5.3:80" entryPointName=web serviceName=httpd-httpd-main serverName=0 routerName=httpd_main#docker
level=debug msg="child http://172.20.5.3:80 now UP"
level=debug msg="Propagating new UP status"
level=debug msg="Added outgoing tracing middleware httpd-httpd-main" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=httpd_main#docker
level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=webs middlewareName=traefik-internal-recovery
level=debug msg="Adding route for mydomain.com with TLS options default" entryPointName=web
level=debug msg="Adding route for mydomain.com with TLS options default" entryPointName=webs
level=debug msg="Trying to challenge certificate for domain [mydomain.com] found in HostSNI rule" rule="Host(`mydomain.com`)" providerName=certbot.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=httpd_main#docker
level=debug msg="Looking for provided certificate(s) to validate [\"mydomain.com\"]..." routerName=httpd_main#docker rule="Host(`mydomain.com`)" providerName=certbot.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory"
level=debug msg="Domains [\"mydomain.com\"] need ACME certificates generation for domains \"mydomain.com\"." providerName=certbot.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=httpd_main#docker rule="Host(`mydomain.com`)"
level=debug msg="Loading ACME certificates [mydomain.com]..." rule="Host(`mydomain.com`)" providerName=certbot.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=httpd_main#docker
level=debug msg="Building ACME client..." providerName=certbot.acme
level=debug msg="https://acme-v02.api.letsencrypt.org/directory" providerName=certbot.acme
level=error msg="Unable to obtain ACME certificate for domains \"mydomain.com\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:44201->127.0.0.11:53: i/o timeout" ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=httpd_main#docker rule="Host(`mydomain.com`)" providerName=certbot.acme
The error is in the last line of your debug output:
Get "https://acme-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:44201->127.0.0.11:53: i/o timeout
To generate your certificates, your Traefik container try to reach acme-v02.api.letsencrypt.org without success. It tries to use a local DNS server 127.0.0.11:53 but fails to contact it.
You need to fix this DNS issues to fix your certificate generation
Have a look at this SO answer, it can guide you to fix this issue:
ACME certificates timeout with traefik

Kubernetes Dashboard TLS cert issue

I am deploying the standard Kubernetes Dashboard (Jetstack) to the K3s cluster I have deployed on my RPI cluster, I am using lets-encrypt to provision the TLS cert and setting the following options on the dashboard deployment:
spec:
args:
- --tls-cert-file=/tls.crt
- --tls-key-file=/tls.key
volumeMounts:
- mountPath: /certs
name: kubernetes-dashboard-certs
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: cluster.smigula.com-tls
The cert is valid when I visit the URL in my browser, however the pod raises this:
http: TLS handshake error from 10.42.0.8:43704: remote error: tls: bad certificate
It appears that the ingress is terminating the TLS connection when the pod expects to terminate it. What should I do? Thanks.
[edit] I changed the resource kind from Ingress to IngressRouteTCP and set passthrough: true in the tls: section. Still same result.

Traefik 2.0 TLS TCP passthrough

I have a VM0 where Traefik is running as a docker and two target system VM1 and VM2 which both have a webserver running.
All domainA.com requests should go to VM1 via TCP router and tls passthrough, because this webservice is handling the certificates itself.
All domainB.com requests should go to VM2 via http router and Traefik should generate the tls certs for this domain.
My problem now is, as soon as I add any tls config to the http router, it seems tcp passthrough doesn't work anymore. In the logs I see this messages:
time="2020-03-15T21:46:18Z" level=debug msg="Serving default certificate for request: \"subdomain.DomainA.com\""
time="2020-03-15T21:46:18Z" level=debug msg="http: TLS handshake error from 192.168.1.116:55103: remote error: tls: unknown certificate"
time="2020-03-15T21:46:18Z" level=debug msg="Serving default certificate for request: \"subdomain.DomainA.com\""
time="2020-03-15T21:46:18Z" level=debug msg="http: TLS handshake error from 192.168.1.116:55104: remote error: tls: unknown certificate"
And if I visit the website through Traefik, it shows me a self signed certificate from Traefik.
If I remove then all tls settings under the http router, passthrough is working again.
My Dynamic File:
http:
routers:
HTTProuter0:
rule: "HostRegexp(`{subdomain:[a-z]+}.domainA.com`)"
service: "domainA"
entryPoints:
- "websecure"
tls:
certResolver: "myresolver"
domains:
- main: "domainA.com"
sans:
- "*.domainA.com"
services:
domainA:
loadBalancer:
servers:
- url: "https://192.168.1.13:4433"
tcp:
routers:
TCProuter0:
rule: "HostSNI(`*`)"
service: "domainB"
entryPoints:
- "websecure"
tls:
passthrough: true
services:
domainB:
loadBalancer:
servers:
- address: "192.168.1.11:443"
My static file:
serversTransport:
insecureSkipVerify: true
entryPoints:
web:
address: ":80"
websecure:
address: ":443"
spain:
address: ":4443"
certificatesResolvers:
myresolver:
acme:
email: email#email.com
storage: /etc/traefik/acme/acme.json
dnsChallenge:
provider: cloudflare
delayBeforeCheck: 60
resolvers:
- "1.1.1.1:53"
- "8.8.8.8:53"
api:
insecure: true
dashboard: true
providers:
docker: {}
file:
directory: /etc/traefik/config
watch: true
log:
filePath: /etc/traefik/traefik.log
level: DEBUG
I'm stuck at this problem now for hours. I am not sure if it is a bug or if I do something wrong?
Any help would be very appreciated!
Thanks a lot
I found the problem. Unbelievable I wasted so much time for this...
It seems Traefik does not support wildcards in combination with domains in HostSNI.
HostSNI(`*`) => Works
HostSNI(`*.mydomain.com`) => DOESN'T WORK !!!!
HostSNI(`www.mydomain.com`,`web.mydomain.com`) => Work
So I added every domain explicit and now it works.

How to tell Traefik 2.1 to use a user-defined wildcard certificate for a router?

I'm trying to get Traefik to use a manually configured wildcard certificate for all routers that have a matching Host rule. I assumed Traefik would try to find a certificate based on the domains used in the Host rule, but it always uses the default generated certificate instead.
traefik.yml
global:
checkNewVersion: false
sendAnonymousUsage: false
log:
level: DEBUG
entryPoints:
web:
address: ":80"
web-secure:
address: ":443"
providers:
file:
directory: /etc/traefik/conf
watch: true
dynamic conf:
http:
routers:
test:
rule: "Host(`subdomain.wildcard.domain.tld`)"
entryPoints: ["web"]
service: service-test
middlewares: ["https_redirect"]
test-secure:
rule: "Host(`subdomain.wildcard.domain.tld`)"
entryPoints: ["web-secure"]
service: service-test
tls: {}
services:
service-test:
loadBalancer:
servers:
- url: "http://helloworld"
middlewares:
https_redirect:
redirectScheme:
scheme: https
permanent: true
Traefik is running inside a Docker container, based on the log it can see and use the mounted certificate files:
time="2020-03-04T10:44:13Z" level=debug msg="No store is defined to add the certificate <...>, it will be added to the default store."
time="2020-03-04T10:44:13Z" level=debug msg="Adding certificate for domain(s) wildcard.domain.tld,*.wildcard.domain.tld"
time="2020-03-04T10:44:13Z" level=debug msg="No default certificate, generating one"
However, when doing a curl request, the default certificate is used:
curl -k -v --header "Host: subdomain.wildcard.domain.tld" https://192.168.173.143/
* Server certificate:
* subject: CN=TRAEFIK DEFAULT CERT
* start date: Mar 4 10:44:13 2020 GMT
* expire date: Mar 4 10:44:13 2021 GMT
* issuer: CN=TRAEFIK DEFAULT CERT
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
When specifyng the certificate as the default certificate, Traefik does use it, making it more obvious that something is wrong when it tries to match the routers to fitting certificates.
tls:
certificates:
- certFile: /ssl/wildcard.crt
keyFile: /ssl/wildcard.key
stores:
default:
defaultCertificate:
certFile: /ssl/wildcard.crt
keyFile: /ssl/wildcard.key

Pass traffic directly to container to answer LetsEncrypt challenge in Traefik

I have a container ('matrix'), based on https://github.com/silvio/docker-matrix (though that's probably not important).
It runs a service on port 8448 and 3478 (not 80 or 443).
Without running traefik, and only running the 'matrix' container, inside of the 'matrix' container, I can run LetsEncrypt's certbot, which requests tells LetsEncrypt to try to contact me on port 443 and provide a ssl cert, like so:
certbot certonly --standalone --test-cert --email admin#amazing.site --agree-tos -d m.amazing.site
The challenge is made, the challenge is accepted, certs get saved in dir /etc/letsencrypt in the container.
Ok now I want to do this when running Traefik.
I pass my parameters for Traefik for this container in my docker-compose file, like so:
labels:
- "traefik.acme=false"
- "traefik.enable=true"
- "traefik.backend=matrix"
- "traefik.frontend.rule=Host:m.amazing.site"
- "traefik.port=443"
restart: always
expose:
- 443
ports:
- "8448:8448"
- "3478:3478"
When I run the challenge in the container (same command as above)
certbot certonly --standalone --test-cert --email admin#amazing.site --agree-tos -d m.amazing.site
I get the following in my Traefik logs
time="2017-07-14T01:04:35Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 453.949201ms"
time="2017-07-14T01:04:35Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 808.788592ms"
time="2017-07-14T01:04:36Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 1.138006833s"
time="2017-07-14T01:04:37Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 2.436785791s"
time="2017-07-14T01:04:40Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 3.055167113s"
time="2017-07-14T01:04:43Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 4.856677044s"
time="2017-07-14T01:04:48Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 7.544878611s"
time="2017-07-14T01:04:55Z" level=error msg="Error getting cert: Cannot find challenge cert for domain b374a9118f855cacdb0096846a3dfa0c.f7c92b61d040f9ba250f14cc533ba4b8.acme.invalid, retrying in 6.313970727s"
time="2017-07-14T01:05:01Z" level=error msg="Error getting cert: Cannot find challenge cert for domain 8b1e27af665c4676b47236f25c3ccc73.1313b1cc8ceaaa7467ba2e5845c08fde.acme.invalid"
time="2017-07-14T01:05:01Z" level=debug msg="ACME got nothing 8b1e27af665c4676b47236f25c3ccc73.1313b1cc8ceaaa7467ba2e5845c08fde.acme.invalid"
2017/07/14 01:05:01 server.go:2753: http: TLS handshake error from 66.133.109.36:55264: EOF
Note these are real logs. No mention of the actual domain name I am trying to verify.
What am I doing wrong?
You cannot disable acme for a particular host out of the box since acme installs a route that is executed before all the rules.
I figured out a way to solve this:
https://gist.github.com/micw/67faf5cd3d4a6f64568ca2bb9a051230
Best regards,
Michael.