I'm setting up a k3s cluster for local development.
To be clear, I do not have a public IP address.
At this moment I'm looking for a solution to get the certificate process automated (via cert-manager).
In order to get this to work I've did the following:
Deployed k3s
Deployed cert-manager
Deployed traefik
Purchased a domain
Created a cloudflare account and added the domain there
Created an API token to do the acme challenge (based on https://cert-manager.io/docs/configuration/acme/dns01/cloudflare/)
Created a simple test website
When a add the test website I get the following error:
Found no Zones for domain _acme-challenge.. (neither in
the sub-domain noir in the SLD) please make sure your domain-entries
in the config are correct and the API is correctly setup with
Zone.read rights.
I have the following configuration:
ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
email: my#emailaddress.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt
solvers:
- dns01:
cloudflare:
email: my#emailaddress.com
apiKeySecretRef:
name: cloudflare-api-key-secret
key: api-key
Test website
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test-ingress
namespace: test
annotations:
kubernetes.io/ingress.class: "traefik"
cert-manager.io/cluster-issuer: letsencrypt
spec:
rules:
- host: test.<mydomain>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-svc
port:
number: 80
tls:
- secretName: test.<mydomain>
I'm trying to setup TLS for a service that's available outside a Kubernetes cluster (AWS EKS). With cert-manager, I've successfully issued a certificate and configured ingress, but I'm still getting error NET::ERR_CERT_AUTHORITY_INVALID. Here's what I have:
namespace tests and hello-kubernetes in it (both deployment and service have name hello-kubernetes-first, serivce is ClusterIP with port 80 and targetPort 8080, deployment is based on paulbouwer/hello-kubernetes:1.8, see details in my previous question)
DNS and ingress configured to show the service:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-kubernetes-ingress
namespace: tests
spec:
ingressClassName: nginx
rules:
- host: test3.projectname.org
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: hello-kubernetes-first
port:
number: 80
Without configuring TLS, I can access test3.projectname.org via http and see the service (well, it tries to redirect me to https, I see NET::ERR_CERT_AUTHORITY_INVALID, I go to insecure anyway and see the hello-kubernetes page).
note: I have nginx-ingress ingress controller; it was installed before me via the following chart:
apiVersion: v2
name: nginx
description: A Helm chart for Kubernetes
type: application
version: 4.0.6
appVersion: "1.0.4"
dependencies:
- name: ingress-nginx
version: 4.0.6
repository: https://kubernetes.github.io/ingress-nginx
and the values overwrites applied with the chart differ from the original ones mostly in extraArgs: default-ssl-certificate: "nginx-ingress/dragon-family-com" is uncommneted
cert-manager installed via kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml
ClusterIssuer created with the following config:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-backoffice
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
# use https://acme-v02.api.letsencrypt.org/directory after everything is fixed and works
privateKeySecretRef: # this secret is created in the namespace of cert-manager
name: letsencrypt-backoffice-private-key
# email: <will be used for urgent alerts about expiration etc>
solvers:
# TODO: add for each domain/second-level domain/*.projectname.org
- selector:
dnsZones:
- test.projectname.org
- test2.projectname.org
- test3.projectname.org
http01:
ingress:
class: nginx
certificate in the tests namespace. It's config is
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: letsencrypt-certificate-31
namespace: tests
spec:
secretName: tls-secret-31
issuerRef:
kind: ClusterIssuer
name: letsencrypt-backoffice
commonName: test3.projectname.org
dnsNames:
- test3.projectname.org
Now, certificate is ready (kubectl get certificates -n tests tells that) and to apply it, I add this to ingress's spec:
tls:
- hosts:
- test3.projectname.org
secretName: tls-secret-31
However, when I try to open test3.projectname.org via https, it still shows me the NET::ERR_CERT_AUTHORITY_INVALID error. What am I doing wrong? How to debug this? I've checked up openssl s_client -connect test3.projectname.org:443 -prexit* and it shows the following chain:
0 s:CN = test3.projectname.org
i:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
1 s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
2 s:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
and tells, among other output
Verification error: unable to get local issuer certificate
Unfortunately, I haven't found anything useful to try further, so any help is appreciated.
Your ClusterIssuer refers to LetsEncrypt staging issuer. Remove that setting / the default should use their production setup. As pointed out in comments: https://acme-v02.api.letsencrypt.org/directory
Deleting the previously generated secrets or switching to new secrets should ensure your certificates would be re-generated, using the right issuer.
The staging issuer could be useful testing LetsEncrypt integration, it shouldn't be used otherwise.
Following the suggestion from SYN, I've fixed this by
switching ACME server in ClusterIssuer config from https://acme-staging-v02.api.letsencrypt.org/directory to https://acme-v02.api.letsencrypt.org/directory. The idea of the staging server seems to be: allow to debug certificate issuing (so that kubectl get certificate [-n <namespace>] shows that READY = true) without providing actual trusted certificates; after certificate issuing is ok, one has to switch to the main server to get production certificates.
Updating certificates, tls secrets and ingress configs. Well, I'm not sure if there's a way to actually update certificates; instead, I've created new ones, which created new secrets, and then updated ingress configs (just secrets' names)
The reason that your certificates didn't work, it not because you used staging server, but because you didn't specify the tls object within the Ingress rules.
Certbot's staging exists only for the purpose of testing, and for not "ban" you while you testing things out if you request more than 5 certificates/hour.
When you verify that everything works as expected, you can use the normal non-staging server.
This is how it should be done:
Cluster Issuer object:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: user#example.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
Ingress Object:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-kubernetes-ingress
namespace: tests
labels:
app: hello-kubernetes-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- test3.projectname.org
secretName: hello-tls
rules:
- host: test3.projectname.org
http:
paths:
- pathType: ImplementationSpecific
path: "/"
backend:
service:
name: hello-kubernetes-ingress
port:
number: 80
The certificate and the key, are stored in a secret called "hello-tls" which you didn't also specify in your initial example, hence the failure you was receiving.
I'm trying configure for TLS (LetsEncrypt) on a multi-tenant GKE+Istio setup.
I mainly followed this guide -> Full Isolation in Multi-Tenant SAAS with Kubernetes & Istio for setting up the multi-tenancy in GKE+Istio, which I was able to successfully pull-off. I'm able to deploy simple apps on their separate namespaces which are accessible through their respective subdomains.
I then tried to move forward and setup the TLS with LetsEncrypt. For this I mainly followed a different guide which is can be found here-> istio-gke . But unfortunately, following this guide didn't produce the result I wanted. When I was done with it, LetsEncrypt aren't even issuing certificates to my deployment or domain.
Thus I tried to follow a different guide which is as follows -> istio-gateway-tls-setup. Here I managed to get LetsEncrypt issue a certificate for my domain, but when I tried to test it out with openssl or other online ssl checkers, it says that I still aren't communicating securely.
Below are the results when I try describe the configurations of my certificates, issuer & gateway:
Certificate: kubectl -n istio-system describe certificate istio-gateway
Issuer: kubectl -n istio-system describe issuer letsencrypt-prod
Gateway: kubectl -n istio-system describe gateway istio-gateway
While here's the dry-run results for my helm install <tenant>
install.go:173: [debug] Original chart version: ""
install.go:190: [debug] CHART PATH: /home/cjcabero/projects/aqt-ott-msging-dev/gke-setup/helmchart
NAME: tenanta
LAST DEPLOYED: Wed Feb 17 21:15:08 2021
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
{}
COMPUTED VALUES:
frontend:
image:
pullPolicy: IfNotPresent
repository: paulbouwer/hello-kubernetes
tag: "1.8"
ports:
containerPort: 8080
service:
name: http
port: 80
type: ClusterIP
HOOKS:
MANIFEST:
---
# Source: helmchart/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tenanta
labels:
istio-injection: enabled
---
# Source: helmchart/templates/frontend-service.yaml
apiVersion: v1
kind: Service
metadata:
name: frontend
namespace: tenanta
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 8080
selector:
app: frontend
---
# Source: helmchart/templates/frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
namespace: tenanta
spec:
replicas: 3
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.8
ports:
- containerPort: 8080
env:
- name: MESSAGE
value: Hello tenanta
---
# Source: helmchart/templates/virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: tenanta-frontend-ingress
namespace: istio-system
spec:
hosts:
- tenanta.cjcabero.dev
gateways:
- istio-gateway
http:
- route:
- destination:
host: frontend.tenanta.svc.cluster.local
port:
number: 80
I don't understand how come even though LetsEncrypt seem to be able issue the certificate for my domain, it still aren't communicating securely.
Google Domains even managed to find that a certificate was issued in the domain in it's Transparency Report.
Anyway, I'm not sure if this could help, but I also tried the check the domain with an online ssl checker and here are the results -> https://check-your-website.server-daten.de/?q=cjcabero.dev.
By the way I did use Istio on GKE which results with Istio v1.4.10 & Kubernetes v1.18.15-gke.1100.
I had a project hosted on kubernetes and using nginx ingress controller and cert manager to handle the SSL certificates. It was all working fine until I change the hosting of the kubernetes cluster to another data center, and I happen shut down the old cluster without backing up the private keys.
Now my sites are showing NET::ERR_CERT_AUTHORITY_INVALID.
Any suggestions on how to either revoke the old cert so that I can regenerate a new certificate in the new cluster, or if there is a way that I can override the old cert?
Thanks for helping out!
Ingress controller:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "prod-letsencrypt"
# nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
tls:
- hosts:
- xxx.example.com
secretName: example-tls
rules:
- host: xxx.example.com
http:
paths:
- backend:
serviceName: example-svc
servicePort: 80
Issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: prod-letsencrypt
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: xxx#example.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: prod-letsencrypt
# Enable the HTTP-01 challenge provider
solvers:
- dns01:
route53:
region: us-east-2
role: arn:aws:iam::xxxx:role/dns-challenge-role
describing the certificate in the new cluster and only see the cert is being requested.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 60m cert-manager Issuing certificate as Secret does not exist
Normal Generated 60m cert-manager Stored new private key in temporary Secret resource "example-tls-f4xt5"
Normal Requested 60m cert-manager Created new CertificateRequest resource "example-tls-xmxqm"
I'm running into an issue handling tls certificates with cert-manager, I'm following the documentation and added some extras to work with Traefik as an ingress.
Currently, I have this YAML files:
cluster-issuer.yaml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: secure-alexguedescom
spec:
acme:
email: user#gmail.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-staging
# Add a single challenge solver, HTTP01 using nginx
solvers:
- selector: {}
http01:
ingress:
class: traefik-cert-manager
traefik-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# add an annotation indicating the issuer to use.
cert-manager.io/cluster-issuer: letsencrypt-staging
name: secure-alexguedescom-ingress-http
namespace: secure-alexguedescom
spec:
rules:
- host: secure.alexguedes.com
http:
paths:
- backend:
serviceName: secure-alexguedescom-nginx
servicePort: 80
path: /
tls:
- hosts:
- secure.alexguedes.com
secretName: secure-alexguedescom-cert
cert-staging.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: secure-alexguedescom-cert
namespace: secure-alexguedescom
spec:
commonName: secure.alexguedes.com
secretName: letsencrypt-staging
dnsNames:
- secure.alexguedes.com
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
Inspecting the certs I have this error message:
Message: Issuing certificate as Secret does not contain a certificate
Reason: MissingData
Also inspecting the certificaterequest I have this log messages:
Status:
Conditions:
Last Transition Time: 2020-08-16T00:32:01Z
Message: Waiting on certificate issuance from order secure-alexguedescom/secure-alexguedescom-cert-q8w5p-1982372682: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 11m cert-manager Created Order resource secure-alexguedescom/secure-alexguedescom-cert-q8w5p-1982372682
Normal OrderPending 11m cert-manager Waiting on certificate issuance from order secure-alexguedescom/secure-alexguedescom-cert-q8w5p-1982372682: ""
I'm not sure which piece is wrong, using Helm v2 with Tiller and k8s v1.7
Any ideas?
Thanks in advance
The typical problem with letsencrypt certs is the letsencrypt itself not being able to validate who you are and that you own the domain. In this case, alexguedes.com.
With cert-manager you can do Domain Validation and HTTP Validation. Based on the posted ClusterIssuer you are doing HTTP Validation. So you need to make sure that secure.alexguedes.com resolves to a globally available IP address and that Traefik port 443 is listening on that IP address.