Unable to self sign certificate nginx ingress k3s - ssl-certificate

I'm new to K3s, and have struggle with this step for a few days.
Environment: Ubuntu 20.04 | K3s installation without Traefik.
K3s installation script:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --no-deploy=traefik" sh -s -
Nginx ingress installation script
helm repo add nginx-stable https://helm.nginx.com/stable
helm repo update
helm install my-release nginx-stable/nginx-ingress
Cert-manager installation script
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.3.1 \
--set installCRDs=true
Verified with Cert-manager verifier
Create a testing namespace to play with kubectl create ns practice-cls
Test service deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: kuard
namespace: practice-cls
spec:
selector:
matchLabels:
app: kuard
replicas: 1
template:
metadata:
labels:
app: kuard
spec:
containers:
- image: gcr.io/kuar-demo/kuard-amd64:1
imagePullPolicy: Always
name: kuard
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: kuard
namespace: practice-cls
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
selector:
app: kuard
Issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-cluster-issuer
namespace: cert-manager
spec:
selfSigned: {}
service ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kuard
namespace: practice-cls
annotations:
cert-manager.io/cluster-issuer: "selfsigned-cluster-issuer"
spec:
tls:
- hosts:
- example.example.com
secretName: quickstart-example-tls
rules:
- host: example.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: kuard
port:
number: 80
ingressClassName: nginx
# kubectl describe ing kuard -n practice-cls
Name: kuard
Labels: <none>
Namespace: practice-cls
Address: 10.227.224.141
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
quickstart-example-tls terminates example.example.com
Rules:
Host Path Backends
---- ---- --------
example.example.com
/ kuard:80 (10.42.0.76:8080)
Annotations: cert-manager.io/cluster-issuer: selfsigned-cluster-issuer
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning AddedOrUpdatedWithWarning 6m9s nginx-ingress-controller Configuration for practice-cls/kuard was added or updated ; with warning(s): TLS secret quickstart-example-tls is invalid: secret doesn't exist or of an unsupported type
I don't know if there was anything wrong with this, the kuard image was just a tutorial service from cert-manager. And I got ERR_SSL_UNRECOGNIZED_NAME_ALERT from the manifests above.
Let me know if there's some more information to troubleshoot this.
Thank you guys

After a while searching and experiment, I manage to handle this by:
Using K8s nginx ingress instead of the official one provide by nginx themself
Enable SSL passthrough either by editing the deployment of nginx controller or enable that right when installing

The nginx ingress controller (produced by Nginx, the company), has picky code that will not support the default Opaque Secret type for the TLS secret. Check that your "quickstart-example-tls" Secret has its type set to: kubernetes.io/tls, or one of the supported types in their list.
// IsSupportedSecretType checks if the secret type is supported.
func IsSupportedSecretType(secretType api_v1.SecretType) bool {
return secretType == api_v1.SecretTypeTLS ||
secretType == SecretTypeCA ||
secretType == SecretTypeJWK ||
secretType == SecretTypeOIDC
}
The community supported Kubernetes Nginx Ingress Controller does not have this restriction, and supports Opaque secret types just fine.

Related

ArgoCD with nginx ingress and cert manager not working

I am trying to expose ArgoCD using ingress and cert-manager in GKE cluster(version 1.21.5-gke.1302) but the certificate is not issued.
Steps to reproduce:
Install cert-manager applying this yaml
Install nginx ingress-controller with helm running:
helm install my-release nginx-stable/nginx-ingress
Create clusterIssuer applying the following:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-cluster-issuer
spec:
acme:
email: example#email.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-cluster-issuer-key
solvers:
- http01:
ingress:
class: nginx
Applied ingress using this guide and the file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argocd-server-ingress
namespace: argocd
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
# If you encounter a redirect loop or are getting a 307 response code
# then you need to force the nginx ingress to connect to the backend using HTTPS.
#
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
rules:
- host: argocd.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: argocd-server
port:
name: https
tls:
- hosts:
- argocd.example.com
secretName: argocd-secret # do not change, this is provided by Argo CD
Map the ip of ingress-controller to your host name.
Expected behaviour: I was expecting a certificate to be created successfully and have access to the app.
Current status:
Certificate describe gives me this:
Conditions:
Last Transition Time: 2022-01-18T14:10:14Z
Message: Existing issued Secret is not up to date for spec: [spec.dnsNames]
Observed Generation: 3
Reason: SecretMismatch
Status: False
Type: Ready
Last Transition Time: 2022-01-18T14:10:14Z
Message: Issuing certificate as Secret was previously issued by Issuer.cert-manager.io/
Observed Generation: 1
Reason: IncorrectIssuer
Status: True
Type: Issuing
Next Private Key Secret Name: argocd-secret-ccjtv
Not After: 2023-01-18T13:39:24Z
Not Before: 2022-01-18T13:39:24Z
Renewal Time: 2022-09-18T21:39:24Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Requested 16m cert-manager Created new CertificateRequest resource "argocd-secret-qm469"
Normal Requested 15m cert-manager Created new CertificateRequest resource "argocd-secret-9ctn4"
Normal Reused 7m19s (x2 over 45h) cert-manager Reusing private key stored in existing Secret resource "argocd-secret"
Finally I can access the provided url by the challenge but status is pending with reason:
Waiting for HTTP-01 challenge propagation: failed to perform self check GET request
Does anyone have any idea what might be wrong? It would be highly appreciated.
Thanks!
Looks like you have a different name for cluster issue in your ingress rule for ArgoCD.
From your example in the ClusterIssuer manifest:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-cluster-issuer
And from ingress rule
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
I think you need to specify:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-cluster-issuer
spec:
acme:
email: example#email.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: argocd-secret # HERE use secrets created by ArgoCD
solvers:
- http01:
ingress:
class: nginx

How can I use Cert manager letsencrypt-prod in my kubernetes service?

I have 4 yaml file
Deployment.yaml
Service.yaml
Ingress.yaml
issuer.yaml
I want to use letsencrypt-prod for my service for certification . But it doesn't work.
When I use to be sure ingress is working or issuer is working both of them are done!
kubectl get ing
kubectl get issuer
But when I run:
kubectl get cert
Cert is not readt during 2 days . Like below:
it creates problem like below. certification is not binding mandrakee.xyz.Mandrakee.xyz looks still not secure! how can I make my website secyre via cert manager?
Deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo-deployment
spec:
replicas: 1
selector:
matchLabels:
app: echo-server
template:
metadata:
labels:
app: echo-server
spec:
containers:
- name: httpapi-host
image: jmalloc/echo-server
imagePullPolicy: Always
resources:
requests:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 80
Service.yaml:
apiVersion: v1
kind: Service
metadata:
name: echo-service
spec:
ports:
- name: http-port
port: 80
targetPort: 8080
selector:
app: echo-server
Ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: ambassador
cert-manager.io/issuer: letsencrypt-prod
name: test-ingress
spec:
tls:
- hosts:
- mandrakee.xyz
secretName: letsencrypt-prod
rules:
- host: mandrakee.xyz
http:
paths:
- backend:
service:
name: echo-service
port:
number: 80
path: /
pathType: Prefix
issuer.yaml:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: ykaratoprak#sphereinc.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
digitalocean:
tokenSecretRef:
name: digitalocean-dns
key: ce28952b5b4e33ea7d98de190f3148a7cc82d31f030bde966ad13b22c1abc524
If you have setup your issuer correctly, which you have assured us, you will see in your namespace a pod belonging to cert manager. This creates a pod that will validate that the server requesting the certificate resolves to the DNS record.
In your case, you would need to point your DNS towards your ingress.
If this is done successfully, then the next stage of debugging is to validate that both 443 and 80 can be resolved. The Validation Pod created by Cert Manager uses port 80 to validate the communication. A common mistake people make is assuming that they will only use port 443 for ssl and disable 80 for security reasons to find out later that letsencrypt can't validate the hostname without port 80.
Otherwise, the common scenario is that cert-manager is installed in the namespace cert-manager and so you should check the logs of the manager. This will provided a limited amount of logs and can be sometimes cryptic to finding the remedy to your issues.
To find the direct error, the pod spawned by cert-manager in the namespace you have deployed the ingress is a good place to focus.
A test I would run is to setup the ingress with both 80 and 443, if you use your domain from your browser you should get some invalid kubernetes generic certificates response on the port 443 and just "Not Found" on port 80. If this is successful, it rules out the limitation I have mentioned before.

Unable to successfully setup TLS on a Multi-Tenant GKE+Istio with LetsEncrypt (via Cert Manager)

I'm trying configure for TLS (LetsEncrypt) on a multi-tenant GKE+Istio setup.
I mainly followed this guide -> Full Isolation in Multi-Tenant SAAS with Kubernetes & Istio for setting up the multi-tenancy in GKE+Istio, which I was able to successfully pull-off. I'm able to deploy simple apps on their separate namespaces which are accessible through their respective subdomains.
I then tried to move forward and setup the TLS with LetsEncrypt. For this I mainly followed a different guide which is can be found here-> istio-gke . But unfortunately, following this guide didn't produce the result I wanted. When I was done with it, LetsEncrypt aren't even issuing certificates to my deployment or domain.
Thus I tried to follow a different guide which is as follows -> istio-gateway-tls-setup. Here I managed to get LetsEncrypt issue a certificate for my domain, but when I tried to test it out with openssl or other online ssl checkers, it says that I still aren't communicating securely.
Below are the results when I try describe the configurations of my certificates, issuer & gateway:
Certificate: kubectl -n istio-system describe certificate istio-gateway
Issuer: kubectl -n istio-system describe issuer letsencrypt-prod
Gateway: kubectl -n istio-system describe gateway istio-gateway
While here's the dry-run results for my helm install <tenant>
install.go:173: [debug] Original chart version: ""
install.go:190: [debug] CHART PATH: /home/cjcabero/projects/aqt-ott-msging-dev/gke-setup/helmchart
NAME: tenanta
LAST DEPLOYED: Wed Feb 17 21:15:08 2021
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
{}
COMPUTED VALUES:
frontend:
image:
pullPolicy: IfNotPresent
repository: paulbouwer/hello-kubernetes
tag: "1.8"
ports:
containerPort: 8080
service:
name: http
port: 80
type: ClusterIP
HOOKS:
MANIFEST:
---
# Source: helmchart/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tenanta
labels:
istio-injection: enabled
---
# Source: helmchart/templates/frontend-service.yaml
apiVersion: v1
kind: Service
metadata:
name: frontend
namespace: tenanta
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 8080
selector:
app: frontend
---
# Source: helmchart/templates/frontend-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
namespace: tenanta
spec:
replicas: 3
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.8
ports:
- containerPort: 8080
env:
- name: MESSAGE
value: Hello tenanta
---
# Source: helmchart/templates/virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: tenanta-frontend-ingress
namespace: istio-system
spec:
hosts:
- tenanta.cjcabero.dev
gateways:
- istio-gateway
http:
- route:
- destination:
host: frontend.tenanta.svc.cluster.local
port:
number: 80
I don't understand how come even though LetsEncrypt seem to be able issue the certificate for my domain, it still aren't communicating securely.
Google Domains even managed to find that a certificate was issued in the domain in it's Transparency Report.
Anyway, I'm not sure if this could help, but I also tried the check the domain with an online ssl checker and here are the results -> https://check-your-website.server-daten.de/?q=cjcabero.dev.
By the way I did use Istio on GKE which results with Istio v1.4.10 & Kubernetes v1.18.15-gke.1100.

SSL Certificates on Kubernetes Using ACME

I have been following this tutorial: https://cert-manager.io/docs/ , and after I have installed my cert manager and made sure they are running with kubectl get pods --namespace cert-manager,
cert-manager-5597cff495-l5hjs 1/1 Running 0 91m
cert-manager-cainjector-bd5f9c764-xrb2t 1/1 Running 0 91m
cert-manager-webhook-5f57f59fbc-q5rqs 1/1 Running 0 91m
I then configured my cert-manager using ACME issuer by following this tutorial https://cert-manager.io/docs/configuration/acme/ .
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: aidenhsy#gmail.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-staging
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
Here is my full ingress config file:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: www.hyhaus.xyz
http:
paths:
- path: /api/?(.*)
backend:
serviceName: devback-srv
servicePort: 4000
- path: /?(.*)
backend:
serviceName: devfront-srv
servicePort: 3000
---
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'
service.beta.kubernetes.io/do-loadbalancer-hostname: 'www.hyhaus.xyz'
labels:
helm.sh/chart: ingress-nginx-2.0.3
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 0.32.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: aidenhsy#gmail.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-staging
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
However when I browse to my site, the browser warns: security certificate is not trusted by your computer's operating system. And when I took a look a my certificate, it shows self-assigned, which is not really what I want. Am I doing something wrong here?
This is a certificate placeholder provided by nginx ingress controller. When you see it, it means there is no other (dedicated) certificate for the endpoint.
Now the first reason why this happened is that your Ingress doesn't have necessary data. Update it with this:
metadata:
annotations:
# which issuer to use
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
tls: # placing a host in TLS config indicates that a certificate should be created
- hosts:
- example.org
- www.example.org
- xyz.example.org
secretName: myingress-cert # cert-manager will store the created certificate in this secret
Documentation for ingress objects is here.
If the above didn't help, try the troubleshooting steps offered by the documentation. In my experience checking CertificateRequest and Certificate resources was enough in most cases to determine the problem.
$ kubectl get certificate
$ kubectl describe certificate <certificate-name>
$ kubectl get certificaterequest
$ kubectl describe certificaterequest <CertificateRequest name>
Remember that these objects are namespaced, meaning that they'll be in the same namespace as the ingress object.
To secure Ingress, First you have to add ClusterIssuer to your Ingress resources and cert-manager will then pick it up and create the Certificate resource for you .
Kind : ingress metadata: annotations : cert-manager.io/cluster-issuer: nameOfClusterIssuer .
Second you have to add tls <= this indicates the creation of certificate(key/cert pair) by Cert-manager via The ClusterIssuer.
Third you have to add secretName: myingress <= here the cert manager will store the tls secrets ( after creating key/cert pair and store them for you)..

GKE, NGINX ingress, HTTPS, and certificates

I have set up a Kubernetes cluster on GCP/GKE and it's all working well except for one thing. When I access the external IP for the service the (default?) "Kubernetes Ingress Controller Fake Certificate" is served.
I am trying to use the NGINX Ingress (https://kubernetes.github.io/ingress-nginx/) and have followed what I believe are the correct instructions for associating a TLS secret with the Ingress. For example:
https://estl.tech/configuring-https-to-a-web-service-on-google-kubernetes-engine-2d71849520d
https://kubernetes.github.io/ingress-nginx/user-guide/tls/
I have created a secret like this:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: default
data:
tls.crt: [removed]
tls.key: [removed]
type: kubernetes.io/tls
And associated that secret (which I can confirm is applied correctly and I can see in the cluster config) with the Ingress like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/affinity: "cookie"
spec:
backend:
serviceName: example-service
servicePort: 80
tls:
- secretName: example-tls
From the documentation I feel that this should work (but, barring a bug, I am obviously mistaken!).
I've also seen some documentation around requiring target proxies for HTTPS. Perhaps that is the way that I should be doing this?
Many thanks for your help in advance.
Cheers,
Ben
PS: This is my load balancer configuration:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Cluster
loadBalancerIP: [removed]
sessionAffinity: ClientIP
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
Edit 1:
Looking at my Ingress I can see this:
➜ gke git:(develop) ✗ kubectl describe ing example-tls-ingress
Name: example-tls-ingress
Namespace: default
Address: [removed]
Default backend: example-webapp-service:80 ([removed])
TLS:
example-tls terminates
Rules:
Host Path Backends
---- ---- --------
* * example-webapp-service:80 ([removed])
So it looks like the secret is picked up.
And this makes me think that there is a difference between Ingress-terminated TLS and Load Balancer-terminated TLS?
You can just refer to this stackoverflow post.
You need to install jetstack cert-Manager, create clusterissuer/issuer, along with a certificate in which you have to pass domain name / hostname and jetstack will automatically create the secret for you, by the name you mentioned in the 'Certificate'.
That secret has to be patched to TLS in ingress rule.