I am having a problem with my TLS. I have my TLS secret created:
kubectl create secret tls ingress-tls --key certificate.key --cert certificate.crt
And I use it in my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "beta"
spec:
tls:
- hosts:
- '*.host.com'
- 'beta.host.com'
secretName: ingress-tls
backend:
serviceName: nginx
servicePort: 443
The ingress is created perfectly, I access through a browser and no problem, the problem comes when I do a curl or using the program postman, I get certificate error.
curl: (60) SSL certificate problem: unable to get local issuer
certificate More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could
not establish a secure connection to it. To learn more about this
situation and how to fix it, please visit the web page mentioned
above.
I'm using the GCE driver, it's the default GKE from google cloud.
I've been reading how to add the ca key, but this error is not fixed.
I did the following:
kubectl create secret generic ca --from-file certificate.ca.crt
And I added the following annotation to the ingress:
ingress.kubernetes.io/auth-tls-secret: ca
But is not working.
Does anyone know how to fix the CA certificate? The certificate is purchased on the DonDominio website and it's a Wildcard.
The problem was basically that I was using the .crt instead of the .pem when I generated the TLS secret. By changing the secret I got curl to detect it as a valid certificate.
New command:
kubectl create secret tls ingress-tls --key certificate.key --cert certificate.pem
Thanks to #Michael-sqlbot!
If you have .ca-bundle + crt file, append .ca-bundle into the end of .crt and create secret from the new crt file will resolve this issue. This is because you need all the immediate keys for your ingress.
Related
I have created a GKE Cluster 1.18.17-gke.1901 and I have installed Istio 1.9.5 on it. My Ingress Gateway Service is of type: LoadBalancer.
I am trying to implement MUTUAL TLS mode in my istio-ingressgateway. The Gateway configuration looks like this:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: mutual-domain
namespace: test
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- mutual.domain.com
port:
name: mutual-domain-https
number: 443
protocol: HTTPS
tls:
credentialName: mutual-secret
minProtocolVersion: TLSV1_2
mode: MUTUAL
I have also setup a corresponding VirtualService and DestinationRule too.
Now, whenever I try to connect to https://mutual.domain.com I get the following error:
* Trying 100.50.76.97...
* TCP_NODELAY set
* Connected to mutual.domain.com (100.50.76.97) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mutual.domain.com:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mutual.domain.com:443
If I change the tls: mode: to SIMPLE I am able to reach the service via the domain name but when it's MUTUAL the error above shows up.
The mutual-secret is a tls type Kubernets secret and it contains the tls.crt and tls.key.
$ kubectl describe mutual-secret
Name: mutual-secret
Namespace: istio-system
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 4585 bytes
tls.key: 1674 bytes
Is there something missing? Why can't I access my service in MUTUAL mode but the same secret works for SIMPLE mode?
Assuming you are following this.
It seems you are missing ca.crt in your secret. Create a new secret with tls.crt, tsl.key and ca.crt, and try again.
The ERR_BAD_SSL_CLIENT_AUTH_CERT error mentioned in the comments is Chrome/Chromium specific. It means the browser does not recognise Certificate Authority.
Add your CA certificate (.pem file) to Chrome/Chromium, you can follow this. Hopefully it will solve your problem.
I team I have followed this link to configure cert manager in for My Istio but still I am not able to access the app through Istio ingress.
my manifest file look like this:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: test-cert
namespace: testing
spec:
secretName: test-cert
dnsNames:
- "example.com"
issuerRef:
name: test-letsencrypt
kind: ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: test-letsencrypt
namespace: testing
spec:
acme:
email: abc#example.com
privateKeySecretRef:
name: testing-letsencrypt-private-key
server: https://acme-staging-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: istio
selector: {}
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
annotations:
certmanager.k8s.io/acme-challenge-type: http01
certmanager.k8s.io/cluster-issuer: test-letsencrypt
name: test-gateway
namespace: testing
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "example.com"
tls:
mode: SIMPLE
credentialName: test-cert
Can anyone help me with what I am missing here?
Error from browser :
Secure Connection Failed
An error occurred during a connection to skydeck-test.asteria.co.in. PR_CONNECT_RESET_ERROR
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the web site owners to inform them of this problem.
Learn more…
these are few logs may be helpful:
Normal Generated 5m13s cert-manager Stored new private key in temporary Secret resource "test-cert-sthkc"
Normal Requested 5m13s cert-manager Created new CertificateRequest resource "test-cert-htxcr"
Normal Issuing 4m33s cert-manager The certificate has been successfully issued
samirparhi#Samirs-Mac ~ % k get certificate -n testing
NAME READY SECRET AGE
test-cert True test-cert 19m
Note: this Namespace (testing) has Istio side car injection enabled and all the http request is working but HTTPS when I try to setup , it fails
I encountered the same problem when my certificate was not authenticated by a trusted third party but instead signed by me. I had to add an exception to my browser in order to access the site. So a simple money issue.
Also I was able to add my certificate to the /etc/ssl directory of the client machine to connect without problems.
Also I was able to add certificates by using TLS secrets and adding them to my virtual service configuration. You can try them too.
Examples:
TLS Secret:
kubectl create -n istio-system secret tls my-tls-secret --key=www.example.com.key --cert=www.example.com.crt
I assumed that you already have your certificate and its key but in case you need it:
Certificate creation:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=My Company Inc./CN=example.com' -keyout example.com.key -out example.com.crt
openssl req -out www.example.com.csr -newkey rsa:2048 -nodes -keyout www.example.com.key -subj "/CN=www.example.com/O=World Wide Example organization"
openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in www.example.com.csr -out www.example.com.crt
Just don't forget to fill -subj fields in a reasonable manner. They are the working factor of authenticity when it comes to SSL certs as I understand. For example the first line of certificate creation creates a key and certificate for your organisation. Which is not approved by authorities to be added to Mozilla's or Chrome's or OS's ssl database.
That is why you get your "Untrusted certificate" message. So, for that reasons you can simply create a key and create your dns records on a trusted third parties dns zone and database and by paying them, you can use their trusted organisation certificates for authenticating your own site.
Gateway:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: mygateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: my-tls-secret # must be the same as secret
hosts:
- www.example.com
Hope it helps.
Feel free to share "app" details.
I'm trying to sort out certificates for applications deployed to a k8s cluster (running on docker-for-win, WSL2 on Windows 10 20H2).
I would like to use the DNS to connect to services, e.g. registry.default.svc.cluster.local, which I've verified is reachable. I created a certificate by following these steps:
Create an openssl.conf with content
[ req ]
default_bits = 2048
prompt = no
encrypt_key = no
distinguished_name = req_dn
req_extensions = req_ext
[ req_dn ]
CN = *.default.svc.cluster.local
[ req_ext ]
subjectAltName = #alt_names
[ alt_names ]
DNS.1 = *.default.svc.cluster.local
Create csr and key file with openssl req -new -config openssl.conf -out wildcard.csr -keyout wildcard.key
Created a certificate signing request with
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: wildcard_csr
spec:
groups:
- system:authenticated
request: $(cat wildcard.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
Approved the request: kubectl certificate approve wildcard_csr
Extracted the crt file: kubectl get csr wildcard_csr -o jsonpath='{.status.certificate}' | base64 -d > wildcard.crt
Deleted the request kubectl delete csr wildcard_csr.
I've then started a pod with the registry:2 image and configured it to use the wildcard.crt and wildcard.key files.
In a different pod, I then tried to push to that registry, but got the error
Error response from daemon: Get https://registry.default.svc.cluster.local:2100/v2/: x509: certificate signed by unknown authority
So it seems that within the pod, the k8s CA isn't trusted. Am I right with this observation? If so, how can I make k8s trust itself (after all, it was a k8s component that signed the certificate)?
I found a way to achieve this with changes to the yaml only: On my machine (not sure how universal that is), the CA certificate is available in the service-account-token secret default-token-7g75m (kubectl describe secrets to find out the name, look for the secret of type kubernetes.io/service-account-token that contains an entry ca.crt).
So to trust this certificate, add a volume
name: "kube-certificate"
secret:
secretName: "default-token-7g75m"
and to the pod that requires the certificate, add a volumeMount
name: "kube-certificate"
mountPath: "/etc/ssl/certs/kube-ca.crt",
subPath: "ca.crt"
I am new to kubernetes and I finally realized how to launch the metrics-server as documented kubernetes-sigs/metrics-server. In case that someone else wonders you need to deploy on Master node and also have minimum one worker in the cluster.
So I get this error:
E0818 15:25:22.835094 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:<hostname-master>: unable to fetch metrics from Kubelet <hostname-master> (<hostname-master>): Get https://<hostname-master>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:<hostname-worker>: unable to fetch metrics from Kubelet <hostname-worker> (<hostname-worker>): Get https://<hostname-worker>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]
I am using my own CAs (not self signed) and I have modified the components.yml file (sample):
args:
- --cert-dir=/tmp/metricsServerCas
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
I know that I can disable the tls by using this flag --kubelet-insecure-tls I have already tried it. I want to use my own CAs for extra security.
I have see other many relevant questions (few samples) e.g.:
x509 certificate signed by unknown authority- Kubernetes and kubectl unable to connect to server: x509: certificate signed by unknown authority
Although that I have applied chown already my $HOME/.kube/config still I see this error.
Where am I going wrong?
Update: On the worker I am creating a directory e.g. /tmp/ca and I add the ca file(s) in the directory.
I am not really good yet with the mounting points and I assume that I am doing something wrong. The default syntax of the images can be found here kubernetes-sigs/metrics-server/v0.3.7 (see components.yml file).
I tried to create a directory on my worker e.g. /tmp/ca and I modified the flag --cert-dir=/tmp/ca and mountPath: /tmp/ca
When I am deploying the file e.g.:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
I keep getting the error from the metrics-server-xxxx:
panic: open /tmp/client-ca-file805316981: read-only file system
Although I have given full access to the directory e.g.:
$ ls -la /tmp/ca
total 8
drwxr-xr-x. 2 user user 20 Aug 19 16:59 .
drwxrwxrwt. 18 root root 4096 Aug 19 17:34 ..
-rwxr-xr-x. 1 user user 1025 Aug 19 16:59 ca.crt
I am not sure where I am going wrong.
How is meant to be configured so someone can use non self signed certificates? I can see that most people are using non SSL which I would like to avoid.
Sample of my args in the image:
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp/ca
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp/ca
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
Update 2: Adding curl command from Master to Worker including error output:
$ curl --cacert /etc/kubernetes/pki/ca.crt https://node_hostname:10250/stats/summary?only_cpu_and_memory=true
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Posting this answer as a community wiki to give better visibility as the solution was posted in the comments.
The version that I used before was 1.18.2 and metrics server v0.3.6. Deployment was through kubeadm. Yes all requirements was exactly as the metrics-server/requirements. The good news is that I got it running by upgrading my k8s version on 1.19.0 and using the latest version v0.3.7. It works with self signed certificates.
The issue was resolved by upgrading:
Kubernetes: 1.18.2 -> 1.19.0
Metrics-server: 0.3.6 -> 0.3.7
This upgrade allowed to run metrics-server with tls enabled (self-signed certificates).
Additional resources that could help when deploying metrics-server with tls:
Github.com: Kubernetes-sigs: Metrics-server: FAQ: How to run metrics-server-securely
How to run metrics-server securely?
Suggested configuration:
Cluster with RBAC enabled
Kubelet read-only port port disabled
Validate kubelet certificate by mounting CA file and providing --kubelet-certificate-authority flag to metrics server
Avoid passing insecure flags to metrics server (--deprecated-kubelet-completely-insecure, --kubelet-insecure-tls)
Consider using your own certificates (--tls-cert-file, --tls-private-key-file)
Github.com: Metrics-server: x509: certificate signed by unknown authority
Ftclausen.github.io: Setting up K8S with metrics-server
Create a configmap to store the ca certificate which was used to generate kubelet serving certificate.
kubectl -n kube-system create configmap ca --from-file=ca.crt=/etc/kubernetes/pki/ca.crt -o yaml
Then use volumeMounts to use it in metrics server pod
spec:
volumes:
- emptyDir: {}
name: tmp-dir
- configMap:
defaultMode: 420
name: ca
name: ca-dir
containers:
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-certificate-authority=/ca/ca.crt
- --kubelet-preferred-address-types=Hostname
volumeMounts:
- mountPath: /tmp
name: tmp-dir
- mountPath: /ca
name: ca-dir
You can follow the same approach and use --tls-cert-file and --tls-private-key-file for using your own certificate instead of self signed certificate.
For my friends on EKS make sure you have the username set (and not set to just the session name like I did):
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: {{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
error: You must be logged in to the server (Unauthorized)
robert ❱ 1 ❱ kubectl edit configmap -n kube-system aws-auth
configmap/aws-auth edited
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: literally_anything:{{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-3-103.us-west-2.compute.internal 341m 17% 1738Mi 52%
...
robert ❱ kubectl logs -n kube-system -l app.kubernetes.io/instance=metrics-server
E0407 22:34:45.879156 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:49.399854 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:50.691133 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:51.827629 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:07.288163 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:08.755492 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:09.801957 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:40:32.405458 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:43:09.791769 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:44:14.244221 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
robert ❱
I want to use cert-manager for issuing my own SSL certificate on AKS.
I already have a signed certificate (https://www.quovadisglobal.de/Zertifikate/SSLCertificates/BusinessSSLCertificates.aspx) which I want to use. In the docs of cert-manager, I find only two relevant Solutions.
https://cert-manager.io/docs/configuration/
SelfSigned: This should be used to sign a certificate by a CSR.
CA: This should be used to sign incoming certificate requests.
I tried the second one. Here what I did:
Install and verify cert-manager:
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml
$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7c5748846c-b4nqb 1/1 Running 0 2d23h
cert-manager-cainjector-7b5965856-bgk4g 1/1 Running 1 2d23h
cert-manager-webhook-5759dd4547-mlgjs 1/1 Running 0 2d23h
Create Secret from private key and cert:
$ sudo kubectl create secret tls ssl-secret-p --cert=mycert.crt --key=mykey.key --namespace=cert-manager
Create issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: ca-issuer
namespace: cert-manager
spec:
ca:
secretName: ssl-secret-p
Error:
$ sudo kubectl get clusterissuers ca-issuer -n cert-manager -o wide
NAME READY STATUS AGE
ca-issuer False Error getting keypair for CA issuer: certificate is not a CA 5m
What I'm doing wrong?
EDIT:
sudo kubectl -n namespace get ing
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress ***.com 51.105.205.128 80, 443 13m
Cert manager will carry out the acme challenge verification, try passing this secret name to the tls in the ingress rule, once the acme challenge appears valid, you will see a corresponding entry in ingress
kubectl -n namespace get ing
will give you that.
Then the certificate shall acquire ready state
I tried it, but I haven't used any pre-created tls secret. You can refer this stackoverflow post, I guess it turns up somewhat helpful to you