We are using tlsmode=SIMPLE since our gateway will accept token auth or client cert auth. Also we set caCertificates to validate client cert if has. how we do this in Istio and EnvoyFilter ?
My goal is make it work exactly like VerifyClientCertIfGiven
I create a secret ssl-certificate-cacert with our valid ca cert
Then I apply the following EnvoyFilter but its not working, any advice ?
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: add-validation-context-to-simple-mode
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
portNumber: 443
patch:
operation: INSERT_BEFORE
value:
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
'#type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
combined_validation_context:
default_validation_context: {}
validation_context_sds_secret_config:
name: kubernetes://ssl-certificate-cacert
sds_config:
ads: {}
resource_api_version: V3
Related
I'm using the cert-manager to manage my ssl certificates in my Kubernetes cluster. The cert-manager creates the pods and the challenges, but the challenges are never getting fulfilled. They're always saying:
Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://somedomain/.well-known/acme-challenge/VqlmMCsb019CCFDggs03RyBLZJ0jo53LO...': Get "http://somedomain/.well-known/acme-challenge/VqlmMCsb019CCFDggs03RyBLZJ0jo53LO...": EOF
But when I open the url (http:///.well-known/acme-challenge/VqlmMCsb019CCFDggs03RyBLZJ0jo53LO...), it returns the expected code:
vzCVdTk1q55MQCNH...zVkKYGvBJkRTvDBHQ.YfUcSfIKvWo_MIULP9jvYcgtsGxwfJMLWUGsB5kFKRc
When I do kubectl get certs, it says that the certs are ready:
NAME
READY
SECRET
AGE
crt1
True
crt1-secret
65m
crt1-secret
True
crt1-secret
65m
crt2
True
crt2-secret
65m
crt2-secret
True
crt2-secret
65m
It looks like Let's Encrypt never calls (or the cert-manager never instructs) these url's to verify.
When I list the challenges kubectl describe challenges, it says:
Name: crt-secret-hcgcf-269956107-974455061
Namespace: default
Labels: <none>
Annotations: <none>
API Version: acme.cert-manager.io/v1
Kind: Challenge
Metadata:
Creation Timestamp: 2021-07-23T10:47:27Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizer.acme.cert-manager.io":
f:ownerReferences:
.:
k:{"uid":"09e39ad0-cc39-421f-80d2-07c2f82680af"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:authorizationURL:
f:dnsName:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:key:
f:solver:
.:
f:http01:
.:
f:ingress:
.:
f:class:
f:ingressTemplate:
UID: 09e39ad0-cc39-421f-80d2-07c2f82680af
Resource Version: 19014474
UID: b914ad18-2f5c-45cd-aa34-4ad7a2786536
Spec:
Authorization URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/1547...9301
Dns Name: mydomain.something
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt
Key: VqlmMCsb019CCFDggs03RyBLZ...nc767h_g.YfUcSfIKv...GxwfJMLWUGsB5kFKRc
Solver:
http01:
Ingress:
Class: nginx
Ingress Template:
Metadata:
Annotations:
nginx.org/mergeable-ingress-type: minion
Service Type: ClusterIP
Token: VqlmMCsb019CC...03RyBLZJ0jo53LOiqnc767h_g
Type: HTTP-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/15...49301/X--4pw
Wildcard: false
Events: <none>
Any idea how I can solve this issue?
Update 1:
When I run curl http://some-domain.tld/.well-known/acme-challenge/VqlmMCsb019CC...gs03RyBLZJ0jo53LOiqnc767h_g in another pod, it returns:
curl: (52) Empty reply from server
When I do it locally (on my PC), it returns me the expected challenge-response.
Make sure your POD is returning something on the home URL or on the Home page of the domain that you are configuring on ingress host
You can also use the DNS-01 method for verification if HTTP-01 is not working
Here example for the DNS-01 :
Wild card certificate with cert-manager example
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: test123#gmail.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- selector:
dnsZones:
- "devops.example.in"
dns01:
route53:
region: us-east-1
hostedZoneID: Z0152EXAMPLE
accessKeyID: AKIA5EXAMPLE
secretAccessKeySecretRef:
name: route53-secret
key: secret-access-key
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: le-crt
spec:
secretName: tls-secret
issuerRef:
kind: Issuer
name: letsencrypt-prod
commonName: "*.devops.example.in"
dnsNames:
- "*.devops.example.in"
Try using dns01 challenge instead of HTTP-01
I am using cert-manager v1.0.0 on GKE, I tried to use the staging environment for acme and it worked fine but when shifting to production I can find the created certificate stuck at Created new CertificateRequest resource and nothing changes after that
I expect to see the creation of the certificate to be succeeded and change the status of the certificate from false to true as happens in staging
Environment details::
Kubernetes version (v1.18.9):
Cloud-provider/provisioner (GKE):
cert-manager version (v1.0.0):
Install method (helm)
Here is my clusterIssuer yaml file
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: i-storage-ca-issuer-prod
namespace: default
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: MY_EMAIL_HERE
privateKeySecretRef:
name: i-storage-ca-issuer-prod
solvers:
- http01:
ingress:
class: gce
And here is my ingress yaml file
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: i-storage-core
namespace: i-storage
annotations:
kubernetes.io/ingress.global-static-ip-name: i-storage-core-ip
cert-manager.io/cluster-issuer: i-storage-ca-issuer-prod
labels:
app: i-storage-core
spec:
tls:
- hosts:
- i-storage.net
secretName: i-storage-core-prod-cert
rules:
- host: i-storage.net
http:
paths:
- path: /*
backend:
serviceName: i-storage-core-service
servicePort: 80
describe certificateRequest output
Name: i-storage-core-prod-cert-stb6l
Namespace: i-storage
Labels: app=i-storage-core
Annotations: cert-manager.io/certificate-name: i-storage-core-prod-cert
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: i-storage-core-prod-cert-2pw26
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2020-10-31T15:44:57Z
Generate Name: i-storage-core-prod-cert-
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-name:
f:cert-manager.io/certificate-revision:
f:cert-manager.io/private-key-secret-name:
f:generateName:
f:labels:
.:
f:app:
f:ownerReferences:
.:
k:{"uid":"f3442651-3941-49af-81de-dcb937e8ba40"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:request:
f:status:
.:
f:conditions:
Manager: controller
Operation: Update
Time: 2020-10-31T15:44:57Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: i-storage-core-prod-cert
UID: f3442651-3941-49af-81de-dcb937e8ba40
Resource Version: 18351251
Self Link: /apis/cert-manager.io/v1/namespaces/i-storage/certificaterequests/i-storage-core-prod-cert-stb6l
UID: 83412862-903f-4fff-a736-f170e840748e
Spec:
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: i-storage-ca-issuer-prod
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2ZUQ0NBV1VDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5HcQovRDRVZlRhV0xFa01GUzdsdVN1RmRlR0NNVjJ4czREcG5Pem1HbjJxSlRUTlBnS2hHbGVEd0p2TkZIaTc5WWxHCmpYcjhjNDFHU1JUT2U4UDdUS3AvWXpBSUtxSXpPMllIeHY5VzA5bEZDWWQ4MTByMUNsOG5jb2NYa3BGZlAxMzAKZURlczZ6SUkwZW9ZTW1uRXQ3cmRUNk52dHhuZ1ZZVmlnai9VcXpxSkZ4NmlLa0R6V1VHK3lNcWtQM1ZKa1lYeApZUFNTNWZsWXlTdkI4emdxb3pnNUNJUndra09KTU1aRlNoWHVxYkpNZnJvQmR2YW9nQWtEYmZYSWs0SVRIaXlYCkV4UDFBaFdieGhPbndDd2h5bXpGWmgzSkZUZHhzeFdtRDZJMmp3MzV1SXZ1WWlIWEJ4VTBCMG50K3FYMVVWaWwKSkRlOFdNcTdjT3AzWmtlT2FHa0NBd0VBQWFBNE1EWUdDU3FHU0liM0RRRUpEakVwTUNjd0dBWURWUjBSQkJFdwpENElOYVMxemRHOXlZV2RsTG01bGREQUxCZ05WSFE4RUJBTUNCYUF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFLMkhhSEQxd3dDZVFqS1diU1N0SFkxMm1Da1A1amQ0RnFmZFFYRG5XR3grK3FCWExGY0F4TVZhbVF2cStQK0gKLzExQjhvdlUydU9icGRHRktoak9aNDJsdjNNMVllRWk5UG5nS0RFdndCbER0Q0Vsa0lHQzV4T1ZENCtheVlmaApEMUI2L20vdEJsdlhYNS8zRDlyejJsTWNRSzRnSTNVQ3Mxd0Y0bmduQ3JYMEhoSDJEendheXI5d2QvY1V1clZlClloYS9HZjcyaEFCcGQxSmkrR2hKaGxzVDlGbTNVZVNUTi9OYkpVWmk4NkM1S1dTRW1DblNjV3dzWGNoVW1vVisKVHpGQmNhOEhqOUxsVFdJVVBSYVl0bFQ2TEhrUjVLUW1EL2tJRTZDajlidTNXMG9oWDZ2UC9CQ012SWdaTVZEUgoyeFVwY3lhUmJad2ttWTQ2MktNZ25wUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
Status:
Conditions:
Last Transition Time: 2020-10-31T15:44:57Z
Message: Waiting on certificate issuance from order i-storage/i-storage-core-prod-cert-stb6l-177980933: "pending"
Reason: Pending
Status: False
Type: Ready
Events: <none>
describe order output
Name: i-storage-core-prod-cert-stb6l-177980933
Namespace: i-storage
Labels: app=i-storage-core
Annotations: cert-manager.io/certificate-name: i-storage-core-prod-cert
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: i-storage-core-prod-cert-2pw26
API Version: acme.cert-manager.io/v1
Kind: Order
Metadata:
Creation Timestamp: 2020-10-31T15:44:57Z
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:cert-manager.io/certificate-name:
f:cert-manager.io/certificate-revision:
f:cert-manager.io/private-key-secret-name:
f:labels:
.:
f:app:
f:ownerReferences:
.:
k:{"uid":"83412862-903f-4fff-a736-f170e840748e"}:
.:
f:apiVersion:
f:blockOwnerDeletion:
f:controller:
f:kind:
f:name:
f:uid:
f:spec:
.:
f:dnsNames:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:request:
f:status:
.:
f:authorizations:
f:finalizeURL:
f:state:
f:url:
Manager: controller
Operation: Update
Time: 2020-10-31T15:44:57Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: CertificateRequest
Name: i-storage-core-prod-cert-stb6l
UID: 83412862-903f-4fff-a736-f170e840748e
Resource Version: 18351252
Self Link: /apis/acme.cert-manager.io/v1/namespaces/i-storage/orders/i-storage-core-prod-cert-stb6l-177980933
UID: 92165d9c-e57e-4d6e-803d-5d28e8f3033a
Spec:
Dns Names:
i-storage.net
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: i-storage-ca-issuer-prod
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2ZUQ0NBV1VDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5HcQovRDRVZlRhV0xFa01GUzdsdVN1RmRlR0NNVjJ4czREcG5Pem1HbjJxSlRUTlBnS2hHbGVEd0p2TkZIaTc5WWxHCmpYcjhjNDFHU1JUT2U4UDdUS3AvWXpBSUtxSXpPMllIeHY5VzA5bEZDWWQ4MTByMUNsOG5jb2NYa3BGZlAxMzAKZURlczZ6SUkwZW9ZTW1uRXQ3cmRUNk52dHhuZ1ZZVmlnai9VcXpxSkZ4NmlLa0R6V1VHK3lNcWtQM1ZKa1lYeApZUFNTNWZsWXlTdkI4emdxb3pnNUNJUndra09KTU1aRlNoWHVxYkpNZnJvQmR2YW9nQWtEYmZYSWs0SVRIaXlYCkV4UDFBaFdieGhPbndDd2h5bXpGWmgzSkZUZHhzeFdtRDZJMmp3MzV1SXZ1WWlIWEJ4VTBCMG50K3FYMVVWaWwKSkRlOFdNcTdjT3AzWmtlT2FHa0NBd0VBQWFBNE1EWUdDU3FHU0liM0RRRUpEakVwTUNjd0dBWURWUjBSQkJFdwpENElOYVMxemRHOXlZV2RsTG01bGREQUxCZ05WSFE4RUJBTUNCYUF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCCkFLMkhhSEQxd3dDZVFqS1diU1N0SFkxMm1Da1A1amQ0RnFmZFFYRG5XR3grK3FCWExGY0F4TVZhbVF2cStQK0gKLzExQjhvdlUydU9icGRHRktoak9aNDJsdjNNMVllRWk5UG5nS0RFdndCbER0Q0Vsa0lHQzV4T1ZENCtheVlmaApEMUI2L20vdEJsdlhYNS8zRDlyejJsTWNRSzRnSTNVQ3Mxd0Y0bmduQ3JYMEhoSDJEendheXI5d2QvY1V1clZlClloYS9HZjcyaEFCcGQxSmkrR2hKaGxzVDlGbTNVZVNUTi9OYkpVWmk4NkM1S1dTRW1DblNjV3dzWGNoVW1vVisKVHpGQmNhOEhqOUxsVFdJVVBSYVl0bFQ2TEhrUjVLUW1EL2tJRTZDajlidTNXMG9oWDZ2UC9CQ012SWdaTVZEUgoyeFVwY3lhUmJad2ttWTQ2MktNZ25wUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==
Status:
Authorizations:
Challenges:
Token: EMTpMo_Jt5YkITiwk_lOuL66Xu_Q38scNMf1o0LPgvs
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/8230128790/0EcdqA
Token: EMTpMo_Jt5YkITiwk_lOuL66Xu_Q38scNMf1o0LPgvs
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/8230128790/9chkYQ
Token: EMTpMo_Jt5YkITiwk_lOuL66Xu_Q38scNMf1o0LPgvs
Type: tls-alpn-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/8230128790/BaReZw
Identifier: i-storage.net
Initial State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/8230128790
Wildcard: false
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/100748195/5939190036
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/100748195/5939190036
Events: <none>
List all certificates that you have:
kubectl get certificate --all-namespaces
Try to figure out the problem using describe command:
kubectl describe certificate CERTIFICATE_NAME -n YOUR_NAMESPACE
The output of the above command contains the name of the associated certificate request. Dig into more details using describe command once again:
kubectl describe certificaterequest CERTTIFICATE_REQUEST_NAME -n YOUR_NAMESPACE
You may also want to troubleshoot challenges with the following command:
kubectl describe challenges --all-namespaces
In my case, to make it work, I had to replace ClusterIssuer with just Issuer for reasons explained in the comment.
Here is my Issuer manifest:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: cert-manager-staging
namespace: YOUR_NAMESPACE
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: example#example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: cert-manager-staging-private-key
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
Here is my simple Ingress manifest:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/issuer: cert-manager-staging
name: YOUR_NAME
namespace: YOUR_NAMESPACE
spec:
tls:
- hosts:
- example.com
secretName: example-com-staging-certificate
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: example.com
port:
number: 80
Hi guys i have set up istio on minikube and set envoy ext-auth filter on the gateways . i have two microservices running in different pods exposing virtual services /auther and /appone to outside world . the ext-auth filter i set will send every single request to /auther/auth to be authenticated and if the response is 200 let the request to pass and reach other the service it wants .
the problem is that istio is authenticating every single request to all endpoints even /auther. i want to exclude requests sent to /auther to be authenticated (cause auther service will handle the authentication itself ) .but its not working .
so here is my ext-auth filter :
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: authn-filter
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
subFilter:
name: "envoy.router"
patch:
operation: INSERT_BEFORE
value:
name: envoy.ext_authz
typed_config:
"#type": "type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz"
http_service:
server_uri:
uri: http://auther.default.svc.cluster.local
cluster: outbound|3000||auther.default.svc.cluster.local
timeout: 1.5s
path_prefix: /auther/auth?user=
authorizationRequest:
allowedHeaders:
patterns:
- exact: "cookie"
- exact: "authorization"
authorizationResponse:
allowedClientHeaders:
patterns:
- exact: "set-cookie"
- exact: "authorization"
and here is the exception filter im trying to implement:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: bypass-filter
namespace: default
spec:
configPatches:
# The first patch adds the lua filter to the listener/http connection manager
- applyTo: HTTP_ROUTE
match:
context: GATEWAY
routeConfiguration:
vhost:
name: auther
route:
name: auther
patch:
operation: MERGE
value:
typed_per_filter_config:
envoy.ext_authz:
"#type": type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthzPerRoute
disabled: true
the first filter is working just fine . but the second one which is going to exclude the auther service from the authentication ext-filter is not working.
You have set #type to envoy.config.filter.http.ext_authz.v2.ExtAuthzPerRoute, but the correct path is envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute.
Furthermore the route name must match the name in the virtual service. And it must be deployed to the istio-system namespace as your authn-filter. This config works for me:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: bypass-authn
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: HTTP_ROUTE
match:
routeConfiguration:
vhost:
route:
name: my-route #from virtual service http route name
patch:
operation: MERGE
value:
name: envoy.ext_authz_disabled
typed_per_filter_config:
envoy.ext_authz:
"#type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
I have tried to create wildcard SSL certificate using k8s certmanager and issuer with acmedns acme provider. I have created the credentials by POST requesting to /register URL and tested the acmedns successfully. However I am unable to create new wildcard SSL certificate using the k8s issuer. I am adding my issuer YAML file below,
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
annotations:
name: letsencrypt-wildcard-prod
namespace: default
spec:
acme:
dns01:
providers:
acmedns:
accountSecretRef:
key: acmedns.json
name: acme-dns
host: http://auth.mydomain.com
email: info#mydomain.com
privateKeySecretRef:
name: letsencrypt-prod
server: https://acme-v02.api.letsencrypt.org/directory
I have created the secret acme-dns using the json output got from the /register output.
Also, adding the k8s certificate YAML here
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: wildcard-mydomain.com
namespace: default
spec:
acme:
config:
- dns01:
provider: acmedns
domains:
- '*.mydomain.com'
commonName: '*.mydomain.com'
dnsNames:
- '*.mydomain.com'
issuerRef:
kind: Issuer
name: letsencrypt-wildcard-prod
secretName: wildcard-mydomain.com-tls
I am getting the following error from the cert-manager:
E1129 16:30:31.881025 1 reflector.go:205]
github.com/jetstack/cert-manager/pkg/client/informers/
externalversions/factory.go:71: Failed to list
*v1alpha1.Issuer: v1alpha1.IssuerList: Items:
[]v1alpha1.Issuer: v1alpha1.Issuer: Spec: v1alpha1.
IssuerSpec: IssuerConfig: ACME: v1alpha1.ACMEIssuer:
DNS01: v1alpha1.ACMEIssuerDNS01Config: Providers:
[]v1alpha1.ACMEIssuerDNS01Provider:
ReadArrayCB:
expect [ or n, but found {, error found in #10 byte
of ...|oviders":{"acmedns":|..., bigger context
...|81551da95"},
"spec":{"acme":{"dns01":{"providers":
{"acmedns":{"accountSecretRef":{"key":"acmedns.json|...
E1129 16:30:32.887374 1 reflector.go:205] github.com/
jetstack/cert-manager/pkg/client/informers/externalversions
/factory.go:71: Failed to list *v1alpha1.Issuer: v1alpha1.
IssuerList: Items: []v1alpha1.Issuer: v1alpha1.Issuer:
Spec: v1alpha1.IssuerSpec: IssuerConfig: ACME: v1alpha1.
ACMEIssuer: DNS01: v1alpha1.ACMEIssuerDNS01Config:
Providers: []v1alpha1.ACMEIssuerDNS01Provider:
ReadArrayCB:
expect [ or n, but found {, error found in #10
byte of ...|oviders":{"acmedns":|...,
bigger context
...|81551da95"},"spec":{"acme":{"dns01":
{"providers":{"acmedns":{"accountSecretRef":
{"key":"acmedns.json|...
I've been struggling for a while trying to get HTTPS access to my Elasticsearch cluster in Kubernetes.
I think the problem is that Kubernetes doesn't like the TLS certificate I'm trying to use, which is why it's not passing it all the way through to the browser.
Everything else seems to work, since when I accept the Kubernetes Ingress Controller Fake Certificate, the requests go through as expected.
In my attempt to do this I've set up:
The cluster itself
An nginx-ingress controller
An ingress resource
Here's the related yaml:
Cluster:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-08-03T03:20:47Z
labels:
run: my-es
name: my-es
namespace: default
resourceVersion: "3159488"
selfLink: /api/v1/namespaces/default/services/my-es
uid: 373047e0-96cc-11e8-932b-42010a800043
spec:
clusterIP: 10.63.241.39
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 9200
selector:
run: my-es
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
The ingress resource
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS
nginx.ingress.kubernetes.io/cors-origins: http://localhost:3425 https://mydomain.ca
https://myOtherDomain.ca
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: 2018-08-12T08:44:29Z
generation: 16
name: es-ingress
namespace: default
resourceVersion: "3159625"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/es-ingress
uid: ece0071d-9e0b-11e8-8a45-42001a8000fc
spec:
rules:
- http:
paths:
- backend:
serviceName: my-es
servicePort: 8080
path: /
tls:
- hosts:
- mydomain.ca
secretName: my-tls-secret
status:
loadBalancer:
ingress:
- ip: 130.211.179.225
The nginx-ingress controller:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-08-12T00:41:32Z
labels:
app: nginx-ingress
chart: nginx-ingress-0.23.0
component: controller
heritage: Tiller
release: nginx-ingress
name: nginx-ingress-controller
namespace: default
resourceVersion: "2781955"
selfLink: /api/v1/namespaces/default/services/nginx-ingress-controller
uid: 755ee4b8-9dc8-11e8-85a4-4201a08000fc
spec:
clusterIP: 10.63.250.256
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 32084
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31182
port: 443
protocol: TCP
targetPort: https
selector:
app: nginx-ingress
component: controller
release: nginx-ingress
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 35.212.6.131
I feel like I'm missing something basic, because it doesn't seem like it should be this hard to expose something this simple...
To get my certificate, I just requested one for mydomain.ca from godaddy.
Do I need to somehow get a certificate using my ingress resource's cluster IP as the common name?
It doesn't seem possible to verify ownership of an IP.
I've seen people mention ways for Kubernetes to automatically create certificates for ingress resources, but those seem to be self signed.
Here are some logs from the nginx-controller:
This one is talking about a PEM with the tls-secret, but it's only a warning.
{
insertId: "1kvvhm7g1q7e0ej"
labels: {
compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.17-5b82n"
container.googleapis.com/namespace_name: "default"
container.googleapis.com/pod_name: "nginx-ingress-controller-58f57fc597-zl25s"
container.googleapis.com/stream: "stderr"
}
logName: "projects/project-7d320/logs/nginx-ingress-controller"
receiveTimestamp: "2018-08-14T02:58:42.135388365Z"
resource: {
labels: {
cluster_name: "my-elasticsearch-cluster"
container_name: "nginx-ingress-controller"
instance_id: "2341889542400230234"
namespace_id: "default"
pod_id: "nginx-ingress-controller-58f57fc597-zl25s"
project_id: "project-7d320"
zone: "us-central1-a"
}
type: "container"
}
severity: "WARNING"
textPayload: "error obtaining PEM from secret default/my-tls-cert: error retrieving secret default/my-tls-cert: secret default/my-tls-cert was not found"
timestamp: "2018-08-14T02:58:37Z"
}
I have a few occurences of this handshake error, which may be a result of the last warning...
{
insertId: "148t6rfg1xmz978"
labels: {
compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.17-5b82n"
container.googleapis.com/namespace_name: "default"
container.googleapis.com/pod_name: "nginx-ingress-controller-58f57fc597-zl25s"
container.googleapis.com/stream: "stderr"
}
logName: "projects/project-7d320/logs/nginx-ingress-controller"
receiveTimestamp: "2018-08-14T15:55:52.438035706Z"
resource: {
labels: {
cluster_name: "my-elasticsearch-cluster"
container_name: "nginx-ingress-controller"
instance_id: "2341889542400230234"
namespace_id: "default"
pod_id: "nginx-ingress-controller-58f57fc597-zl25s"
project_id: "project-7d320"
zone: "us-central1-a"
}
type: "container"
}
severity: "ERROR"
textPayload: "2018/08/14 15:55:50 [crit] 1548#1548: *860 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:442"
timestamp: "2018-08-14T15:55:50Z"
}
The above logs make it seem like my tls secret isnt working, but when I run kubectl describe ingress, it says my secret terminates.
aaronmw#project-7d320:~$ kubectl describe ing
Name: es-ingress
Namespace: default
Address: 130.221.179.212
Default backend: default-http-backend:80 (10.61.3.7:8080)
TLS:
my-tls-secret terminates mydomain.ca
Rules:
Host Path Backends
---- ---- --------
*
/ my-es:8080 (<none>)
Annotations:
Events: <none>
I figured it out!
What I ended up doing was adding a default ssl certificate to my nginx-ingress controller on creation using the following command
helm install --name nginx-ingress --set controller.extraArgs.default-ssl-certificate=default/search-tls-secret stable/nginx-ingress
Once I had that, it was passing the cert as expected, but I still had the wrong cert as the CN didn't match my load balancer IP.
So what I did was:
Make my load balancer IP static
Add an A record to my domain, to map a subdomain to that IP
Re-key my cert to match that new subdomain
And I'm in business!
Thanks to #Crou, who's comment reminded me to look at the logs and got me on the right track.