Https not working locally for ingress on my kubernetes cluster - ssl

Question

I am trying to make my ingress run over https but I'm getting HTTP ERROR 403, and I don't know what am I missing here.
I am trying to access them on my-local-domain.com and I added that entry on my etc/hosts file as
127.0.0.1 my-local-domain.com so I can access it that way.
Here's my Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: cert-issuer
namespace: my-namespace
spec:
selfSigned: {}
Here's my Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: self-signed-cert
namespace: my-namespace
spec:
secretName: my-secret-tls
dnsNames:
- my-local-domain.com
- www.my-local-domain.com
issuerRef:
name: cert-issuer
While here is my nginx ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
labels:
name: ingress-service
namespace: my-namespace
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
namespace: my-namespace
spec:
tls:
- hosts:
- my-local-domain.com
- www.my-local-domain.com
secretName: my-secret-tls
rules:
- host: my-local-domain.com
http:
paths:
- path: /api/(.*)
pathType: Prefix
backend:
service:
name: web-api
port:
number: 443
- path: /(.*)
pathType: Prefix
backend:
service:
name: web-client
port:
number: 80
nginx.conf for my web-client service
server {
listen 80;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
try_files $uri $uri/ /index.html;
}
gzip on;
}
And when I try to access it with https://my-local-domain.com
Access to my-local-domain.com was denied
You don't have authorization to view this page.
HTTP ERROR 403
I also tried to run
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "//CN=my-local-domain.com\O=my-local-domain.com" -addext "subjectAltName = DNS:my-local-domain.com"
kubectl create secret tls tls-secret --key tls.key --cert tls.crt -n my-namespace
remove Certificate and CertificateIssuer and add tls-secret as secretName in ingress, as well as added those tls.key and tls.crt to my web-client nginx and updated it to:
server {
listen 443 ssl default_server;
ssl_protocols SSLv3 TLSv1;
server_name www.my-local-domain.com my-local-domain.com;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log info;
keepalive_timeout 75 75;
ssl_certificate /etc/ssl/certs/self-signed.crt;
ssl_certificate_key /etc/ssl/private/self-signed.key;
ssl_session_timeout 5m;
add_header Strict-Transport-Security "max-age=7200";
location / {
root /usr/share/nginx/html;
index index.html index.htm;
try_files $uri $uri/ /index.html;
}
}
p.s. when doing this, i run everyrhing on port 443 instead of 80.
When I remove the tls part from my ingress, then it does work through HTTP. And I also tried to change my dns/host from my-local-domain.com to be only localhost but still the same. And even if I try to access https://localhost I get that HTTP ERROR 403.

Answer 1

I don't know if this will help, but here are some advices:
Determine the source of your error
You provide us a lot of code, but no details of the error. Is the 403 thrown by your webservice, your api or by your ingress-controller? You can check that in the error-page. Errors from ingress-controller usally contains somethink like "openresty" in footer. But it hardly depends on your ingress-controller which are you using in your k8s.
You can also check the logs of your containers (nginx write the logs to /var/log/nginx/error.log). Here you can find some more detailed informations why you're getting a 403.
Let the cluster handle SSL-termination
Don't put the ssl-certifications into the containers. It will increase hardly the complexity of your app(s).
It is a lot easier, when the cluster take care about that. So your app can be faster deployed on other clusters with others environments.
Avoid the cert-manager, when using self-signed certs
As you already tested, but I suggest to use the cert-manager only, when you need certificates from lets encrypt. Otherwise it will increase the complexity again.
Don't mix up HTTP and HTTPS
Your webapi using HTTPS for the internal communication (internal means: ingress-controller -> webapi-service, which is usually unencrypted) ... but not for your web-server.
Keep things simple - HTTPS for all or for nobody.
Try to avoid url-rewrite
Whenever when possible, try to avoid url-rewrite. I see very often problems with it, so I suggest to avoid it.
I know this is not always possible, but when all componets are developed by you ... why did you need it at all? Here is your ingress in "easy":
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
labels:
name: ingress-service
namespace: my-namespace
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
spec:
tls:
- hosts:
- my-local-domain.com
- www.my-local-domain.com
secretName: my-secret-tls
rules:
- host: my-local-domain.com
http:
paths:
- path: /api/
backend:
service:
name: web-api
port:
number: 80
- path: /
backend:
service:
name: web-client
port:
number: 80
Rootless is the best
Not very important for your problem, but instead of the standard nginx-image, use the unprivileged nginx-image. If your service (in this case nginx) got hacked, then the hacker has the same privileges like nginx. This means, he can easily install tools into the nginx-container and do very bad things.
If you're using the unprivileged image, than the hacker can ... almost nothing. He needs a second exploit to escalate his rights.
Good luck to find your error. We are happy, when you share your next steps with us.