K3s ingress TLS enabled accessing TLS enabled backend, how to? - ssl

Question

I have a local K3s kubernetes cluster with its traefik ingress controller.
(Mac OSX, Multipass Hyper-V based local VMs: v1.18.3+k3s1 Ubuntu 16.04.6 LTS 4.4.0-179-generic containerd://1.3.3-k3s2)
what I want is having an ingress that is tls enabled AND forwarding to vault port 8200 via tls/https
k get -n kube-system svc traefik
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik LoadBalancer 10.43.105.6 192.168.64.5 80:30303/TCP,443:30142/TCP 4h21m
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/vault-0 1/1 Running 0 4h31m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 4h31m
service/vault ClusterIP 10.43.8.235 <none> 8200/TCP,8201/TCP 4h31m
NAME READY AGE
statefulset.apps/vault 1/1 4h31m
I deployed vault via helm chart standalone vault (non dev mode) and tls enabled (values.yaml see below)
vault's cert is signed by k3s itself: kubectl -n "${NAMESPACE}" certificate approve "${CSR_NAME}"
certinfo tmp/localK3s/certs/vault/vault.crt
Version: 3 (0x2)
Serial Number:
ed:8f:07:da:0d:3d:8d:55:3d:73:aa:93:9d:98:d2:69
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN=k3s-server-ca#1591718124
Validity
Not Before: Jun 9 15:53:56 2020 GMT
Not After : Jun 9 15:53:56 2021 GMT
Subject: CN=vault.vault.svc
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
DNS:vault, DNS:vault.vault, DNS:vault.vault.svc, DNS:vault.vault.svc.iac.local, DNS:localhost, IP Address:127.0.0.1
now I can access the vault service directly by e.g.:
$ kubectl -n vault port-forward service/vault 8200:8200 &
$
$ export VAULT_ADDR=https://127.0.0.1:8200
$ export VAULT_CAPATH=$(pwd)/tmp/localK3s/certs/localK3s_root.ca
$ export VAULT_CLIENT_CERT=$(pwd)/tmp/localK3s/certs/vault/vault.crt
$ export VAULT_CLIENT_KEY=$(pwd)/tmp/localK3s/certs/vault/vault.key
$
$ vault status
Handling connection for 8200
Handling connection for 8200
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.4.2
Cluster Name vault-cluster-5bc9e954
Cluster ID ca5496a6-525d-2b86-22dd-f771da82d5e0
HA Enabled false
now what I want is having an ingress that is tls enabled AND forwarding to vault port 8200 via tls/https
so I have
$ kubectl get ingress vault -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: vault
namespace: vault
annotations:
meta.helm.sh/release-name: vault
meta.helm.sh/release-namespace: vault
labels:
helm.sh/chart: vault-0.6.0
spec:
rules:
- host: vault.iac.local
http:
paths:
- backend:
serviceName: vault
servicePort: 8200
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- vault.iac.local
secretName: vault-tls
status:
loadBalancer: {}
$ export VAULT_ADDR=https://vault.iac.local
$ export VAULT_CAPATH=$(pwd)/tmp/localK3s/certs/localK3s_root.ca
$ export VAULT_CLIENT_CERT=$(pwd)/tmp/localK3s/certs/vault/vault.crt
$ export VAULT_CLIENT_KEY=$(pwd)/tmp/localK3s/certs/vault/vault.key
$
$ vault status
vault status -tls-skip-verify
Error checking seal status: Error making API request.
URL: GET https://vault.iac.local/v1/sys/seal-status
Code: 404. Raw Message:
404 page not found
helm vault values.yaml
global:
enabled: true
tlsDisable: false
injector:
enabled: false
server:
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
extraVolumes:
- type: secret
name: vault-tls
standalone:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = "false" # 1
# address = "0.0.0.0:8200"
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
}
storage "file" {
path = "/vault/data"
}
ingress:
enabled: true
hosts:
- host: vault.iac.local
tls:
- secretName: vault-tls
hosts:
- vault.iac.local
any ideas anybody?

Answer 1

ok, it's always helpful to read the logs (sigh), e.g. of the ingress controller itself, e.g.:
INGCTRL=traefik && \
kubectl -n kube-system logs \
pod/$(kubectl -n kube-system get pods -l app=$INGCTRL | sed -n -E "s/^($INGCTRL-[a-z0-9-]+).*$/\1/p")
if you use generic secrets for ingress tls, beware the secret keys have to be tls.crt and tls.key (or use kubectl create secret tls and not generic in the first place)
also check that your target service has an endpoint at all and not
k describe svc theService
...
Endpoints: <none>
...