When I enable kubernetes auth method at default path (-path=kubernetes) it works. However, if it is enabled at custom path, the vault init and sidecar containers don't start.
kubernetes auth method enable at auth/prod
vault auth enable -path=prod/ kubernetes
vault write auth/prod/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_ca_cert=#/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
vault write auth/prod/role/internal-app \
bound_service_account_names=internal-app \
bound_service_account_namespaces=default \
policies=internal-app \
ttl=24h
What could be wrong with these auth configurations?
Not sure how you have deployed the vault but if your injector is true
injector:
enabled: true
vault will be injecting the sidecars and init container. You should check the logs of side car or init container which is failing.
If you are using the K8s method to authenticate you should check out below annotation example and use them
annotations:
vault.hashicorp.com/agent-image: registry.gitlab.com/XXXXXXXXXXX/vault-image/vault:1.4.1
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-secret-secrets: kv/secret-path-location
vault.hashicorp.com/auth-path: auth/<K8s-cluster-auth-name>
vault.hashicorp.com/role: app
You can keep the multiple auth-path for different K8s clusters to authenticate with a single vault instance also.
If the vault is injecting the sidecar you should check the logs of it.
https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar
Related
I'm curious about something in the kube config file generated by the aws eks update-kubeconfig command. At the bottom of the file, there is this construct:
- name: arn:aws:eks:us-west-2:redacted:cluster/u62d2e14b31f0270011485fd3
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- us-west-2
- eks
- get-token
- --cluster-name
- u62d2e14b31f0270011485fd3
command: aws
It is clearly an invocation of the aws eks get-token command. Why is this here? Does this command get automatically called?
Why is this here?
The command gets an IAM token using your IAM account and pass along to EKS via the HTTP header Authorization: Bearer <token> for authentication. See here for details.
Does this command get automatically called?
Yes, by kubectl.
I have a Cassandra cluster over Kubernetes deployed as stateful sets. Is it possible to implement security in Cassandra?
Yes. In your Docker entrypoint use sed to inject whichever auth classes you wish to use:
sed -i "s/authorizer: CassandraAuthorizer/authorizer: ${AUTHORIZER_CLASS}/" ${CASSANDRA_CONF}/cassandra.yaml
sed -i "s/authenticator: PasswordAuthenticator/authenticator: ${AUTHENTICATOR_CLASS}/" ${CASSANDRA_CONF}/cassandra.yaml
This will create the default cassandra/cassandra role, which you can use to create your new roles.
what about Cassandra Client-to-node encryption?
That's a little trickier. Basically, you need to inject the certificate or keystore and the keystore's password into your image (from a password/secret store or other secure location). Then, your entrypoint should add/update the following standard SSL settings to the end of the cassandra.yaml:
if [ "${SSL_ENABLED}" == "true" ]; then
cat << EOF >> ${CASSANDRA_CONF}/cassandra.yaml
client_encryption_options:
enabled: ${SSL_ENABLED}
optional: ${SSL_OPTIONAL}
keystore: ${CASSANDRA_CONF}/server-keystore.jks
keystore_password: ${KEYSTORE_PASSWORD}
EOF
fi
If you also require two-way SSL (will need require_client_auth: true set) or Node-to-Node encryption, you'll also need to inject or build a Java truststore in the same manner.
I created a pod (an Alpine "BusyBox" to run commands in) which then gets the default service account associated with it. I then created a RoleBinding (and later ClusterRoleBinding when the first didn't work) but it still won't let me call the K8s API.
What am I doing wrong?
First I created a container to run commands in:
# Create a namespace to install our pod
kubectl create namespace one
# Now create a pod that we can run stuff in
kubectl run runner -n one --image alpine -- sleep 3600
Then I created a role binding:
# My understanding of this command is that I'm doing the following:
# 1. Creating a binding for the "default" service account in the "one" namespace
# 2. Tying that to the cluster role for viewing things
# 3. Making this binding work in the "default" namespace, so that it can call
# the API there FROM its own namespace (one)
kubectl create rolebinding default-view --clusterrole=view --serviceaccount=one:default --namespace=default
Then I connected to the pod's terminal and tried to call the API to list all services in its own namespace:
kubectl exec --stdin --tty use-rest -n one -- /bin/ash
# Now I run all these inside that terminal:
# Point to the internal API server hostname
APISERVER=https://kubernetes.default.svc
# Path to ServiceAccount token
SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
# Read the ServiceAccount bearer token
TOKEN=$(cat ${SERVICEACCOUNT}/token)
# Reference the internal certificate authority (CA)
CACERT=${SERVICEACCOUNT}/ca.crt
# The wget installed with Alpine cannot do SSL
apk --no-cache add ca-certificates
apk add wget
wget --ca-certificate=${CACERT} --header="Authorization: Bearer ${TOKEN}" ${APISERVER}/api/v1/namespaces/$NAMESPACE/services
The above gives the error:
--2021-04-20 01:04:54-- https://kubernetes.default.svc/api/v1/namespaces/default/services/
Resolving kubernetes.default.svc (kubernetes.default.svc)... 10.43.0.1
Connecting to kubernetes.default.svc (kubernetes.default.svc)|10.43.0.1|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-04-20 01:04:54 ERROR 403: Forbidden.
But that should be allowed! I get the same error when using a cluster role binding.
USING:
k3d version v4.4.1
k3s version v1.20.5-k3s1 (default)
Calico
You can only have one ServiceAccount per pod and once you've assigned an account to that pod, the default account no longer applies. I was trying to bind the role to the default account, but passing the token of another account I'd created for the pod.
kubectl is using aws eks get-token and works perfectly.
But when I try to login to kubernetes-dashboard with the token generated below I get Unauthorized (401): Invalid credentials provided:
AWS_PROFILE=MYPROFILE aws eks get-token --cluster-name myclustername | jq -r '.status.token'
But if I use the token generated with:
AWS_PROFILE=MYPROFILE aws-iam-authenticator -i myclustername token --token-only
then I can login to kubernetes-dashboard.
So in which way are those tokens different? I thought they were equivalent.
There should be not difference between the tokens generated by aws-iam-authenticator or aws eks get-token.
Make sure that you spelled the cluster name right in both commands as you can generate tokens for clusters that do not exist.
Double check that both commands authenticate:
kubectl --token=`AWS_PROFILE=MYPROFILE aws-iam-authenticator -i MYCLUSTERNAME token --token-only` get nodes
kubectl --token=`AWS_PROFILE=MYPROFILE aws --region eu-north-1 eks get-token --cluster-name MYCLUSTERNAME | jq -r '.status.token'` get nodes
Sometimes is very easy to misspell the cluster name and the tools will happily generate a token for it without producing any visible error or warning.
I have Traefik running with a Consul KV store. How do I remove a record from the acme certificate storage in Consul, or force a renewal for just one domain/frontend?
Problem:
Somehow one of the frontend domains has saved with the wrong certificate. It's referencing a certificate from a different domain (which is also a frontend in Traefik).
I was able to inspect the acme json by getting the consul value for the traefik/acme/account/object key, decode and unzip it and this is the record from the Certs array:
{
"Domains":{
"Main":"my.domain1.com",
"SANs":null
},
"Certificate":{
"Domain":"my.domain2.com",
"CertURL":"https://acme-v02.api.letsencrypt.org/acme/cert/idfordomain2",
"CertStableURL":"https://acme-v02.api.letsencrypt.org/acme/cert/idfordomain2",
"PrivateKey":"...",
"Certificate":"..."
}
}
As you can see, somehow the cert for my.domain2.com has been saved against the record for my.domain1.com so this results in an invalid certificate warning in the browser. I want to clear out the whole record so Traefik will get a fresh cert. I'm using Consul and its saved in binary so I can't just edit the json.
Here is how I solved this issue:
Your traefik network should be marked as attachable: true
Run on host:
docker run -it --rm --name consul-client --network traefik_traefik consul sh
Then run in created container:
export CONSUL_HTTP_ADDR=consul:8500
# get value from consul and store it to acme.json
consul kv get traefik/acme/account/object | gzip -dc > acme.json
# remove invalid domain and store it to acme-fixed.json
cat acme.json | jq -r 'del (.DomainsCertificate.Certs[] | select(.Domains.Main=="'yourdomain.com'"))' > acme-fixed.json
# gzip it
cat acme-fixed.json | gzip -c > acme-fixed.json.gz
# upload fixed and gzipped json back to consul
consul kv put traefik/acme/account/object #acme-fixed.json.gz
Simplest way is to use consul CLI utility. The utility is also used to run server and ideally you should use same version as the one used for your servers. Make sure you export environment variables: CONSUL_HTTP_ADDR - points to consul server, default is http://127.0.0.1:8500 and CONSUL_HTTP_TOKEN - is ACL token, if you have ACLs on your server enabled, as it should be on production environments.
Then you just run following command
consul kv put traefik/acme/account/object #traefik.json
Where trafik.json is json file that has updated values, that you wish to change in Consul KV store.
Or you can use HTTP API: Consul Create/Update Key
curl -X PUT --data #traefik.json http://<your-server-url>:<port>/v1/kv/traefik/acme/account/object
If your server is ACL enabled, you need to add following header to curl request, with <your-acl-token> that was issued to you: -H "X-Consul-Token: <your-acl-token>"