Created Storage Class for AWS EKS for EBS
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ebs-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
Created Persistent volume Claim with ReadWriteOnce
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ebs-claim
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp2
resources:
requests:
storage: 4Gi
Deploy app using Deployment. Everything worked fine. PV got created and got claimed by PVC. 2 pods started.
Tried to delete one POD but got below error while starting new POD
Multi-Attach error for volume "pvc-5a4ee84b-7afb-40cf-8596-65b76829d540" Volume is already used by pod(s)
Tried PVC with ReadWriteMany but PV failed to create PV
Understood that ReadWriteMany is not supported by EBS and ReadWriteOnce doesnt allow to scale, delete or rollout restart.
Could you please suggest any solution or WA?
As you may already be aware, that as the error suggests there are multiple pods that try to mount this EBS PV, which can be mount only to a single POD at any moment. While you delete the POD you may see that there's another instance of the POD that tries to come up. You'd ideally need to delete the deployment and redeploy for a single instance of your POD to use your EBS PV.
Also EBS PV would bound you to a single Zone.
As part of your deployment, you could scale down the deployment that
uses EBS PV and your new deployment should scale up the deployment.
Related
I've installed an OpenShift cluster for testing purposes, and since I'm behind a corporate network, I need to include some Root Certificates in any Pod that wants to make external requests. What can I do to inject those certificates automatically at Pod creation?
I'm running OpenShift Origin (OKD) 3.11 in a local CentOS 7 VM, with a GlusterFS storage provisioning on top of it. I already had multiple issues with the VM itself, which gave me errors when trying to access the network: x509: certificate signed by unknown authority. I fixed that by adding my corporation root certificates in /etc/pki/ca-trust/source/anchors and by running the update-ca-trust command.
When I was running for example the docker-registry deployment in the OpenShift cluster, since the created Pods didn't have access to the host root certificates, they gave again x509: certificate signed by unknown authority errors when trying to pull images from docker.io. I resolved that by creating a ConfigMap containing all needed root certificates, and mounting them in a volume on the registry deployment config.
I thought I only needed to mount a volume in all deployment configs which want to request the external network. But then I provisioned a Jenkins instance and I realised something new: When a pipeline runs, Jenkins creates a Pod with an adapted agent (example: a Spring Boot app will need a Maven agent). Since I have no control to those created pods, they can't have the mounted volume with all root certificates. So for instance I have a pipeline that runs helm init --client-only before releasing my app chart, and this command gives a x509: certificate signed by unknown authority error, because this pod hasn't the root certificates.
x509 Error screenshot
I found that a PodPreset could be the perfect way to resolve my problem, but when I enable this feature in the cluster and create the PodPreset, no new pod is populated. I read on the OpenShift documentation that PodPresets are no longer supported as of 3.7, so I think that it could be the reason it is not working.
OpenShift docs screenshot
Here is my PodPreset definition file:
kind: PodPreset
apiVersion: settings.k8s.io/v1alpha1
metadata:
name: inject-certs
spec:
selector: {}
volumeMounts:
- mountPath: /etc/ssl/certs/cert1.pem
name: ca
subPath: cert1.pem
- mountPath: /etc/ssl/certs/cert2.pem
name: ca
subPath: cert2.pem
- mountPath: /etc/ssl/certs/cert3.pem
name: ca
subPath: cert3.pem
- mountPath: /etc/ssl/certs/cert4.pem
name: ca
subPath: cert4.pem
- mountPath: /etc/ssl/certs/cert5.pem
name: ca
subPath: cert5.pem
- mountPath: /etc/ssl/certs/cert6.pem
name: ca
subPath: cert6.pem
volumes:
- configMap:
defaultMode: 420
name: ca-pemstore
name: ca
I don't know if there is any way to make PodPresets work on OpenShift 3.11, or if there is another solution to inject certs file like this in created pods. This would be really great.
The RedHat COP on GitHub contains a project with a podpresent admission webhook controller you can use:
https://github.com/redhat-cop/podpreset-webhook
basically you deploy that project and change the apiVersion in your PodPresent to apiVersion: redhatcop.redhat.io/v1alpha1
There are cases when you want to restart a specific container instead of deleting the pod and letting Kubernetes recreate it.
i am having one pod running apache container. i did editing in apache config file. for SSL certificate virtual host port changes etc.
now i want to restart apache2 service but without recreating pod.
i tried inside pod with
service apache2 restart
but it also recreate pod and configuration also change again.
This is not how it supposed to work.
You should not change anything inside the POD.
If your POD dies or crushes, Kubernetes should just start a new one and everything should work.
Also keep in mind that you cannot scale the POD that had configuration altered.
Please check the Kubernetes docs Configure a Pod to Use a ConfigMap
You can use ConfigMap to create configuration file.
ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. This page provides a series of usage examples demonstrating how to create ConfigMaps and configure Pods using data stored in ConfigMaps.
ConfigMap can be created and read a content of a file :
$ kubectl create configmap config_data --from-file=config_data.txt
or it can be declared in .yml
config_map:
data:
db_name=colors_db
table_name=purple
name: config_data
version: v1
Also this might be done by creating a secret
or secret can be declared:
secret:
data:
username: my-username
password: my-password
name: secret_data
version: v1
I recommend reading Kubernetes recipe: store nginx config with ConfigMap and reverse-proxy requests from your domain to your Github page.
There are also other options like mounting path with needed configuration on new POD.
I advice you to check Configure a Pod to Use a PersistentVolume for Storage
check this
You can also create a new dockerfile for override the apache dockerfile and change de CMD line, but it's more complicated
Environment:
AWS managed Kubernetes cluster (EKS)
Action:
Create a PodPreset object by applying the following:
apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
meta data:
name: sample
spec:
selector:
matchLabels:
app: microservice
env:
- name: test_env
value: "6379"
volumeMounts:
- name: shared
mountPath: /usr/shared
volumes:
- name: shared
emptyDir: {}
Observation:
unable to recognize "podpreset.yaml": no matches for kind "PodPreset" in version "settings.k8s.io/v1alpha1"
Looks like that the API version settings.k8s.io/v1alpha1 is not supported by default by EKS.
I'm using EKS as well, I just run this commands to check it out:
$ kubectl proxy
Starting to serve on 127.0.0.1:8001
The I run
curl localhost:8001/apis
And clearly in my case settings.k8s.io/v1alpha1 was not supported. I recommend running the same checks.
Also checking here it's mentioned that
You should have enabled the API type settings.k8s.io/v1alpha1/podpreset
I don't know how can the settings.k8s.io/v1alpha1 can be enabled in EKS.
EKS does not enable any k8s Alpha feature and as of today, PodPreset is a k8s Alpha feature. So if you want to achieve something like above, you have will have to create a Mutating Admission webhook which is supported by EKS now. But it is not sure simple use cases, PodPreset can handle most of the simple use cases hopefully it will enter Beta Phase soon.
As of 03.11.2020 there is still an open GitHub request for this.
In my 10-machines bare-metal Kubernetes cluster, one service needs to call another https-based service which is using a self-signed certificate.
However, since this self-signed certificate is not added into pods' trusted root ca, the call failed saying can't validate x.509 certificate.
All pods are based on ubuntu docker images. However the way to add ca cert to trust list on ubuntu (using dpkg-reconfigure ca-certificates) is not working on this pod any longer. Of course even I succeeded adding the ca cert to trust root on one pod, it's gone when another pod is kicked.
I searched Kubernetes documents, and surprised not found any except configuring cert to talk to API service which is not what I'm looking for. It should be quite common scenario if any secure channel needed between pods. Any ideas?
If you want to bake the cert in at buildtime, edit your Dockerfile adding the commands to copy the cert from the build context and update the trust. You could even add this as a layer to something from docker hub etc.
COPY my-cert.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates
If you're trying to update the trust at runtime things get more complicated. I haven't done this myself, but you might be able to create a configMap containing the certificate, mount it into your container at the above path, and then use an entrypoint script to run update-ca-certificates before your main process.
Updated Edit read option 3:
I can think of 3 options to solve your issue if I was in your scenario:
Option 1.) (The only complete solution I can offer, my other solutions are half solutions unfortunately, credit to Paras Patidar/the following site:)
https://medium.com/#paraspatidar/add-ssl-tls-certificate-or-pem-file-to-kubernetes-pod-s-trusted-root-ca-store-7bed5cd683d
1.) Add certificate to config map:
lets say your pem file is my-cert.pem
kubectl -n <namespace-for-config-map-optional> create configmap ca-pemstore — from-file=my-cert.pem
2.) Mount configmap as volume to exiting CA root location of container:
mount that config map’s file as one to one file relationship in volume mount in directory /etc/ssl/certs/ as file for example
apiVersion: v1
kind: Pod
metadata:
name: cacheconnectsample
spec:
containers:
- name: cacheconnectsample
image: cacheconnectsample:v1
volumeMounts:
- name: ca-pemstore
mountPath: /etc/ssl/certs/my-cert.pem
subPath: my-cert.pem
readOnly: false
ports:
- containerPort: 80
command: [ "dotnet" ]
args: [ "cacheconnectsample.dll" ]
volumes:
- name: ca-pemstore
configMap:
name: ca-pemstore
So I believe the idea here is that /etc/ssl/certs/ is the location of tls certs that are trusted by pods, and the subPath method allows you to add a file without wiping out the contents of the folder, which would contain the k8s secrets.
If all pods share this mountPath, then you might be able to add a pod present and configmap to every namespace, but that's in alpha and is only helpful for static namespaces. (but if this were true then all your pods would trust that cert.)
Option 2.) (Half solution/idea + doesn't exactly answer your question but solves your problem, I'm fairly confident will work in theory, that will require research on your part, but I think you'll find it's the best option:)
In theory you should be able to leverage cert-manager + external-dns + Lets Encrypt Free + a public domain name to replace the self signed cert with a Public Cert.
(there's cert-manager's end result is to auto gen a k8s tls secret signed by Lets Encrypt Free in your cluster, they have a dns01 challenge that can be used to prove you own the cert, which means that you should be able to leverage that solution even without an ingress / even if the cluster is only meant for private network.)
Edit: Option 3.) (After gaining more hands on experience with Kubernetes)
I believe that switchboard.op's answer is probably the best/should be the accepted answer. This "can" be done at runtime, but I'd argue that it should never be done at runtime, doing it at runtime is super hacky and full of edge cases/there's not a universal solution way of doing it.
Also it turns out that my Option 1 doing it is only half correct.
mounting the ca.crt on the pod alone isn't enough. After that file is mounted on the pod you'd need to run a command to trust it. And that means you probably need to override the pods startup command. Example you can't do something like connect to database (the default startup command) and then update trusted CA Certs's command. You'd have to override the startup file to be a hand jammed, overwrite the default startup script, update trusted CA Certs's, connect to the database. And the problem is Ubuntu, RHEL, Alpine, and others have different locations where you have to mount the CA Cert's and sometimes different commands to trust the CA Certs so a universal at runtime solution that you can apply to all pods in the cluster to update their ca.cert isn't impossible, but would require tons of if statements and mutating webhooks/complexity. (a hand crafted per pod solution is very possible though if you just need to be able to dynamically update it for a single pod.)
switchboard.op's answer is the way I'd do it if I had to do it. Build a new custom docker image with your custom ca.cert being trusted baked into the image. This is a universal solution, and greatly simplifies the YAML side. And it's relatively easy to do on the docker image side.
Just for curiosity, here is an example of manifest utilizing the init container approach.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: demo
data:
# in my case it is CloudFlare CA used to sign certificates for origin servers
origin_ca_rsa_root.pem: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
apiVersion: v1
kind: Pod
metadata:
name: demo
labels:
name: demo
spec:
nodeSelector:
kubernetes.io/os: linux
initContainers:
- name: init
# image: ubuntu
# command: ["/bin/sh", "-c"]
# args: ["apt -qq update && apt -qq install -y ca-certificates && update-ca-certificates && cp -r /etc/ssl/certs/* /artifact/"]
# # alternative image with preinstalled ca-certificates utilities
image: grafana/alpine:3.15.4
command: ["/bin/sh", "-c"]
args: ["update-ca-certificates && cp -r /etc/ssl/certs/* /artifact/"]
volumeMounts:
- name: demo
# note - we need change extension to crt here
mountPath: /usr/local/share/ca-certificates/origin_ca_rsa_root.crt
subPath: origin_ca_rsa_root.pem
readOnly: false
- name: tmp
mountPath: /artifact
readOnly: false
containers:
- name: demo
# note - even so init container is alpine base, and this one is ubuntu based everything still works
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: tmp
mountPath: /etc/ssl/certs
readOnly: false
volumes:
- name: demo
configMap:
name: demo
# will be used to pass files between init container and actual container
- name: tmp
emptyDir: {}
and its usage:
kubectl apply -f demo.yml
kubectl exec demo -c demo -- curl --resolve foo.bar.com:443:10.0.14.14 https://foo.bar.com/swagger/v1/swagger.json
kubectl delete -f demp.yml
notes:
replace foo.bar.com to your domain name
replace 10.0.14.14 to ingress controller cluster IP
you may want to add -vv flag to see more details
Indeed it is kind of ugly and monstrous, but at least it does work and is proof of concept. Workarounds with simple ConfigMap do not work because curl reads ca-certificates.crt, which is not modified in that approach.
I'm wondering whether there is a way using the kubernetes API to get the the details of the pods that belong to a given replication controller. I've looked at the reference and the only way as I see it, is getting the pods list and go through each one checking whether it is belongs to a certain RC by analysing the 'annotations' section. It's again a hard job since the json specifies the whole 'kubernetes.io/created-by' part as a single string.
Every Replication Controller has a selector which defines the set of pods managed by it:
selector:
label_name_1: some_value
label_name_2: another_value
You can use the selector to get all the pods with a corresponding set of labels:
https://k8s.example.com/api/v1/pods?labelSelector=label_name_1%3Dsome_value,label_name_2%3Danother_value
To get the details of pods belonging to a particular replication controller we need to include selector field in the yaml file that defines the replication controller and matching label fields in the template of the pod to be created. An example of a replication controller yaml file is given below:
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx
spec:
replicas: 3
selector:
app: nginx
template:
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
To list out the pod names use the command:
pods=$(kubectl get pods --selector=app=nginx --output=jsonpath={.items..metadata.name})
echo $pods
In the above command the --output=jsonpath option specifies the expression that just gets the name of each pod.