The current default cidr block in EKS for the Pods is clashing with Subnet cidr block. Any place where we can mention the cidr blocks in the cloud formation template.
Is it true that the pods becomes part of the same network as the worker nodes .
https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html
This will create lot of problem as the ips will get exhausted.
You can add subnets for your worker nodes to the EKS cluster with SubnetIds under EKS Cluster ResourcesVpcConfig. This currently requires a recreation of the cluster!
Yes, the pods use the same subnet as the node they are being scheduled on. Consider larger CIDR blocks (/22 and above) for EKS subnets. Kubernetes may place pods that will get stuck in ContainerCreating due to no available IP addresses in the subnet.
Update:
With CNI Custom Networking, you can have pods use different subnets.
Related
NOTE: I am new to redis cluster setup.
I am working setting up redis cluster. The cluster setup is complete. However when I want to setup the cluster using dns as follows
redis-cli --cluster create redis-01.internal:6379 redis-02.internal:6379 redis-03.internal:6379
I get the following errors indicating
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Node redis-02.internal:6379 replied with error:
ERR Invalid node address specified: redis-01.internal:6379
I did also go through few of the similar questions which suggest dns based clustering is not available in redis.
So, is there any solution or some way to make sure the dns based cluster works for redis.
Unfortunately, Redis doesn't support creating cluster by specifying hostnames; their IPs need to be used instead.
I am trying to set up hive using mr3 on a kubernetes cluster hosted on AWS ec2. When I run the command run-hive.sh, Hive-server starts and the master-DAg is initialised but then it gets stuck on pending. When I describe the pod. This is the error message shows. I have kept the resources to minimum so it should not be that issue and I do not have any tainted nodes. If you know any alternative for running hive on Kubernetes with access to S3 or a better way to implement mr3 hive on Kubernetes cluster, please share.
One of the node description
Based on the topic i think the problem here is your cluster have not enough resources on your worker nodes, and a master node is tainted.
So the option here is either inreasing the resources on workers or taint the master node so You would be able to schedule pods there.
Control plane node isolation
By default, your cluster will not schedule pods on the control-plane node for security reasons. If you want to be able to schedule pods on the control-plane node, e.g. for a single-machine Kubernetes cluster for development, run:
kubectl taint nodes --all node-role.kubernetes.io/master-
This will remove the node-role.kubernetes.io/master taint from any nodes that have it, including the control-plane node, meaning that the scheduler will then be able to schedule pods everywhere
we are using GKE for NET Core containers with ASP. Each ASP container uses at least one inotify instance (to watch Razer templates) and can use another to watch config files (if not explicitly disabled).
Linux default limit for number of inotify instances per host is 128 (fs.inotify.max_user_instances=128). Some instances are consumed by kubernetes itself (e.g. fluend daemons). So when lots of pods are deployed on single host, host runs out of free inotify instances and containers are stuck in crash loop.
Since we use GKE, we cannot manage worker nodes and alter sysctl settings directly.
My questions are:
Can I somehow alter sysctl setting for host VM through privileged container?
Is there a way to setup kubernetes scheduler to take number of free inotify instances (or at least a number of pods deployed) into account when selecting a node to deploy new pods?
As noted here, "Sysctls with no namespace are called node-level sysctls. If you need to set them, you must manually configure them on each node’s operating system, or by using a DaemonSet with privileged containers".
Regarding scheduling pods, there doesn't seem to be a way for the scheduler to take inotify or number of pods into account when scheduling. The scheduler is only aware of available resources (CPU and memory) and pod specs such as pod or node affinity.
To attain the kind of spread you are looking for will take a good deal of planning and use of both resource requests and pod affinity/anti-affinity. You can review this.
I am trying to setup redis cluster on Kubernetes. One of my requirements is that my redis cluster should be resilient in case of kubernetes cluster restart(due to issues like power failure).
I have tried Kubernetes statefulset and deployment.
In case of statefulset, on reboot a new set of IP addresses are assigned to Pods and since redis cluster works on IP addresses, it is not able to connect to other redis instance and form cluster again.
In case of services with static IP over individual redis instance deployment, again redis stores IP of Pod even when I created cluster using static service IP addresses, so on reboot it is not able to connect to other redis instance and form cluster again.
My redis-cluster statefulset config
My redis-cluster deployment config
Redis-4.0.0 has solved this problem by adding support for cluster announce node IP and Port
Set cluster-announce-ip as static IP of service over redis instance kubernetes deployment.
Link to setup instructions: https://github.com/zuxqoj/kubernetes-redis-cluster/blob/master/README-using-statefulset.md
Are you able to use DNS names instead of IP addresses? I think that is the preferred way to route your traffic to individual nodes in a statefulset:
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-network-id
Playing a bit with Kubernetes (v1.3.2) I’m checking the ability to load balance calls inside the cluster (3 on-premise CentOS 7 VMs).
If I understand correctly the documentation in http://kubernetes.io/docs/user-guide/services/ ‘Virtual IPs and service proxies’ paragraph, and as I see in my tests, the load balance is per node (VM). I.e., if I have a cluster of 3 VMs and deployed a service with 6 pods (2 per VM), the load balancing will only be between the pods of the same VM which is somehow disappointing.
At least this is what I see in my tests: Calling the service from within the cluster using the service’s ClusterIP, will load-balance between the 2 pods that reside in the same VM that the call was sent from.
(BTW, the same goes when calling the service from out of the cluster (using NodePort) and then the request will load-balance between the 2 pods that reside in the VM which was the request target IP address).
Is the above correct?
If yes, how can I make internal cluster calls load-balance between all the 6 replicas? (Must I employ a load balancer like nginx for this?)
No, the statement is not correct. The loadbalancing should be across nodes (VMs). This demo demonstrates it. I have run this demo on a k8s cluster with 3 nodes on gce. It first creates a service with 5 backend pods, then it ssh into one gce node and visits the service.ClusterIP, and the traffic is loadbalanced to all 5 pods.
I see you have another question "not unique ip per pod" open, it seems you hadn't set up your cluster network properly, which might caused what you observed.
In your case, each node will be running a copy of the service - and load-balance across the nodes.