I ran into this problem and solved it, so this Q&A is here just in case somebody else spent time on sifting through AWS, Keda, and/or Kubernetes' docs trying to deduce this answer
My team deployed Keda with the goal of horizontally autoscaling pods in Kubernetes based on Redis queue length, CPU utilization, and memory utilization. Post-deployment, we noticed the horizontal pod autoscaler as viewed in ArgoCD was throwing the error:
unable to get metrics for resource memory: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Some other errors we saw:
$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1"
Error from server (NotFound): the server could not find the requested resource
$ kubectl top nodes
error: Metrics API not available
Of course, these errors are in addition to the human-visible issue of pods not autoscaling when CPU/Memory utilization thresholds are reached.
This occurs despite the redis queue pod scaling as expected via Keda/HPA.
What can we do to make sure the CPU and Memory utilization causes scaling as expected?
As it turns out, this is caused by:
Keda uses metrics-server under the hood for CPU/Memory utilization metrics; this is not noted on the CPU and Memory utilization pages in their documentation, but rather is noted in a sidebar on their Keda metrics querying page (as of this writing, https://keda.sh/docs/2.8/operate/metrics-server/) that reads: Note: There are 2 exceptions in querying metrics and those are cpu and memory scalers. When KEDA creates the HPA object, it uses standard cpu and memory metrics from the Kubernetes Metrics Server. If you want to query these 2 specific values, you should do it using /apis/metrics.k8s.io/v1beta1 instead of /apis/external.metrics.k8s.io/v1beta1
AWS EKS does not install metrics-server by default. For reference, as of this writing AWS docs read "In Amazon EKS, Metrics Server isn't installed by default." (https://aws.amazon.com/premiumsupport/knowledge-center/eks-metrics-server/)
To resolve this, we installed metrics-server via its Helm chart to our cluster.
Related
While I upgraded the Ignite that is deployed in Kubernetes (EKS) for Log4j vulnerability, I get the error below
[ignite-1] Caused by: class org.apache.ignite.spi.IgniteSpiException: BaselineTopology of joining node (54b55de4-7742-4e82-9212-7158bf51b4a9) is not compatible with BaselineTopology in the cluster. Joining node BlT id (4) is greater than cluster BlT id (3). New BaselineTopology was set on joining node with set-baseline command. Consider cleaning persistent storage of the node and adding it to the cluster again.
The setup is a 3 node cluster, with native persistence enabled (PVC). This seems to be occurring many times in our journey with Apache Ignite, having followed the official guide.
I cannot clean the storage as the pod gets restarted every now and then, by the time I get the pod shell the pod crash & restarts.
This might happen to be due to the wrong startup order, starting nodes manually in reverse order may resolve this, but I'm not sure if that is possible in K8s. Another possible issue might be related to the baseline auto-adjustment that might change your baseline unexpectedly, I suggest you turn it off if it's enabled.
One of the workarounds to clean a DB of a failing POD might be (quite tricky) - to replace Ignite image with some simple image like a plain Debian or Alpine docker images (just to be able to access CLI) keeping the same PVC attached, and once you fix the persistence issue, set the Ignite image back. The other one is - to access underlying PV directly if possible and do surgery in place.
I am doing load test to tune my apache to server maximum concurrent https request. Below is the details of my test.
System
I dockerized my httpd and deployed in openshift with pod configuration is 4CPU, 8GB RAM.
Running load from Jmeter with 200 thread, 600sec ramup time, loop is for infinite. duration is long run (Jmeter is running in same network with VM configuration 16CPU, 32GB RAM ).
I compiled by setting module with worker and deployed in openshift.
Issue
Httpd is not scaling more than 90TPS, even after tried multiple mpm worker configuration (no difference with default and higher configuration)
2.Issue which i'am facing after 90TPS, average time is increasing and TPS is dropping.
Please let me know what could be the issue, if any information is required further suggestions.
I don't have the answer, but I do have questions.
1/ What does your Dockerfile look like?
2/ What does your OpenShift cluster look like? How many nodes? Separate control plane and workers? What version?
2b/ Specifically, how is traffic entering the pod (if you are going in via a route, you'll want to look at your load balancer; if you want to exclude OpenShift from the equation then for the short term, expose a NodePort and have Jmeter hit that directly)
3/ Do I read correctly that your single pod was assigned 8G ram limit? Did you mean the worker node has 8G ram?
4/ How did you deploy the app -- raw pod, deployment config? Any cpu/memory limits set, or assumed? Assuming a deployment, how many pods does it spawn? What happens if you double it? Doubled TPS or not - that'll help point to whether the problem is inside httpd or inside the ingress route.
5/ What's the nature of the test request? Does it make use of any files stored on the network, or "local" files provisioned in a network PV.
And,
6/ What are you looking to achieve? Maximum concurrent requests in one container, or maximum requests in the cluster? If you've not already look to divide and conquer -- more pods on more nodes.
Most likely you have run into a bottleneck/limitation at the SUT. See the following post for a detailed answer:
JMeter load is not increasing when we increase the threads count
I'm running a task at fargate with CPU as 2048 and memory as 8192. Task after running some time is stopped with error
container was stopped as it ran out of memory.
Thing is that task does not fails every time. If I run the same task 10 time it fails 5 times and works 5 times. However If I take an ec2 machine with 2 vcpu and 4GB memory and try to run the same container it runs successfully.(Infact the memory usage on ec2 instance is very low).
Can somebody please guide me how to figure out the memory issue while running a fargate task?
Thanks
The way to start would be enabling memory metrics from container insights for your fargate tasks and Further correlating the Memory Usage graph with Application logs. help here
The difference between running on EC2 vs Fargate could probably be due to the fact that when you run a container on ECS Fargate, it runs on AWS's internal EC2 Instances. Now, here could possibly arise a Noisy Neighbour Situation although the chances would be pretty low.
I am trying to set up hive using mr3 on a kubernetes cluster hosted on AWS ec2. When I run the command run-hive.sh, Hive-server starts and the master-DAg is initialised but then it gets stuck on pending. When I describe the pod. This is the error message shows. I have kept the resources to minimum so it should not be that issue and I do not have any tainted nodes. If you know any alternative for running hive on Kubernetes with access to S3 or a better way to implement mr3 hive on Kubernetes cluster, please share.
One of the node description
Based on the topic i think the problem here is your cluster have not enough resources on your worker nodes, and a master node is tainted.
So the option here is either inreasing the resources on workers or taint the master node so You would be able to schedule pods there.
Control plane node isolation
By default, your cluster will not schedule pods on the control-plane node for security reasons. If you want to be able to schedule pods on the control-plane node, e.g. for a single-machine Kubernetes cluster for development, run:
kubectl taint nodes --all node-role.kubernetes.io/master-
This will remove the node-role.kubernetes.io/master taint from any nodes that have it, including the control-plane node, meaning that the scheduler will then be able to schedule pods everywhere
After some readings, it seems there is no sustainable solution for auto-scaling Redis on Kubernetes without adding a controller like Maestro. Unfortunatly the project seems a bit dead.
What are some alternatives for autoscaling Redis ?
Edit: Redis is a statefull app.
If you want to autoscale anything on Kubernetes, it requires some type of controller. For general autoscaling, the community is rallying around the Horizontal Pod Autoscaler. By default, you configure it to scale based on CPU utilization.
If you want to scale based on metrics other than CPU utilization and you're using the Redis helm chart, you can easily configure it to run a Prometheus metric sidecar and can set the autoscaler to scale based on one of those values.