I'm using AWS EKS with fargate, in my deployment, I set the resources.requests.memory to 512 Mi, but the fargate node will still request 1Gi. So I looking the prometheus kube states for that pod, it show that pod request 762 Mi.
Because 762-512=250, so I was checked other deployments, found that it always request memory more than I defined about 200 Mi.
Is this a k8s issue, or just a fargate rule?
resources:
limits:
memory: 512Mi
cpu: "0.25"
requests:
memory: 512Mi
cpu: "0.25"
Annotations: CapacityProvisioned: 0.25vCPU 1GB
Memory Request: 762 Mi
After update requests to 250 Mi:
resources:
limits:
memory: 500Mi
cpu: "0.25"
requests:
memory: 250Mi
cpu: "0.25"
Annotations: CapacityProvisioned: 0.25vCPU 0.5GB
I found the explaination in Monitoring Amazon EKS on AWS Fargate using Prometheus and Grafana
The current version of the dashboard doesn’t consider initContainers’ requests. This is because kube-state-metrics doesn’t expose resources requested by initContainers.
The requests metric in the graph will be absent if none of the long-running containers request any resources. The request metric should not be confused with the total CPU and memory the pod has at its disposal. The pod’s CPU and memory is determined by the calculated Fargate configuration of the pod, as explained above.
sum(kube_pod_container_resource_requests{resource="memory"}+262144000)
If I request 512 Mi, the fargate node will request another 250 Mi for the init container,
my final memory request will be 762 Mi.
So if I want to keep the fargate capacity provisioned is 0.5GB, my request memory should not be greater than 262 Mi.
I don't think this is reasonable.
The additional 256m is used for kubelet and other daemon processes rather than your init container.
Fargate adds 256 MB to each pod’s memory reservation for the required Kubernetes components (kubelet, kube-proxy, and containerd).
Related
I am trying to add redis request rate limiter to a gateway project. Redis cluster is already up with 6 nodes in docker. But it seems the redis request rate limiter not working in the gateway project.
Here is the config
spring:
redis:
cluster:
nodes: ${REDIS_CLUSTER_NODES}
maxRedirects: ${REDIS_CLUSTER_MAX_REDIRECTS}
...
filters:
- name: RequestRateLimiter
args:
key-resolver: "#{#userRemoteAddressResolver}"
redis-rate-limiter.replenishRate: 1
redis-rate-limiter.burstCapacity: 2
redis-rate-limiter.requestedTokens: 1
There is no error message and no 429 HttpStatus in responses. Does RequestRateLimiter not work with Redis-Cluster? Am i missing something? Thanks in advance
I'm trying to create a minimal cluster with 1 node and 1 GPU/node. My command:
gcloud container clusters create cluster-gpu --num-nodes=1 --zone=us-central1-a --machine-type="n1-highmem-2" --accelerator="type=nvidia-tesla-k80,count=1" --scopes="gke-default,storage-rw"
creates the cluster. Now when the following pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: gke-training-pod-gpu
spec:
containers:
- name: my-custom-container
image: gcr.io/.../object-classification:gpu
resources:
limits:
nvidia.com/gpu: 1
is applied to my cluster, I can see in the GKE dashboard that the gke-training-pod-gpu pod is never created. When I do the same as above, only replacing num-nodes=1 by num-nodes=2, this time I get the following error:
ERROR: (gcloud.container.clusters.create) ResponseError: code=403, message=Insufficient regional quota to satisfy request: resource "NVIDIA_K80_GPUS": request requires '2.0' and is short '1.0'. project has a quota of '1.0' with '1.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED&project=...
Is there any way to use a GPU when the quota is 1?
EDIT:
when pod has been created with kubectl apply command, a kubectl describe pod gke-training-pod-gpu command shows following event:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 48s (x2 over 48s) default-scheduler 0/1 nodes are available: 1 Insufficient nvidia.com/gpu.
Looks like you need to install the NVIDIA CPU device driver on your worker node(s).
Running
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
should do the trick.
The best solution as I see it is to request a quota increase in the IAM & Admin Quotas page.
As for the reason this is happening, I can only imagine that both the node and the pod are requesting GPUs, but only the node is getting it because of the capped quota.
Question 1.)
Given the scenario a multi-container pod, where all containers have a defined CPU request:
How would Kubernetes Horizontal Pod Autoscaler calculate CPU Utilization for Multi Container pods?
Does it average them? (((500m cpu req + 50m cpu req) /2) * X% HPA target cpu utilization
Does it add them? ((500m cpu req + 50m cpu req) * X% HPA target cpu utilization
Does it track them individually? (500m cpu req * X% HPA target cpu utilization = target #1, 50m cpu req * X% HPA target cpu utilization = target #2.)
Question 2.)
Given the scenario of a multi-container pod, where 1 container has a defined CPU request and a blank CPU request for the other containers:
How would Kubernetes Horizontal Pod Autoscaler calculate CPU Utilization for Multi Container pods?
Does it work as if you only had a 1 container pod?
Question 3.)
Do the answers to questions 1 and 2 change based on the HPA API version? I noticed stable/nginx-ingress helm chart, chart version 1.10.2, deploys an HPA for me with these specs:
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
(I noticed apiVersion: autoscaling/v2beta2 now exists)
Background Info:
I recently had an issue with unexpected wild scaling / constantly going back and forth between min and max pods after adding a sidecar(2nd container) to an nginx ingress controller deployment (which is usually a pod with a single container). In my case, it was an oauth2 proxy, although I image istio sidecar container folks might run into this sort of problem all the time as well.
apiVersion: apps/v1
kind: Deployment
spec:
replicas: 3
template:
spec:
containers:
- name: nginx-ingress-controller #(primary-container)
resources:
requests:
cpu: 500m #baseline light load usage in my env
memory: 2Gi #according to kubectl top pods
limits:
memory: 8Gi #(oom kill pod if this high, because somethings wrong)
- name: oauth2-proxy #(newly-added-2nd-sidecar-container)
resources:
requests:
cpu: 50m
memory: 50Mi
limits:
memory: 4Gi
I have an HPA (apiVersion: autoscaling/v1) with:
min 3 replicas (to preserve HA during rolling updates)
targetCPUUtilizationPercentage = 150%
It occurred to me that my misconfiguration leads to unexpected wild scaling was caused by 2 issues:
I don't actually understand how HPAs work when the pod has multiple containers
I don't know how to dig deep to get metrics of what's going on.
To address the first issue: I brainstormed my understanding of how it works in the single container scenario (and then realized I don't know the multi-container scenario so I decided to ask this question)
This is my understanding of how HPA (autoscaling/v1) works when I have 1 container (temporarily ignore the 2nd container in the above deployment spec):
The HPA would spawn more replicas when the CPU utilization average of all pods shifted from my normal expected load of 500m or less to 750m (150% x 500m request)
To address the 2nd issue: I found out how to dig to see concrete numeric value-based metrics vs relative percentage-based metrics to help figure out what's happening behind the scenes:
bash# kubectl describe horizontalpodautoscaler nginx-ingress-controller -n=ingress | grep Metrics: -A 1
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 5% (56m) / 100%
(Note: kubectl top pods -n=ingress, showed cpu usage of the 5 replicas as 36m, 34m, 88m, 36m, 91m, so that 57m current which ~matches 56m current)
Also now it's a basic proportions Math Problem that allows solving for target static value:
(5% / 56m) = (100% / x m) --> x = 56 * 100 / 5 = 1120m target cpu
(Note: this HPA isn't associated with the deployment mentioned above, that's why the numbers are off.)
Basing on stackoverflow community member answer in other case
"HPA calculates pod cpu utilization as total cpu usage of all containers in pod divided by total request. I don't think that's specified in docs anywhere, but the relevant code is here"
You have got more informations,with examples in the link above.
Basing on documentation
Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization (or, with beta support, on some other, application-provided metrics).
So basically:
apiVersion autoscaling/v1 HPA base on cpu.
apiVersion autoscaling/v2beta2 base on cpu,memory,custom metrics.
More informations here
For question 2 if we look at source code we can see that it looks all containers individually and returns if container doesn't have request.
func calculatePodRequests(pods []*v1.Pod, resource v1.ResourceName) (map[string]int64, error) {
requests := make(map[string]int64, len(pods))
for _, pod := range pods {
podSum := int64(0)
for _, container := range pod.Spec.Containers {
if containerRequest, ok := container.Resources.Requests[resource]; ok {
podSum += containerRequest.MilliValue()
} else {
return nil, fmt.Errorf("missing request for %s", resource)
}
}
requests[pod.Name] = podSum
}
return requests, nil
}
I appear to have some problem with my installation of OpenShift Origin.
When I get endpoints for the router, I get the following:
oc get endpoints --namespace=default --selector=router
NAME ENDPOINTS AGE
router-west <none> 21m
Obviously the router should have at least one endpoint.
Im trying to follow the troubleshooting guide on https://docs.openshift.com/enterprise/3.1/admin_guide/sdn_troubleshooting.html#debugging-the-router however it does not provide assistance in the situation where the router has not endpoints.
When I get my list of nodes, I get:
oc get nodes
NAME LABELS STATUS AGE
openshift.hughestech.space kubernetes.io/hostname=openshift.mydomain.com NotReady 38d
When I describe the node, I get the following:
oc describe node openshift.mydomain.com
Name: openshift.mydomain.com
Labels: kubernetes.io/hostname=openshift.mydomain.com
CreationTimestamp: Sat, 06 Feb 2016 21:44:23 +0100
Phase:
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
──── ────── ───────────────── ────────────────── ────── ───────
Ready Unknown Fri, 04 Mar 2016 18:50:39 +0100 Fri, 04 Mar 2016 18:51:21 +0100 NodeStatusUnknown Kubelet stopped posting node status.
Addresses: 88.198.37.183,88.198.37.183
Capacity:
memory: 24515560Ki
pods: 40
cpu: 8
System Info:
Machine ID: bafaea4f3c4c4cf6a632047c1d14db1a
System UUID: 00000000-0000-0000-0000-002421DDE3D7
Boot ID: f9febe14-ec61-41d5-b7c3-db2e42f9b452
Kernel Version: 3.10.0-327.4.5.el7.x86_64
OS Image: Red Hat Enterprise Linux
Container Runtime Version: docker://1.8.2-el7
Kubelet Version: v1.1.0-origin-1107-g4c8e6f4
Kube-Proxy Version: v1.1.0-origin-1107-g4c8e6f4
ExternalID: openshift.mydomain.com
Non-terminated Pods: (0 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
───────── ──── ──────────── ────────── ─────────────── ─────────────
Allocated resources:
(Total limits may be over 100%, i.e., overcommitted. More info: http://releases.k8s.io/HEAD/docs/user-guide/compute-resources.md)
CPU Requests CPU Limits Memory Requests Memory Limits
──────────── ────────── ─────────────── ─────────────
0 (0%) 0 (0%) 0 (0%) 0 (0%)
No events.
Where have I gone wrong? What do I need to do?
Thanks
Restart the node service and see if that makes a difference in oc get nodes output.
systemctl restart origin-node
Unless your node is running you can cannot make a running router pod and resulting in no endpoints.
I run a very simple website (basically a redirect based on a php database) which gets on average 5 visits per second throughout the day, but at peak times (usually 2-3 times a day), this may go up to even 300 visits/s or more. I've modified the default apache settings as follows (based on various info found online as I'm not an expert):
Start Servers: 5 (default) / 25 (now)
Minimum Spare Servers: 5 (default) / 50 (now)
Maximum Spare Servers: 10 (default) / 100 (now)
Server Limit: 256 (default) / 512 (now)
Max Clients: 150 (default) / 450 (now)
Max Requests Per Child: 10000 (default)
Keep-Alive: On (default) / Off (now)
Timeout: 300 (default)
Server (VPS) specs:
4x 3199.998 MHz, 512 KB Cache, QEMU Virtual CPU version (cpu64-rhel6)
8GB RAM (Memory: 8042676k/8912896k available (5223k kernel code, 524700k absent, 345520k reserved, 7119k data, 1264k init)
70GB SSD disk
CENTOS 6.5 x86_64 kvm – server
During average loads the server handles just fine. Problems occur almost every day during peak traffic times, as in http time-outs or extremely long response/load times.
Question is, do I need to get a better server or can I improve response times during peak traffic by further tuning Apache config? Any help would be appreciated. Thank you!
maybe you need to enable mod_cache with mod_mem_cache, another parameter that i always configure is ulimits:
nofile to get more sockets
nproc to get more processes
http://www.webperformance.com/load-testing/blog/2012/12/setting-apache2-ulimit-for-maximum-prefork-performance/
finally TCP Tuning and Network, check all net.core and net.ipv4 parameters to get less latency
http://fasterdata.es.net/host-tuning/linux/