My colleague and I have a question on how one Kubernetes Pod can serve multiple users. Our belief is/was that one Pod serves a single end user. Here is our experiment.
We have a basic Angular Frontend talking to a .Net API Server that uses a MySQL PVClaim structure running on a Linode Kubernetes Cluster. The Cluster has 3 nodes with plenty of RAM, disk space, and CPUs. Here is a print out of 'kubectl get all':
C:\development\Kubernetes >kubectl get all
NAME READY STATUS RESTARTS AGE
pod/dataapi-server-57c87f56c6-2fcdz 1/1 Running 0 21h
pod/gems-deployment-c9f6dbdbb-2qhff 1/1 Running 0 20h
pod/mysql-6d4b9f6f46-8m66h 1/1 Running 0 21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dataapi-server ClusterIP 10.128.6.155 <none> 80/TCP 21h
service/gems-frontend-service LoadBalancer 10.128.76.185 104.***.**.** 80:31190/TCP 20h
service/kubernetes ClusterIP 10.128.0.1 <none> 443/TCP 4d23h
service/mysql ClusterIP None <none> 3306/TCP 21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dataapi-server 1/1 1 1 21h
deployment.apps/gems-deployment 1/1 1 1 20h
deployment.apps/mysql 1/1 1 1 21h
My colleague and I, working side by side on two different client machines, launched the frontend in our respective Chrome browsers and both did some basic independent work. Surprisingly to us, we were both able to hit the same Pod ('gems-deployment-c9...') and work productively. Based on everything we understood on Pods and virtual machines, this did not make sense to us. I will note here we are reasonably new to the Kubernetes and containers world.
Can anyone explain why this works? Are there any papers one could point us to? My Google searches have only turned up papers on multiple containers per Pod, which is not our case.
Thanks in advance!
When you make a regular HTTP GET request to your Angular app, the Load Balancer (Service) determines which pod should take the request based on the strain on the nodes. Because you have one frontend pod, it only resides on one node, so that node is selected.
The node then creates a new thread that lasts as long as it takes to send you a response, which is the webpage. If your requests were simultaneous, the node would typically just open another thread and serve you both simultaneously, which is the same thing that would happen if you were hosting this frontend off of Kubernetes.
For what you're worrying about to become a problem, the Node would have to be receiving more requests than it could effectively deal with. In that case, optimizing the Load Balancer or improving Node power are both valid responses.
Related
The scenario: Selenium is a browser automation tool that can be run in a K8s cluster, it consists of Selenium-hub (master) and selenium-nodes (workers) where the hub receives test requests and creates nodes (pods) ondemand (dynamically) to run the test-case, after execution of a test-case the runner node (pod) gets thrown away. also, Selenium supports live-preview of the test being run by the runner and a client (outside of K8s) can basically watch this live preview, there is a little change that when a client is watching the live preview of the test and it ends, another pod gets created with the same IP that the client is actually is still watching, this is a problem since the client may continue watching the run of another test because the client's software is not aware of the length of run and may still fetch the traffic with same user/pass/IP combination.
The question: is it possible to change the way Kubernetes assigns IP addresses?
let's say the first pod to be created gets IP 1.1.1.1 and the second one gets 1.1.1.2 and third 1.1.1.3, and before the fourth request the first pod dies and its IP is free then the fourth pod would be created with IP 1.1.1.1,
What I am trying to do is to tell to the Kubernetes to use previously assigned IP after some time or change the sequence of IP assignment or something similar.
Any ideas?
Technically: yes you can either configure or edit-the-code of your CNI plugin (or write one from scratch).
In practice: I know of none that work quite that way. I know Calico does allow having multiple IP pools so you could have a small one just for Selenium pods but I think it still attempts to minimize reuse. But check the docs for your CNI plugin and see what it offers.
I am doing load test to tune my apache to server maximum concurrent https request. Below is the details of my test.
System
I dockerized my httpd and deployed in openshift with pod configuration is 4CPU, 8GB RAM.
Running load from Jmeter with 200 thread, 600sec ramup time, loop is for infinite. duration is long run (Jmeter is running in same network with VM configuration 16CPU, 32GB RAM ).
I compiled by setting module with worker and deployed in openshift.
Issue
Httpd is not scaling more than 90TPS, even after tried multiple mpm worker configuration (no difference with default and higher configuration)
2.Issue which i'am facing after 90TPS, average time is increasing and TPS is dropping.
Please let me know what could be the issue, if any information is required further suggestions.
I don't have the answer, but I do have questions.
1/ What does your Dockerfile look like?
2/ What does your OpenShift cluster look like? How many nodes? Separate control plane and workers? What version?
2b/ Specifically, how is traffic entering the pod (if you are going in via a route, you'll want to look at your load balancer; if you want to exclude OpenShift from the equation then for the short term, expose a NodePort and have Jmeter hit that directly)
3/ Do I read correctly that your single pod was assigned 8G ram limit? Did you mean the worker node has 8G ram?
4/ How did you deploy the app -- raw pod, deployment config? Any cpu/memory limits set, or assumed? Assuming a deployment, how many pods does it spawn? What happens if you double it? Doubled TPS or not - that'll help point to whether the problem is inside httpd or inside the ingress route.
5/ What's the nature of the test request? Does it make use of any files stored on the network, or "local" files provisioned in a network PV.
And,
6/ What are you looking to achieve? Maximum concurrent requests in one container, or maximum requests in the cluster? If you've not already look to divide and conquer -- more pods on more nodes.
Most likely you have run into a bottleneck/limitation at the SUT. See the following post for a detailed answer:
JMeter load is not increasing when we increase the threads count
I'm trying to optimize Docker-Swarm load-balancing in a way that it will first route requests to services by the following priority
Same machine
Same DC
Anywhere else.
Given the following setup:
DataCenter-I
Server-I
Nginx:80
Server-II
Nginx:80
Worker
DataCenter-II
Server-I
Nginx:80
Worker
In case and DataCenter-I::Server-II::Worker will issue an API request over port 80, The desired behavior is:
Check if there are any tasks (containers) mapped to port:80 on local server (DataCenter-I::Server-II)
Fallback and check in local DataCenter (i.e DataCenter-I::Server-I)
Fallback and check in all clusters (i.e DataCenter-II::Server-I)
This case is very useful when using workers and response time doesn't matter while bandwidth does.
Please advise,
Thanks!
According to this question I asked before, docker swarm is currently only using round-robin and no indication to be pluginable yet.
However, Nginx Plus support least_time load balancing method, which I think there will be an similar open-source module, and it is similar to what you need, with perhaps the least effort.
ps: Don't run Nginx with the docker swarm. Instead, run Nginx with regular docker or docker-compose in the same docker network of your app. You don't want docker swarm to load balancing your load balancer.
Playing a bit with Kubernetes (v1.3.2) I’m checking the ability to load balance calls inside the cluster (3 on-premise CentOS 7 VMs).
If I understand correctly the documentation in http://kubernetes.io/docs/user-guide/services/ ‘Virtual IPs and service proxies’ paragraph, and as I see in my tests, the load balance is per node (VM). I.e., if I have a cluster of 3 VMs and deployed a service with 6 pods (2 per VM), the load balancing will only be between the pods of the same VM which is somehow disappointing.
At least this is what I see in my tests: Calling the service from within the cluster using the service’s ClusterIP, will load-balance between the 2 pods that reside in the same VM that the call was sent from.
(BTW, the same goes when calling the service from out of the cluster (using NodePort) and then the request will load-balance between the 2 pods that reside in the VM which was the request target IP address).
Is the above correct?
If yes, how can I make internal cluster calls load-balance between all the 6 replicas? (Must I employ a load balancer like nginx for this?)
No, the statement is not correct. The loadbalancing should be across nodes (VMs). This demo demonstrates it. I have run this demo on a k8s cluster with 3 nodes on gce. It first creates a service with 5 backend pods, then it ssh into one gce node and visits the service.ClusterIP, and the traffic is loadbalanced to all 5 pods.
I see you have another question "not unique ip per pod" open, it seems you hadn't set up your cluster network properly, which might caused what you observed.
In your case, each node will be running a copy of the service - and load-balance across the nodes.
I've been using ServiceStack PooledRedisClientManager with success. I'm now adding Twemproxy into the mix and have 4 Redis instances fronted with Twemproxy running on a single Ubuntu server.
This has caused problems with light load tests (100 users) connecting to Redis through ServiceStack. I've tried the original PooledRedisClientManager and BasicRedisClientManager, both are giving the error No connection could be made because the target machine actively refused it
Is there something I need to do to get these two to play nice together? This is the Twemproxy config
alpha:
listen: 0.0.0.0:12112
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
redis: true
timeout: 400
server_retry_timeout: 30000
server_failure_limit: 3
server_connections: 1000
servers:
- 0.0.0.0:6379:1
- 0.0.0.0:6380:1
- 0.0.0.0:6381:1
- 0.0.0.0:6382:1
I can connect to each one of the Redis server instances individually, it just fails going through Twemproxy.
I haven't used twemproxy before but I would say your list of servers is wrong. I don't think you are using 0.0.0.0 correctly.
Your servers would need to be (for your local testing):
servers:
- 127.0.0.1:6379:1
- 127.0.0.1:6380:1
- 127.0.0.1:6381:1
- 127.0.0.1:6382:1
You use 0.0.0.0 on the listen command to tell twemproxy to listen on all available network interfaces on the server. This mean twemproxy will try to listen on:
the loopback address 127.0.0.1 (localhost),
on your private IP (i.e. 192.168.0.1) and
on your public IP (i.e. 134.xxx.50.34)
When you are specifying servers, the server config needs to know the actual address it should connect on. 0.0.0.0 doesn't make sense. It needs a real value. So when you come to use different Redis machines you will want to use, the private IPs of each machine like this:
servers:
- 192.168.0.10:6379:1
- 192.168.0.13:6379:1
- 192.168.0.14:6379:1
- 192.168.0.27:6379:1
Obviously your IP addresses will be different. You can use ifconfig to determine the IP on each machine. Though it may be worth using a hostname if your IPs are not statically assigned.
Update:
As you have said you are still having issues, I would make these recommendations:
Remove auto_eject_hosts: true. If you were getting some connectivity, then after time you end up with no connectivity, it's because something has caused twemproxy to think there was something wrong with the Redis hosts and reject them.
So eventually when your ServiceStack client connects to twemproxy, there will be no hosts to pass the request onto and you get the error No connection could be made because the target machine actively refused it.
Do you actually have enough RAM to stress test your local machine this way? You are running at least 4 instances of Redis, which require real memory to store the values, twemproxy consumes a large amount of memory to buffer the requests it passes to Redis, this memory pool is never released, see here for more information. Your ServiceStack app will consume memory - more so in Debug mode. You'll probably have Visual Studio or another IDE open, the stress test application, and your operating system. On top of all that there will likely be background processes and other applications you haven't closed.
A good practice is to try to run tests on isolated hardware as far as possible. If it is not possible, then the system must be monitored to check the benchmark is not impacted by some external activity.
You should read the Redis article here about benchmarking.
As you are using this in a localhost situation use the BasicRedisClientManager not the PooledRedisClientManager.