We started using express-gateway as a simple proxy with some rewrite and jwt verification (jwt comes from Auth0). No problem there.
For some endpoints I needed basic-auth so I needed to set up persistence.
We use Kubernetes to run our apps and managed databases all on the same cloud provider.
I noticed long response time for endpoints using basic-auth.
I started running some basic benchmarks. Here are the results.
Results Summary
Directly Calling my App
284 Requests/sec
Express Gateway - Proxy Only
285 Requests/sec
Express Gateway - Proxy + Logs
270 Requests/sec
Express Gateway (Managed Redis) - Proxy + Logs + Basic Auth
7 Requests/sec
Express Gateway (Embedded Redis) - Proxy + Logs + Basic Auth
7 Requests/sec
I first though that my managed Redis had poor performances (its supposed to be on the same data-center has the kubernetes cluster) but the embedded Redis has the same performances.
I don't need huge performances but it seams to me that there is an issue with the Basic Auth / Redis implementation.
Is this expected ? Do I need a huge Redis database ?
What can I do to troubleshoot this issue ?
Results Details
Shared
HTTP benchmarks are done with HEY.
Express-Gateway docker image : express-gateway:1.16.x
Express-Gateway gateway.config.yml (redacted) :
http:
port: ${HTTP_PORT:-8080}
admin:
port: ${ADMIN_PORT:-9876}
host: ${ADMIN_HOSTNAME:-localhost}
apiEndpoints:
app-api:
- host: ${APP_HOST}
paths:
- '/some/path'
methods: ["POST"]
scopes: ["scope:write", "scope:read"]
serviceEndpoints:
app-api:
url: ${SERVICE_HOST_APP}
policies:
- log
- proxy
- basic-auth
pipelines:
app-api:
apiEndpoints:
- app-api
policies:
- log:
- action:
message: "${req.method} ${req.originalUrl}"
- basic-auth:
- proxy:
- action:
changeOrigin: false
serviceEndpoint: app-api
Express-Gateway Resources :
resources:
requests:
memory: "512M"
cpu: "500m"
limits:
memory: "1024M"
cpu: "1000m"
App Resources :
resources:
requests:
memory: "512M"
cpu: "500m"
limits:
memory: "1024M"
cpu: "1000m"
1. Directly Calling my App
Command :
./hey_linux_amd64 -n 10000 -m POST -T "application/json" -d <body> <app url>
Results :
Summary:
Total: 35.2033 secs
Slowest: 0.6221 secs
Fastest: 0.0335 secs
Average: 0.1685 secs
Requests/sec: 284.0639
Response time histogram:
0.034 [1] |
0.092 [2689] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.151 [1830] |■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.210 [1781] |■■■■■■■■■■■■■■■■■■■■■■■■■■
0.269 [2197] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.328 [1271] |■■■■■■■■■■■■■■■■■■■
0.387 [130] |■■
0.446 [31] |
0.504 [37] |■
0.563 [22] |
0.622 [11] |
Latency distribution:
10% in 0.0435 secs
25% in 0.0852 secs
50% in 0.1643 secs
75% in 0.2434 secs
90% in 0.2834 secs
95% in 0.3056 secs
99% in 0.3944 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0007 secs, 0.0335 secs, 0.6221 secs
DNS-lookup: 0.0001 secs, 0.0000 secs, 0.0316 secs
req write: 0.0000 secs, 0.0000 secs, 0.0083 secs
resp wait: 0.1676 secs, 0.0334 secs, 0.6217 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0027 secs
Status code distribution:
[200] 10000 responses
Resource Usage :
> kubectl top pods
NAME CPU(cores) MEMORY(bytes)
app-6fb8b4787d-9qpf2 920m 254Mi
2. Express Gateway - Proxy Only
Command :
./hey_linux_amd64 -m POST -T "application/json" -d <body> <express-gateway url>
Results:
Summary:
Total: 35.0445 secs
Slowest: 0.7521 secs
Fastest: 0.0349 secs
Average: 0.1672 secs
Requests/sec: 285.3516
Response time histogram:
0.035 [1] |
0.107 [3145] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.178 [2298] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.250 [2286] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.322 [2046] |■■■■■■■■■■■■■■■■■■■■■■■■■■
0.394 [153] |■■
0.465 [32] |
0.537 [25] |
0.609 [9] |
0.680 [4] |
0.752 [1] |
Latency distribution:
10% in 0.0450 secs
25% in 0.0865 secs
50% in 0.1628 secs
75% in 0.2436 secs
90% in 0.2796 secs
95% in 0.2989 secs
99% in 0.3538 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0009 secs, 0.0349 secs, 0.7521 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0074 secs
req write: 0.0000 secs, 0.0000 secs, 0.0042 secs
resp wait: 0.1662 secs, 0.0348 secs, 0.7520 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0028 secs
Status code distribution:
[200] 10000 responses
Resource Usage :
> kubectl top pods
NAME CPU(cores) MEMORY(bytes)
express-gateway-56699967d8-hnvzw 320m 50Mi
app-6fb8b4787d-9qpf2 743m 252Mi
3. Express Gateway - Proxy + Logs
Command :
./hey_linux_amd64 -m POST -T "application/json" -d <body> <express-gateway url>
Results:
Summary:
Total: 36.9277 secs
Slowest: 0.6278 secs
Fastest: 0.0356 secs
Average: 0.1796 secs
Requests/sec: 270.7991
Response time histogram:
0.036 [1] |
0.095 [1595] |■■■■■■■■■■■■■■■■■■■■■■■■
0.154 [2616] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.213 [2604] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.272 [1756] |■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.332 [996] |■■■■■■■■■■■■■■■
0.391 [301] |■■■■■
0.450 [71] |■
0.509 [22] |
0.569 [32] |
0.628 [6] |
Latency distribution:
10% in 0.0734 secs
25% in 0.1195 secs
50% in 0.1709 secs
75% in 0.2316 secs
90% in 0.2932 secs
95% in 0.3262 secs
99% in 0.4061 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0007 secs, 0.0356 secs, 0.6278 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0038 secs
req write: 0.0000 secs, 0.0000 secs, 0.0053 secs
resp wait: 0.1787 secs, 0.0355 secs, 0.6275 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0063 secs
Status code distribution:
[200] 10000 responses
Resource Usage :
> kubectl top pods
NAME CPU(cores) MEMORY(bytes)
express-gateway-56699967d8-hnvzw 760m 94Mi
app-6fb8b4787d-9qpf2 898m 253Mi
4. Express Gateway (Managed Redis) - Proxy + Logs + Basic Auth
Command :
./hey_linux_amd64 -m POST -H "Authorization: Basic ..." -T "application/json" -d <body> <express-gateway url>
Results:
Summary:
Total: 1386.7479 secs
Slowest: 11.4025 secs
Fastest: 0.2489 secs
Average: 6.9121 secs
Requests/sec: 7.2111
Response time histogram:
0.249 [1] |
1.364 [15] |
2.480 [77] |■
3.595 [148] |■
4.710 [236] |■■
5.826 [493] |■■■■
6.941 [2836] |■■■■■■■■■■■■■■■■■■■■■
8.056 [5292] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
9.172 [595] |■■■■
10.287 [231] |■■
11.403 [76] |■
Latency distribution:
10% in 5.8535 secs
25% in 6.8129 secs
50% in 7.0067 secs
75% in 7.1767 secs
90% in 7.8473 secs
95% in 8.5065 secs
99% in 10.0107 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0008 secs, 0.2489 secs, 11.4025 secs
DNS-lookup: 0.0001 secs, 0.0000 secs, 0.0252 secs
req write: 0.0000 secs, 0.0000 secs, 0.0072 secs
resp wait: 6.9111 secs, 0.2488 secs, 11.4024 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0090 secs
Status code distribution:
[200] 9990 responses
[502] 10 responses
Resource Usage :
> kubectl top pods
NAME CPU(cores) MEMORY(bytes)
express-gateway-56699967d8-hnvzw 966m 62Mi
app-6fb8b4787d-9qpf2 20m 245Mi
5. Express Gateway (Embedded Redis) - Proxy + Logs + Basic Auth
Command :
./hey_linux_amd64 -m POST -H "Authorization: Basic ..." -T "application/json" -d <body> <express-gateway url>
Results:
Summary:
Total: 1395.9956 secs
Slowest: 11.2510 secs
Fastest: 0.2258 secs
Average: 6.9731 secs
Requests/sec: 7.1633
Response time histogram:
0.226 [1] |
1.328 [3] |
2.431 [13] |
3.533 [3] |
4.636 [12] |
5.738 [145] |■
6.841 [2597] |■■■■■■■■■■■■■■■
7.943 [6804] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
9.046 [375] |■■
10.148 [29] |
11.251 [18] |
Latency distribution:
10% in 6.3767 secs
25% in 6.8170 secs
50% in 6.9824 secs
75% in 7.1205 secs
90% in 7.5867 secs
95% in 7.8832 secs
99% in 8.5095 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0007 secs, 0.2258 secs, 11.2510 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0093 secs
req write: 0.0000 secs, 0.0000 secs, 0.0053 secs
resp wait: 6.9722 secs, 0.2255 secs, 11.2504 secs
resp read: 0.0001 secs, 0.0000 secs, 0.0126 secs
Status code distribution:
[200] 9993 responses
[502] 7 responses
Resource Usage :
> kubectl top pods
NAME CPU(cores) MEMORY(bytes)
express-gateway-56699967d8-hnvzw 651m 57Mi
app-6fb8b4787d-9qpf2 2m 252Mi
I've replicated the same setup that you've posted here as much as I could and I'm not getting the same poor performances you're experiencing.
I created an Azure Redis Cache on the Cloud and running EG locally on my machine (MacBook Pro 2019) and here are the results:
Concurrency Level: 50
Time taken for tests: 3.425 seconds
Complete requests: 500
Failed requests: 0
Non-2xx responses: 500
Total transferred: 99500 bytes
HTML transferred: 6000 bytes
Requests per second: 145.99 [#/sec] (mean)
Time per request: 342.478 [ms] (mean)
Time per request: 6.850 [ms] (mean, across all concurrent requests)
Transfer rate: 28.37 [Kbytes/sec] received
I've been using the same gateway configuration. I would say there's something going on in your Kubernetes Networking configuration.
Related
I try to apply limit_conn to my server for test purposes but it doesn't work.
There is my nginx.conf
events {}
http {
include mime.types;
limit_conn_zone $binary_remote_addr zone=addr:10m;
server {
listen 80;
location /downloads/ {
limit_conn addr 1;
return 200 "Hello";
}
}
}
Then I run Apache Benchmark
ab -n 10 -c 10 http://MY_IP_ADDRESS_OF_SERVER/downloads/
I get this output data:
This is ApacheBench, Version 2.3 <$Revision: 1757674 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking MY_IP_ADDRESS_OF_SERVER (be patient).....done
Server Software: nginx/1.16.1
Server Hostname: MY_IP_ADDRESS_OF_SERVER
Server Port: 80
Document Path: /downloads/
Document Length: 16 bytes
Concurrency Level: 10
Time taken for tests: 0.004 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 1760 bytes
HTML transferred: 160 bytes
Requests per second: 2693.97 [#/sec] (mean)
Time per request: 3.712 [ms] (mean)
Time per request: 0.371 [ms] (mean, across all concurrent requests)
Transfer rate: 463.03 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 2 0.3 2 2
Processing: 0 1 0.6 1 2
Waiting: 0 1 0.5 1 2
Total: 2 3 0.3 3 3
Percentage of the requests served within a certain time (ms)
50% 3
66% 3
75% 3
80% 3
90% 3
95% 3
98% 3
99% 3
100% 3 (longest request)
But I expect to get in output something like this
Complete requests: 10
Failed requests: 1
Non-2xx responses: 9
It seems that limit_conn doesn't work in my server. Why doesn't it work and how can i solve this problem?
I have been working with apache bench for a while now and until now it worked just fine. However, today I started getting several Non-2xx responses:. In order to investigate further, I tried to run a test with a simple website, so I run:
ab -n 100 -c 10 http://www.yahoo.com/
And this is what I got:
This is ApacheBench, Version 2.3 <$Revision: 1796539 $> Copyright 1996
Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to
The Apache Software Foundation, http://www.apache.org/
Benchmarking www.yahoo.com (be patient).....done
Server Software: ATS
Server Hostname: www.yahoo.com
Server Port: 80
Document Path: /
Document Length: 8 bytes
Concurrency Level: 10
Time taken for tests: 4.898 seconds
Complete requests: 100
Failed requests: 0
Non-2xx responses: 100
Total transferred: 36875 bytes
HTML transferred: 800 bytes
Requests per second: 20.42 [#/sec] (mean)
Time per request: 489.817 [ms] (mean)
Time per request: 48.982 [ms] (mean, across all concurrent requests)
Transfer rate: 7.35 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 39 48 6.3 47 66
Processing: 50 416 89.3 415 521
Waiting: 49 254 121.0 261 512
Total: 93 464 92.1 460 575
Percentage of the requests served within a certain time (ms)
50% 460
66% 476
75% 511
80% 541
90% 569
95% 572
98% 574
99% 575
100% 575 (longest request)
As the output shows, even with an external url, I get 100% Non-2xx responses. Does anyone know how I could fix this?
Thank you!
It might be because when accessing Yahoo.com they are redirecting you, and therefore you'll get 30X responses and not 20X directly.
In the book Tomcat The Definitive Guide written by Jason Brittain with Ian F.Darwin, when using the ab tool to benchmark, the writters says,
you should benchmark by running a minimum of 100,000 HTTP requests.
Also , you may configure the test client to spawn as many client threads as you would like,
but you will not get helpful results if you set it higher than the maxThreads you set for you Connector in your Tomcat's conf/server.xml file.
By default, it is set to 150.
Then the writter recommends 149.
In my case, with 149 client threads, the running result is:
[user#apachetomcat ~]$ ab -k -n 100000 -c 149 http://10.138.0.2:8080/test.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.138.0.2 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname: 10.138.0.2
Server Port: 8080
Document Path: /test.html
Document Length: 13 bytes
Concurrency Level: 149
Time taken for tests: 45.527 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 99106
Total transferred: 23195530 bytes
HTML transferred: 1300000 bytes
Requests per second: 2196.48 [#/sec] (mean)
Time per request: 67.836 [ms] (mean)
Time per request: 0.455 [ms] (mean, across all concurrent requests)
Transfer rate: 497.54 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 6.8 0 70
Processing: 66 67 5.6 67 870
Waiting: 66 67 5.6 67 870
Total: 66 68 8.8 67 870
Percentage of the requests served within a certain time (ms)
50% 67
66% 67
75% 67
80% 67
90% 67
95% 68
98% 69
99% 133
100% 870 (longest request)
After increasing to 1000 client threads, the result is:
[user#apachetomcat ~]$ ab -k -n 100000 -c 1000 http://10.138.0.2:8080/test.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.138.0.2 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname: 10.138.0.2
Server Port: 8080
Document Path: /test.html
Document Length: 13 bytes
Concurrency Level: 1000
Time taken for tests: 7.205 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 99468
Total transferred: 23197340 bytes
HTML transferred: 1300000 bytes
Requests per second: 13879.80 [#/sec] (mean)
Time per request: 72.047 [ms] (mean)
Time per request: 0.072 [ms] (mean, across all concurrent requests)
Transfer rate: 3144.28 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 8.1 0 68
Processing: 66 69 22.3 67 1141
Waiting: 66 69 22.3 67 1141
Total: 66 70 27.5 67 1141
Percentage of the requests served within a certain time (ms)
50% 67
66% 67
75% 68
80% 68
90% 69
95% 71
98% 87
99% 139
100% 1141 (longest request)
The Requests per second increases from 2196.48/sec to 13879.80/sec, so I think this change is meaningful.
Why does the writer think it's not helpful when we set it higher than the maxThreads?
What does the incrementation of requests per second mean in my case?
I'm confused with the requests per second. It's very important to understand the writer's benchmarks in the following chapters of the book.
I see some strange results while using apache benchmark with nginx. Please see details below -
terminal:~ directory$ ab -c 100 -n 10000 http://localhost/banner.jpg
This is ApacheBench, Version 2.3
Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: nginx/1.2.0
Server Hostname: localhost
Server Port: 80
Document Path: /banner.jpg
Document Length: 16697 bytes
Concurrency Level: 100
Time taken for tests: 1.224 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 169226562 bytes
HTML transferred: 167094858 bytes
Requests per second: 8170.40 [#/sec] (mean)
Time per request: 12.239 [ms] (mean)
Time per request: 0.122 [ms] (mean, across all concurrent requests)
Transfer rate: 135024.21 [Kbytes/sec] received
But when I increase the number of requests from 10K to 15K, then the time for test increases from 1.124 seconds to 6.760 seconds.
terminal:~ directory$ ab -c 100 -n 15000 http://localhost/banner.jpg
This is ApacheBench, Version 2.3
Benchmarking localhost (be patient)
Completed 1500 requests
Completed 3000 requests
Completed 4500 requests
Completed 6000 requests
Completed 7500 requests
Completed 9000 requests
Completed 10500 requests
Completed 12000 requests
Completed 13500 requests
Completed 15000 requests
Finished 15000 requests
Server Software: nginx/1.2.0
Server Hostname: localhost
Server Port: 80
Document Path: /banner.jpg
Document Length: 16697 bytes
Concurrency Level: 100
Time taken for tests: 6.760 seconds
Complete requests: 15000
Failed requests: 0
Write errors: 0
Total transferred: 254201718 bytes
HTML transferred: 250999689 bytes
Requests per second: 2218.92 [#/sec] (mean)
Time per request: 45.067 [ms] (mean)
Time per request: 0.451 [ms] (mean, across all concurrent requests)
Transfer rate: 36722.28 [Kbytes/sec] received
Did anyone notice this issue? Is there any tool to analyze nginx or apache benchmark tool and see what is taking some much of time?
I have a 1 GB VPS and Apache slows to a crawl almost from start up. I ran ApacheBench on a static.html file and things don't differ. However, the site will have both MySQL and PHP and a high volume of AJAX requests, so I'd like to tune for that.
When I restart, error logs show this almost immediately:
[error] server reached MaxClients setting, consider raising the MaxClients setting
ab -n 1000 -c 1000
shows:
Document Path: /static.html
Document Length: 7 bytes
Concurrency Level: 1000
Time taken for tests: 57.784 seconds
Complete requests: 1000
Failed requests: 64
(Connect: 0, Receive: 0, Length: 64, Exceptions: 0)
Write errors: 0
Total transferred: 309816 bytes
HTML transferred: 6552 bytes
Requests per second: 17.31 [#/sec] (mean)
Time per request: 57784.327 [ms] (mean)
Time per request: 57.784 [ms] (mean, across all concurrent requests)
Transfer rate: 5.24 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 25 13.4 25 48
Processing: 1070 16183 15379.4 9601 57737
Waiting: 0 14205 15176.5 9591 42516
Total: 1070 16208 15385.0 9635 57783
Percentage of the requests served within a certain time (ms)
50% 9635
66% 20591
75% 20629
80% 36357
90% 42518
95% 42538
98% 42556
99% 42560
100% 57783 (longest request)
If I run ab on a php file, it finishes sometimes, most of the time it won't and sometimes gets errors like
apr_socket_recv: Connection reset by peer (104)
and
socket: No buffer space available (105)
httpd.conf items:
Timeout 10
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 1
<IfModule prefork.c>
StartServers 3
MinSpareServers 5
MaxSpareServers 9
ServerLimit 40
MaxClients 40
MaxRequestsPerChild 5000
</IfModule>
Top... (CPU and Load 1min are very erratic during testing):
top - 10:44:51 up 11:50, 3 users, load average: 0.17, 0.42, 0.90
Tasks: 84 total, 2 running, 82 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.8%us, 3.1%sy, 0.0%ni, 94.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1793072k total, 743604k used, 1049468k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21831 mysql 18 0 506m 71m 6688 S 0.7 4.1 4:03.18 mysqld
1828 root 15 0 113m 52m 2052 S 0.0 3.0 0:02.85 spamd
1830 popuser 18 0 113m 51m 956 S 0.0 2.9 0:00.00 spamd
8012 apache 15 0 327m 35m 17m S 3.7 2.0 0:11.83 httpd
8041 apache 15 0 320m 28m 15m S 0.0 1.6 0:11.83 httpd
8022 apache 15 0 321m 27m 14m S 2.3 1.6 0:11.05 httpd
8033 apache 15 0 320m 27m 14m S 1.7 1.6 0:10.06 httpd
Is there something obvious that is wrong here? or what would be my next step in troubleshooting?
Sounds like you don't have enough memory -- 1GB isn't much when you're running PHP with prefork and MySQL on the same server. Your MaxClients should probably be 10-20, not 40.
A few weeks ago I wrote a script to tune Apache httpd that would probably help determine the maximum values for your server. You can find the weblog entry here http://surniaulula.com/2012/11/09/check-apache-httpd-mpm-config-limits/ and the script is on Google Code as well.
Enjoy!
js.