Latency Between Apache And Tomcat - apache

Question

We have a setup where we have a Elastic LB that spreads load over two Apache servers A1 and A2. These apache servers render a few php pages and primarily redirects the API requests to tomcat appplication servers T1 and T2 as per the following diagram:
Incoming Request
|
|
|
\/
LB
/\
/ \
/ \
A1 A2
|\ /|
| \ / |
| \ / |
| / \ |
T1 T2
We have recently started to note delays between apache and tomcat. Here is an example log lines from apache mod_slow log and tomcat access log for the same request:
APACHE_MOD_SLOW: VNSdtwoAAJkAACnXb-cAAACJ [06/Feb/2015:16:25:51 +0530] elapsed: 50.58 cpu: 0.00(usr)/0.00(sys) pid: 10711 ip: 10.0.0.153 host: www.example.com:443 reqinfo: GET /data/v1/url?url=test-508324 HTTP/1.1
TOMCAT: [06/Feb/2015:16:26:42 +0530] "GET /data/v1/url?url=test-508324 HTTP/1.1" 200 65 10
Apache says the incoming request came at 06/Feb/2015:16:25:51 +0530 and it took 50s to process the request. Whereas tomcat says it took only 10ms to process the request whereas it received the request at 06/Feb/2015:16:26:42 +0530.
It means it took nearly 50s for apache to connect and send the whole request to tomcat. Apache is using mod_proxy_ajp to connect to apache. Here is the configuration:
<Proxy balancer://prod>
BalancerMember ajp://127.0.0.1:8009 route=jvmRoute-8009 connectiontimeout=1 retry=300
BalancerMember ajp://10.0.0.153:8009 route=jvmRoute-8009 connectiontimeout=1 retry=300
ProxySet lbmethod=byrequests
</Proxy>
Here is the connector configuration from tomcat:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="4096" minSpareThreads="25" maxSpareThreads="75"/>
As per connectiontimeout value, I am assuming apache shouldn't take more than 1 sec to establish connection. Since both apache and tomcat are both on the same machine, there shouldn't be much time lag once connection is established.
If it helps, we are using https requests. But that, I don'y think has anything to do with this. We have done ab test in order to compare performance using https, http and connecting tomcat directly. Here are the stats:
ab -n5000 -c5 https://example.com/test/100001
Requests per second: 13.67 [#/sec] (mean)
Time per request: 365.851 [ms] (mean)
Time per request: 73.170 [ms] (mean, across all concurrent requests)
Transfer rate: 79.96 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 236 267 95.5 247 3401
Processing: 83 98 58.6 89 1959
Waiting: 82 96 57.5 87 1959
Total: 319 365 134.0 338 3571
Percentage of the requests served within a certain time (ms)
50% 338
66% 347
75% 356
80% 364
90% 399
95% 477
98% 689
99% 869
100% 3571 (longest request)
ab -n5000 -c5 http://example.com/test/100001
Time per request: 186.015 [ms] (mean)
Time per request: 37.203 [ms] (mean, across all concurrent requests)
Transfer rate: 155.55 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 74 79 33.4 76 1278
Processing: 83 107 82.3 91 3964
Waiting: 82 105 60.9 89 940
Total: 157 186 90.1 168 4042
Percentage of the requests served within a certain time (ms)
50% 168
66% 174
75% 180
80% 184
90% 211
95% 259
98% 379
99% 507
100% 4042 (longest request)
ab -n5000 -c5 http://IP:8080/test/100001
Requests per second: 31.32 [#/sec] (mean)
Time per request: 159.624 [ms] (mean)
Time per request: 31.925 [ms] (mean, across all concurrent requests)
Transfer rate: 181.30 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 71 76 68.4 73 3079
Processing: 78 84 13.1 81 594
Waiting: 77 83 6.5 81 185
Total: 149 159 71.2 154 3313
Percentage of the requests served within a certain time (ms)
50% 154
66% 157
75% 160
80% 161
90% 166
95% 171
98% 177
99% 189
100% 3313 (longest request)
Following observation leads me to believe this that bad performance is depending on sequence of events because none of the requests in the test performed that badly.
Versions:
Apache: 2.2.4
Tomcat: 8.0.16
Any ideas where the lag is coming from and how to reduce the same?