No matter what changes we've made TTFB is high!
Surprisingly, server side guy insists that everything is set correct and the server runs fast enough but webpagetest report didn't change at all!
After so much optimization, I can't believe it doesn't change and I started to suspect TLS, gZIP and redirections...
Did I miss something?
It appears that the page itself takes about a second to generate. Here's a curl output (using this to get timestamps), and notice the value of time_appconnect
$ curl -w "#curl_format.txt" -so /dev/null https://jimmydance.com/
time_namelookup: 0.004
time_connect: 0.217
time_appconnect: 0.921
time_pretransfer: 0.921
time_redirect: 0.000
time_starttransfer: 1.348
----------
time_total: 1.352
This would suggest that the bottleneck is the application generating the page. Looking at the page itself, it shouldn't take so long, I would look at the framework you're using or the resources allocated to the server.
Using #frederik-deweerdt's answer and a breakdown of curl timing figures
Your server is spending nearly 1 second doing its TLS handshake (time_appconnect: 0.921)
TTFB is time_starttransfer - time_appconnect (1.348 - 0.921 = 0.427)
the time the server spent generating the html is TTFB - (time_connect - time_namelookup) (0.427 - (0.217 - 0.004)) = 0.214
so I'd say the bottleneck isn't the app, instead it's the server's slow TLS negotiation and some latency in the network.
Have a look here for a good explanation of these curl timing figures: https://blog.cloudflare.com/a-question-of-timing/
The diagram below shows what each of those timings refer to against a typical HTTP over TLS 1.2 connection (TLS 1.3 setup needs one less round trip):
time_namelookup in this example takes a long time. To exclude DNS resolver performance from the figures, you can resolve the IP for cURL: --resolve www.zasag.mn:443:218.100.84.167. It may also be worth looking for a faster resolver :).
time_connect is the TCP three-way handshake from the client’s perspective. It ends just after the client sends the ACK - it doesn't include the time taken for that ACK to reach the server. It should be close to the round-trip time (RTT) to the server. In this example, RTT looks to be about 200 ms.
time_appconnect here is TLS setup. The client is then ready to send it’s HTTP GET request.
time_starttransfer is just before cURL reads the first byte from the network (it hasn't actually read it yet). time_starttransfer - time_appconnect is practically the same as Time To First Byte (TTFB) from this client - 250 ms in this example case. This includes the round trip over the network, so you might get a better guess of how long the server spent on the request by calculating TTFB - (time_connect - time_namelookup), so in this case, the server spent only a few milliseconds responding, the rest of the time was the network.
time_total is just after the client has sent the FIN connection tear down.
Related
I tried posting this on ask.openstack but it has been stuck in the moderator for 5 days now. I thought I'd try here.
I was trying to debug a Nova issue and wanted to decode the SSL / TLS packets being exchanged using Wireshark. Part of the changes I was making was setting Nova up to use SSL / TLS and I wanted to be sure that part of it I had set correctly. I eventually figure out my issues from the various log files but I'm somewhat assuming that being able to watch the network traffic may help in some very difficult cases.
The exchange uses TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 at one point. According to this security stackexchannge question, there is a "pre-master secret" or various other terms. I've wrestled with this before in a previous life doing IPSec. Usually you can set debug in the application and it will spew out the secret into the log file. I tried "debug = true" under Default in nova.conf and got lots of debug but no secret. There was two items that looked interesting that were reported as **** in the log: keystone_authtoken.memcache_secret_key and neutron.metadata_proxy_shared_secret. I wasn't sure if those were the secrets I was looking for or not. In this case, I'm looking at the nova-api traffic going to port 8774.
Also, since all of openstack is Python and uses the same "request" and "certifi" packages, it may be possible to generalize this to all of the openstack components.
nova --version report 9.1.1
I have a problem with our subversion server. Doing small commits works fine, but as soon as someone tried to commit a large collection sizeable files the commit stalls halfway through and the client finally time out. My test set consists of roughly 2000 files and the total size of the commit is about 1 GB. When I commit the files the file uploading starts but about halfway through the transfer rate drops to 0kb/s and the commit just stalls and never recovers. If I splitting the commit into smaller pieces (<150 Mb) everything works just fine, but that breaks the atomicity of the commit structure and is something I really want to avoid.
When I look at the logs generate by Apache there is no error messages.
When I bumped the loglevel from debug to trace6 on the Apache server, there is some errors appearing at the moment when the upload stalls:
...
OpenSSL: I/O error, 2229 bytes expected to read on BIO
OpenSSL: read 1460/2229 bytes from BIO
...
Versions used:
We are running the connection to the subversion via apache, mod_dav, mod_dav_svn, mod_authz_svn and mod_auth_digest. The client connects via https.
Server:
OpenSuse 42.3
svnserve: 1.9.7
Apache: 2.4.23
Client:
Windows 10 enterprise
svn client: 1.10.0-dev.
What I tried so far:
I have tried increasing the TimeOut value in the apache configuration. The only difference is that the client ends up in stalled mode longer before posting the timeout message.
I have tried increasing the MaxKeepAliveRequests from 100 to 1000. No change.
I have tried adding SVNAllowBulkUpdates Prefer to the svn settings. No change.
Have anyone got any hints on how to debug these types of errors?
I have an Apache server with 16GB of Ram. The script cap.php returns a very small chunk of data (500B). It starts a mysql connection and makes a simple query.
However, the response from the server is, in my opinion, too lengthy.
I attach a screenshot of the Developer Tool Panel in Chrome.
Beside SSL and the TTFB there is a strange delay of 300ms (Stalled).
If I try a curl from the WebServer:
curl -w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' -k -H 'miyazaki' https://127.0.0.1/ui/cap.php
Lookup time: 0.000
Connect time: 0.000
PreXfer time: 0.182
StartXfer time: 0.266
Total time: 0.266
Does anyone know what that is?
Eventually, I found that if you use SSL it is really better and it does really matter to switch on the KeepAlive directive into Apache. See the picture below.
According to the Chrome documentation:
Stalled/Blocking
Time the request spent waiting before it could be sent. This time is inclusive of any time spent in proxy negotiation.
Additionally, this time will include when the browser is waiting for
an already established connection to become available for re-use,
obeying Chrome's maximum six TCP connection per origin rule.
So this appears to be a client issue with Chrome talking to the network rather than a server config issue. As you are only making one request I think we can rule out the TCP limit per origin (unless you have lots of other tabs using up these connections) so would guess either limitations on your PC (network card, RAM, CPU) or infrastructure issues (e.g. You connect via a proxy and it takes time to set up that connection).
Your curl request doesn't seem to show this delay as it has just a 0.182 wait time to send the request (which is easily explained with https negotiation) and then a 0.266 total time to download (including the 0.182). This compares with 0.700 seconds when using Chrome so don't understand why you say "total time is similar" when to me it's clearly not?
Finally I do not understand your follow up answer. It looks to me like you have made request, presumably after a recent other request as this has skipped the whole network connection stage (including any grey stalling, blue DNS lookup, orange initial connection and purple https connection). So of course this quicker. But it's not comparing like for like with your first screenshot in your question and is not addressing your question.
But yes you absolutely should be using keep-alives (they are on by default in most web server so usually takes extra efforts to turn them off) and https resumption techniques (not on by default unless you explicitly add this to your https config) to benefit any additional requests sent shortly after the first. But these will not benefit the first connection of the session.
I tested my web site with 100 users with http and https. The response time obtained in https is much higher compared to the response time obtained in http. The response time of https is nearly four times greater than http. Can anyone explain me why the response time is higher in https compared to http? or do i need to change any SSL property in jmeter system.properties? Thanks in Advance..!
SSL Handshake assumes 4 requests for establishing a connection so first request should be something like 4x times longer than in case of HTTP. See The SSL handshake diagram for more info
However if you receive 4 times performance degradation for all requests - that doesn't sound right.
There are following JMeter properties which control SSL flows:
https.sessioncontext.shared - controls whether SSL session contexts are created per thread (if it's set to false) or shared (if it's set to true)
https.use.cached.ssl.context - controls if cached SSL context is being reused between iterations
These properties live in jmeter.properties file under /bin folder of your JMeter installation. It's also possible to override them using -J command line key as follows:
jmeter -Jhttps.sessioncontext.shared=true -Jhttps.use.cached.ssl.context=true
See Apache JMeter Properties Customization Guide for more details.
If above setting won't help you'll need to review your test plan and perhaps profile application to see where this extra time is spent.
How do I find the HTTP timeout set on the WebLogic 8.1 application server?
I only have Weblogic 9 and 10 available but on those platforms, you can go to the console, click on the name of your domain, then (in "Configuration" tab) "Web Applications". There you will have 3 parameters:
Post Timeout: The amount of time this server waits between receiving chunks of data in an HTTP POST data before it times out. (This is used to prevent denial-of-service attacks that attempt to overload the server with POST data.)
Maximum Post Time: Max Post Time (in seconds) for reading HTTP POST data in a servlet request. MaxPostTime < 0 means unlimited
Maximum Post Size: The maximum post size this server allows for reading HTTP POST data in a servlet request. A value less than 0 indicates an unlimited size.
However, there might be other parameters involved depending on what your problem exactly is.