If I have the following action in a controller
def give_a
print a
a = a+1
end
What happens in each webserver when a request comes and when multiple requests are recieved?
I know that webrick and thin and single threaded so I guess that means that the a request doesn't get processed until the current request is done.
What happens in concurrent webservers such as puma or unicorn (perhaps others)
If there are 2 requests coming and 2 unicorn threads handle them, would both responses give the same a value? (in a situation when both request enter the method in the same time)
or does it all depend on what happens on the server itself and the access to data is serial?
Is there a way to have a mutex/semaphore for the concurrent webservers?
afaik, the rails application makes a YourController.new with each request env.
from what you post, it is not possible to see, what a means. when it is some shared class variable, then it is mutuable state and could be modified from both request threads.
Related
I have a Perl web app served by Apache httpd using plain mod_cgi or optionally mod_perl with PerlHandler ModPerl::Registry. Recently the app encountered the error Script timed out before returning headers on some invocations and behaved differently afterwards: While some requests seemed to be processed successfully in the background, after httpd sent status 504 to the client, others didn't.
So how exactly behaves httpd AFTER it reached its configured timeout and sent the error to the client? The request/response cycle is finished now, so I guess things like KeepAlive come into play to decide if the TCP connections stays alive or not etc. But what happens to the running script in which environment, e.g. mod_cgi vs. mod_perl?
Especially in mod_cgi, where new processes are started for each request, I would have guessed that httpd keeps the processes simply running. Because all our Perl files have a shebang, I'm not even sure if httpd is able to track the processes and does so or not. That could be completely different with mod_perl, because in that case httpd is aware of the interpreters and what they are doing etc. In fact, the same operation which timed out using plain mod_cgi, succeeded using mod_perl without any timeout, but even with a timeout in mod_cgi at least one request succeeded afterwards as well.
I find this question interesting for other runtimes than Perl as well, because they share the concepts of plain mod_cgi vs. some persistent runtime embedded into the httpd processes or using some external daemons.
So, my question is NOT about how to get the error message away. Instead I want to understand how httpd behaves AFTER the error occurred, because I don't seem to find much information on that topic. It's all just about increasing configuration values and try to avoid the problem in the first place, which is fine, but not what I need to know currently.
Thanks!
Both mod_cgi and mod_cgid set a cleanup function on the request scope to kill the child process, but they do it slightly different ways. This would happen shortly after the timeout is reported (a little time for mod_cgi to return control, the error response to be written, the request logged, etc)
mod_cgi uses a core facility in httpd that does SIGTERM, sleeps for 3 seconds, then does a SIGKILL.
I have a Ruby on Rails application with following versions:
Ruby: 1.9.3-p547
Rails: 3.2.16
I am facing some performance issues in this application. My initial efforts on diagnosing this landed me up on using the RackTimer gem (https://github.com/joelhegg/rack_timer) mentioned in article at http://www.codinginthecrease.com/news_article/show/137153 to record the Middleware timestamps.
Using it I found that Rack::Lock consumes a lot of time.For e.g. following
are some of the RackTimer logs from the logs provided below:
Rack Timer (incoming) -- Rack::Lock: 73.77910614013672 ms
Rack Timer (incoming) -- Rack::Lock: 67.05522537231445 ms
Rack Timer (incoming) -- Rack::Lock: 87.3713493347168 ms
Rack Timer (incoming) -- Rack::Lock: 59.815168380737305 ms
Rack Timer (incoming) -- Rack::Lock: 55.583953857421875 ms
Rack Timer (incoming) -- Rack::Lock: 111.56821250915527 ms
Rack Timer (incoming) -- Rack::Lock: 119.28486824035645 ms
Rack Timer (incoming) -- Rack::Lock: 69.2741870880127 ms
Rack Timer (incoming) -- Rack::Lock: 75.4690170288086 ms
Rack Timer (incoming) -- Rack::Lock: 86.68923377990723 ms
Rack Timer (incoming) -- Rack::Lock: 113.18349838256836 ms
Rack Timer (incoming) -- Rack::Lock: 116.78934097290039 ms
Rack Timer (incoming) -- Rack::Lock: 118.49355697631836 ms
Rack Timer (incoming) -- Rack::Lock: 132.1699619293213 ms
As can be seen Rack::Lock middleware processing time fluctuates between 10 ms to more than 130 seconds. And the majority of these comes into picture while serving assets on my Home Page.
BTW I have Asset Pipeline enabled in my config/application.rb
# Enable the asset pipeline
config.assets.enabled = true
I have my Production-version application configured to be monitored by NewRelic.There too the graph-charts highlights highest % and time taken by Rack::Lock.
I am totally blank on what is contributing towards making Rack::Lock take so many milliseconds.Would appreciate if anybody from the community can provide their valuable guidance in figuring out what might be causing this and how to fix it?
Below can be found the Gemfile, what all Middlewares are involved and Dev environment Logs.
Gemfile:
https://gist.github.com/JigneshGohel-BoTreeConsulting/1b10977de58d09452e19
Middlewares Involved:
https://gist.github.com/JigneshGohel-BoTreeConsulting/91c004686de21bd6ebc1
Development Environment Logs:
----- FIRST TIME LOADED THE HOME INDEX PAGE
https://gist.github.com/JigneshGohel-BoTreeConsulting/990fab655f156a920131
----- SECOND TIME LOADED THE HOME INDEX PAGE WITHOUT RESTARTING THE SERVER
https://gist.github.com/JigneshGohel-BoTreeConsulting/f5233302c955e3b31e2f
Thanks,
Jignesh
I am posting my findings here after I posted my question above. Hoping that somebody else benefits from these findings when ends up in a situation like above.
Discussing about the Rack::Lock issue with one of my senior associates Juha Litola, below were his first thoughts (quoting his own words as it is):
Could it just be possible that you are seeing measuring artifact in sense that you are just seeing Rack::Lock as taking a lot of time, but that is just because it is wrapping the actual call? So the Rack::Lock time is cumulative time from everything that happens in the request processing. See
https://github.com/rack/rack/blob/master/lib/rack/lock.rb .
As for the performance issues, could you elaborate on what kind of problems you have so I could help?
To which I thought it could be a possibility. However I could not convince myself with this possibility because of the following doubt:
Rack::Lock is at the second position in the Middlewares chain with a Rails application (please refer the Middleware List I mentioned in post above at https://gist.github.com/JigneshGohel-BoTreeConsulting/91c004686de21bd6ebc1 ). And each middleware is processed in sequential order in the chain. Thus Rack::Lock would be the second one to process the request
and then others in the chain would get a chance to jump in.
In such a case as per my understanding I don't think the timestamps recorded for Rack::Lock middleware is the cumulative time from everything that happens in the request processing.And it should be the time taken by Rack::Lock middleware itself.
Later Juha, after spending a few minutes looking at the server(see note below) config provided following inputs:
With a quick look I think that there is a quite clear problem in how the application has been configured.
Application doesn’t have config.threadsafe! enabled, which means that Rack::Lock is enabled, and request processing is limited to one thread / process. Now puma is configured only to have one process, but 16-32 threads. What this means in effect that puma is currently processing only one request at given moment.
Best solution would of course be if you could enable thread safe mode, but that will require thorough testing.
If that fails or is not an option puma should be configured with multiple workers with 1 thread each.
Note: I forgot to add any details about the configuration of web server on which my application is deployed. We are using Puma Web Server (https://github.com/puma/puma)
With that I got a hint to dig more into the config.threadsafe!. Doing a search on web I landed up on following articles
http://www.sitepoint.com/config-threadsafe/
http://tenderlovemaking.com/2012/06/18/removing-config-threadsafe.html
shedding great insights on how enabling or disabling the option config.threadsafe! impacts the performance of the application deployed on a multi-threaded or multi-process webserver on production.
A Brief Summary Of What The Above Articles Conveyed
What is Rack::Lock?
Rack::Lock is a middleware that is inserted to the Rails middleware stack in order to protect our applications from the multi-threaded Bogeyman. This middleware is supposed to protect us from nasty race conditions and deadlocks by wrapping our requests with a mutex. The middleware locks a mutex at the beginning of the request, and unlocks the mutex when the request finishes.
Let's assume that there is a program running and sending 5 requests simultaneously 100 times to an application whose code (say Controller) is NOT thread-safe.
Now lets observe the impact of combination of Rack::Lock middleware, config.threadsafe! option enabled or disabled, Thread-unsafe code in the application, and Multi-Threaded or Multi-Process Web Server, after the program gets finished or is killed
Multi-Threaded Web Server (Puma)
# Combination 1:
config.threadsafe! option : Disabled
Rack::Lock middleware : Available in app's middleware stack because of config.threadsafe! option disabled
With this combination the web server is successfully able to entertain all the 500 requests received.
This is because each request is augmented by Rack::Lock so as to execute it synchronously.In other words
Rack::Lock ensures we have only one concurrent request at a time.Thus each of the 500 requests gets a chance to
execute.
# Combination 2:
config.threadsafe! option : Enabled
Rack::Lock middleware : Unavailable in app's middleware stack because of config.threadsafe! option enabled
With this combination the web server is able to entertain only 200 out of 500 requests received.
This is because of the absence of Rack::Lock middleware, which ensures that we have only one concurrent request
at a time and thus each request gets a chance.
However there are advantages as well as disadvantages of each combination mentioned above:
# Combination 1
Advantage:
Each of the request received gets chance to be processed
Disadvantage:
* The runtime to process all of the 500 requests took 1 min 46 secs (compare it to runtime of Combination 2)
* Using a multi-threaded webserver is useless, if Rack::Lock remains available in middleware stack.
# Combination 2
Advantage:
The runtime to process 200 requests took 24 secs (compare it to runtime of Combination 1).
The reason being the multi-threaded nature of webserver is being leveraged in this case to entertain concurrent requests coming in.
Disadvantage:
* Not all 500 requests got a chance to be processed
Note: Examples and Runtime statistics have been quoted from http://tenderlovemaking.com/2012/06/18/removing-config-threadsafe.html
Multi-Process Web Server (Unicorn)
# Combination 1:
config.threadsafe! option : Disabled
Rack::Lock middleware : Available in app's middleware stack because of config.threadsafe! option disabled
Since multiple processes are forked by the webserver and each of them listens for requests and also
Rack::Lock middleware is available, the web server is successfully able to entertain all the 500 requests received.
# Combination 2:
config.threadsafe! option : Enabled
Rack::Lock middleware : Unavailable in app's middleware stack because of config.threadsafe! option enabled
Here too multiple processes are forked by the webserver and each of them listens for requests,
however Rack::Lock middleware is unavailable which enables multi-threading, which in turn means that we'll
get a race condition in the thread-unsafe code we have in the application.But strangely with this combination
too the web server is successfully able to entertain all the 500 requests received.
The reason being a process-based web server creates worker processes and each process holds one instance of
our application. When a request is received webserver spawns a child process for handling it.
Conclusion:
In a multi-process environment Rack::Lock becomes redundant if we keep config.threadsafe! option disabled.
This is because in multi-process environment the socket is our lock and we don't need any additional lock.
Thus it is beneficial to enable config.threadsafe! and remove Rack::Lock overhead in production environment.
In a multi-threaded environment if we keep config.threadsafe! enabled the developers needs to ensure the application's code is thread-safe.
And the advantage of keeping config.threadsafe! is that lesser runtime is needed to process the incoming requests.
In my application's context we tweaked the Puma server's config by increasing the workers. I hope the performance improves.
We have developed an application where after a successful request, session has to be destroyed. This is working fine, when we have a single tomcat.
But, this is not happening when we use multiple tomcats under Apache simple load balancer (We are using Load Balancer, for balancing the requests between two tomcats, which are hosting the same application).
The SessionID that is created and have processed successfully, can be used for one more transaction, after which it is getting killed.
Moreover, the SessionID value is being appended with either 'n1' or 'n2' (SessionID-n1). I am not sure of why is this happening.
Please help me to resolve this issue.
We have a configuration setup as below:
Load Balancer
/ \
Cluster1 Cluster2
| |
Tomcat1 Tomcat2
Thanks,
Sandeep
If you have configured each Tomcat node to have a "jvmRoute", then the string you specify there will be appended to the session identifier. This can help your load-balancer determine which back-end server should be used to serve a particular request. It sounds like this is exactly what you have done. Check your CATALINA_BASE/conf/server.xml file for the word "jvmRoute" to confirm.
If you only use a session for a single transaction, why are you bothering to create the session in the first place? Is a request == transaction?
If you are sure you are terminating the session when the transaction is complete, you should be okay even if the client wants to try to make a new request with the same session id. It will no longer be valid and therefore useless to the client.
It's not clear from your question whether there is an actual problem with the session because you claim it is "getting killed" which sounds like what you want it to do. If you provide more detail on the session expiration thing, I'll modify my answer accordingly.
I'm developing a Rails 3.2.16 app and deploying to a Heroku dev account with one free web dyno and no worker dynos. I'm trying to determine if a (paid) worker dyno is really needed.
The app sends various emails. I use delayed_job_active_record to queue those and send them out.
I also need to check a notification count every minute. For that I'm using rufus-scheduler.
rufus-scheduler seems able to run a background task/thread within a Heroku web dyno.
On the other hand, everything I can find on delayed_job indicates that it requires a separate worker process. Why? If rufus-scheduler can run a daemon within a web dyno, why can't delayed_job do the same?
I've tested the following for running my every-minute task and working off delayed_jobs, and it seems to work within the single Heroku web dyno:
config/initializers/rufus-scheduler.rb
require 'rufus-scheduler'
require 'delayed/command'
s = Rufus::Scheduler.singleton
s.every '1m', :overlap => false do # Every minute
Rails.logger.info ">> #{Time.now}: rufus-scheduler task started"
# Check for pending notifications and queue to delayed_job
User.send_pending_notifications
# work off delayed_jobs without a separate worker process
Delayed::Worker.new.work_off
end
This seems so obvious that I'm wondering if I'm missing something? Is this an acceptable way to handle the delayed_job queue without the added complexity and expense of a separate worker process?
Update
As #jmettraux points out, Heroku will idle an inactive web dyno after an hour. I haven't set it up yet, but let's assume I'm using one of the various keep-alive methods to keep it from sleeping: Easy way to prevent Heroku idling?.
According to this
https://blog.heroku.com/archives/2013/6/20/app_sleeping_on_heroku
your dyno will go to sleep if he hasn't serviced requests for an hour. No dyno, no scheduling.
This could help as well: https://devcenter.heroku.com/articles/clock-processes-ruby
When I run my Rails application in Apache using Passenger and open two browser log the request with thread id using log4r.
I see both the request uses same thread id. How is it possible?
If I do sleep in one request until sleep expire another request is blocked.
Where can I configure use different thread for each request or maxThreadCount?
Is it the behavior for development environment or in production too? how to overcome with this?
config.threadsafe!
put it in your production.rb or development.rb.
I have same problem when calling a local webservice inside a controller action.
Puma also has better concurrency, but that threadsafe confgi make webrick multi-thread for me.