I need to modify the number of threads available in my Weblogic 10.0 MP2 environment for some perf benchmarking but I cannot seem to be able to find where exactly that option lies.
Can anyone share this info please? thank you.
Weblogic 10 does not use execute thread queues like in previous versions (i.e. Weblogic 8.1 and older)
This concept is now replaced with Work Managers.
These are self-tuned, i.e. WLS will auto-tune the number of threads every 2 seconds based on how it sees the need to increase threads for the application load.
You can confirm this from the console, it will show the increasing number of execute threads as the load increases.
You can use the work manager and constraints to make sure your applications get certain criteria met.
Such as certain web apps or EJBs can get a higher share of threads and so on.
For a quick read see http://www.oracle.com/technetwork/articles/entarch/workload-management-088692.html
and
http://m-button.blogspot.com/2009/02/tuning-default-workmanager-on-weblogic.html
Secondly, are you running in dev mode or production mode.
If dev mode, you can try this cmd line parameter
-Dweblogic.threadpool.MinPoolSize=100
but I am not sure if it will work, so it's better to leave it to Work Managers
Related
We are currently testing an upgrade from CF11 to CF2018 for my company's intranet. To give you an idea how long this site has been running, our first version of CF was 3.1! It is still using application.cfm, and there is code from 1998, when I started writing this thing. Yes, 21 years -- I'm astonished, too. It is a hodgepodge of all kinds of older frameworks, too, including Fusebox.
Anyway, we're running Win 2012 VM connected to a SQL 2016 farm. Everything looked OK initially, but in the Week I've been testing, the server has come to a slowdown once (a page took more than 5 seconds to run, something that usually takes 100ms, no DB involvement), and another time, the server came to a grinding halt. The only way I could restart CF App service was by connecting to the server with another server via Services, because doing it via Remote Desktop was so slow.
Now keep in mind -- it's just me testing. This is a site that doesn't have a ton of users, but still, having 5 concurrent connections is normal and there are upwards of 200-400 users hitting this thing every day.
I have FusionReactor running on this thing now, so the next time a lockup happens, I will be able to take a closer look, but what do you think is the best way I can test this? Our site is mostly transactional, users going and filling out forms to put internal orders through. We also connect to XML web services and REST services; we also provide REST services, too. Obviously there's no way to completely replicate a production server's requests onto a test server, but I need to do more thorough testing. Any advice would be hugely appreciated.
I realize your focus for now is trying to recreate the problem on test. That may not be as easy as hoped. Instead, you should be able to understand and resolve it in production. FusionReactor can help, but the answer may well be in the cf logs.
You don't mention assessing the logs at the time of the hangup. See especially the coldfusion-error log, for outofmemory conditions.
You mention raising the heap, but the problem may be with the metaspace instead. If so, consider simply removing the maxmetaspace setting in the jvm args. That may be the sole and likely cause of such new and unexpected outages.
Or if it's not, and there's nothing in the logs at the time, THEN do consider FR. Does IT show anything happening at the time?
If not then consider a need to tune the cf/web server connector. I assume you're using iis. How many sites do you have? And how many connectors (folders in the cf config/wsconfig folder)? What are the settings in their workers.properties file? Are they optimized for the number of sites using that connector?
Also, have you updated cf2018? Are there any errors in the update error log? Did you update the web server connector also?
Are you running the cf2018 pmt (performance monitoring tool set)? Have you updated it?
There could be still more to consider, but let's see how it goes with those. I have blog posts on these and many more topics that would elaborate on things, both at my site (carehart.org) and the Adobe cf portal (coldfusion.adobe.com).
But let's hear if any of this gets you going.
I am a java developer. My hard time i have to improve performance in production box when 150 concurrent users hits our application(app is deployed in weblogic server).
Our production system is configured with 4 JVM, 2 clustors. i found that each JVM has 12 processors(i am not sure, may be 3 per box, totally 12 processors).
Presently we deployed our JEE application in that weblogic which uses default thread pool. Weblogic version is 10.x. I know that, after version 8.1 we can't change the thread size of default threadpool(work manager). Problem we are getting is our application works perfectly when 20 concurrent users hits our server(4 JVMs). i see there are 8 threads created for each JVM and i assume there are 24 threads services the concurrent user request(totally 4 box, so 24 threads). But we need to support totally 600 request and 150 request per JVM(load balancer helps us here). But the problem is instead of handling all the request in the avg time of 300 ms it takes 8000 ms. it hurts us.
To fix this i created a work manager with 75 threads and assigned to my application and i see the performance is imported significantly. Note: there are 20 applications are deployed in the same weblogic. if i use 75 threads for my application how it will affect other application?
advice me on this to handle 150 request without any lag.
The limit of the number of threads is based on you host machine and not your JVM. You can check the limit your host can handle using this cat /proc/sys/kernel/threads-max from this article. Maximum number of threads per process in Linux?
Now once you know your limit, you can use this to fine tune your server to use a specified number of threads.
http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/WLSTuning.html#1140013
Work managers do not provide you a max number of threads. They cap the max that can be. This is for applications having a long thread life and consuming more resources when they start having stuck threads. In theory, you have infinite amount of threads. My suggestion is to go with increasing the value as mentioned in the config.xml and find your sweet spot. If you are not able to achieve it with 4 JVMs, it would be much wiser to request for 2 more JVMs. Some things can just not be done by software alone.
We have a few applications which are running in Windows 2K, 2008 servers. They are written in java.
These applications needs to do many automation tasks. We are having difficulty to monitor these applications. Sometime due to XYZ reasons application either hangs or fail to perform desired job. We only come to know about this after a few days when some one reports that desired function hasn't been executed.
To come out of this issue, we added emails for each imp exceptions but then developer needs to spend time to check those 1000 emails everyday. Which is again not feasible & efficient solution.
Now we are looking for a alert, alarms, notification display & monitoring system. We need to have a remote application which can receive alarms from these java applications & then based on certain information/Condition/Configuration, remote application can display some red, orange, green text on the screen. Based on red text, users can be visually see that there is an issue in the system. If required users can be notified that there is a serious issue in the application.
Please help us to identify any existing mechanism, tool, package to achieve this goal. Any suggestion would be highly appreciated.
Thanks
There are myriad ways to achieve this, but all of them will require some effort. Which way to proceed depends on your needs and abilities. A couple of options occur to me:
Have your processes log their exceptions to a Syslog daemon, running on some central server. Then you could have an admin read through the log file for serious problems, but there are many ways to post-process syslog messages, a web search on it might give some more hints.
Is there any way, when logged into the server, to observe whether or not the process is running properly or not? You could install something like Nagios on a sever, and write a plugin that monitors your particular process on all the servers. The plugin can basically be a shell script that checks "ps", or a log file, or whatever you want.
If you are in an IT department, your organization might already have some system like this (NMS).
I'm not sure why this question is tagged "snmp", but it's technically possible to install an SNMP agent on each server, and have them send traps on certain conditions. I do think it would be slightly overkill because you would also have to get a good SNMP manager to receive the traps and alert a sysadmin.
I would go for a combination of the check_logfiles plugin to parse log exceptions and raise alerts, and check_jmx/jmxquery to check metrics inside the JVM such as heap usage and thread count.
check_logfiles
check_jmx
I am new to weblogic server. I am using work manager. I want to know what is work manager and why we need it. What is the difference between normal request with out work manager and with work manager !!
I think the documentation is rather good on this subject.
WebLogic Server prioritizes work and allocates threads based on an
execution model that takes into
account administrator-defined
parameters and actual run-time
performance and throughput.
Administrators can configure a set of
scheduling guidelines and associate
them with one or more applications, or
with particular application
components. For example, you can
associate one set of scheduling
guidelines for one application, and
another set of guidelines for other
application. At run-time, WebLogic
Server uses these guidelines to assign
pending work and enqueued requests to
execution threads.
Essentially, with work managers you can attach a scheduling policy to an application to e.g. make sure that a specific application gets a fair share of the available computing resources under a heavy load situation. Or you might want to restict the maximum number of threads that will be allocated to an application to prevent a buggy/untested application to bring the whole application server to its knees. (But surely all apps have been tested not to do anything like that.... ;) )
Outside of modifying the default allocation algorithms, the Work Manager is also useful if you are using a Foreign JMS Provider (such as IBM MQ) and need to process more than 16 messages at a time.
I am just wondering why is restarting IIS 7 application pool consider a good practice? I know that it'll clean up orphan resources like threads or session state, but how does other web server deal with this kind of problem? Do java developer/admin have to restart weblogic/websphere/apache/tomcat every so often to clean up junk in memory?
Typically it's used to compensate for bad programming. Sometimes yours, sometimes third parties depending on the libraries you are using.
And, as with all recommendations, it's best to look at your specific environment to see if it's really necessary before doing it.
Start with profiling the app. A simple way is to just let it run for a while under load. Look for things like does memory utilization settle down or does it continually grow?
If it keeps growing, you will need to set IIS to recycle every so often and go fix your app. If it grows to a point then settles down then you should be just fine. Make sure you test even the edge cases of your app.
With IIS it is possible to have non-managed code - the best example is a COM object - that don't behave well. As a result you can set your IIS to recycle processes periodically, to allow for the memory leakage that might occur with such code.
It's not necessary to do it though. You can eliminate the restart thresholds, if you prefer not to restart the processes.
If this idea doesn't exist in other servers, then it may be because other servers allow only managed code.