How to monitor GlassFish thread pool via asadmin interface - glassfish

I'm trying to use the asadmin interface to monitor a thread-pool on GlassFish 3.1.1. I'm executing the following command:
asadmin get -m server.network.my-listener.thread-pool.*
and I'm getting data back, but most of it has lastsampletime = -1 (so the related data is zero; and is worthless).
Note: I've also tried the REST interface, which I believe asadmin delegates to, and the JMX interface. Same problem: much of the data has lastsampletime = -1.
I've already turned monitoring to HIGH for all modules. What am I missing?

It seems like redeploying my application was necessary for the monitoring to actually get values. Perhaps I interpreted the manual incorrectly but it seems to suggest that a restart/redeploy wouldn't be required:
Oracle GlassFish Server 3.1 Administration Guide
Also, it is weird that the following shows there is no monitoring data:
asadmin get -m server.thread-pools.thread-pool.http-thread-pool.*
Instead you must go through a specific network listener like:
asadmin get -m server.network.http-listener-2.thread-pool.*
It also took me by surprise that enabling thread-pool monitoring IS NOT enough to see thread pool statistics. You must also enable http-service monitoring:
asadmin enable-monitoring
asadmin set server.monitoring-service.module-monitoring-levels.thread-pool=HIGH
asadmin set server.monitoring-service.module-monitoring-levels.http-service=HIGH

That's all you should need to do.
Enable monitoring, set to HIGH, for the http-service module on the DAS, stand-alone instance, or cluster you want to monitor.
Deploy an app to the DAS, stand-alone instance, or cluster and make http-requests.
asadmin get -m *instancename*.network.*listener*.thread-pool.*
Looks like you are monitoring DAS, since you are using asadmin get -m server.network.my-listener.thread-pool.*.
I deployed a simple war to DAS and made a bunch of http requests. I see the corethreads-count and maxthreads-count have last sample time as -1. And the remaining statistics have actual last sample times.
asadmin get -m "server.network.http-listener-1.thread-pool.*"
server.network.http-listener-1.thread-pool.corethreads-count = 0
server.network.http-listener-1.thread-pool.corethreads-description = Core number of threads in the thread pool
server.network.http-listener-1.thread-pool.corethreads-lastsampletime = -1
server.network.http-listener-1.thread-pool.corethreads-name = CoreThreads
server.network.http-listener-1.thread-pool.corethreads-starttime = 1320764890444
server.network.http-listener-1.thread-pool.corethreads-unit = count
server.network.http-listener-1.thread-pool.currentthreadcount-count = 5
server.network.http-listener-1.thread-pool.currentthreadcount-description = Provides the number of request processing threads currently in the listener thread pool
server.network.http-listener-1.thread-pool.currentthreadcount-lastsampletime = 1320765351708
server.network.http-listener-1.thread-pool.currentthreadcount-name = CurrentThreadCount
server.network.http-listener-1.thread-pool.currentthreadcount-starttime = 1320764890445
server.network.http-listener-1.thread-pool.currentthreadcount-unit = count
server.network.http-listener-1.thread-pool.currentthreadsbusy-count = 0
server.network.http-listener-1.thread-pool.currentthreadsbusy-description = Provides the number of request processing threads currently in use in the listener thread pool serving requests
server.network.http-listener-1.thread-pool.currentthreadsbusy-lastsampletime = 1320765772814
server.network.http-listener-1.thread-pool.currentthreadsbusy-name = CurrentThreadsBusy
server.network.http-listener-1.thread-pool.currentthreadsbusy-starttime = 1320764890445
server.network.http-listener-1.thread-pool.currentthreadsbusy-unit = count
server.network.http-listener-1.thread-pool.dotted-name = server.network.http-listener-1.thread-pool
server.network.http-listener-1.thread-pool.maxthreads-count = 0
server.network.http-listener-1.thread-pool.maxthreads-description = Maximum number of threads allowed in the thread pool
server.network.http-listener-1.thread-pool.maxthreads-lastsampletime = -1
server.network.http-listener-1.thread-pool.maxthreads-name = MaxThreads
server.network.http-listener-1.thread-pool.maxthreads-starttime = 1320764890443
server.network.http-listener-1.thread-pool.maxthreads-unit = count
server.network.http-listener-1.thread-pool.totalexecutedtasks-count = 31
server.network.http-listener-1.thread-pool.totalexecutedtasks-description = Provides the total number of tasks, which were executed by the thread pool
server.network.http-listener-1.thread-pool.totalexecutedtasks-lastsampletime = 1320765772814
server.network.http-listener-1.thread-pool.totalexecutedtasks-name = TotalExecutedTasksCount
server.network.http-listener-1.thread-pool.totalexecutedtasks-starttime = 1320764890444
server.network.http-listener-1.thread-pool.totalexecutedtasks-unit = count
Command get executed successfully.

To instantly enable monitoring without restart use enable-monitoring command
enable-monitoring
enable-monitoring --modules jvm=LOW
enable-monitoring --modules thread-pool=HIGH
enable-monitoring --modules http-service=HIGH
enable-monitoring --modules jdbc-connection-pool=HIGH
The trick is that thread-pool and http-service modules must have high level to get monitoring info.
For more info refer https://docs.oracle.com/cd/E26576_01/doc.312/e24928/monitoring.htm#GSADG00558

Related

Task queues and result queues with Celery and Rabbitmq

I have implemented Celery with RabbitMQ as Broker. I rely on Celery v4.4.7 since I have read that v5.0+ doesn't support RabbitMQ anymore. RabbitMQ is a MUST in my case.
Everything has been containerized then deployed as pods within Kubernetes 1.19. I am able to execute long running tasks and everything apparently looks fine at first glance. However, I have few concerns which require your expertise.
I have declared inbound and outbound queues but Celery created his owns and I do not see any message within those queues (inbound or outbound) :
inbound_queue = "_IN"
outbound_queue = "_OUT"
app = Celery()
app.conf.update(
broker_url = 'pyamqp://%s//' % path,
broker_heartbeat = None,
broker_connection_timeout = int(timeout)
result_backend = 'rpc://',
result_persistent = True,
task_queues = (
Queue(algorithm_queue, Exchange(inbound_queue), routing_key='default', auto_delete=False),
Queue(result_queue, Exchange(outbound_queue), routing_key='default', auto_delete=False),
),
task_default_queue = inbound_queue,
task_default_exchange = inbound_exchange,
task_default_exchange_type = 'direct',
task_default_routing_key = 'default',
)
#app.task(bind=True,
name='osmq.tasks.add',
queue=inbound_queue,
reply_to = outbound_queue,
autoretry_for=(Exception,),
retry_kwargs={'max_retries': 5, 'countdown': 2})
def execute(self, data):
<method_implementation>
I have implemented callbacks to get results back via REST APIs. However, randomly, it can return or not some results when the status is successfull. This is probably related to message persistency. In details, when I implement flower API to get info, status is successfull and the result is partially displayed (shortened json messages) - when I call AsyncResult, for the same status, result is either None or the right one. I do not understand the mechanism between rabbitmq queues and kombu which seems to cache the resulting message. I must guarantee to retrieve results everytime the task has been successfully executed.
def callback(uuid):
task = app.AsyncResult(uuid)
Specifically, it was that Celery 5.0+ did not support amqp:// as a result back end anymore. However, as your example, rpc:// is supported.
The relevant snippet is here: https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/index.html#rabbitmq
We tend to always ignore_results=True in our implementation, so I can't give any practical tips of how to use rpc://, other than to infer that any response is put on an application-specific queue, instead of being able to put on a specified queue (or even different broker / rabbitmq instance) via amqp://.

Inconsistent behavior of Quartz2 scheduler in Apache Camel

I have an Apache Camel project that is using Quartz2 as the scheduler. The requirement is to make it a cluster. The code is deployed to weblogic 12c. the quartz is configured as per many samples with clustering enabled.
This is my properties file (without the datasource)
org.quartz.scheduler.instanceName = MyScheduler
org.quartz.scheduler.instanceId = AUTO
org.quartz.scheduler.skipUpdateCheck = true
org.quartz.scheduler.jobFactory.class = org.quartz.simpl.SimpleJobFactory
org.quartz.threadPool.class = org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.threadCount = 10
org.quartz.threadPool.threadPriority = 5
org.quartz.jobStore.misfireThreshold = 60000
org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
org.quartz.jobStore.useProperties=true
org.quartz.JobBuilder.requestRecovery=true
org.quartz.jobStore.isClustered = true
org.quartz.jobStore.clusterCheckinInterval = 20000
When I deploy and start both nodes I see that the QRTZ_SCHEDULER_STATE table has extra entry for one of the nodes:
MyScheduler-routerContext server_node21567108546690
MyScheduler-routerContext-1 server_node11565896495100
MyScheduler-routerContext-1 server_node11567108547295
And I am guessing because of that the one node is being called once in a while while the other node gets called all the time (so occasionally both nodes are invoked at the same time).
I have tried to do a clean restart of weblogic nodes but the issue is still there
This is how my route(s) look like:
from("quartz2://provRegGroup/createUsersTrigger?cron={{create_users_cron}}&job.name=createUsersJob")
.routeId("createUsersRB")
.log("**** starting check for create users");
//where
//create_users_cron=0+0,5,10,15,20,25,30,35,40,45,50,55+*+*+*+?
//expecting one node being called by the scheduler at a time..
I figured out what caused the issue. apparently there were orphan weblogic processes that were running on one (or even both nodes) - this would be a question to our tech archs - why this was such a mess.. ps was showing two weblogic servers running on a node - one that I started recently and one that was there for say a month..
expecting this would never happen to production environment I assume the issue has been resolved..

MQ PCFParameter returning different values for Linux and Windows

When using IBM PCF Messages to monitor a queue, getting values of Input Count (MQIA_OPEN_INPUT_COUNT), it works perfectly for MQ Servers installed in Windows environment, but not for Linux. Not sure if it is a code or environment issue.
If we connect to a Windows service and perform que query there are more parameters in the response if compared to the Linux.
Same code, different results. Not sure if it is a configuration on the Channel, permissions or any other environment issue. On both MQ Servers the queues are local.
I've tried using IBM.WMQ.MQC.MQCMD_INQUIRE_Q_STATUS, with no success. Didn't find any workaround to get MQIA_OPEN_INPUT_COUNT.
PCFMessages documentation is very limited, so I didn't find anything related to this problem at MQIA_OPEN_INPUT_COUNT documentation:
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.ref.adm.doc/q087810_.htm
Any idea of how to solve this?
Public Function GetQtyQConnections(ByVal MQQueueName As String) As Integer
Dim queueManager As IBM.WMQ.MQQueueManager = Nothing
queueManager = New IBM.WMQ.MQQueueManager(AppSettings("MQQueueManagerName"), AppSettings("MQChannelName"), AppSettings("MQConnectionName"))
Dim oPCFMessageAgent As IBM.WMQ.PCF.PCFMessageAgent = New IBM.WMQ.PCF.PCFMessageAgent
oPCFMessageAgent.Connect(queueManager)
Dim pcfMsg As IBM.WMQ.PCF.PCFMessage = New IBM.WMQ.PCF.PCFMessage(IBM.WMQ.MQC.MQCMD_INQUIRE_Q)
pcfMsg.AddParameter(IBM.WMQ.MQC.MQCA_Q_NAME, MQQueueName)
Dim pcfResponse() As IBM.WMQ.PCF.PCFMessage = oPCFMessageAgent.Send(pcfMsg)
Dim pcfResponseLen As Integer = pcfResponse.Length
Dim inputcount As Integer = -1
For i As Integer = 0 To pcfResponseLen - 1
Dim oParams() As IBM.WMQ.PCF.PCFParameter = pcfResponse(i).GetParameters
For Each oParam As IBM.WMQ.PCF.PCFParameter In oParams
Select Case oParam.Parameter
Case IBM.WMQ.MQC.MQIA_OPEN_INPUT_COUNT
inputcount = Integer.Parse(oParam.GetValue())
End Select
Next
Next
Return inputcount
End Function
On Windows:
---------------
2016-QUEUENAME
20-1
134--3
2027-2018-03-12
2028-13.59.40
2019-
22-0
2030-
2029-
2124-
96-0
95-0
98--3
2004-2018-03-12
2005-13.59.40
3-0
2119-
61-0
6-0
5-1
184-1
188-0
4-2
7-1
2013-
34-0
9-0
8-1
272-2
2008-
17-0
15-5000
13-104857600
123--3
16-0
24-0
78-0
18-0
2012-
10-0
190-0
40-80
41-20
43-0
44-0
42-1
46-0
54-999999999
21-999999999
45-1
23-1
128--3
2023-
29-1
26-0
28-1
12-0
On Linux:
---------------
2016-QUEUENAME
20-6
2027-2019-03-11
2028-17.38.24
2030-
2029-
96-0
95-0
2119-
61-1
6-0
5-1
184-1
2013-QUEUEDESCRIPTION
10-0
2017-QUEUEMANAGER
2018-QUEUENAME
45-1
2024-QUEUEMANAGER
From your output I can see that the queue you have looked at on Windows is a local queue. The second parameter you display (20) is MQIA_Q_TYPE and it has a value of (1) MQQT_LOCAL.
The queue you have looked at on Linux however is a remote queue. It's MQIA_Q_TYPE (20) parameter has a value of (6) MQQT_REMOTE.
There are many differences between local queues and remote queues, and their attributes are quite different. Try using runmqsc and display a few local and remote queues to understand the differences. These differences have not occurred because of the different platform, just because of the different queue type.
You say in your question that on both MQ Servers the queues are local, but I'm afraid that is not what your output is showing.
Also, if you want to use the Inquire Queue command, please be sure you know that OpenInputCount and OpenOutputCount are only shown for local queues, not remote queues.
If your Linux and Windows releases of IBM MQ are at the same version level then you should get the same response parameters returned.
Why don't you format your output so that it is readable to the average human? Nobody will know what you mean by 2016, 2028, etc. (except for me and a few others)
Issues:
You did not specify the "Status Type" for the request. i.e. QUEUE vs HANDLE
You did not specify any attributes for the request.
Have a look at MQListQueueStatus01.java code I posted here: IBM MQ fetch LGETTIME using Java
Finally, why don't you use C# rather than VB? You could simply use all of the Java/MQ/PCF code that I post since C# is a clone of Java (so to speak).

Payara asadmin command to monitor a specific resource

Does anyone know the asadmin command line equivalent to display the Resource data as shown in the image below (ie the Resource __TimerPool)?
I'm using Payara 4.1.1.171.1.
I typed asadmin monitor --help and it provided this as
monitor [--help]
--type type
[--filename filename]
[--interval interval]
[--filter filter]
instance-name
The type field only accepts "httplistener", "jvm" and "webmodule" as inputs.
So I can't use a "resource" or "jdbcpool" as a type.
Oddly enough in the old glassfish 2.1 https://docs.oracle.com/cd/E19879-01/821-0185/gelol/index.html you can select "jdbcpool" as the type
Any help is appreciated.
I couldn't really find the answer on the payara documentation https://docs.payara.fish/documentation/payara-server/monitoring-service/monitoring-service.html
But using part of the glassfish documentation https://docs.oracle.com/cd/E18930_01/html/821-2416/ghmct.html#gipzv I was able to get what I needed.
The command is asadmin get --monitor server.resources.__TimerPool.*
This then returns (this is a partial output):
server.resources.__TimerPool.numconnused-highwatermark = 2
server.resources.__TimerPool.numconnused-lastsampletime =
1559826720029 server.resources.__TimerPool.numconnused-lowwatermark =
0 server.resources.__TimerPool.numconnused-name = NumConnUsed
server.resources.__TimerPool.numconnused-starttime = 1559823838730
server.resources.__TimerPool.numconnused-unit = count
server.resources.__TimerPool.numpotentialconnleak-count = 0
server.resources.__TimerPool.numpotentialconnleak-description = Number
of potential connection leaks
server.resources.__TimerPool.numpotentialconnleak-lastsampletime = -1
server.resources.__TimerPool.numpotentialconnleak-name =
NumPotentialConnLeak
server.resources.__TimerPool.numpotentialconnleak-starttime =
1559823838735 server.resources.__TimerPool.numpotentialconnleak-unit =
count server.resources.__TimerPool.waitqueuelength-count = 0
server.resources.__TimerPool.waitqueuelength-description = Number of
connection requests in the queue waiting to be serviced.
server.resources.__TimerPool.waitqueuelength-lastsampletime = -1
server.resources.__TimerPool.waitqueuelength-name = WaitQueueLength
server.resources.__TimerPool.waitqueuelength-starttime = 1559823838735
server.resources.__TimerPool.waitqueuelength-unit = count
Command get executed successfully.
It's important to add the .* at the end of the asadmin command in asadmin get --monitor server.resources.__TimerPool.*
If you neglect that and just enter asadmin get --monitor server.resources.__TimerPool it'll return
No monitoring data to report.
Command get executed successfully.
To see thelist of resources you have available to you to monitor type /asadmin list --monitor server.resources.*

How to start and stop multiple weblogic managed servers at one go through WLST

I am writing a code to start , stop, undeploy and deploy my application on weblogc.
My components need to be deployed on few managed servers.
When I do new deployments manually I can start and stop the servers in parallel, by ticking multiple boxes and selecting start and stop from the dop down. See below.
but when trying from WLST, i could do that in one server at a time.
ex:
start(name='ServerX',type='Server',block='true')
start(name='ServerY',type='Server',block='true')
shutdown(name='ServerX',entityType='Server',ignoreSessions='true',timeOut=600,force='true',block='true')
shutdown(name='ServerY',entityType='Server',ignoreSessions='true',timeOut=600,force='true',block='true')
Is there a way I can start stop multiple servers in once command?
Instead of directly starting and stopping servers, you create tasks, then wait for them to complete.
e.g.
tasks = []
for server in cmo.getServerLifeCycleRuntimes():
# to shut down all servers
if (server.getName() != ‘AdminServer’ and server.getState() != ‘RUNNING’ ):
tasks.append(server.start())
#or to start them up:
#if (server.getName() != ‘AdminServer’ and server.getState() != ‘SHUTDOWN’ ):
# tasks.append(server.shutdown())
#wait for tasks to complete
while len(tasks) > 0:
for task in tasks:
if task.getStatus() != ‘TASK IN PROGRESS’ :
tasks.remove(task)
java.lang.Thread.sleep(5000)
I know this is an old post, today I was reading this book "Advanced WebLogic Server Automation" written by Martin Heinzl so in the page 282 I found this.
def startCluster(clustername):
try:
start(clustername, 'Cluster')
except Exception, e:
print 'Error while starting cluster', e
dumpStack()
I tried it and it started managed servers in parallel.
Just keep in mind the AdminServer must be started first and your script must connect to the AdminServer before trying it.
Perhaps this would not be useful for you as the servers should be in a cluster, but I wanted to share this :)