Splunk:
{ [-]
guid: ABC
level: warn
message: Analytics Audit: analyticsLoaded
source: client
timestamp: 2017-08-07T16:38:38+00:00 }
{ [-]
guid: BAC
level: warn
message: Analytics Audit: doneWithAnalytics
source: client
timestamp: 2017-08-07T16:38:38+00:00 }
These messages show up for each guid. I would like to get duration between the first mesage " Analytics Audit: analyticsLoaded" showing up and the second message "Analytics Audit: doneWithAnalytics" by guid. And get the average duration for both messages showing up after the two messages to a guid.
Do basically, get the duration per guid. Get the average duration.
How can I do that in splunk?
Try this
index=blah | transaction guid startswith="analyticsLoaded" endswith="doneWithAnalytics" | timechart avg(duration)
Related
Need Help to understand this..
My application is listening to IBM MQ (On new Message).This MQ is subscribed to a topic., when messages are loaded to topic, my application processes them.
Having said that., attaching the logs, here mule event ids are not unique for each message.
Also, im logging correlationId as my job guid to track one end to end transaction in mule.
But none of the Ids are unique.
Also, im guessing the MessageID is being logged as eventID of mule - Correct me if wrong.
Note: I have also set 'disable Message ID' to true in my IBM MQ listener.
I just want to know why event ids or corelation id are not unique and to track one complete transaction in mule., what can be used?
Edited:
Logs as described - event Id of 2 different applications.
INFO 2023-02-07 07:23:28,506 [[MuleRuntime].uber.65573: [app-name].app-name-1-Flow.CPU_LITE #19dd2f11] [processor: app-name-1-Flow/processors/0; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
INFO 2023-02-07 07:23:28,119 [[MuleRuntime].uber.65571: [app-name].app-name-1-Flow.CPU_LITE #19dd2f11] [processor: app-name-1-Flow/processors/0; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
INFO 2023-02-07 07:21:34,373 [[MuleRuntime].uber.91422: [app-name].app-name-Flow.CPU_LITE #60d70fe6] [processor: app-name-Flow/processors/0; event: ID:414d51205150415a303555582020202005aa0563048ba823] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
INFO 2023-02-07 07:21:34,355 [[MuleRuntime].uber.91422: [app-name].app-name-Flow.CPU_LITE #60d70fe6] [processor: app-name-Flow/processors/0; event: ID:414d51205150415a303555582020202005aa0563048ba823] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Payload Received:
Logs for jobGuid (set as correlation ID) is same as eventID
INFO 2023-02-07 07:23:28,123 [[MuleRuntime].uber.65572: [app-name].app-name-1-Flow.CPU_INTENSIVE #10f2a539] [processor: flow-name-SubFlow/processors/0; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","txnGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"ProcessedSub","sourceEpoc":"1675772608120","now":"1675772608122"}}
INFO 2023-02-07 07:23:28,122 [[MuleRuntime].uber.65572: [app-name].app-name-1-Flow.CPU_INTENSIVE #10f2a539] [processor: app-name-1-Flow/processors/3; event: ID:414d5120515030355558202020202020d0752b63a5f3a921] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","txnGuid":"ID:414d5120515030355558202020202020d0752b63a5f3a921","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"Received","sourceEpoc":"1675772608120","now":"1675772608121"}}
INFO 2023-02-07 07:21:34,656 [[MuleRuntime].uber.91422: [app-name].app-name-2-Flow.CPU_INTENSIVE #1ae0a3cf] [processor: app-name-2-Flow/processors/4/route/0/processors/2; event: ID:414d51205150415a303555582020202005aa0563048ba823] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","txnGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"Received","sourceEpoc":"1675772494374","now":"1675772494656"}}
INFO 2023-02-07 07:21:34,653 [[MuleRuntime].uber.91419: [app-name].app-name-2-Flow.CPU_INTENSIVE #1ae0a3cf] [processor: app-name-2-Flow/processors/4/route/0/processors/2; event: ID:414d51205150415a303555582020202005aa0563048ba823] com.mule: {"jobControl":{"message":"","jobGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","txnGuid":"ID:414d51205150415a303555582020202005aa0563048ba823","appName":"app-name","source":"sourceSystem","sourceType":"QUEUE","status":"Received","sourceEpoc":"1675772494355","now":"1675772494653"}}
Event Ids -also called correlation ids- are usually unique when generated by Mule automatically. When overridden with the message id from a queue broker, like IBM MQ, it is up to the message generator to send unique ids. In the log snippets you shared the event id looks to be a sequence of bytes in hexadecimal. That's usual for IBM MQ message ids. Default Mule event ids are formatted GUIDs.
have two message threads, each thread consists of ten messages. I need to request to display these two chains in one.
The new thread must consist of ten different messages: five messages from one system, five messages from another (backup) system. Messages from the system use the same SrcMsgId value. Each system has a unique SrcMsgId within the same chain. The message chain from the backup system enters the splunk immediately after the messages from the main system. Messages from the standby system also have a Mainsys_srcMsgId value - this value is identical to the main system's SrcMsgId value. Tell me how can I display a chain of all ten messages? Perhaps first messages from the first system (main), then from the second (backup) with the display of the time of arrival at the server.
Specifically, we want to see all ten messages one after the other, in the order in which they arrived at the server. Five messages from the primary, for example: ("srcMsgId": "rwfsdfsfqwe121432gsgsfgd71") and five from the backup: ("srcMsgId": "rwfsdfsfqwe121432gsgsfgd72"). The problem is that messages from other systems also come to the server, all messages are mixed (chaotically), which is why we want to organize all messages from one system and its relative in the search. Messages from the backup system are associated with the main system only by this parameter: "Mainsys_srcMsgId" - using this key, we understand that messages come from the backup system (secondary to the main one).
Examples of messages from the primary and secondary system:
Main system:
{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F32",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"Mainsys_srcMsgId": "",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
.....................................
Message from backup system:
{
"event": "Sourcetype test please",
"sourcetype": "testsystem-2",
"host": "some-host-123",
"fields":
{
"messageId": "ED280816-E404-444A-A2D9-FFD2D171F23",
"srcMsgId": "rwfsdfsfqwe121432gsgsfgd72",
"Mainsys_srcMsgId": "rwfsdfsfqwe121432gsgsfgd71",
"baseSystemId": "abc1",
"routeInstanceId": "abc2",
"routepointID": "abc3",
"eventTime": "1985-04-12T23:20:50Z",
"messageType": "abc4",
"GISGMPRequestID": "PS000BA780816-E404-444A-A2D9-FFD2D1712345",
"GISGMPResponseID": "PS000BA780816-E404-444B-A2D9-FFD2D1712345",
"resultcode": "abc7",
"resultdesc": "abc8"
}
}
When we want to combine in a query only five messages from one chain, related: "srcMsgId".
We make the following request:
index="bl_logging" sourcetype="testsystem-2"
| транзакция maxpause=5m srcMsgId Mainsys_srcMsgId messageId
| таблица _time srcMsgId Mainsys_srcMsgId messageId продолжительность eventcount
| сортировать srcMsgId_time
| streamstats current=f window=1 значения (_time) as prevTime по теме
| eval timeDiff=_time-prevTime
| delta _time как timediff
I have a Keda Scaledjob configured to spawn 1 job per message having the state 'ready' in RabbitMQ.
It has a max replica count set to 70.
Observed:
When there are many messages in the queue, let's say 300, Keda correctly creates new jobs to reach the max replica count limit => So there are 70 running jobs each consuming 1 message from the queue.
When there are few messages in the queue, let's say 1 Ready and 1 Unacked, Keda refuses to create a new job even if there's enough resources in the cluster.
It's like waiting until the current running job finishes to spawn a new job.
Here's my Keda configuration :
---
# Reference - https://keda.sh/docs/2.0/concepts/scaling-jobs/
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: scaledjob-puppeteer
labels:
environment: development
app: puppeteer-display
spec:
jobTargetRef:
parallelism: 1 # [max number of desired pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
completions: 1 # [desired number of successfully finished pods](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#controlling-parallelism)
activeDeadlineSeconds: 7200 # (2 hours) Specifies the duration in seconds relative to the startTime that the job may be active before the system tries to terminate it; value must be positive integer
backoffLimit: 2 # Specifies the number of retries before marking this job failed. Defaults to 6
template:
spec:
volumes:
...
containers:
...
pollingInterval: 10
successfulJobsHistoryLimit: 0
failedJobsHistoryLimit: 0
maxReplicaCount: 75
triggers:
- type: rabbitmq
metadata:
protocol: amqp
queueName: tasks
mode: QueueLength
value: "1"
authenticationRef:
name: keda-trigger-auth-rabbitmq-conn
---
How to make Keda to create a job whenever the queue has >= 1 message ?
Edit: It seems like it waits for at least 1 hour before creating the new job.
The problem seems to be the missing scalingStrategy setting. You can add following configuration:
scalingStrategy:
strategy: accurate
The accurate setting is used when you consume messages from your queue instead of locking the messages. This is often used in other message queues.
For reference you can look into https://keda.sh/docs/2.7/concepts/scaling-jobs/
You can find further information about the scaling strategies in the details section.
I have the following query with the following output:
Query:
filter #message like /A:|B:/
Output:
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
[INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':0}
I would like to print ONLY the A message when in the B message 'MyValue' = 0. For the above example, I would have to have the following output
Output:
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
For the next example
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
[INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':12}
The output should be empty
I can't do something like this because I miss the A message:
filter #message like /A:|B:/
filter MyValue = 0
Any ideas?
If anyone still interested, there IS ways to get the first and last from grouping by a field. So if you can fit your data into pairs of messages, it might help.
For example, given API Gateway access log (each row is a #message):
2021-09-14T14:09:00.452+03:00 (01c53288-5d25-*******) Extended Request Id: ***************
2021-09-14T14:09:00.452+03:00 (01c53288-5d25-*******) Verifying Usage Plan for request: 01c53288-5d25-*******. API Key: API Stage: **************/dev
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) API Key authorized because method 'ANY /path/{proxy+}' does not require API Key. Request will not contribute to throttle or quota limits
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) Usage Plan check succeeded for API Key and API Stage **************/dev
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) Starting execution for request: 01c53288-5d25-*******
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) HTTP Method: GET, Resource Path: /path/json.json
2021-09-14T14:09:00.468+03:00 (01c53288-5d25-*******) Method completed with status: 304
We can get method, uri and return code from the last 2 rows.
To do this, I parse the relevant data into params, and then get them by doing aggregation by request id (that i also parse)
The magic is: using stats likesortsFirst() and sortsLast() and grouping by #reqid. (AWS Docs
Note: IMO, don't use earliest() and latest() as they depend on built-in #timestamp and worked weird for me where 2 sequential messages had the same timestamp
So, for example, using this query:
filter #message like "Method"
| parse #message /\((?<#reqid>.*?)\) (.*?) (Method: (?<#method>.*?), )?(.*?:)* (?<#data>[^\ ]*)/
| sort #timestamp desc
| stats sortsFirst(#method) as #reqMethod, sortsFirst(#data) as #reqPath, sortsLast(#data) as #reqCode by #reqid
| limit 20
We would get the following desired output:
#reqid #reqMethod #reqPath #reqCode
f42e2b44-b858-45cb-***************** GET /path-******.json 304
fecddb03-3804-4ff5-***************** OPTIONS /path-******.json 200
e8e47185-6280-4e1e-***************** GET /path-******.json 304
e4fa9a0c-6d75-4e26-***************** GET /path-******.json 304
Sql server 2005 service pack 2 version: 9.00.3042.00
All maintenance plans fail with the same error.
The details of the error are:-
Execute Maintenance Plan
Execute maintenance plan. test7 (Error)
Messages
Execution failed. See the maintenance plan and SQL Server Agent job history logs for details.
The advanced information section shows the following;
Job 'test7.Subplan_1' failed. (SqlManagerUI)
Program Location:
at Microsoft.SqlServer.Management.SqlManagerUI.MaintenancePlanMenu_Run.PerformActions()
At this point the following appear in the windows event log:
Event Type: Error
Event Source: SQLISPackage
Event Category: None
Event ID: 12291
Date: 28/05/2009
Time: 16:09:08
User: 'DOMAINNAME\username'
Computer: SQLSERVER4
Description:
Package "test7" failed.
and also this:
Event Type: Warning
Event Source: SQLSERVERAGENT
Event Category: Job Engine
Event ID: 208
Date: 28/05/2009
Time: 16:09:10
User: N/A
Computer: SQLSERVER4
Description:
SQL Server Scheduled Job 'test7.Subplan_1' (0x96AE7493BFF39F4FBBAE034AB6DA1C1F) - Status: Failed - Invoked on: 2009-05-28 16:09:02 - Message: The job failed. The Job was invoked by User 'DOMAINNAME\username'. The last step to run was step 1 (Subplan_1).
There are no entries in the SQl Agent log at all.
Probably no points for this, but you're likely to get more help on this over at ServerFault.com now that they are open.