I have two Cloudwatch insights queries that I would love to be able to run side by side and compare the results of both two.
stats count(*) as requestIdCount by #requestId
| filter #message like /START RequestId/
| filter requestIdCount > 1
stats count(*) as requestIdCount by #requestId
| filter #message like /END RequestId/
| filter requestIdCount > 1
It would be great to be able to do
fields (
stats count(*) as requestIdCount by #requestId
| filter #message like /END RequestId/
| filter requestIdCount > 1) as EndRequestCount,
(
stats count(*) as requestIdCount by #requestId
| filter #message like /START RequestId/
| filter requestIdCount > 1) as StartRequestCount
But I don't see any way to do subqueries in insights right now. Is there a method to combine queries like this?
Try this:
parse #message 'START RequestId' as #startRequestId
| parse #message 'END RequestId' as #endRequestId
| stats count(#startRequestId) as startRequestIdCount , count(#endRequestId) as endRequestIdCount by bin(5m)
| filter startRequestIdCount > 1
| filter endRequestIdCount > 1
CloudWatch Logs Insights Query Syntax
You can create a logic via API or CLI in order to use the output of a query as the input of another query
Amazon CloudWatch Logs API Reference
AWS CLI Command Reference - logs
It works as a script where you make a request, interpret the results, and then issue another requests with the results of the first one. It's a bit more work but I'm not aware of another way to do so
Related
I have the following AWS Cloudwatch query:
fields #timestamp, #message
| filter #message like /(?i)(error|except)/
| filter !ispresent(level) and !ispresent(eventType)
| stats count(*) as ErrorCount by #message
| sort ErrorCount desc
Results end up looking something like this with the message and a count:
The first 4 results are actualy the same error. However, since they have different (node:*) values at the beginning of the message, it ends up grouping them as different errors.
Is there a way for the query to parse/ignore the (node:*) part so that the first 4 results in the image would be considered just one result with a total count of 2,997?
With this count query by bin:
filter #message like / error /
| stats count() as exceptionCount by bin(30m)
I get a discontinuous graph, which is hard to grasp:
Is is possible for AWS Cloudwatch Log Insights to consider the empty bin as zero count to get a continuous graph?
Found your question looking for my own answer to this.
The best that I came up with is to calculate a 'presence' field and then use sum to get 0's in the time bins.
I used strcontains, which returns a 1 when matched or 0 when not. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html#CWL_QuerySyntax-operations-functions
Mine looks like this:
fields #timestamp, #message
| fields strcontains(#message, 'Exit status 1') as is_exit_message
| stats sum(is_exit_message) as is_exit_message_count by bin(15m) as time_of_crash
| sort time_of_crash desc
So, yours would be:
fields strcontains(#message, 'error') as is_error
| stats sum(is_error) as exceptionCount by bin(30m)
Use strcontains + sum or parse + count.
The point is not using filter. You should query all of logs.
I am working on a dashboard query where I want to have a count of all the transactions that took more than a certain amount to complete.
My query is something like:
fields message
| filter kubernetes.namespace_name = 'feature-7355'
| filter message like "INFO"
| filter message like "Metric"
| parse '[*] * [*] * - *' as logLevel, timeStp, threadName, classInfo, logMessage
| parse logMessage 'Header: [*]. Metric: [*]. TimeSpent: [*]. correlationId: [*]' as headers, metric, timeSpent, correlationId
| filter ispresent(correlationId)
| stats sum(timeSpent) as TotalTimeSpentByTransaction by correlationId
| filter TotalTimeSpentByTransaction > 2000
| stats count(timeCorrelationId) as correlationIdCount
When I try to execute this I am getting an error:
mismatched input 'stats' expecting {K_PARSE, K_SEARCH, K_FIELDS, K_DISPLAY, K_FILTER, K_SORT, K_ORDER, K_HEAD, K_LIMIT, K_TAIL}
Is there a way to work this out, Can someone help me resolve this?
I wonder whether someone can help me please.
I'd made the following post about Splunk query I'm trying to write:
https://answers.splunk.com/answers/724223/in-a-table-powered-by-a-stats-count-search-can-you.html
I received some great help, but despite working on this for a few days now concentrating on using eval if statements, I still have the same issue with the "Successful" and "Unsuccessful" columns showing blank results. So I thought I'd cast the net a little wider and ask please whether someone maybe able to look at this and offer some guidance on how I may get around the problem.
Many thanks and kind regards
Chris
I tried exploring your use-case with splunkd-access log and came up with a simple SPL to help you.
In this query I am actually joining the output of 2 searches which aggregate the required results (Not concerned about the search performance).
Give it a try. If you've access to _internal index, this will work as is. You should be able to easily modify this to suit your events (eg: replace user with ClientID).
index=_internal source="/opt/splunk/var/log/splunk/splunkd_access.log"
| stats count as All sum(eval(if(status <= 303,1,0))) as Successful sum(eval(if(status > 303,1,0))) as Unsuccessful by user
| join user type=left
[ search index=_internal source="/opt/splunk/var/log/splunk/splunkd_access.log"
| chart count BY user status ]
I updated your search from splunk community answers (should look like this):
w2_wmf(RequestCompleted)`request.detail.Context="*test"
| dedup eventId
| rename request.ClientID as ClientID detail.statusCode AS statusCode
| stats count as All sum(eval(if(statusCode <= 303,1,0))) as Successful sum(eval(if(statusCode > 303,1,0))) as Unsuccessful by ClientID
| join ClientID type=left
[ search w2_wmf(RequestCompleted)`request.detail.Context="*test"
| dedup eventId
| rename request.ClientID as ClientID detail.statusCode AS statusCode
| chart count BY ClientID statusCode ]
I answered in Splunk
https://answers.splunk.com/answers/724223/in-a-table-powered-by-a-stats-count-search-can-you.html?childToView=729492#answer-729492
but using dummy encoding, it looks like
w2_wmf(RequestCompleted)`request.detail.Context="*test"
| dedup eventId
| rename request.ClientId as ClientID, detail.statusCode as Status
| eval X_{Status}=1
| stats count as Total sum(X_*) as X_* by ClientID
| rename X_* as *
Will give you ClientID, count and then a column for each status code found, with a sum of each code in that column.
As I gather you can't get this working, this query should show dummy encoding in action
`index=_internal sourcetype=*access
| eval X_{status}=1
| stats count as Total sum(X_*) as X_* by source, user
| rename X_* as *`
This would give an output of something like
I want to make time chart table like this:
Currently I using two queries
1.Get transaction column :
sourcetype="mysource" host="myhost" | timechart count span=1h
2.Get transaction_success column :
sourcetype="mysource" host="myhost" status="2" | timechart count span=1h
Then combine them manually with Excel.
How to search that data with only one query?
currently i found how to join the queries:
sourcetype="mysource" host="myhost" | timechart count as transaction count(eval(status="2")) as transaction_success span=1h
I hope append should work in your case. Query 1 append Query 2
Please go thru the following posts in splunk
https://answers.splunk.com/answers/28621/combine-2-splunk-queries.html
https://answers.splunk.com/answers/182453/how-to-combine-my-two-search-queries-using-join-or.html
https://answers.splunk.com/answers/30909/combine-two-queries-into-a-single-value.html
https://answers.splunk.com/answers/123204/how-to-combine-two-queries-into-one-without-using-eventtypes.html