I don't think this is possible, but wanted to ask anyway -- is there a way in Cloudwatch Insights where I can find the count of how often a log with a dynamic value is emitted with the same value from distinct logs? The use case I have is we want to compare log statements from two different code paths, so we attach the same requestID to both log statements. So to illustrate what might happen, two logs may get emitted
Log 1:
{
message: "SPECIAL_LOG_EMITTED Operation1_Emitted"
requestID: "123456"
}
Log2:
{
message: "SPECIAL_LOG_EMITTED Operation2_Emitted"
requestID: "123456"
}
So ideally I could do something like
fields #timestamp, #message, requestID
| filter #message like "SPECIAL_LOG_EMITTED"
| parse #message '*_Emitted' as operation
| stats count(*) as all, sum (operation LIKE 'Operation1') as Op1, sum (operation LIKE 'Operation2') as Operation2 by bin(5m)
And then from this find out where the requestID is matching. The requestID is dynamic, though, so I can't just hard-code it -- I want to find how often logs are emitted with matching requestIDs.
I've considered looking into count_distinct but that seems like the wrong approach (correct me if I'm wrong)
Related
Small question for a Splunk query please.
May I ask if there is a way to search for “a first log that got printed, but the second was not printed” statement please? Background, I have a very simple piece of Java logic as follow:
LOGGER.info("START/END compute something that might result in a bad exception for id START " + id);
invote_method_which_can_fail(id);
LOGGER.info("START/END compute something that might result in a bad exception for id END " + id);
Which results in something like (snippet from a million):
START/END compute something that might result in a bad exception for id START 12345
START/END compute something that might result in a bad exception for id END 12345
START/END compute something that might result in a bad exception for id START 88888
START/END compute something that might result in a bad exception for id START 98765
START/END compute something that might result in a bad exception for id END 98765
As you can see, the id 88888 in my example got the start statement printed, but not the end statement, because something bad happened in the java code. (the question is not about how to make the java code reliable)
May I ask if there is a Splunk query which can find me those id please?
What I tried: So far, I am downloading the search result containing all the starts. Then, downloading the search results with all the ends. Once having both, I am running another offline script in order to find all the id from the first search result that are not there from the second...
I do not think this is "the smart thing to do" and was wondering if there is a smarter query which can give me the expected result directly in Splunk.
Thank you
You can try something along these lines (with rex and stats):
index=... "START/END compute something that might result in a bad exception for id"
| rex "(?<operation>(START|END))\s+(?<id>\d+)"
| stats count(eval(operation="START")) as start count(eval(operation="END")) as end by id
| where NOT start=end
I have not tested this SPL code
I am using AWS Cloudwatch Insights and running a query like this:
fields #message, #timestamp
| filter strcontains(#message, "Something of interest happened")
| stats count() as interestCount by bin(10m) as tenMinuteTime
| stats max(interestCount) by datefloor(tenMinuteTime, 1d)
However, on the last line, I get the following error:
mismatched input 'stats' expecting {K_PARSE, K_SEARCH, K_FIELDS, K_DISPLAY, K_FILTER, K_SORT, K_ORDER, K_HEAD, K_LIMIT, K_TAIL}
It would seem to mean from this that I cannot take multiple layers of stat queries in Insights, and thus cannot take a statistic of a statistic. Is there a way around this?
You cannot currently use multiple stat commands and from what I know there is no direct way around that at this time. You can however thicken up your single stat command and separate by comma, like so:
fields #message, #timestamp
| filter strcontains(#message, "Something of interest happened")
| stats count() as #interestCount,
max(interestCount) as #maxInterest,
interestCount by bin(10m) as #tenMinuteTime
You define fields and use functions after stats and then process those result fields.
I have events something like:
{
taskId:5a6d
category:created
when:1517131461
...
}
{
taskId:5a6d
category:started
when:1517131609
...
}
{
taskId:5a6d
category:ended
when:1517134657
...
}
For each task (task id is same), we have events when it is created / started / ended.
I'd like to search if there is any task never be processed (task is created but not started). Here is my search statement:
index=XXX sourcetype=XXX category=created | search NOT [search index=XXX sourcetype=XXX category=started | fields taskId]
This statement works correctly if the time range is less than 48 hours.
If the time range is set to, for example, latest 7 days, the above search statement works incorrectly. It returns a lot of tasks (category=created) which means these tasks are never processed. Actually, they are processed, I can search the events (category=started) by taskId.
I have no idea what's wrong with it. it seems subsearch doesn't return correct results in the range of main search.
This will be hard to debug without seeing your exact data.
To make it simpler, you can try something like this to do everything with one search:
index=XXX sourcetype=XXX category=created
| eventstats values(category) as categories by taskId
| search categories = created NOT categories = started
Suppose you have a Keen IO collection called "survey-completed" that contains events matching the following pattern:
keen.id: <unique autogenerated id>
keen.timestamp: <autogenerated overridable timestamp>
userId: <hex string for user>
surveyScore: <integer from 1 to 10>
...
How would you create a report of only the most up-to-date satisfaction score for each user that responded to one or more surveys within a given amount of time (such as one week)?
There isn't a really elegant way to make it happen, but for a given userId you could successfully return your the most up-to-date event create a count query with a group_by on [surveyScore, keen.timestamp] and an order_by on the keen.timestamp property. You will want to set limit=1 to select only the most recent surveyScore.
If you'd like to use an extraction, the most straight forward way would be to run an extraction with property_names set to ["userId","keen.timestamp","surveyScore"]. Once you receive the results you can then do some client-side post processing. This is probably the best way if you want to take a look at all of your userIds.
If you're interested in a given userId and want to use an extraction, you can run an extraction with a filter on the userId eq X, define the optional parameter latest set to latest=1. The latest property is an integer containing the number of most recent events to extract. Note: The use of latest will call upon the keen.created_at timestamp instead of keen.timestamp (https://keen.io/docs/api/#the-keen-object).
I'm writing a query to determine what percentage of events are error events for a camera-based system.
To narrow logged events down to camera events, I have event=camera* in the initial query.
What I want to do next is treat the event as bad if it's in a subset, so I want something like:
event=camera* | eval bad_event=IF(event IN (camera-failed, camera-error, ...))
but I am not sure of the correct syntax for this in Splunk.
I tried eval bad_event=IF(event=camera-failed OR event=camera-error), but got the message Error in 'eval' command: The arguments to the 'if' function are invalid.
How do I check if the event is in a subset of its possible values?
You could do with with coalesce and case, or if and match (documentation):
Using case:
| eval event_type=coalesce(case(event=='camera-failed','bad',event=='camera-error','bad'), 'good')
Using match:
| eval event_type=if(match(event_type, 'camera-(failed|error)'),'bad', 'good')