Splunk query filter out based on other event in same index - splunk

I have a index named Events
It contains a bunch of different events, all events have a property called EventName.
Now I want to do a query where I return everything that matches the following:
IF AccountId exists in event with EventName AccountCreated AND there is at least 1 event with EventName FavoriteCreated with the same AccountId -> return all events where EventName == AccountCreated
Example events:
AccountCreated
{
"AccountId": 1234,
"EventName": "AccountCreated",
"SomeOtherProperty": "Some value",
"Brand": "My Brand",
"DeviceType": "Mobile",
"EventTime": "2020-06-01T12:13:14Z"
}
FavoriteCreated
{
"AccountId": 1234,
"EventName": "FavoritesCreated,
"Brand": "My Brand",
"DeviceType": "Mobile",
"EventTime": "2020-06-01T12:13:14Z"
}
Given the following two events, I would like to create 1 query that returns the AccountCreated event.
I've tried the following but it does not work, surely I must be missing something simple?
index=events EventName=AccountCreated
[search index=events EventName=FavoriteCreated | dedup AccountId | fields AccountId]
| table AccountId, SomeOtherProperty
Im expecting ~6000 hits here but Im only getting 2298 events. What am I missing?
UPDATE
Based on the answer given by #warren below, the following query works. The only problem is that it's using a JOIN which limits us to 50K results from the subsearch. When running this query I get 5900 results in total = Correct.
index=events EventName=AccountCreated AccountId=*
| stats count by AccountId, EventName
| fields - count
| join AccountId
[ | search index=events EventName=FavoriteCreated AccountId=*
| stats count by AccountId ]
| fields - count
| table AccountId, EventName
I then tried to use his updated example like this but the problem seems to be that it returns FavoriteCreated events instead of AccountCreated.
When running this query I get 25 494 hits = Incorrect.
index=events AccountId=* (EventName=AccountCreated OR EventName=FavoriteCreated)
| stats values(EventName) as EventName by AccountId
| eval EventName=mvindex(EventName,-1)
| search EventName="FavoriteCreated"
| table AccountId, EventName
Update 2 - WORKING
#warren is awesome, here is a full working query that only returns data from the AccountCreated events IF 1 or more FavoriteCreated event exists.
index=events AccountId=* (EventName=AccountCreated OR EventName=FavoriteCreated)
| stats
values(Brand) as Brand,
values(DeviceType) as DeviceType,
values(Email) as Email,
values(EventName) as EventName
values(EventTime) as EventTime,
values(Locale) as Locale,
values(ClientIp) as ClientIp
by AccountId
| where mvcount(EventName)>1
| eval EventName=mvindex(EventName,0)
| eval EventTime=mvindex(EventTime,0)
| eval ClientIp=mvindex(ClientIp,0)
| eval DeviceType=mvindex(DeviceType,0)

You found, perhaps, one factor of your issues - that subsearches are capped at 50,000 (when doing a join) events (or 60 seconds run time (or 10,000 results when you use a "normal" subsearch)).
Start by dumping dedup in favor of stats:
index=events EventName=AccountCreated AccountId=*
| stats count by AccountId, SomeOtherProperty [, more, fields, as, desired]
| fields - count
| search
[ | search index=events EventName=FavoriteCreated AccountId=*
| stats count by AccountId
| fields - count]
<rest of search>
If that doesn't get you where you want to be (ie, you still have too many results in your subsearch), you can try join:
index=events EventName=AccountCreated AccountId=*
| stats count by AccountId, SomeOtherProperty [, more, fields, as, desired]
| fields - count
| join AccountId
[ | search index=events EventName=FavoriteCreated AccountId=*
| stats count by AccountId ]
| fields - count
<rest of search>
There are yet more ways of doing what you're looking for - but these two should get you a long ways toward your goal
Here's a join-less approach which'll show only "FavoriteCreated" events:
index=events AccountId=* (EventName=AccountCreated OR EventName=FavoriteCreated)
| stats values(EventName) as EventName values(SomeOtherProperty) as SomeOtherProperty by AccountId
| eval EventName=mvindex(EventName,-1)
| search EventName="FavoriteCreated"
And here's one that shows "FavoriteCreated" only if there was also an "AccountCreated" event in the same timeframe:
index=events AccountId=* (EventName=AccountCreated OR EventName=FavoriteCreated)
| stats values(EventName) as EventName values(SomeOtherProperty) as SomeOtherProperty by AccountId
| where mvcount(EventName)>1
And if you want to 'pretend' the values() didn't happen (ie, throw-out the "favoriteCreated" entry), add this:
| eval EventName=mvindex(EventName,0)

Turns out that Splunk truncates the SubSearch result if its bigger than 10 000 results...

Related

SPLUNK use result from first search in second search

Say I have a query such as
index="example" source="example.log" host="example" "ERROR 1234"
| stats distinct_count by id
This will give me all the events with that error code per id.
I then want to combine this query to search the same log file for another string but only on the unique id's returned from the first search. Because the new string will appear on a separate event I can't just do an 'AND'.
There are a few ways to do that, including using subsearches, join, or append, but those require multiple passes through the data. Here is a way that makes a single pass through the index.
index=example source="example.log" ("ERROR 1234" OR "ERROR 5678")
``` Check for the presence of each string in the event ```
| eval string1=if(searchmatch("ERROR 1234"), 1, 0)
| eval string2=if(searchmatch("ERROR 5678"), 1, 0)
``` Count string occurrences by id ```
| stats sum(string1) as string1, sum(string2) as string2 by id
``` Keep only the ids that have both strings ```
| where (string1 > 0 AND string2 > 0)
You can search for "some other string" in subsearch and then join the queries on the id:
index="example" source="example.log" host="example" "ERROR 1234"
| join id [search index="example" source="example.log" host="example" "some other string" ]
| stats distinct_count by id
Presuming your id field is the same and available in both indices, this form should work:
(index=ndxA sourcetype=srctpA id=* source=example.log host=example "ERROR 1234") OR (index=ndxB sourcetype=srctpB id=* "some other string")
| rex field=_raw "(?<first_field>ERROR 1234)"
| rex field=_raw "(?<second_field>some other string)"
| fillnull value="-" first_field second_field
| stats count by id first_string second_string
| search NOT (first_string="-" OR second_string="-")
If your id field has a different name in the other index, do a rename like this before the stats line:
| rename otherIdFieldName as id
Advantages of this format:
you are not limited by subsearch constraints (search must finish in 60 seconds, no more than 50k rows)
the Search Peers (ie Indexers) will handle all of the overhead instead of having to wait on the Search Head that initiated the search to do lots of post-processing (all the SH is doing is sending the distributed search, then a post-stats filter to ensure both first_string and second_string have the values you are looking for)

How to show the time difference between two events in a Splunk join query?

Consider the following log data,
{"level":"info","msg":"Enqueued a recurring task","time":"2022-04-01T18:46:22.854684-07:00","uuid":"4e9fc098-f611-4128-ae0b-2e6b9cb232c8"}
{"level":"info","msg":"Reported a result","path":"/results","time":"2022-04-01T18:46:22.955999-07:00","uuid":"4e9fc098-f611-4128-ae0b-2e6b9cb232c8"}
{"level":"info","msg":"Reported a result","path":"/results","time":"2022-04-01T18:46:23.056295-07:00","uuid":"4e9fc098-f611-4128-ae0b-2e6b9cb232c8"}
{"level":"info","msg":"Enqueued a recurring task","time":"2022-04-01T18:46:23.056376-07:00","uuid":"28e9bea9-5d0c-4dd5-af4f-c22944fc4fcd"}
It represents enqueuing a recurring task with a certain uuid, the results of which can be reported multiple times (or not at all) with the same uuid. I'm interested in determining the interval elapsed between when the task was enqueued and when the result was reported for the first time. So far, I can display the results this outer join table,
msg="Enqueued a recurring task"
| join type=outer left=L right=R where L.uuid = R.uuid
[ search msg="Reported a result" | dedup uuid sortby +_time]
| fillnull value=""
| table _time, L.msg, L.uuid, R.msg, L.time, R.time
What I'd like for convenience is to add an additional column with the difference between R.time and L.time. As far as I can tell from how to calculate duration between two events Splunk, one way to do this is to use strptime to convert those time fields into time values and then determine their difference. However, the time of the events was already parsed when importing the data (as seen from the built-in _time field) so this seems inefficient to me. Is there a more efficient way?
Update
Even the 'inefficient' method using strptime is proving tricky; I've added eval fields t to each of the searches,
msg="Enqueued a recurring task"
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")
| join type=outer left=L right=R where L.uuid = R.uuid
[ search msg="Reported a result"
| dedup uuid sortby +_time
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")]
| fillnull value=""
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, R.t-L.t
and it appears to parse the time correctly, yet the difference R.t-L.t shows up empty in the table:
Any idea why it's not showing the difference?
Update 2
I've tried both of RichG's answers, but they both lead to the following error in Splunk:
Error in 'eval' command: Type checking failed. '-' only takes numbers.
(See screenshots below). The way I'm running Splunk is with a Docker container,
docker run -p 8000:8000 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=supersecret" --name splunk splunk/splunk:latest
Update 3
I finally got this to work using RichG's updated answer, combining both an evaluated strftime field with a diff field where R.t and L.t are quoted:
msg="Enqueued a recurring task"
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")
| join type=outer left=L right=R where L.uuid = R.uuid
[ search msg="Reported a result"
| dedup uuid sortby +_time
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")]
| fillnull value=""
| eval diff='R.t'-'L.t'
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, diff
The time difference is empty in the table because the table command does not perform arithmetic. You must calculate the difference in a separate eval and display the result in the table.
index=foo msg="Enqueued a recurring task"
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")
| join type=outer left=L right=R where L.uuid = R.uuid
[ search index=foo msg="Reported a result"
| dedup uuid sortby +_time
| eval t=strptime(time, "%Y-%m-%dT%H:%M:%S.%6N%:z")]
| fillnull value=""
| eval diff='R.t' - 'L.t'
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, diff
You should be able to use each event's _time field to avoid having to parse the time field.
index=foo msg="Enqueued a recurring task"
| join type=outer left=L right=R where L.uuid = R.uuid
[ search index=foo msg="Reported a result"
| dedup uuid sortby +_time ]
| fillnull value=""
| eval diff='R._time' - 'L._time'
| table _time, L.msg, L.uuid, R.msg, L.time, R.time, L.t, R.t, diff
Fairly confident something akin to this should work (without using join:
index=ndx sourtype=srctp uuid=* msg=*
| stats min(_time) as first_time max(_time) as last_time earliest(msg) as first_msg latest(msg) as last_msg by uuid
| eval diff_seconds=last_time-first_time
| eval first_time=strftime(first_time,"%c"), last_time=strftime(last_time,"%c")
This approach will presume that _time has been set properly in the sourcetype's props.conf, but if it has, this gets you what you're looking for in one pass.

Splunk query to get non matching ID from two query

Index=* sourcetype="publisher" namespace="app_1" | table ID message | where message="published"
Index=* sourcetype="consumer" namespace="app_1" | table ID message | where message="consumed"
I want to display non matching ID by comparing both the query, how can I achieve this.
If query 1 is giving 100 records and query 2 is giving 90 records and all 90 records are present in query 1 them I want to see 10 records which are not present in query 2.
There are several ways to achieve this outcome.
The following counts the number of consumer and producers events for each ID, then shows the IDs of the events that occur only once.
index=* sourcetype="publisher" OR sourcetype="consumer" namespace="app_1" ID="*" | stats count by ID | where count<2
In this next method, we use a sub search and a join. This has the benefit of giving you the full event, not just the ID.
index=* sourcetype="publisher" namespace="app_1" ID="*" | join type=outer ID [ search index=* sourcetype="consumer" namespace="app_1" ID="*" | eval does_match=1 ] | where isnull(does_match)

Join two Splunk queries without predefined fields

I am trying to join 2 splunk queries. However in this case the common string between the 2 queries is not a predefined splunk field and is logged in a different manner. I have created the regex which individually identifies the string but when I try to combine using join, I do not get the result.
I have logs like this -
Logline 1 -
21-04-2019 11:01:02.001 server1 app1 1023456789 1205265352567565 1234567Z-1234-1234-1234-123456789123 Application Completed
Logline 2 -
21-04-2019 11:00:00.000 journey_ends server1 app1 1035625855585989 .....(lots of text) commonID:1234567Z-1234-1234-1234-123456789123 .....(lots of text) status(value) OK
the second Logline can be NOTOK as well
Logline 2 -
21-04-2019 11:00:00.000 journey_ends server1 app1 1035625855585989 .....(lots of text) commonID:1234567Z-1234-1234-1234-123456789123 .....(lots of text) status(value) NOTOK
I have tried multiple things but the best that I can come up with is -
index=test "journey_ends" | rex "status(value) (?<StatusType>[A-Z][A-Z]*)" | rex "commonID\:(?<commonID>[^\t]{37})" | table StatusType, commonID | join type=inner commonID [ search index=test "Application Completed" | rex "^(?:[^\t\n]*\t){7}(?P<commonID>[^\t]+)" | table _time, commonID] | chart count over StatusType by commonID
However the above query does not provide me the stats. In verbose mode, I can just see the events of query 1. Please note that the above 2 queries run correctly individually.
However currently I have to initially run the query to fetch the commonIDs from "Application Completed" logline and then in another query give the list of commonIDs found in the result first query as input and find the status value for each commonId from logline 2.
Expected Result (in a table):
StatusType commonID OK 1234567Z-1234-1234-1234-123456789123 NOTOK 1234567Z-1234-1234-1234-985625623541
Can you try the below query,
index=main
AND "Application Completed"
| rex "(?<common_id>[[:alnum:]]+-[[:alnum:]]+-[[:alnum:]]+-[[:alnum:]]+-[[:alnum:]]+)"
| table _time, common_id
| join type=inner common_id [
search index=main
| rex "status\(value\)\s+(?<status>.+)$"
| rex "(?<common_id>[[:alnum:]]+-[[:alnum:]]+-[[:alnum:]]+-[[:alnum:]]+-[[:alnum:]]+)"
| table status, common_id
]

How to use rex command to extract two fields and chart the count for both in one search query?

I have a log statement like 2017-06-21 12:53:48,426 INFO transaction.TransactionManager.Info:181 -{"message":{"TransactionStatus":true,"TransactioName":"removeLockedUser-1498029828160"}} .
How can i extract TransactionName and TranscationStatus and print in table form TransactionName and its count.
I tried below query but didn't get any success. It is always giving me 0.
sourcetype=10.240.204.69 "TransactionStatus" | rex field=_raw ".TransactionStatus (?.)" |stats count((status=true)) as success_count
Solved it with this :
| makeresults
| eval _raw="2017-06-21 12:53:48,426 INFO transaction.TransactionManager.Info:181 -{\"message\":{\"TransactionStatus\":true,\"TransactioName\":\"removeLockedUser-1498029828160\"}}"
| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"
| rex "{\"TransactionStatus\":(?[^,]),\"TransactioName\":\"(?[^\"])\""
| chart count OVER TransactioName BY TransactionStatus