I have results like below:
1. DateTime=2019-07-02T16:17:20,913 Thread=[], Message=[Message(userId=124, timestamp=2019-07-02T16:17:10.859Z, notificationType=CREATE, userAccount=UserAccount(firstName=S, lastName=K, emailAddress=abc#xyz.com, status=ACTIVE), originalValues=OriginalValue(emailAddress=null)) Toggle : true]
2. DateTime=2019-07-02T16:18:20,913 Thread=[], Message=[Message(userId=124, timestamp=2019-07-02T16:17:10.859Z, notificationType=CREATE, userAccount=UserAccount(firstName=S, lastName=K, emailAddress=abc#xyz.com, status=ACTIVE), originalValues=OriginalValue(emailAddress=new#xyz.com)) Toggle : true]
3. DateTime=2019-07-02T16:19:20,913 Thread=[], Message=[Message(userId=124, timestamp=2019-07-02T16:17:10.859Z, notificationType=CREATE, userAccount=UserAccount(firstName=S, lastName=K, emailAddress=abc#xyz.com, status=ACTIVE), originalValues=OriginalValue(emailAddress=new#xyz.com)) Toggle : true]
And I am trying to group results where the contents of the entire "Message" field is same and "emailAddress=null" is not contained in the Message.
So in the results above 2 and 3 should be the output.
The following query works fine for me but I need to optimize it further according to the following conditions:
Working Query: index=app sourcetype=appname host=appname* splunk_server_group=us-east-2 | fields Message | search Message= "[Message*" | regex _raw!="emailAddress=null" | stats count(Message) as count by Message | where count > 1
Conditions to optimize
Cannot rex against raw
Message key/value pair needs to be in the main search, not a sub-search
You don't have any subsearches in your current query. A subsearch is a query surrounded by square brackets.
What's wrong with rex against _raw?
Try this:
index=app sourcetype=appname host=appname* splunk_server_group=us-east-2 Message="[Message*"
| fields Message
| regex Message!="emailAddress=null"
| stats count(Message) as count by Message | where count > 1
Related
I have multiple log messages each containing a list of JobIds -
IE -
1. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890039","db7a18ae-ea59-4987-87d5-c80adefa4475"]}`
2. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890040","db7a18ae-ea59-4987-87d5-c80adefa4489"]}`
3. `{"JobIds":["661ce07c-b5f3-4b37-8b4c-a0b76d890070"]}`
I have a rex to get those jobIds. Next I want to count the number of jobIds
My query looks like this -
| rex field=message "\"(?<job_ids>(?:\w+-\w+-\w+-\w+-\w+)+),?\""
| stats count(job_ids)
But this will only give me a count of 3 when I am looking for 5. How can I get a count of all jobIds? I am not sure if this is a splunk limitation or I am missing something in my regex.
Here is my regex - https://regex101.com/r/vqlq5j/1
Also with max-match=0 but with mvcount() instead of mvexpand():
| makeresults count=3 | streamstats count
| eval message=case(count=1, "{\"JobIds\":[\"a1a2a2-b23-b34-d4d4d4\", \"x1a2a2-y23-y34-z4z4z4\"]}", count=2, "{\"JobIds\":[\"a1a9a9-b93-b04-d4d4d4\", \"x1a9a9-y93-y34-z4z4z4\"]}", count=3, "{\"JobIds\":[\"a1a9a9-b93-b04-d14d14d14\"]}")
``` above is test data setup ```
``` below is the actual query ```
| rex field=message max_match=0 "\"(?<id>[\w\d]+\-[\w\d]+\-[\w\d]+\-[\w\d]+\")"
| eval cnt=mvcount(id)
| stats sum(cnt)
In Splunk, to capture multiple matches from a single event, you need to add max_match=0 to your rex, per docs.Splunk
But to get them then separated into a singlevalue field from the [potential] multivalue field job_ids that you made, you need to mvxepand or similar
So this should get you closer:
| rex field=message max_match=0 "\"(?<job_id>(?:\w+-\w+-\w+-\w+-\w+)+),?\""
| mvexpand job_id
| stats dc(job_id)
I also changed from count to dc, as it seems you're looking for a unique count of job IDs, and not just a count of how many in total you've seen
Note: if this is JSON data (and not JSON-inside-JSON) coming into Splunk, and the sourcetype is configured correctly, you shouldn't have to manually extract the multivalue field, as Splunk will do it automatically
Do you have a full set of sample data (a few entire events) you can share?
Say I have a query such as
index="example" source="example.log" host="example" "ERROR 1234"
| stats distinct_count by id
This will give me all the events with that error code per id.
I then want to combine this query to search the same log file for another string but only on the unique id's returned from the first search. Because the new string will appear on a separate event I can't just do an 'AND'.
There are a few ways to do that, including using subsearches, join, or append, but those require multiple passes through the data. Here is a way that makes a single pass through the index.
index=example source="example.log" ("ERROR 1234" OR "ERROR 5678")
``` Check for the presence of each string in the event ```
| eval string1=if(searchmatch("ERROR 1234"), 1, 0)
| eval string2=if(searchmatch("ERROR 5678"), 1, 0)
``` Count string occurrences by id ```
| stats sum(string1) as string1, sum(string2) as string2 by id
``` Keep only the ids that have both strings ```
| where (string1 > 0 AND string2 > 0)
You can search for "some other string" in subsearch and then join the queries on the id:
index="example" source="example.log" host="example" "ERROR 1234"
| join id [search index="example" source="example.log" host="example" "some other string" ]
| stats distinct_count by id
Presuming your id field is the same and available in both indices, this form should work:
(index=ndxA sourcetype=srctpA id=* source=example.log host=example "ERROR 1234") OR (index=ndxB sourcetype=srctpB id=* "some other string")
| rex field=_raw "(?<first_field>ERROR 1234)"
| rex field=_raw "(?<second_field>some other string)"
| fillnull value="-" first_field second_field
| stats count by id first_string second_string
| search NOT (first_string="-" OR second_string="-")
If your id field has a different name in the other index, do a rename like this before the stats line:
| rename otherIdFieldName as id
Advantages of this format:
you are not limited by subsearch constraints (search must finish in 60 seconds, no more than 50k rows)
the Search Peers (ie Indexers) will handle all of the overhead instead of having to wait on the Search Head that initiated the search to do lots of post-processing (all the SH is doing is sending the distributed search, then a post-stats filter to ensure both first_string and second_string have the values you are looking for)
My splunk result looks like this:
9/1/20
5:00:14.487 PM
2020-09-01 16:00:14.487, 'TOTALITEMS'="Number of items registered in the last 2 hours ", COUNT(*)="1339"
I am trying to table the number that appears in the end in quotes.
index=my_db sourcetype=no_of_items_registered source=P_No_of_items_registered_2hours | rex field=_raw "\"Number of items registered in the last 2 hours \", COUNT(\*)=\"(?P<itm_ct>\d+)\"$" | table itm_ct
This displays a blank table without any numbers. The number of rows in the table however matches the the number of events.
Any help much appreciated
The regular expression doesn't match the sample data. Literal parentheses must be escaped in the regex. Try this:
index=my_db sourcetype=no_of_items_registered source=P_No_of_items_registered_2hours
| rex "COUNT\(\*\)="(?<itm_ct>\d+)" | table itm_c
I have 3 fields in my splunk result like message, id and docId.
Need to group the results by id and doc id which has specific messages
message="successfully added" id=1234 docId =1345
message="removed someUniqueId" id=1234 docId =1345
I have to group based on the results by both id's which has the specific message
search query | rex "message=(?<message[\S\s]*>)" | where message="successfully added"
which is giving result for the first search, when i tried to search for second search query which is not giving result due to the someUniqueId"
search query | rex "message=(?<message[\S\s]*>)" | where match(message, "removed *")
Could you pelase help me to filter the results which has the 2 messages and group by id and docID
The match function expects a regular expression, not a pattern, as the second argument. Try search query | rex "message=(?<message>[\S\s]*)" | where match(message, "removed .*").
BTW, the regex strings in the rex commands are invalid, but that may be a typing error in the question.
I have a log statement like 2017-06-21 12:53:48,426 INFO transaction.TransactionManager.Info:181 -{"message":{"TransactionStatus":true,"TransactioName":"removeLockedUser-1498029828160"}} .
How can i extract TransactionName and TranscationStatus and print in table form TransactionName and its count.
I tried below query but didn't get any success. It is always giving me 0.
sourcetype=10.240.204.69 "TransactionStatus" | rex field=_raw ".TransactionStatus (?.)" |stats count((status=true)) as success_count
Solved it with this :
| makeresults
| eval _raw="2017-06-21 12:53:48,426 INFO transaction.TransactionManager.Info:181 -{\"message\":{\"TransactionStatus\":true,\"TransactioName\":\"removeLockedUser-1498029828160\"}}"
| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"
| rex "{\"TransactionStatus\":(?[^,]),\"TransactioName\":\"(?[^\"])\""
| chart count OVER TransactioName BY TransactionStatus