Splunk - Lookup values + static search string = output with count - splunk

I want to perform a search where I need to use a static search string + input from a csv file with usernames:
Search query-
index=someindex host=host*p* "STATIC_SEARCH_STRING"
Value from users.csv where the list is like this- Please note that User/UserList is NOT a field in my Splunk:
**UserList**
User1
User2
User3
.
.
UserN
I have tried using multiple one of them being-
| inputlookup users.csv | join [search index=someindex host=host*p* "STATIC_SEARCH_STRING"] | lookup users.csv UserList OUTPUT UserList as User| stats count by User
The above one just outputs the list of users with count as '1' - which I assume it is getting from the table itself.
When I try searching events for a single user like-
index=someindex host=host*p* "User1" "STATIC_SEARCH_STRING". I get 100's of events for that user.
Can someone please help me with this?
Sorry if this is a noob question, I have been trying to learn splunk in order to reduce my workload and am stuck here.
Thanks in advance!

index=someindex host=host*p* "STATIC_SEARCH_STRING" [ | inputlookup users.csv | fields UserList | rename UserList as query]
What is happening here is that there is a sub-search, which does an inputlookup on the users.csv file. We then use fields to ensure there is only a single field (UserList) in the data. We then rename that field to query. This is a special field in sub-searches; when the sub-search returns the field query, it is expanded out into the expression (field_value_1) OR (field_value_2) OR ....
This expression is then appended to the original search string, so the final search that Splunk executes is index=someindex host=host*p* "STATIC_SEARCH_STRING" ("alice") OR ("bob") OR ("charlie")
This approach is outlined at https://docs.splunk.com/Documentation/Splunk/8.0.3/Search/Changetheformatofsubsearchresults
You can also look at the Splunk format command, https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Format if you need to alter the sub-search's expression format, for example, adding * around each returned expression.

I think you're doing the search inside out
What I think you may want is the following:
index=ndx sourcetype=srctp host=host*p* User=*
| search
[| inputlookup users.csv ]
| stats count by User
If I understand your question correctly, you want to use the values in your lookup as a filter on the data (ie, only where User is in that list)
If that is the case, the above will do just that
If you need to make the fieldnames match because the lookup table has a different name, change the subsearch to the following:
[| inputlookup users.csv
| rename lookup_field_name as User ]

Related

Splunk - I want to add a value from stats count() to a value from a lookup table and show that value in a table

The objective of the query im trying to write is to take a count of raw data from the previous month and add that to a count from a lookup table (.csv)
What I have attempted to do is…
index=*** source=***
| stats count(_raw) as monthCount
| join
[ | inputlookup Log_Count_YTD.csv]
| eval countYTD = toNumber(monthCount) + toNumber(TOTAL_COUNT_YTD)
| table countYTD
This query doesn’t return any value on a table. The TOTAL_COUNT_YTD is the only field from the inputlookup file. Let me know if there is any other information you need to help me out with this one. Thanks!
The stats command transforms the data so it has only 1 field: monthCount. The inputlookup returns only the TOTAL_COUNT_YTD field. The join command works by comparing values of common fields between the main search and the subsearch. Since there are no common fields no events are joined.
There is no need for join in this case. The appendcols command will do, assuming the CSV contains a single field in a single row.
index=*** source=***
| stats count() as monthCount
| appendcols
[ | inputlookup Log_Count_YTD.csv]
| eval countYTD = toNumber(monthCount) + toNumber(TOTAL_COUNT_YTD)
| table countYTD
FWIW, the tonumber function is unnecessary, but doesn't hurt.

Filtering out holidays in Splunk

I am attempting to use a search lookup table csv to filter out holidays for some Splunk queries.
To do this, I created a holidays.csv in the following style:
dateof,dateafter,description
01/17/2022,01/18/2022,MLK Day 22
02/21/2022,02/22/2022,Presidents Day 22
05/30/2022,05/31/2022,Memorial Day 22
[...]
Some of the queries run the day after the holiday, which is why I created a dateof and datefater field. I am trying to pipe the condition onto the end of the existing queries.
environment=staging "message=\"This line would contain the original search query"
| eval date=strftime(_time,"%m/%d/%Y")
| search NOT [ inputlookup holidays.csv | fields dateof ]
Note that an Event from the original query will look something like this:
time=2022-08-31T12:01:39,495Z [...] message="This line would contain the original search query"
Despite giving read access to the csv, the above condition does not filter out anything, regardless of whether it is a holiday listed in the csv file or not.
I suspect something is missing. However, I have a limited knowledge of the Spunk querying language. Would anyone be able to give guidance on this? Thanks in advance!
Subsearches can be tricky. If the result of the subsearch isn't just right, the query will fail. It helps to run the subsearch by itself to confirm the string produced makes sense as part of a query. In this case, check that
| inputlookup holidays.csv | fields dateof | rename dateof as date | format
produces something that works with search NOT.
An alternative to try is to explicitly look for the date field in the lookup.
environment=staging "message=\"This line would contain the original search query"
| eval date=strftime(_time,"%m/%d/%Y")
| where NOT [ | inputlookup holidays.csv | fields dateof | rename dateof as date | format ]
Here is another way to do it without a subsearch. A null description field tells us the date was not found in the lookup file and so is not a holiday.
environment=staging "message=\"This line would contain the original search query"
| eval date=strftime(_time,"%m/%d/%Y")
| lookup holidays.csv dateof as date output description
| where isnull(description)

Merge url with parameters into 1 in Splunk

I am creating a dashboard for our service. And I want to create metrics for url requests.
Lets say have a similar url like this one:
/api/v1/users/{userId}/settings
And I have following query in Splunk
url=*/api/v1/users/*/settings
| stats avg(timeTaken) as avg_latency, p99(timeTaken) as "p99(ms)", perc75(timeTaken) as "p75(ms)", count as total_requests, count(eval(responseStatus=500)) as failed_requests by url
| eval "success_rate"=round((total_requests - failed_requests)/total_requests*100,2)
| eval avg = round(avg)
| sort success_rate
All I want is to have a table with one common url showing all the metrics. But instead, I get a table with a list of all urls with different parameters.
You want to create a field which is the URL minus the UserId part, And therefore the stats will be grouped by which url is called.
You can do this by using split(url,"/") to make a mv field of the url, and take out the UserId by one of two ways depending on the URLs.
Mvfilter: Eg: mvfilter(eval(x!=userId))
Or created a new mvfield with the userId removed by it's index in the mvfield using this: Add/Edit/Delete mvfield
Instead of removing you could also choose to replace the UserId with "{userId}", so long as you do the same for all Urls.
And then you can rejoin the url using mvjoin(url,"/")
I hope I understood your question correctly and this helps you!
You could try doing a replace() on your URL field with eval before calling stats:
| eval url=replace(url,"\/\d+\/settings","/settings")
If it turns out the userid is important to hold onto, pull it into its own field prior to running replace():
| rex field=url "\/(?<userid>\d+)\/settings"
expansion for comment
For multiple possible endings of your URL, try something like this:
index=ndx sourcetype=srctp URL IN("*/api/v1/users/*/settings","*/api/v1/users/*/logout","*/api/v1/users/*/profile")
| rex field=url "\/(?<url_type>\w+)$"
| eval url=replace(url,"\/\d+\/\w+$","")
| stats avg(timeTaken) as avg_latency, p99(timeTaken) as "p99(ms)", perc75(timeTaken) as "p75(ms)", count as total_requests, count(eval(responseStatus=500)) as failed_requests by url type
| eval "success_rate"=round((total_requests - failed_requests)/total_requests*100,2)
| eval avg = round(avg)
| sort success_rate
This will extract the "type" (logout, profile, settings) into a new field, then cleanup the URL by removing everything from userid to the end

Splunk Count Specific String in a Field

In Splunk, I need to get the count of events from the below msg field value which matches factType=COMMERCIAL and has filters.
Using the basic Splunk query with wildcard does not work efficiently. Could you please assist
app_name="ABC" cf_space_name=prod msg="*/facts?factType=COMMERCIAL&sourceSystem=ADMIN&sourceOwner=ABC&filters*"
msg: abc.asia - [2021-08-23T00:27:08.152+0000] "GET
/facts?factType=COMMERCIAL&sourceSystem=ADMIN&sourceOwner=ABC&filters=%257B%2522stringMatchFilters%2522:%255B%257B%2522key%2522:%2522BFEESCE((json_data-%253E%253E'isNotSearchable')::boolean,%2520false)%2522,%2522value%2522:%2522false%2522,%2522operator%2522:%2522EQ%2522%257D%255D,%2522multiStringMatchFilters%2522:%255B%257B%2522key%2522:%2522json_data-%253E%253E'id'%2522,%2522values%2522:%255B%25224970111%2522%255D%257D%255D,%2522containmentFilters%2522:%255B%255D,%2522nestedMultiStringMatchFilter%2522:%255B%255D,%2522nestedStringMatchFilters%2522:%255B%255D%257D&sorts=%257B%2522sortOrders%2522:%255B%257B%2522key%2522:%2522id%2522,%2522order%2522:%2522DESC%2522%257D%255D%257D&pagination=null
Try this:
index=ndx sourcetype=srctp msg=*
| rex field=msg "factType=(?<facttype>\w+).(?<params>.+)"
| stats count by facttype params
| fields - count
| search facttype="commercial"
The rex will extract the facttype and any following parameters (note - if the URL is submitted with the arguments in a different order, you'll need to adjust the regular expression)
Then use a | stats count by to bin them together
Lastly, search only where there is both a facttype="commercial" and the URL has additional parameters

Splunk: find what a user is searching for?

I am trying to write a Splunk SPL query that will show me the most popular search terms that a user is looking for in one of my web apps. I have the logs already in Splunk but I am having a hard time extract the search parameter from the event. The event shows the full SQL select statement that looks like the query below:
select result from table where search_term = 'searched for this text'
How can I have this:
index=my_app search_term | top result
How do I actually capture the search term?
Thank you
You can use rex to extract the search term. Something like this
index=my_app | rex "search_term = '(?<search_term>[^']+)"
If you want individual words then use the split function followed by mvexpand to make each word a separate event.
... | eval words=split(search_term, " ") | mvexpand words