Query to loop through data in splunk - splunk

Question

I've below lines in my log:
...useremail=abc#fdsf.com id=1234 ....
...useremail=pqr#fdsf.com id=4565 ....
...useremail=xyz#fdsf.com id=5773 ....
Capture all those userids for the period from -1d#d to #d
For each user, search from beginning of index until -1d#d & see if the userid is already present by comparing actual id field
If it is not present, then add it into the counter
Display this final count.
Can I achieve this in Splunk?
Thanks!

Answer 1

Yes, there are several ways to do this in Splunk, each varying in degrees of ease and ability to scale. I'll step through the subsearch method:
1) Capture all those userids for the period from -1d#d to #d
You want to first validate a search that returns only a list of ids, which will then be turned into a subsearch:
sourcetype=<MY_SOURCETYPE> earliest=-1d#d latest=-#d | stats values(id) AS id
2) For each user, search from beginning of index until -1d#d & see if the userid is already present by comparing actual id field
Construct a main search with a different timeframe that using the subsearch from (1) to match against those ids (note that the subsearch must start with search):
sourcetype=<MY_SOURCETYPE> [search sourcetype=<MY_SOURCETYPE> earliest=-1d#d latest=-#d | stats values(id) AS id] earliest=0 latest=-1d#d
This will return a raw dataset of all events from the start of the index up to but not including 1d#d that contain the ids from (1).
3) If it is not present, then add it into the counter
Revise that search with a NOT against the entire subsearch and pipe the outer search to stats to see the ids it matched:
sourcetype=<MY_SOURCETYPE> NOT [search sourcetype=<MY_SOURCETYPE> earliest=-1d#d latest=-#d | stats values(id) AS id] earliest=0 latest=-1d#d | stats values(id)
4) Display this final count.
Revise the last stats command to return a distinct count number instead:
sourcetype=<MY_SOURCETYPE> NOT [search sourcetype=<MY_SOURCETYPE> earliest=-1d#d latest=-#d | stats values(id) AS id] earliest=0 latest=-1d#d | stats dc(id)
Performance considerations:
The above method works reasonably well for datasets under 1 million rows, on commodity hardware. The issue is that the subsearch is blocking, thus the outer search needs to wait. If you have larger datasets to deal with, then alternative methods need to be employed to make this an efficient search.
FYI, Splunk has a dedicated site where you can get answers to questions like this much faster: http://splunk-base.splunk.com/answers/