How to use a token for a rex in Splunk? - splunk

Question

I have a token $token_rex$ set up as follows in the dashboard:
<set>mvjoin(mvmap('token_keywords_mv',"(?<".'token_keywords_mv'.">".'token_keywords_mv'."+?)"), "|")</set>
token_keyrowrds_mv is basically the following:
lorem
ipsum
situs
The resulting token looks like this:
(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)
The query I am trying to run is as follows:
index=* | rex field _raw "(?i)".$token_rex$
Ideally the above should be rendered like so:
index=* | rex field _raw "(?i)(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)"
I have tried using the token filters $token_rex|s$ and $token_rex|n$ but neither work. I have even tried to return the value from a subsearch:
index=* | rex field _raw [| makeresults | eval string_rex=$token_rex$ | return $token_rex]
Update: Found out that the token is being given as verbose to the rex command. My token is set as follows:
<set token="token_rex">mvjoin(mvmap('token_keywords_mv',"(?>".'token_keywords_mv'."<".'token_keywords_mv'."+?)"), "|")</set>
When I used it as ... | rex field=_raw '(?i)$token_rex$' it gives me the following error:
Error in 'rex' command: Encountered the following error while compiling the regex ''(?i)mvjoin(mvmap('token_keywords_mv'': Regex: missing closing parenthesis.
When I set it as the value directly, however, it works:
<set token="token_rex">(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)</set>
Update: Here's a sample dashboard.
<dashboard theme="dark">
<init>
<set token="token_keywords_mv">split("lorem,ipsum,situs", ",")</set>
<set token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</set>
<set token="token_rex">"(?i)".mvjoin(mvmap('token_keywords_mv', "(?<".'token_keywords_mv'.">".'token_keywords_mv'."+?)"), "|")</set>
<set token="token_raw">(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)</set>
</init>
<label>Test Search</label>
<description>Multivalue</description>
<row>
<panel>
<table>
<search>
<query>index=*
[| makeresults
| eval string_search=$token_keywords_starred$
| return $string_search]
| rex field=_raw $token_raw|s$
| stats count(lorem) AS Lorem, count(ipsum) AS Ipsum, count(situs) AS Situs
| eval header="Count"
| transpose column_name="String" header_field=header</query>
<earliest>-24h#h</earliest>
<latest>now</latest>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</table>
</panel>
</row>
<row>
<panel>
<table>
<title>ECHO</title>
<search>
<query>| localop
| makeresults
| eval token_keywords_starred=$token_keywords_starred$, token_keywords_mv=$token_keywords_mv$, token_rex=$token_rex$, token_raw=$token_raw|s$</query>
<earliest>-24h#h</earliest>
<latest>now</latest>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</table>
</panel>
</row>
</dashboard>
If any other token replaces $token_raw|s$, there will be an error. Using Splunk 8.0.3.

Answer 1

When I try to reproduce your results with this dashboard code:
<form>
<label>test</label>
<fieldset submitButton="false">
<input type="text" token="token_rex">
<label>field1</label>
</input>
</fieldset>
<row>
<panel>
<event>
<title>test</title>
<search>
<query>| makeresults | eval _raw="lorem ipsum dolor"
| rex "(?i)$token_rex$"
| table lorem ipsum situs</query>
<earliest>$earliest$</earliest>
<latest>$latest$</latest>
</search>
<option name="refresh.display">progressbar</option>
</event>
</panel>
</row>
</form>
I get this optimized search (as reported by Job Inspector):
| makeresults | rex field=_raw "(?i)(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)"
It looks like the concatenation is not needed.
Update: Change all the set tags to eval and use mvzip instead of mvmap. The set tag does not compute, while the eval tag does.
<eval token="token_keywords_mv">split("lorem,ipsum,situs", ",")</eval>
<eval token="token_keywords_starred">"*".mvjoin($token_keywords_mv$, "* OR *")."*"</eval>
<eval token="token_rex">"(<".mvjoin(mvzip('token_keywords_mv','token_keywords_mv',">"),"+?)|(?<")."+?)"</eval>
<set token="token_raw">(?<lorem>lorem+?)|(?<ipsum>ipsum+?)|(?<situs>situs+?)</set>