Multisearch not doing what I expect - splunk

Question

The message format we chose uses a field called scope to control the level of aggregation you want (by request_type, site, zone, cluster). The scope is set with a dropdown and passed in as a token. I wanted to use multi-search to coalesce the results of 4 different searches. So that if the scope was site, only the results from the site search would be shown.
Actual Search:
index=cloud_aws namespace=cloudship lambda=SCScloudshipStepFunctionStats metric_type=*_v0.3
| spath input=message
| multisearch
[search $request_type_token$ | where "$scope_token$" == "request_type" ]
[search $request_type_token$ $site_token$ | where "$scope_token$" == "site"]
[search $request_type_token$ $site_token$ $zone_token$ | where "$scope_token$" == "zone"]
[search scope=$scope_token$ $request_type_token$ $site_token$ $zone_token$ $cluster_token$ | where "$scope_token$" == "cluster"]
| timechart cont=FALSE span=$span_token$ sum(success) by request_type
Search after token substitution with literal values.
index=cloud_aws namespace=cloudship lambda=SCScloudshipStepFunctionStats metric_type=*_v0.3
| spath input=message
| multisearch
[search request_type="*"
| where "site" == "request_type" ]
[search request_type="*" site="RTP"
| where "site" == "site"]
[search request_type="*" site="RTP" zone="*"
| where "site" == "zone"]
[search scope=site request_type="*" site="RTP" zone="*" cluster="*"
| where "site" == "cluster"]
| timechart cont=FALSE span=hour sum(success) by request_type
BUT ... the results of this query are equivalent to no search at all and I basically do not filter anything.
index=cloud_aws namespace=cloudship lambda=SCScloudshipStepFunctionStats metric_type=*_v0.3
| spath input=message
| timechart cont=FALSE span=hour sum(success) by request_type
This query and the one above give the same result. What am I missing here? When I execute each part of the multi-search separately, the results are correct. I get empty results for all but the 'where "site" == "site"' search. But when I run the whole query I get no filtering at all. Help!

Answer 1

First, I think what you're looking for is the value of site to match request_type (in the initial multisearch search line) - but what you're actually checking for in the where clause is whether the text "site" equals the text "request_type". And, of course, that is not the case!
Start by removing the second line of the multisearch (since comparing site to site will always be true), and using upper() and match():
index=cloud_aws namespace=cloudship lambda=SCScloudshipStepFunctionStats metric_type=*_v0.3
| spath input=message
| multisearch
[search request_type="*" site=*
| eval request_type=upper(request_type), site=upper(site)
| where "site" == "request_type" ]
[search request_type="*" site="RTP" zone="*"
| eval zone=upper(zone), site=upper(site)
| where match(site,zone)]
[search scope=site request_type="*" site="RTP" zone="*" cluster="*"
it would be even easier to do cluster="rtp" instead of cluster=* here, but I've left the idiom of upper()ing and match()ing for reading consistency
| where match(site,cluster)]
| timechart cont=FALSE span=hour sum(success) by request_type