Cloudwatch Logs Insights working with multiple #messages - amazon-cloudwatch

Question

I have the following query with the following output:
Query:
filter #message like /A:|B:/
Output:
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
[INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':0}
I would like to print ONLY the A message when in the B message 'MyValue' = 0. For the above example, I would have to have the following output
Output:
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
For the next example
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
[INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':12}
The output should be empty
I can't do something like this because I miss the A message:
filter #message like /A:|B:/
filter MyValue = 0
Any ideas?

Answer 1

If anyone still interested, there IS ways to get the first and last from grouping by a field. So if you can fit your data into pairs of messages, it might help.
For example, given API Gateway access log (each row is a #message):
2021-09-14T14:09:00.452+03:00 (01c53288-5d25-*******) Extended Request Id: ***************
2021-09-14T14:09:00.452+03:00 (01c53288-5d25-*******) Verifying Usage Plan for request: 01c53288-5d25-*******. API Key: API Stage: **************/dev
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) API Key authorized because method 'ANY /path/{proxy+}' does not require API Key. Request will not contribute to throttle or quota limits
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) Usage Plan check succeeded for API Key and API Stage **************/dev
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) Starting execution for request: 01c53288-5d25-*******
2021-09-14T14:09:00.454+03:00 (01c53288-5d25-*******) HTTP Method: GET, Resource Path: /path/json.json
2021-09-14T14:09:00.468+03:00 (01c53288-5d25-*******) Method completed with status: 304
We can get method, uri and return code from the last 2 rows.
To do this, I parse the relevant data into params, and then get them by doing aggregation by request id (that i also parse)
The magic is: using stats likesortsFirst() and sortsLast() and grouping by #reqid. (AWS Docs
Note: IMO, don't use earliest() and latest() as they depend on built-in #timestamp and worked weird for me where 2 sequential messages had the same timestamp
So, for example, using this query:
filter #message like "Method"
| parse #message /\((?<#reqid>.*?)\) (.*?) (Method: (?<#method>.*?), )?(.*?:)* (?<#data>[^\ ]*)/
| sort #timestamp desc
| stats sortsFirst(#method) as #reqMethod, sortsFirst(#data) as #reqPath, sortsLast(#data) as #reqCode by #reqid
| limit 20
We would get the following desired output:
#reqid #reqMethod #reqPath #reqCode
f42e2b44-b858-45cb-***************** GET /path-******.json 304
fecddb03-3804-4ff5-***************** OPTIONS /path-******.json 200
e8e47185-6280-4e1e-***************** GET /path-******.json 304
e4fa9a0c-6d75-4e26-***************** GET /path-******.json 304