How to use Regex in kusto query - azure-log-analytics

So I have a Kusto query like this
exceptions
| extend A_= tostring(customDimensions.A)
| where A_ in~ ("Could not get notes: From:", "failed to call", "Custom conference list" )
// | where A_ contains 'Could not get notes: From:' //This is working when I use "contains" operator but fails to check below 2 items when the first item doesn't exist
// | where A_ contains 'Custom conference list'
// | where A_ contains 'failed to call'
The first item in the list is "Could not get notes: From:", where it has multiple ID at the end of the string
For example;
Could not get notes: From: abcd
Could not get notes: From: abcdef
What I am trying to do is take all the items that start with "Could not get notes: From:" and use them in the "in~" operator.
So far I have tried using contains operator like below
|where A_ contains 'Could not get notes: From:' // This seems to be working as it will outputting every item that starts with "Could not get notes: From:" but when I am trying to use it in the "in~" it is failing.
The problem with using the "contains" operator for each unique item is if any of the items(let's say for 3 items) will not there, the query returns no results even if 2 items still exist.
Not sure if this has a better solution to it.

You should use has_any instead:
exceptions
| extend A_= tostring(customDimensions.A)
| where A_ has_any ("Could not get notes: From:", "failed to call", "Custom conference list")
Also, note that contains is significantly slower than has/has_any, because the latter uses the index to only fetch relevant records, while contains scans all the records. Note, however, that there's a semantic difference between the two: contains looks for substrings, while has only looks for full tokens. For example, "hello world" contains "hell" will return true, while "hello world" has "hell" will return false. See more info in the String Operators doc.

Related

How to correctly use in for string list in KQL

the following code works
let names = dynamic(['Windows Installer', 'Software Protection']);
ConfigurationChange
| where Computer like "SRV"
| where SvcPreviousState == "Running"
| where SvcState == "Stopped"
// | where SvcDisplayName in (names)
| order by TimeGenerated
as commented out I would like to only check for a list of SvcDisplayName's.
According to the documentation this should work but does complain
: Failed to resolve table or column or scalar expression named 'names'
How would I correctly use in with a list for SvcDisplayName ?
The blank line is considered separator between queries, unless you select the whole code for execution.
See screenshots below.
Select the whole code for execution.
=> Valid query
Put the cursor on the query for execution.
There is no blank line after the let statement.
=> Valid query
Put the cursor on the query for execution.
There is a blank line after the let statement.
=> Invalid query
Please note how the query is marked by a pale blue color, but not the let statement

Asserting Against Large Dynamic Response

I have a very large response array I want to assert against, but without knowing the order. I have a variable with the expected response values so I can do a single giant comparison, but I'm unable to load the entire response and compare it with the entire expected response variable at the same time.
* def obligationsQuery = Java.type("tests.account.sql.Obligations").getObligations(division, account)
* def getObligations = db.readRows(obligationsQuery)
Given path "account", "v1", "accounts", systemId, "obligations"
And header api-key = gatewayKey
When method GET
Then status 200
And match $.data != null
And match $.data[*].transactionType contains any "<transactionTypeResponse>"
And match $.data[*] contains only getObligations
Examples:
| description | transactionType | transactionTypeResponse |
| Invoice | 001 | invoice
The error I get is:
get_obligations_collection.feature:49 - path: $.data[*][*], actual: [{"object1"}, {"object2"}, {"etc"}], expected: {"object1"}, reason: actual value does not contain expected
I've also tried:
And match each $.data[*] contains only getObligations
But then I get:
get_obligations_collection.feature:49 - path: $[0], actual: [{"object1"}, expected: [{"object1"}, {"object2"}, {"etc"}, reason: actual value is not list-like
I assume $.data is a JSON array so no need to use json-path to again get the data into another array by calling as $.data[*].
so,
And match $.data contains only getObligations
should work.
If this still not working, please provide some proper response and getObligations values to investigate further.
I assumed that contains only would show the complete value of my variable, but smartly, it only shows the object that fails to match the api response object. Was able to verify that through a simpler assert, and then checking the error message saw that an ID was missing padding that the API adds, but the DB does not.
As usual, just looking more closely at the data returned provides a simple explanation.

Log Analtyics - How to use "inverted commas" within search query

I am trying to create a search query for when a Public IP is assigned to a NIC, and then create an alert off that. I can find the part which identifies the assignment, but I need to use "inverted commas" within my search, but I can't...
My query:
AzureActivity
| where OperationName == "Microsoft.Network/networkInterfaces/write" and ActivityStatus == "Started"
| where Properties contains "<>"
Within that "contains", I need to use the following JSON pulled from the properties JSON (which I found doing a search without Properties Contains):
\"provisioningState\":\"Succeeded"\
However, I know I can't use "inverted commas" within an already inverted comma area. Is there a way to allow me to put that inside, perhaps with some sort of cancelling or bracketing?
You can use # for escaping - see here:
https://docs.loganalytics.io/docs/Language-Reference/Data-types/string
or, possibly better yet, you can use extractjson (or parsejson) functions
https://docs.loganalytics.io/docs/Language-Reference/Scalar-functions/extractjson()
I have found my solution, thanks to the links submitted by #Oleg Ananiev.
AzureActivity
| sort by TimeGenerated desc nulls last
| where OperationName == "Microsoft.Network/networkInterfaces/write" and ActivityStatus == "Started"
| where Properties contains '\\"provisioningState\\":\\"Succeeded\\"'
A better way to read to the nested property in JSON format using parse_json. For example if you would like to read to provisioningState property's value, simple do the following query
| where parse_json(Properties).provisioningState == 'Succeeded'
Please let me know if that helps!

BigQuery Java API to read an Array of Record : "Retrieving field value by name is not supported" exception

My current table in BigQuery has a column that uses complex types. The "family" column is actually a list ("repeated" feature) of records (with 2 fields: id & name).
When I try to get the 1st "id" value of 1 row with the following syntax:
FieldValueList c = qr.getValues().iterator().next();
c.get("family").getRepeatedValue().get(0).getRecordValue().get("id");
I get the exception:
Method threw 'java.lang.UnsupportedOperationException' exception.
Retrieving field value by name is not supported when there is no fields schema provided
This is a bit annoying because my table has a clearly defined schema. And when I do the "read" query with the same Java call, I can also see that this schema is correctly found:
qr.getSchema().getFields().get("family").getSubFields().toString();
-->
[Field{name=id, type=INTEGER, mode=NULLABLE, description=null}, Field{name=name, type=STRING, mode=NULLABLE, description=null}]
Due to this exception, the workaround that I have found is to pass the "index" of the record field instead of giving it its name
c.get("family").getRepeatedValue().get(0).getRecordValue().get(0).getLongValue();
However, this seeks awkward to pass an index instead of a name.
Is there a better way to get the value of a field in a record inside an array (if my column is only a record, without array, then I don't get the exception) ?
Is this exception normal?
You can wrap the unnamed FieldValueList with a named one using the "of" static method:
FieldList subSchema = qr.getSchema().getFields().get("family").getSubFields();
FieldValueList c = qr.getValues().iterator().next();
FieldValueList.of(
c.get("family").getRepeatedValue().get(0).getRecordValue(),
subSchema).get("id");
The "of" method takes a FieldValueList (returned by getRecordValue() in this case) and a FieldList (subSchema here), and returns the same FieldValueList but with named access.

Splunk: Extracting multiple fields with the same name

I am using Splunk to index logs with multiple fields with the same name. All fields have the same meaning:
2012-02-22 13:10:00,ip=127.0.0.1,to=email1#example.com,to=email2#example.com
In the automatic extraction for this event, I only get "email1#example.com" extracted for the "to" field. How can I make sure all the values are extracted?
Thanks!
I think adding this to the end of the search this may do it:
| extract pairdelim="," kvdelim="=" mv_add=t | table to
(the 'table' is just for demonstration).
So, I think, in 'transforms.conf' (from http://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf) put:
[my-to-extraction]
DELIMS = ",", "="
MV_ADD = true
and reference it in 'props.conf':
[eventtype::my_custom_eventtype]
REPORT-to = my-to-extraction
where 'eventtype::my_custom_eventtype' could be anything that works as a 'props.conf' specification (<spec> in http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf).