KQL Remove from array if array item contains equals condition - kql

Question

I have a log source which has the following format:
//file1.png is the filename, the parenthesis hold the filesize in bytes.
// Each item is separated by a semicolon.
file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);
I have this regex to split the items into a list of lists, with 0 being the filename, 1 the extension and 2 the size in bytes. This can be copied as a test:
datatable (item:string ) [
'file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);',
'file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);'
]
| extend list = extract_all(#'([^<>:"\/\\\|\?\*;]+)\.([a-zA-z0-9]+) \((\d+)b\);', item)
This outputs as the following:
[["file1", ".png", "343"], ["file2", ".pdf", "232"],...]
I want to remove items from the list if the sublist contains certain filetypes (such as .png). How would I go about doing this? KQL has no iteration features as far as I know.
I have tried using regex to exclude the queries I do not want to match. It was the following: (?:^|; =?)([^<>:"\/\\\|\?\*;]+\.(?!jpg\b|png\b)\w+ \(\d+b\))(?=;|$). Unfortunately, KQL does not support negative lookaheads.

Answer 1

mv-apply operator
datatable (item:string ) [
'file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);',
'file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);'
]
| mv-apply f = extract_all(#'([^<>:"\/\\\|\?\*;]+)\.([a-zA-z0-9]+) \((\d+)b\);', item) on
(
where f[1] !in~ ("png", "jpg")
| summarize make_list(pack_array(f))
)
item
list_
file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);
[["file2","pdf","2345","b"],["file4","docx","3243","b"]]
file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);
[["file1","src","3243","b"],["file2","ps2","24","b"]]
Fiddle

Answer 2

replace_regex()
datatable (item:string ) [
'file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);',
'file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);'
]
| extend clean_item = replace_regex(item, #'([^<>:"\/\\\|\?\*;]+)\.(?i:png|jpg) \((\d+)b\);', "")
| extend extract_all(#'([^<>:"\/\\\|\?\*;]+)\.([a-zA-z0-9]+) \((\d+)b\);', clean_item)
item
clean_item
Column1
file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);
file2.pdf (2345b); file4.docx (3243b);
[[" file2","pdf","2345"],[" file4","docx","3243"]]
file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);
file1.src (3243b); file2.ps2 (24b);
[["file1","src","3243"],[" file2","ps2","24"]]
Fiddle

Answer 3

And another option that doesn't rely on (arguably) hard to read regular expressions:
datatable (item:string ) [
'file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);',
'file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);'
]
| extend s=split(item, "; ")
| mv-expand s
| parse s with fileName "." extension " (" fileSize "b)" rest
| where extension !in ("png", "jpg")
| extend image = bag_pack("fileName", fileName, "extension", extension, "fileSizeBytes", fileSize)
| summarize make_list(image) by item
item
list_image
file1.png (445b); file2.pdf (2345b); file3.jpg (343b); file4.docx (3243b);
[ { "fileName": "file2", "extension": "pdf", "fileSizeBytes": "2345" }, { "fileName": "file4", "extension": "docx", "fileSizeBytes": "3243" } ]
file1.src (3243b); file2.ps2 (24b); file3.jpg (300b); file4.jpg (326b);
[ { "fileName": "file1", "extension": "src", "fileSizeBytes": "3243" }, { "fileName": "file2", "extension": "ps2", "fileSizeBytes": "24" } ]
According to the original regular expression ; isn't part of the filename which means simply splitting on ; should be safe.