In Azure Log Analytics, I am trying to analyze events created by the Task Scheduler and group them by the executed task's name.
The basic query looks like
Event
| where Source == "Microsoft-Windows-TaskScheduler"
and TimeGenerated > ago(24h)
and EventLog == "Microsoft-Windows-TaskScheduler/Operational"
and EventID == 201
I am struggling with the task of grouping the results by the scheduled tasks' names as the names are contained in XML-encoded data in the ParameterXML and EventData attributes which have the following formats:
ParamterXML:
<Param>\MyScheduledTasksName</Param>
<Param>{1F1893C6-0696-430C-9738-50B068DDE37B}</Param>
<Param>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe</Param>
<Param>0</Param>
<Param>9684</Param>
EventData:
<DataItem type="System.XmlData" time="2019-06-13T11:18:45.4806563+02:00" sourceHealthServiceId="D4165670-2EBA-67E0-FF6B-1D838007CE5B">
<EventData Name="ActionSuccess" xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<Data Name="TaskName">\MyScheduledTasksName</Data>
<Data Name="TaskInstanceId">{1F1893C6-0696-430C-9738-50B068DDE37B}</Data>
<Data Name="ActionName">C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe</Data>
<Data Name="ResultCode">0</Data><Data Name="EnginePID">9684</Data>
</EventData>
</DataItem>
How would I go about parsing the XML task name so I could use it for grouping / summarizing in a pipe?
here's an example for using parse_xml(), combined with mv-apply to access only specific elements in the XML payload, e.g. TaskName in this case:
datatable(id:int, EventData:string)
[
1, '<DataItem type="System.XmlData" time="2019-06-13T11:18:45.4806563+02:00" sourceHealthServiceId="D4165670-2EBA-67E0-FF6B-1D838007CE5B">\n <EventData Name="ActionSuccess" xmlns="http://schemas.microsoft.com/win/2004/08/events/event">\n <Data Name="TaskName">\\MyScheduledTasksName</Data>\n <Data Name="TaskInstanceId">{1F1893C6-0696-430C-9738-50B068DDE37B}</Data>\n <Data Name="ActionName">C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe</Data>\n <Data Name="ResultCode">0</Data><Data Name="EnginePID">9684</Data>\n </EventData>\n</DataItem>',
2, '<DataItem type="System.XmlData" time="2019-06-13T11:18:45.4806563+02:00" sourceHealthServiceId="D4165670-2EBA-67E0-FF6B-1D838007CE5B">\n <EventData Name="ActionSuccess" xmlns="http://schemas.microsoft.com/win/2004/08/events/event">\n <Data Name="TaskName">\\MyOtherTask</Data>\n <Data Name="TaskInstanceId">{1F1893C6-0696-430C-9738-50B068DDE37B}</Data>\n <Data Name="ActionName">C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe</Data>\n <Data Name="ResultCode">0</Data><Data Name="EnginePID">9684</Data>\n </EventData>\n</DataItem>'
]
| mv-apply EventData = parse_xml(EventData).DataItem.EventData.Data on
(
where EventData['#Name'] == 'TaskName'
| project TaskName = EventData['#text']
)
The query above returns the following table:
| id | TaskName |
|----|-----------------------|
| 1 | \MyScheduledTasksName |
| 2 | \MyOtherTask |
It seems that you are looking for the parse_xml() function: https://learn.microsoft.com/en-us/azure/kusto/query/parse-xmlfunction
Related
I have
mytable consists of one XMLTYPE column called 'RS'. RS contains:
<test>
<mycol>
<name>a</name>
<number>1</number>
<number>2</number>
<number>50</number>
<number>60</number>
</mycol>
<mycol>
<name>b</name>
<number>5</number>
<number>820</number>
<number>601</number>
</mycol>
<mycol>
<name>c</name>
<number>6</number>
<number>8</number>
<number>62</number>
</mycol>
etc...
</test>
I'm looking to run a select statement that will display ALL names and up to 2 numbers from mytable.
something like this select statement but for all rows and without calling mycol[] several times.
select a.RS.extract('/test/mycol[1]/name[1]/text()').getstringval() as Names,
a.RS.extract('/test/mycol[1]/a[1]/text()').getstringval() ||''||
a.RS.extract('/test/mycol[1]/a[2]/text()').getstringval() ||''||
a.RS.extract('/test/mycol[1]/a[3]/text()').getstringval()
as num
from mytable a;/
output should be:
Names | num
a | 1 2
b | 5 820
c | 6 8
etc...
Thanks in advance.
Xml_table and string-join may be helpful
I'm running following Log Analytics Kusto query to get data what uses and thus generetes our Log Analytics cost
Usage
| where IsBillable == true
| summarize BillableDataGB = sum(Quantity) by Solution, DataType
| sort by Solution asc, DataType asc
and then the output is following:
What kinda query should I use if I want to deep dive more eg to ContainerInsights/InfrastructureInsights/ServiceMap/VMInsights/LogManagement so to get more detailed data what name or namespaces really cost?
Insightmetrics table have e.g these names and namespaces.
I was able maybe able to get something out using following query but something is still missing. Not totally sure if I'm on right or wrong way
union withsource = tt *
| where _IsBillable == true
| extend Namespace, Name
Here is the code for getting the name and namespace details. using Kusto query
let startTimestamp = ago(1h);
KubePodInventory
| where TimeGenerated > startTimestamp
| project ContainerID, PodName=Name, Namespace
| where PodName contains "name" and Namespace startswith "namespace"
| distinct ContainerID, PodName
| join
(
ContainerLog
| where TimeGenerated > startTimestamp
)
on ContainerID
// at this point before the next pipe, columns from both tables are available to be "projected". Due to both
// tables having a "Name" column, we assign an alias as PodName to one column which we actually want
| project TimeGenerated, PodName, LogEntry, LogEntrySource
| summarize by TimeGenerated, LogEntry
| order by TimeGenerated desc
For more information you can go through the Microsoft document and here is the Kust Query Tutorial.
I recreated the dashboard using the report query and have the search returning all of the table results. I have an input for the reference number as a text box. The token name is: purchCostReferenceToken
I want to limit the table results based on this token. This is the query:
<form>
<label>Thru Train Dashboard</label>
<fieldset submitButton="false" autoRun="true">
<input type="text" token="purchCostReferenceToken" searchWhenChanged="true">
<label>Enter a TMS Reference Number to Filter Table</label>
<default>*</default>
<initialValue>*</initialValue>
</input>
</fieldset>
<row>
<panel>
<title>Thru Train Data</title>
<table>
<search>
<query>index=... "<billingMethod>RULE</billingMethod>" "createMessage MsgSource" | xmlkv | rex max_match=0 "\<purchasedCostTripSegment\>(?P<segment>[^\<]+)" |eval Segments = mvrange(1,mvcount(mvindex(segment, 0, 2))+1,1) | rex max_match=0 "\<carrier\>(?P<Carriers>[^\<]+)" | rex max_match=0 "\<billingMethod\>(?P<BillingMethod>[^\<]+)" | rex max_match=0 "<purchasedCostTripSegment>[\s\S]*?<origin>\s*<ns2:numberCode>(?P<Origin>\d+)" | rex max_match=0 "<purchasedCostTripSegment>[\s\S]*?<destination>\s*<ns2:numberCode>(?P<Destination>\d+)" | rex max_match=0 "<purchasedCostTripSegment>[\s\S]*?<stopOff>\s*<ns2:stopOffLocation>\s*<ns2:numberCode>(?P<StopOffLocation>\d+)" | eval Time =_time | convert timeformat="%m-%d-%Y %H:%M:%S" ctime(Time) | table purchCostReference, eventType, Time, Segments, Carriers, BillingMethod, Origin, Destination, StopOffLocation | sort Time</query>
<earliest>-30d#d</earliest>
<latest>now</latest>
</search>
<option name="drilldown">none</option>
</table>
</panel>
</row>
</form>
Where do I add the token to limit the search?
I tried adding this to the end of the query before the table command:
... | eval Time =_time | convert timeformat="%m-%d-%Y %H:%M:%S" ctime(Time) purchCostReference=$purchCostReferenceToken$ | table purchCostReference, eventType, Time, Segments, Carriers, BillingMethod, Origin, Destination, StopOffLocation | sort Time
I get an error...error in convert command: the argument purchCostReference- is invalid
I would like to add filters in several of the table columns. The purchCostReference value is an extracted field in the query using xmlkv
Technically, a token can be placed anywhere in a query, but the query has to be valid when the token is replaced with its value. convert timeformat="%m-%d-%Y %H:%M:%S" ctime(Time) purchCostReference=4 is not valid SPL.
If the field referenced by the token is extracted automatically then it's usually best to put the token in the base search. That's not the case here.
You should use a search or where command to filter events based on the token value. Something like xmlkv | search purchCostReference=$purchCostReferenceToken$.
I'm currently working with a database that stores XML record for all of its field, please see below example. Lets name the table CUSTOMER table.
customer table
------------------------------------------------------
| RECID | XMLRECORD |
| 1 | <row id='1' xml:space="preserve"><c1>... |
| 2 | <row id='2' xml:space="preserve"><c1>... |
| 3 | <row id='3' xml:space="preserve"><c1>... |
------------------------------------------------------
All of the record of each customer is stored in 1 field called XMLRECORD, below is one example of XML RECORD of a customer.
<row id="1" xml:space="preserve">
<c1>James</c1>
<c2>Anderson</c2>
<c3>25</c3>
<c4>District 2 1657</c4>
<c4 m="2">Riverside Drive Redding</c4>
<c4 m="3">California, USA</c4>
</row>
Where c1 would be the customer's first name, c2 for last name, c3 for age and c4 would be the customer's address.
To query or extract values for each column, I usually use .value function to extract and return single value.
SELECT XMLRECORD.value('(/row/c4)[1]','NVARCHAR(20)') as ADDRESS
FROM CUSTOMER
Now my problem is this function only returns a single value, what I want is to return all the values under c4, which is multi value field. Can someone advise a way to do this?
Initiating the table
declare #xml as table
(
recid int,
xmlrecord xml
)
insert into #xml
values
( 1 , '<row id="1" xml:space="preserve">
<c1>James</c1>
<c2>Anderson</c2>
<c3>25</c3>
<c4>District 2 1657</c4>
<c4 m="2">Riverside Drive Redding</c4>
<c4 m="3">California, USA</c4>
</row>' )
DECLARE #XMLRECORD as xml =
'<row id="1" xml:space="preserve">
<c1>James</c1>
<c2>Anderson</c2>
<c3>25</c3>
<c4>District 2 1657</c4>
<c4 m="2">Riverside Drive Redding</c4>
<c4 m="3">California, USA</c4>
</row>' ;
Using .nodes functionality to fetch all the nodes
SELECT T.C.value('.','NVARCHAR(1000)') as c4_nodes FROM #XMLRECORD.nodes('(/row/c4)') as T(C)
Output -
c4_nodes
---------
District 2 1657
Riverside Drive Redding
California, USA
Since this fetches multiple records, using stuff command to concatenate the rows
SELECT
recid
,STUFF((
SELECT ',' + T.C.value('.','NVARCHAR(1000)')
FROM XMLRECORD.nodes('(/row/c4)') as T(C)
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 1, '') c4
FROM #xml
Output -
recid | c4
-----------
1 District 2 1657,Riverside Drive Redding,California, USA
c4 should not repeat multiple time. data should be stored in single node. For all node value, you should use [*] to get all node value.
You can try something like this
SELECT Tmp.record.value('.','NVARCHAR(20)')
FROM [customer]
CROSS APPLY [XMLRECORD].nodes('/row/c4') as Tmp(record)
Using MSSQL 2008 and XQUERY
Consider the following XML stored in a table:
<ROOT>
<WrapperElement>
<ParentElement ID=1>
<Title>parent1</Title>
<Description />
<ChildElement ID="6">
<Title>Child 4</Title>
<Description />
<StartDate>2010-01-25T00:00:00</StartDate>
<EndDate>2010-01-25T00:00:00</EndDate>
</ChildElement>
<ChildElement ID="0">
<Title>Child1</Title>
<Description />
<StartDate>2010-01-25T00:00:00</StartDate>
<EndDate>2010-01-25T00:00:00</EndDate>
</ChildElement>
<ChildElement ID="8">
<Title>Child6</Title>
<Description />
<StartDate>2010-01-25T00:00:00</StartDate>
<EndDate>2010-01-25T00:00:00</EndDate>
</ChildElement>
</ParentElement>
</WrapperElement>
</Root>
I want to decompose this xml into something like
PE!ID | PE!Title | PE!Description | CE!ID | CE!Title | CE!StartDate |...
1 | parent1 | | 6 | child 4 | 2010-... |
1 | parent1 | | 0 | child1 | 2010-... |
etc.
Note: there may be many ChildElements per ParentElement, in this example.
I've been experimenting with xquery however i've not been able to navigate through complex elements as such.
Basically, i'm trying to do the exact opposite of what FOR XML does to a table, only with a much more simplistic set of data to work with.
Any ideas on where to go next or how to accomplish this?
Thanks
How about this (I declared #input to be a XML datatype variable with your XML content - replace accordingly):
SELECT
Parent.Elm.value('(#ID)[1]', 'int') AS 'ID',
Parent.Elm.value('(Title)[1]', 'varchar(100)') AS 'Title',
Parent.Elm.value('(Description)[1]', 'varchar(100)') AS 'Description',
Child.Elm.value('(#ID)[1]', 'int') AS 'ChildID',
Child.Elm.value('(Title)[1]', 'varchar(100)') AS 'ChildTitle',
Child.Elm.value('(StartDate)[1]', 'DATETIME') AS 'StartDate',
Child.Elm.value('(EndDate)[1]', 'DATETIME') AS 'EndDate'
FROM
#input.nodes('/ROOT/WrapperElement/ParentElement') AS Parent(Elm)
CROSS APPLY
Parent.Elm.nodes('ChildElement') AS Child(Elm)
You basically iterate over all the /ROOT/WrapperElement/ParentElemet nodes (as Parent(Elm) pseudo table), and for each of those entries, you then do a CROSS APPLY for the child elements contained inside that ParentElement and pluck out the necessary information.
Should work - I hope!