Given below is a snippet of splunk event. My requirement is to find all the occurrences of "isOutstanding": true. Here the point to note is that one event may/may not have multiple occurrences. Need to find the total count from multiple events over a period of time.
{
\"school\": {
\"schoolId\": \"1\",
\"schoolName\": \"SchoolX\",
\"schoolType\": \"private\",
\"students\": [
{
\"id\": \"1\",
\"isOutstanding\": true,
},
{
\"id\": \"2\",
\"isOutstanding\": false,
},
{
\"id\": \"3\",
\"isOutstanding\": false,
}
]
}
}
The below Splunk query index=myIndex "isOutstanding":true gives the count of events having "isOutstanding": true. But it doesn't consider the count of multiple occurrences in one event.
How can I get the count of all the occourences in an event? TIA
You can combine the rex feature to extract all instances of the pattern you're looking for, then use the mvcount to count them.
index=syslog sourcetype=testing isOutstanding
| rex field=school max_match=0 "(?<outs>isOutstanding\": true")
| eval total=mvcount(outs)
| table total
Finally got the query for my requirement
index=myindex sourcetype=mysourceType
| rex max_match=0 "(?<isOutstanding>isOutstanding\\\\\":true)"
| stats count(isOutstanding) as total
Related
I have some events that capture the times when different jobs start or end. Here are two events that capture the start and end times of a job-
[
{
"appName": "a1",
"eventName": "START",
"eventTime": "t1"
},
{
"appName": "a1",
"eventName": "END",
"eventTime": "t2"
},
{
"appName": "a1",
"eventName": "START",
"eventTime": "t3"
},
{
"appName": "a2",
"eventName": "START",
"eventTime": "t4"
}
]
I am looking to visualize this information in a table showing the latest start and end times of each application, something like this -
--AppName--Last Start Time--Last End Time--
--a1--t3--t2--
--a2--t4--null--
The above table is assuming t3 comes after t1. How do i get to this ? I am able to extract the latest events for each into separate rows with this -
stats latest(eventTime) by appName, eventName but need them to be combined into one single tuple.
Create separate fields for start and end times, then use stats to get the latest for each.
| eval start_time=if(eventName="START", eventTime, null())
| eval end_time=if(eventName="END", eventTime, null())
| stats latest(start_time) as last_start, latest(end_time) as last_end by appName
XXX Note that for the latest function to work properly, eventTime must be in epoch format. If it isn't, use the strptime function in a eval to convert it. XXX (Disregard this last part.)
I've a Splunk json event like this:
{
name: "my-name"
tasks: [
{
id: 1,
value: 1
},
{
id: 2,
value: 2
},
]
}
How to write a SPL command to return output with 2 records like:
{name: "my-name", id: 1, value: 1}
{name: "my-name", id: 2, value: 3}
Please help me, thank you guys !
Assuming you actually have ingested a valid JSON object, (as copied here your "json" event is not actually JSON, See the spec at json.org ) but it's possible you copied the prettified version instead of the raw event.
And assuming you're on a new enough version of Splunk to have JSON Functions with eval...
<your search>
| fields name
| spath tasks{} output=task
| mvexpand task
| eval _raw=json_set(task,"name",name)
I simulated an event with this:
| makeresults | eval _raw=json_object("name", "my-name", "tasks", json_array(json_object("id", 1, "value", 1), json_object("id", 2, "value", 2))) | spath
Admittedly you could also use | windbag | head 1 instead of | makeresults for simulation but that gets a bit into the obscure undocumented testing commands that happen to ship with the product.
I receive JSON from API in the following format:
[
{
"scId": "000DD2",
"sensorId": 2,
"metrics": [
{
"s": 5414,
"dateTime": "2018-02-02T13:03:30+01:00"
},
{
"s": 5526,
"dateTime": "2018-02-02T13:04:56+01:00"
},
{
"s": 5631,
"dateTime": "2018-02-02T13:06:22+01:00"
}
}, .... ]
Currently trying to display these metrics on the linear chart with dateTime for the X-axis and "s" for Y.
I use the following search query:
index="main" source="rest://test3" | spath input=metrics{}.s| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime | rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime| table s,dateTime
And I receive the data in the following format which is not applicable for linear chart. The point is - how to correctly parse the JSON to apply date-time from dateTime field in JSON to _time in Splunk.
Query results
#Max Zhylochkin,
Can you please try following search?
index="main" source="rest://test3"
| spath input=metrics{}.s
| mvexpand metrics{}.s
| mvexpand metrics{}.dateTime
| rename metrics{}.s as s
| rename metrics{}.dateTime as dateTime
| table s,dateTime
| eval _time = strptime(dateTime,"%Y-%m-%dT%H:%M:%S.%3N")
Thanks
I'm performing the following lookback snapshot. I got the data back but i also need the ValidFrom Date of the Completed record. It's not a part of the _PreviousValues record. How can i get that to come back with my query?
Thanks!
find: {
'_TypeHierarchy': 'HierarchicalRequirement',
'Children':null,'ScheduleState':'Accepted',
'_PreviousValues.ScheduleState':'Completed',
'_ValidFrom': { '$gte':startDate},
'_ValidTo': { '$lte': endDate},
},
fetch: ['FormattedID','Name','_ValidFrom','_ValidTo','BlockedReason','_User','WorkProduct','ScheduleState','_PreviousValues.ScheduleState','AcceptedDate'],
// order: 'OpenedDate DESC',
hydrate: ['FormattedID','Name','_ValidFrom','_ValidTo','BlockedReason','_User','WorkProduct','ScheduleState','_PreviousValues.ScheduleState','AcceptedDate'],
compress: true,
It looks like what we need is something like _PreviousValues.ScheduleState._ValidFrom, but it does not exist.
I think it is not possible to get _ValidFrom value of the _PreviousValues.ScheduleState from the same query, and a separate query is needed.
For example, this query:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/1234/artifact/snapshot/query.js?find={"Project":5678,"_TypeHierarchy":"HierarchicalRequirement","ScheduleState":"Accepted", "_PreviousValues.ScheduleState": "Completed"}&fields=["ObjectID","_ValidFrom","_ValidTo","ScheduleState","_PreviousValues.ScheduleState"]&hydrate=["ScheduleState","_PreviousValues.ScheduleState"]&compress=true
will return _PreviousValues object which only includes state value:
_PreviousValues: {
ScheduleState: "Completed"
}
Let's say one of the results has ObjectID 777.
The second query will use ObjectID(s) of the results of the first query to get the time interval when the story was in the "Completed" state:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/1234/artifact/snapshot/query.js?find={"ObjectID":777,"ScheduleState": "Completed","_PreviousValues.ScheduleState": "In-Progress"}&fields=["ObjectID","_ValidFrom","_ValidTo","ScheduleState"]&hydrate=["ScheduleState"]
It may return more than one snapshot, and depending on what fields are fetched there may not be an indication what changed between those snapshots (e.g. in this case TaskStatus and TaskRemainingTotal) but in any case the earliest snapshot's _ValidFrom value, _ValidFrom: "2013-06-17T18:51:36.931Z" is the date you are looking for
Results:
[
{
_ValidFrom: "2013-06-17T18:51:36.931Z",
_ValidTo: "2013-06-17T18:51:44.382Z",
ObjectID: 12353154323,
ScheduleState: "Completed"
},
{
_ValidFrom: "2013-06-17T18:55:50.897Z",
_ValidTo: "2013-06-18T20:53:01.755Z",
ObjectID: 12353154323,
ScheduleState: "Completed"
}
]
If you are writing a code, you will get the _ValidFrom of the first element of the array of objects.
Probably a lookback newbie question, but how do I return all of the history for stories based on an attribute that gets set later in their history?
Specifically, I want to load all of the history for all stories/defects in my project that have an accepted date in the last two weeks.
The following query (below) doesn't work because it (of course) only returns those history records where accepted date matches the query. What I actually want is all of the history records for any defect/story that is eventually accepted after that date...
filters :
[
{
property: "_TypeHierarchy",
value: { $nin: [ -51009, -51012, -51031, -51078 ] }
},
{
property: "_ProjectHierarchy",
value: this.getContext().getProject().ObjectID
},
{
property: "AcceptedDate",
value: { $gt: Ext.Date.format(twoWeeksBack, 'Y-m-d') }
}
]
Thanks to Nick's help, I divided this into two queries. The first grabs the final history record for stories/defects with an accepted date. I accumulate the object ids from that list, then kick off the second query, which finds the entire history for each object returned from the first query.
Note that I'm caching some variables in the "window" scope - that's my lame workaround to the fact that I can't ever quite figure out the context of "this" when I need it...
window.projectId = this.getContext().getProject().ObjectID;
I also end up flushing window.objectIds (where I store the results from the first query) when I exec the query, so I don't accumulate results across reloads. I'm sure there's a better way to do this, but I struggle with scope in javascript.
filter for first query
filters : [ {
property : "_TypeHierarchy",
value : {
$nin : [ -51009, -51012, -51031, -51078 ]
}
}, {
property : "_ProjectHierarchy",
value : window.projectId
}, {
property : "AcceptedDate",
value : {
$gt : Ext.Date.format(monthBack, 'Y-m-d')
}
}, {
property : "_ValidTo",
value : {
$gt : '3000-01-01'
}
} ]
Filter for second query:
filters : [ {
property : "_TypeHierarchy",
value : {
$nin : [ -51009, -51012, -51031, -51078 ]
}
}, {
property : "_ProjectHierarchy",
value : window.projectId
}, {
property : "ObjectID",
value : {
$in : window.objectIds
}
}, {
property : "c_Kanban",
value : {
$exists : true
}
} ]
Here's an alternative query that will return only the snapshots that represent transition into the Accepted state.
find:{
_TypeHierarchy: { $in : [ -51038, -51006 ] },
_ProjectHierarchy: 999999,
ScheduleState: { $gte: "Accepted" },
"_PreviousValues.ScheduleState": {$lt: "Accepted", $exists: true},
AcceptedDate: { $gte: "2014-02-01TZ" }
}
A second query is still required if you need the full history of the stories/defects. This should at least give you a cleaner initial list. Also note that Project: 999999 limits to the given project, while _ProjectHierarchy finds stories/defects in the child projects, as well.
In case you are interested, the query is similar to scenario #5 in the Lookback API documentation at https://rally1.rallydev.com/analytics/doc/.
If I understand the question, you want to get stories that are currently accepted, but you want that the returned results include snapshots from the time when they were not accepted. Before you write code, you may test an equivalent query in the browser and see if the results look as expected.
Here is an example - you will have to change OIDs.
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"_ProjectHierarchy":12352608219,"_TypeHierarchy":"HierarchicalRequirement","ScheduleState":"Accepted",_ValidFrom:{$gte: "2013-11-01",$lt: "2014-01-01"}}},sort:[{"ObjectID": 1},{_ValidFrom: 1}]&fields=["Name","ScheduleState","PlanEstimate"]&hydrate=["ScheduleState"]
You are correct that a query like this: find={"AcceptedDate":{$gt:"2014-01-01T00:00:00.000Z"}}
will return one snapshot per story that satisfies it.
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"AcceptedDate":{$gt:"2014-01-01T00:00:00.000Z"}}&fields=true&start=0&pagesize=1000
but a query like this: find={"ObjectID":{$in:[16483705391,16437964257,14943067452]}}
will return the whole history of the 3 artifacts:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"ObjectID":{$in:[16483705391,16437964257,14943067452]}}&fields=true&start=0&pagesize=1000
To illustrate, here are some numbers: the last query returns 17 results for me. I check each story's revision history, and the number of revisions per story are 5, 5, 7 respectively, sum of which is equal to the total result count returned by the query.
On the other hand the number of stories that meet find={"AcceptedDate":{$gt:"2014-01-01T00:00:00.000Z"}} is 13. And the query based on the accepted date returns 13 results, one snapshot per story.