I have following message generated. This is default structure generated by filebeat. Here the metadata is nested, I am looking for a way to restructure the beat fields to root level as shown in second example.
{
"#timestamp": "2017-03-27T08:14:09.508Z",
"beat": {
"hostname": "stage-kube03",
"name": "stage-kube03",
"version": "5.2.1"
},
"input_type": "log",
"message": {
"message": {
"activityType": null
},
"offset": 3783008,
"source": "/var/log/audit.log",
"type": "log"
}
would want to take beat fields a level up as shown below.
{
"#timestamp": "2017-03-27T08:14:09.508Z",
"hostname": "stage-kube03",
"name": "stage-kube03",
"version": "5.2.1"
"input_type": "log",
"message": {
"message": {
"activityType": null
},
"offset": 3783008,
"source": "/var/log/audit.log",
"type": "log"
}
For this sort of manipulation you could use the Logstash Output: https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html
Related
Hi Can someone help me simulate this scenario, Example this is the response I got, I want to extract all alertId with the name parameter contains test. You response is highly appreciated. Thank you so much.
Response:
[
{
"duplicateCount": 0,
"fqdn": "qa-ubuntu14-4",
"appName": "TEST_APD_UB14",
"stateString": "OPEN",
"category": "FILESCAN",
"alkey": {
"agentId": "8470ea64-a710-3e46-ba6b-ccd37ebc4074",
"role": "AD SERVER",
"alertId": "0258a7ca-bc72-3a53-aa98-3098c87411ba",
"id": "6695a7fa-ab9f-43fa-871b-620cd1eeb75054af7770-604b-11e9-b486-8d59ab9344597cea0ea2-d897-3696-852d-5f3cb36f270e8470ea64-a710-3e46-ba6b-ccd37ebc4074/var/log/test321.txttest321.txtA",
"applicationContextId": "7cea0ea2-d897-3696-852d-5f3cb36f270e"
},
"properties": {
"name": "test321.txt",
"acl": ""
}
},
{
"duplicateCount": 0,
"fqdn": "qa-ubuntu14-4",
"appName": "TEST_APD_UB18",
"stateString": "OPEN",
"category": "FILESCAN",
"alkey": {
"agentId": "8470ea64-a710-3e46-ba6b-ccd37ebc4074",
"role": "AD SERVER",
"alertId": "0258a7ca-bc72-3a53-aa98-3098c8741CDA",
"id": "6695a7fa-ab9f-43fa-871b-620cd1eeb75054af7770-604b-11e9-b486-8d59ab9344597cea0ea2-d897-3696-852d-5f3cb36f270e8470ea64-a710-3e46-ba6b-ccd37ebc4074/var/log/test321.txttest321.txtA",
"applicationContextId": "7cea0ea2-d897-3696-852d-5f3cb36f270e"
},
"properties": {
"name": "test555.txt",
"acl": ""
}
}
]
Screenshot:
Expected Result:
I want to extract all alertId with the name parameter contains test
You could use the following JSON query to extract the values:
[*].[?(#.properties.name contains 'test')]alkey.agentId
I found this reference with JSON Path Syntax is really useful.
I'm trying to query the TinkerPop server (hosted inside docker container) via CosmosDB client library, which uses under the hood Gremlin.Net. So I managed to connect it and insert the data, here's intercepted WebSocket request:
!application/vnd.gremlin-v1.0+json{
"requestId": "b64bd2eb-46c3-4095-9eef-768bca2a14ed",
"op": "eval",
"processor": "",
"args": {
"gremlin": "g.addV(\"User\").property(\"UserId\",2).property(\"CustomerId\",1)"
}
}
The response:
{
"requestId": "b64bd2eb-46c3-4095-9eef-768bca2a14ed",
"status": {
"message": "",
"code": 200,
"attributes": {
"host": "/172.19.0.1:38848"
}
},
"result": {
"data": [
{
"id": 0,
"label": "User",
"type": "vertex",
"properties": {}
}
],
"meta": {}
}
}
Problem is that I see those properties when I'm connected via gremlin console
gremlin> g.V().hasLabel("User").has("CustomerId",1).has("UserId",2).limit(1).valueMap()
==>{UserId=[2], CustomerId=[1]}
Also, I'm able to query the TinkerPop server with Gremlin.Net:
!application/vnd.gremlin-v1.0+json{
"requestId": "de35909f-4bc1-4aae-aa5f-28361b3c0933",
"op": "eval",
"processor": "",
"args": {
"gremlin": "g.V().hasLabel(\"User\").has(\"CustomerId\",1).has(\"UserId\",2).limit(1)"
}
}
But it returns a payload with zero-valued ID and without any properties included:
{
"requestId": "de35909f-4bc1-4aae-aa5f-28361b3c0933",
"status": {
"message": "",
"code": 200,
"attributes": {
"host": "/172.19.0.1:38858"
}
},
"result": {
"data": [
{
"id": 0,
"label": "User",
"type": "vertex",
"properties": {}
}
],
"meta": {}
}
}
Tried to swap between GraphSON v1, v2, v3 with no luck. Documentation says that script serializers should include all the properties. Do I have to tweak the config somehow to make this work and return properties?
So it seems that with a version of 3.4 of the Gremlin server ReferenceElementStrategy
was added by default to traversals, to preserve compatibility between binary and script serializers. In our case we wanted to mimic the behavior of the CosmosDB, so to adjust and receive desired behavior just remove the strategy from init script (in our case it was empty-sample.groovy
globals << [g : graph.traversal().withStrategies(ReferenceElementStrategy.instance())]
to
globals << [g : graph.traversal()]
I have completed a Node.js app using LINE APIs.I have the following request object. How can I define the array of different objects, here the messagesfield which contains different object structure for different message types. I hope swagger permits this very common scenario.
Request Body:
{
"to": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"messages":[
{
"type":"text",
"text":"Hello, world1"
},
{
"type": "audio",
"originalContentUrl": "https://example.com/original.m4a",
"duration": 240000
}
{
"type": "location",
"title": "my location",
"address": "〒150-0002 東京都渋谷区渋谷２丁目２１−１",
"latitude": 35.65910807942215,
"longitude": 139.70372892916203
}
]
}
My swagger definition for messages array.
"Messages Object": {
"type": "array",
"items": {
"allOf": [
{
"$ref": "#/definitions/multicast Message Error Response"
},
{
"$ref": "#/definitions/multicast Message Error Response"
}
]
}
}
And this is the rendered messages array. It has only one entry. I want to include many different entries
"messages": [
{
"code": 500,
"httpCode": 400,
"name": "string",
"message": "string"
}
]
I create measurements at reception of an event, I can get them using the API, but they are not represented graphically in the Device Management interface. I there a specific format they would have to respect to be representable automatically? If so, is there a place I can find all the formats supported by Cumulocity? I infered the c8y_TemperatureMeasurement from the examples in the doc but I didn't find an exhaustive list of the native formats.
Here are examples of the measurements I have at the moment:
{
"time": "2016-06-29T12:10:02.000+02:00",
"id": "27006",
"self": "https://<tenant-id>/measurement/measurements/27006",
"source": {
"id": "26932",
"self": "https://<tenant-id>/inventory/managedObjects/26932"
},
"type": "c8y_BatteryMeasurement",
"c8y_BatteryMeasurement": {
"unit": "V",
"value": 80
}
},
{
"time": "2016-06-29T10:15:22.000+02:00",
"id": "27010",
"self": "https://<tenant-id>/measurement/measurements/27010",
"source": {
"id": "26932",
"self": "https://<tenant-id>/inventory/managedObjects/26932"
},
"type": "c8y_TemperatureMeasurement",
"c8y_TemperatureMeasurement": {
"T": {
"unit": "C",
"value": 24
}
}
}
The measurements have to be sent to Cumulocity in the following format:
{
"fragment": {
"series": {
"unit": "x",
"value": y
}
}
}
I am trying to list all the types for a particular id:
{
"id": "/en/sony",
"type": [{
"name": "Topic",
"id": null
}]
}
This query giving me the following result:
http://tinyurl.com/lubavey
{
"result": {
"type": [
{
"id": "/common/topic",
"name": "Topic"
},
{
"id": "/base/audiobase/topic",
"name": "Topic"
},
{
"id": "/base/fblinux/topic",
"name": "Topic"
},
{
"id": "/base/digitalcameras/topic",
"name": "Topic"
},
{
"id": "/base/popstra/topic",
"name": "Topic"
},
{
"id": "/base/televisions/topic",
"name": "Topic"
},
{
"id": "/base/ps3games/topic",
"name": "Topic"
},
{
"id": "/base/filmcameras/topic",
"name": "Topic"
},
{
"id": "/m/04mny2g",
"name": "Topic"
}
],
"id": "/en/sony"
}
}
I want exactly the opposite result. I want all the types which do not have name as "Topic" with them.
How can I achieve this? I tried to use ! operator with property name which is suggested in reference guide of MQL, but it's giving me error:
"Can't use unqualified property names with ! reversing".
What should I do to remove this error with ! and to obtain opposite result of the query?
Try with !=:
{
"id": "/en/sony",
"type": [{
"name!=": "Topic",
"id": null
}]
}
The != operator says that the constrained property can be anything but
the specified value. (It does require that the property be something,
however: it does not match object for which the property is null.)
Read more about != operator here: http://wiki.freebase.com/wiki/MQL_operators#The_.22but_not.22_Operator_.21.3D