Use sprintf syntax inside logstash's sprintf syntax - printf

For the below data structure:
{
"sprints": [
{
"id": 17193,
"name": "Sprint 12"
},
{
"id": 16510,
"name": "Sprint 11"
}
],
"velocityStatEntries": {
"16510": {
"estimated": {
"value": 49
},
"completed": {
"value": 36
}
},
"17193": {
"estimated": {
"value": 52
},
"completed": {
"value": 70
}
}
}
}
Given this, I want to be able to produce an Elasticsearch object that's easier to handle, by adding the values of the Estimated and Completed fields to the sprints with their matching IDs.
Ideally, I would like to handle this without writing Ruby, but I am not finding a logstash-native solution that handles this scnenario.
First, I split the data on the sprints field using split, so, I only have a single sprints object, and can use [sprints][id] to know what sprint I'm processing.
Then, I have attempted to work with the mutate filter, in one of two ways:
- using merge to add the [velocityStateEntries][] object to the
current sprint
- using add_field to add the two fields I need
Syntactically, is this possible? Ideally, I would want to be able to do a 'double substitution' of sorts, obtaining the estimated time for the current sprint something like:
add_field => {
"estimatedTime" => "%{[velocityStatEntries][%{[sprints][id]}][estimated][value]}"
}
but this only seems to work with a hardcoded format such as "estimatedTime" => "%{[velocityStatEntries][1234][estimated][value]}"
Do I have to use the Ruby format for this?

For what it's worth, the Ruby solution is very simple:
ruby {
code => "
sprintId = event.get('[sprints][id]');
estimated = event.get('[velocityStatEntries]['+(sprintId).to_s+'][estimated][value]');
completed = event.get('[velocityStatEntries]['+(sprintId).to_s+'][completed][value]');
event.set('[sprints][estimatedUnits]', estimated);
event.set('[sprints][completedUnits]', completed);
"
}

Related

Handling multiple rows returned by IMPORTJSON script on GoogleSheets

I am trying to populate a google sheet using an API. But the API has more than one row to be returned for a single query. Following is the JSON returned by API.
# https://api.dictionaryapi.dev/api/v2/entries/en/ABANDON
[
{
"word": "abandon",
"phonetics": [
{
"text": "/əˈbændən/",
"audio": "https://lex-audio.useremarkable.com/mp3/abandon_us_1.mp3"
}
],
"meanings": [
{
"partOfSpeech": "transitive verb",
"definitions": [
{
"definition": "Cease to support or look after (someone); desert.",
"example": "her natural mother had abandoned her at an early age",
"synonyms": [
"desert",
"leave",
"leave high and dry",
"turn one's back on",
"cast aside",
"break with",
"break up with"
]
},
{
"definition": "Give up completely (a course of action, a practice, or a way of thinking)",
"example": "he had clearly abandoned all pretense of trying to succeed",
"synonyms": [
"renounce",
"relinquish",
"dispense with",
"forswear",
"disclaim",
"disown",
"disavow",
"discard",
"wash one's hands of"
]
},
{
"definition": "Allow oneself to indulge in (a desire or impulse)",
"example": "they abandoned themselves to despair",
"synonyms": [
"indulge in",
"give way to",
"give oneself up to",
"yield to",
"lose oneself in",
"lose oneself to"
]
}
]
},
{
"partOfSpeech": "noun",
"definitions": [
{
"definition": "Complete lack of inhibition or restraint.",
"example": "she sings and sways with total abandon",
"synonyms": [
"uninhibitedness",
"recklessness",
"lack of restraint",
"lack of inhibition",
"unruliness",
"wildness",
"impulsiveness",
"impetuosity",
"immoderation",
"wantonness"
]
}
]
}
]
}
]
By using the following calls via IMPORTJSON,
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/phonetics/text", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/partOfSpeech", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/definitions/definition", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/definitions/synonyms", "noHeaders")
=ImportJSON(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2), "/meanings/definitions/example", "noHeaders")
I am able to get the following in GoogleSheets,
Whereas, the actual output according to JSON should be,
As you can see a complete row is being overwritten. How can this be fixed?
EDIT
Following is the link to sheet for viewing only.
I believe your goal as follows.
You want to achieve the bottom image in your question on Google Spreadsheet.
Unfortunately, I couldn't find the method for directly retrieving the bottom image using ImportJson. So in this answer, I would like to propose a sample script for retrieving the values you expect using Google Apps Script. I thought that creating a sample script for directly achieving your goal might be simpler rather than modifying ImportJson.
Sample script:
function SAMPLE(url) {
var res = UrlFetchApp.fetch(url, {muteHttpExceptions: true});
if (res.getResponseCode() != 200) return res.getContentText();
var obj = JSON.parse(res.getContentText());
var values = obj[0].meanings.reduce((ar, {partOfSpeech, definitions}, i) => {
definitions.forEach(({definition, example, synonyms}, j) => {
var v = [definition, Array.isArray(synonyms) ? synonyms.join(",") : synonyms, example];
var phonetics = obj[0].phonetics[i];
ar.push(j == 0 ? [(phonetics ? phonetics.text : ""), partOfSpeech, ...v] : ["", "", ...v]);
});
return ar;
}, []);
return values;
}
When you use this script, please put =SAMPLE(CONCATENATE("https://api.dictionaryapi.dev/api/v2/entries/en/"&$A2)) to a cell as the custom formula.
Result:
When above script is used, the following
Note:
In this sample script, when the structure of the JSON object is changed, it might not be able to be used. So please be careful this.
References:
Class UrlFetchApp
Custom Functions in Google Sheets

Figure out different values to send partial update to server

From a form submission I receive two objects: the original values and the dirty values. I like to figure out how to create a diff to send to the server using the following rules:
id field of the root object should always be included
all changed primitive values should be included
all nested changes should be included as well.
if a nested value other than id changed, it should include id as well.
Original values:
{
"id":10,
"name": "tkvw"
"locale": "nl",
"address":{
"id":2,
"street": "Somewhere",
"zipcode": "8965",
},
"subscriptions":[8,9,10],
"category":{
"id":6
},
}
Example expected diff objects:
1) User changes field name to "Foo"
{
"id":10,
"name":"foo"
}
2) User changes field street on address node and category
{
"id":10,
"address":{
"id": 2,
"street":"Changed"
},
"category":{
"id":5
}
}
I do understand the basics of functional programming, but I just need a hint in the right direction (some meta code maybe).
Take a look at JSON Patch (rfc6902), JSON Patch is a format for describing changes to a JSON document. For example:
[
{ "op": "replace", "path": "/baz", "value": "boo" },
{ "op": "add", "path": "/hello", "value": ["world"] },
{ "op": "remove", "path": "/foo"}
]
You generate a patch by comparing to JS objects/arrays, and then you can apply the patch to the original object (on the server side for example) to reflect changes.
You can create a patch using the fast-json-patch lib.
const obj1 = {"id":10,"name":"tkvw","locale":"nl","address":{"id":2,"street":"Somewhere","zipcode":"8965"},"subscriptions":[8,9,10],"category":{"id":6}};
const obj2 = {"id":10,"name":"cats","locale":"nl","address":{"id":2,"street":"Somewhere","zipcode":"8965"},"subscriptions":[8,9,10,11],"category":{"id":7}};
const delta = jsonpatch.compare(obj1, obj2);
console.log('delta:\n', delta);
const doc = jsonpatch.applyPatch(obj1, delta).newDocument;
console.log('patched obj1:\n', doc);
<script src="https://cdnjs.cloudflare.com/ajax/libs/fast-json-patch/2.0.6/fast-json-patch.min.js"></script>

Query with a logical expression comparing field values

I need to do a simple mongo query which resembles like this SQL
Select * from insights where category = 1 and param_count > param_mean + 1
Ideally you would use $redact as an aggregation expression for this, coupled with an initial $match to at least possibly use an index for the non-calculated expression:
db.collection.aggregate([
{ "$match": { "category": 1 } },
{ "$redact": {
"$cond": {
"if": { "$gt": [ "$param_count", { "$add": [ "$param_mean", 1 ] } ] },
"then": "$$KEEP",
"else": "$$PRUNE"
}
}}
])
If your MongoDB "server" version is less than 2.6 without the $redact operator, then you can use $where which evaluates a JavaScript expression to boolean to return results:
db.collection.find({
"category": 1,
"$where": "this.param_count > this.param_mean + 1"
})
Which while shorter in syntax, it takes considerable more processing time due to the need to evaluate the JavaScript expression.
Where possible, then you should use $redact, or avoid calculations altogether and store the calculated evaluation in the document instead. That last statement is true for "all" databases really.

square connect api batch processing

I need assistance with batch processing, especially in adding tax codes to items.
I'm experimenting with the square batch processing feature and my sample cases are create 2 items and add the tax code to them. In all 4 requests - 2 for creating item, 2 to 'put' the tax code. I have tried the following orders:
1. create the two items; add the taxes
2. create one item; add tax code to that item; create second item, add code to the second item.
In both instances, the result is the same - the taxes are applied to only one item. For the second item, the response I get is:
{
"status_code":404,
"body":{
"type":"not_found",
"message":"NotFound"
},
"request_id":4
}
To help with the investigation, here's the sample json that I use in the cURL request.
{
"requests":[
{
"method":"POST",
"relative_path":"\/v1\/me\/items",
"access_token":"XXX-YYY",
"body":
{
"id":126,
"name":"TestItem",
"description":"TestItemDescription",
"category_id":"DF1F51FB-11D6-4232-B138-2ECE3D89D206",
"variations":[
{
"name":"var1",
"pricing_type":"FIXED_PRICING",
"price_money":
{
"currency_code":"CAD",
"amount":400
},
"sku":"123444:QWEFASDERRG"
}
]},
"request_id":1
},
{
"method":"PUT",
"relative_path":"\/v1\/me\/items\/126\/fees\/7F2D50D8-43C1-4518-8B8D-881CBA06C7AB",
"access_token":"XXX-YYY",
"request_id":2
},
{
"method":"POST",
"relative_path":"\/v1\/me\/items",
"access_token":"XXX-YYY",
"body":
{
"id":127,
"name":"TestItem1",
"description":"TestItemDescription1",
"category_id":"DF1F51FB-11D6-4232-B138-2ECE3D89D206",
"variations":[
{
"name":"var1",
"pricing_type":"FIXED_PRICING",
"price_money":
{
"currency_code":"CAD",
"amount":400
},
"sku":"123444:QWEFASDERRG1"
}
]
},
"request_id":3
},
{
"method":"PUT",
"relative_path":"\/v1\/me\/items\/127\/fees\/7F2D50D8-43C1-4518-8B8D-881CBA06C7AB",
"access_token":"XXX-YYY",
"request_id":4
}
]
}
Below is the full response that I receive indicating successful creation of two items and only one successful tax push.
[
{
"status_code":200,
"body":
{
"visibility":"PUBLIC",
"available_online":false,
"available_for_pickup":false,
"id":"126",
"description":"TestItemDescription",
"name":"TestItem",
"category_id":"DF1F51FB-11D6-4232-B138-2ECE3D89D206",
"category":
{
"id":"DF1F51FB-11D6-4232-B138-2ECE3D89D206",
"name":"Writing Instruments"
},
"variations":[
{
"pricing_type":"FIXED_PRICING",
"track_inventory":false,
"inventory_alert_type":"NONE",
"id":"4c70909b-90bd-4742-b772-e4fabe636557",
"name":"var1",
"price_money":
{
"currency_code":"CAD",
"amount":400
},
"sku":"123444:QWEFASDERRG",
"ordinal":1,
"item_id":"126"
}
],
"modifier_lists":[],
"fees":[],
"images":[]
},
"request_id":1
},
{
"status_code":200,
"body":{},
"request_id":2
},
{
"status_code":200,
"body":
{
"visibility":"PUBLIC",
"available_online":false,
"available_for_pickup":false,
"id":"127",
"description":"TestItemDescription1",
"name":"TestItem1",
"category_id":"DF1F51FB-11D6-4232-B138-2ECE3D89D206",
"category":
{
"id":"DF1F51FB-11D6-4232-B138-2ECE3D89D206",
"name":"Writing Instruments"
},
"variations":[
{
"pricing_type":"FIXED_PRICING",
"track_inventory":false,
"inventory_alert_type":"NONE",
"id":"6de8932f-603e-4cd9-99ad-67f6c7777ffd",
"name":"var1",
"price_money":
{
"currency_code":"CAD",
"amount":400
},
"sku":"123444:QWEFASDERRG1",
"ordinal":1,
"item_id":"127"
}
],
"modifier_lists":[],
"fees":[],
"images":[]
},
"request_id":3
},
{
"status_code":404,
"body":
{
"type":"not_found",
"message":"NotFound"
},
"request_id":4
}
]
I have checked through going for the list of items and both items with their item ID's are present in the inventory. So the questions I have are, Why the tax is applied to one item and not to the other? How to resolve it?
From the Square docs:
Note the following when using the Submit Batch endpoint:
You cannot include more than 30 requests in a single batch.
Recursive
requests to the Submit Batch endpoint are not allowed (i.e., none of
the requests included in a batch can itself be a request to this
endpoint).
There is no guarantee of the order in which batched
requests are performed.
(emphasis mine).
If you want to use the batch API, you will have to create parent entities like items first, then in a separate batch request apply any child entities like fees, discounts, etc... Alternately, you can just make separate requests. There may not be much benefit from using the batch API in this case.

Unable to filter out n shingle(n - gram) facets using the "exclude" words option provided in the "facets" query

I am trying to make a tagcloud of words and phrases using the facets feature of elasticsearch.
My mapping:
curl -XPOST http://localhost:9200/myIndex/ -d '{
...
"analysis":{
"filter":{
"myCustomShingle":{
"type":"shingle",
"max_shingle_size":3,
"output_unigrams":true
}
},
"analyzer":{ //making a custom analyzer
"myAnalyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":[
"lowercase",
"myCustomShingle",
"stop"
]
}
}
}
...
},
"mappings":{
...
"description":{ //the field to be analyzed for making the tag cloud
"type":"string",
"analyzer":"myAnalyzer",
"null_value" : "null"
},
...
}
Query for generating facets:
curl -X POST "http://localhost:9200/myIndex/myType/_search?&pretty=true" -d '
{
"size":"0",
"query": {
match_all:{}
},
"facets": {
"blah": {
"terms": {
"fields" : ["description"],
"exclude" : [ 'evil' ], //remove facets that contain these words
"size": "50"
}
}
}
}
My problem is, when I insert a word say 'evil' in the "exclude" option of "facets", it successfully removes the facets containing the words(or single shingles) that match 'evil'. But it doesn't remove the 2/3 word shingles, "resident evil" , "evil computer", "my evil cat". How do I remove the facets of phrases containing the "exclude words"?
It isn't completely clear what you want to achieve. You usually wouldn't make facets on analyzed fields. Maybe you could explain why you're making shingles so that we can help achieving what you want in a better way.
With the exclude facet parameter you can exclude some specific entry, but evil is not the same as resident evil. If you want to exclude it you need to specify it. Facets are made based on indexed terms, and resident evil is in fact a single term in the index, which is not the same as the term evil.
Given the choice that you already made for indexing and faceting, there is a way to achieve what you want. Elasticsearch has a really powerful scripting module. You can use a script to decide whether each entry should be included in the facet or not like this:
{
"query": {
"match_all" : {}
},
"facets": {
"tags": {
"terms": {
"field" : "tags",
"script" : "term.contains('evil') ? true : false"
}
}
}
}