Pushing array to Firebase via REST API - api

Question
How do I (with a single HTTP request to the REST API) write an array to Firebase and give each array element a (non-integer) unique ID?
As described here.
Data
The data I have to write looks like the following.
data-to-write.js
myArray = [ {"user_id": "jack", "text": "Ahoy!"},
{"user_id": "jill", "text": "Ohai!"} ];
Goal
When finished, I want my Firebase to look like this following.
my-firebase.firebaseio.com
{
"posts": {
"-JRHTHaIs-jNPLXOQivY": { // <- unique ID (non-integer)
"user_id": "jack",
"text": "Ahoy!"
},
"-JRHTHaKuITFIhnj02kE": { // <- unique ID (non-integer)
"user_id": "jill",
"text": "Ohai!"
}
}
}
I do not want it to look like this following...
my-anti-firebase.firebaseio.com
// NOT RECOMMENDED - use push() instead!
{
"posts": {
"0": { // <- ordered array index (integer)
"user_id": "jack",
"text": "Ahoy!"
},
"1": { // <- ordered array index (integer)
"user_id": "jill",
"text": "Ohai!"
}
}
}
I note this page where it says:
[...] if all of the keys are integers, and more than half of the keys between 0 and the maximum key in the object have non-empty values, then Firebase will render it as an array.
Code
Because I want to do this in a single HTTP request, I want to avoid iterating over each element in the array and, instead, I want to push a batch in a single request.
In other words, I want to do something like this:
pseudocode.js
curl -X POST -d '[{"user_id": "jack", "text": "Ahoy!"},
{"user_id": "jill", "text": "Ohai!"}]' \
// I want some type of batch operation here
'https://my-firebase.firebaseio.com/posts.json'
However, when I do this, I get exactly what I describe above that I don't want (i.e., sequential integer keys).
I want to avoid doing something like this:
anti-pseudocode.js
for(i=0; i<=myArray.length; i++;){ // I want to avoid iterating over myArray
curl -X POST -d '{"user_id": myArray[i]["user_id"],
"text": myArray[i]["text"]}' \
'https://my-firebase.firebaseio.com/posts.json'
}
Is it possible to accomplish what I have described? If so, how?

I don't think there is a way to use the Firebase API to do this as described in the OP.
However, it can be done with a server script as follows:
Iterate through each array element.
Assign each element a unique id (generated by server script).
Create a return object with keys being the unique IDs and values being the corresponding array elements.
Write object to Firebase with a single HTTP request using the patch method. Because post creates a new Firebase generated ID for the entire object itself. Whereas, patch does not; it writes directly to the parent node.
script.js
var myObject = {},
i = myArray.length;
while(i--){
var key = function(){ /* return unique ID */ }();
myObject[key] = myArray[i];
}
curl -X PATCH -d JSON.stringify(myObject) \
'https://my-firebase.firebaseio.com/posts.json'

Your decision to use POST is correct. The one which cause numeric indexes as a key is because your payload is an array. Whenever you post/put and array, the key will always be indexes. Post your object one by one if you want the server generate key for you.

Firebase will generate unique ID only if you use POST. If you use PATCH unique ID is not generated.
Hence for the given case, will need to iterate through using some server/ client side code to save data in firebase.
Peseudo Code:
For each array
curl -X POST -d
"user_id": "jack",
"text": "Ahoy!"
'https://my-firebase.firebaseio.com/posts.json'
Next

Related

Lua script access object path inside an array

I'm trying to access the object with property optionId = 'a386ead3-08ca-486e-aeb1-23add87292e7' to set its weight.
my object is like following:
weight": {
"options": [
{
"optionId": "a386ead3-08ca-486e-aeb1-23add87292e7",
"weight": 10
},
{
"optionId": "a386ead3-08ca-486e-aeb1-23add87292e7",
"weight": 20
}
],
"value": 100
}
and i'm using the following function to get its path but with no luck:
local GetFieldOptionWeightPath = function (optionId)
return "$.weight.options[\"optionId\"==\""..optionId.."\"]";
end
You need to compare each optionId like you would programmatically. To do that, you can use a filter script expression:
$.weight.options[?(#.optionId=="a386ead3-08ca-486e-aeb1-23add87292e7")]
Here, $() is the filter and # will point to each of the elements that's getting filtered.
Note that since it's a filter, it may potentially yield multiple results. In fact, in your example case it will yield both entries as they have the same optionId.
In the end your Lua function generating the path can look like:
local GetFieldOptionWeightPath = function (optionId)
return ("$.weight.options[?(#.optionId==%q)]"):format(optionId)
end
This answer assumes JSONPath support which was implemented in RedisJSON v2 (late 2021).

How to set nested values (objects) using ReJSON

If I insert the following object using ReJSON:
JSON.SET testing . '{"person":{"name":"John","surname":"Doe"}}'
Is there a way to "append" a nested structure? I would like to add "address.name" for an example to get the following JSON:
{
"person": {
"name": "John",
"surname": "Doe"
},
"address": {
"name": "Imaginary Street"
}
}
I was trying to use JSON.SET testing .address.name '"Imaginary Street 7"' but this results in (error) ERR missing key at non-terminal path level.
The docs read:
A key (with its respective value) is added to a JSON Object (in a
Redis ReJSON data type key) if and only if it is the last child in the
path.
Is "address.name" not the last child in the path? What am I doing wrong?
Since you're adding a dictionary ('address'), the way to go about this is:
JSON.SET testing .address '{"name": "Imaginary Street"}'
Alternatively, if you do just:
JSON.SET testing .address '{}'
you'll be able to use the command from your question without any errors.

Collapsing a group using Google Sheets API

So as a workaround to difficulties creating a new sheet with groups I am trying to create and collapse these groups in a separate call to batchUpdate. I can call request an addDimensionGroup successfully, but when I request updateDimensionGroup to collapse the group I just created, either in the same API call or in a separate one, I get this error:
{
"error": {
"code": 400,
"message": "Invalid requests[1].updateDimensionGroup: dimensionGroup.depth must be \u003e 0",
"status": "INVALID_ARGUMENT"
}
}
But I'm passing depth as 0 as seen by the following JSON which I send in my request:
{
"requests":[{
"addDimensionGroup":{
"range":{
"dimension":"ROWS",
"sheetId":0,
"startIndex":2,
"endIndex":5}
}
},{
"updateDimensionGroup":{
"dimensionGroup":{
"range": {
"dimension":"ROWS",
"sheetId":0,
"startIndex":2,
"endIndex":5
},
"depth":0,
"collapsed":true
},
"fields":"*"
}
}],
"includeSpreadsheetInResponse":true}',
...
I'm not entirely sure what I am supposed to provide for "fields", the documentation for UpdateDimensionGroupRequest says it is supposed to be a string ("string ( FieldMask format)"), but the FieldMask definition itself shows the possibility of multiple paths, and doesn't tell me how they are supposed to be separated in a single string.
What am I doing wrong here?
The error message is actually instructing you that the dimensionGroup.depth value must be > 0:
If you call spreadsheets.get() on your sheet, and request only the DimensionGroup data, you'll note that your created group is actually at depth 1:
GET https://sheets.googleapis.com/v4/spreadsheets/{SSID}?fields=sheets(rowGroups)&key={API_KEY}
This makes sense, since the depth is (per API spec):
depth numberThe depth of the group, representing how many groups have a range that wholly contains the range of this group.
Note that any given particular DimensionGroup "wholly contains its own range" by definition.
If your goal is to change the status of the DimensionGroup, then you need to set its collapsed property:
{
"requests":
[
{
"updateDimensionGroup":
{
"dimensionGroup":
{
"range":
{
"sheetId": <your sheet id>,
"dimension": "ROWS",
"startIndex": 2,
"endIndex": 5
},
"collapsed": true,
"depth": 1
},
"fields": "collapsed"
}
}
]
}
For this particular Request, the only attribute you can set is collapsed - the other properties are used to identify the desired DimensionGroup to manipulate. Thus, specifying fields: "*" is equivalent to fields: "collapsed". This is not true for the majority of requests, so specifying fields: "*" and then omitting a non-required request parameter is interpreted as "Delete that missing parameter from the server's representation".
To change a DimensionGroup's depth, you must add or remove other DimensionGroups that encompass it.

Possible to use angular-datatables with serverside array sourced data instead of object sourced data

I'm trying to use angular-datatables with serverside processing. However, it seems that angular-datatables expects that the data from the server is in object format (object vs array data described) with column names preceding each table datapoint. I'd like to configure angular-datatables to accept array based data since I can't modify my server side output which only outputs data in array format.
I'm configuring Datatables in my javascript like so:
var vm = this;
vm.dtOptions = DTOptionsBuilder.newOptions()
.withOption('ajax', {
url: 'output/ss_results/' + $routeParams.uuid,
type: 'GET'
})
.withDataProp('data')
.withOption('processing', true)
.withOption('serverSide', true);
My data from the server looks like this in array format:
var data = [
[
"Tiger Nixon",
"System Architect",
"$3,120"
],
[
"Garrett Winters",
"Director",
"$5,300"
]
]
But as far as I can tell, angular-datatables is expecting the data in object format like so:
[
{
"name": "Tiger Nixon",
"position": "System Architect",
"extn": "5421"
},
{
"name": "Garrett Winters",
"position": "Director",
"extn": "8422"
}
]
I tried not defining dtColumns or setting it to an empty array like vm.dtColumns = []; but I get an error message when I do that. When I configure dtColumns with a promise to load the column data via ajax I get datatables error #4 because it can't find the column name preceding my table datapoints in the data retrieved from the server.
Is it possible to configure angular-datatables to accept array based data? I can't find anything on the angular-datatables website that indicates it can be configured this way.
Edit: So I removed the .withDataProp('data') which I think was causing the problem. The table works a little better now but it's still broken. After it loads, I get the message No matching records found. Even though right below it it says Showing 1 to 10 of 60,349 entries
Previous1…456…6035Next Does anyone know why this might be?
If you want to use an array of arrays instead of an array of objects, simply refer to the array indexes instead of the object names :
$scope.dtColumns = [
DTColumnBuilder.newColumn(0).withTitle('Name'),
DTColumnBuilder.newColumn(1).withTitle('Position'),
DTColumnBuilder.newColumn(2).withTitle('Office'),
DTColumnBuilder.newColumn(3).withTitle('Start date'),
DTColumnBuilder.newColumn(4).withTitle('Salary')
]
demo using the famous "Tiger Nixon" array loaded via AJAX -> http://plnkr.co/edit/16UoRqF5hvg2YpvAP8J3?p=preview

What is the best way to create a subset of my data in Elasticsearch?

I have an index in elasticsearch containing apache log data. Here is what I want to do:
Identify all visitors (by ip number) that accessed a certain file (e.g. /signup.php).
Do a search/query/aggregation on my data, but limit the documents that are examined to those containing an ip number found in step 1.
In the sql world, I would just create a temporary table and insert all the matching IP numbers from step one. Next I would query my main table and limit the result set by joining in my temporary table on IP number.
I understand joins are not possible in elasticsearch. The elasticsearch documentation suggests a few ways to handle situations like this:
Application side joins
This does not seem practical, because the list of IP numbers may be very large and it seems inefficient to send the results to the client and then pass it back to elasticsearch in one huge terms filter.
Denormalizing the data
This would involve iterating over the matching IP numbers and updating every document in the index for any given IP number with something like "in_group": true, so I can use that in my query later on. This also seems very impractical and inefficient, especially since the source query (step 1) is dynamic.
Nested Object and/or parent-Child relationship
I'm not sure if dynamically creating new documents with nested objects is practical in this case. It seems to me that I would end up copying huge parts of my data.
I'm new to elasticsearch and noSQL in general, so perhaps I'm just looking at the problem the wrong way and I shouldn't be trying to emulate a JOIN in the first place.
But this seems like such a common case for segmenting a dataset, it makes me wonder if I am overlooking some other obvious way of doing this?
Any help would be appreciated!
If I understood your question correctly, you are trying to get a subset of your documents based on certain condition and use that sub set to query/search/aggregate it further.
If true, why would you like to store it in another view(sql types). The main power of elasticsearch is it's caching capability of filters and thus it highly reduces your query time. Using this feature, all the queries/searches/aggregation you need to perform on, would require a term filter which would specify the condition you are trying to do in step 1. Now, whatever other operations you want to do, you can do it in the same query on the already shrinked dataset.
If you have other different use cases, then the storage of document(mapping) might be considered to get changed for easier and faster retrieval.
This is a current workaround that I use:
Run this bash script to save the first query ip-list to a temp index, then use a terms-query filter (in Kibana) to query using the ip-list from step1.
#!/usr/bin/env bash
es_host='https://************'
elk_user='************'
cred=($(pass ELK/************ | tr "\n" " ")) ##password
index_name='iis-************'
index_hostname='"************"'
temp_index_path='temp1/_doc/1'
results_limit=1000
timestamp_gte='"2018-03-20T13:00:00"' #UTC
timestamp_lte='"now"' #UTC
resp_data="$(curl -X POST $es_host/$index_name/_search -u $elk_user:${cred[0]} -H 'Content-Type: application/json; charset=utf-8' -d #- << EOF
{
"query": {
"bool": {
"must": [{
"match": {
"index_hostname": {
"query": $index_hostname
}
}
},
{
"regexp": {
"iis.access.url":{
"value": ".*((jpg)|(jpeg)|(png))"
}
}
}],
"must_not": {
"match": {
"iis.access.agent": {
"query": "Amazon+CloudFront"
}
}
},
"filter": {
"range": {
"#timestamp": {
"gte": $timestamp_gte,
"lte": $timestamp_lte
}
}
}
}
},
"aggs" : {
"whatever" : {
"terms" : { "field" : "iis.access.remote_ip", "size":$results_limit }
}
},
"size" : 0
}
EOF
)"
ip_list="$(echo "$resp_data" | jq '.aggregations.whatever.buckets[].key' | tr "\n" ",\ " | head -c -1)"
resp_data2="$(curl -X PUT $es_host/$temp_index_path -u $elk_user:${cred[0]} -H 'Content-Type: application/json; charset=utf-8' -d #- << EOF
{
"ips" : [$ip_list]
}
EOF
)"
echo "$resp_data2"
Query DSL - "terms-query" filter:
{
"query": {
"terms": {
"iis.access.remote_ip": {
"id": "1",
"index": "temp1",
"path": "ips",
"type": "_doc"
}
}
}
}