Amazon Athena returning `HIVE_CURSOR_ERROR: HIVE_CURSOR_ERROR` on geo json - hive

I'm trying to read a geojson file using Amazon Athena.
The head of my input data looks like this:
{
"type": "FeatureCollection",
"name": "sql_statement",
"features": [
{ "type": "Feature", "properties": { "gridsize": 500.0, "big_gid": 353, "little_gid": 22482 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -981739.267883020918816, 30855.609566356935829 ], [ -981739.267883020918816, 31355.354737498135364 ], [ -981241.022986860014498, 31355.354737498135364 ], [ -981241.022986860014498, 30855.609566356935829 ], [ -981739.267883020918816, 30855.609566356935829 ] ] ] } },
{ "type": "Feature", "properties": { "gridsize": 500.0, "big_gid": 353, "little_gid": 22483 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -981241.022986860014498, 30855.609566356935829 ], [ -981241.022986860014498, 31355.354737498135364 ], [ -980742.778090699226595, 31355.354737498135364 ], [ -980742.778090699226595, 30855.609566356935829 ], [ -981241.022986860014498, 30855.609566356935829 ] ] ] } },
{ "type": "Feature", "properties": { "gridsize": 500.0, "big_gid": 353, "little_gid": 22484 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -980742.778090699226595, 30855.609566356935829 ], [ -980742.778090699226595, 31355.354737498135364 ], [ -980244.533194538322277, 31355.354737498135364 ], [ -980244.533194538322277, 30855.609566356935829 ], [ -980742.778090699226595, 30855.609566356935829 ] ] ] } },
{ "type": "Feature", "properties": { "gridsize": 500.0, "big_gid": 353, "little_gid": 22485 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -980244.533194538322277, 30855.609566356935829 ], [ -980244.533194538322277, 31355.354737498135364 ], [ -979746.288298377417959, 31355.354737498135364 ], [ -979746.288298377417959, 30855.609566356935829 ], [ -980244.533194538322277, 30855.609566356935829 ] ] ] } },
{ "type": "Feature", "properties": { "gridsize": 500.0, "big_gid": 353, "little_gid": 22486 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -979746.288298377417959, 30855.609566356935829 ], [ -979746.288298377417959, 31355.354737498135364 ], [ -979248.043402216513641, 31355.354737498135364 ], [ -979248.043402216513641, 30855.609566356935829 ], [ -979746.288298377417959, 30855.609566356935829 ] ] ] } },
{ "type": "Feature", "properties": { "gridsize": 500.0, "big_gid": 353, "little_gid": 22487 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -979248.043402216513641, 30855.609566356935829 ], [ -979248.043402216513641, 31355.354737498135364 ], [ -978749.798506055609323, 31355.354737498135364 ], [ -978749.798506055609323, 30855.609566356935829 ], [ -979248.043402216513641, 30855.609566356935829 ] ] ] } },
and I defined the table using the geospatial documentation as a template:
CREATE external TABLE IF NOT EXISTS testdb.grid_500
(
gridsize double,
big_gid int,
little_gid int,
geometry binary
)
ROW FORMAT SERDE 'com.esri.hadoop.hive.serde.JsonSerde'
STORED AS INPUTFORMAT 'com.esri.json.hadoop.EnclosedJsonInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://jdl-athena/grid/'
;
Any query I run against the table, however, returns the same, rather unhelpful, error:
Your query has the following error(s):
HIVE_CURSOR_ERROR: HIVE_CURSOR_ERROR
This query ran against the "testdb" database, unless qualified by the query.
Please post the error message on our forum or contact customer support with
Query Id: 25d3da93-5cfd-46bb-9f77-4eb014679ba6.
Any ideas why I might be receiving this error or how I might go about debugging it? I can't even figure out how to get enough info to diagnose what might be happening.

So after much Google Fu, I have discovered that (of course) ESRI has created their own JSON format for geographic data. I am using the open standard GeoJson and the ESRI format is called ESRI Enclosed JSON. For geometric JSON, Athena supports only the ESRI format, not GEOJSON.
I discovered this distinction here:
https://github.com/Esri/gis-tools-for-hadoop/issues/29
So it looks like I need my input data in a different format.

Related

VEGA Sunburst using table based raw data

I am working on a Sunburst that would eventually take input data without a configured parent-child structure but a column based data so I employed a nest transform instead of a stratify as in the given example:
https://vega.github.io/editor/#/gist/c1eeb9142fd7f611513f5a4edf7e180e/spec.json
The problem with it, is that it generates internal nodes even if some fields are empty as long as my data object got value available for leaf nodes.
How can i transform my data to get a visual as in the following example
https://vega.github.io/editor/#/gist/112c45cfed3b9c9013ea0b63a318292f/spec.json
The idea is to "transform" the column based data into a parent, child structure in order to apply a Stratify transform.
Starting data:
{
"value": 140,
"level1": "France",
"level2": "Reins",
"level3": "Jean Monnet",
"level4": "rue 3 ",
"level5": "no",
"level6": "a level"
}
First group by all the levels or depth of the sunburst according to your value or quantity.
"transform": [
{
"type": "aggregate",
"groupby": [
"level1",
"level2",
"level3",
"level4",
"level5",
"level6"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
}
]
Secondly, we need to have a root note for stratify, so based on available data we can transform our data as following to define our root node having level1 elements as child nodes:
{
"name": "isolatedLevel1",
"source": [
"mainData"
],
"transform": [
{
"type": "project",
"fields": [
"level1",
"currentNode",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "aggregate",
"groupby": [
"parent",
"currentNode"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
},
{
"type": "formula",
"as": "parent",
"expr": "!isDefined(datum.parent)?'rootID':datum.parent"
},
{
"type": "project",
"fields": [
"currentNode",
"parent",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "filter",
"expr": "datum.currentNode"
}
]
},
{
"name": "rootNode",
"source": "isolatedLevel1",
"transform": [
{
"type": "project",
"fields": [
"parent",
"level2"
],
"as": [
"currentNode",
"parent"
]
},
{
"type": "aggregate",
"groupby": ["currentNode","parent"]
},
{
"type": "project",
"fields": [
"currentNode",
"parent"
],
"as": [
"currentNode",
"parent"
]
}
]
},
{
"name": "isolatedLevel1MergedWithParent",
"source": [
"isolatedLevel1",
"rootNode"
]
},
Then, the idea is to determine for the upcoming levels, the parent and child between 2 levels.
We can apply this data transform and repeat it till the last level of our data.
{
"name": "isolatedLevel2",
"source": "mainData",
"transform": [
{
"type": "project",
"fields": [
"level1",
"level2",
"value"
],
"as": [
"parent",
"currentNode",
"value"
]
},
{
"type": "aggregate",
"groupby": [
"parent",
"currentNode"
],
"fields": [
"value"
],
"ops": [
"sum"
],
"as": [
"value"
]
},
{
"type": "project",
"fields": [
"currentNode",
"parent",
"value"
],
"as": [
"currentNode",
"parent",
"value"
]
},
{
"type": "filter",
"expr": "datum.currentNode"
},
{
"type": "filter",
"expr": "datum.parent"
}
]
}
Finally, we merge all these "decomposed" transforms all together to get back our data, transformed into a parent child hierarchy, with values associated to its nodes only. This is a way to suit your data to a Stratify VEGA transform.
{
"name": "mergedlevelsFinal",
"source": [
"isolatedLevel1",
"isolatedLevel2",
"isolatedLevel3",
"isolatedLevel4",
"isolatedLevel5",
"isolatedLevel6",
"rootNode"
],
"transform": [
{
"type": "formula",
"as": "value",
"expr": "!indata('parents','parent',datum.currentNode)? datum.value:''"
}
]
}
The hard part gone, we can now apply our stratify to the transformed data as we want for our sunburst.
{
"name": "tree",
"source": [
"mergedlevelsFinal"
],
"transform": [
{
"type": "stratify",
"key": "currentNode",
"parentKey": "parent"
},
{
"type": "partition",
"field": "value",
"sort": {
"field": "value"
},
"size": [
{
"signal": "2 * PI"
},
{
"signal": "width / 3"
}
],
"as": [
"a0",
"r0",
"a1",
"r1",
"depth",
"children"
]
},
{
"type": "formula",
"expr": "!datum.children?datum.value:''",
"as": "value"
}
]
}
Full Spec Solution available here https://vega.github.io/editor/#/gist/0f66d06894b61f8ba131f13f564e8c1f/spec.json

Postgresql using json_build_obj

I'm trying to get a column of geometries from a table with PostgreSQL and PostGIS
SELECT json_build_object(
'type', 'FeatureCollection',
'features', json_agg(ST_AsGeoJSON(t.*)::json)
) FROM (SELECT position FROM permit) as t
This returns an Array with a key json_build_object.
[
{
"json_build_object": {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-97.74318277,
30.25239259
]
},
"properties": {}
}
]
}
}
]
I would like a single FeatureCollection object.
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
-97.74318277,
30.25239259
]
},
"properties": {}
}
]
}
Is this the expected output of the query? And how can I get a clean FeatureCollection otherwise?

How to add a tooltip to a d3.js path?

I am building a floormap from geojson data using d3.js topo and d3.js path. For clarification, I am also using vue.js. I want to add a tooltip when the user hovers over a room (aka the d3.js path). First I added just a console log to when the user hovers over a path but that did not work. I noticed that every time I load the app it does a console log but not when a user clicks / hovers over the d3.js path. I heard someone say that I would have to create an invisible circle or rectangle which would have the tooltip property bind to it but I don't think that route would work once the floor maps get complex. I dont care about adding any actual data to the tooltip right now but later I would want to. Can someone point me in the right direction, please? Thank you.
const data = {
"type": "FeatureCollection",
"features": [{
"type": "Feature",
"properties": {},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
0, 0
],
[
0, 11.4
],
[
7, 11.4
],
[
7, 0
],
[
0, 0
]
]
]
}
},
{
"type": "Feature",
"properties": {},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
7, 0
],
[
7, 11.4
],
[
12, 11.4
],
[
12, 0
],
[
7, 0
]
]
]
}
},
{
"type": "Feature",
"properties": {},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
12, 0
],
[
12, 11.4
],
[
19, 11.4
],
[
19, 0
],
[
12, 0
]
]
]
}
}
]
};
var svg = d3.select("svg")
var width = +svg.attr("width")
var height = +svg.attr("height")
svg.attr("transform", "translate(" + width / 2 + ",0)")
var projection = d3.geoIdentity().fitSize([width, height], data)
var path = d3.geoPath(projection)
svg.selectAll("path")
.data(data.features)
.enter()
.append("path")
.attr("d", path)
.attr("fill", "grey")
.attr("fill-opacity", .2)
.attr("stroke", "black")
.attr("stroke-width", 1.5)
.on("mouseover", console.log("hello"));
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>
<svg width="400" height="200"></svg>
There are several ways to do this. The simplest - but most limited - one is to create a <title> element in each path. It will show on hover, as it's browser native.
const data = {
"type": "FeatureCollection",
"features": [{
"type": "Feature",
"properties": { "name": "Kitchen" },
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
0, 0
],
[
0, 11.4
],
[
7, 11.4
],
[
7, 0
],
[
0, 0
]
]
]
}
},
{
"type": "Feature",
"properties": { "name": "Living room" },
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
7, 0
],
[
7, 11.4
],
[
12, 11.4
],
[
12, 0
],
[
7, 0
]
]
]
}
},
{
"type": "Feature",
"properties": { "name": "Toilet" },
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
12, 0
],
[
12, 11.4
],
[
19, 11.4
],
[
19, 0
],
[
12, 0
]
]
]
}
}
]
};
var svg = d3.select("svg")
var width = +svg.attr("width")
var height = +svg.attr("height")
svg.attr("transform", "translate(" + width / 2 + ",0)")
var projection = d3.geoIdentity().fitSize([width, height], data)
var path = d3.geoPath(projection)
svg.selectAll("path")
.data(data.features)
.enter()
.append("path")
.attr("d", path)
.attr("fill", "grey")
.attr("fill-opacity", .2)
.attr("stroke", "black")
.attr("stroke-width", 1.5)
.append("title")
// Do this to maintain access to the features you drew
.datum(function(d) { return d; })
.text(function(d) {
return d.properties.name;
});
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>
<svg width="400" height="200"></svg>
More complicated ones can use an absolutely positioned div, for example:
const data = {
"type": "FeatureCollection",
"features": [{
"type": "Feature",
"properties": {
"name": "Kitchen"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
0, 0
],
[
0, 11.4
],
[
7, 11.4
],
[
7, 0
],
[
0, 0
]
]
]
}
},
{
"type": "Feature",
"properties": {
"name": "Living room"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
7, 0
],
[
7, 11.4
],
[
12, 11.4
],
[
12, 0
],
[
7, 0
]
]
]
}
},
{
"type": "Feature",
"properties": {
"name": "Toilet"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
12, 0
],
[
12, 11.4
],
[
19, 11.4
],
[
19, 0
],
[
12, 0
]
]
]
}
}
]
};
var svg = d3.select("svg")
var width = +svg.attr("width")
var height = +svg.attr("height")
var tooltip = d3.select("#tooltip")
svg.attr("transform", "translate(" + width / 2 + ",0)")
var projection = d3.geoIdentity().fitSize([width, height], data)
var path = d3.geoPath(projection)
svg.selectAll("path")
.data(data.features)
.enter()
.append("path")
.attr("d", path)
.attr("fill", "grey")
.attr("fill-opacity", .2)
.attr("stroke", "black")
.attr("stroke-width", 1.5)
.on("mousemove", function(d) {
// +3 as some offset to make sure spawning the tooltip doesn't
// accidentally also cause unhover and thus removing itself
tooltip
.html(d.properties.name)
.style("display", "block")
.style("left", d3.event.x + 3 + 'px')
.style("top", d3.event.y + 3 + 'px');
})
.on("mouseleave", function() {
tooltip
.style("display", null)
.style("left", null)
.style("top", null);
});
#tooltip {
position: absolute;
top: 0;
left: 0;
display: none;
border: solid 1px red;
padding: 5px;
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/d3/5.7.0/d3.min.js"></script>
<svg width="400" height="200"></svg>
<div id="tooltip"></div>

turf.within() fails but turf.inside() is successful

I am trying to use turf.js to return points within a polygon, but turf.within() generates an empty feature list even for points I know to be within a polygon - turf.inside() returns true for the same points and polygon. I provide an example below. I can successfully run the example at turf.within() and am not sure what it is about my geometry that is the problem.
<script type="text/javascript" src="https://npmcdn.com/#turf/turf/turf.min.js"></script>
....
var thePoint = [-118.049551, 33.866965]
var test = {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {},
"geometry": {
"type": "Point",
"coordinates": thePoint
}
}, {
"type": "Feature",
"properties": {},
"geometry": {
"type": "Point",
"coordinates": thePoint
}
}
]
};
var cerrito = { "type": "Feature", "properties": { "GEO_ID": "1600000US0612552", "STATE": "06", "PLACE": "12552", "NAME": "Cerritos", "LSAD": "city", "CENSUSAREA": 8.725, "Shape_STAr": 246572616.215, "Shape_STLe": 118080.76087 }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ -118.028887999836968, 33.882629000032836 ], [ -118.028681155924062, 33.873316057140322 ], [ -118.028772420533898, 33.873315497179512 ], [ -118.028772240680681, 33.873285861714471 ], [ -118.028761755934283, 33.872110544548683 ], [ -118.028717388339928, 33.866241889545748 ], [ -118.028792999919617, 33.866240999737094 ], [ -118.035119980483032, 33.866020314132811 ], [ -118.035591866241305, 33.866016576824734 ], [ -118.035818357122793, 33.866015383719059 ], [ -118.035983055259194, 33.866014509762415 ], [ -118.036147752309518, 33.8660136310734 ], [ -118.036148847050185, 33.866013630744973 ], [ -118.037598000415159, 33.866012999764315 ], [ -118.037597881676277, 33.866006075268416 ], [ -118.037624048408333, 33.866005950805395 ], [ -118.037575372135976, 33.86388909386703 ], [ -118.037562273006358, 33.863314044528884 ], [ -118.037540454019165, 33.86236905889217 ], [ -118.037606327718564, 33.862368507358319 ], [ -118.037979218998117, 33.862365367620129 ], [ -118.039225723817765, 33.862354938919154 ], [ -118.039350888987059, 33.862353881198253 ], [ -118.039574899257346, 33.862352021930512 ], [ -118.039782079613744, 33.862350277788913 ], [ -118.039963633664883, 33.86234876086143 ], [ -118.040128290807047, 33.862347360175178 ], [ -118.040309482829713, 33.862345842847105 ], [ -118.040474501989308, 33.862344468612086 ], [ -118.040655693999881, 33.862342951664914 ], [ -118.040820383551591, 33.862341577067333 ], [ -118.041001575536058, 33.862340031653119 ], [ -118.041166594672802, 33.862338656470193 ], [ -118.041522357556943, 33.86233567659459 ], [ -118.041555328956449, 33.86233539026582 ], [ -118.04158826686961, 33.86233513188656 ], [ -118.041667975939376, 33.862334444439369 ], [ -118.041884905794092, 33.862332720355347 ], [ -118.041879045435579, 33.861679208477895 ], [ -118.041878661410877, 33.861635133586446 ], [ -118.041877779312557, 33.861530668241585 ], [ -118.041877117333357, 33.861453668349675 ], [ -118.041875622573116, 33.86128871703405 ], [ -118.041874434390664, 33.861151243763409 ], [ -118.04187324622599, 33.86101379844164 ], [ -118.041872056970632, 33.860876325175354 ], [ -118.041870868813831, 33.86073887985755 ], [ -118.041869613384449, 33.860598713795902 ], [ -118.041868145114535, 33.860428321848957 ], [ -118.04186653720933, 33.860242294996745 ], [ -118.041865207827939, 33.860088170827417 ], [ -118.041854341147129, 33.858838905318855 ], [ -118.041854202456747, 33.8588251662961 ], [ -118.041853336146318, 33.858724067229424 ], [ -118.046188000416265, 33.858768000030139 ], [ -118.046186748924541, 33.858687270846296 ], [ -118.046200389259099, 33.858687186283355 ], [ -118.046186014574644, 33.857785576122218 ], [ -118.046163305133547, 33.856407842019244 ], [ -118.046152668322023, 33.855664249739483 ], [ -118.046110568906883, 33.853648342941831 ], [ -118.04611084525645, 33.853610518941778 ], [ -118.047386006877716, 33.853589266798537 ], [ -118.048728911992114, 33.853575817845297 ], [ -118.049101279351319, 33.853572084045524 ], [ -118.049609117085879, 33.853568105873244 ], [ -118.05041069369274, 33.85356182224956 ], [ -118.050410420346836, 33.853533108261765 ], [ -118.050405543181043, 33.853033914419463 ], [ -118.050399505365675, 33.852413871433946 ], [ -118.050398863336483, 33.852345210650313 ], [ -118.050398171596498, 33.852276536365281 ], [ -118.050395364025476, 33.851988181322085 ], [ -118.050394029188837, 33.851850845358634 ], [ -118.0503926949755, 33.851712741307736 ], [ -118.050392161744654, 33.85165789437027 ], [ -118.050391361261092, 33.851575460340229 ], [ -118.050390027518588, 33.85143812528397 ], [ -118.050388692733023, 33.851300845222546 ], [ -118.050387326600486, 33.851163537229304 ], [ -118.050386025351628, 33.851026311248653 ], [ -118.050383464434944, 33.850765297230296 ], [ -118.050382854871259, 33.850696677914939 ], [ -118.050382130718276, 33.850627962186074 ], [ -118.050379462200738, 33.850353264157327 ], [ -118.050375711015633, 33.849966976406435 ], [ -118.053892316950595, 33.849848893473471 ], [ -118.054516818610182, 33.849844709837626 ], [ -118.054590588300428, 33.849844214857512 ], [ -118.054722683312946, 33.849843325535495 ], [ -118.054722478592268, 33.849821017688754 ], [ -118.054723000454345, 33.849821000322997 ], [ -118.054716601109988, 33.84918063060708 ], [ -118.054705566290011, 33.84797815444341 ], [ -118.054700499743277, 33.847440996646093 ], [ -118.054689180780002, 33.846217950124199 ], [ -118.054776697011036, 33.846215943261306 ], [ -118.056046115878445, 33.846208952311493 ], [ -118.057043290604042, 33.846203779261714 ], [ -118.058278859805654, 33.846196339066665 ], [ -118.059040364775043, 33.846192198135391 ], [ -118.059040016006662, 33.846124967096777 ], [ -118.060671000051059, 33.846177999563118 ], [ -118.06338900026266, 33.84615300022368 ], [ -118.065990999985502, 33.846131999565429 ], [ -118.072090999789609, 33.846084000390604 ], [ -118.081980999696881, 33.84600600023699 ], [ -118.08202099972182, 33.849640999592488 ], [ -118.086370999820048, 33.84963300020425 ], [ -118.086368999500323, 33.845970999859297 ], [ -118.086803999550057, 33.845968999565606 ], [ -118.092398000044483, 33.845925000070416 ], [ -118.098939000208091, 33.845875000320532 ], [ -118.09921100033479, 33.845875999972094 ], [ -118.100211999558695, 33.848065999601282 ], [ -118.101733000020189, 33.853478000147867 ], [ -118.103054999461108, 33.858318000193101 ], [ -118.108316999981298, 33.858371000240467 ], [ -118.108428999611235, 33.865586999998889 ], [ -118.108536000018645, 33.872715000058548 ], [ -118.106910000099688, 33.872711000330419 ], [ -118.106974000434647, 33.872942999957047 ], [ -118.108003999515034, 33.876790000315744 ], [ -118.108546999473432, 33.87984299979373 ], [ -118.108563999545112, 33.886942999906346 ], [ -118.108417999623612, 33.886938999677412 ], [ -118.108160000123547, 33.886939999577628 ], [ -118.104370000310539, 33.886968000366373 ], [ -118.10425600018489, 33.887381999848436 ], [ -118.100572999781591, 33.887395000134291 ], [ -118.100547000184832, 33.886982999787215 ], [ -118.09988300024132, 33.886990000446659 ], [ -118.09988699995877, 33.88740200023738 ], [ -118.095532999746752, 33.887425999876328 ], [ -118.095537999774862, 33.88211299995568 ], [ -118.095466999498029, 33.880314000301986 ], [ -118.09541300051032, 33.87852799995575 ], [ -118.09380399970361, 33.87855300002056 ], [ -118.091064999993833, 33.878859999883623 ], [ -118.091085999524168, 33.88032599969636 ], [ -118.087806000080136, 33.880338000092316 ], [ -118.087776999582886, 33.876750000365199 ], [ -118.091033999660468, 33.876416000198674 ], [ -118.090991999619234, 33.8729809995612 ], [ -118.090990999805427, 33.865683999854205 ], [ -118.090901000460704, 33.861313000431181 ], [ -118.089982999574389, 33.861319000332337 ], [ -118.089973999821112, 33.860224999604107 ], [ -118.086482999908952, 33.860250000054762 ], [ -118.086474999522395, 33.85842399971353 ], [ -118.086450000287186, 33.856003999941251 ], [ -118.082092000279232, 33.856040000312333 ], [ -118.082110999618877, 33.860495000048402 ], [ -118.078810999737044, 33.85847399969186 ], [ -118.075151000444947, 33.856228000446528 ], [ -118.072288000163809, 33.856968999864691 ], [ -118.072240999730425, 33.854512000252328 ], [ -118.070459000360898, 33.853411000398474 ], [ -118.068915000458702, 33.853423000135727 ], [ -118.069006999624165, 33.858557000390903 ], [ -118.071223999523767, 33.858534999730935 ], [ -118.070178000376117, 33.860339000416801 ], [ -118.070294000094378, 33.865815999626136 ], [ -118.070313999561719, 33.869456999865655 ], [ -118.0724970001271, 33.869453999697591 ], [ -118.070334999534126, 33.869835999844774 ], [ -118.070384000066582, 33.873111000077408 ], [ -118.08004299960642, 33.873053999696083 ], [ -118.07964899968843, 33.876462000326832 ], [ -118.07927399950583, 33.87674000007636 ], [ -118.075086999544695, 33.8767670000233 ], [ -118.074840999957502, 33.876769000254562 ], [ -118.074895000371953, 33.880383000438925 ], [ -118.072708000101883, 33.880391999903772 ], [ -118.072825000397017, 33.885832999950722 ], [ -118.068447000489812, 33.885887999807679 ], [ -118.06846999965768, 33.887112000073593 ], [ -118.068475999850449, 33.887559000148578 ], [ -118.066323999719302, 33.887571000149549 ], [ -118.06528799964515, 33.887577000140304 ], [ -118.064333000067364, 33.887581999621858 ], [ -118.06408499971505, 33.887165000021376 ], [ -118.055376000176864, 33.887216000217663 ], [ -118.055382999695453, 33.887649999683752 ], [ -118.046538999840678, 33.887713000322073 ], [ -118.038572000150765, 33.88777300012908 ], [ -118.028887999836968, 33.882629000032836 ] ] ] ] } }
var cerritoPolygon = turf.polygon(cerrito.geometry.coordinates[0]) //turn cerrito coordinates into polygon
var cerritoFeatureCollection = turf.featureCollection(cerrito) //create a feature collection
var ptsWithin = turf.within(test, cerritoFeatureCollection); //see if test points are within the cerrito featureCollection
var pt = turf.point(thePoint);
var isInside = turf.inside(pt, cerritoPolygon);
console.log(ptsWithin)//returns a featureCollection with an empty features array
console.log(isInside)//returns true - thePoint is in the cerrito polygon

Store and query extremely large amount of JSON data

Context:
300000 lines of JSON data are generated approximately every five seconds by a game's remote public API. This data disappears immediately after the next update and is never accessible again. I want to be able to save compressed historical records of this data, which can be reliably and efficiently indexed and searched.
Question:
What is the best database system to store JSON data on this scale, considering possible compression methods and most importantly query efficiency?
EDIT Adding example unpacked JSON (one object out of around 30000):
{
"accountName": "Kabancheg11",
"lastCharacterName": "KabanNeedBuff",
"id": "427d66d977b5a63af08b3c39173a1a567eb783fde6d6f97df505dbe6c1fa5988",
"stash": "~b/o 30 chaos",
"stashType": "PremiumStash",
"items": [
{
"verified": false,
"w": 1,
"h": 1,
"ilvl": 71,
"icon": "http://web.poecdn.com/image/Art/2DItems/Amulets/Amulet37.png?scale=1&w=1&h=1&v=25bebeac11abfd9f7931d574765521093",
"league": "Hardcore",
"id": "a7bb36a28e55865b9d766dcebd5bc0c6893d5ecb6cd9870dc0bcd3c9f1164f29",
"sockets": [],
"name": "",
"typeLine": "Onyx Amulet",
"identified": false,
"corrupted": false,
"lockedToCharacter": false,
"implicitMods": [
"+15 to all Attributes"
],
"frameType": 3,
"x": 11,
"y": 0,
"inventoryId": "Stash1",
"socketedItems": []
},
{
"verified": false,
"w": 1,
"h": 3,
"ilvl": 75,
"icon": "http://web.poecdn.com/image/Art/2DItems/Weapons/OneHandWeapons/Daggers/Heartbreaker.png?scale=1&w=1&h=3&v=b4f18d5c0602a17381df6a45b4c57be33",
"league": "Hardcore",
"id": "c955f054ffae9439e5d257c4a444b02423b2dbe304d1fa581227d21a0f203aa9",
"sockets": [
{
"group": 0,
"attr": "D"
},
{
"group": 0,
"attr": "D"
}
],
"name": "<<set:MS>><<set:M>><<set:S>>Heartbreaker",
"typeLine": "Royal Skean",
"identified": true,
"corrupted": false,
"lockedToCharacter": false,
"properties": [
{
"name": "Dagger",
"values": [],
"displayMode": 0
},
{
"name": "Physical Damage",
"values": [
[
"15-59",
0
]
],
"displayMode": 0
},
{
"name": "Critical Strike Chance",
"values": [
[
"6.60%",
0
]
],
"displayMode": 0
},
{
"name": "Attacks per Second",
"values": [
[
"1.45",
0
]
],
"displayMode": 0
}
],
"requirements": [
{
"name": "Level",
"values": [
[
"50",
0
]
],
"displayMode": 0
},
{
"name": "Dex",
"values": [
[
"71",
0
]
],
"displayMode": 1
},
{
"name": "Int",
"values": [
[
"102",
0
]
],
"displayMode": 1
}
],
"implicitMods": [
"40% increased Global Critical Strike Chance"
],
"explicitMods": [
"45% increased Spell Damage",
"+50 to maximum Mana",
"+50 to maximum Energy Shield",
"10% faster start of Energy Shield Recharge",
"Your Spells have Culling Strike"
],
"flavourText": [
"A heart can be stabbed by thoughts, \r",
"If your mind is sharp enough."
],
"frameType": 3,
"x": 0,
"y": 9,
"inventoryId": "Stash2",
"socketedItems": []
},
{
"verified": false,
"w": 2,
"h": 3,
"ilvl": 75,
"icon": "http://web.poecdn.com/image/Art/2DItems/Weapons/OneHandWeapons/Scepters/MontregulsGrasp.png?scale=1&w=2&h=3&v=fed985831c32e8e25b7663608a2b7a5c3",
"league": "Hardcore",
"id": "a78e6bfff62145c78b92714cc7de10e4484c42745d48b585ef905b3972c2a319",
"sockets": [
{
"group": 0,
"attr": "S"
},
{
"group": 0,
"attr": "S"
}
],
"name": "",
"typeLine": "Void Sceptre",
"identified": false,
"corrupted": false,
"lockedToCharacter": false,
"properties": [
{
"name": "One Handed Mace",
"values": [],
"displayMode": 0
},
{
"name": "Physical Damage",
"values": [
[
"42-63",
0
]
],
"displayMode": 0
},
{
"name": "Critical Strike Chance",
"values": [
[
"6.50%",
0
]
],
"displayMode": 0
},
{
"name": "Attacks per Second",
"values": [
[
"1.25",
0
]
],
"displayMode": 0
}
],
"requirements": [
{
"name": "Str",
"values": [
[
"104",
0
]
],
"displayMode": 1
},
{
"name": "Int",
"values": [
[
"122",
0
]
],
"displayMode": 1
}
],
"implicitMods": [
"15% increased Elemental Damage"
],
"frameType": 3,
"x": 0,
"y": 6,
"inventoryId": "Stash3",
"socketedItems": []
},
{
"verified": false,
"w": 2,
"h": 4,
"ilvl": 74,
"icon": "http://web.poecdn.com/image/Art/2DItems/Weapons/TwoHandWeapons/TwoHandAxes/TwoHandAxe11Unique.png?scale=1&w=2&h=4&v=49f4f7aff220d356fc6300d7454df63a3",
"league": "Hardcore",
"id": "c8d722c909939d1fda4431722c732e7af4c17781d9d9c58da613bf776ea1e993",
"sockets": [
{
"group": 0,
"attr": "D"
}
],
"name": "",
"typeLine": "Superior Karui Chopper",
"identified": false,
"corrupted": false,
"lockedToCharacter": false,
"properties": [
{
"name": "Two Handed Axe",
"values": [],
"displayMode": 0
},
{
"name": "Quality",
"values": [
[
"+10%",
1
]
],
"displayMode": 0
},
{
"name": "Physical Damage",
"values": [
[
"88-138",
1
]
],
"displayMode": 0
},
{
"name": "Critical Strike Chance",
"values": [
[
"5.00%",
0
]
],
"displayMode": 0
},
{
"name": "Attacks per Second",
"values": [
[
"1.15",
0
]
],
"displayMode": 0
}
],
"requirements": [
{
"name": "Str",
"values": [
[
"151",
0
]
],
"displayMode": 1
},
{
"name": "Dex",
"values": [
[
"43",
0
]
],
"displayMode": 1
}
],
"frameType": 3,
"x": 2,
"y": 0,
"inventoryId": "Stash4",
"socketedItems": []
},
{
"verified": false,
"w": 2,
"h": 3,
"ilvl": 51,
"icon": "http://web.poecdn.com/image/Art/2DItems/Armours/BodyArmours/StrInt3AUnique2.png?scale=1&w=2&h=3&v=3ef1aa6bc50b635ca334653ca95485a63",
"league": "Hardcore",
"id": "26c55af00fb31f63bec7229b622c3c79fb6efcfa03d3d2595672a92778d04d99",
"sockets": [
{
"group": 0,
"attr": "I"
},
{
"group": 1,
"attr": "I"
}
],
"name": "<<set:MS>><<set:M>><<set:S>>Ambu's Charge",
"typeLine": "Crusader Chainmail",
"identified": true,
"corrupted": false,
"lockedToCharacter": false,
"properties": [
{
"name": "Armour",
"values": [
[
"477",
1
]
],
"displayMode": 0
},
{
"name": "Energy Shield",
"values": [
[
"141",
1
]
],
"displayMode": 0
}
],
"requirements": [
{
"name": "Level",
"values": [
[
"43",
0
]
],
"displayMode": 0
},
{
"name": "Str",
"values": [
[
"64",
0
]
],
"displayMode": 1
},
{
"name": "Int",
"values": [
[
"64",
0
]
],
"displayMode": 1
}
],
"explicitMods": [
"182% increased Armour and Energy Shield",
"+15% to all Elemental Resistances",
"Gain an Endurance Charge when you take a Critical Strike",
"2% of Life Regenerated per second while on Low Life",
"Share Endurance Charges with nearby party members"
],
"flavourText": [
"Nothing stops the pain like a courageous rush into battle."
],
"frameType": 3,
"x": 0,
"y": 0,
"inventoryId": "Stash5",
"socketedItems": []
},
{
"verified": false,
"w": 1,
"h": 1,
"ilvl": 64,
"icon": "http://web.poecdn.com/image/Art/2DItems/Jewels/basicint.png?scale=1&w=1&h=1&v=cd579ea22c05f1c6ad2fd015d7a710bd3",
"league": "Hardcore",
"id": "34103c553f0ae3e36c695664a7540dd6e547338596c33554ca3a8443dd5349c1",
"sockets": [],
"name": "<<set:MS>><<set:M>><<set:S>>Cataclysm Heart",
"typeLine": "Cobalt Jewel",
"identified": true,
"corrupted": false,
"lockedToCharacter": false,
"explicitMods": [
"2% increased Cast Speed",
"7% increased maximum Energy Shield",
"+13% to Lightning Resistance"
],
"descrText": "Place into an allocated Jewel Socket on the Passive Skill Tree. Right click to remove from the Socket.",
"frameType": 2,
"x": 0,
"y": 3,
"inventoryId": "Stash6",
"socketedItems": []
},
{
"verified": false,
"w": 1,
"h": 1,
"ilvl": 0,
"icon": "http://web.poecdn.com/image/Art/2DItems/Gems/VaalGems/VaalArc.png?scale=1&w=1&h=1&v=b4f32328e279496ebb227521e8dce6793",
"support": false,
"league": "Hardcore",
"id": "be9a23cfe0a53942b19c22da32b7d461b464838749110d04689e15498dcad86b",
"sockets": [],
"name": "",
"typeLine": "Vaal Arc",
"identified": true,
"corrupted": true,
"lockedToCharacter": false,
"properties": [
{
"name": "Vaal, Spell, Chaining, Lightning",
"values": [],
"displayMode": 0
},
{
"name": "Level",
"values": [
[
"20 (Max)",
0
]
],
"displayMode": 0
},
{
"name": "Mana Cost",
"values": [
[
"0",
0
]
],
"displayMode": 0
},
{
"name": "Souls Per Use",
"values": [
[
"32",
0
]
],
"displayMode": 0
},
{
"name": "Can Store %0 Use",
"values": [
[
"1",
0
]
],
"displayMode": 3
},
{
"name": "Cast Time",
"values": [
[
"0.80 sec",
0
]
],
"displayMode": 0
},
{
"name": "Critical Strike Chance",
"values": [
[
"5.00%",
0
]
],
"displayMode": 0
},
{
"name": "Damage Effectiveness",
"values": [
[
"80%",
0
]
],
"displayMode": 0
}
],
"requirements": [
{
"name": "Level",
"values": [
[
"70",
0
]
],
"displayMode": 0
},
{
"name": "Int",
"values": [
[
"155",
0
]
],
"displayMode": 1
}
],
"secDescrText": "An arc of lightning stretches from the caster to a targeted nearby enemy and chains on to many additional targets.",
"explicitMods": [
"Deals 49 to 938 Lightning Damage",
"Chain +40 Times",
"100% chance to Shock enemies"
],
"descrText": "Place into an item socket of the right colour to gain this skill. Right click to remove from a socket.",
"frameType": 4,
"x": 0,
"y": 4,
"inventoryId": "Stash7",
"socketedItems": []
}
],
"public": true
}
An example query could be asking how many of the items nested inside every object like the one shown above have an ilvl field greater than or equal to an integer.