I would like to transform a column of
array(map(varchar, varchar))
to string as rows of a table on presto db by pyspark hive sql programmatically from jupyter notebook python3.
example
user_id sport_ids
'aca' [ {'sport_id': '5818'}, {'sport_id': '6712'}, {'sport_id': '1065'} ]
expected results
user_id. sport_ids
'aca'. '5815'
'aca'. '5712'
'aca'. '1065'
I have tried
sql_q= """
select distinct, user_id, transform(sport_ids, x -> element_at(x, 'sport_id')
from tab """
spark.sql(sql_q)
but got error:
'->' cannot be resolved
I have also tried
sql_q= """
select distinct, user_id, sport_ids
from tab"""
spark.sql(sql_q)
but got error:
org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column request_features[0] is map<string,string>;;
Did I miss something ?
I have tried this, but helpful
hive convert array<map<string, string>> to string
Extract map(varchar, array(varchar)) - Hive SQL
thanks
Lets try use higher order functions to find map values and explode into individual rows
df.withColumn('sport_ids', explode(expr("transform(sport_ids, x->map_values(x)[0])"))).show()
+-------+---------+
|user_id|sport_ids|
+-------+---------+
| aca| 5818|
| aca| 6712|
| aca| 1065|
+-------+---------+
You can process json data (json_parse, cast to array of json and json_extract_scalar - for more json functions - see here) and flatten (unnest) on presto side:
-- sample data
WITH dataset(user_id, sport_ids) AS (
VALUES
('aca', '[ {"sport_id": "5818"}, {"sport_id": "6712"}, {"sport_id": "1065"} ]')
)
-- query
select user_id,
json_extract_scalar(record, '$.sport_id') sport_id
from dataset,
unnest(cast(json_parse(sport_ids) as array(json))) as t(record)
Output:
user_id
sport_id
aca
5818
aca
6712
aca
1065
I have a Postgres extended with Postgis version 2.5 database in Heroku.
I want to use the function:
ST_Area( a_polygon )
Specifically I want a generated column in my table:
alter table buildings add building_area float generated always as ( st_area( base_polygon ) ) stored;
Where base_polygon is of type polygon.
However, I am getting this error:
ERROR: function st_area(polygon) does not exist Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Aren't these commands supposed to be available after I run CREATE EXTENSION postgis?
Or, is there something else I have to do?
It seems your polygon column data type is postgre base built in polygon.
ST_Area expects postgis geometry type as a parameter.
As in this example from docs https://postgis.net/docs/ST_Area.html
select ST_Area(geom) sqft,
ST_Area(ST_Transform(geom, 26986)) As sqm
from (
select
'SRID=2249;POLYGON((743238 2967416,743238 2967450,
743265 2967450,743265.625 2967416,743238 2967416))' :: geometry
geom
) subquery;
Check if this example works, it means that ST_Area function exists.
You can add a column with postgis geometry type. https://postgis.net/docs/AddGeometryColumn.html
SELECT AddGeometryColumn ('my_schema','my_spatial_table','geom',4326,'POLYGON',2);
Then convert your polygons into postgis format, by postgis functions.
For example https://postgis.net/docs/ST_MakePolygon.html
SELECT ST_MakePolygon( ST_GeomFromText('LINESTRING(75 29,77 29,77 29, 75 29)'));
I have a simple table called"imposm3_restaurant" with columns [ id, name, geometry] I want to convert these data into geoJSON, I am using this function
CREATE VIEW imposm3_restaurants_geojson AS SELECT row_to_json(fc) AS geojson FROM
(SELECT 'FeatureCollection' As type, array_to_json(array_agg(f))
As features FROM
(SELECT
'Feature' As type,
ST_AsGeoJSON((lg.geometry),15,0)::json As geometry,
row_to_json((id, name)) As properties
FROM imposm3_restaurants As lg) As f ) As fc;
and the result is this:
{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"LineString","coordinates":[[2615020.47191046,5899232.25158985],[2615034.50527113,5899231.67978097],[2615033.86145338,5899215.4513157],[2615032.35921198,5899215.51938806],[2615031.96732292,5899205.64890158],[2615034.97180572,5899205.51275702],[2615034.36531075,5899190.07397728],[2615018.19522163,5899190.71385561],[2615018.77372453,5899205.40384137],[2615020.47191046,5899205.32215463],[2615020.91045298,5899216.48601561],[2615019.83742341,5899216.52685903],[2615020.47191046,5899232.25158985]]},"properties":{"f1":2719,"f2":"Atelierul de Pizza"}},{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[2615018.19522163,5899190.71385561],[2615018.77372453,5899205.40384137],[2615020.47191046,5899205.32215463],[2615020.91045298,5899216.48601561],[2615019.83742341,5899216.52685903],[2615020.47191046,5899232.25158985],[2615034.50527113,5899231.67978097],[2615033.86145338,5899215.4513157],[2615032.35921198,5899215.51938806],[2615031.96732292,5899205.64890158],[2615034.97180572,5899205.51275702],[2615034.36531075,5899190.07397728],[2615018.19522163,5899190.71385561]]]},"properties":{"f1":2720,"f2":"Atelierul de Pizza"}},{"type":"Feature","geometry":{"type":"LineString","coordinates":[[2624875.82864931,5903443.39761349],[2624897.49451598,5903452.78251964],[2624901.44139867,5903443.67003443],[2624879.78486269,5903434.29875908],[2624875.82864931,5903443.39761349]]},"properties":{"f1":2986,"f2":"Pizza Acrobatica"}},{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[2624875.82864931,5903443.39761349],[2624897.49451598,5903452.78251964],[2624901.44139867,5903443.67003443],[2624879.78486269,5903434.29875908],[2624875.82864931,5903443.39761349]]]},"properties":{"f1":2988,"f2":"Pizza Acrobatica"}},{"type":"Feature","geometry":{"type":"LineString","coordinates":[[2622460.22447654,5904586.41424973],[2622479.10046632,5904587.95362911],[2622480.25747212,5904573.81314552],[2622461.39081303,5904572.26014582],[2622460.22447654,5904586.41424973]]},"properties":{"f1":3248,"f2":"Casa Vikingilor"}},{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[2622460.22447654,5904586.41424973],[2622479.10046632,5904587.95362911],[2622480.25747212,5904573.81314552],[2622461.39081303,5904572.26014582],[2622460.22447654,5904586.41424973]]]},"properties":{"f1":3249,"f2":"Casa Vikingilor"}},{"type":"Feature","geometry":{"type":"LineString","coordinates":[[2625201.09657005,5897608.45120294],[2625224.46062264,5897614.30435379],[2625241.33051365,5897576.653689],[2625213.43174478,5897570.82778714],[2625201.09657005,5897608.45120294]]},"properties":{"f1":6152,"f2":"Silva"}},{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[2625201.09657005,5897608.45120294],[2625224.46062264,5897614.30435379],[2625241.33051365,5897576.653689],[2625213.43174478,5897570.82778714],[2625201.09657005,5897608.45120294]]]},"properties":{"f1":6153,"f2":"Silva"}},{"type":"Feature","geometry":{"type":"LineString","coordinates":[[2622825.25980629,5904372.27967993],[2622826.15555271,5904353.45341631],[2622834.51585268,5904353.1673446],[2622854.22227404,5904346.00193242],[2622860.03529512,5904362.26715407],[2622856.61093118,5904374.66361634],[2622825.25980629,5904372.27967993]]},"properties":{"f1":6322,"f2":"Restaurant Sinaia"}},{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[2622825.25980629,5904372.27967993],[2622856.61093118,5904374.66361634],[2622860.03529512,5904362.26715407],[2622854.22227404,5904346.00193242],[2622834.51585268,5904353.1673446],[2622826.15555271,5904353.45341631],[2622825.25980629,5904372.27967993]]]},"properties":{"f1":6323,"f2":"Restaurant Sinaia"}}]}
which does not have a current geometry, do you know what is wrong in function: I am using Postgres 9.3 and PostGIS 2.2
Your output is a valid geojson file but the geometries are projected using the projection EPSG:3857.
You can load the data without problems in the most gis desktop applications, in example Quantum Gis.
Probably geojson.io supports only long/lat coordinates EPSG:4326, also try reprojecting the geometries to long/lat coordinates using the function St_Transform
Change this line:
ST_AsGeoJSON((lg.geometry),15,0)::json As geometry,
in this:
ST_AsGeoJSON(ST_Transform(lg.geometry, 4326),15,0)::json As geometry,
Does anybody know what is the problem with my query. I am trying to calculate area using geographical coordinates, but result seems to be too small to be true. 0.00118 sqm. Can anybody help?
SELECT ST_Area(the_geom) As sqm
FROM (SELECT
ST_GeomFromText('POLYGON
(
(14.604514925547997 121.0968017578125,
14.595212295624522 121.08512878417969,
14.567302046916149 121.124267578125,
14.596541266841905 121.14761352539062,
14.604514925547997 121.0968017578125)
)',4326) ) As foo(the_geom)
How accurate should be the calculation?
A solution is to cast GEOMETRY to GEOGRAPHY, which is acceptably accurate for the most use cases:
SELECT ST_Area(the_geom::GEOGRAPHY ) As sqm
FROM (SELECT
ST_GeomFromText('POLYGON
(
(14.604514925547997 121.0968017578125,
14.595212295624522 121.08512878417969,
14.567302046916149 121.124267578125,
14.596541266841905 121.14761352539062,
14.604514925547997 121.0968017578125)
)',4326) ) As foo(the_geom)
The geography type automatically converts degrees to meters.
Depending on your scenario you could also use directly the geography constructor St_GeographyFromText, which accept a WKT string as argument, very similar to ST_GeomFromText
ST_GeographyFromText('POLYGON((14.604514925547997 121.0968017578125,
14.595212295624522 121.08512878417969,
14.567302046916149 121.124267578125,
14.596541266841905 121.14761352539062,
14.604514925547997 121.0968017578125))'
)
I have the following sample data and I am trying to explode it in hive.. I used split but I know I am missing something..
["[[-80.742426,35.23248],[-80.740424,35.23184],[-80.739583,35.231562],[-80.735935,35.23041],[-80.728624,35.228069],[-80.727753,35.227836],[-80.727294,35.227741],[-80.726762,35.227647],[-80.726321,35.227594],[-80.725687,35.227544],[-80.725134,35.227535],[-80.721502,35.227615],[-80.691298,35.216202],[-80.688009,35.215396],[-80.686516,35.215016],[-80.598433,35.234307]]"]
I used the below query
select explode(split(col, ',')) from sample2;
and the result is this
["[[-80.742426
35.23248]
[-80.740424
35.23184]
[-80.739583
35.231562]
[-80.735935
35.23041]
[-80.728624
35.228069]
[-80.727753
35.227836]
[-80.71143
35.227831]
[-80.711007
35.227795]
[-80.710638
35.227741]
[-80.673884
35.21014]
[-80.672358
35.209481]
[-80.672036
35.209356]
[-80.671686
35.209234]
[-80.67124
35.209099]
[-80.670815
35.209006]
[-80.670267
35.208906]
[-80.669612
35.208833]
[-80.668924
35.208806]
[-80.598433
35.234307]]"]
I need it in below format
[-80.742426,35.23248]
[-80.740424,35.23184]
[-80.739583,35.231562]
[-80.735935,35.23041]
[-80.728624,35.228069]
[-80.727753,35.227836]
[-80.727294,35.227741]
[-80.726762,35.227647]
[-80.726321,35.227594]
[-80.725687,35.227544]
[-80.725134,35.227535]
[-80.721502,35.227615]
[-80.691298,35.216202]
[-80.688009,35.215396]
[-80.686516,35.215016]
[-80.684281,35.214466]
[-80.68396,35.214395]
[-80.683375,35.214231]
[-80.682908,35.214079]
[-80.682444,35.213905]
[-80.682045,35.213733]
[-80.68062,35.213112]
[-80.678078,35.211983]
[-80.676836,35.211447]
[-80.598433,35.234307]
Any help over here..?
You have your data set as arrays of array and you want to explode your data at first level only, so use LATERAL VIEW explode(colname) to explode at the first level.
Below is the SELECT query with explode():
SELECT col1 FROM sample2 LATERAL VIEW EXPLODE(col) explodeVal AS col1;
output generated from your input data set as below:
[-80.742426,35.23248]
[-80.740424,35.23184]
[-80.739583,35.231562]
[-80.735935,35.23041]
[-80.728624,35.228069]
[-80.727753,35.227836]
[-80.727294,35.227741]
[-80.726762,35.227647]
[-80.726321,35.227594]
[-80.725687,35.227544]
[-80.725134,35.227535]
[-80.721502,35.227615]
[-80.691298,35.216202]
[-80.688009,35.215396]
[-80.686516,35.215016]
[-80.684281,35.214466]
[-80.68396,35.214395]
[-80.683375,35.214231]
[-80.682908,35.214079]
[-80.682444,35.213905]
[-80.682045,35.213733]
[-80.68062,35.213112]
[-80.678078,35.211983]
[-80.676836,35.211447]
[-80.598433,35.234307]