I've encountered a problem in Spark SQL. It is interpreting a datetime.date object as a mathematical formula, or integer, in a SQL statement I am writing.
currentDateAndTime = datetime,now()
current_month = currentDateAndTie.strftime("%m")
current_year = currentDateAndTime.strftime("%Y")
first_day_of_month = date(int(current_year), int(current)month), 1)
print(first_day_of_month)
type(first_day_of_month)
and you get:
2022-10-01
datetime.date
Then when I do
df = spark.sql("""
SELECT * FROM table_A
WHERE IncidentCreatedDate < {}
""".format(first_day_of_month))
I get an error that says AnalysisException: cannot resolve '(table_A.IncidentCreatedDate < ((2022 - 10) - 1' due to data type mismatch: differing types in '(tableA.IncidentCreatedDate < ((2022 - 10 - 1))' (date and int).;......
There might be a typo in everything above because I had to type everything out on another laptop since the other one is my work laptop and they don't like me sending anything from that laptop to anywhere else.)
pyspark doesn't support prepared statements.
format will replace the pace holder, but strings mus be in single quotes, so simply add them
df = spark.sql("""
SELECT * FROM table_A
WHERE IncidentCreatedDate < '{}'
""".format(first_day_of_month))
We have geojson polygons we would like to convert to a geo object in bigquery using ST_GeogFromGeoJSON. The conversion fails in bigquery while is successful in postgres using the equivalent command ST_GeomFromGeoJSON.
I am familiar with the SAFE prefix that can be added to the the bigquery call, but we would like to use the object and not just ignore it in case the conversion fails. I tried converting the object using ST_CONVEXHULL but wasn't able to make it work.
Is there some work around in bigquery?
Example:
Running the following command in bigquery
select ST_GeogFromGeoJSON('{"type":"Polygon","coordinates":[[[-82.022982,26.69785],[-81.606813,26.710698],[-81.999574,26.109253],[-81.615053,26.105558],[-82.022982,26.69785]]]}')
returns
Query failed: ST_GeogFromGeoJSON failed: Invalid polygon loop: Edge 4 crosses edge 9
While runs successfully in postgres
select ST_GeomFromGeoJSON('{"type":"Polygon","coordinates":[[[-82.022982,26.69785],[-81.606813,26.710698],[-81.999574,26.109253],[-81.615053,26.105558],[-82.022982,26.69785]]]}')
October 2020 Update for this post
No more any tricks needed - ST_GEOGFROMGEOJSON and ST_GEOGFROMTEXT geographic functions now support a new make_valid parameter. If set to TRUE, the function attempts to correct polygon issues when importing geography data.
So, below simple statement works perfectly now ...
select ST_GeogFromGeoJSON(
'{"type":"Polygon","coordinates":[[[-0.49044,51.4737],[-0.4907,51.4737],[-0.49075,51.46989],[-0.48664,51.46987],[-0.48664,51.47341],[-0.48923,51.47336],[-0.48921,51.4737],[-0.49072,51.47462],[-0.49114,51.47446],[-0.49044,51.4737]]]}'
, make_valid => true
)
and returns expected output
Below is for BigQuery Standard SQL
Query failed: ST_GeogFromGeoJSON failed: Invalid polygon loop: Edge 4 crosses edge 9
... Is there some work around in bigquery? ...
Proposed workaround is obviously naive and simple way of fixing specific issue while easily can be extended to more generic cases. The idea here is to extract coordinates and reorder them to eliminate the problem ...
WITH test AS (
SELECT '{"type":"Polygon","coordinates":[[[-82.022982,26.69785],[-81.606813,26.710698],[-81.999574,26.109253],[-81.615053,26.105558],[-82.022982,26.69785]]]}' AS geojson
)
SELECT ST_GEOGFROMGEOJSON('{"type":"Polygon","coordinates":' || fixed_coordinates || '}') AS geo
FROM (
SELECT '[[[' || STRING_AGG(lat_lon, '],[') || '],[' || ANY_VALUE(ordered_coordinates[OFFSET(0)]) || ']]]' fixed_coordinates
FROM (
SELECT
ARRAY( SELECT lon_lat
FROM UNNEST(REGEXP_EXTRACT_ALL(JSON_EXTRACT(geojson, '$.coordinates'), r'\[+(.*?)\]+')) lon_lat
ORDER BY CAST( SPLIT(lon_lat)[OFFSET(0)] AS FLOAT64), CAST(SPLIT(lon_lat)[OFFSET(1)] AS FLOAT64)
) ordered_coordinates
FROM test
) t, t.ordered_coordinates lat_lon
)
This produces correct output
POLYGON((-82.022982 26.69785, -81.999574 26.109253, -81.8073135 26.1074055, -81.615053 26.105558, -81.606813 26.710698, -81.8148975 26.704274, -82.022982 26.69785))
and respective visualization is
Below is for BigQuery Standard SQL
My previous answer is based on oversimplified logic of re-ordering coordinates. Obviously it will not work in more complex cases like below one
{‘type’:‘Polygon’,‘coordinates’:[[[-0.49044,51.4737],[-0.4907,51.4737],[-0.49075,51.46989],[-0.48664,51.46987],[-0.48664,51.47341],[-0.48923,51.47336],[-0.48921,51.4737],[-0.49072,51.47462],[-0.49114,51.47446],[-0.49044,51.4737]]]}
Is there some more advanced sorting logic that can be applied?
So more complex logic can be used to address this
#standardSQL
WITH test AS (
SELECT '{"type":"Polygon","coordinates":[[[-0.49044,51.4737],[-0.4907,51.4737],[-0.49075,51.46989],[-0.48664,51.46987],[-0.48664,51.47341],[-0.48923,51.47336],[-0.48921,51.4737],[-0.49072,51.47462],[-0.49114,51.47446],[-0.49044,51.4737]]]}' geojson
), coordinates AS (
SELECT CAST(SPLIT(lon_lat)[OFFSET(0)] AS FLOAT64) lon, CAST(SPLIT(lon_lat)[OFFSET(1)] AS FLOAT64) lat
FROM test, UNNEST(REGEXP_EXTRACT_ALL(JSON_EXTRACT(geojson, '$.coordinates'), r'\[+(.*?)\]+')) lon_lat), stats AS (
SELECT ST_CENTROID(ST_UNION_AGG(ST_GEOGPOINT(lon, lat))) centroid FROM coordinates
)
SELECT ST_MAKEPOLYGON(ST_MAKELINE(ARRAY_AGG(point ORDER BY sequence))) AS polygon
FROM (
SELECT point,
CASE
WHEN ST_X(point) > ST_X(centroid) AND ST_Y(point) > ST_Y(centroid) THEN 3.14 - angle
WHEN ST_X(point) > ST_X(centroid) AND ST_Y(point) < ST_Y(centroid) THEN 3.14 + angle
WHEN ST_X(point) < ST_X(centroid) AND ST_Y(point) < ST_Y(centroid) THEN 6.28 - angle
ELSE angle
END sequence
FROM (
SELECT point, centroid,
ACOS(ST_DISTANCE(centroid, anchor) / ST_DISTANCE(centroid, point)) angle
FROM (
SELECT centroid,
ST_GEOGPOINT(lon, lat) point,
ST_GEOGPOINT(lon, ST_Y(centroid)) anchor
FROM coordinates, stats
)
)
)
This approach produces correct output
POLYGON((-0.49075 51.46989, -0.48664 51.46987, -0.48664 51.47341, -0.48923 51.47336, -0.48921 51.4737, -0.49072 51.47462, -0.49114 51.47446, -0.49044 51.4737, -0.4907 51.4737, -0.49075 51.46989))
which is visualized as below
I am trying to convert an Access query to one that works in SQL server. The original query in Access works perfectly well (just terribly slow).
I only changed things slightly to make it compatible with SQL server instead of Access, like changing "NOW()" to "GETDATE()" and we can no longer divide aliases.
Running this query in SQL Server:
SELECT batches.[price-group],
[development].verifier,
Count([development].company) AS SENT,
Sum([order] *- 1) AS ORDS,
Count([development].company) / Sum([order] *- 1) AS PCT
FROM [development]
INNER JOIN batches
ON [development].batch = batches.batch
WHERE (( ( [development].[mail-date] ) < Getdate() - 50 ))
GROUP BY batches.[price-group],
[development].verifier
HAVING (( ( batches.[price-group] ) = 'pgb' ))
ORDER BY batches.[price-group],
[development].verifier,
Count([development].company) DESC;
Returns this error:
Msg 8134, Level 16, State 1, Line 1 Divide by zero error encountered.
Only real change, was like I said, in Access we could do this
[ords] / [sent] AS PCT
Any help will be appreciated, I'm not sure exactly why it isn't working! Removing the converted line above, does work in SQL server without any errors.
Thank you!
Use NULLIF():
Count([development].company) / NULLIF(Sum([order] * -1), 0) AS PCT
I have a DQL query object which we've implemented (copying a legacy application). The output has to match the legacy verbatim.
The static fields worked like a charm - but now we've encountered more complex computed fields, such as:
IF(
wo.date_approved = 0,
0,
IF(
ship.date_shipped > wo.date_approved,
ROUND(
IF(
(ship.date_shipped - wo.date_approved) > wo.time_inactive,
(ship.date_shipped - wo.date_approved) - wo.time_inactive,
ship.date_shipped - wo.date_approved
) / 86400, 2),
0
)
) AS TAT,
This is not possible to express using the query builder/DQL. I had hoped to possibly adjust the query right before execution (after parameters have been bound but before execution).
Using a placeholder or similar I would search and replace that with the series of computed fields...
I can't figure out a way to make this happen?!?! :o
Alex
I'm trying to get a query to work which returns a list of locId's from the database when fed a long and a lat.
Here's the sql:
eg: lat = "-37.8333300" : lon = "145.000000" : radius = (5 * 0.621371192) ^ 2
SELECT locId,longitude,latitude FROM tbliplocations WHERE (69.1*([longitude]- "&lon&") * cos("&lat&"/57.3))^2 + (69.1*([latitude]- "&lat&"))^2 < "&radius
Here's the error I receive:
The data types float and int are incompatible in the '^' operator.
I'm unsure of a workaround, can anyone point me in the right direction?
Answer:
Using SQL Server 2008 R2
SELECT city FROM tbliplocationsnew WHERE POWER((69.1*([longitude]- "&lon&") * cos("&lat&"/57.3)),2) + POWER((69.1*([latitude]- "&lat&")),2) < "&radius
Not sure what database you use, but I think that "^2" in SQL does not mean "squared" like in maths. You should use a math "power" function, like POWER(number,2) in SQL Server (since you use VB maybe you use SQL Server ?)
You need to have two of the same data type it's saying. SQL thinks "5" is an int. So, you should be able to trick it into treating it as a float, by putting "5.0" instead.