A library I’m trying to use takes a given latitude and longitude and work out with entries in a table have a lat/lng within a certain distance from this. The generated SQL query works with MySQL, but not with PostgreSQL.
Here’s the entry in my server.log file detailing the error psql is giving and the full query:
ERROR: column "distance" does not exist at character 507
STATEMENT: select *, ( '3959' * acos( cos( radians('53.49') ) * cos( radians( places.lat ) ) * cos( radians( places.lng ) - radians('-2.38') ) + sin( radians('53.49') ) * sin( radians( places.lat ) ) ) ) AS distance from (
Select *
From places
Where places.lat Between 53.475527714192 And 53.504472285808
And places.lng Between -2.4043246788967 And -2.3556753211033
) As places where "places"."deleted_at" is null having "distance" <= $1 order by "distance" asc
Knowing what the SQL should be I can then edit the PHP code that generates it and send a PR back to the library.
The query uses the syntax specific for MySql. In Postgres (and all other known to me RDBMS) you should use a derived table:
select *
from (
select *,
(3959 * acos( cos( radians(53.49) ) * cos( radians( places.lat ) )
* cos( radians( places.lng ) - radians(-2.38) )
+ sin( radians(53.49) ) * sin( radians( places.lat ) ) ) ) AS distance
from (
select *
from places
where places.lat between 53.475527714192 and 53.504472285808
and places.lng between -2.4043246788967 and -2.3556753211033
) as places
where places.deleted_at is null
) sub
where distance <= $1
order by distance asc
I have also removed quotes from numeric constants.
What your query does is a replace the ST_Dwithin function of postgresql with a long query.
ST_DWithin — Returns true if the geometries are within the specified
distance of one another. For geometry units are in those of spatial
reference and For geography units are in meters and measurement is
defaulted to use_spheroid=true (measure around spheroid), for faster
check, use_spheroid=false to measure along sphere.
it's very simple just do
ST_DWithin(geometry1, geometry2)
where geometry1 and geometry2 can be point fields (but they can be other geometric data types two)
So your query might become
SELECT * FROM places WHERE
ST_DWithin(ST_GeomFromText('POINT(-71.060316 48.432044)', 4326), places.geom) ORDER BY ST_Distance (ST_GeomFromText('POINT(-71.060316 48.432044)', 4326), places.geom)
The only change that you will need to do is to put your lat, lng together into a single POINT Field. (your probably should do that even in mysql). You can continue to have your lat,lng in separate columns but you will be sacrificing the gains you can get by using a proper geo type.
Using ST_Dwithin will be faster than a manual distance query because it's designed to make use of geometry columns on postgresql which can be indexed.
What if you want to go back to mysql someday? Well mysql 5.7 added the St_Dwithin feature.
Related
We have geojson polygons we would like to convert to a geo object in bigquery using ST_GeogFromGeoJSON. The conversion fails in bigquery while is successful in postgres using the equivalent command ST_GeomFromGeoJSON.
I am familiar with the SAFE prefix that can be added to the the bigquery call, but we would like to use the object and not just ignore it in case the conversion fails. I tried converting the object using ST_CONVEXHULL but wasn't able to make it work.
Is there some work around in bigquery?
Example:
Running the following command in bigquery
select ST_GeogFromGeoJSON('{"type":"Polygon","coordinates":[[[-82.022982,26.69785],[-81.606813,26.710698],[-81.999574,26.109253],[-81.615053,26.105558],[-82.022982,26.69785]]]}')
returns
Query failed: ST_GeogFromGeoJSON failed: Invalid polygon loop: Edge 4 crosses edge 9
While runs successfully in postgres
select ST_GeomFromGeoJSON('{"type":"Polygon","coordinates":[[[-82.022982,26.69785],[-81.606813,26.710698],[-81.999574,26.109253],[-81.615053,26.105558],[-82.022982,26.69785]]]}')
October 2020 Update for this post
No more any tricks needed - ST_GEOGFROMGEOJSON and ST_GEOGFROMTEXT geographic functions now support a new make_valid parameter. If set to TRUE, the function attempts to correct polygon issues when importing geography data.
So, below simple statement works perfectly now ...
select ST_GeogFromGeoJSON(
'{"type":"Polygon","coordinates":[[[-0.49044,51.4737],[-0.4907,51.4737],[-0.49075,51.46989],[-0.48664,51.46987],[-0.48664,51.47341],[-0.48923,51.47336],[-0.48921,51.4737],[-0.49072,51.47462],[-0.49114,51.47446],[-0.49044,51.4737]]]}'
, make_valid => true
)
and returns expected output
Below is for BigQuery Standard SQL
Query failed: ST_GeogFromGeoJSON failed: Invalid polygon loop: Edge 4 crosses edge 9
... Is there some work around in bigquery? ...
Proposed workaround is obviously naive and simple way of fixing specific issue while easily can be extended to more generic cases. The idea here is to extract coordinates and reorder them to eliminate the problem ...
WITH test AS (
SELECT '{"type":"Polygon","coordinates":[[[-82.022982,26.69785],[-81.606813,26.710698],[-81.999574,26.109253],[-81.615053,26.105558],[-82.022982,26.69785]]]}' AS geojson
)
SELECT ST_GEOGFROMGEOJSON('{"type":"Polygon","coordinates":' || fixed_coordinates || '}') AS geo
FROM (
SELECT '[[[' || STRING_AGG(lat_lon, '],[') || '],[' || ANY_VALUE(ordered_coordinates[OFFSET(0)]) || ']]]' fixed_coordinates
FROM (
SELECT
ARRAY( SELECT lon_lat
FROM UNNEST(REGEXP_EXTRACT_ALL(JSON_EXTRACT(geojson, '$.coordinates'), r'\[+(.*?)\]+')) lon_lat
ORDER BY CAST( SPLIT(lon_lat)[OFFSET(0)] AS FLOAT64), CAST(SPLIT(lon_lat)[OFFSET(1)] AS FLOAT64)
) ordered_coordinates
FROM test
) t, t.ordered_coordinates lat_lon
)
This produces correct output
POLYGON((-82.022982 26.69785, -81.999574 26.109253, -81.8073135 26.1074055, -81.615053 26.105558, -81.606813 26.710698, -81.8148975 26.704274, -82.022982 26.69785))
and respective visualization is
Below is for BigQuery Standard SQL
My previous answer is based on oversimplified logic of re-ordering coordinates. Obviously it will not work in more complex cases like below one
{‘type’:‘Polygon’,‘coordinates’:[[[-0.49044,51.4737],[-0.4907,51.4737],[-0.49075,51.46989],[-0.48664,51.46987],[-0.48664,51.47341],[-0.48923,51.47336],[-0.48921,51.4737],[-0.49072,51.47462],[-0.49114,51.47446],[-0.49044,51.4737]]]}
Is there some more advanced sorting logic that can be applied?
So more complex logic can be used to address this
#standardSQL
WITH test AS (
SELECT '{"type":"Polygon","coordinates":[[[-0.49044,51.4737],[-0.4907,51.4737],[-0.49075,51.46989],[-0.48664,51.46987],[-0.48664,51.47341],[-0.48923,51.47336],[-0.48921,51.4737],[-0.49072,51.47462],[-0.49114,51.47446],[-0.49044,51.4737]]]}' geojson
), coordinates AS (
SELECT CAST(SPLIT(lon_lat)[OFFSET(0)] AS FLOAT64) lon, CAST(SPLIT(lon_lat)[OFFSET(1)] AS FLOAT64) lat
FROM test, UNNEST(REGEXP_EXTRACT_ALL(JSON_EXTRACT(geojson, '$.coordinates'), r'\[+(.*?)\]+')) lon_lat), stats AS (
SELECT ST_CENTROID(ST_UNION_AGG(ST_GEOGPOINT(lon, lat))) centroid FROM coordinates
)
SELECT ST_MAKEPOLYGON(ST_MAKELINE(ARRAY_AGG(point ORDER BY sequence))) AS polygon
FROM (
SELECT point,
CASE
WHEN ST_X(point) > ST_X(centroid) AND ST_Y(point) > ST_Y(centroid) THEN 3.14 - angle
WHEN ST_X(point) > ST_X(centroid) AND ST_Y(point) < ST_Y(centroid) THEN 3.14 + angle
WHEN ST_X(point) < ST_X(centroid) AND ST_Y(point) < ST_Y(centroid) THEN 6.28 - angle
ELSE angle
END sequence
FROM (
SELECT point, centroid,
ACOS(ST_DISTANCE(centroid, anchor) / ST_DISTANCE(centroid, point)) angle
FROM (
SELECT centroid,
ST_GEOGPOINT(lon, lat) point,
ST_GEOGPOINT(lon, ST_Y(centroid)) anchor
FROM coordinates, stats
)
)
)
This approach produces correct output
POLYGON((-0.49075 51.46989, -0.48664 51.46987, -0.48664 51.47341, -0.48923 51.47336, -0.48921 51.4737, -0.49072 51.47462, -0.49114 51.47446, -0.49044 51.4737, -0.4907 51.4737, -0.49075 51.46989))
which is visualized as below
I have a website in Classic ASP with an Access database. I'm using the Google maps API. Now I want to find Companies (from the database) within a radius of 20 km from a given location.
I have found a formular (haversine), but it seems like I can't use ACOS(). I get this error:
Microsoft JET Database Engine error '80040e14'
Undefined function 'acos' in expression.
And here is my sql:
lat = 56.113517
lng = 10.145928
sql = "SELECT companyId, ( 3959 * acos( cos( radians(37) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(-122) ) + sin( radians(37) ) * sin( radians( lat ) ) ) ) AS distance FROM company HAVING distance < 25 ORDER BY distance"
Set rs = Connect.Execute(sql)
I have search and search but I cant find a solution.
Can anybody help?
It is true that Access SQL does not have a built-in ACOS() function. If you were running your query in Access itself then you could create your own ACOS() function in VBA to use in an SQL query, but that option is not available to SQL queries against an Access database from other applications (like your ASP app).
So, you will need to do the precise distance calculations in ASP after you retrieve the records that might qualify. That is, you could calculate the latitude and longitude boundaries of a 40km by 40km "square" centered on the target location (to exclude records that are guaranteed to be too far away), then calculate the actual distance for each of the returned records using your ASP function (example code here).
I am using CActiveDataProvider with CDbCriteria to search through some related models, using multiple related models. The code to generate the results is as below:
$criteria->select = '*, ( 3959 * acos( cos( radians(' . $latitude . ') )
* cos( radians( latitude ) ) * cos( radians( longitude ) -
radians(' . $longitude . ') ) + sin( radians(' . $latitude . ') )
* sin( radians( latitude ) ) ) ) * 1.609344 AS distance';
//Basically just calculating distance from an input point
$criteria->with = array('keywords', 'coupons', 'jobs');
$criteria->order = 'distance asc';
$criteria->having = 'distance < 20';
$criteria->compare('name', $this->searchTerm, true, 'AND');
$dataProvider = new CActiveDataProvider('Store', array(
'criteria'=>$criteria));
The search works fine and gets the results as expected. The problem is that the CListview reports the number of results correctly, but shows pagination anyways. For example: 'Displaying 1-7 of 31 results, and pagination is shown. Clicking on pages 2, 3, 4 show no results.
Is this a bug, or am I doing something wrong?
Sometimes with a complex query, you have to manually provide the number of rows as item count, try passing the count with your dataprovider with an attribute like:
'totalItemCount'=>$count,
http://www.yiiframework.com/doc/api/1.1/CDataProvider#totalItemCount-detail
Try commenting out this line
$criteria->with = array('keywords', 'coupons', 'jobs');
and seeing if the pager and item count responds correctly. I found that this was my case, so I have to take a performance hit and let the thing lazy load.
I've had a similar mismatch between the count and the actual list of records when I tried to use a query that included a GROUP BY clause. I notice that you have a HAVING clause in your query. I imagine this could cause similar problems and a quick look at the Yii source here shows that the presence of either a GROUP BY or HAVING clause causes the counting to be handled differently than without either of those clauses.
It looks like you used $criteria->having to allow using the calculated column distance. A quick test if this is the problem would be to use $criteria->condition instead and set it equal to the complete calculation like this:
$criteria->condition = '( 3959 * acos( cos( radians(' . $latitude . ') )
* cos( radians( latitude ) ) * cos( radians( longitude ) -
radians(' . $longitude . ') ) + sin( radians(' . $latitude . ') )
* sin( radians( latitude ) ) ) ) * 1.609344 < 20';
The condition property is used to generate the WHERE clause, which does not cause problems with count like HAVING may.
By the way, it's extremely useful to be able to inspect the actual queries that Yii is making. It can be surprising and show problems or inefficiencies. You can get Yii to output the query info at the bottom of every webpage by setting this in your protected/config/main.php:
'components'=>array(
'db'=>array(
// DB connection info as usual
),
'log'=>array(
'routes'=>array(
array(
'class'=>'CWebLogRoute',
'levels'=>'trace',
),
),
),
),
I had to hack CJoinElement in CActiveFinder to fix this. The count function in CJoinElement is the one that does the real work to display the total count in the summary. This resets the group and having components of the criteria. Removing the reset fixed the problem. See https://github.com/yiisoft/yii/issues/167
This is dragging at my nerves!
Scenario:
I am using the geocoder plugin: http://github.com/alexreisner/geocoder/
I have two models: Address and Item
the latitude and the longitude are in Address and an Item belongs_to :address
Now, when I try to do a near search:
Item.near(Person.first.address.coordinates, 20)
on an Item it fails with the following SQL:
Unknown column 'latitude' in 'field list':
SELECT *, 3956 * 2 * ASIN(SQRT(POWER(SIN((51.3883 - latitude) * PI() / 180 / 2), 2) + COS(51.3883 * PI()/180) * COS(latitude * PI() / 180) * POWER(SIN((6.65326 - longitude) * PI() / 180 / 2), 2) )) AS distance FROM `items` WHERE ((latitude BETWEEN 51.0984449275362 AND 51.6781550724638 AND longitude BETWEEN 6.188777824785 AND 7.117742175215) AND (`items`.`accepted` IS NULL AND `items`.`taken_by` IS NULL)) ORDER BY distance ASC
Don't look at the weird Math stuff in the query,
the problem lies within the FROM and the WHERE clause because
the latitude and the longitude attributes are part of the address table,
so a simple addition to the Query would solve the problem:
... FROM items, addresses WHERE ...
... null)) AND items.address_id = addresses.id ORDER BY ...
I just want to add a table to the FROM field and a condition :(
I tried using several named_scopes using :include and :joins, but none worked!
Does anybody know a solution to this? How could I insert this into a scoped ActiveRecord object?
So the Item has address(es) and that's where the longitude and latitude actually exist, right? That's where I think the query is going wrong for you. You could try adding a :joins=>:address ... but I like the following better:
Since all the math occurs in the database, I think you could do:
Address.near(Person.first.address.coordinates, 20)
and then get the items that the addresses belong to, maybe even an :include=>:item or something.
I have been testing my geolocation query for some time now and I haven't found any issues with it until now.
I am trying to search for all cities within a given radius, often times I'm searching for cities surrounding a city using that city's coords, but recently I tried searching around a city and found that the city itself was not returned.
I have these cities as an excerpt in my database:
city latitude longitude
Saint-Mathieu 45.316708 -73.516253
Saint-Édouard 45.233374 -73.516254
Saint-Michel 45.233374 -73.566256
Saint-Rémi 45.266708 -73.616257
But when I run my query around the city of Saint-Rémi, with the following query...
SELECT tblcity.city, tblcity.latitude, tblcity.longitude,
truncate((degrees(acos( sin(radians(tblcity.latitude))
* sin(radians(45.266708))
+ cos(radians(tblcity.latitude))
* cos(radians(45.266708))
* cos(radians(tblcity.longitude - -73.616257) ) ) )
* 69.09*1.6),1) as distance
FROM tblcity HAVING distance < 10 ORDER BY distance desc
I get these results:
city latitude longitude distance
Saint-Mathieu 45.316708 -73.516253 9.5
Saint-Édouard 45.233374 -73.516254 8.6
Saint-Michel 45.233374 -73.566256 5.3
The town of Saint-Rémi is missing from the search.
So I tried a modified query hoping to get a better result:
SELECT tblcity.city, tblcity.latitude, tblcity.longitude,
truncate(( 6371 * acos( cos( radians( 45.266708 ) )
* cos( radians( tblcity.latitude ) )
* cos( radians( tblcity.longitude )
- radians( -73.616257 ) )
+ sin( radians( 45.266708 ) )
* sin( radians( tblcity.latitude ) ) ) ),1) AS distance
FROM tblcity HAVING distance < 10 ORDER BY distance desc
But I get the same result...
However, if I modify Saint-Rémi's coords slighly by changing the last digit of the lat or long by 1, both queries will return Saint-Rémi. Also, if I center the query on any of the other cities above, the searched city is returned in the results.
Can anyone shed some light on what may be causing my queries above to not display the searched city of Saint-Rémi? I have added a sample of the table (with extra fields removed) below.
I'm using MySQL 5.0.45, thanks in advance.
CREATE TABLE `tblcity` (
`IDCity` int(1) NOT NULL auto_increment,
`City` varchar(155) NOT NULL default '',
`Latitude` decimal(9,6) NOT NULL default '0.000000',
`Longitude` decimal(9,6) NOT NULL default '0.000000',
PRIMARY KEY (`IDCity`)
) ENGINE=MyISAM AUTO_INCREMENT=52743 DEFAULT CHARSET=latin1 AUTO_INCREMENT=52743;
INSERT INTO `tblcity` (`city`, `latitude`, `longitude`) VALUES
('Saint-Mathieu', 45.316708, -73.516253),
('Saint-Édouard', 45.233374, -73.516254),
('Saint-Michel', 45.233374, -73.566256),
('Saint-Rémi', 45.266708, -73.616257);
In your first query, I believe you've inverted the longitudes in the subtraction. The Spherical Law of Cosines is:
d = acos(sin(lat1)*sin(lat2) + cos(lat1)*cos(lat2)*cos(long2−long1))*R
If lat1 is substituted with tblcity.latitude, long1 must be substituted with tblcity.longitude. I think you've accidentally substituted long2 in your query. Does this one work better?
SELECT tblcity.city, tblcity.latitude, tblcity.longitude,
truncate((degrees(acos( sin(radians(tblcity.latitude))
* sin(radians(45.266708))
+ cos(radians(tblcity.latitude))
* cos(radians(45.266708))
* cos(radians(-73.616257 - tblcity.longitude) ) ) )
* 69.09*1.6),1) as distance
FROM tblcity HAVING distance < 10 ORDER BY distance desc
I haven't looked into your second query yet, but hopefully that helps.
You are using the "spherical law of cosines" formula, which is susceptable to rounding error at small distances (like zero!) -- see this discussion. The long expression that is fed into acos() is evaluating to slightly more than 1.0, which is out of bounds.
Here's the problem illustrated using Python to do the calculations:
>>> from math import sin, cos, acos, radians
>>> lat = radians(45.266708)
>>> long_expression = sin(lat) * sin(lat) + cos(lat) * cos(lat) * cos(0.0)
>>> repr(long_expression)
'1.0000000000000002'
>>> acos(long_expression)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: math domain error
>>>
It seems that MySQL is substituting NULL instead of raising an exception. I know little about MySQL, but you should be able to overcome that by doing something like ifnull(acos(long_expression), 0.0) or coalesce(acos(long_expression), 0.0).
Alternatively you could use the haversine formula, which shifts the rounding problem from your door-step to the opposite side of the earth.
Update: I've tested using that formula in Python to calculate the should-be-zero distance between a point and the same point, for each of the 37582 unique (lat, lon) 2-tuples in a file of US zip codes.
Of these:
31591 (84.1%) produced a zero distance
4244 (11.3%) produced a distance of 9.5 cm.
831 (2.2%) produced a distance of 13.4 cm.
916 (2.4%) produced a "cos" value of 1.0000000000000002 which would cause an exception in acos() if not detected and avoided.
It appears that explicit testing for lat1 = lat2 and lon1 = lon2 and avoiding the formula in that case (just use zero) might be a good idea -- it would give a consistent answer and avoid puzzlement.