I'm creating a database that is being hosted on a MS SQL 2012 server. The primary function of this database is to return results that is within a certain distance from an origin. Locations are being stored as latitude / longitude.
By reading here on Stack Overflow i found a very nice way to query the database for exactly what i am looking for and it works like a charm! However I'm thinking of a possible way to optimize this.
Original SQL query
DECLARE #orig_lat DECIMAL(12, 9)
DECLARE #orig_lng DECIMAL(12, 9)
SET #orig_lat=56.xxxxxx
SET #orig_lng=14.xxxxxx
DECLARE #orig geography = geography::Point(#orig_lat, #orig_lng, 4326);
SELECT *
FROM foobar
WHERE #orig.STDistance(geography::Point(foobar.latitude, foobar.longitude, 4326)) < 2000
My guess is that this query does a linear search of the foobar table only returning the matching columns. However since this table contains positions all over the world I want to know if I can help the database by reducing the amount of rows it has to run the distance calculation on. My guess is that this calculation is heavy for the server.
I know the origin of the request being made and I also know that the maximum distance between the points will never be larger than lets say 100km.
Hypothesis
Since I know that I don't have to search the whole world only up to 100km from point of origin I can improve upon the WHERE statement as seen below. By creating a minimum and maximum bound for the latitude and longitude that is done by moving the position by some number in each direction.
I explain:
Origin latitude 56.xxxxxx
Min latitude 55.xxxxxx
Max latitude 57.xxxxxx
Origin longitude 14.xxxxxx
Min longitude 13.xxxxxx
Max longitude 15.xxxxxx
By doing this I create a zone around the origin reaching about 126km. By adding this to the WHERE statement I first make sure the requested position is within the correct bounds. After that I run the distance calculation to get exact distance. The distance calculation is now only run against the rows that is within the min and max bounds instead of the whole world.
Optimization proposal
DECLARE #orig_lat DECIMAL(12, 9)
DECLARE #orig_lng DECIMAL(12, 9)
DECLARE #orig_latMin DECIMAL(12, 9)
DECLARE #orig_latMax DECIMAL(12, 9)
DECLARE #orig_lngMin DECIMAL(12, 9)
DECLARE #orig_lngMax DECIMAL(12, 9)
SET #orig_lat=56.xxxxxx
SET #orig_lng=14.xxxxxx
SET #orig_latMin=55.xxxxxx
SET #orig_latMax=57.xxxxxx
SET #orig_lngMin=13.xxxxxx
SET #orig_lngMax=15.xxxxxx
DECLARE #orig geography = geography::Point(#orig_lat, #orig_lng, 4326);
SELECT *
FROM foobar
WHERE ([latitude] > #orig_latMin
AND [latitude] < #orig_latMax
AND [longitude] > #orig_lngMin
AND [longitude] < #orig_lngMax)
AND #orig.STDistance(geography::Point(foobar.latitude, foobar.longitude, 4326)) < 2000
I don't know database implementation details but does this improve the query or does it make it worse? My guess is that it depends on how the WHERE statement actually work and in what order it dose things. My hope is that the boundary checks will be run before the distance calculation in order to reduce the amount of time a distance calculation is done.
EDIT
Just implemented the suggested index proposal with the following results.
Without indexing:
With optimized statement have a cost of 0,025352
Without optimized statement have a cost of 0,025323
With indexing:
With optimized statement have a cost of 0,0104057
Without optimized statement have a cost of 0,0253234
A good rule of thumb is that the execution time of a database query depends on the number of disk pages that have to be read. The CPU time can usually be ignored.
According to this rule, your proposed optimization will improve the execution time if it makes a difference for the number of disk pages. This will be the case if there is an index on latitude and longitude that will allow to skip many table rows and therefore many disk pages. If that's the case, the optimizer will certainly evalute that part of the WHERE clause before the distance.
If there's no index that helps with those two columns, I doubt you will see a big difference.
You can analyze the query time using MS Management Studio, run a big query with differents where's, it will even show you what part of the query requires which amount of time.
You can click CTRL+L: Display estimated executionplan
or CTRL+M: display actual execution plan (when you run it)
Run it once with the "boundaries" first, then again with boundaries after.
you will be able to see which is slower, then try again without the boundaries.
If you don't have enough data the difference might not be visible.
Related
I have an SQL table with geo-tagged values (Longitude, Latitude, value). The table is accumulated quickly and has thousands entries. Therefore, querying the table for values in some area return very large data-set.
I would like to know the way to average value with close location proximity to one value, here is an illustration:
Table:
Long lat value
10.123001 53.567001 10
10.123002 53.567002 12
10.123003 53.567003 18
10.124003 53.568003 13
lets say my current location is 10.123004, 53.567004. If I am querying for the values near by I will get the four raws with values 10, 12, 18, and 13. This works if the data-set is relatively small. If the data is large I would like to query sql for rounded location (10.123, 53.567) and need sql to return something like
Long lat value
10.123 53.567 10 (this is the average of 10, 12, and 18)
10.124 53.568 13
Is this possible? how we can average large data set based on locations?
Is sql database is the right choice in the first place?
GROUP BY rounded columns, and the AVG aggregate function should work fine for this:
SELECT ROUND(Long, 3) Long,
ROUND(Lat, 3) Lat,
AVG(value)
FROM Table
GROUP BY ROUND(Long, 3), ROUND(Lat, 3)
Add a WHERE clause to filter as needed.
Here's some rough pseudocode that might be a start. You need to provide the proper precision arguments for the round function in the dialect of SQL you are using for your project, so understand that the 3 I provide as the second argument to Round is the number of decimals of precision to which the number is rounded, as indicated by your original post.
Select round(lat,3),round(long,3),avg(value)
Group by round(lat,3),round(long,3)
The problem with the rounding approach is the boundary conditions -- what happens when points are close to the bounday.
However, for the neighborhood of a given point it is better to use something like:
select *
from table
where long between #MyLong - #DeltaLong and #MyLong + #DeltaLong and
lat between #MyLat - #DeltaLat and #MyLat + #DeltaLat
For this, you need to define #DeltaLong and #DeltaLat.
Rounding works fine for summarization, if that is your problem.
We have a SQL 2008 database with 20million+ geoWe locations (and growing) , each location includes the standard name/address/Geography/ID/Etc columns.
We need a way to efficiently search through the records based on a distance but also a "contains" keyword via the full text index. The basic idea is we search for locations near us based on a max distance.
Right now when we search for full strings such as StarBucks within 1 mile the search returns in a few seconds. However if we search for "star" within 1 mile the search can sometimes take several min to return.
We have been playing around logic such as this :
DECLARE #geoSearchLocation GEOGRAPHY, #geoSearchPolygon GEOGRAPHY, #returncount smallint = 50
SET #geoSearchLocation = geography::Point(40.729047, -74.010086, 4326); --NYC
SET #geoSearchPolygon = geography::STGeomFromText('POLYGON((-74.015086 40.734047,
-74.015086 40.718047,
-74.005086 40.718047,
-74.005086 40.734047,
-74.015086 40.734047))', 4326);
SET #geoSearchLocation = geography::Point(40.729047, -73.010086, 4326);
SELECT TOP (100) --WITH TIES
*, gt.LocationGeog.STDistance(#geoSearchLocation) AS dist
FROM dbo.GeoLocation_Locations gt WITH (NOLOCK, INDEX(geolocation_HHHH128_sidx))
WHERE gt.LocationGeog.STIntersects(#geoSearchPolygon) = 1
ORDER BY gt.LocationGeog.STDistance(#geoSearchLocation)
This introduces dupes and other issues in the search however. We have also been trying to use the POWER forumlas we have found online.
We have other queries working great which are only based on distance or a certain category ID, those return in a under a second. The big problem is the wildcard string matches.
Does anyone have a awesome SQL or CLR proc which accepts name (wildcard support) and distance when dealing with 20million+ records ?
Right now we are very stuck :(
thanks in advance,
Jeff
Simple LIKE predicates will only work almost well if you are searching for 'star%' and that column has an index. Performance however will continue to degrade as the dataset gets larger. If you can set up Sql Server's Full-Text mechanism you will be better off, it's not hard and it is way faster than indexed LIKE's
I need to write a query which allows me to find all locations within a range (Miles) from a provided location.
The table is like this:
id | name | lat | lng
So I have been doing research and found: this my sql presentation
I have tested it on a table with around 100 rows and will have plenty more! - Must be scalable.
I tried something more simple like this first:
//just some test data this would be required by user input
set #orig_lat=55.857807; set #orig_lng=-4.242511; set #dist=10;
SELECT *, 3956 * 2 * ASIN(
SQRT( POWER(SIN((orig.lat - abs(dest.lat)) * pi()/180 / 2), 2)
+ COS(orig.lat * pi()/180 ) * COS(abs(dest.lat) * pi()/180)
* POWER(SIN((orig.lng - dest.lng) * pi()/180 / 2), 2) ))
AS distance
FROM locations dest, locations orig
WHERE orig.id = '1'
HAVING distance < 1
ORDER BY distance;
This returned rows in around 50ms which is pretty good!
However this would slow down dramatically as the rows increase.
EXPLAIN shows it's only using the PRIMARY key which is obvious.
Then after reading the article linked above. I tried something like this:
// defining variables - this when made into a stored procedure will call
// the values with a SELECT query.
set #mylon = -4.242511;
set #mylat = 55.857807;
set #dist = 0.5;
-- calculate lon and lat for the rectangle:
set #lon1 = #mylon-#dist/abs(cos(radians(#mylat))*69);
set #lon2 = #mylon+#dist/abs(cos(radians(#mylat))*69);
set #lat1 = #mylat-(#dist/69);
set #lat2 = #mylat+(#dist/69);
-- run the query:
SELECT *, 3956 * 2 * ASIN(
SQRT( POWER(SIN((#mylat - abs(dest.lat)) * pi()/180 / 2) ,2)
+ COS(#mylat * pi()/180 ) * COS(abs(dest.lat) * pi()/180)
* POWER(SIN((#mylon - dest.lng) * pi()/180 / 2), 2) ))
AS distance
FROM locations dest
WHERE dest.lng BETWEEN #lon1 AND #lon2
AND dest.lat BETWEEN #lat1 AND #lat2
HAVING distance < #dist
ORDER BY distance;
The time of this query is around 240ms, this is not too bad, but is slower than the last. But I can imagine at much higher number of rows this would work out faster. However anEXPLAIN shows the possible keys as lat,lng or PRIMARY and used PRIMARY.
How can I do this better???
I know I could store the lat lng as a POINT(); but I also haven't found too much documentation on this which shows if it's faster or accurate?
Any other ideas would be happily accepted!
Thanks very much!
-Stefan
UPDATE:
As Jonathan Leffler pointed out I had made a few mistakes which I hadn't noticed:
I had only put abs() on one of the lat values. I was using an id search in the WHERE clause in the second one as well, when there was no need. In the first query was purely experimental the second one is more likely to hit production.
After these changes EXPLAIN shows the key is now using lng column and average time to respond around 180ms now which is an improvement.
Any other ideas would be happily accepted!
If you want speed (and simplicity) you'll want some decent geospatial support from your database. This introduces geospatial datatypes, geospatial indexes and (a lot of) functions for processing / building / analyzing geospatial data.
MySQL implements a part of the OpenGIS specifications although it is / was (last time I checked it was) very very rough around the edges / premature (not useful for any real work).
PostGis on PostgreSql would make this trivially easy and readable:
(this finds all points from tableb which are closer then 1000 meters from point a in tablea with id 123)
select
myvalue
from
tablea, tableb
where
st_dwithin(tablea.the_geom, tableb.the_geom, 1000)
and
tablea.id = 123
The first query ignores the parameters you set - using 1 instead of #dist for the distance, and using the table alias orig instead of the parameters #orig_lat and #orig_lon.
You then have the query doing a Cartesian product between the table and itself, which is seldom a good idea if you can avoid it. You get away with it because of the filter condition orig.id = 1, which means that there's only one row from orig joined with each of the rows in dest (including the point with dest.id = 1; you should probably have a condition AND orig.id != dest.id). You also have a HAVING clause but no GROUP BY clause, which is indicative of problems. The HAVING clause is not relating any aggregates, but a HAVING clause is (primarily) for comparing aggregate values.
Unless my memory is failing me, COS(ABS(x)) === COS(x), so you might be able to simplify things by dropping the ABS(). Failing that, it is not clear why one latitude needs the ABS and the other does not - symmetry is crucial in matters of spherical trigonometry.
You have a dose of the magic numbers - the value 69 is presumably number of miles in a degree (of longitude, at the equator), and 3956 is the radius of the earth.
I'm suspicious of the box calculated if the given position is close to a pole. In the extreme case, you might need to allow any longitude at all.
The condition dest.id = 1 in the second query is odd; I believe it should be omitted, but its presence should speed things up, because only one row matches that condition. So the extra time taken is puzzling. But using the primary key index is appropriate as written.
You should move the condition in the HAVING clause into the WHERE clause.
But I'm not sure this is really helping...
The NGS Online Inverse Geodesic Calculator is the traditional reference means to calculate the distance between any two locations on the earth ellipsoid:
http://www.ngs.noaa.gov/cgi-bin/Inv_Fwd/inverse2.prl
But above calculator is still problematic. Especially between two near-antipodal locations, the computed distance can show an error of some tens of kilometres !!! The origin of the numeric trouble was identified long time ago by Thaddeus Vincenty (page 92):
http://www.ngs.noaa.gov/PUBS_LIB/inverse.pdf
In any case, it is preferrable to use the reliable and very accurate online calculator by Charles Karney:
http://geographiclib.sourceforge.net/cgi-bin/Geod
Some thoughts on improving performance. It wouldn't simplify things from a maintainability standpoint (makes things more complex), but it could help with scalability.
Since you know the radius, you can add conditions for the bounding box, which may allow the db to optimize the query to eliminate some rows without having to do the trig calcs.
You could pre-calculate some of the trig values of the lat/lon of stored locations and store them in the table. This would shift some of the performance cost when inserting the record, but if queries outnumber inserts, this would be good. See this answer for an idea of this approach:
Query to get records based on Radius in SQLite?
You could look at something like geohashing.
When used in a database, the structure of geohashed data has two advantages. ,,, Second, this index structure can be used for a quick-and-dirty proximity search - the closest points are often among the closest geohashes.
You could search SO for some ideas on how to implement:
https://stackoverflow.com/search?q=geohash
If you're only interested in rather small distances, you can approximate the geographical grid by a rectangular grid.
SELECT *, SQRT(POWER(RADIANS(#mylat - dest.lat), 2) +
POWER(RADIANS(#mylon - dst.lng)*COS(RADIANS(#mylat)), 2)
)*#radiusOfEarth AS approximateDistance
…
You could make this even more efficient by storing radians instead of (or in addition to) degrees in your database. If your queries may cross the 180° meridian, some extra care would be neccessary there, but many applications don't have to deal with those locations. You could also try to change POWER(x) to x*x, which might get computed faster.
I have a table of 320000 rows which contains lat/lon coordinate points. When a user selects a location my program gets the coordinates from the selected location and executes a query which brings all the points from the table that are near. This is done by calculating the distance between the selected point and each coordinate point from my table row. This is the query I use:
select street from locations
where ( ( (lat - (-34.594804)) *(lat - (-34.594804)) ) + ((lon - (-58.377676 ))*(lon - (-58.377676 ))) <= ((0.00124)*(0.00124)))
group by street;
As you can see the WHERE clause is a simple Pythagoras formula to calculate the distance between two points.
Now my problem is that I can not get an INDEX to be usable. I've tried with
CREATE INDEX indx ON location(lat,lon)
also with
CREATE INDEX indx ON location(street,lat,lon)
with no luck. I've notice that when there is math operation with lat or lon, the index is not being called . Is there any way I can optimize this query for using an INDEX so as to gain speed results?
Thanks in advance!
The problem is that the sql engine needs to evaluate all the records to do the comparison (WHERE ..... <= ...) and filter the points so the indexes don’t speed up the query.
One approach to solve the problem is compute a Minimum and Maximum latitude and longitude to restrict the number of record.
Here is a good link to follow: Finding Points Within a Distance of a Latitude/Longitude
Did you try adjusting the page size? A table like this might gain from having a different (i.e. the largest?) available page size.
PRAGMA page_size = 32768;
Or any power of 2 between 512 and 32768. If you change the page_size, don't forget to vacuum the database (assuming you are using SQLite 3.5.8. Otherwise, you can't change it and will need to start a fresh new database).
Also, running the operation on floats might not be as fast as running it on integers (big maybe), so that you might gain speed if you record all your coordinates times 1 000 000.
Finally, euclydian distance will not yield very accurate proximity results. The further you get from the equator, the more the circle around your point will flatten to ressemble an ellipse. There are fast approximations which are not as calculation intense as a Great Circle Distance Calculation (avoid at all cost!)
You should search in a square instead of a circle. Then you will be able to optimize.
Surely you have a primary key in locations? Probably called id?
Why not just select the id along with the street?
select id, street from locations
where ( ( (lat - (-34.594804)) *(lat - (-34.594804)) ) + ((lon - (-58.377676 ))*(lon - (-58.377676 ))) <= ((0.00124)*(0.00124)))
group by street;
I have a table where I'm storing Lat/Long coordinates, and I want to make a query where I want to get all the records that are within a distance of a certain point.
This table has about 10 million records, and there's an index over the Lat/Long fields
This does not need to be precise. Among other things, I'm considering that 1 degree Long == 1 degree Lat, which I know is not true, but the ellipse I'm getting is good enough for this purpose.
For my examples below, let's say the point in question is [40, 140], and my radius, in degrees, is 2 degrees.
I've tried this 2 ways:
1) I created a UDF to calculate the Square of the Distance between 2 points, and I'm running that UDF in a query.
SELECT Lat, Long FROM Table
WHERE (Lat BETWEEN 38 AND 42)
AND (Long BETWEEN 138 AND 142)
AND dbo.SquareDistance(Lat, Long, 40, 140) < 4
I'm filtering by a square first, to speed up the query and let SQL use the index, and then refining that to match only the records that fall within the circle with my UDF.
2) Run the query to get the square (same as before, but without the last line), feed ALL those records to my ASP.Net code, and calculate the circle in the ASP.Net side (same idea, calculate the square of the distance to save the Sqrt call, and compare to the square of my radius).
To my suprise, calculating the circle in the .Net side is about 10 times faster than using the UDF, which leads me to believe that I'm doing something horribly wrong with that UDF...
This is the code I'm using:
CREATE FUNCTION [dbo].[SquareDistance]
(#Lat1 float, #Long1 float, #Lat2 float, #Long2 float)
RETURNS float
AS
BEGIN
-- Declare the return variable here
DECLARE #Result float
DECLARE #LatDiff float, #LongDiff float
SELECT #LatDiff = #Lat1 - #Lat2
SELECT #LongDiff = #Long1 - #Long2
SELECT #Result = (#LatDiff * #LatDiff) + (#LongDiff * #LongDiff)
-- Return the result of the function
RETURN #Result
END
Am I missing something here?
Shouldn't using a UDF within SQL Server be much faster than feeding about 25% more records than necessary to .Net, with the overhead of the DataReader, the communication between processes and whatnot?
Is there something I'm doing horribly wrong in that UDF that makes it run slow?
Is there any way to improve it?
Thank you very much!
You can improve the performance of this UDF by NOT declaring variables and doing your calculations more in-line. This will likely improve performance a little but (but probably not much).
CREATE FUNCTION [dbo].[SquareDistance]
(#Lat1 float, #Long1 float, #Lat2 float, #Long2 float)
RETURNS float
AS
BEGIN
Return ( SELECT ((#Lat1 - #Lat2) * (#Lat1 - #Lat2)) + ((#Long1 - #Long2) * (#Long1 - #Long2)))
END
Even better would be to remove the function and put the calculations in the original query.
SELECT Lat, Long FROM Table
WHERE (Lat BETWEEN 38 AND 42)
AND (Long BETWEEN 138 AND 142)
AND ((Lat - 40) * (Lat - 40)) + ((Long - 140) * (Long - 140)) < 4
There is a little bit of overhead with calling a user defined function. By removing the function, you are likely to gain a little in performance.
Also, I encourage you to check your execution plan just to make sure you are getting index seeks like you expect.
There is a lot of overhead in using a UDF.
Even coding it in-line may not be good because an index can not be used, although here the BETWEEN clauses should reduce the data that needs crunched.
To extend G Mastros' idea, separate the select bit from the square bit. It may help the optimiser.
SELECT
Lat, Long
FROM
(
SELECT
Lat, Long
FROM
Table
WHERE
(Lat BETWEEN 38 AND 42)
AND
(Long BETWEEN 138 AND 142)
) foo
WHERE
((Lat - 40) * (Lat - 40)) + ((Long - 140) * (Long - 140)) < 4
Edit: You may be able to reduce the actual calculations involved.
This next idea may reduce the number of calcs from 7 to 5
...
SELECT
Lat, Long,
Lat - 40 AS LatDiff, Long - 140 AS LongDiff
FROM
...
(LatDiff * LatDiff) + (LongDiff * LongDiff) < 4
...
Basically, try the 3 solutions offered and see what works.
The optimiser may ignore the derived table, it may use it, or it may generate an even worse plan.
Check this article that describes why UDF in SQL Server are generically speaking a bad idea. Unless you're pretty sure the table you're invoking the UDF will not grow up a lot beware that UDF functions are always called on ALL the rows in your tables and not (as one can wrongly guess) only on resultset. This can give you a big performance hit when database grow.
The very good article linked details also some ways to overcome the problem but the real fact is that the SQL Server TSQL dialect misses a way to create a scalar function or a deterministic one (like Oracle does).
Updates:
GMastros: You were absolutely right. Doing the math in the query itself is infinitely faster than the UDF. I'm using the SQUARE() function to do the multiplication, which makes it a bit more concise, but performance is the same.
However, doing it this way is still twice as slow as doing the math in .Net.
I can't really understand that, but i've come to a compromise that is useful for my particular situation (which sucks, because I need to duplicate code, but it's the best scenario, unless we can find a way to make the circle calculation in SQL be faster)
Thanks!