I have a list with nearly 100,000 GPS coordinates in lat/long format in a CSV file. A lot of these are only a few inches away from each other, so I would like to merge them somehow, or filter those out that are too close together within a certain radius.
Do you guys know of a script or a service that can do this automatically?
There is a reference to a nice paper that explains how to find nearest lat/lng points inside a specified bounding box in another thread, which you can find here: latitude/longitude find nearest latitude/longitude - complex sql or complex calculation
Here is the direct link to the paper: Geo Distance Search with MySQL
I think you can adapt the idea from the paper to your domain in order to set up a filter procedure.
Related
I followed several example about geospatial support in Redis. I tried to add POINT features to my Redis dataset without any problem, and subsequently I can query POINT names within certain radius (in meters, km, miles) of a certain coordinate (or a certain member of POINT).
The next immediate feature i need to try is POINT-in-POLYGON query. Now I am curious :
Does Redis geo have the capability for storing polygon?
If yes, does this polygon capability come as native or need another stack of software/extension?
I think it is still not possible to store polygons with redis geospatial.
But I found an article explaining how to load a custom lua script to query points (stored in redis) within a polygon. The GEOSEARCH command allows you to do that for a bounding box out of the box if it is enough for you.
I am working on a project where I save the Latitude and Longuite of a vehicle each an interval. I have also a route saved as an array of gps coordinates. So I would like to know if there is some library, that helps me to know if a point is inside the rout and other basic calculations with the coordinates as distance calculations for ex.
Any tool an any language helps!
Based on your comment, since you're not building a typical internet map, I might recommend you use a combination of Python and the Shapely library. You can see some nice examples on this post over at GIS.SE.
GIS Analyses: Geometry Types, Buffering, Intersection, etc.
In order to treat several individual Lat/Long positions as a "route", you'll need to format them as points in a LineString geometry type. Also beware: In most GIS software, points are arranged as X,Y. That means you'll be adding your points as Long,Lat. Inverting this is a common mistake that can be frustrating if you're not aware of it.
Next, in order to test whether any given point is within your route, you'll need to Buffer your route (LineString). I would use the accuracy of the GPS unit, + a few extra meters, as my buffering radius. This will give you a proper geometry (Polygon) for a Point-In-Polygon test (i.e. Intersection) that will calculate whether a given point is within the bounds of the route.
The GIS.SE post I linked to provides examples for both buffering and intersection using Python and Shapely.
Some notes about coordinates: Geodetic vs. Cartesian
I'm not confident if Shapely will perform reliable calculations on geodetic data, which is what we call the familiar coordinates you get from GPS. Before doing operations in Shapely, you may need to translate your long/lat points into projected X/Y coordinates for an appropriate coordinate system, such as UTM, etc. (Hopefully someone will comment whether this is necessary.)
Assuming this is necessary, you could add the PyProj library to give you a bridge between the GPS coordinates you have and the Cartesian coordinates you need. PyProj is the one-size-fits-all solution to this problem. However if UTM coordinates will work you might find the library cited here to be easier to implement.
If you decide to go with PyProj, it will help to know that your GPS data is described by the EPSG:4326 coordinate system. And if you are comfortable with UTM for your projected coordinates, you'll need need to determine an appropriate UTM zone for your area and get its Proj4 coordinate definition from SpatialReference.org.
For example I live in South Carolina, USA, which is UTM 17 North. So if I go to SpatialReference.org, search for "EPSG UTM zone 17N", select the option which references "WGS 1984" (I happen to know this means units in meters), then click on the Proj4 link, the site provides the coordinate system definition I'm after in Proj4 notation:
+proj=utm +zone=17 +ellps=WGS84 +datum=WGS84 +units=m +no_defs
If you're not comfortable diving into the world of coordinate systems, EPSG codes, Proj4 strings and such, you might want to favor that alternate coordinate translation library I mentioned earlier rather than PyProj. On the other hand, if you will benefit from a more localized coordinate system (most countries have their own localized systems), or if you need to keep your code portable for use in many areas, I'd recommend using PyProj and make sure to keep your Proj4 definition string in a config file, and NOT hard-coded throughout your app!
I want to be able to determine if a GPS location is in an inhabited or uninhabited zone.
I have tried several reverse geocoding API like Nominatim, but failed to get good results. It always returns the nearest possible address, even when I selected a location in the middle of a forest.
Is there any way to determine this with reasonable accuracy? Are there any databases or web services for this ?
If you have to calculate that youself, then the interesting things start:
The information whether or not a region is inhabited is stored in digital maps in layer "Land_Use". There are values for Forest, Water, Industry, Cemetary, etc.
You would have to import these Land_use polygons into a special DB (PostGres).
Such a spatial DB provides fast geo indizeds for searching only the relevant polygons.
Some countries may also fit in main memory, but then you need some kind of geo spatial index, like Quad-Tree or k-d tree to store the polygons.
Once you have imported the polygons, it is a simple "point in polygon" query, or "polygons within radius r". The typoe of th epolygon denotes the land use.
OpenStreetMap provides these polygons for free.
Otherwise you have to buy them from TomTom or probably NavTeq (Nokia Maps). But this makes only sense for major companies.
Since you're using Nominatim, you're getting the coordinates of the nearest address back in the reply.
Since the distance between two coordinates can be calculated, you can just use that to calculate the distance to the closest address found, and from that figure out if you're close to populated areas or not.
We're building a GIS interface to display GPS track data, e.g. imagine the raw data set from a guy wandering around a neighborhood on a bike for an hour. A set of data like this with perhaps a new point recorded every 5 seconds, will be large and displaying it in a browser or a handheld device will be challenging. Also, displaying every single point is usually not necessary since a user can't visually resolve that much data anyway.
So for performance reasons we are looking for algorithms that are good at 'reducing' data like this so that the number of points being displayed is reduced significantly but in such a way that it doesn't risk data mis-interpretation. For example, if our fictional bike rider stops for a drink, we certainly don't want to draw 100 lat/lon points in a cluster around the 7-Eleven.
We are aware of clustering, which is good for when looking at a bunch of disconnected points, however what we need is something that applies to tracks as described above. Thanks.
A more scientific and perhaps more math heavy solution is to use the Ramer-Douglas-Peucker algorithm to generalize your path. I used it when I studied for my Master of Surveying so it's a proven thing. :-)
Giving your path and the minimum angle you can tolerate in your path, it simplifies the path by reducing the number of points.
Typically the best way of doing that is:
Determine the minimum number of screen pixels you want between GPS points displayed.
Determine the distance represented by each pixel in the current zoom level.
Multiply answer 1 by answer 2 to get the minimum distance between coordinates you want to display.
starting from the first coordinate in the journey path, read each next coordinate until you've reached the required minimum distance from the current point. Repeat.
We are using Solr 3.3 with Solr.NET and we have put a dynamic "location_p" location type field on our documents and now we need the ability to do spatial searches.
I have got the radius searches (distance from a given point) working like this;
{!geofilt sfield=location_p pt=33.882518712472255,-84.05531775646972 d=1.7}
Now we need the ability to do a Polygon squery to get all documents with the "location_p" field 'inside' a given set of Points (something along the lines of the Polygon search capabilities of ElasticSearch).
This is really different than the BBox query filter as the points of the Polygon are not symmetrical, more random based on user 'click' points.
Any ideas or suggestions would be appreciated.
As far as I know Solr doesn't currently implement polygon spatial search.
There are a couple of efforts towards implementing this (SOLR-2155, SOLR-2268). Try applying one of these patches, test it, contribute to the project.
There's also Spatial Solr plugin, which implements polygon search but is only compatible with Solr 1.4.
See also http://wiki.apache.org/incubator/SpatialProposal