Can you tell me if there is a formula to change a latitude and longitude into a single number?
I plan to use this for a database table in software that provides routing for deliveries. The table row would have that number as well as the postal address. The database table would be sorted in ascending numeric order so the software can figure out which address the truck would need to go to first, second etc.
Please can you respond showing VB or VB.Net syntax so I can understand how it works?
For example I would use the following numbers for the latitude and longitude:
Lat = 40.71412890
Long = -73.96140740
Additional Information:
I'm developing an Android app using Basic4Android. Basic4Android uses a VB or VB.Net syntax with SQLite as the database.
Part of this app will have route planning. I want to use this number as the first column in an SQLite table and the other columns will be for the address. If I do a query within the app that sorts the rows in numerical ascending order, I will be able to figure out which postal address are closest to each other so it will take less time for me to go from house to house.
For example, if the numbers were:
194580, 199300, 178221
I can go to postal address 178221 then to 194580 and finally to 199300 and I won't need to take the long way around town to do my deliveries after they were sorted.
As an alternative, I would be happy if there was an easy way to call a web service that returns maybe a json response that has the single number if I send a postal address to the web site. Basic4Android does have http services that can send requests to a web site.
A latitude an longitude, can both be represented as 4 byte integer, such that the coordinates has an accuracy of 3cm which is sufficent for most applications.
Steps to create one 8 byte value of type long from latitude and longitude:
1) convert lat and lon to int by: int iLat = lat * 1E7;
2) Use a 8 byte long value to store both 4 byte int.
set upper 4byte to latitude, and lower 4 to longitude.
Now you have a 8 byte long representing a point on world up to 3cm accuracy.
There are other, better solutions, such ones that maintain similar numbers for near locations, but these are more complex.
You can add them up, but it makes little sense.
For instance a total of "10" - 8 lat and 2 long would then be the same as "10" - 3 lat and 7 long.
You can concatenate them, maybe with a dash.
But why do either? They are both really bad choices. A delivery system would want real x-y co-ordinates and if planning a route would want them seperate in order to calculate things like Euclidean distances.
Is this a homework question? I doubt a delivery service is designing their service structure on SO. Least hope not.
Based on AlexWien's anwser this is a solution in JavaScript:
pairCoordinates = function(lat, lng) {
return lat * 1e7 << 16 & 0xffff0000 | lng * 1e7 & 0x0000ffff;
}
How about this:
(lat+90)*180+lng
From Tom Clarkson's comment in Geospatial Indexing with Redis & Sinatra for a Facebook App
If you want to treat location as "one thing", the best way to handle this is to create a data structure that contains both values. A Class for OO languages, or a struct otherwise. Combining them into a single scalar value has little value, even for display.
Location is a really rich problem space, and there are dozens of ways to represent it. Lat/Lon is the tip of the iceberg.
As always, the right answer depends on what you're using it for, which you haven't mentioned.
I have created a method of putting the latitude and longitude into one base-36 number which for now I'm calling a geohexa.
The method works by dividing the world into a 36 x 36 grid. The first character is a longitude and the second character is a latitude. The latitude and longitude those two characters represent is the midpoint of that 'rectangle'. You just keep adding characters, alternating between longitude and latitude. Eventually the geohexa, when converted back to a lat and lon will be close enough to your original lat and lon.
Nine characters will typically get you within 5 meters of a randomly generated lat and lon.
The geohexa for London Bridge is hszaounu and for Tower Bridge is hszaqu88.
It is possible to sort the geohexa, and locations that are near each other will tend to be next to each other in a sorted list to some extent. However it by no means solves the travelling salesman problem!
The project, including a full explanation, implementations in Python, Java and JavaScript can be found here: https://github.com/Qarj/geohexa
You can use the Hilbert space filling curve to convert latitude,longitude into a single number: e.g., https://geocode.xyz/40.71413,-73.96141?geoit=xml 2222211311031 and https://geocode.xyz/40.71413,-73.96151?geoit=xml 2222211311026
The source code is here: https://github.com/eruci/geocode
In a nutshell:
Let X,Y be latitude,longitude
Truncate both to the 5th decimal point and convert to integers multiplying by 100000
Let XY = X+Y and YX = X-Y
Convert XY,YX to binary, and merge them into XYX by alternating the bits
Convert XYX to decimal
Add an extra number (1,2,3,4) to indicate when one or both XY,YX are negative numbers.
Now you have a single number that can be converted back to latitude,longitude and which preserves all their positional properties.
I found I can get good results by adding the latitude and longitude of a particular address by not including the house number and sorting the results in the database table by the added number following by a 2nd sort on the house number in ascending order.
I used this url to get the numbers I needed to add together:
http://where.yahooapis.com/geocode?q=stedman+st,+lowell,+ma
Related
I'm trying to convert and upload latitude and longitude data into a database through an ETL process I created where we take the source data from a .csv file and convert it to DECIMAL. Here you have an example of what the two values look like:
Latitude (first column): 41.896585191199556
Longitude (second column):-87.66454238198166
I set the data type on the database as for:
Latitude DECIMAL(10,8)
Longitude DECIMAL(11,8)
The main problem arises when I try to convert data from file to database and then I get the message
[Flat File Source [85]] Error: Data conversion failed. The data conversion for column "Latitude" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
View of my process:
When trying to ignore the error Latitude and Longitude values in the database are changed to NULL... The flat file encoding is 65001.
I tried doing conversions for data types: float, DECIMAL, int and nothing helped.
My questions are:
what data type for these above values should I use in the target database.
what data type should i choose on input for flat file ?
what data type to set for conversion (I suspect the one we will have on the database) ?
please note that some records in the file are missing the location
view from Data:
view from Data Conversion:
UPDATE
When FastParse is run I receive an error message as below:
What data type should I choose in this case ? I set everything up as #billinkc suggested. When I set an integer, for example DT_I4, it results in NULL and the same error as before (in this message there is no possibility to select some data type for the value of Latitude, i.e. DECIMAL or STRING).
You need DECIMAL(11,8). That has three digits before the decimal place and either digits after.
The conversion failure is no doubt happening when you have longitudes above 100 or less than -100.
The error reported indicates the failure point is the Flat File Source
[Flat File Source [85]] Error: Data conversion failed. The data conversion for column "Latitude" returned status value 2 and status text "The value could not be converted because of a potential loss of data.".
I'm on a US locale machine so you could be running into issues with the decimal separator. If that's the case, then in your Flat File Source, right click and select Show Advanced Editor. Go to Input and Output Properties, and under the Flat File Source Output, expand Output Columns and for each column that is a floating point number, check the FastParse option.
If that works, great, you have a valid Flat File Source.
I was able to get this working two different ways. I defined two Flat File Connection Managers in my package: FFCM Dec and FFCM String While I prefer to minimize the number of operations and transforms I apply to my packages, declaring the data types as strings can help you get past the hurdle of "I can't even get my data flow to start because of bad data"
Source data
I created a CSV saved as UTF-8
Latitude,Longitude
41.896585191199556,-87.66454238198166
FFCM Dec
I configured a standard CSV
I defined my columns with the DataType of DT_DECIMAL
FFCM String
Front page is the same but on the columns in the Advanced section, I left the data type as DT_WSTR with a length of 50
At this point, we've defined the basic properties of how the source data is structured.
Destination
I went with consistency on the size for the destination. You're not going to save anything by using 10 vs 11 and I'm too lazy to look up the allowable domain for lat/long numbers
CREATE TABLE dbo.SO_65909630
(
[Latitude] decimal(18,15)
, [Longitude] decimal(18,15)
)
Data Flow
I need to run but you either use the correctly typed data when you bring it in (DFT DEC) or you transform it.
The blanks I see in your source data will likely need to be dealt with (either you have a column that needed to be escaped or there is no data - which will cause the data conversion to fail so I'd advocate this approach
Row counts are there just to provide a place to put a data viewer while I was building the answer
What data type should I use for lat and long
Decimal is an exact data type so it will store the exact value you supply. When used it takes the form of decimal(scale, precision). Before my current role, I had never used any other data type for non-whole numbers.
Books On Line on decimal and numeric (Transact-SQL) https://learn.microsoft.com/en-us/sql/t-sql/data-types/decimal-and-numeric-transact-sql?view=sql-server-ver15
Scale
The maximum total number of decimal digits to be stored. This number includes both the left and the right sides of the decimal point. The precision must be a value from 1 through the maximum precision of 38. The default precision is 18.
Precision
The number of decimal digits that are stored to the right of the decimal point. This number is subtracted from p to determine the maximum number of digits to the left of the decimal point. Scale must be a value from 0 through p, and can only be specified if precision is specified. The default scale is 0 and so 0 <= s <= p. Maximum storage sizes vary, based on the precision.
Precision Storage bytes
1 - 9 5
10-19 9
20-28 13
29-38 17
For the table I defined above, it will cost us 18 bytes (2 * 9) for each lat/long to store.
But let's look at the actual domain for latitude and longitude (on Earth) This magnificent answer on GIS.se is printed out and hangs from my work monitor https://gis.stackexchange.com/questions/8650/measuring-accuracy-of-latitude-and-longitude
Pasting the relevant bits here
The sixth decimal place is worth up to 0.11 m: you can use this for laying out structures in detail, for designing landscapes, building roads. It should be more than good enough for tracking movements of glaciers and rivers. This can be achieved by taking painstaking measures with GPS, such as differentially corrected GPS.
The seventh decimal place is worth up to 11 mm: this is good for much surveying and is near the limit of what GPS-based techniques can achieve.
The eighth decimal place is worth up to 1.1 mm: this is good for charting motions of tectonic plates and movements of volcanoes. Permanent, corrected, constantly-running GPS base stations might be able to achieve this level of accuracy.
The ninth decimal place is worth up to 110 microns: we are getting into the range of microscopy. For almost any conceivable application with earth positions, this is overkill and will be more precise than the accuracy of any surveying device.
Ten or more decimal places indicates a computer or calculator was used and that no attention was paid to the fact that the extra decimals are useless. Be careful, because unless you are the one reading these numbers off the device, this can indicate low quality processing!
Your input values show more than 10 digits of precision so I'm guessing it's a calculated value and not a "true observation". That's good, that gives us more wiggle room to work with.
Why, we could dial that decimal declaration down the following for half* the storage cost of the first one
CREATE TABLE dbo.SO_65909630_alt
(
[Latitude] decimal(8,5)
, [Longitude] decimal(8,5)
);
Well that's good, we've stored the "same" data at lower the cost. Maybe your use case is just "where are my stores" and even if you're Walmart with under 12000 stores, who cares? That's a trivial cost. But if you need to also store the coordinates of their customers, the storage cost per record might start to matter. Or use Amazon or Alibaba or whatever very large consumer retailer exists when you read this.
In my work, I deal with meteorological data and it comes in all shapes and sizes but a common source for me is Stage IV data It's just hourly rainfall amounts across the contiguous US. So 24 readings per coordinate, per day. Coordinate system is 1121 x 881 (987,601 points) so expressing hourly rainfall in the US for a day is 23,702,424 rows. The difference between 18 bytes versus 10 bytes can quickly become apparent given that Stage IV data is available back to 2008.
We actually use a float (or real) to store latitude and longitude values because it saves us a 2 bytes per coordinate.
CREATE TABLE dbo.SO_65909630_float
(
[Latitude] float(24)
, [Longitude] float(24)
);
INSERT INTO dbo.SO_65909630_alt
(
Latitude
, Longitude
)
SELECT * FROM dbo.SO_65909630 AS S
Now, this has caused me pain because I can't use an exact filter in queries because of the fun of floating point numbers.
My decimal typed table has this in it
41.89659 -87.66454
And my floating type table has this in it
41.89658 -87.66454
Did you notice the change to the last digit in Latitude? 8 not 9 as the decimal table has but either way, it doesn't matter
SELECT * FROM dbo.SO_65909630_float AS S WHERE S.Latitude = 41.89658
This won't find a row because of floating point rounding exact match nonsense. Instead, your queries become very tight range queries, like
SELECT * FROM dbo.SO_65909630_float AS S WHERE S.Latitude >= (41.89658 - .00005) AND S.Latitude <= (41.89658 + .00005)
where .00005 is a value that you'll have to experiment with given your data to find out how much you need to adjust the numbers to find it again.
Finally, for what it's worth, if you convert lat and long into the Geography Point it's going to coerce the input data type to float as it is.
I'm looking at the freely available Solar potential dataset on Google BigQuery that may be found here: https://bigquery.cloud.google.com/table/bigquery-public-data:sunroof_solar.solar_potential_by_censustract?pli=1&tab=schema
Each record on the table has the following border definitions:
lat_max - maximum latitude for that region
lat_min - minimum latitude for that region
lng_max - maximum longitude for that region
lng_min - minimum longitude for that region
Now I have a coordinate (lat/lng pair) and I would like to query to see whether or not that coordinate is within the above range. How do I do that with BQ Standard SQL?
I've seen the Geo Functions here: https://cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions
But I'm still not sure how to write this query.
Thanks!
Assuming the points are just latitude and longitude as numbers, why can't you just do a standard numerical comparison?
Note: The first link doesn't work without a google account, so I can't see the data.
But if you want to become spatial, I'd suggest you're going to need to take the border coordinates that you have and turn them into a polygon using one of: ST_MAKEPOLYGON, ST_GEOGFROMGEOJSON, or ST_GEOGFROMTEXT. Then create a point using the coords you wish to test ST_MAKEPOINT.
Now you have two geographies you can compare them both using ST_INTERSECTION or ST_DISJOINT depending on what outcome you want.
If you want to get fancy and see how far aware from the border you are (which I guess means more efficient?) you can use ST_DISTANCE.
Agree with Jonathan, just checking if each of the lat/lon value is within the bounds is simplest way to achieve it (unless there are any issues around antimeridian, but most likely you can just ignore them).
If you do want to use Geography objects for that, you can construct Geography objects for these rectangles, using
ST_MakePolygon(ST_MakeLine(
[ST_GeogPoint(lon_min, lat_min), ST_GeogPoint(lon_max, lat_min),
ST_GeogPoint(lon_max, lat_max), ST_GeogPoint(lon_min, lat_max),
ST_GeogPoint(lon_min, lat_min)]))
And then check if the point is within particular rectangle using
ST_Intersects(ST_GeogPoint(lon, lat), <polygon-above>)
But it will likely be slower and would not provide any benefit for this particular case.
I have an application with a sqlite database that contains 7000+ records in it with city names, longitudes and latitudes.. also these "cities" are connected to relevant city fields on the database too.
What my app doing is, query the current location with core location, fetch the lon and lat values, and then find the closest location from the database.
The result doesn't have to be super accurate (i just want to match cities), so I want to use Hypotenuse formula for finding the closest point:
closest city in db: min((x1-x2)^2 +(y1-y2)^2)^(1/2)
x1, y1: lon and lat for user
x2, y2: lon and lat for points in database.
If I was using ms-sql or sqlite database, I could easily create a query but when it comes to core data, I'm out of ideas.
I don't want to fetch all the data (and fill the memory) then aggregate this formula on all fields so is there a way to create a query and get the result from the db?
Am I overthinking this problem, and missing a simple solution?
If I'm understanding your problem correctly, you're wanting to find the closest "n" cities to your current location.
I had something similar and here's how I approached it.
In essence, you probably need to take each city's lat/lon and hash it into some index. We use a Mercator Projection to convert the lat/lon to x/y, then hash that value in a manner similar to how Google/Bing/Apple Maps hash their map tiles. Fortunately, MapKit has a built-in Mercator Projection function.
In pseudocode:
for each city's lat/lon {
CLLocationCoordinate2D coordinate = (CLLocationCoordinate2D){lat, lon};
MKMapPoint point = MKMapPointForCoordinate(coordinate);
//256 represents the size of a map tile at zoomLevel 20. You can use whatever zoomLevel
//you want here, but we need something to quickly lookup close-by cities.
//this is the formula you can use to determine how granular your index is
//(256 * pow(2, (20 - zoomLevel)))
NSInteger x = point.x/256.0;
NSInteger y = point.y/256.0;
save x & y in a CityHashIndex table
}
Now, you get the current location's lat/lon, hash that into the index as above, and just simply write a query against this CityHashIndex table.
So say that, for simplicity sake, you're current location is indexed at 1000, 1000. So to find close by cities, maybe you search for cities with indexes in the range of `900-1100, 900-1100'.
From there, you're now only pulling in a much smaller set of cities and the memory requirements to process your Hypotenuse Formula isn't so bad.
I can elaborate more if you're interested.
This is directly related to a commonly asked question about Core Data.
Searching for surrounding suburbs based on latitude & longitude using Objective C
Calculate a bounding box around the point you need (min lat/long max lat/long) then use an NSPredicate against those values to find everything within the box. From there you can do a distance calculation on the results that return and sort them.
I would suggest setting this up so that it can search at multiple distances then you can see if a city is within 10 miles, 100 miles, etc. Slowly increasing the bounding box until you get one or more results back.
I would use NSPredicate to define my search criteria it will act as a filter. I'm not sure how optimized is this and if it will pull all your registers but I'm assuming that coreData has some kind of indexing mechanism that will optimize the search.
You can take a look of this document
https://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/CoreData/Articles/cdFetching.html
Check the section named
Retrieving Specific Objects
I have a coordinate, assume any latitude and longitude values.
I have a domain setup on SimpleDB that has many items (simple strings) with attributes of 'Latitude' and 'Longitude'. Now what I want to do is query SimpleDB and see if the current location coordinates are 'x' meters apart from SimpleDB's items' coordinates. 'x' should be 10.
My app uploads an item to SimpleDB with an attribute that contains the latitude and longitude. I detect the users location, get the coordinates and I want to use a select expression to see if the coordinates are 'x' meters apart. So is their a better approach to doing this? Or is this is the best way, if so, how can I do it?
Here is an example of what the select expression may look like, I just have no idea how to use it in this case and what the '%#' values would be filled in by. This whole format could be off, its just my idea.
select * from test-app-simpledb where Latitude >= '%#' AND Latitude <= '%#' AND Longitude >= '%#' AND Longitude <= '%#'
So "test-app-simpledb" is my SimpleDB domain name, Latitude and Longitude are the attributes I compare to the coordinates. They are all converted to string.
So, how can I do location comparative select expressions. Querying if the item's coordinate (latitude and longitude attribute) are 'x' (in this case 10) meters apart.
Any way to do this? Thanks!
In the revised question, it has become clear that the actual question is how to conduct a query against a Amazon SimpleDB database to see if locations in the database are within a certain radius of a location provided by an iOS app.
As you correctly identify, you really want to do this server-side, rather than client-side, if possible. And the particular solutions will be highly dependent upon the particular database technology (SimpleDB in this case).
This question is touched upon in Spatial queries on AWS SimpleDB. I would suggest checking that out for more information.
As a proxy for a proper distance algorithm, you could translate the distance in meters into a ranges of latitudes and longitudes. Thus, you could, in iOS, calculate a minimum and maximum for both latitude and longitude and then pass those along in the WHERE clause to your remote database. Then the server could filter results based upon those criteria. That admittedly gives you a square-shaped region (rather than a circular region that you get by calculating distances properly), but it makes it really easy to quickly limit the result set with no special geolocation logic required on the server. To do this, you could define a region with MKCoordinateRegionMakeWithDistance, and then grab its span.
MKCoordinateRegion region = MKCoordinateRegionMakeWithDistance(originalCoordinate, 200, 200);
MKCoordinateSpan span = region.span;
That gives you span.latitudeDelta and span.longitudeDelta which you can center around a given location's latitude and longitude to define a square shaped region around a location. To demonstrate that in action, here the center pin is my location at Times Square, and the shaded area is a region (constructed using the above span which is 200m wide and tall), which covers longitudes between 100m east and 100m west of my current location, as well as latitudes 100m north and 100m south of my current location.
This is a way to employ MapKit.framework functions to greatly streamline your SQL queries for remote databases to filter locations based upon geographic distance. If the square region is problematic, you could also further filter the results once they're downloaded to the iOS app using the CLLocation instance method, distanceFromLocation to determine the actual distance. But by limiting the longitudes and latitudes of locations retrieved by the server, you dramatically streamline the server retrieval process while not encumbering it with too much geographic location calculations.
But you really want to have SimpleDB do the full, proper distance calculation, I'll have to leave that to others.
Below, is my original answer. In the original question, I misinterpreted it as being "how do I construct a list of coordinates going in a circle around a particular location?" As made clear by the revised question, that was not the issue at all, but I'll keep my old answer here for historical reference.
Original answer:
If you used Calculate new coordinate x meters and y degree away from one coordinate, the implementation might look like:
NSInteger numberOfPoints = 10;
for (double bearing = 0.0; bearing < 360.0; bearing += (360.0 / numberOfPoints))
{
CLLocationCoordinate2D coordinate = [self coordinateFromCoord:originalCoordinate
atDistanceKm:distanceKm
atBearingDegrees:bearing];
// do whatever you want with this coordinate
}
It seems to work fine. For example, I had an app use this routine to drop 10 pins 100m from me in Times Square:
In the table definition I saw:
Latitude -> varchar(50)
Longitude -> nvarchar(50)
Immediately, obviously, I queried the thinking behind this - being positively sure these values are in fact numerical by nature. Long story short: I postulated that these will be numerical, decimal in fact, and we would discard the 'thinking-in-strings' philosophy.
Now for the horns of my dilemma, I just went ahead and typed:
Latitude -> decimal(2, 4)
But hold on a second, 4 ain't right, right? Right. So I thought I'd up the threshold before realising (in a split second might I add) that 6 or 8 might not not cut it either. So, first things first...
Am I right in insisting we even go about it this way? And if so...
To what precision ought these values be stored to ensure we can persist the entire value which is to be inserted? For example, is there anything predefined by specification?
I don't just want to use something like Latitude -> decimal(2, 16) simply for it to be just as flawed as decimal(2, 2) in principle. And a similar question arises for Longitude specifically but I'm assuming the answer to one will suffice for the other, i.e decimal(3, answer).
We are using MSSQL Server 2005.
It seems I am educating myself with SQL Server by manual experience and therefore rendering parts of this question irrelevant: I can only use decimal(x, max(x)) not decimal(x, y) anyway! Will leave the question as is for input.
Decimal(2, 4) means 2 total digits of precision and 4 behind the decimal. SQL Server won't let you do that, but I think it would means you can store values from -0.0099 to 0.0099.
I'd recommend decimal(9, 6). This will store with an accuracy down to about 1/6th of an inch at the equator. Using 9 or less as the precision requires 5 bytes of storage, using 10-19 requires 9 bytes.
The maximum precision of decimals in sql server is currently 38. The scale can be up to 38. At it's max, a decimal will take up 17 bites, where as a varchar takes up whatever the length is plus 2. So if you went with a varchar(38), at it's max you're taking up 40 bits of data. The flip side is that a varchar is not as limited in size as a decimal is. So what you really need to do is figure out how many decimal points you're going to allow and then figure out your data type for it.
Source Info