In BigQuery, given a country in ISO-2 code I need to get its centroids coordinates (lat and long).
There is a way to do this?
Looking into the geography functions of BQ I did not find a way.
You can use the bigquery-public-data.geo_openstreetmap.planet_features table from BigQuery public datasets to calculate the centroids of countries. The inner query is based on this answer from Stack Overflow. Consider the below query and output.
SELECT -- Extract country code, lat and long
CountryISO2Code,
ST_X(centroid) AS longitude,
ST_Y(centroid) AS latitude
FROM (SELECT (SELECT value FROM UNNEST(all_tags)
WHERE key = 'ISO3166-1:alpha2') AS CountryISO2Code,
ST_CENTROID(geometry) centroid -- calculate the centroid with the geometry values
FROM `bigquery-public-data.geo_openstreetmap.planet_features`
WHERE EXISTS (SELECT 1 FROM UNNEST(all_tags) WHERE key='boundary' AND value= 'administrative')
AND EXISTS (SELECT 1 FROM UNNEST(all_tags) WHERE key='admin_level' AND value = '2')
AND EXISTS (SELECT 1 FROM UNNEST(all_tags) WHERE key='ISO3166-1:alpha2'))
WHERE CountryISO2Code="IN" -- country code to filter the output
Output of the above query
Please note that there are some country codes that are not available in the table. Also, OSM dataset in BigQuery itself is produced as a public good by volunteers, and there are no guarantees about data quality.
I also found some community-maintained data from Github like this one which can be imported into BigQuery and is ready-to-use.
Related
I have three SQL tables:
A list of 100k weather stations with a latitude and longitude coordinate
A list of 15 cities of interest with a latitude and longitude coordinate
A list of weather data for each weather station
My interest right now is only with the first two tables. How do I filter the list of weather stations to those within e.g. 100km of each city of interest?
I have a Microsoft SQL Server and I'd prefer to do it within SQL if possible.
Basically, if you try to do this yourself, you end up with a Cartesian product:
select c.*, ws.*
from cities c cross apply
(select ws.*
from (select ws.*,
<complicated expression to calculate distance> as distance
from weather_station ws
) ws
where distance < 100
) ws;
In order to get the list of weather stations, all cities and weather stations have to be compared. The distance calculation is often rather expensive, so you can cut down on this by "prefiltering". For instance, in most inhabited places, 100 km is within 1 degree latitude and 2 degrees longitude:
select c.*, ws.*
from cities c cross apply
(select ws.*
from (select ws.*,
<complicated expression to calculate distance> as distance
from weather_station ws
where ws.latitutde between c.latitude - 1 and c.latitude + 1 and
ws.longitude between c.longitude - 2 and c.longitude + 2
) ws
where distance < 100
) ws;
Although that helps, this is still essentially a filtered Cartesian product.
So, what should you really do? If you care about coordinates as spatial data, you should look into SQL Server's spatial extensions (the documentation is here, particularly the geography type because that is most relevant to your needs).
As Gordon mentioned, you can define spatial geography datatype for your needs. You can follow below steps to achieve the goal.
Store the latitude, longitude data in the Point
Now, use the STDistance to calculate the distance between two points
You can leverage common scenario of finding nearest neighbor
I have a polygons table in Postgres (using PostGIS extension) named polygon having two fields (geom, id).
If I want to query the id of the polygon which intersects with the geometry of input geo-coordinate then I can do it with the below query.
SELECT id, geom
FROM polygon
WHERE ST_Intersects(polygon.%s, ST_GeometryFromText(POINT(latitude logitude), 4326));
But now I have a use case where I am getting a lot of geo-coordinates in request(~60k), now I am breaking this into lists of 1k Geo-coordinate each and querying the id of the polygon intersecting with each geo-coordinate.
I am struggling with how to write. a query for this, or if anyone has a better solution for this please suggest.
Keep in mind that the right order of coordinate pairs is lon, lat, so creating a point with lat, lon in your query will return wrong results. Your query also misses the single quotes ' around the WKT coordinate, e.g. 'POINT(1 2)'.
That all being said, you could simply paginate your result sets using ORDER BY, LIMIT and OFFSET, e.g.
Getting the first 1000 records
SELECT id, geom FROM polygon
WHERE ST_Intersects(geom, 'SRID=4326;POINT(1 2)')
ORDER BY id
LIMIT 1000 OFFSET 0;
By changing the OFFSET value you are able to retrieve the next pages.
LIMIT 1000 OFFSET 1000;
and so on ..
LIMIT 1000 OFFSET 2000;
EDIT: One way to apply this query using multiple input points is to use a CTE / subquery (see comments), e.g.
WITH j(g) AS (
VALUES
('SRID=4326;POINT(1 1)'),
('SRID=4326;POINT(1 2)')
-- ... add as many geometries as you want
)
SELECT id, geom FROM polygon, j
WHERE ST_Intersects(geom, g::geometry)
I currently have an Azure postgresql database containing openstreetmap data and I was wondering if there's a SQL query that can get the total distance of a way by using the lat/longs of the nodes the way uses.
I would like the SQL query to return way_id and distance.
My current approach is using C# to download all the ways and all the nodes into dictionaries (with their id's being the key). I then loop through all the ways, grouping all the nodes that belong to that way and then use their lat/longs (value divided by 10000000) to calculate the distance. This part works as excepted but rather it be done on the server.
The SQL I have attempted is below but I'm stuck on calculating the total distance per way based on the lat/longs.
Update: Postgis extension is installed.
SELECT current_ways.id as wId, node_id, (CAST(latitude as float)) / 10000000 as lat, (CAST(longitude as float)) / 10000000 as lon FROM public.current_ways
JOIN current_way_nodes as cwn ON current_ways.id = cwn.way_id
JOIN current_nodes as cn ON cwn.node_id = cn.id
*output*
wId node_id latitude longitude
2 1312575 51.4761127 -3.1888786
2 1312574 51.4759647 -3.1874216
2 1312573 51.4759207 -3.1870016
2 1213756 51.4758761 -3.1865223
3 ....
*desired_output*
way_id length
2 x.xxx
3 ...
**Tables**
current_nodes
id
latitude
longitude
current_ways
id
current_way_nodes
way_id
node_id
sequence_id
It would be much simpler should you also had the geometry in your table, i.e. the actual point instead of just the coordinates, or, even better, the actual lines.
That being said, here is a query to get what you are looking for:
SELECT w.way_id,
ST_Length( -- compute the length
ST_MAKELINE( --of a new line
ST_SetSRID( --made of an aggregation of NEW points
ST_MAKEPOINT((CAST(longitude as float)) / 10000000,(CAST(latitude as float)) / 10000000), --created using the long/lat from your text fields
4326) -- specify the projection
ORDER BY w.sequence_id -- order the points using the given sequence
)::geography --cast to geography so the output length will be in meters and not in degrees
) as length_m
FROM current_way_nodes w
JOIN current_nodes n ON w.node_id = n.node_id
GROUP BY w.way_id;
New to PostGIS/PostgreSQL...any help would be greatly appreciated!
I have two tables in a postgres db aliased as gas and ev. I'm trying to choose a specific gas station (gas.site_id=11949) and locate all EV/alternative fuel charging stations within a 1000m radius. When I run the following though, PostGIS returns a number of ev stations that are all stacked on top of each other in the map (see screenshot).
Anyone have any idea why this is happening? How can I get PostGIS to visualize the points within a 1000m radius of the specified gas station?
with myplace as (
SELECT gas.geom
from nj_gas gas
where gas.site_id = 11949 limit 1)
select myplace.*, ev.*
from alt_fuel ev, myplace
where ST_DWithin(ev.geom1, myplace.geom, 1000)
The function ST_DWithin does not compute distances in meters using geometry typed parameters.
From the documentation:
For geometry: The distance is specified in units defined by the
spatial reference system of the geometries. For this function to make
sense, the source geometries must both be of the same coordinate
projection, having the same SRID.
So, if you want compute distances in meters you have to use the data type geography:
For geography units are in meters and measurement is defaulted to
use_spheroid=true, for faster check, use_spheroid=false to measure
along sphere.
That all being said, you have to cast the data type of your geometries. Besides that your query looks just fine - considering your data is correct :-)
WITH myplace as (
SELECT gas.geom
FROM nj_gas gas
WHERE gas.site_id = 11949 LIMIT 1)
SELECT myplace.*, ev.*
FROM alt_fuel ev, myplace
WHERE ST_DWithin(ev.geom1::GEOGRAPHY, myplace.geom::GEOGRAPHY, 1000)
Sample data:
CREATE TABLE t1 (id INT, geom GEOGRAPHY);
INSERT INTO t1 VALUES (1,'POINT(-4.47 54.22)');
CREATE TABLE t2 (geom GEOGRAPHY);
INSERT INTO t2 VALUES ('POINT(-4.48 54.22)'),('POINT(-4.41 54.18)');
Query
WITH j AS (
SELECT geom FROM t1 WHERE id = 1 LIMIT 1)
SELECT ST_AsText(t2.geom)
FROM j,t2 WHERE ST_DWithin(t2.geom, j.geom, 1000);
st_astext
--------------------
POINT(-4.48 54.22)
(1 Zeile)
You are cross joining those tables and have PostgreSQL return the cartesian product of both when selecting myplace.* & ev.*.
So while there is only one row in myplace, its geom will be merged with every row of alt_fuel (i.e. the result set will have all columns of both tables in every possible combination of both); since the result set thus has two geometry columns, your client application likely chooses either the first, or the one called geom (as opposed to alt_fuel.geom1) to display!
I don't see that you are interested in myplace.geom in the result set anyway, so I suggest to run
WITH
myplace as (
SELECT gas.geom
FROM nj_gas gas
WHERE gas.site_id = 11949
LIMIT 1
)
SELECT ev.*
FROM alt_fuel AS ev
JOIN myplace AS mp
ON ST_DWithin(ev.geom1, mp.geom, 1000) -- ST_DWithin(ev.geom1::GEOGRAPHY, mp.geom::GEOGRAPHY, 1000)
;
If, for some reason, you also want to display myplace.geom along with the stations, you'd have to UNION[ ALL] the above with a SELECT * on myplace; note that you will also have to provide the same column list and structure (same data types!) as alt_fuel.* (or better, the other side of the UNION[ ALL]) in that SELECT!
Note the suggestions made by #JimJones about units; if your data is not projected in a meter based CRS (but in a geographic reference system; 'LonLat'), use the cast to GEOGRAPHY to have ST_DWithin consider the input as meter (and calculate using spheroidal algebra instead of planar (Euclidean))!
Resolved by using:
WITH
myplace as (
SELECT geom as g
FROM nj_gas
WHERE site_id = 11949 OR site_id = 11099 OR site_id = 11679 or site_id = 480522
), myresults AS (
SELECT ev.*
FROM alt_fuel AS ev
JOIN myplace AS mp
ON ST_DWithin(ev.geom, mp.g, 0.1))
select * from myresults```
Thanks so much for your help #ThingumaBob and #JimJones ! Greatly appreciate it.
Output of what I would like to get
This is what I'd like to get to:
I have a Table which holds my store locations and their long and lat in three columns, and another table which holds my customer key their long and lat. in my output id like to see a column with closest store and a column with distance to the closest store.
Thank you so much for your help!
For future readers, I found how to do it:
1) DECLARE #start_location GEOGRAPHY = GEOGRAPHY::Point(Lat,Long,4326)
2) SELECT ID, #start_location.STDistance(Location)/1609.344 [YourTableName] from (SELECT ID, GEOGRAPHY::Point(Latitude, Longitude, 4326) as Location FROM [TheTableWhereYourZipInfoIsStored]
notes: /1609.344 is because you want to convert to miles from meters.