I want to transform the corner coordinates of a UTM projected file into longlat. If I read coordinates from a GeoTIFF, the function works as expected. But, after converting the GeoTiff to ENVI raw file (with gdal_translate), the coordinates are flipped:
from osgeo import gdal
from osgeo import osr
def corner2longlat(fname):
dataset = gdal.Open(fname, gdal.GA_ReadOnly)
ds_geotrans = dataset.GetGeoTransform()
X = ds_geotrans[0]
Y = ds_geotrans[3] + dataset.RasterYSize * ds_geotrans[5]
srs_projection = dataset.GetProjectionRef()
srs = osr.SpatialReference()
srs.ImportFromWkt(srs_projection)
srsLatLong = srs.CloneGeogCS()
ct = osr.CoordinateTransformation(srs, srsLatLong)
latlon = ct.TransformPoint(X, Y)
print(srs_projection)
print([latlon[0], latlon[1]])
image_tif = "all_bands.tif"
image_raw = "all_bands.raw"
corner2longlat(image_tif) gives then:
PROJCS["WGS 84 / UTM zone 43N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",75],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32643"]]
[41.463751886708124, 74.99976050569691]
corner2longlat(image_raw) gives then:
PROJCS["WGS 84 / UTM zone 43N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",75],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH]]
[74.99976050569691, 41.463751886708124]
Does anybody have an idea how that could happen?
Thank you in advance
Lukas
Related
I need to build a relief profile graph by coordinates, I have a csv file with 12,000,000 lines. searching through a csv file of the same height takes about 2 - 2.5 seconds. I rewrote the csv to parquet and it helped me save some time, it takes about 1.7 - 1 second to find one height. However, I need to build a profile for 500 - 2000 values, which makes the time very long. In the future, you may have to increase the base of the csv file, which will slow down this process even more. In this regard, my question is, is it possible to somehow reduce the processing time of values?
Code example:
import dask.dataframe as dk
import numpy as np
import pandas as pd
import time
filename = 'n46_e032_1arc_v3.csv'
df = dk.read_csv(filename)
df.to_parquet('n46_e032_1arc_v3_parquet')
Latitude1y, Longitude1x = 46.6276, 32.5942
Latitude2y, Longitude2x = 46.6451, 32.6781
sec, steps, k = 0.00027778, 1, 11.73
Latitude, Longitude = [Latitude1y], [Longitude1x]
sin, cos = Latitude2y - Latitude1y, Longitude2x - Longitude1x
y, x = Latitude1y, Longitude1x
while Latitude[-1] < Latitude2y and Longitude[-1] < Longitude2x:
y, x, steps = y + sec * k * sin, x + sec * k * cos, steps + 1
Latitude.append(y)
Longitude.append(x)
time_start = time.time()
long, elevation_data = [], []
df2 = dk.read_parquet('n46_e032_1arc_v3_parquet')
for i in range(steps + 1):
elevation_line = df2[(Longitude[i] <= df2['x']) & (df2['x'] <= Longitude[i] + sec) &
(Latitude[i] <= df2['y']) & (df2['y'] <= Latitude[i] + sec)].compute()
elevation = np.asarray(elevation_line.z.tolist())
if elevation[-1] < 0:
elevation_data.append(0)
else:
elevation_data.append(elevation[-1])
long.append(30 * i)
plt.bar(long, elevation_data, width = 30)
plt.show()
print(time.time() - time_start)
Here's one way to solve this problem using KD trees. A KD tree is a data structure for doing fast nearest-neighbor searches.
import scipy.spatial
tree = scipy.spatial.KDTree(df[['x', 'y']].values)
elevations = df['z'].values
long, elevation_data = [], []
for i in range(steps):
lon, lat = Longitude[i], Latitude[i]
dist, idx = tree.query([lon, lat])
elevation = elevations[idx]
if elevation < 0:
elevation = 0
elevation_data.append(elevation)
long.append(30 * i)
Note: if you can make assumptions about the data, like "all of the points in the CSV are equally spaced," faster algorithms are possible.
It looks like your data might be on a regular grid. If (and only if) every combination of x and y exist in your data, then it probably makes sense to turn this into a labeled 2D array of points, after which querying the correct position will be very fast.
For this, I'll use xarray, which is essentially pandas for N-dimensional data, and integrates well with dask:
# bring the dataframe into memory
df = dk.read('n46_e032_1arc_v3_parquet').compute()
da = df.set_index(["y", "x"]).z.to_xarray()
# now you can query the nearest points:
desired_lats = xr.DataArray([46.6276, 46.6451], dims=["point"])
desired_lons = xr.DataArray([32.5942, 32.6781], dims=["point"])
subset = da.sel(y=desired_lats, x=desired_lons, method="nearest")
# if you'd like, you can return to pandas:
subset_s = subset.to_series()
# you could do this only once, and save the reshaped array as a zarr store:
ds = da.to_dataset(name="elevation")
ds.to_zarr("n46_e032_1arc_v3.zarr")
I have a few annotations that is originally in .csv format. I would need to convert it to yolo darknet format inorder to train my model with yolov4.
my .csv file :
YOLO format is : object-class x y width height
where, object_class, widht, height is know from my .csv format. But finding x,y is confusing .Note that x and y are center of rectangle (are not top-left corner).
Any help would be appreciated :)
You can use this function to convert bounding boxes to the yolo format. Of course you will need to write some code to read the csv. Just use this function as a template for your needs.
This function was extracted from the labelimg app:
https://github.com/tzutalin/labelImg/blob/master/libs/yolo_io.py
def BndBox2YoloLine(self, box, classList=[]):
xmin = box['xmin']
xmax = box['xmax']
ymin = box['ymin']
ymax = box['ymax']
xcen = float((xmin + xmax)) / 2 / self.imgSize[1]
ycen = float((ymin + ymax)) / 2 / self.imgSize[0]
w = float((xmax - xmin)) / self.imgSize[1]
h = float((ymax - ymin)) / self.imgSize[0]
# PR387
boxName = box['name']
if boxName not in classList:
classList.append(boxName)
classIndex = classList.index(boxName)
return classIndex, xcen, ycen, w, h
I am following the example from Astropy docs for 2D Cutout.
The header of my FITS file:
SIMPLE = T / file does conform to FITS standard
BITPIX = -32 / number of bits per data pixel
NAXIS = 3 / number of data axes
NAXIS1 = 512 / length of data axis 1
NAXIS2 = 512 / length of data axis 2
NAXIS3 = 3 / length of data axis 3
EXTEND = T / FITS dataset may contain extensions
COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
SURVEY = 'DECaLS '
VERSION = 'DR8-south'
IMAGETYP= 'image '
BANDS = 'grz '
BAND0 = 'g '
BAND1 = 'r '
BAND2 = 'z '
CTYPE1 = 'RA---TAN' / TANgent plane
CTYPE2 = 'DEC--TAN' / TANgent plane
CRVAL1 = 186.11382 / Reference RA
CRVAL2 = 0.15285422 / Reference Dec
CRPIX1 = 256.5 / Reference x
CRPIX2 = 256.5 / Reference y
CD1_1 = -7.27777777777778E-05 / CD matrix
CD1_2 = 0. / CD matrix
CD2_1 = 0. / CD matrix
CD2_2 = 7.27777777777778E-05 / CD matrix
IMAGEW = 512. / Image width
IMAGEH = 512. / Image height
So far what I have tried :
from astropy.coordinates import SkyCoord
from astropy.wcs import WCS
position = SkyCoord(hdu[0].header['CRVAL1']*u.deg,hdu[0].header['CRVAL2']*u.deg)
size = 200*u.pixel
wcs1 = WCS(hdu[0].header)
cutout = Cutout2D(hdu[0].data[0], position ,size, wcs = wcs1 )
I run into error in the last line.
Error :
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-142-7cc21a13e941> in <module>
----> 1 cutout = Cutout2D(hdu[0].data[0], position ,size, wcs = wcs1 )
/Applications/anaconda3/lib/python3.7/site-packages/astropy/nddata/utils.py in __init__(self, data, position, size, wcs, mode, fill_value, copy)
714 if wcs is not None:
715 self.wcs = deepcopy(wcs)
--> 716 self.wcs.wcs.crpix -= self._origin_original_true
717 self.wcs.array_shape = self.data.shape
718 if wcs.sip is not None:
ValueError: operands could not be broadcast together with shapes (3,) (2,) (3,)
My guess is it is because naxis =3 in my file and the documentation assumes naxis = 2. Though I am not sure if that is the actual issue here. Can anybody help fix this error?
Since your WCS is 3d, but you're getting a 2d cutout, you need to drop the 3rd dimension. Try cutout = Cutout2D(hdu[0].data[0], position ,size, wcs = wcs1.celestial ), where .celestial is a convenience tool to drop the third dimension in the WCS.
Can anyone help me with parameters for SetGeoTransform? I'm creating raster layers with GDAL, but I can't find description of 3rd and 5th parameter for SetGeoTransform. It should be definition of x and y axis for cells. I try to find something about it here and here, but nothing.
I need to find description of these two parameters... It's a value in degrees, radians, meters? Or something else?
The geotransform is used to convert from map to pixel coordinates and back using an affine transformation. The 3rd and 5th parameter are used (together with the 2nd and 4th) to define the rotation if your image doesn't have 'north up'.
But most images are north up, and then both the 3rd and 5th parameter are zero.
The affine transform consists of six coefficients returned by
GDALDataset::GetGeoTransform() which map pixel/line coordinates into
georeferenced space using the following relationship:
Xgeo = GT(0) + Xpixel*GT(1) + Yline*GT(2)
Ygeo = GT(3) + Xpixel*GT(4) + Yline*GT(5)
See the section on affine geotransform at:
https://gdal.org/tutorials/geotransforms_tut.html
I did do like below code.
As a result I was able to do same with SetGeoTransform.
# new file
dst = gdal.GetDriverByName('GTiff').Create(OUT_PATH, xsize, ysize, band_num, dtype)
# old file
ds = gdal.Open(fpath)
wkt = ds.GetProjection()
gcps = ds.GetGCPs()
dst.SetGCPs(gcps, wkt)
...
dst.FlushCache()
dst = Nonet
Given information from the aforementioned gdal datamodel docs, the 3rd & 5th parameters of SatGeoTransform (x_skew and y_skew respectively) can be calculated from two control points (p1, p2) with known x and y in both "geo" and "pixel" coordinate spaces. p1 should be above-left of p2 in pixelspace.
x_skew = sqrt((p1.geox-p2.geox)**2 + (p1.geoy-p2.geoy)**2) / (p1.pixely - p2.pixely)`
y_skew = sqrt((p1.geox-p2.geox)**2 + (p1.geoy-p2.geoy)**2) / (p1.pixelx - p2.pixelx)`
In short this is the ratio of Euclidean distance between the points in geospace to the height (or width) of the image in pixelspace.
The units of the parameters are "geo"length/"pixel"length.
Here is a demonstration using the corners of the image stored as control points (gcps):
import gdal
from math import sqrt
ds = gdal.Open(fpath)
gcps = ds.GetGCPs()
assert gcps[0].Id == 'UpperLeft'
p1 = gcps[0]
assert gcps[2].Id == 'LowerRight'
p2 = gcps[2]
y_skew = (
sqrt((p1.GCPX-p2.GCPX)**2 + (p1.GCPY-p2.GCPY)**2) /
(p1.GCPPixel - p2.GCPPixel)
)
x_skew = (
sqrt((p1.GCPX-p2.GCPX)**2 + (p1.GCPY-p2.GCPY)**2) /
(p1.GCPLine - p2.GCPLine)
)
x_res = (p2.GCPX - p1.GCPX) / ds.RasterXSize
y_res = (p2.GCPY - p1.GCPY) / ds.RasterYSize
ds.SetGeoTransform([
p1.GCPX,
x_res,
x_skew,
p1.GCPY,
y_skew,
y_res,
])
I'm trying to copy the Scipy Cookbook function:
from scipy import ogrid, sin, mgrid, ndimage, array
x,y = ogrid[-1:1:5j,-1:1:5j]
fvals = sin(x)*sin(y)
newx,newy = mgrid[-1:1:100j,-1:1:100j]
x0 = x[0,0]
y0 = y[0,0]
dx = x[1,0] - x0
dy = y[0,1] - y0
ivals = (newx - x0)/dx
jvals = (newy - y0)/dy
coords = array([ivals, jvals])
newf = ndimage.map_coordinates(fvals, coords)
by using my own function that has to work for many scenarios
import scipy
import numpy as np
"""N-D interpolation for equally-spaced data"""
x = np.c_[plist['modx']]
y = np.transpose(np.c_[plist['mody']])
pdb.set_trace()
#newx,newy = np.meshgrid(plist['newx'],plist['newy'])
newx,newy = scipy.mgrid[plist['modx'][0]:plist['modx'][-1]:-plist['remapto'],
plist['mody'][0]:plist['mody'][-1]:-plist['remapto']]
x0 = x[0,0]
y0 = y[0,0]
dx = x[1,0] - x0
dy = y[0,1] - y0
ivals = (newx - x0)/dx
jvals = (newy - y0)/dy
coords = scipy.array([ivals, jvals])
for i in np.arange(ivals.shape[0]):
nvals[i] = scipy.ndimage.map_coordinates(ivals[i], coords)
I'm having difficulty getting this code to work properly. The problem areas are:
1.) Recreating this line: newx,newy = mgrid[-1:1:100j,-1:1:100j]. In my case I have a dictionary with the grid in vector form. I've tried to recreate this line using np.meshgrid but then I get an error on line coords = scipy.array([ivals, jvals]). I'm looking for some help in recreating this Cookbook function and making it more dynamic
any help is greatly appreciated.
/M
You should have a look at the documentation for map_coordinates. I don't see where the actual data you are trying to interpolate is in your code. What I mean is, presumably you have some data input which is a function of x and y; i.e. input = f(x,y) that you want to interpolate. In the first example you show, this is the array fvals. This should be your first argument to map_coordinates.
For example, if the data you are trying to inperpolate is input, which should be a 2-dimensional array of shape (len(x),len(y)), then the interpolated data would be:
interpolated_data = map_coordinates(input, coords)