I run a complex cpp application, where gdal (3.5.3) is a part to get elevation data from eudem v 1.1. The eudem files are quite large and I want to reduce their file size, since I don't need an accuracy of 25m. This is one file I use and want to downsize.
gdalinfo E20N20.TIF
Driver: GTiff/GeoTIFF
Files: E20N20.TIF
Size is 40000, 40000
Coordinate System is:
PROJCS["ETRS89_ETRS_LAEA",
GEOGCS["ETRS89",
DATUM["European_Terrestrial_Reference_System_1989",
SPHEROID["GRS 1980",6378137,298.2572221010042,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6258"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4258"]],
PROJECTION["Lambert_Azimuthal_Equal_Area"],
PARAMETER["latitude_of_center",52],
PARAMETER["longitude_of_center",10],
PARAMETER["false_easting",4321000],
PARAMETER["false_northing",3210000],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]]]
Origin = (2000000.000000000000000,3000000.000000000000000)
Pixel Size = (25.000000000000000,-25.000000000000000)
Metadata:
AREA_OR_POINT=Area
DataType=Elevation
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 2000000.000, 3000000.000) ( 20d45'24.21"W, 45d41'42.74"N)
Lower Left ( 2000000.000, 2000000.000) ( 16d36'13.25"W, 37d23'21.20"N)
Upper Right ( 3000000.000, 3000000.000) ( 8d 7'39.52"W, 48d38'23.47"N)
Lower Right ( 3000000.000, 2000000.000) ( 5d28'46.07"W, 39d52'33.70"N)
Center ( 2500000.000, 2500000.000) ( 12d41'50.60"W, 43d 6' 4.82"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
NoData Value=-3.4028234663852886e+38
Metadata:
BandName=Band_1
RepresentationType=ATHEMATIC
```
I tried the following cmd, but w/out success. Its probably about wrong ts or tr values;
```
gdalwarp -r average -tr 1024 1024 -wm 4096 -multi -wo NUM_THREADS=ALL_CPUS -co TILED=YES \
-co NUM_THREADS=ALL_CPUS E20N20.TIF dz_E20N20.TIF
Creating output file that is 977P x 977L.
Processing E20N20.TIF [1/1] : 0Using internal nodata values (e.g. -3.40282e+38) for image E20N20.TIF.
Copying nodata values from source E20N20.TIF to destination dz_E20N20.TIF.
ERROR 1: Integer overflow : nSrcXSize=19989, nSrcYSize=19989
ERROR 1: Integer overflow : nSrcXSize=20012, nSrcYSize=19989
```
I'm lost, here.
Where is your GDAL coming from, this is a 32-bit error because your data exceeds 4GB?
Also, unless you are reprojecting, I strongly advise you to use gdal_translate instead which has much better performance when resampling:
gdal_translate -r average -tr 1024 1024 -wo NUM_THREADS=ALL_CPUS -co TILED=YES \
-co NUM_THREADS=ALL_CPUS E20N20.TIF dz_E20N20.TIF
You can probably use the official Docker images if you need another GDAL build.
Thank you for your time. I found the problem in -wm 4096. I use the Linux Subsystem for windows to make the resampling. I used 1024 a power of 2, because I didn't really understand, how -ts or -ts worked. I finally use:
gdalwarp -r average -ts 8000 8000 -multi -wo NUM_THREADS=ALL_CPUS -co TILED=YES \
-co NUM_THREADS=ALL_CPUS E20N20.TIF dz_E20N20.TIF
This reduces my file size about 75% and the pixel dim moves from 25x25m to 125x125m.
Related
I created a backup for a file then compressed it and store it using tar.
At the time I didn't know it was a sparse file, So I didn't use the -S flag.
Now I am trying to retrieve the data, but I can't since when I extract I get a non sparse file.
Is their a way to retrieve that info, or is it lost for good ?
Thanks in advance.
Sparsity information is kind of redundant. You can determine whether some parts of a file should be sparse by checking whether those parts only contain zeros.
head -c $(( 1024 * 1024 )) /dev/urandom > foo
head -c $(( 1024 * 1024 )) /dev/zero >> foo
head -c $(( 1024 * 1024 )) /dev/urandom >> foo
stat foo
Size: 3145728 Blocks: 6144
fallocate --dig-holes foo
stat foo
Size: 3145728 Blocks: 4096
As you can see from the block count, making it sparse was successful, and all those block that were completely zeroed out have been successfully removed.
I am cropping a tif image file using gdal_translate but the resulting file is of bigger size compared to the original file.
Here is the sample command that I am using to crop the image
gdal_translate -srcwin 4000 4500 2000 3000 Ortho.tif Ortho_cropped.tif
You can compress the output too with the following
gdal_translate -srcwin 4000 4500 2000 3000 -co COMPRESS=DEFLATE -co PREDICTOR=2 Ortho.tif Ortho_cropped.tif
This uses deflate compression which is often quite effective. A predictor of 2 is often nice for integer values, while you can change the predictor to 3 if your values are floats.
You can use other compressions too like ZSTD if you are using gdal >= 2.3 which should be faster and archive similar compression rates. If you do not care about lossy compression, you can even use JPEG.
If you do not know the compression of your original image you can see what it is using the command gdalinfo Ortho.tif where the compression is described under Image Structure Metadata
I am trying to merge a number of geotiffs into one large geotiff with overviews, however the final merged geotiff shows a number of horizontal artifacts around the edges of the original merged geotiffs (see here for an example).
I create the merged file using the following code:
'''Produce Combined VRT'''
string ='gdalbuildvrt -srcnodata "0 0 0 0" -hidenodata -r bilinear %s -overwrite %s' %(tmp_vrt, GDal_merge_string)
os.system(string)
'''Convert VRT to Geotiff'''
string ='gdal_translate -b 1 -b 2 -b 3 -mask 4 --config GDAL_TIFF_INTERNAL_MASK YES -of GTiff %s %s' %(tmp_vrt,tmp_fname)
os.system(string)
I have a hunch that this might have to do with using gdal_translate on a vrt, as the erorrs occur on the edges of the original geotiffs, and in this case it might be related or similar to the issue found in this post.
This code is using VRTs to combine the geotiffs for speed purposes, but perhaps it might be better to just merge these with gdalwarp?
Edit: I have reduced the number of flags and left out the overviews in the code above, as suggested in the comment below by Benjamin. The error seems to be produced in the code above. I think the issue may lie in the masking process. I guess at some point in the process of stacking the bands, the inputs are distorted. Is it generally inadvisable to gdal_translate VRTs?
although there are several pages dedicated to this task, it seems that i can not make that work. I have a non georeferenced tif file which I provide you the output from gdalinfo
Driver: GTiff/GeoTIFF
Files: Arctic_r05c02.2017235.terra.250m.tif
Size is 4096, 4096
Coordinate System is `'
Metadata:
TIFFTAG_SOFTWARE=ppm2geotiff v0.0.9
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 4096.0)
Upper Right ( 4096.0, 0.0)
Lower Right ( 4096.0, 4096.0)
Center ( 2048.0, 2048.0)
Band 1 Block=4096x1 Type=Byte, ColorInterp=Red
Band 2 Block=4096x1 Type=Byte, ColorInterp=Green
Band 3 Block=4096x1 Type=Byte, ColorInterp=Blue
I do have the corner coordinate extention in lat/lon. Hence I am creating Ground Control Points with thee following command:
gdal_translate -of GTiff -gcp 0 0 -180.0000 +63.1066 -gcp 0 4096 -161.5651 +68.5979 -gcp 4096 0 +161.5651 +68.5979 -gcp 4096 4096 -180.0000 +76.3728 input.tif output.tif
which results to:
Driver: GTiff/GeoTIFF
Files: Arctic_r05c02.2017235.terra.250mproj.tif
Size is 4096, 4096
Coordinate System is `'
GCP Projection =
GCP[ 0]: Id=1, Info=
(0,0) -> (-180,63.1066,0)
GCP[ 1]: Id=2, Info=
(0,4096) -> (-161.5651,68.5979,0)
GCP[ 2]: Id=3, Info=
(4096,0) -> (161.5651,68.5979,0)
GCP[ 3]: Id=4, Info=
(4096,4096) -> (-180,76.3728,0)
Metadata:
TIFFTAG_SOFTWARE=ppm2geotiff v0.0.9
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 4096.0)
Upper Right ( 4096.0, 0.0)
Lower Right ( 4096.0, 4096.0)
Center ( 2048.0, 2048.0)
Band 1 Block=4096x1 Type=Byte, ColorInterp=Red
Band 2 Block=4096x1 Type=Byte, ColorInterp=Green
Band 3 Block=4096x1 Type=Byte, ColorInterp=Blue
The next step is to resample with gdalwarp in order to create a projected file.
gdalwarp -r near -s_srs epsg:4326 -t_srs epsg:3413 -tr 250 250 input.tif output.tif
But the file seems distort:
Any idea what I can further do? Is there a way to directly change the corner coordinates inside the tiff file instead of adding GCP's? Would that help?
Some more information which i forgot. The data can be downloaded at:
https://lance-modis.eosdis.nasa.gov/imagery/subsets/?subset=Arctic_r05c02.2017235.terra.250m
and for the bounding box definition:
https://lance-modis.eosdis.nasa.gov/imagery/subsets/?subset=Arctic_r05c02.2017235.terra.250m.met
Since the box definition from NASA's side looks a bit odd, i tried the same procedure for the following tile:
https://lance-modis.eosdis.nasa.gov/imagery/subsets/?subset=Arctic_r04c03.2017235.terra.250m
and box definition:
https://lance-modis.eosdis.nasa.gov/imagery/subsets/?subset=Arctic_r04c03.2017235.terra.250m.met
Although the box looks ok the following image was produced:
The data you downloaded is already projected (in epsg:3413), you are assigning a projection (epsg:4326) and bounding-box which is not correct.
You can view the correct settings by visiting:
https://lance-modis.eosdis.nasa.gov/imagery/subsets/?subset=Arctic_r05c02.2017235.terra.250m.gdal
About halfway down on the page you'll see the gdal_translate command, use that with your in- and outputs.
For example:
gdal_translate -of GTiff -outsize 4096 4096 -projwin -2097152 2097152 -1048576 1048576 -a_srs "+proj=stere +lat_0=90 +lat_ts=70 +lon_0=-45 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs" Arctic_r05c02.2017235.terra.250m.tif Arctic_r05c02.2017235.terra.250m_output.tif
I have a big GeoTIFF that I want to stream through a WMS within GeoServer (v.2.11). The size of the image is about 7GB, consisting on a very large high resolution RGB image. I have allowed enough heap space within JVM in order to display the imagery. However, I would like to compress the image so it can be more responsive when exploring through and so it will allocate less memory. I have followed some of the recommendations here.
My strategy was to compress the GeoTIFF with JPEG compression and use that as a data store in GeoServer. However, this seems not to work. This is the gdal command I have used to translate the image:
gdal_translate -of GTiff -co "BIGTIFF=YES" -co "COMPRESS=JPEG" -co "TILED=YES" -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -a_srs "EPSG:3057" D:\raster\image.tif
D:\raster\image_translate.tif
When previewing the image with openlayers, I got nothing, just a blank basemap. The log from GeoServer told me that something in the projection went bad:
2017-06-09 13:16:47,551 INFO [geoserver.wms] -
Request: getServiceInfo
2017-06-09 13:16:47,561 WARN [lite.gridcoverage2d] - Could not reduce the grid geometry inside the valid area bounds: ReferencedEnvelope[-1.7976931348623157E308 : 1.7976931348623157E308, -85.0 : 85.0]
Grid geometry isGridGeometry2D[GeneralGridEnvelope[0..357, 0..357], PARAM_MT["Affine",
PARAMETER["num_row", 3],
PARAMETER["num_col", 3],
PARAMETER["elt_0_0", 0.7353351955307262],
PARAMETER["elt_0_2", 584219.1848475977],
PARAMETER["elt_1_1", -0.7353351955307262],
PARAMETER["elt_1_2", 383937.61122240225]]]
2017-06-09 13:16:47,566 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Error rendering coverage on the fast path
I then tried to use another compression strategy with GDAL, i.e. "DEFLATE":
gdal_translate -of GTiff -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 -co "BIGTIFF=YES" -a_srs "EPSG:3057" D:\raster\image.tif D:\raster\image_translate2.tif
And that worked when previewing in openlayers. Here is the GeoServer log:
2017-06-09 13:28:27,137 INFO [geoserver.wms] -
Request: getServiceInfo
2017-06-09 13:28:27,146 WARN [lite.gridcoverage2d] - Could not reduce the grid geometry inside the valid area bounds: ReferencedEnvelope[-1.7976931348623157E308 : 1.7976931348623157E308, -85.0 : 85.0]
Grid geometry isGridGeometry2D[GeneralGridEnvelope[0..357, 0..357], PARAM_MT["Affine",
PARAMETER["num_row", 3],
PARAMETER["num_col", 3],
PARAMETER["elt_0_0", 0.7353351955307262],
PARAMETER["elt_0_2", 584219.1848475977],
PARAMETER["elt_1_1", -0.7353351955307262],
PARAMETER["elt_1_2", 383937.61122240225]]]
2017-06-09 13:28:27,231 INFO [geoserver.wms] -
Request: getMap
I have also tried to perform gdal_translate using JPEG compression and no tiling, and I got also errors with the GeoServer log and the openlayers preview displayed nothing.
So my question is, what is the best strategy to compress GeoTIFF files to be used in a WMS? At the moment, seems that DEFLATE is the only one working, but the compression is not the best. Has anyone been able to successfully upload a JPEG compressed GeoTIFF to Geoserver?
If it's any help the way I do it is as follows.
First I chop the raster up into smaller tiles, size is not important, for me it's generally either 256x256, 512x512 or 1024x124.
I use a number of different programs from gdal2tiles.py to my own homegrown c# apps.
What's important is that the tiles are square.
Once I have the tiles in a folder, I then use gdaltindex
This creates a shapefile with one square for each tile, correctly georeferenced (assuming your raster was) along with the name of each tile, I usually tell gdaltindex to write absolute paths to the shapefile.
I then reference the shape as a tile layer in mapserver, I can't say if geoserver will accept a shape based tile index, but since gdal can make them and the other WMS based server available open source (mapserver) can use them, then I would be very suprised if geoserver was not able to use them.
This would work in the case of tiff greater than 2GB. GeoServer can efficiently deal with large TIFF with overviews, as long as the TIFF is below the 2GB size limit.
Using image pyramid makes the tiff load faster as it makes multiple mosaic of images at different zoom levels.
Use the following command:-
mkdir bmpyramid
gdal_retile.py -v -r bilinear -levels 4 -ps 2048 2048 -co "TILED=YES" -co "COMPRESS=JPEG" -targetDir bmpyramid bmreduced.tiff
You can check here
https://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html