I'm reading an avi file with approx 2MB size, 301 frames, 20 frames/sec (15 sec long video) and a size of 1024 * 1096 per frame.
When I'm reading the single frames with cv2 and resaving them in original size as png, then I'm getting a size of approx 600KB per picture/frame. So, I have in total 301 * 600KB = 181MB (original avi had 2MB).
Any idea why this is happening and how to reduce the file size of the single frames without changing the resolution? Idea is to somehow generate single frames from the original video, do detections with CNN and to resave the original video again with included detections and the output video shall be somehow very similar to input video (approx same file size, must not be avi format)
PNG files or single frames are in the most cases always larger than the original video file (compressed in the most cases by a codec https://www.fourcc.org/codecs.php). Use for example the following command on Linux to create a compressed avi:
ffmpeg -i FramePicName%d.png -vcodec libx264 -f avi aviFileName
You can get the used codec to create the original video file by the following python cv2 code
cap = cv2.VideoCapture(videoFile)
fourcc = cap.get(cv2.CAP_PROP_FOURCC) # or cv2.cv.CV_CAP_PROP_FOURCC
"".join([chr((int(fourcc) >> 8 * i) & 0xFF) for i in range(4)])
Related
I have a big GeoTIFF that I want to stream through a WMS within GeoServer (v.2.11). The size of the image is about 7GB, consisting on a very large high resolution RGB image. I have allowed enough heap space within JVM in order to display the imagery. However, I would like to compress the image so it can be more responsive when exploring through and so it will allocate less memory. I have followed some of the recommendations here.
My strategy was to compress the GeoTIFF with JPEG compression and use that as a data store in GeoServer. However, this seems not to work. This is the gdal command I have used to translate the image:
gdal_translate -of GTiff -co "BIGTIFF=YES" -co "COMPRESS=JPEG" -co "TILED=YES" -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -a_srs "EPSG:3057" D:\raster\image.tif
D:\raster\image_translate.tif
When previewing the image with openlayers, I got nothing, just a blank basemap. The log from GeoServer told me that something in the projection went bad:
2017-06-09 13:16:47,551 INFO [geoserver.wms] -
Request: getServiceInfo
2017-06-09 13:16:47,561 WARN [lite.gridcoverage2d] - Could not reduce the grid geometry inside the valid area bounds: ReferencedEnvelope[-1.7976931348623157E308 : 1.7976931348623157E308, -85.0 : 85.0]
Grid geometry isGridGeometry2D[GeneralGridEnvelope[0..357, 0..357], PARAM_MT["Affine",
PARAMETER["num_row", 3],
PARAMETER["num_col", 3],
PARAMETER["elt_0_0", 0.7353351955307262],
PARAMETER["elt_0_2", 584219.1848475977],
PARAMETER["elt_1_1", -0.7353351955307262],
PARAMETER["elt_1_2", 383937.61122240225]]]
2017-06-09 13:16:47,566 ERROR [geoserver.ows] -
org.geoserver.platform.ServiceException: Error rendering coverage on the fast path
I then tried to use another compression strategy with GDAL, i.e. "DEFLATE":
gdal_translate -of GTiff -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 -co "BIGTIFF=YES" -a_srs "EPSG:3057" D:\raster\image.tif D:\raster\image_translate2.tif
And that worked when previewing in openlayers. Here is the GeoServer log:
2017-06-09 13:28:27,137 INFO [geoserver.wms] -
Request: getServiceInfo
2017-06-09 13:28:27,146 WARN [lite.gridcoverage2d] - Could not reduce the grid geometry inside the valid area bounds: ReferencedEnvelope[-1.7976931348623157E308 : 1.7976931348623157E308, -85.0 : 85.0]
Grid geometry isGridGeometry2D[GeneralGridEnvelope[0..357, 0..357], PARAM_MT["Affine",
PARAMETER["num_row", 3],
PARAMETER["num_col", 3],
PARAMETER["elt_0_0", 0.7353351955307262],
PARAMETER["elt_0_2", 584219.1848475977],
PARAMETER["elt_1_1", -0.7353351955307262],
PARAMETER["elt_1_2", 383937.61122240225]]]
2017-06-09 13:28:27,231 INFO [geoserver.wms] -
Request: getMap
I have also tried to perform gdal_translate using JPEG compression and no tiling, and I got also errors with the GeoServer log and the openlayers preview displayed nothing.
So my question is, what is the best strategy to compress GeoTIFF files to be used in a WMS? At the moment, seems that DEFLATE is the only one working, but the compression is not the best. Has anyone been able to successfully upload a JPEG compressed GeoTIFF to Geoserver?
If it's any help the way I do it is as follows.
First I chop the raster up into smaller tiles, size is not important, for me it's generally either 256x256, 512x512 or 1024x124.
I use a number of different programs from gdal2tiles.py to my own homegrown c# apps.
What's important is that the tiles are square.
Once I have the tiles in a folder, I then use gdaltindex
This creates a shapefile with one square for each tile, correctly georeferenced (assuming your raster was) along with the name of each tile, I usually tell gdaltindex to write absolute paths to the shapefile.
I then reference the shape as a tile layer in mapserver, I can't say if geoserver will accept a shape based tile index, but since gdal can make them and the other WMS based server available open source (mapserver) can use them, then I would be very suprised if geoserver was not able to use them.
This would work in the case of tiff greater than 2GB. GeoServer can efficiently deal with large TIFF with overviews, as long as the TIFF is below the 2GB size limit.
Using image pyramid makes the tiff load faster as it makes multiple mosaic of images at different zoom levels.
Use the following command:-
mkdir bmpyramid
gdal_retile.py -v -r bilinear -levels 4 -ps 2048 2048 -co "TILED=YES" -co "COMPRESS=JPEG" -targetDir bmpyramid bmreduced.tiff
You can check here
https://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html
Actually, I am dealing with many pictures which are from different videos, so I use tf.SequenceExample() to save them as different sequences and their labels attached into TFRcord.
But after running my code to generate TFRecord, it generates a TFRecord which is 29GB larger than my original pictures 3GB.
Is that normal to create TFRecord larger than the original data?
You may be storing the decoded images instead of the jpeg encoded ones. TFRecord has no concept of image formats so you can use any encoding you want. To keep the size the same, convert the original image file contents to a BytesList and store that without calling decode_image or using any image libraries or anything that understands image formats.
Another possibility is you might be storing the image as an Int64List full of bytes which would be 8x the size. Instead, store it as a BytesList containing a single Bytes.
Check the the type of data you load. I guess you load images as pixel-data. Every pixel is unit8 (8 bit) and likely to be converted to float (32 bit). Hence you have to expect that it gets 4 times the original size (3 GB -> 12 GB).
Also, the original format might have (better) compression than tfrecords. (I'm not sure if tfrecords can use compression)
I am using GIMP to convert grayscale PNM-files (scanned documents) to PDF.
My goal is a small filesize. (Ideally: viewable on different devices without any problems and maybe suitable long-term preservation - PDF/A?)
So far, so good. Trying to reproduce that process with ImageMagick in a batch script doesn't give me that same small filesize as in GIMP.
GIMP (Ver. 2.8.14) workflow:
Open File
Change resolution (density) to 300x300 Pixel/in
Set threshold to 127 (=50%)
Export as OutGIMP.pdf
ImageMagick (Ver. 6.7.9-0 2012-09-16 Q16) workflow:
convert Scan.pnm -density 300x300 -threshold 50% -monochrome OutA.pdf
convert Scan.pnm -density 300x300 -threshold 50% -monochrome OutB.png
convert OutB.png OutC.pdf
Using an example File this results in:
OutGIMP.pdf: 141.195 Byte
OutA.pdf: 684.245 Byte
OutB.png: 137.246 Byte
OutC.pdf: 217.860 Byte
How can I get a PDF with ImageMagick that is at least as small as the GIMP-PDF?
Edit
Continuing the GIMP (Ver. 2.8.14) workflow from above with:
Scale to 100x100 Pixel/in while keeping the Imagesize
Export as OutGIMP_100ppi.pdf
strangely results in:
OutGIMP_100ppi.pdf: 179.123 Byte
I'm using imagemagick to create an animated GIF out of ~60 JPG 640x427px photos. The combined size of the JPGs is about 4MB.
However, the output GIF is ~12MB. Is there a reason why the GIF is considerably bigger? Can I conceivably achieve a GIF size of ~4MB?
The command I'm using is:
convert -channel RGB # no improvement in size
-delay 2x10 \
-size 640 \
-loop 0 \
-dispose Background # no improvement in size
-layers Optimize # about 2MB improvement
portrait/*.jpg portrait.gif
Using gifsicle didn't seem to improve either.
JPG is lossy compression.
GIF is lossless compression.
A better comparison would be to convert all the source images to GIF first, then combine them..
First google hit for GIF compression is http://ezgif.com/optimize which claims lossy GIF compresion, might work for you but I offer no warranty as I haven't tried it.
JPEG achieves it's compression through a (lossy) transform, where an 16x16 / 8x8 block of pixels is transformed to frequency representation and then quantized. Instead of selecting e.g. 256 levels (i.e. 8 bits) of red/green/blue per component, JPEG can ignore some frequency components, or use just 1 or 2 bits to represent them.
GIF on the other hand works by identifying repeated patterns from a paletted image (upto 256 entries), which occur exactly in the previously encoded/decoded stream. Both because of the JPEG compression, and the source of the images typically encoded by JPEG (natural full color), the probability of (long) exact matches is quite low.
60 RGB images with the size 640x427 is about 16 million pixels. To represent that much in 4 MB, requires a compression of 2 bits per pixel. To achieve this with GIF would require a very lossy algorithm, that would select (vector) quantization of true color pixels not to the closest pixel in the target GIF palette, but based also on the fact how good dictionary of code words this particular selection will make. The dictionary builds slowly and to achieve 2 bits/pixel, the average length of the decoded code word would have to map to 5.5 matching pixels in the close neighborhood.
By contrast, imagemagick has been able to compress the 16 million pixels (each selected from a palette of 256 elements) to 75% already!
I have a camera which produces photographs of 3008x2000 pixels. I use Image Magick to scale and resize the photos to be put up on my website. The size of the images I am using on the website is 602x400. I use this command to reduce the size:
convert DSC_0124.JPG -scale 20% -size 24% img1.jpg
This produces an image which is 602x400 pixels in size. But the file size will be always above 250KB. More images on a single html page means the page will be heavier and loading time will be longer. Are there any features in image magic that will help me to keep the file size as small as possible, possibly, below 100KB. But the image size should be the same, that is, 602x400px. I have achieved similar optimisation with SEAMonster tool for MS Windows. As it doean't have a commandline alternative, it wouldn't be of much help when there are hundreds of images to be converted.
Use command as Delan proposed with additional "-strip" flag to remove EXIF data, this have reduced the size of some of my images drastically. Here is a bash script for unix platforms, but you can use the second part only for individual images.
for X in *.jpg; do convert "$X" -resize 602x400 -strip -quality 86 "$X"; done
This will convert all images in the directory.
Use -quality to set the compression level:
convert DSC_0124.JPG -scale 20% -size 24% -quality [0..100] img1.jpg
You can define the maximum size of the output image at 100KB like this:
convert DSC_0124.JPG -resize 602x400! -strip -define jpeg:extent=100KB img1.jpg
If you are running your website on PHP, you might want to consider the SLIR image resizing script, it does a great job resizing to various constraints (see below) and caches the results.
Parameters:
w Maximum width
h Maximum height
c Crop ratio
q Quality
b Background fill color
p Progressive
http://shiftingpixel.com/2008/03/03/smart-image-resizer/
http://code.google.com/p/smart-lencioni-image-resizer/