PHP ImageMagick use 10gb~ memory when resizing a big image - resize

I have problem when I try to resize a very big image like a 30,000px x 20,000px with ImageMagick software. But I do not have enought RAM. I have 8GB, but the software requires about 10GB~. I need to use all data in RAM. I tried two options:
1 using convert, writes to a different image file:
convert -resize 90% source.jpg destination.jpg
using mogrify overwrites the original image file
mogrify -resize 90% source.jpg
Maybe someone knows how I can use less memory or some other tricks.
More information about Imagemagick: http://www.imagemagick.org/script/mogrify.php

If your attempting to configure ImageMagick's memory polices to use more RAM, before caching to disk, define the AREA limit by modifying policy.xml or set an environment variable.
Setting Memory limit in Environment
This is probably the quickest way for one-off process.
MAGICK_AREA_LIMIT=8GB convert -resize 90% source.jpg destination.jpg
Also attempt adjusting other environment vars
MAGICK_MAP_LIMIT - Maximum amount of memory map in megabytes to allocate for the pixel cache.
MAGICK_MEMORY_LIMIT - Maximum amount of memory in megabytes to allocate for the pixel cache from the heap.
Setting Memory limit with Policy.xml
Edit the policy.xml under your {$PREFIX}/lib/ImageMagic-X.X.X/config directory.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
<!ELEMENT policymap (policy)+>
<!ELEMENT policy (#PCDATA)>
<!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
<!ATTLIST policy name CDATA #IMPLIED>
<!ATTLIST policy rights CDATA #IMPLIED>
<!ATTLIST policy pattern CDATA #IMPLIED>
<!ATTLIST policy value CDATA #IMPLIED>
]>
<policymap>
<policy domain="resource" name="memory" value="8GB"/>
<policy domain="resource" name="map" value="8GB"/>
<policy domain="resource" name="area" value="8GB"/>
</policymap>
Verification
Whatever method you choose, you can verify by running the identify utility.
$ identify -list resource
File Area Memory Map Disk Thread Throttle Time
--------------------------------------------------------------------------------
1920 8GB 8GB 8GB unlimited 1 0 unlimited

It's not clear what you are trying to achieve overall, but you can resize an image while reading it like this and it should take less memory:
convert source.jpg'[15000x10000]' destination.jpg

Edit the policy.xml which resides under /etc/ImageMagick/policy.xml
In my case it was using more than 2GiB, but here you can limit so that script should be running under the provided limit. I set it for 512MiB for map and memory.

Related

Can man pass an option to the roff formatter?

SYNOPSIS
From man(1):
-l
Format and display local manual files instead of
searching through the system's manual collection.
-t
Use groff -mandoc to format the manual page to stdout.
From groff_tmac(5):
papersize
This macro file is already loaded at start-up by troff so it
isn't necessary to call it explicitly. It provides an interface
to set the paper size on the command line with the option
-dpaper=size. Possible values for size are the same as
the predefined papersize values in the DESC file (only
lowercase; see groff_font(5) for more) except a7–d7.
An appended l (ell) character denotes landscape orientation.
Examples: a4, c3l, letterl.
Most output drivers need additional command-line switches -p
and -l to override the default paper length and orientation
as set in the driver-specific DESC file. For example, use the
following for PS output on A4 paper in landscape orientation:
sh# groff -Tps -dpaper=a4l -P-pa4 -P-l -ms foo.ms > foo.ps
THE PROBLEM
I would like to use these to format local and system man pages to print out, but want to switch the paper size from letter to A4. Unfortunately I couldn't find anything in man(1) about passing options to the underlying roff formatter.
Right now I can use
zcat `man -w man` | groff -tman -dpaper=a4 -P-pa4
to format man(1) on stdout, but that's kind of long and I'd rather have man build the pipeline for me if I can. In addition the above pipeline might need changing for more complicated man pages, and while I could use grog, even it doesn't detect things like accented characters (for groff's -k option), while man does (perhaps using locale settings).
The man command is typically intended only for searching for and displaying manual pages on a TTY device, not for producing typeset and paper printed output.
Depending on the host system, and/or the programs of interest, the a fully typeset printable form of a manual page can sometimes be generated when a program (or the whole system) is compiled. This is more common for system documents and less common for manual pages though.
Note that depending on which manual pages you are trying to print there may be additional steps required. Traditionally the following pipeline would be used to cover all the bases:
grap $MANFILE | pic | tbl | eqn /usr/pub/eqnchar | troff -tman -Tps | lpr -Pps
Your best solution for simplifying your command line would probably be to write a little tiny script which encapsulates what you're doing. Note that man -w might find several different filenames, so you would probably want to print each separately (or maybe only print the first one).

GDAL - XYZ to GeoTIFF slow

I have an XYZ raster file, 1.1GB in EPSG:23700 (EOV), 50 meters resolution.
The aim is to create a GeoTIFF file to be published via GeoServer (EPSG:4326), but I have some performance issues.
If I open the XYZ file from QGIS (2.14.0, Essen), choose Raster » Conversion » Translate and start it with the default options, it completes in several minutes, which is acceptable.
But if I copy the generated gdal_translate command and run it from CLI, than it takes more than an hour or so.
I've tried -co "GDAL_CACHEMAX=500", -co "NUM_THREADS=3", but has no effect. In the process monitor, the QGIS version uses 1 core fully (25% CPU) and the default max memory of 10MByte, but the CLI version only <10% and <10Mbyte mem. The --degub ON option shows "XYZ: New stepX=50.000000000000000" and hangs there.
I've tried to run it from the QGIS Directory \bin folder and the separately downloaded GDAL instance (C:\OSGeo4W64\bin), same results.
Windows Server 2012, 16GB RAM, 2,6 GHz 4 core Xenon CPU.
Any thoughts on this?
Looks like there's some problem with the environment variables. If I use a modified version of the QGIS startup batch file, it all works as expected.
#echo off
call "%~dp0\o4w_env.bat"
#echo off
path %OSGEO4W_ROOT%\apps\qgis\bin;%PATH%
set QGIS_PREFIX_PATH=%OSGEO4W_ROOT:\=/%/apps/qgis
set GDAL_FILENAME_IS_UTF8=YES
set GDAL_CACHEMAX = 500
rem Set VSI cache to be used as buffer, see #6448
set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000
set QT_PLUGIN_PATH=%OSGEO4W_ROOT%\apps\qgis\qtplugins;%OSGEO4W_ROOT%\apps\qt4\plugins
REM This line changed to run my batch file instead of starting QGIS.
call "d:\gdaltest.bat"

Apache Spark to S3 upload performance Issue

I'm seeing a major performance issue when Apache Spark uploads its results to S3. As per my understanding it goes these steps...
Output of final stage is written to _temp/ table in HDFS and the same is moved into "_temporary" folder inside the specific S3 folder.
Once the whole process is done - Apache spark completes the saveAsTextFile stage and then files inside "_temporary" folder in S3 are moved into the main folder. This is actually taking a long time [ approximately 1 min per file (average size : 600 MB BZ2) ]. This part is not getting logged in the usual stderr log.
I'm using Apache Spark 1.0.1 with Hadoop 2.2 on AWS EMR.
Has anyone encountered this issue ?
Update 1
How can I increase the number of threads that does this move process ?
Any suggestion is highly appreciated...
Thanks
This was fixed with SPARK-3595 (https://issues.apache.org/jira/browse/SPARK-3595). Which was incorporated in builds 1.1.0.e and later (see https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark).
I use below functions . it uploads file to s3. it uploads around 60 gb , gz files in 4-6 mins.
ctx.hadoopConfiguration().set("mapred.textoutputformat.separator",
",");
counts.saveAsHadoopFile(s3outputpath, Text.class, Text.class,
TextOutputFormat.class);
Make sure that you create more output files . more number of smaller files will make upload faster.
API details
saveAsHadoopFile[F <: org.apache.hadoop.mapred.OutputFormat[_, ]](path: String, keyClass: Class[], valueClass: Class[], outputFormatClass: Class[F], codec: Class[ <: org.apache.hadoop.io.compress.CompressionCodec]): Unit
Output the RDD to any Hadoop-supported file system, compressing with the supplied codec.

How do you convert ttf font to pff2?

I'm looking into grub2 themes, and i can't seem to convert ttf to grub's native format.
I tried to do this:
➜ asdfafdsgsgsg ttf2bdf -p 32 Oxygen_Regular_Oxygen.ttf > oxygen.bdf
➜ asdfafdsgsgsg grub-mkfont -s 32 -o oxygen.pf2 oxygen.bdf
grub-mkfont: error: can't set 32x32 font size.
Generating Grub2 PF2 fonts requires significant patience, as I had to attempt the conversion multiple times.
If your going from TTF -> PF2, you should first try skipping the BDF intermediate stage. Something like this command,
grub2-mkfont -o ./oxygen.pf2 -s 24 ./Oxygen_Regular_Oxygen.ttf
If it was only the -p 32 which caused that error, I would accept that output at face value. Unless you know of other fonts, which successfully specified such a large point size. Additionally I would hope grub-mkfont wouldn't be allowed to generate a font, whose size exceeds Grub2 limitations. However I should mention, grub-mkfont historically had been lacking in proper maintenance. This might not currently be the case, and would expect upstream has rectified this long ago, but distributions often lag behind.

find out what is the right block size for SWAP?

i found the following method to create swap, from here
dd if=/dev/zero of=/root/myswapfile bs=1M count=1024
Is there any thumb rule that to decide block size ? What is the best block size for Swap in any machine ? Here it is 1M..
When creating a regular file for swap using dd the blocksize is a convenience to allow the count parameter to create a file of the specified size.
The underlying disk structure is unchanged by the bs= in the dd command.