I am currently working on VMWare virtualization; I am trying to make a converted image from qcow2 to vmdk work with ESXi Server 6.0.
I have myImage.qcow2 with a disk which is thin provisioned for 300GB.
I used the image converter tool qemu with the following command
qemu-img convert -f qcow2 myImage.qcow2 -O vmdk myNewImage.vmdk
This command gives me a vmdk image which is only VMWare Workstation compatible. Therefore, in order to make it ESXi compatible I have to use the vmkfstools with the following command.
vmkfstools -i myImage.vmdk outputName.vmdk -d thin
The vmkfstools command gives me two files, an metadata.vmdk and the actual data.vmdk.
As mentioned above my disk is thin provisioned for 300GB and when I apply vmkfstools it expands the disk and gives me a size of 300GB.
Deploying the image through the vSphere Client works without any problem; however, for the purpose of this project I want to use the ovftool and doing so with such a large image is not feasible.
Is there a way for me to make my .vmdk ESXi compatible without vmkfstools expanding my image to 300GB?
Or Is there any other method for me to deploy those 300GB using the ovftool while the disk image is on the datastore, so that it doesn't have to be downloaded/uploaded through the deployment process?
I have been stuck on this for weeks and any help will be highly appreciated.
FYI: This support has been added in Qemu 2.1 and above as per changelogs
qemu-img convert -f qcow2 -O vmdk -o adapter_type=lsilogic,subformat=streamOptimized,compat6 SC-1.qcow2 SC-1.vmdk
This worked for me with VMware 6.7
The TL;DR is
qemu-img convert -f qcow2 -O vmdk -o subformat=streamOptimized source_qcow_image_path destination_path_to_vmdk
For example:
qemu-img convert -f qcow2 -O vmdk -o subformat=streamOptimized \
CentOS-7-x86_64-GenericCloud-1503.qcow2 \
CentOS-7-x86_64-GenericCloud-1503.vmdk
Update the vmdk version setting embedded in the converted image using this script:
* This is what actually worked for me *
printf '\x03' | dd conv=notrunc of=<vmdk file name> bs=1 seek=$((0x4))
For example:
printf '\x03' | dd conv=notrunc of=CentOS-7-x86_64-GenericCloud-1503.vmdk bs=1 seek=$((0x4))
source: https://kb.vmware.com/s/article/2144687
Related
I'm absolutely new to stackoverflow, so I'm extremely sorry if my question has formatting issues.
I created a VM using virt-install:
virt-install \
--name guest \
--memory 2048 \
--vcpus 2 \
--disk size=8 \
--location /var/lib/libvirt/images/centos.iso \
--os-variant centos7.0 \
--check disk_size=off \
--graphics none \
--extra-args console=ttyS0
The above command works fine, but when I want to use a specific image which is a qcow2 image, Virt-install throws the error:
ERROR Command '['isoinfo', '-J', '-i', '/var/lib/libvirt/images/centos.qcow2', '-f']' returned non-zero exit status 1
Looks like virt-install is looking for an isofile since its running the isoinfo command to verify an iso image, eventhough I have a qcow2 image.
I might be able to pass a qcow2 using the --disk parameter instead of --location, but --extra-args only works with --location.
Kindly help me out, thank you in advance!
You seem to be mixing up two different concepts here. The --location flag is intended to provide the installation media, which will be an ISO image or a URL to a installable tree. The --disk flag would be to provide the storage to back the virtual disk of the VM being provisioned, which will typically be a qcow2 file or a raw file, or a block device.
I'm assuming the centos.iso image would be your installation media, and the centos.qcow2 image would be an empty volume for the virtual disk into which you'll install the CentOS guest.
VM Disk Information
I wanted to change Disk Size upto 20G before vm Installation. The image already has Virtual size of 50G.
First, You should consider trying to compress the disk size.
sudo qemu-img -O qcow2 vmf5.img vmf5-compact.qcow2
If you really need those disk spaces, what you need to do:
sudo qemu-img resize /var/lib/libvirt/images/vmf5.img 20G
There is a document about qemu-img
Good day,
So I am following this coreboot v3 + OpenBIOS tutorial Here .
In the instructions I have the following...
mkdir foo
cd foo
wget http://www.coreboot.org/images/9/9d/Qemu_coreboot_openbios.zip
wget http://www.coreboot.org/images/0/0d/Vgabios-cirrus.zip
unzip Qemu_coreboot_openbios.zip
unzip Vgabios-cirrus.zip
mv qemu_coreboot_openbios.bin bios.bin
cd ..
qemu -L foo -hda /dev/zero -serial stdio
I noticed that qemu has been replace or is implemented with qemu-system.
command I am running
qemu-x86_64 -L foo -hda /dev/zero -serial stdio
When I run the command, I see just qemu run it's typical and not find a disk.(which I expect since the disk switch points to /dev/zero) but none of the payloads run as I would expect from the tutorial.
What am I doing incorrectly?
Should I use a different version of qemu?
Should I create a dummy disk for this?
Qemu seems to be ignoring the files in the foo directory.
The examples are not up to date, as you have noticed by the renaming of qemu to qemu-system-x86_64.
I managed to get the examples to work using only the cirrus video card, and by renaming the outputs of the zips (bin - bios files to bios-256k.bin). I did this because by adding the -L option I specify the bios location and qemu will look for a file called bios-256k.bin as the bios. The command to run the bios with cirrus (all done while in the foo directory) was
qemu-system-x86_64 -L . -vga cirrus -serial stdio
Both machine types pc and q35 worked.
I normally compress using tar zcvf and decompress using tar zxvf (using gzip due to habit).
I've recently gotten a quad core CPU with hyperthreading, so I have 8 logical cores, and I notice that many of the cores are unused during compression/decompression.
Is there any way I can utilize the unused cores to make it faster?
You can also use the tar flag "--use-compress-program=" to tell tar what compression program to use.
For example use:
tar -c --use-compress-program=pigz -f tar.file dir_to_zip
You can use pigz instead of gzip, which does gzip compression on multiple cores. Instead of using the -z option, you would pipe it through pigz:
tar cf - paths-to-archive | pigz > archive.tar.gz
By default, pigz uses the number of available cores, or eight if it could not query that. You can ask for more with -p n, e.g. -p 32. pigz has the same options as gzip, so you can request better compression with -9. E.g.
tar cf - paths-to-archive | pigz -9 -p 32 > archive.tar.gz
Common approach
There is option for tar program:
-I, --use-compress-program PROG
filter through PROG (must accept -d)
You can use multithread version of archiver or compressor utility.
Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:
$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive
Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):
$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz
Input and output of singlethread and multithread are compatible. You can compress using multithread version and decompress using singlethread version and vice versa.
p7zip
For p7zip for compression you need a small shell script like the following:
#!/bin/sh
case $1 in
-d) 7za -txz -si -so e;;
*) 7za -txz -si -so a .;;
esac 2>/dev/null
Save it as 7zhelper.sh. Here the example of usage:
$ tar -I 7zhelper.sh -cf OUTPUT_FILE.tar.7z paths_to_archive
$ tar -I 7zhelper.sh -xf OUTPUT_FILE.tar.7z
xz
Regarding multithreaded XZ support. If you are running version 5.2.0 or above of XZ Utils, you can utilize multiple cores for compression by setting -T or --threads to an appropriate value via the environmental variable XZ_DEFAULTS (e.g. XZ_DEFAULTS="-T 0").
This is a fragment of man for 5.1.0alpha version:
Multithreaded compression and decompression are not implemented yet, so this
option has no effect for now.
However this will not work for decompression of files that haven't also
been compressed with threading enabled. From man for version 5.2.2:
Threaded decompression hasn't been implemented yet. It will only work
on files that contain multiple blocks with size information in
block headers. All files compressed in multi-threaded mode meet this
condition, but files compressed in single-threaded mode don't even if
--block-size=size is used.
Recompiling with replacement
If you build tar from sources, then you can recompile with parameters
--with-gzip=pigz
--with-bzip2=lbzip2
--with-lzip=plzip
After recompiling tar with these options you can check the output of tar's help:
$ tar --help | grep "lbzip2\|plzip\|pigz"
-j, --bzip2 filter the archive through lbzip2
--lzip filter the archive through plzip
-z, --gzip, --gunzip, --ungzip filter the archive through pigz
You can use the shortcut -I for tar's --use-compress-program switch, and invoke pbzip2 for bzip2 compression on multiple cores:
tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 DIRECTORY_TO_COMPRESS/
If you want to have more flexibility with filenames and compression options, you can use:
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec \
tar -P --transform='s#/my/path/##g' -cf - {} + | \
pigz -9 -p 4 > myarchive.tar.gz
Step 1: find
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec
This command will look for the files you want to archive, in this case /my/path/*.sql and /my/path/*.log. Add as many -o -name "pattern" as you want.
-exec will execute the next command using the results of find: tar
Step 2: tar
tar -P --transform='s#/my/path/##g' -cf - {} +
--transform is a simple string replacement parameter. It will strip the path of the files from the archive so the tarball's root becomes the current directory when extracting. Note that you can't use -C option to change directory as you'll lose benefits of find: all files of the directory would be included.
-P tells tar to use absolute paths, so it doesn't trigger the warning "Removing leading `/' from member names". Leading '/' with be removed by --transform anyway.
-cf - tells tar to use the tarball name we'll specify later
{} + uses everyfiles that find found previously
Step 3: pigz
pigz -9 -p 4
Use as many parameters as you want.
In this case -9 is the compression level and -p 4 is the number of cores dedicated to compression.
If you run this on a heavy loaded webserver, you probably don't want to use all available cores.
Step 4: archive name
> myarchive.tar.gz
Finally.
A relatively newer (de)compression tool you might want to consider is zstandard. It does an excellent job of utilizing spare cores, and it has made some great trade-offs when it comes to compression ratio vs. (de)compression time. It is also highly tweak-able depending on your compression ratio needs.
Here is an example for tar with modern zstd compressor, as finding out good examples on this one was difficult:
apt poem to install zstd and pv utilities for Ubuntu
Compress multiple files and folders (zstd command alone can only do single files)
Display progress using pv - shows the total bytes compressed and compression speed GB/sec real-time
Use all physical cores with -T0
Set compression level higher than the default with -8
Display the resulting wall clock and CPU time used after the operation is finished using time
apt install zstd pv
DATA_DIR=/path/to/my/folder/to/compress
TARGET=/path/to/my/arcive.tar.zst
time (cd $DATA_DIR && tar -cf - * | pv | zstd -T0 -8 -o $TARGET)
I have some virtual machines with me. I want to write a script which automates the following process...
It mounts the virtual machine (with linux as the os) to a location say /mnt/image
It modifies the /etc/passwd (or the equivalent file) to change the password of the user
Unmount the virtual machine
Since, I am using libvirt I am having some qcow2 images of the virtual machine. to mount the image on my ubuntu, I am using nbd module. Here are the commands that I am trying :
modprobe nbd max_part=63
qemu-nbd -c /dev/nbd0 image.qcow2
mount /dev/nbd0p1 /mnt/image
It gives me the error:
mount: special device /dev/nbd0p1 does not exist
When I replace nbd0p1 with nbdo I am getting the following error (though I am not sure what I am trying to do by this)
mount: you must specify the filesystem type
Any suggestions so as what could be the problem... ?
Check that /sys/modules/nbd/parameters/max_part has the expected value. If it's 0 or too low, the partitions /dev/nbd0p1, etc. will not be made available by the kernel. This can happen if the nbd kernel module was already loaded (with a different max_part parameter) when you ran modprobe.
You can fix that by unloading the module and modprobing it again.
[Not a direct answer to the question, but an alternate]
You can try to convert qcow2 image to raw and then, mount the raw image.
convert:
qemu-img convert -f qcow2 image.qcow2 -O raw image_raw.raw
mount:
sudo losetup /dev/loop0 image_raw.raw
sudo kpartx -a /dev/loop0
sudo mount /dev/mapper/loop0p3 /mnt/image
sudo mount /dev/mapper/loop0p2 /mnt/image/boot
Could it be that the partition isn't in the first slot in the MBR, or an extended partition is in use? Check to see if any other nbdXpY device nodes are being created, or run fdisk on it and print the partition table.
I stumbled on the same issue and same error but on a vdi
qemu-nbd -c /dev/nbd0 image.vdi
for me the solution was simple I just changed nbd0 to nbd1
qemu-nbd -c /dev/nbd1 image.vdi
and then:
sudo mount /dev/nbd1p1 /media/eddie/virtual
worked.
Please leave a comment if this worked for you also and on what type of image.