I am using a 64 MB QSPI formatted in some UBI partitions.
df is an applet of busybox 1.27.2
Actually,
~ # df -h
Filesystem Size Used Available Use% Mounted on
/dev/ubi0_0 3.1T 1.9T 1.2T 63% /
/dev/ubi1_0 1.6T 21.8G 1.5T 1% /conf
But, obviously, the size cannot be that! Anyway, the use % seems to be correct, for the files contained in the partitions weight few MB.
How do you explain that?
I have been able to fix the issue.
Busybox 1.28.0 commit d1535216 substitutes use of statfs with statvfs (https://github.com/mirror/busybox/commit/d1535216ca27047e3962d61b975bd2a638aa45a2).
I applied the commit to my project using Busybox 1.27.2 and, now, sizes are correct!
Thanks anyway.
Related
I am trying to find out the block size of a file system. I found these 2 commands:
[root#node1 test]# stat -fc %s .
4096
[root#node1 test]# blockdev --getbsz /dev/mapper/node1_test
512
Why is the result different? Which is the correct one?
Many thanks.
I think it's likely that both answers are correct. It's just that blockdev --getbsz returns the result in bytes. Your stat command must be returning bits.. So 512 X 8 = 4096.
It seems that the result of stat would only show in bytes with that format specifier if it was not a filesystem. See the note about the formats on this page for stat.
Reference for blockdev.
I'm trying to base my cluster memory allocation for a given rule on the file size of an input file. Is this possible in snakemake and if so how?
So far I have tried specifying it in the resource: section like so:
rule compute2:
input: "input1.txt"
output: "input2.txt"
resources:
mem_mb=lambda wildcards, input, attempt: int(os.path.getsize(str(input))/(1024*1024))
shell: "touch input2.txt"
But it seems snakemake attempts to calculate this upfront before the file gets created as I'm getting this error:
InputFunctionException in line 35 of test_snakemake/Snakefile:
FileNotFoundError: [Errno 2] No such file or directory: 'input1.txt'
Im running my snakemake with the following command:
snakemake --verbose -j 10 --cluster-config cluster.json --cluster "sbatch -n {cluster.n} -t {cluster.time} --mem {resources.mem_mb}"
If you want to do this dynamically per rule as per the question, you can use something along these lines:
resources: mem_mb=lambda wildcards, input, attempt: (input.size//1000000) * attempt * 10
Where input.size//1000000 is used convert the cumulative size of input files in bytes to mb, and the tailing 10 could be any arbitrary number based on the specifics of your shell/script requirements.
This is possible using the --default-resources option. As explained in Snakemake's --help information:
In addition to plain integers, python expressions over input size are
allowed (e.g. '2*input.size_mb'). When specifying this without any
arguments (--default-resources), it defines
'mem_mb=max(2*input.size_mb, 1000)''disk_mb=max(2*input.size_mb, 1000)',
i.e., default disk and mem usage is twice the input file size
but at least 1GB.
I want to generate some trace file from disk IO, but the problem is I need the actual input data along with timestamp, logical address and access block size, etc.
I've been trying to solve the problem by using the "blktrace | blkparse" with "iozone" on the ubuntu VirtualBox environment, but it seems not working.
There is an option in blkparse for setting the output format to show the packet data, -f "%P", but it dose not print anything.
below is the command that i use:
$> sudo blktrace -a issue -d /dev/sda -o - | blkparse -i - -o ./temp/blktrace.sda.iozone -f "%-12C\t\t%p\t%d\t%S:%n:%N\t\t%P\n"
$> iozone -w -e -s 16M -f ./mnt/iozone.dummy -i 0
In the printing format "%-12C\t\t%p\t%d\t%S:%n:%N\t\t%P\n", all other things are printed well, but the "%P" is not printed at all.
Is there anyone who knows why the packet data is not displayed?
OR anyone who knows other way to get the disk IO packet data with actual input value?
As far as I know blktrace does not capture the actual data. It just capture the metadata. One way to capture real data is to write your own kernel module. Some students at FIU.edu did that in this paper:
"I/O deduplication: Utilizing content similarity to ..."
I would ask this question in linux-btrace mailing list as well:
http://vger.kernel.org/majordomo-info.html
I need to fill a USB memory and I want others to be able to repeat this in an easy way. SO I dont want to write "find a file that filles the memory" so they have to look around for such a file.
Rather I want to generate X MB of data and write that to a file that can then be transferrred to the USB stick.
How would you do that (on Windows)?
If you are on Windows, you can use fsutil:
fsutil file createnew D:\fatboy.tmp SIZE
If you are on Linux or OSX or somesuch, you can use mkfile to make a big file:
mkfile SIZE PathToFileOnUSB
e.g.
mkfile 10m /some/where/on/USB/10MegFile
Or, if your system lacks mkfile, use dd to fill storage quickly and easily.
So, if you want 10MB, use:
dd if=/dev/zero of=PathToFileOnUSB bs=1m count=10
That says... dump data, reading from /dev/zero (which supplies an endless sream of zeroes) and writing to the file called PathToFileOnUSB using a blocksize (bs) of 1 megabyte, and do this 10 times (cnt).
If you want X MB, use:
dd if=/dev/zero of=PathToFileOnUSB bs=1m count=X
If you want to fill the device, write until error without specifying a count:
dd if=/dev/zero of=PathToFileOnUSB bs=1m
Hallo.
I have a big video file. ffmpeg, tcprobe and other tool say, it is an h264-stream in an AVI-container.
Now i'd like to cut out small chunks form the video.
Problem: The index of the video seam corrupted/destroyed. I kind of fixed this via mplayer -forceidx -saveidx <IndexFile> <BigVideoFile>. The Problem here is, that I'm now stuck with mplayer/mencoder which can use this index file via -loadidx <IndexFile>. I have tried correcting the index like described in man aviindex (mplayer -frames 0 -saveidx mpidx broken.avi ; aviindex -i mpidx -o tcindex ; avimerge -x tcindex -i broken.avi -o fixed.avi), but this didn't fix my video - meaning that most tools i've tested couldn't search in the video file.
Problem: I cut out parts of the video via following command: mencoder -loadidx in.idx -ss 8578 -endpos 20 -oac faac -ovc x264 -sws 9 -lavfopts format=mp4 -x264encopts <LotsOfOpts> -of lavf -vf scale=800:-10,harddup in.avi -o out.mp4. Now here the problem is, that some videos are corrupted at the beginning. I think this is because the fact, that i do not necessarily cut at keyframe.
Questions:
What is the best way to fix the index of an avi "inline" so that every tool can again work as expected with it?
How can i split at the keyframes? Is there an mencoder-option for this?
Are Keyframes coming in a frequency? How to find out this frequency? (So with a bit of math it should be possible to calculate the next keyframe and cut there)
Is ther perhaps some completely other way to split this movie? Doing it by hand is no option, i've to cut out 1000+ chunks ...
Thanks a lot!
https://spreadsheets.google.com/ccc?key=0AjWmZ0umsuZHdHNzZVhuMTkxTHdYbUdCQzF3cE51Snc&hl=en lists the various options available for splitting accurately.
I would attempt to use avidemux to repair the file before doing anything. You may also have better results using an MP4 Based Container than AVI.
As for ensuring your specified intervals are right at the keyframe, I would suggest encoding with FFMPEG with the: -g 1 option before using the split below to ensure every frame is in fact a keyframe. FFMPEG refers to keyframs as GOP or Groups of Pictures instead.
ffmpeg -i input.avi -g 1 -vcodec copy -acodec copy out.avi
Then multiple splits (with FFMPEG) :
ffmpeg -i input.avi -ss 00:00:10 -t 00:00:30 out1.avi -ss 00:00:35 -t 00:00:30 out2.avi
Some more options to try:
x264 Mapping FFMPEG encoding in linux