I am trying to find out the block size of a file system. I found these 2 commands:
[root#node1 test]# stat -fc %s .
4096
[root#node1 test]# blockdev --getbsz /dev/mapper/node1_test
512
Why is the result different? Which is the correct one?
Many thanks.
I think it's likely that both answers are correct. It's just that blockdev --getbsz returns the result in bytes. Your stat command must be returning bits.. So 512 X 8 = 4096.
It seems that the result of stat would only show in bytes with that format specifier if it was not a filesystem. See the note about the formats on this page for stat.
Reference for blockdev.
Related
I am using a Splunk query to calculate the size of logs files sent to Splunk. This is the Splunk query I have used:
index="<my_index>" path="/<my_path>/<my_log_file>"
| eval raw_len=len(_raw)
| eval raw_len_kb = raw_len/1024
| eval raw_len_mb = raw_len/1024/1024
| eval raw_len_gb = raw_len/1024/1024/1024
| stats sum(raw_len) as Bytes sum(raw_len_kb) as KB sum(raw_len_mb) as MB sum(raw_len_gb) as GB by source
| addcoltotals
Splunk reports the size as 17 GB. On the other hand, when I do this on the Unix host:
ls -l /<my_path>/<my_log_file>
the value is just a few MB.
Any idea why there is so much difference?
One should not expect the size of data indexed in Splunk to exactly match the size reported by an OS. This is because Splunk by default removes line ends and because the len function counts characters rather than bytes.
Also, the query shown does not account for multiple hosts sending data to Splunk. There's no time window indicated so we don't know if the file may have been truncated at time point while Splunk still retains all of the data the file ever held.
I am using a 64 MB QSPI formatted in some UBI partitions.
df is an applet of busybox 1.27.2
Actually,
~ # df -h
Filesystem Size Used Available Use% Mounted on
/dev/ubi0_0 3.1T 1.9T 1.2T 63% /
/dev/ubi1_0 1.6T 21.8G 1.5T 1% /conf
But, obviously, the size cannot be that! Anyway, the use % seems to be correct, for the files contained in the partitions weight few MB.
How do you explain that?
I have been able to fix the issue.
Busybox 1.28.0 commit d1535216 substitutes use of statfs with statvfs (https://github.com/mirror/busybox/commit/d1535216ca27047e3962d61b975bd2a638aa45a2).
I applied the commit to my project using Busybox 1.27.2 and, now, sizes are correct!
Thanks anyway.
I'm trying to base my cluster memory allocation for a given rule on the file size of an input file. Is this possible in snakemake and if so how?
So far I have tried specifying it in the resource: section like so:
rule compute2:
input: "input1.txt"
output: "input2.txt"
resources:
mem_mb=lambda wildcards, input, attempt: int(os.path.getsize(str(input))/(1024*1024))
shell: "touch input2.txt"
But it seems snakemake attempts to calculate this upfront before the file gets created as I'm getting this error:
InputFunctionException in line 35 of test_snakemake/Snakefile:
FileNotFoundError: [Errno 2] No such file or directory: 'input1.txt'
Im running my snakemake with the following command:
snakemake --verbose -j 10 --cluster-config cluster.json --cluster "sbatch -n {cluster.n} -t {cluster.time} --mem {resources.mem_mb}"
If you want to do this dynamically per rule as per the question, you can use something along these lines:
resources: mem_mb=lambda wildcards, input, attempt: (input.size//1000000) * attempt * 10
Where input.size//1000000 is used convert the cumulative size of input files in bytes to mb, and the tailing 10 could be any arbitrary number based on the specifics of your shell/script requirements.
This is possible using the --default-resources option. As explained in Snakemake's --help information:
In addition to plain integers, python expressions over input size are
allowed (e.g. '2*input.size_mb'). When specifying this without any
arguments (--default-resources), it defines
'mem_mb=max(2*input.size_mb, 1000)''disk_mb=max(2*input.size_mb, 1000)',
i.e., default disk and mem usage is twice the input file size
but at least 1GB.
I'm using Perl v5.22.1, Storable 2.53_01, and IO::Uncompress::Gunzip 2.068.
I want to use Perl to gunzip a Storable file in memory, without using an intermediate file.
I have a variable $zip_file = '/some/storable.gz' that points to this zipped file.
If I gunzip directly to a file, this works fine, and %root is correctly set to the Storable hash.
gunzip($zip_file, '/home/myusername/Programming/unzipped');
my %root = %{retrieve('/home/myusername/Programming/unzipped')};
However if I gunzip into memory like this:
my $file;
gunzip($zip_file, \$file);
my %root = %{thaw($file)};
I get the error
Storable binary image v56.115 more recent than I am (v2.10)`
so the Storable's magic number has been butchered: it should never be that high.
However, the strings in the unzipped buffer are still correct; the buffer starts with pst which is the correct Storable header. It only seems to be multi-byte variables like integers which are being broken.
Does this have something to do with byte ordering, such that writing to a file works one way while writing to a file buffer works in another? How can I gunzip to a buffer without it ruining my integers?
That's not related to unzip but to using retrieve vs. thaw. They both expect different input, i.e. thaw expect the output from freeze while retrieve expects the output from store.
This can be verified with a simple test:
$ perl -MStorable -e 'my $x = {}; store($x,q[file.store])'
$ perl -MStorable=freeze -e 'my $x = {}; print freeze($x)' > file.freeze
On my machine this gives 24 bytes for the file created by store and 20 bytes for freeze. If I remove the leading 4 bytes from file.store the file is equivalent to file.freeze, i.e. store just added a 4 byte header. Thus you might try to uncompress the file in memory, remove the leading 4 bytes and run thaw on the rest.
I tried carl function
http://www.rebol.com/article/0281.html
with 155 Mo it works.
Then I tested with 7 Go it fails without saying the limit.
Why is there a limit I can't see anything in code that puts a limit.
There's no error message
>> copy-file to-rebol-file "D:\#mirror_ftp\cpmove.tar" to-rebol-file "D:\#mirror_ftp\testcopy.tar"
0:00
== none
>>
REBOL uses 32-bit signed integers, so it can't read files bigger than 2147483647 bytes (2^31-1) which is roughly 2GB. REBOL3 uses 64-bit integers, so won't have such limitation.