Massif visualizer can't display text in legend - valgrind
I already generate a massif output file use command
valgrind --tool=massif --time-unit=ms ./prog
and then open this output file through massif visualizer.
Most of them work well, but the text in label can't display properly(empty square)
Are there any suggestion for me to let text show up properly ? Thanks!
Following output is another method to draw graph: $ ms_print massif.out.16245
--------------------------------------------------------------------------------
Command: ./prog
Massif arguments: --time-unit=B --stacks=yes
ms_print arguments: massif.out.16245
--------------------------------------------------------------------------------
KB
20.13^ #
| #
| #:
| #:
| ###:
| # #:
| # #::
| # #::
| ## #:::
| ## #::: :
| ### #:::::::#::
| ### #:::::::#::
| #### #:::::::#::
| #### #:::::::#::
| ##:: #### #:::::::#::
| # : :#### #:::::::#::
| :# : : :#### #:::::::#::
| :# : : :#### #:::::::#::
| :# : : ::: :#### #:::::::#::
| : :# : : :: :# ::::::::::#::::#::#::::#::::::::::: :::#### #:::::::#::
0 +----------------------------------------------------------------------->KB
0 198.5
Number of snapshots: 73
Detailed snapshots: [4, 7, 11, 22, 27, 30, 35, 51, 52, 53, 54, 55, 57, 59 (peak), 69]
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
0 0 0 0 0 0
1 3,344 1,120 0 0 1,120
2 5,832 824 0 0 824
3 10,016 5,008 0 0 5,008
4 13,800 7,176 0 0 7,176
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
5 18,176 6,400 0 0 6,400
6 22,688 5,008 0 0 5,008
7 26,872 824 0 0 824
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
8 29,760 3,056 0 0 3,056
9 33,240 2,680 0 0 2,680
10 37,624 1,528 0 0 1,528
11 41,144 2,056 0 0 2,056
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
12 44,840 952 0 0 952
13 48,040 1,160 0 0 1,160
14 51,520 1,568 0 0 1,568
15 54,584 1,608 0 0 1,608
16 58,920 1,416 0 0 1,416
17 61,272 1,416 0 0 1,416
18 64,200 1,416 0 0 1,416
19 68,504 1,416 0 0 1,416
20 71,432 1,416 0 0 1,416
21 74,360 1,416 0 0 1,416
22 77,288 1,416 0 0 1,416
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
23 80,216 1,416 0 0 1,416
24 83,144 1,416 0 0 1,416
25 86,072 1,416 0 0 1,416
26 89,000 1,416 0 0 1,416
27 91,912 1,416 0 0 1,416
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
28 94,840 1,416 0 0 1,416
29 98,744 1,416 0 0 1,416
30 101,248 1,312 0 0 1,312
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
31 103,624 1,416 0 0 1,416
32 106,128 1,312 0 0 1,312
33 108,504 1,416 0 0 1,416
34 111,008 1,312 0 0 1,312
35 113,360 1,312 0 0 1,312
00.00% (0B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
36 116,912 1,136 0 0 1,136
37 119,216 1,568 0 0 1,568
38 123,832 1,400 0 0 1,400
39 127,616 1,088 0 0 1,088
40 130,080 1,312 0 0 1,312
41 133,584 1,600 0 0 1,600
42 136,096 1,568 0 0 1,568
43 138,392 1,416 0 0 1,416
44 140,928 1,312 0 0 1,312
45 143,352 1,416 0 0 1,416
46 145,888 1,312 0 0 1,312
47 148,176 224 0 0 224
48 151,816 1,432 1,000 16 416
49 156,520 5,496 5,000 80 416
50 158,872 7,528 7,000 112 416
51 158,872 7,528 7,000 112 416
92.99% (7,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->92.99% (7,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
52 160,048 8,544 8,000 128 416
93.63% (8,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->93.63% (8,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
53 161,224 9,560 9,000 144 416
94.14% (9,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->94.14% (9,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
54 162,400 10,576 10,000 160 416
94.55% (10,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->94.55% (10,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
55 164,584 12,600 12,000 168 432
95.24% (12,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->79.37% (10,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->15.87% (2,000B) 0x108717: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
->15.87% (2,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
56 168,768 16,624 16,000 176 448
57 168,768 16,624 16,000 176 448
96.25% (16,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->60.15% (10,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->24.06% (4,000B) 0x108706: g (in /home/steven/Desktop/linux-design/lab0-c/prog)
| ->24.06% (4,000B) 0x10871C: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
| ->24.06% (4,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->12.03% (2,000B) 0x108717: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
->12.03% (2,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
58 172,984 20,616 20,000 184 432
59 172,984 20,616 20,000 184 432
97.01% (20,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->48.51% (10,000B) 0x108749: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->38.80% (8,000B) 0x108706: g (in /home/steven/Desktop/linux-design/lab0-c/prog)
| ->19.40% (4,000B) 0x10871C: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
| | ->19.40% (4,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
| |
| ->19.40% (4,000B) 0x10876A: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->09.70% (2,000B) 0x108717: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
->09.70% (2,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
60 175,352 18,568 18,000 152 416
61 177,704 16,536 16,000 120 416
62 180,056 14,504 14,000 88 416
63 182,408 12,472 12,000 56 416
64 184,760 10,440 10,000 24 416
65 187,048 10,472 10,000 24 448
66 189,408 11,488 10,000 24 1,464
67 191,696 10,528 10,000 24 504
68 193,984 10,464 10,000 24 440
69 196,296 10,472 10,000 24 448
95.49% (10,000B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->76.39% (8,000B) 0x108706: g (in /home/steven/Desktop/linux-design/lab0-c/prog)
| ->38.20% (4,000B) 0x10871C: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
| | ->38.20% (4,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
| |
| ->38.20% (4,000B) 0x10876A: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->19.10% (2,000B) 0x108717: f (in /home/steven/Desktop/linux-design/lab0-c/prog)
| ->19.10% (2,000B) 0x108765: main (in /home/steven/Desktop/linux-design/lab0-c/prog)
|
->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
--------------------------------------------------------------------------------
n time(B) total(B) useful-heap(B) extra-heap(B) stacks(B)
--------------------------------------------------------------------------------
70 198,632 10,536 10,000 24 512
71 200,936 10,472 10,000 24 448
72 203,272 10,536 10,000 24 512
Related
AWK ouput to separated file
I am trying to split my file based on the patterns that it contains and store them as separate text files with that pattern.
I am using AWK to split my lines by two different separators ("\t" "_") and use $2 as file name and append that line ($0) into related file that is generated.
My command is:
awk 'BEGIN {FS="[_ \t]"}; {fn=$2"_HP100.txt"}; {print $0 >fn}' my_file.txt
Awk can generate text file named after my $2, and also it can give $0 as whole line when I printed separately. But My generated text files are always empty... Am I missing something?
My text file contains:
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
Expected output is CGTCGCCAAACTTAAC_HP100.txt file that contains all the lines with _CGTCGCCAAACTTAAC tag.
You can just use this awk command:
awk -F '[\t_]' '{print $0 >> ($2 "_HP100.txt")}' file
With your shown samples and attempts please try following awk code. Its a combination of awk + sort + cut + awk solution, this will help us in case you have multiple lines where 2nd fields are same and you want to save all same $2 lines in same output file. This is sorting them from $2(by making 2nd field as 1st field to sort it and then using only required/actual portion of input only).
awk -F'_|\t' '{print $2,$0}' Input_file | sort | cut -d' ' -f2- |
awk -F'_|\t' '
prev!=$2{
close(outputFile)
outputFile=$2"_HP100.txt"
}
{
print $0 > (outputFile)
prev=$2
}
'
NOTE: Pass your Input_file to first used awk in above code. Also this code is good for small files.
The approach I have in mind to deal with the 50GB issue is to collect a certain amount each round, then when the input arrays reach a fixed threshold, output the rows according to their file names
instead of nonstop concat-ing into a very very long string - the current approach uses one array for the rows themselves, and one more for mapping filenames to row #s (aka NR) :
collect until reaching threshold (i didn't set one that's up to u)
dump out all data in arrays to files
— also allows for re-creation of original input row order
use print (. . .) >> (fn) instead of >, just in case
use split("", arr) to reset both arrays to a mint state
— it's cleaner and more portable than delete arr statement
host system resource friendly
minimizes concurrent resource usage on the host system by only requiring one single instance of awk
the threshold you set would also constrain RAM usage at any moment to a sane level
avoids the need to use shell loops, xargs, or gnu-parallel
avoids nonstop close(fn)
avoids wasting time performing any amount of data sort-ing
mawk '1;1;1;1;1;1;1' raw_input.txt | shuf |
mawk2 '
BEGIN { OFS = "_HP100.txt"
FS = "^[^_]+_|[ \t].+$"
} { ___[ NR ] = $_
sub(OFS, _, $!(NF = NF))
sub( "$", "=" NR,____[$_])
} END {
OFS = "\f\r\t\t"
for(_ in ___) { print _, ___[_] }
for(_ in ____) { print _,____[_] } } '
1
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
2
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
3
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
4
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
5
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
6
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
7
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
8
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
9
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
10
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
11
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
12
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
13
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
14
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
15
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
16
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
17
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
18
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
19
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
20
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
21
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
22
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
23
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
24
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
25
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
26
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
27
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
28
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
29
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
30
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
31
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
32
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
33
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
34
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
35
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
36
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
37
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
38
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
39
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
40
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
41
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
42
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
43
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
44
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
45
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
46
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
47
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
48
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
49
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
50
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
51
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
52
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
53
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
54
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
55
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
56
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
57
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
58
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
59
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
60
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
61
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
62
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
63
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
TCATACCAAGTCTCCG_HP100.txt
=5=12=15=23=24=40=48
CGTCGCCATCGCTAGG_HP100.txt
=29=30=34=38=44=46=50
TCATCGAACCTCGTTG_HP100.txt
=9=25=27=31=36=49=59
AGCGTCTTTCATGCTG_HP100.txt
=3=8=14=17=18=22=26
GCCTATTCCCTCGTTG_HP100.txt
=11=37=52=54=56=57=61
GATGCGTTTTCTGGTT_HP100.txt
=6=7=10=20=39=41=55
TCGAACGACCGTTGCG_HP100.txt
=16=19=32=42=43=60=63
TGGATCAACAGGACCA_HP100.txt
=1=13=33=35=45=47=62
CGTCGCCAAACTTAAC_HP100.txt
=2=4=21=28=51=53=58
Transpose data in awk
I have a benchmarking tool that has an output looking like this:
Algorithm Data Size CPU Time (ns)
----------------------------------------
bubble_sort 1 16.1
bubble_sort 2 19.1
bubble_sort 4 32.8
bubble_sort 8 74.3
bubble_sort 16 257
bubble_sort 32 997
bubble_sort 64 4225
bubble_sort 128 18925
bubble_sort 256 83565
bubble_sort 512 313589
bubble_sort 1024 1161146
insertion_sort 1 16.1
insertion_sort 2 17.7
insertion_sort 4 26.5
insertion_sort 8 43.7
insertion_sort 16 96.1
insertion_sort 32 263
insertion_sort 64 770
insertion_sort 128 2807
insertion_sort 256 10775
insertion_sort 512 38956
insertion_sort 1024 135419
std_sort 1 17.3
std_sort 2 20.7
std_sort 4 24.4
std_sort 8 32.7
std_sort 16 59.6
std_sort 32 173
std_sort 64 345
std_sort 128 762
std_sort 256 1769
std_sort 512 3982
std_sort 1024 18500
And I'm trying to transform this to become more like this:
Data Size bubble_sort insertion_sort std_sort
1 16.1 16.1 17.3
2 19.1 17.7 20.7
4 32.8 26.5 24.4
8 74.3 43.7 32.7
16 257 96.1 59.6
32 997 263 173
64 4225 770 345
128 18925 2807 762
256 83565 10775 1769
512 313589 38956 3982
1024 1161146 135419 18500
Is there a simple way to achieve this using awk? I'm mostly interested in the numbers in the final table, so the header line isn't essential.
==============================
EDIT:
I was actually able to achieve this using the following code
{
map[$1][$2] = $3
}
END {
for (algo in map) {
some_algo = algo
break;
}
printf "size "
for (algo in map) {
printf "%s ", algo
}
print ""
for (size in map[some_algo]) {
printf "%s ", size
for (algo in map) {
printf "%s ", map[algo][size]
}
printf "\n"
}
}
This works. However, it has two minor problems: It looks a little bit difficult to read, therefore, is there a better and more idiomatic way to do the job? Also, the order of the resulting columns is different from the order in the original data rows. Is there a simple way to fix this order?
here is an alternative
$ sed 1,2d file |
pr -w200 -3t |
awk 'NR==1{print "Data_Size", $1,$4,$7} {print $2,$3,$6,$9}' |
column -t
Data_Size bubble_sort insertion_sort std_sort
1 16.1 16.1 17.3
2 19.1 17.7 20.7
4 32.8 26.5 24.4
8 74.3 43.7 32.7
16 257 96.1 59.6
32 997 263 173
64 4225 770 345
128 18925 2807 762
256 83565 10775 1769
512 313589 38956 3982
1024 1161146 135419 18500
Here is a Ruby to do this.
Ruby is very much like awk but with additional functions and data structures. The advantage to this is it will correctly deal with missing values by inserting n/a if one of the data values is missing.
$ sed 1,2d file |
ruby -lane 'BEGIN{
h=Hash.new {|n,k| n[k]={} }
l="Data_Size"
}
h[l][$F[1]]=$F[1]
h[$F[0]][$F[1]]=$F[2]
END{
puts h.keys.join("\t")
h[l]=h[l].sort{|a,b| a[0].to_i<=>b[0].to_i}.to_h
h[l].each_key { |k|
a=[]
h.each_key { |j|
a.push(h[j][k] || "n/a")
}
puts a.join("\t")
}
}' | column -t
Taking your example and removing the line bubble_sort 4 32.8 prints:
Data_Size bubble_sort insertion_sort std_sort
1 16.1 16.1 17.3
2 19.1 17.7 20.7
4 n/a 26.5 24.4
8 74.3 43.7 32.7
16 257 96.1 59.6
32 997 263 173
64 4225 770 345
128 18925 2807 762
256 83565 10775 1769
512 313589 38956 3982
1024 1161146 135419 18500
How to interpret the log output of docplex optimisation library
I am having a problem interpreting this log that I get after trying to maximise an objective function using docplex: Nodes Cuts/ Node Left Objective IInf Best Integer Best Bound ItCnt Gap 0 0 6.3105 0 10.2106 26 0 0 5.9960 8 Cone: 5 34 0 0 5.8464 5 Cone: 8 47 0 0 5.8030 11 Cone: 10 54 0 0 5.7670 12 Cone: 13 64 0 0 5.7441 13 Cone: 16 72 0 0 5.7044 9 Cone: 19 81 0 0 5.6844 14 5.6844 559 * 0+ 0 4.5362 5.6844 25.31% 0 0 5.5546 15 4.5362 Cuts: 322 1014 22.45% 0 0 5.4738 15 4.5362 Cuts: 38 1108 20.67% * 0+ 0 4.6021 5.4738 18.94% 0 0 5.4296 16 4.6021 Cuts: 100 1155 17.98% 0 0 5.3779 19 4.6021 Cuts: 34 1204 16.86% 0 0 5.3462 17 4.6021 Cuts: 80 1252 16.17% 0 0 5.3396 19 4.6021 Cuts: 42 1276 16.03% 0 0 5.3364 24 4.6021 Cuts: 57 1325 15.96% 0 0 5.3269 17 4.6021 Cuts: 66 1353 15.75% 0 0 5.3188 20 4.6021 Cuts: 42 1369 15.57% 0 0 5.2975 21 4.6021 Cuts: 62 1387 15.11% 0 0 5.2838 24 4.6021 Cuts: 72 1427 14.81% 0 0 5.2796 21 4.6021 Cuts: 70 1457 14.72% 0 0 5.2762 24 4.6021 Cuts: 73 1471 14.65% 0 0 5.2655 24 4.6021 Cuts: 18 1479 14.42% * 0+ 0 4.6061 5.2655 14.32% * 0+ 0 4.6613 5.2655 12.96% 0 0 5.2554 26 4.6613 Cuts: 40 1492 12.75% 0 0 5.2425 27 4.6613 Cuts: 11 1511 12.47% 0 0 5.2360 23 4.6613 Cuts: 3 1518 12.33% 0 0 5.2296 19 4.6613 Cuts: 7 1521 12.19% 0 0 5.2213 18 4.6613 Cuts: 8 1543 12.01% 0 0 5.2163 24 4.6613 Cuts: 15 1552 11.91% 0 0 5.2106 21 4.6613 Cuts: 4 1558 11.78% 0 0 5.2106 21 4.6613 Cuts: 3 1559 11.78% * 0+ 0 4.6706 5.2106 11.56% 0 2 5.2106 21 4.6706 5.2106 1559 11.56% Elapsed time = 9.12 sec. (7822.43 ticks, tree = 0.01 MB, solutions = 5) 51 29 4.9031 3 4.6706 5.1575 1828 10.42% 260 147 4.9207 1 4.6706 5.1575 2699 10.42% 498 242 infeasible 4.6706 5.0909 3364 9.00% 712 346 4.7470 6 4.6706 5.0591 4400 8.32% 991 497 4.7338 6 4.6706 5.0480 5704 8.08% 1358 566 4.8085 11 4.6706 5.0005 7569 7.06% 1708 708 4.7638 14 4.6706 4.9579 9781 6.15% 1985 817 cutoff 4.6706 4.9265 11661 5.48% 2399 843 infeasible 4.6706 4.9058 15567 5.04% 3619 887 4.7066 4 4.6706 4.7875 23685 2.50% Elapsed time = 17.75 sec. (10933.85 ticks, tree = 3.05 MB, solutions = 5) 4623 500 4.6863 13 4.6706 4.7274 35862 1.22% What I don't understand is the following: What is the difference between the third (Objective) and fifth column (Best integer ) How come that the third column (Objective) has higher values than the actual solution of the problem given by CPLEX which is (4.6706) Does the values in the third column take into consideration the constraints given to the optimization problem? This webpage didn't help me to understand neither, the explanation of Best Integer is really confusing. Thank you in advance for your feedback. Regards.
The user manual includes a detailed explanation of this log in section CPLEX->User's Manual for CPLEX->Discrete Optimization->Solving Mixed Integer Programming Problems (MIP)->Progress Reports: interpreting the node log (see https://www.ibm.com/support/knowledgecenter/SSSA5P_12.8.0/ilog.odms.cplex.help/CPLEX/UsrMan/topics/discr_optim/mip/para/52_node_log.html)
I suggest to have a look at in https://fr.slideshare.net/mobile/IBMOptimization/2013-11-informsminingthenodelog
Is pow(x, 2.0) fast as x * x in GLSL? [duplicate]
Which is faster in GLSL: pow(x, 3.0f); or x*x*x; ? Does exponentiation performance depend on hardware vendor or exponent value?
I wrote a small benchmark, because I was interested in the results.
In my personal case, I was most interested in exponent = 5.
Benchmark code (running in Rem's Studio / LWJGL):
package me.anno.utils.bench
import me.anno.gpu.GFX
import me.anno.gpu.GFX.flat01
import me.anno.gpu.RenderState
import me.anno.gpu.RenderState.useFrame
import me.anno.gpu.framebuffer.Frame
import me.anno.gpu.framebuffer.Framebuffer
import me.anno.gpu.hidden.HiddenOpenGLContext
import me.anno.gpu.shader.Renderer
import me.anno.gpu.shader.Shader
import me.anno.utils.types.Floats.f2
import org.lwjgl.opengl.GL11.*
import java.nio.ByteBuffer
import kotlin.math.roundToInt
fun main() {
fun createShader(code: String) = Shader(
"", null, "" +
"attribute vec2 attr0;\n" +
"void main(){\n" +
" gl_Position = vec4(attr0*2.0-1.0, 0.0, 1.0);\n" +
" uv = attr0;\n" +
"}", "varying vec2 uv;\n", "" +
"void main(){" +
code +
"}"
)
fun repeat(code: String, times: Int): String {
return Array(times) { code }.joinToString("\n")
}
val size = 512
val warmup = 50
val benchmark = 1000
HiddenOpenGLContext.setSize(size, size)
HiddenOpenGLContext.createOpenGL()
val buffer = Framebuffer("", size, size, 1, 1, true, Framebuffer.DepthBufferType.NONE)
println("Power,Multiplications,GFlops-multiplication,GFlops-floats,GFlops-ints,GFlops-power,Speedup")
useFrame(buffer, Renderer.colorRenderer) {
RenderState.blendMode.use(me.anno.gpu.blending.BlendMode.ADD) {
for (power in 2 until 100) {
// to reduce the overhead of other stuff
val repeats = 100
val init = "float x1 = dot(uv, vec2(1.0)),x2,x4,x8,x16,x32,x64;\n"
val end = "gl_FragColor = vec4(x1,x1,x1,x1);\n"
val manualCode = StringBuilder()
for (bit in 1 until 32) {
val p = 1.shl(bit)
val h = 1.shl(bit - 1)
if (power == p) {
manualCode.append("x1=x$h*x$h;")
break
} else if (power > p) {
manualCode.append("x$p=x$h*x$h;")
} else break
}
if (power.and(power - 1) != 0) {
// not a power of two, so the result isn't finished yet
manualCode.append("x1=")
var first = true
for (bit in 0 until 32) {
val p = 1.shl(bit)
if (power.and(p) != 0) {
if (!first) {
manualCode.append('*')
} else first = false
manualCode.append("x$p")
}
}
manualCode.append(";\n")
}
val multiplications = manualCode.count { it == '*' }
// println("$power: $manualCode")
val shaders = listOf(
// manually optimized
createShader(init + repeat(manualCode.toString(), repeats) + end),
// can be optimized
createShader(init + repeat("x1=pow(x1,$power.0);", repeats) + end),
// can be optimized, int as power
createShader(init + repeat("x1=pow(x1,$power);", repeats) + end),
// slightly different, so it can't be optimized
createShader(init + repeat("x1=pow(x1,${power}.01);", repeats) + end),
)
for (shader in shaders) {
shader.use()
}
val pixels = ByteBuffer.allocateDirect(4)
Frame.bind()
glClearColor(0f, 0f, 0f, 1f)
glClear(GL_COLOR_BUFFER_BIT or GL_DEPTH_BUFFER_BIT)
for (i in 0 until warmup) {
for (shader in shaders) {
shader.use()
flat01.draw(shader)
}
}
val flops = DoubleArray(shaders.size)
val avg = 10 // for more stability between runs
for (j in 0 until avg) {
for (index in shaders.indices) {
val shader = shaders[index]
GFX.check()
val t0 = System.nanoTime()
for (i in 0 until benchmark) {
shader.use()
flat01.draw(shader)
}
// synchronize
glReadPixels(0, 0, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixels)
GFX.check()
val t1 = System.nanoTime()
// the first one may be an outlier
if (j > 0) flops[index] += multiplications * repeats.toDouble() * benchmark.toDouble() * size * size / (t1 - t0)
GFX.check()
}
}
for (i in flops.indices) {
flops[i] /= (avg - 1.0)
}
println(
"" +
"$power,$multiplications," +
"${flops[0].roundToInt()}," +
"${flops[1].roundToInt()}," +
"${flops[2].roundToInt()}," +
"${flops[3].roundToInt()}," +
(flops[0] / flops[3]).f2()
)
}
}
}
}
The sampler function is run 9x 512² pixels * 1000 times, and evaluates the function 100 times each.
I run this code on my RX 580, 8GB from Gigabyte, and collected the following results:
Power
#Mult
GFlops*
GFlopsFp
GFlopsInt
GFlopsPow
Speedup
2
1
1246
1429
1447
324
3.84
3
2
2663
2692
2708
651
4.09
4
2
2682
2679
2698
650
4.12
5
3
2766
972
974
973
2.84
6
3
2785
978
974
976
2.85
7
4
2830
1295
1303
1299
2.18
8
3
2783
2792
2809
960
2.90
9
4
2836
1298
1301
1302
2.18
10
4
2833
1291
1302
1298
2.18
11
5
2858
1623
1629
1623
1.76
12
4
2824
1302
1295
1303
2.17
13
5
2866
1628
1624
1626
1.76
14
5
2869
1614
1623
1611
1.78
15
6
2886
1945
1943
1953
1.48
16
4
2821
1305
1300
1305
2.16
17
5
2868
1615
1625
1619
1.77
18
5
2858
1620
1625
1624
1.76
19
6
2890
1949
1946
1949
1.48
20
5
2871
1618
1627
1625
1.77
21
6
2879
1945
1947
1943
1.48
22
6
2886
1944
1949
1952
1.48
23
7
2901
2271
2269
2268
1.28
24
5
2872
1621
1628
1624
1.77
25
6
2886
1942
1943
1942
1.49
26
6
2880
1949
1949
1953
1.47
27
7
2891
2273
2263
2266
1.28
28
6
2883
1949
1946
1953
1.48
29
7
2910
2279
2281
2279
1.28
30
7
2899
2272
2276
2277
1.27
31
8
2906
2598
2595
2596
1.12
32
5
2872
1621
1625
1622
1.77
33
6
2901
1953
1942
1949
1.49
34
6
2895
1948
1939
1944
1.49
35
7
2895
2274
2266
2268
1.28
36
6
2881
1937
1944
1948
1.48
37
7
2894
2277
2270
2280
1.27
38
7
2902
2275
2264
2273
1.28
39
8
2910
2602
2594
2603
1.12
40
6
2877
1945
1947
1945
1.48
41
7
2892
2276
2277
2282
1.27
42
7
2887
2271
2272
2273
1.27
43
8
2912
2599
2606
2599
1.12
44
7
2910
2278
2284
2276
1.28
45
8
2920
2597
2601
2600
1.12
46
8
2920
2600
2601
2590
1.13
47
9
2925
2921
2926
2927
1.00
48
6
2885
1935
1955
1956
1.47
49
7
2901
2271
2279
2288
1.27
50
7
2904
2281
2276
2278
1.27
51
8
2919
2608
2594
2607
1.12
52
7
2902
2282
2270
2273
1.28
53
8
2903
2598
2602
2598
1.12
54
8
2918
2602
2602
2604
1.12
55
9
2932
2927
2924
2936
1.00
56
7
2907
2284
2282
2281
1.27
57
8
2920
2606
2604
2610
1.12
58
8
2913
2593
2597
2587
1.13
59
9
2925
2923
2924
2920
1.00
60
8
2930
2614
2606
2613
1.12
61
9
2932
2946
2946
2947
1.00
62
9
2926
2935
2937
2947
0.99
63
10
2958
3258
3192
3266
0.91
64
6
2902
1957
1956
1959
1.48
65
7
2903
2274
2267
2273
1.28
66
7
2909
2277
2276
2286
1.27
67
8
2908
2602
2606
2599
1.12
68
7
2894
2272
2279
2276
1.27
69
8
2923
2597
2606
2606
1.12
70
8
2910
2596
2599
2600
1.12
71
9
2926
2921
2927
2924
1.00
72
7
2909
2283
2273
2273
1.28
73
8
2909
2602
2602
2599
1.12
74
8
2914
2602
2602
2603
1.12
75
9
2924
2925
2927
2933
1.00
76
8
2904
2608
2602
2601
1.12
77
9
2911
2919
2917
2909
1.00
78
9
2927
2921
2917
2935
1.00
79
10
2929
3241
3246
3246
0.90
80
7
2903
2273
2276
2275
1.28
81
8
2916
2596
2592
2589
1.13
82
8
2913
2600
2597
2598
1.12
83
9
2925
2931
2926
2913
1.00
84
8
2917
2598
2606
2597
1.12
85
9
2920
2916
2918
2927
1.00
86
9
2942
2922
2944
2936
1.00
87
10
2961
3254
3259
3268
0.91
88
8
2934
2607
2608
2612
1.12
89
9
2918
2939
2931
2916
1.00
90
9
2927
2928
2920
2924
1.00
91
10
2940
3253
3252
3246
0.91
92
9
2924
2933
2926
2928
1.00
93
10
2940
3259
3237
3251
0.90
94
10
2928
3247
3247
3264
0.90
95
11
2933
3599
3593
3594
0.82
96
7
2883
2282
2268
2269
1.27
97
8
2911
2602
2595
2600
1.12
98
8
2896
2588
2591
2587
1.12
99
9
2924
2939
2936
2938
1.00
As you can see, a power() call takes exactly as long as 9 multiplication instructions. Therefore every manual rewriting of a power with less than 9 multiplications is faster.
Only the cases 2, 3, 4, and 8 are optimized by my driver. The optimization is independent of whether you use the .0 suffix for the exponent.
In the case of exponent = 2, my implementation seems to have lower performance than the driver. I am not sure, why.
The speedup is the manual implementation compared to pow(x,exponent+0.01), which cannot be optimized by the compiler.
Because the multiplications and the speedup align so perfectly, I created a graph to show the relationship. This relationship kind of shows that my benchmark is trustworthy :).
Operating System: Windows 10 Personal
GPU: RX 580 8GB from Gigabyte
Processor: Ryzen 5 2600
Memory: 16 GB DDR4 3200
GPU Driver: 21.6.1 from 17th June 2021
LWJGL: Version 3.2.3 build 13
While this can definitely be hardware/vendor/compiler dependent, advanced mathematical functions like pow() tend to be considerably more expensive than basic operations. The best approach is of course to try both, and benchmark. But if there is a simple replacement for an advanced mathematical functions, I don't think you can go very wrong by using it. If you write pow(x, 3.0), the best you can probably hope for is that the compiler will recognize the special case, and expand it. But why take the risk, if the replacement is just as short and easy to read? C/C++ compilers don't always replace pow(x, 2.0) by a simple multiplication, so I wouldn't necessarily count on all GLSL compilers to do that.
Syntax error with AMPL
I get syntax error while running this script in AMPL. Can someone help me to solve this?
param K;
param N;
param PT;
param beta_lower{1..K};
param beta_upper{1..K};
set KSET := {1 . . K};
set NSET := {1 . . N};
param Channel {KSET,NSET};
var V
var C {KSET, NSET} binary;
#==================================
data;
param K:=2;
param N:=64;
param PT:= 1;
param beta_lower:= 1 1.99 2 3.99;
param beta_upper:= 1 2.01 2 4.01;
param Channel : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 :=
1 1366 1474 1583 1690 1790 1881 1963 2036 2101 2161 2217 2268 2315 2355 2385 2402 2403 2386 2350 2295 2223 2137 2041 1939 1835 1734 1639 1553 1479 1419 1375 1347 1335 1339 1357 1386 1421 1459 1494 1523 1542 1548 1540 1520 1490 1451 1409 1364 1321 1279 1239 1201 1164 1127 1092 1060 1034 1016 1012 1024 1055 1107 1178 1265
2 1297 1281 1250 1201 1135 1055 963 867 772 685 611 555 519 504 510 536 579 636 702 775 851 928 1002 1074 1143 1209 1276 1345 1420 1503 1596 1698 1808 1921 2033 2137 2225 2290 2327 2333 2309 2256 2180 2089 1989 1890 1796 1712 1641 1582 1533 1493 1458 1425 1393 1364 1337 1314 1298 1289 1288 1292 1297 1301;
I write this piece of code in tex file (.rtf) and upload this to neos-server
The output from the solver is:
amplin, line 7 (offset 54):
syntax error
context: >>> {\ <<< rtf1\ansi\ansicpg1252\cocoartf12processing commands.
Executing on neos-2.neos-server.org
Error (2) in /opt/ampl/ampl -R amplin
The problem is that you mix syntax for AMPL model and data in your code. In particular, you should first declare parameter beta_lower in the model:
param beta_lower{1..K}; # you may want a different indexing expression here
and then provide data for it in the data section:
data;
param beta_lower:= 1 1.99 2 3.99;
Your updated formulation looks correct, but the error indicates that it is in RTF format. To make it work you should convert it into plain text.