Convert n number of rows to columns repeatedly using awk - awk

My data is a large text file that consists of 12 rows repeating. It looks something like this:
{
1
2
3
4
5
6
7
8
9
10
}
repeating over and over. I want to turn every 12 rows into columns. so the data would look like this:
{ 1 2 3 4 5 6 7 8 9 10 }
{ 1 2 3 4 5 6 7 8 9 10 }
{ 1 2 3 4 5 6 7 8 9 10 }
I have found some examples of how to convert all the rows to columns using awk: awk '{printf("%s ", $0)}', but no examples of how to convert every 12 rows into columns and then repeat the process.

Here is an idiomatic way (read golfed down version of Tom Fenech's answer) of doing it with awk:
$ awk '{ORS=(NR%12?FS:RS)}1' file
{ 1 2 3 4 5 6 7 8 9 10 }
{ 1 2 3 4 5 6 7 8 9 10 }
{ 1 2 3 4 5 6 7 8 9 10 }
ORS stands for Output Record Separator. We set the ORS to FS which by default is space for every line except the 12th line where we set it to RS which is a newline by default.

You could use something like this:
awk '{printf "%s%s", $0, (NR%12?OFS:RS)}' file
NR%12 evaluates to true except when the record number is exactly divisible by 0. When it is true, the output field separator is used (which defaults to a space). When it is false, the record separator is used (by default, a newline).
Testing it out:
$ awk '{printf "%s%s", $0, (NR%12?OFS:RS)}' file
{ 1 2 3 4 5 6 7 8 9 10 }

Related

Using awk, extract the first and last numbers between two strings in a column in a text file, and difference those?

I have a text file that looks similar to below.
Code 1 (3)
5 10 10
6 10 10
7 10 10
Code 2 (2)
9 11 11
10 8 8
Code 3 (1)
12 10 9
Code 4 (2)
14 8 10
15 8 10
I am only interested in the first and last numbers, in the first column. I would like to extract the first, last, and difference (1+last-first) to a new text file with a column for each first, last, and difference. The result should look like below. Technically, the difference column could be the number between the parentheses, as this number would always be the 1+difference between the last and first numbers between each string. Note, the last row in the input text file does not have a string below it.
5 7 3
9 10 2
12 12 1
14 15 2
Trying awk '/Code/{flag=1;next}/Code/{flag=0}flag' gives me all the lines and columns between each string. Trying awk '$1 ~ /Code/{flag=1;next},$1 ~ 1 /Code/{flag=0}flag' results in a syntax error at ,.
You may use this awk:
awk -v OFS='\t' '/^Code/ {
if (NR > 1)
print first, prev, (prev-first+1)
first = prev = ""
next
}
(first == "") {
first = $1
}
{
prev = $1
}
END {
print first, prev, (prev-first+1)
}' file
5 7 3
9 10 2
12 12 1
14 15 2

Transpose column to row using awk

In my data file, there is a certain column I am interested. So I used awk to print out only that column (awk '{print $4}') and put a condition to eliminate using "if". However, I could not figure out how to transpose every nth line on that column to new row.
input:
1
2
3
4
5
6
7
8
9
desired output:
1 4 7
2 5 8
3 6 9
I have checked out the other solutions and tried but none of them gave me what I want. I will appreciate if anyone could help me with that.
you can also use pr here
$ seq 9 | pr -3ts' '
1 4 7
2 5 8
3 6 9
$ seq 9 | pr -5ts' '
1 3 5 7 9
2 4 6 8
where the number indicates how many columns you need and the s option allows to specify the delimiter between columns
Using awk:
$ seq 9 |
awk ' {
i=((i=NR%3)?i:3) # index to hash a
a[i]=a[i] (a[i]==""?"":" ") $1 # space separate items to a[i]
}
END {
for(i=1;i<=3;i++) # from 1 to 3 (yes, hardcoded)
print a[i] # output
}'
Output:
1 4 7
2 5 8
3 6 9
The columns program from the autogen package can do this, e.g.:
seq 9 | columns --by-column -w1 -c3
Output:
1 4 7
2 5 8
3 6 9

AWK: print colums of a matrix using first column as reference

I want to read first colum in a matrix, and then print columns of this matrix using this first colum as reference. And example:
mat.txt
2 10 6 12 3
4 11 1 22 6
5 15 3 18 9
Using first column as reference, I would like to get columns 2, 4 and 5, and also put the value of first colum at the begining.
2 10 12 3
4 11 22 6
5 15 18 9
I try this, but doesn't work well:
awk 'FNR==NR{c++;cols[c]=$1;end}
{for(i=1;i&lt=c;i++) printf("%s%s",$(cols[i]+1),i&ltc ? OFS : "\n")}' mat.txt mat.txt
This may do:
awk 'FNR==NR {a[NR]=$1;next} {printf "%s ",a[FNR];for (i in a) printf "%s ",$(a[i]);print ""}' mat.txt{,}
2 10 12 3
4 11 22 6
5 15 18 9
The {,} make the file be used two times.

Lining up columns using awk

I'm trying to use awk to pull every 9th column out of a dataset with 210 columns. How can I get the columns to line up evenly if the data in each column do not contain the same number of characters?
Use for loop to skip over the column you don't need:
awk '{
for(i=1;i<=8;i++) {
printf "%s%s",$i,FS
}
for(i=10;i<=NF;i++) {
printf "%s%s",$i,(i==NF?RS:FS)
}
}' file
Note: Please set your field separator accordingly. You haven't stated what it is so I am going with default (that is space)
Sample Test: (skipping over 3rd column)
$ cat file
1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
$ awk '{
for(i=1;i<=2;i++) {
printf "%s%s",$i,FS
}
for(i=4;i<=NF;i++) {
printf "%s%s",$i,(i==NF?RS:FS)
}
}' file
1 2 4 5 6
1 2 4 5 6
1 2 4 5 6

Get identical rows

I have a file like this: (data.dat)
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 7
5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 9
6 6 6 6 6 6 6 6 6 6 6 7 6 7
7 9 7 7 7 7 7 7 7 7 7 8 7 9
8 10 8 9 8 9 8 8 8 8 8 9
9 11 9 10 9 9 9 9 9 10
10 12 10 11 10 10 10 11
The odd columns are simple line counters (NR), the even columns are simple values. I would like to get those values, in which the second (or even) colum values are the same in all even columns, i.e. I should get this output:
1
2
3
9
I have already tried to make this line, but something is wrong:
awk '{arr1[$1]=$2;arr2[$3]=$4;arr3[$5]=$6;arr4[$7]=$8;arr5[$9]=$10;arr6[$11]=$12;arr7[$13]=$14;arr8[$15]=$16;}END{for(x in arr1) if(x in arr2 && x in arr3 && x in arr4 && x in arr5 && x in arr6 && x in arr7 && x in arr8) print arr1[x];}' data.dat | sort -n
Is there a better way, by the way?
UPDATE: The real problem is that the array indices are different. So, the arr[...] method does not work... :(
This would work -
awk '
BEGIN{x=0}
{if (x<NF) x=NF;for (i=2;i<=NF;i+=2) a[$i]++}
END{x=x/2;for (y in a) if (x==a[y]) print y}' INPUT_FILE
Explanation:
We set a variable x=0 in the BEGIN statement.
We use this variable to get to find out maximum number of fields (This is useful later).
We store value of every second column to an array and get their number of occurrences.
We divide the variable x by 2 to verify maximum number a value can occur in every second column.
If the occurrences of numbers in an array matches this variable it means they are present in every second column.
Test: with your sample file
[jaypal:~/Temp] awk '
BEGIN{x=0}
{if (x<NF) x=NF;for (i=2;i<=NF;i+=2) a[$i]++}
END{x=x/2;for (y in a) if (x==a[y]) print y}' file
2
3
9
1
You can either pipe the output to sort -n to get it in order or use this -
awk '
BEGIN{x=0}
{if (x<NF) x=NF;for (i=2;i<=NF;i+=2) a[$i]++}
END{x=x/2;for (i=1;i<=length(a);i++) if (x==a[i]) print i}' INPUT_FILE
Your example works with just a simple;
awk '{if($2==$4 && $2==$6 && $2==$8 && $2==$10 && $2==$12 && $2==$14 && $2==$16) print $1}' test.txt | sort -n
Any other requirements I'm missing?
EDIT: Apparently with the missing columns you added :) Try
awk '{if(NF>1) { found=1; for(i=4; i<NF+1; i+=2) { if($2!=$i) { found=0; } } } if(found) print $1}' test.txt | sort -n
In your input data row # 9 doesn't have all even columns same so not sure how you show 9 in your desired output. You can try following awk command to print 1st col for your task:
awk '{same=0; prev=-1; for(i=2;i<=NF;i+=2) {if (prev != -1 && prev != $i) {same=1; break;} else prev=$i;} if (same==0) print $1;}' awk '{same=0; prev=-1; for(i=2;i<=NF;i+=2) {if (prev != -1 && prev != $i) {same=1; break;} else prev=$i;} if (same==0) print $1;}'