I'm working with a tab separated file on MacOS. The file contains 15 columns and thousands of rows. I want to cut columns 1, 2, and 3 and then append them with columns 11, 12, and 13. I was hoping to do this in a pipe so that no extra files need to be created. The only post I found used a command sponge but I evidently don't have that on MacOS, or it isn't in my BASH.
The input tsv file is actually being generated within the same line of code,
arbitrary command to generate input.tsv | cut -f1-3,11-13 | <Step to cut -f4-6 and append -f1-3> | sort > out.file
Input tsv
chr1 21018 21101 A B C D E F G chr1 20752 21209
chr10 74645 74836 A B C D E F G chr10 74638 74898
chr10 75267 75545 A B C D E F G chr10 75280 75917
chr4 212478 212556 A B C D E F G chr4 212491 213285
Desired Output tsv
chr1 21018 21101
chr1 20752 21209
chr10 74638 74898
chr10 74645 74836
chr10 75280 75917
chr4 212478 212556
chr4 212491 213285
Using perl and awk :
code
perl -pe 's/chr[0-9]+/\n$&/g' file | awk '/./{print $1, $2, $3}'
Output
chr1 21018 21101
chr1 20752 21209
chr10 74645 74836
chr10 74638 74898
chr10 75267 75545
chr10 75280 75917
chr4 212478 212556
chr4 212491 213285
here is short awk solution:
awk '{print $1, $2, $3, "\n" $1, $12, $13;}' input.tsv
output:
chr1 21018 21101
chr1 20752 21209
chr10 74645 74836
chr10 74638 74898
chr10 75267 75545
chr10 75280 75917
chr4 212478 212556
chr4 212491 213285
Explanation
{ # for each input line
print $1, $2, $3; # print 1st field, append 2nd and 3rd fields. Terminate with new line
print $1, $12, $13; #print 1st field, append 12th and 13th field. Terminate with new line
}
Related
I am trying to use awk to check if each $2 in file1 falls between $2 and $3 of the matching $4 line of file2. If it does then in $5 of file2, exon if it does not intron. I think the awk below will do that, but I am struggling trying to is add a calculation that if the difference is less than or equal to 10, then $5 is splicing. I have added an example of line 1 as well.
The 6th line is an example of the splicing, because the $2 value in file1 is 2 away from the $2 value in file2. My actual data is very large with file2 always being several hundreds of thousand lines. File 1 will be variable but usually ~100 lines. The files are hardcoded in this example but will be gotten from a bash for loop. That will provide the input. Thank you :).
file1 tab-delimited with whitespace after $3 and $4
chr1 17345304 17345315 SDHB
chr1 17345516 17345524 SDHB
chr1 93306242 93306261 RPL5
chr1 93307262 93307291 RPL5
chrX 153295819 153296875 MECP2
chrX 153295810 153296830 MECP2
file2 tab-delimited
chr1 17345375 17345453 SDHB_cds_0_0_chr1_17345376_r 0 -
chr1 17349102 17349225 SDHB_cds_1_0_chr1_17349103_r 0 -
chr1 17350467 17350569 SDHB_cds_2_0_chr1_17350468_r 0 -
chr1 17354243 17354360 SDHB_cds_3_0_chr1_17354244_r 0 -
chr1 17355094 17355231 SDHB_cds_4_0_chr1_17355095_r 0 -
chr1 17359554 17359640 SDHB_cds_5_0_chr1_17359555_r 0 -
chr1 17371255 17371383 SDHB_cds_6_0_chr1_17371256_r 0 -
chr1 17380442 17380514 SDHB_cds_7_0_chr1_17380443_r 0 -
chr1 93297671 93297674 RPL5_cds_0_0_chr1_93297672_f 0 +
chr1 93298945 93299015 RPL5_cds_1_0_chr1_93298946_f 0 +
chr1 93299101 93299217 RPL5_cds_2_0_chr1_93299102_f 0 +
chr1 93300335 93300470 RPL5_cds_3_0_chr1_93300336_f 0 +
chr1 93301746 93301949 RPL5_cds_4_0_chr1_93301747_f 0 +
chr1 93303012 93303190 RPL5_cds_5_0_chr1_93303013_f 0 +
chr1 93306107 93306196 RPL5_cds_6_0_chr1_93306108_f 0 +
chr1 93307322 93307422 RPL5_cds_7_0_chr1_93307323_f 0 +
chrX 153295817 153296901 MECP2_cds_0_0_chrX_153295818_r 0 -
chrX 153297657 153298008 MECP2_cds_1_0_chrX_153297658_r 0 -
chrX 153357641 153357667 MECP2_cds_2_0_chrX_153357642_r 0 -
desired output tab-delimited
chr1 17345304 17345315 SDHB intron
chr1 17345516 17345524 SDHB intron
chr1 93306242 93306261 RPL5 intron
chr1 93307262 93307291 RPL5 intron
chrX 153295819 153296875 MECP2 exon
chrX 153295810 153296800 MECP2 splicing
awk
awk '
FNR==NR{
a[$4];
min[$4]=$2;
max[$4]=$3;
next
}
{
split($4,array,"_");
print $0,(array[1] in a) && ($2>=min[array[1]] &&
$2<=max[array[1]])?"exon":"intron"
}' file1 OFS="\t" file2 > output
example of line 1
a[$4] = SDHB
min[$4] = 17345304
max[$4] = 17345315
array[1] = SDHB, 17345304 >= 17345375 && array[1] = SDHB, 17345315 <= 17345453 ---- intron
I have a tab separated text file and want to do some math operation on one column and make a new tab separated text file.
this is an example of my file:
chr1 144520803 144520804 12 chr1 144520813 58
chr1 144520840 144520841 12 chr1 144520845 36
chr1 144520840 144520841 12 chr1 144520845 36
chr1 144520848 144520849 14 chr1 144520851 32
chr1 144520848 144520849 14 chr1 144520851 32
i want to change the 4th column. in fact I want to divide every single element in the 4th column by sum of all elements in the 4th column and then multiply by 1000000 . like the expected output.
expected output:
chr1 144520803 144520804 187500 chr1 144520813 58
chr1 144520840 144520841 187500 chr1 144520845 36
chr1 144520840 144520841 187500 chr1 144520845 36
chr1 144520848 144520849 218750 chr1 144520851 32
chr1 144520848 144520849 218750 chr1 144520851 32
I am trying to do that in awk using the following command but it does not return what I want. do you know how to fix it:
awk '{print $1 "\t" $2 "\t" $3 "\t" $4/{sum+=$4}*1000000 "\t" $5 "\t" $6 "\t" $7}' myfile.txt > new_file.txt
you need two passes, one to compute the sum and then to scale the field
something like this
$ awk -v OFS='\t' 'NR==FNR {sum+=$4; next}
{$4*=(1000000/sum)}1' file{,} > newfile
In the below awk I am trying to capture all conditions ofKCNMA1, the line in gene (which is a one column list of names) that are in $8 of file which is tab-delimited
So in the below example all instances/lines where KCNMA1 appear in $8would be printed to output.
There could also be multiple ;, however the name, in this case KCNMA1 will be included. The awk seems to capture 2 of the possible 4 conditions but not all instances as shown by the current output. Thank you :).
gene
KCNMA1
file
R_Index Chr Start End Ref Alt Func.IDP.refGene Gene.IDP.refGene GeneDetail.IDP.refGene
4629 chr10 78944590 78944590 G A intergenic NONE;KCNMA1 dist=NONE;dist=451371
4630 chr10 79396463 79396463 C T intronic KCNMA1 .
4631 chr10 79397777 79397777 C - exonic KCNMA1;X1X .
4632 chr10 81318663 81318663 C G exonic SFTPA2 .
4633 chr10 89397777 89397777 - GAA exonic NONE;X1X;KCNMA1 .
current output
R_Index Chr Start End Ref Alt Func.IDP.refGene Gene.IDP.refGene GeneDetail.IDP.refGene
1 chr10 79396463 79396463 C T intronic KCNMA1 .
2 chr10 79397777 79397777 C - exonic KCNMA1;X1X .
desired output (tab-delimeted)
R_Index Chr Start End Ref Alt Func.IDP.refGene Gene.IDP.refGene GeneDetail.IDP.refGene
4629 chr10 78944590 78944590 G A intergenic NONE;KCNMA1 dist=NONE;dist=451371
4630 chr10 79396463 79396463 C T intronic KCNMA1 .
4631 chr10 79397777 79397777 C - exonic KCNMA1;X1X .
4633 chr10 89397777 89397777 - GAA exonic NONE;X1X;KCNMA1 .
awk
awk -F'\t' 'NR==FNR{a[$0];next} FNR==1{print} {x=$8; sub(/;.*/,"",x)} x in a{$1=++c; print}' gene file > out
For the single gene, just pass as a variable
$ awk -v gene='KCNMA1' -v d=';' 'NR==1 || d $8 d ~ d gene d' file
the counter you're using seems unnecessary since you want to have the first field.
If you want to support a file based gene list, you can use this
$ awk -v d=';' 'NR==FNR {genes[$0]; next}
FNR==1;
{for(g in genes)
if(d $8 d ~ d g d) print}' genes file
I have a file :
chr1 1234 2345 EG1234:E1
chr1 2350 2673 EG1234:E2
chr1 2673 2700 EG1234:E2
chr1 2700 2780 EG1234:E2
chr2 5672 5700 EG2345:E1
chr2 5705 5890 EG2345:E2
chr2 6000 6010 EG2345:E3
chr2 6010 6020 EG2345:E3
As you can see there is a specific ID before ':' and there is an id that is repeated after ':' which might be common to more than one row , I want an output that looks something like this:
chr1 1234 2345 EG1234:E1 (output as it is since it doesn't have duplicate id in the next row)
chr1 2350 2780 EG1234:E2 (since duplicate the 1st and 2nd column of 1st occurrence &
3rd and 4 th column of the last occurence)
similarly
chr2 5672 5700 EG2345:E1
chr2 5705 5890 EG2345:E2
chr2 6000 6020 EG2345:E3
I was trying to use a key to move to next column but I am not quiet sure as to how would I extract the column wise values
awk '{key=$4; if (!(key in data)) c[++n]=key; data[key]=$0} END{for (i=1; i<=n; i++) print data[c[i]]}' file1
In short I want to extract the first two columns of first occurrence and last two columns from the last occurrence of any rows with duplicate 4 th column
This one only messes up the record order:
($1 FS $4 in a) { # combination of $1 and $4 is the key
split(a[$1 FS $4],b) # split to get the old $2
a[$1 FS $4]=b[1] FS b[2] FS $3 FS b[4] # update $3
next
}
{
a[$1 FS $4]=$0 # new key found
}
END {
for(i in a) # print them all
print a[i]
}
Test it:
$ awk -f foo.awk foo.txt
chr1 EG1234:E2 2350 2780
chr2 EG2345:E1 5672 5700
chr2 EG2345:E2 5705 5890
chr2 EG2345:E3 6000 6020
chr1 EG1234:E1 1234 2345
One-liner:
$ awk '($1 FS $4 in a) {split(a[$1 FS $4],b); a[$1 FS $4]=b[1] FS b[2] FS $3 FS b[4]; next} {a[$1 FS $4]=$0} END {for(i in a) print a[i]}' foo.txt
Using awk, considering the key1:key2 as a unique combination and if applying it to filter duplicates. Here $4 represents the key1:key2 from your file.
awk '!seen[$4]++' file
chr1 1234 2345 EG1234:E1
chr1 2350 2673 EG1234:E2
chr2 5672 5700 EG2345:E1
chr2 5705 5890 EG2345:E2
chr2 6000 6010 EG2345:E3
The logic is straight forward, the line identified by key1:key2 is printed only if it is not seen already.
I am trying to find the differences between file1.txt and file2.txt and output the differences. I tried diff and sed and the output does not return any differences. I also tried awk and matching on $2, but I think the syntax is wrong as a file gets created but it is 0kb. The actual data I am using is quite large but I know there should be 18 differences. Thank you :).
awk 'NR==FNR{a[$2]++;next} !($2 in a){print $2}' file1.txt file2.txt > diff.txt
file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6
Desired output
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6
Diff result (since these are the two records that are not in both files)
1,52058c1,52040
< chr1 955542 955763
< chr1 957570 957852
< chr1 976034 976270
I'm curious while diff isn't working like you want, but your awk logic isn't correct:
You're checking the second field's (delimited by spaces) value only. In your example the second field is all identical so nothing is being printed out. Using the whole line instead works as expected:
Using your example text where all is different:
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
Here's with the second line identical just to show it working in a more obvious way.
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
EDIT
Based on a comment stating:
"There should be 18 differences out of 52,000 lines. File1.txt is 52,058 entries and file2.txt has 52,040 entries in it. I am trying to find out what the 18 are"
Given you said file1 has more lines, you need to process file2 first. The first file read is populating the array and then the second is checking for lines existing in that array. You need to process the smaller file first so that the additional lines you're interested in aren't in the array. It'd be the same logic above, just with the file order switched, e.g.:
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
New Line!
Not in file2!
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file2.txt file1.txt > diff.txt
$ cat diff.txt
chr1 955542 955763
chr1 976034 976270
New Line!
Not in file2!
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
Note that reading file1 first doesn't emit the additional lines.
If you don't care about the additional text on the lines, just the text in the second field, then you could use $2 as you originally did.
$ awk 'NR==FNR{a[$2];next} !($2 in a)' file1 file2
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6