I have a large file that I want to use awk to count the lines in a specific column $5, before the: and only count -uniq entries, but seem to be having trouble getting the syntax correct. Thank you :).
Sample Input
chr1 955542 955763 + AGRN:exon.1 1 0
chr1 955542 955763 + AGRN:exon.1 2 0
chr1 955542 955763 + AGRN:exon.1 3 0
chr1 955542 955763 + AGRN:exon.1 4 1
chr1 955542 955763 + AGRN:exon.1 5 1
awk -F: ' NR > 1 { count += $5 } -uniq' Input
Desired output
1
$ awk -F'[ \t:]+' '{a[$5]=1;} END{for (k in a)n++; print n;}' Input
1
-F'[ \t:]+'
This tells awk to use spaces, tabs, or colons as the field separator.
a[$5]=1
As we loop through each line, this adds an entry into associative array a for each value of $5 encountered.
END{for (k in a)n++; print n;}
After we have finished reading the file, this counts the number of keys in associative array a and prints the total.
The idiomatic, portable awk approach:
$ awk '{sub(/:.*/,"",$5)} !seen[$5]++{unq++} END{print unq}' file
1
The briefer but gawk-only (courtesy of length(array)) approach:
$ awk '{seen[$5]} END{print length(seen)}' file
1
Related
I am trying to use awk to check if each $2 in file1 falls between $2 and $3 of the matching $4 line of file2. If it does then in $5 of file2, exon if it does not intron. I think the awk below will do that, but I am struggling trying to is add a calculation that if the difference is less than or equal to 10, then $5 is splicing. I have added an example of line 1 as well.
The 6th line is an example of the splicing, because the $2 value in file1 is 2 away from the $2 value in file2. My actual data is very large with file2 always being several hundreds of thousand lines. File 1 will be variable but usually ~100 lines. The files are hardcoded in this example but will be gotten from a bash for loop. That will provide the input. Thank you :).
file1 tab-delimited with whitespace after $3 and $4
chr1 17345304 17345315 SDHB
chr1 17345516 17345524 SDHB
chr1 93306242 93306261 RPL5
chr1 93307262 93307291 RPL5
chrX 153295819 153296875 MECP2
chrX 153295810 153296830 MECP2
file2 tab-delimited
chr1 17345375 17345453 SDHB_cds_0_0_chr1_17345376_r 0 -
chr1 17349102 17349225 SDHB_cds_1_0_chr1_17349103_r 0 -
chr1 17350467 17350569 SDHB_cds_2_0_chr1_17350468_r 0 -
chr1 17354243 17354360 SDHB_cds_3_0_chr1_17354244_r 0 -
chr1 17355094 17355231 SDHB_cds_4_0_chr1_17355095_r 0 -
chr1 17359554 17359640 SDHB_cds_5_0_chr1_17359555_r 0 -
chr1 17371255 17371383 SDHB_cds_6_0_chr1_17371256_r 0 -
chr1 17380442 17380514 SDHB_cds_7_0_chr1_17380443_r 0 -
chr1 93297671 93297674 RPL5_cds_0_0_chr1_93297672_f 0 +
chr1 93298945 93299015 RPL5_cds_1_0_chr1_93298946_f 0 +
chr1 93299101 93299217 RPL5_cds_2_0_chr1_93299102_f 0 +
chr1 93300335 93300470 RPL5_cds_3_0_chr1_93300336_f 0 +
chr1 93301746 93301949 RPL5_cds_4_0_chr1_93301747_f 0 +
chr1 93303012 93303190 RPL5_cds_5_0_chr1_93303013_f 0 +
chr1 93306107 93306196 RPL5_cds_6_0_chr1_93306108_f 0 +
chr1 93307322 93307422 RPL5_cds_7_0_chr1_93307323_f 0 +
chrX 153295817 153296901 MECP2_cds_0_0_chrX_153295818_r 0 -
chrX 153297657 153298008 MECP2_cds_1_0_chrX_153297658_r 0 -
chrX 153357641 153357667 MECP2_cds_2_0_chrX_153357642_r 0 -
desired output tab-delimited
chr1 17345304 17345315 SDHB intron
chr1 17345516 17345524 SDHB intron
chr1 93306242 93306261 RPL5 intron
chr1 93307262 93307291 RPL5 intron
chrX 153295819 153296875 MECP2 exon
chrX 153295810 153296800 MECP2 splicing
awk
awk '
FNR==NR{
a[$4];
min[$4]=$2;
max[$4]=$3;
next
}
{
split($4,array,"_");
print $0,(array[1] in a) && ($2>=min[array[1]] &&
$2<=max[array[1]])?"exon":"intron"
}' file1 OFS="\t" file2 > output
example of line 1
a[$4] = SDHB
min[$4] = 17345304
max[$4] = 17345315
array[1] = SDHB, 17345304 >= 17345375 && array[1] = SDHB, 17345315 <= 17345453 ---- intron
I have a tab separated text file and want to do some math operation on one column and make a new tab separated text file.
this is an example of my file:
chr1 144520803 144520804 12 chr1 144520813 58
chr1 144520840 144520841 12 chr1 144520845 36
chr1 144520840 144520841 12 chr1 144520845 36
chr1 144520848 144520849 14 chr1 144520851 32
chr1 144520848 144520849 14 chr1 144520851 32
i want to change the 4th column. in fact I want to divide every single element in the 4th column by sum of all elements in the 4th column and then multiply by 1000000 . like the expected output.
expected output:
chr1 144520803 144520804 187500 chr1 144520813 58
chr1 144520840 144520841 187500 chr1 144520845 36
chr1 144520840 144520841 187500 chr1 144520845 36
chr1 144520848 144520849 218750 chr1 144520851 32
chr1 144520848 144520849 218750 chr1 144520851 32
I am trying to do that in awk using the following command but it does not return what I want. do you know how to fix it:
awk '{print $1 "\t" $2 "\t" $3 "\t" $4/{sum+=$4}*1000000 "\t" $5 "\t" $6 "\t" $7}' myfile.txt > new_file.txt
you need two passes, one to compute the sum and then to scale the field
something like this
$ awk -v OFS='\t' 'NR==FNR {sum+=$4; next}
{$4*=(1000000/sum)}1' file{,} > newfile
The below awk will produce the tab-delimeted file1 with the difference between $3-$2 calulated for each line and printed in $6. Before the awk is executed only 5 fields exist.
What I am having trouble with updated each $2 value in file2 with the $7 value of file1 if the $1 value of file2 matches the $5 of file1 and $6 in file1 is not intron. If the value of $5 is intron then then the value of $7 in file1 is zero. So for example line 1 in file1 is intron so that is equvilant to zero or skipped (those lines are not needed in the calculation).
It is possible that a $1 value in file2 may not exist in file1 and in this case the value of $2 in file2 is zero. Line3 infile2 is an example and is set to zero because it does not exist in file1. Thank you:).
Awk w/ output
awk '
FNR==NR{ # process same line
b[$4]=$3-$2;
next # process next line
}
{
a[$5]+=($3-$2)
}
{
split($1, b, " "); print b[0], a[b[0]]
}' OFS="\t" file1 file2
Output
-2135
-2135
-2222
-2351
-2351
-2414
File1 tab-delimited
chr5 86667863 86667879 RASA1 intron 16
chr5 86669977 86669995 RASA1 splicing 18
chr5 86670703 86670805 RASA1 exon 102
chr5 86679453 86679547 RASA1 intron 94
chr5 86679571 86679673 RASA1 exon 102
chr19 15088950 15088961 NOTCH2 intron 50
chr19 15288950 15288961 NOTCH3 intron 11
chr19 15308240 15308275 NOTCH3 exon 35
File2 space delimited
RASA1 2135
NOTCH2 0
GIMAP8 87
NOTCH3 129
FOXF2 0
PRB3 63
Desired out after file2 is updated
RASA1 222 `(102+102+18)`
NOTCH2 0
GIMAP8 0
NOTCH3 35 `(35)`
FOXF2 0
PRB3 0
Maybe adding a | after the first awk with:
awk 'FNR==NR { a[$1]=$7; next } { if(a[$5]){$1=a[$5] }; print }'
To update file2
Could you please try following. It will provide you sequence of output in same order of Input_file's order.
awk '
FNR==NR{
if(!b[$1]++){
c[++count]=$1
}
a[$1]
next
}
($4 in a) && $5!="intron"{
a[$4]+=$NF
}
END{
for(i=1;i<=count;i++){
print c[i],a[c[i]]?a[c[i]]:0
}
}' Input_file2 Input_file1
Since your Input_file1 is NOT TAB delimited as per your claim, so in case it is then edit Input_file2 Input_file1 -----> Input_file2 FS="\t" Input_file1. To get output as TAB delimited either append above code's output to | column -t command or set OFS="\t" near to FS="\t" too.
Output will be as follows.
RASA1 222
NOTCH2 0
GIMAP8 0
NOTCH3 35
FOXF2 0
PRB3 0
if I understood it correctly, this should do what you expect
$ awk 'FNR==NR && $5!="intron" {a[$4]+=$3-$2; next}
{$2=($1 in a)?a[$1]:0}1' file1 file2 > file2.updated
Input: A tab-separated input file with 15 columns where column 15 is an integer.
Output: The number of lines that satisfy the conditional.
My code:
$ closest-features --closest --no-overlaps --delim '\t' --dist --ec megatrans_enhancers.sorted.bed ../../data/alu_repeats.sorted.bed | awk -v OFS='\t' '{if ($15 <= 1000 || $15 >= -1000) print $0}' | wc -l
1188
The || conditional in this case is failing to work (the total number of lines in the file are 1188 and I know for certain at least some lines do not satisfy the condition), because if I remove the OR conditional then suddenly it works:
$ closest-features --closest --no-overlaps --delim '\t' --dist --ec megatrans_enhancers.sorted.bed ../../data/alu_repeats.sorted.bed | awk -v OFS='\t' '{if ($15 <= 1000) print $0}' | wc -l
926
Not sure what i'm doing wrong. Any advice?
Example Input to Awk command:
chr1 378268 378486 chr1-798_Enhancer 17.2 + chr1 375923 376219 AluY|SINE|Alu-HOMER529 0 + E:375923 0.044 -2050
chr1 1079471 1079689 chr1-929_Enhancer 14.6 - chr1 1071271 1071563 AluSx1|SINE|Alu-HOMER1669 0 - E:1071271 0.13 -7909
chr1 1080259 1080477 chr1-830_Enhancer 16.7 - chr1 1071271 1071563 AluSx1|SINE|Alu-HOMER1669 0 - E:1071271 0.13 -8697
chr1 6611744 6611962 chr1-241_Enhancer 46.6 + chr1 6611431 6611723 AluSc|SINE|Alu-HOMER10257 0 + E:6611431 0.089 -22
chr1 6959639 6959857 chr1-58_Enhancer 100.1 - chr1 6966612 6966911 AluSx|SINE|Alu-HOMER11041 0 - E:6966612 0.137 6756
chr1 6960593 6960811 chr1-202_Enhancer 51.6 - chr1 6966612 6966911 AluSx|SINE|Alu-HOMER11041 0 - E:6966612 0.137 5802
chr1 7447888 7448106 chr1-2_Enhancer 181.9 - chr1 7449489 7449799 AluSz|SINE|Alu-HOMER11879 0 + E:7449489 0.119 1384
chr1 10752461 10752679 chr1-131_Enhancer 65.4 - chr1 10752754 10753065 AluSq2|SINE|Alu-HOMER19455 0 + E:10752754 0.106 76
chr1 12485694 12485912 chr1-353_Enhancer 36.7 + chr1 12487328 12487634 AluSx3|SINE|Alu-HOMER23581 0 + E:12487328 0.085 1417
chr1 12486469 12486687 chr1-141_Enhancer 63.6 + chr1 12487328 12487634 AluSx3|SINE|Alu-HOMER23581 0 + E:12487328 0.085 642
Try to put && condition because a digit should be greater than -1000 and lesser than 1000.
Your_command | awk '$15<=1000 && $15>=-1000{count++} END{print count}'
Add -F"\t" in above awk in case your Input to it is coming TAB delimited too. Also there is no need to use wc -l after awk. I have written logic for that so give the count of lines which are satisfying the condition by creating a variable named count and printing it at very last of Input_file.
Also for your provided samples output is coming as 3 which I believe is correct one.
I am trying to find the differences between file1.txt and file2.txt and output the differences. I tried diff and sed and the output does not return any differences. I also tried awk and matching on $2, but I think the syntax is wrong as a file gets created but it is 0kb. The actual data I am using is quite large but I know there should be 18 differences. Thank you :).
awk 'NR==FNR{a[$2]++;next} !($2 in a){print $2}' file1.txt file2.txt > diff.txt
file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6
Desired output
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6
Diff result (since these are the two records that are not in both files)
1,52058c1,52040
< chr1 955542 955763
< chr1 957570 957852
< chr1 976034 976270
I'm curious while diff isn't working like you want, but your awk logic isn't correct:
You're checking the second field's (delimited by spaces) value only. In your example the second field is all identical so nothing is being printed out. Using the whole line instead works as expected:
Using your example text where all is different:
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852 + AGRN:exon.2
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
Here's with the second line identical just to show it working in a more obvious way.
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
EDIT
Based on a comment stating:
"There should be 18 differences out of 52,000 lines. File1.txt is 52,058 entries and file2.txt has 52,040 entries in it. I am trying to find out what the 18 are"
Given you said file1 has more lines, you need to process file2 first. The first file read is populating the array and then the second is checking for lines existing in that array. You need to process the smaller file first so that the additional lines you're interested in aren't in the array. It'd be the same logic above, just with the file order switched, e.g.:
$ cat file1.txt
chr1 955542 955763
chr1 957570 957852
chr1 976034 976270
New Line!
Not in file2!
$ cat file2.txt
chr1 955542 955763 + AGRN:exon.1
chr1 957570 957852
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file2.txt file1.txt > diff.txt
$ cat diff.txt
chr1 955542 955763
chr1 976034 976270
New Line!
Not in file2!
$ awk 'NR==FNR{a[$0]++;next} !($0 in a){print $0}' file1.txt file2.txt > diff.txt
$ cat diff.txt
chr1 955542 955763 + AGRN:exon.1
chr1 976034 976270 + AGRN:exon.2;AGRN:exon.3;AGRN:exon.4
Note that reading file1 first doesn't emit the additional lines.
If you don't care about the additional text on the lines, just the text in the second field, then you could use $2 as you originally did.
$ awk 'NR==FNR{a[$2];next} !($2 in a)' file1 file2
chr1 976542 976787 + AGRN:exon.3;AGRN:exon.5
chr1 976847 977092 + AGRN:exon.6