The awk below works perfect, thanks to #RavinderSingh13, if the id in $4 (like in f) is unique. However most of my data is like f1 where the same id can appear multiple times. I think that is the reason why the awk is not working as expected. I added a comment on the line that I can not change without causing the script to abort. Each line in f2 is searched and must contain the id, in this case COL1A2 but that id may not be a single entry. That is the id may appear 5 times, but each line with that is in f2 is searched. Using the $4 in f1 as the id and reading each $1, $2, and $3 value into a variable min and max.
The $4 is then split on the _ in f2 and read into array. The same id from f1 may appear in multiple lines of f2 however each will have unique $2 and $3 values. Each value in the split will match a $4 id in f1. The min and max must match the $1 of f2 and be between the $2 and $3 values in f2. An exact match is not needed rather just that the min or max variables falls within $2 and $3. If that is true then exon is printed in $5 of f2 if it is not true then intron is printed in $5. Most of this works as expected I just did not account for the possibity for multiple enteries and am nut sure how to adjust for it. Thank you :)
For example using the contents of the f1 where COL1A2 appears 3 times, each entry or line is searched in f2. Currently, I believe since COL1A2 is not unique not match is found in f2 as the min and max are not set per entry or line. Thank you :).
awk w/ desired output
awk '
BEGIN{
SUBSEP=","
}
FNR==NR{
max[$1,$NF]=$3
min[$1,$NF]=$2
next
}
{
split($4,array,"_") # How do I change/modify this so it only looks a each line with this id `COL1A2` in it?
}
(($1,array[1]) in max){
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>max[array[5],array[1]] && $3<max[array[5],array[1]])){
print array[5],array[1],min[array[5],array[1]],max[array[5],array[1]],"exon"
next
}
}
{
print $0,"intron"}' f f2
chr7 94024333 94024423 COL1A2_cds_0_0_chr7_94024344_f 0 + intron
chr7 94027049 94027080 COL1A2_cds_1_0_chr7_94027060_f 0 + intron
chr7 COL1A2 94027591 94027701 exon
awk w/ current output
.... }' f1 f2
chr7 94024333 94024423 COL1A2_cds_0_0_chr7_94024344_f 0 + intron
chr7 94027049 94027080 COL1A2_cds_1_0_chr7_94027060_f 0 + intron
chr7 94027683 94027718 COL1A2_cds_2_0_chr7_94027694_f 0 + intron
contents of f single COL1A2 entry
chr7 94027591 94027701 COL1A2
contents of f1 multiple COL1A2 entry, this is most of the actual data, very few are single entries though there are some
chr7 94027591 94027701 COL1A2
chr7 94027799 94027811 COL1A2
chr7 94030799 94030847 COL1A2
contents of f2 always the same format
chr7 94024333 94024423 COL1A2_cds_0_0_chr7_94024344_f 0 +
chr7 94027049 94027080 COL1A2_cds_1_0_chr7_94027060_f 0 +
chr7 94027683 94027718 COL1A2_cds_2_0_chr7_94027694_f 0 +
Related
In the below awk (which executes but produces an empty output) I am using the $4 in file1 as a unique id and reading each $1, $2, and $3 value into a variable chr, min, and max.
The $4 is then split on the _ in file2 and read into array. Each value in the split will match a $4 id in file1 The chr needs to match the $1, the min and max must be between the $2 and $3 values in file2.
An exact match is not needed rather just that the min or max variables falls within $2 and $3. If that is true then exon is printed in $5 of file1, if it is not true then intron is printed in $5.
The desired output has the exon/intron added to it but there is another part where the values in $2 or $3 are adjusted but I am trying to script that before I ask. I am not sure if the below is the best way but hopefully it is a start. Thank you :).
file1 tab delimited except for whitespace after $3 and $4
chr7 94027591 94027701 COL1A2
chr6 31980068 31980074 TNXB
file2 tab delimited
chr7 94027059 94027070 COL1A2_cds_1_0_chr7_94027060_f 0 +
chr7 94027693 94027708 COL1A2_cds_2_0_chr7_94027694_f 0 +
chr6 32009125 32009227 TNXB_cds_0_0_chr6_32009126_r 0 -
chr6 32009547 32009711 TNXB_cds_1_0_chr6_32009548_r 0 -
desired output
chr7 94027683 94027701 COL1A2 exon
chr6 31980068 31980074 TNXB intron
awk w/ comments
awk '
FNR==NR{ open block process matching line in file 1 and file2
a[$4]; # use as a key with unique id
chr[$4]=$1; # store $1 value in chr
min[$4]=$2; # store $2 value in min
max[$4]=$3; # store $3 value in max
next # process next line
} # close block
{ # open block
split($4,array,"_"); # spilt $4 on underscore
print $0,(array[1] in a) && ($2<=min[array[1]] && $2<=max[array[1] && $1=chr[array[1]])?"exon":"intron"
}' file1 OFS="\t" file2 > output # close block, mention input with field separators and output
IMHO, your shown final output is NOT looking correct by logic, since Input_file2 has multiple entries and Input_file1 has only single ones(I am going by samples shown only). Could you please check this one once? If any changes in your output or logic then please do mention them clearly.
awk '
BEGIN{
SUBSEP=","
}
FNR==NR{
max[$1,$NF]=$3
min[$1,$NF]=$2
next
}
{
split($4,array,"_")
}
(($1,array[1]) in max){
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>max[array[5],array[1]] && $3<max[array[5],array[1]])){
print array[5],array[1],min[array[5],array[1]],max[array[5],array[1]],"exon"
next
}
}
{
print $0,"intron"
}' Input_file1 Input_file2 | column -t
What this command is doing it is checking Input_file2's 2nd field OR 3rd field either they are coming in range of Input_file1's 2nd and 3rd field. If anyone of them is coming then I am printing Input_file1's output adding exon in it or else printing Input_file2's output adding intron string at last of it.
i have a tab separated text file like this small example:
chr1 100499714 100499715 1
chr1 100502177 100502178 10
chr1 100502181 100502182 2
chr1 100502191 100502192 18
chr1 100502203 100502204 45
in the new file that I will make:
1- I want to select the rows based on the 4th column meaning in the value of 4th column is at least 10, I will keep the entire row otherwise will be filtered out.
2- in the next step the 4th column will be removed.
the result will look like this:
chr1 100502177 100502178
chr1 100502191 100502192
chr1 100502203 100502204
to get such results I have tried the following code in awk:
cat input.txt | awk '{print $1 "\t" $2 "\t" $3}' > out.txt
but I do not know how to implement the filtering step. do you know how to fix the code?
Just put the condition before output:
cat input.txt | awk '$4 >= 10 {print $1 "\t" $2 "\t" $3}' > out.txt
here is another, might work better if you have many more fields
$ awk '$NF>=10{sub(/\t\w+$/,""); print}' file
In the below awk I am trying to match $2 in file1 up until the ., with $4 in file2 up to the first undescore _. If a match is found then that portion of file2 is up dated with the matching $1 value in file1. I think it is close but not sure how to account for the . in file1. In my real data there are thousands of lines, but they are all in the below format and a match may not always be found. The awk as is does execute but file2 is not updated, I think because the . is not matching. Thank you :).
file 1 space delimited
TGFBR1 NM_004612.3
TGFBR2 NM_003242.5
TGFBR3 NM_003243.4
file 2 tab-delimited
chr1 92149295 92149414 NM_003243_cds_0_0_chr1_92149296_r
chr1 92161228 92161336 NM_003243_cds_1_0_chr1_92161229_r
chr1 92163645 92163687 NM_003243_cds_2_0_chr1_92163646_r
chr3 30648375 30648469 NM_003242_cds_0_0_chr3_30648376_f
chr3 30686238 30686407 NM_003242_cds_1_0_chr3_30686239_f
chr9 101867487 101867584 NM_004612_cds_0_0_chr9_101867488_f
chr9 101904817 101904985 NM_001130916_cds_3_0_chr9_101904818_f
desired output tab-delimited
chr1 92149295 92149414 TGFBR3_cds_0_0_chr1_92149296_r
chr1 92161228 92161336 TGFBR3_cds_1_0_chr1_92161229_r
chr1 92163645 92163687 TGFBR3_cds_2_0_chr1_92163646_r
chr3 30648375 30648469 TGFBR2_cds_0_0_chr3_30648376_f
chr3 30686238 30686407 TGFBR2_cds_1_0_chr3_30686239_f
chr9 101867487 101867584 TGFBR1_cds_0_0_chr9_101867488_f
awk
awk 'FNR==NR {A[$1]=$1; next} $4 in A {sub ($4, $4 "_" A[$4]) }1' OFS='\t' file1 FS='\t' file2
Following awk may help you on same. Also you could change you FS field separator as per your Input_file(s) too, eg--> Input_file1 is space delimited then use FS=" " before it and Input_file2 is TAB delimited then use FS="\t" before its name.
awk '
FNR==NR{
val=$2;
sub(/\..*/,"",val);
a[val]=$1;
next
}
{
split($4,array,"_")
}
((array[1]"_"array[2]) in a){
sub(/.*_cds/,a[array[1]"_"array[2]]"_cds",$4);
print
}
' Input_file1 Input_file2
Output will be as follows:
chr1 92149295 92149414 TGFBR3_cds_0_0_chr1_92149296_r
chr1 92161228 92161336 TGFBR3_cds_1_0_chr1_92161229_r
chr1 92163645 92163687 TGFBR3_cds_2_0_chr1_92163646_r
chr3 30648375 30648469 TGFBR2_cds_0_0_chr3_30648376_f
chr3 30686238 30686407 TGFBR2_cds_1_0_chr3_30686239_f
chr9 101867487 101867584 TGFBR1_cds_0_0_chr9_101867488_f
I have an awk that seemed straight-forward, but I seem to be having a problem. In the file below if $5 starts with a ( then to that string a ) is added at the end. However if$5does not start with a(then nothing is done. The out is separated by a tab. Theawkis almost right but I am not sure how to add the condition to only add a)if the field starts with a(`. Thank you :).
file
chr7 100490775 100491863 chr7:100490775-100491863 ACHE
chr7 100488568 100488719 chr7:100488568-100488719 ACHE;DJ051769
chr1 159174749 159174770 chr1:159174749-159174770 (ACKR1
chr1 159175223 159176240 chr1:159175223-159176240 (ACKR1
awk tried
awk -v OFS='\t' '{print $1,$2,$3,$4,""$5")"}' file
current output
chr7 100490775 100491863 chr7:100490775-100491863 ACHE)
chr7 100488568 100488719 chr7:100488568-100488719 ACHE;DJ051769)
chr1 159174749 159174770 chr1:159174749-159174770 (ACKR1)
chr1 159175223 159176240 chr1:159175223-159176240 (ACKR1)
desired output (line 1 and 2 nothing is done but line 3 and 4 have a ) added to the end)
chr7 100490775 100491863 chr7:100490775-100491863 ACHE
chr7 100488568 100488719 chr7:100488568-100488719 ACHE;DJ051769
chr1 159174749 159174770 chr1:159174749-159174770 (ACKR1)
chr1 159175223 159176240 chr1:159175223-159176240 (ACKR1)
$ awk -v OFS='\t' '{p = substr($5,1,1)=="(" ? ")" : ""; $5=$5 p}1' mp.txt
chr7 100490775 100491863 chr7:100490775-100491863 ACHE
chr7 100488568 100488719 chr7:100488568-100488719 ACHE;DJ051769
chr1 159174749 159174770 chr1:159174749-159174770 (ACKR1)
chr1 159175223 159176240 chr1:159175223-159176240 (ACKR1)
Check the first character of the 5th field. If it is ( append a ) to the end, otherwise append the empty string.
By appending something (where one of the somethings is "nothing" :) in all cases, we force awk to reconstitute the record with the defined (tab) output separator, which saves us from having to print the individual fields. The trailing 1 acts as an always-true pattern whose default action is simply to print the reconstituted line.
I am getting the wrong counts using the awk below. The unique text in $5 before the - is supposed to be counted.
input
chr1 955543 955763 chr1:955543-955763 AGRN-6|gc=75 1 15
chr1 955543 955763 chr1:955543-955763 AGRN-6|gc=75 2 16
chr1 955543 955763 chr1:955543-955763 AGRN-6|gc=75 3 16
chr1 1267394 1268196 chr1:1267394-1268196 TAS1R3-46|gc=68.2 553 567
chr1 1267394 1268196 chr1:1267394-1268196 TAS1R3-46|gc=68.2 554 569
chr1 9781175 9781316 chr1:9781175-9781316 PIK3CD-276|gc=63.1 46 203
chr1 9781175 9781316 chr1:9781175-9781316 PIK3CD-276|gc=63.1 47 206
chr1 9781175 9781316 chr1:9781175-9781316 PIK3CD-276|gc=63.1 48 206
chr1 9781175 9781316 chr1:9781175-9781316 PIK3CD-276|gc=63.1 49 207
current output
1
desired output (AGRN,TAS1R3, PIK3CD) are unique and counted
3
awk
awk -F '[- ]' '!seen[$6]++ {n++} END {print n}' file
Try
awk -F '-| +' '!seen[$6]++ {n++} END {print n}' file
Your problem is that when ' ' (a space) is included as part of a regex to form FS (via -F) it loses its special default-value behavior, and only matches spaces individually as separators.
That is, the default behavior of recognizing runs of whitespace (any mix of spaces and tabs) as a single separator no longer applies.
Thus, [- ] won't do as the field separator, because it recognizes the empty strings between adjacent spaces as empty fields.
You can verify this by printing the field count - based on your intended parsing, you're expecting 9 fields:
$ awk -F '[- ]' '{ print NF }' file
17 # !! 8 extra fields - empty fields
$ awk -F '-| +' '{ print NF }' file
9 # OK, thanks to modified regex
You need alternation -| + to ensure that runs of spaces are treated as a single separator; if tabs should also be matched, use '-|[[:blank:]]+'
Including "-" in FS might be fine in some cases, but in general if the actual field separator is something else (e.g. whitespace, as seems to be the case here, or perhaps a tab), it would be far better to set FS according to the specification of the file format. In any case, it's easy to extract the subfield of interest. In the following, I'll assume the FS is whitespace.
awk '{split($5, a, "-"); if (!(count[a[1]]++)) n++ }
END {print n}'
If you want the details:
awk '{split($5, a, "-"); count[a[1]]++}
END { for(i in count) {print i, count[i]}}'
Output of the second incantation:
AGRN 3
PIK3CD 4
TAS1R3 2