Find duplicates and give sum of values in column next to it (UNIX) (with solution -> need faster way) - sum

Question

I am writing a script for bioinformatical use. I have a file with 2 columns, in which column A shows a number and column B a specific string. I need a script that will search the file for the string in column B, IF any duplicates are found the number in column A should all be added up, duplicates should be removed and only one line with column A having the sum and column B the string should remain.
I have written something that does exactly that, but because I am not really a programmer I am sure there is a much faster way. My files contain sometimes 500k lines and my code takes to long for such files. Please have a look at it and see what I could change to speed things up. Also I can't use uniq because for this Id have to also use sort but the order of the lines have to stay the way they are!
13 ABCD
15 BGDA
12 ABCD
10 BGDA
10 KLMN
17 BGDA
should become
25 ABCD
42 BGDA
10 KLMN
This does it but for a file with 500k lines it takes too long:
for AASEQUENCE in file.txt;
do
#see how many lines the file has and save that number in $LN
LN="$(wc -l $AASEQUENCE | cut -d " " -f 1)"
for ((i=1;i<=${LN};i++));
do
#Create a variable that will have just the string from column B
#save it in $STRING
STRING="$(cut -f2 $AASEQUENCE | head -n $i| tail -n 1 | cut -f1)";
#create $UNIQ: a variable that will have number+string of that
#line. This will be used in the ELSE-statement, IF there are no
#duplicates of the string, it will just be added to the
# output file without further processing
UNIQ="$(head -n $i $AASEQUENCE | tail -n 1)"
for DUPLICATE in $AASEQUENCE;
do
#create variable that will display the number of lines
#of duplicates. IF its 1 the IF-statement will jump to the ELSE
#part as there are no duplicates
VAR="$(grep -w "${STRING}" $DUPLICATE | wc -l)"
#Now add up all the numbers from column A that have $STRING in
#column B
TOTALCOUNT="$(grep -w "${STRING}" $DUPLICATE | cut -f1 | awk
'{SUM += $1} END {print SUM}')"
#Create a file that the single line can be put into it
touch MERGED_`basename $AASEQUENCE`
#The IF-statement checks if the AA occurs more than once
#If it does a second IF-statement checks if this AA-sequence has
#already been added.
#If it hasnt been added, it will be, if not nothing happens.
ALREADYMATCHED="$(grep -w "${STRING}" MERGED_`basename
$AASEQUENCE` | wc -l)"
if [[ "$VAR" > 1 ]];
then if [[ "$ALREADYMATCHED" != 0 ]]; then paste <(echo
"$TOTALCOUNT") <(echo "$STRING") --delimiters ' '>>
MERGED_`basename $AASEQUENCE` ;fi;
else echo $UNIQ >> MERGED_`basename $AASEQUENCE` ;fi
done;
done;
done;
P.S: When I have fileA.txt fileB.txt ... and use file* the loop still always stops after the first file. Any suggestions why?

Answer 1

maybe pure awk solution?
$ cat > in
13 ABCD
15 BGDA
12 ABCD
10 BGDA
10 KLMN
17 BGDA
$ awk '{dc[$2] += $1} END{for (seq in dc) {print dc[seq], seq}}' in
25 ABCD
42 BGDA
10 KLMN
$