AWK ouput to separated file - awk

Question

I am trying to split my file based on the patterns that it contains and store them as separate text files with that pattern.
I am using AWK to split my lines by two different separators ("\t" "_") and use $2 as file name and append that line ($0) into related file that is generated.
My command is:
awk 'BEGIN {FS="[_ \t]"}; {fn=$2"_HP100.txt"}; {print $0 >fn}' my_file.txt
Awk can generate text file named after my $2, and also it can give $0 as whole line when I printed separately. But My generated text files are always empty... Am I missing something?
My text file contains:
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
Expected output is CGTCGCCAAACTTAAC_HP100.txt file that contains all the lines with _CGTCGCCAAACTTAAC tag.

Answer 1

You can just use this awk command:
awk -F '[\t_]' '{print $0 >> ($2 "_HP100.txt")}' file

Answer 2

With your shown samples and attempts please try following awk code. Its a combination of awk + sort + cut + awk solution, this will help us in case you have multiple lines where 2nd fields are same and you want to save all same $2 lines in same output file. This is sorting them from $2(by making 2nd field as 1st field to sort it and then using only required/actual portion of input only).
awk -F'_|\t' '{print $2,$0}' Input_file | sort | cut -d' ' -f2- |
awk -F'_|\t' '
prev!=$2{
close(outputFile)
outputFile=$2"_HP100.txt"
}
{
print $0 > (outputFile)
prev=$2
}
'
NOTE: Pass your Input_file to first used awk in above code. Also this code is good for small files.

Answer 3

The approach I have in mind to deal with the 50GB issue is to collect a certain amount each round, then when the input arrays reach a fixed threshold, output the rows according to their file names
instead of nonstop concat-ing into a very very long string - the current approach uses one array for the rows themselves, and one more for mapping filenames to row #s (aka NR) :
collect until reaching threshold (i didn't set one that's up to u)
dump out all data in arrays to files
— also allows for re-creation of original input row order
use print (. . .) >> (fn) instead of >, just in case
use split("", arr) to reset both arrays to a mint state
— it's cleaner and more portable than delete arr statement
host system resource friendly
minimizes concurrent resource usage on the host system by only requiring one single instance of awk
the threshold you set would also constrain RAM usage at any moment to a sane level
avoids the need to use shell loops, xargs, or gnu-parallel
avoids nonstop close(fn)
avoids wasting time performing any amount of data sort-ing
mawk '1;1;1;1;1;1;1' raw_input.txt | shuf |
mawk2 '
BEGIN { OFS = "_HP100.txt"
FS = "^[^_]+_|[ \t].+$"
} { ___[ NR ] = $_
sub(OFS, _, $!(NF = NF))
sub( "$", "=" NR,____[$_])
} END {
OFS = "\f\r\t\t"
for(_ in ___) { print _, ___[_] }
for(_ in ____) { print _,____[_] } } '
1
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
2
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
3
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
4
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
5
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
6
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
7
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
8
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
9
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
10
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
11
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
12
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
13
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
14
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
15
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
16
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
17
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
18
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
19
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
20
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
21
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
22
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
23
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
24
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
25
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
26
K00295:342:HNYVTBBXY:5:1101:1955:1578_AGCGTCTTTCATGCTG 0 9 41603892 1 50M
27
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
28
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
29
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
30
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
31
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
32
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
33
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
34
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
35
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
36
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
37
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
38
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
39
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
40
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
41
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
42
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
43
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
44
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
45
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
46
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
47
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
48
K00295:342:HNYVTBBXY:5:1101:1976:1578_TCATACCAAGTCTCCG 16 9 113429747 42 49M
49
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
50
K00295:342:HNYVTBBXY:5:1101:1773:1578_CGTCGCCATCGCTAGG 0 12 115297976 24 51M
51
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
52
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
53
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
54
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
55
K00295:342:HNYVTBBXY:5:1101:2057:1578_GATGCGTTTTCTGGTT 0 14 68520409 42 50M
56
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
57
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
58
K00295:342:HNYVTBBXY:5:1101:2098:1578_CGTCGCCAAACTTAAC 0 8 94503004 42 50M
59
K00295:342:HNYVTBBXY:5:1101:1996:1578_TCATCGAACCTCGTTG 16 20 21594558 42 51M
60
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
61
K00295:342:HNYVTBBXY:5:1101:1935:1578_GCCTATTCCCTCGTTG 16 18 54707729 42 51M
62
K00295:342:HNYVTBBXY:5:1101:2016:1578_TGGATCAACAGGACCA 0 16 13244975 27 51M
63
K00295:342:HNYVTBBXY:5:1101:1834:1578_TCGAACGACCGTTGCG 16 2 22709262 42 50M
TCATACCAAGTCTCCG_HP100.txt
=5=12=15=23=24=40=48
CGTCGCCATCGCTAGG_HP100.txt
=29=30=34=38=44=46=50
TCATCGAACCTCGTTG_HP100.txt
=9=25=27=31=36=49=59
AGCGTCTTTCATGCTG_HP100.txt
=3=8=14=17=18=22=26
GCCTATTCCCTCGTTG_HP100.txt
=11=37=52=54=56=57=61
GATGCGTTTTCTGGTT_HP100.txt
=6=7=10=20=39=41=55
TCGAACGACCGTTGCG_HP100.txt
=16=19=32=42=43=60=63
TGGATCAACAGGACCA_HP100.txt
=1=13=33=35=45=47=62
CGTCGCCAAACTTAAC_HP100.txt
=2=4=21=28=51=53=58