Separate lines with keys and store in different files - awk

Question

How to separate (get) the entire line related to hexadecimal number keys and the entire line for DEBUG in a text file, then store in different file, where the key is in this format: "[ uid key]"?
i.e. ignore any lines that is not DEBUG.
in.txt:
[ uid 28fd4583833] DEBUG web.Action
[ uid 39fd5697944] DEBUG test.Action
[ uid 56866969445] DEBUG test2.Action
[ uid 76696944556] INFO test4.Action
[ uid 39fd5697944] DEBUG test7.Action
[ uid 85483e10256] DEBUG testing.Action
The output files are named as "out" + i + ".txt", where i = 1, 2, 3, 4.
i.e.
out1.txt:
[ uid 28fd4583833] DEBUG web.Action
out2.txt:
[ uid 39fd5697944] DEBUG test.Action
[ uid 39fd5697944] DEBUG test7.Action
out3.txt:
[ uid 56866969445] DEBUG test2.Action
out4.txt:
[ uid 85483e10256] DEBUG testing.Action
I tried:
awk 'match($0, /uid ([^]]+)/, a) && /DEBUG/ {print > (a[1] ".txt")}' in.txt

Answer 1

If you are willing to change the output file names to include the keys (frankly, this seems more useful that a one-up counter in the names), you can do:
awk '/DEBUG/{print > ("out-" $3 ".txt")}' FS='[][ ]*' in.txt
This will put all lines that match the string DEBUG with key 85483e10256 into the file out-85483e10256.txt, etc.
If you do want to keep the one-up counter, you could do:
awk '/DEBUG/{if( ! a[$3] ) a[$3] = ++counter;
print > ("out" a[$3] ".txt")}' FS='[][ ]*' in.txt
Basically, the idea is to use the regex [][ ]* as the field separator, which matches a string of square brackets or spaces. This way, $1 is the text preceding the initial [, $2 is the string uid, and $3 is the key. This will (should!) correctly get the key for lines that might have slightly different white space. We use an associative array to keep track of which keys have already been seen to keep track of the counter. But it really is cleaner just to use the key in the output file name.

Answer 2

Using GNU sort for -s (to guarantee retaining input line order for every key value) and any awk:
$ sort -sk3,3 in.txt |
awk '$4!="DEBUG"{next} $3!=prev{close(out); out="out"(++i)".txt"; prev=$3} {print > out}'
$ head out*.txt
==> out1.txt <==
[ uid 28fd4583833] DEBUG web.Action
==> out2.txt <==
[ uid 39fd5697944] DEBUG test.Action
[ uid 39fd5697944] DEBUG test7.Action
==> out3.txt <==
[ uid 56866969445] DEBUG test2.Action
==> out5.txt <==
[ uid 85483e10256] DEBUG testing.Action
If you don't have GNU sort then you can apply the DSU (Decorate/Sort/Undecorate) idiom using any sort:
$ awk -v OFS='\t' '{print NR, $0}' in.txt | sort -k4,4 -k1,1n | cut -f2- |
awk '$4!="DEBUG"{next} $3!=prev{close(out); out="out"(++i)".txt"; prev=$3} {print > out}'
Note that with the above only sort has to handle all of the input in memory and it's designed to use demand paging, etc. to handle extremely large amounts of input, while awk only processes 1 line at a time and keeps almost nothing in memory and only has 1 output file open at a time and so the above is far more likely to succeed for large files than an approach that stores a lot in memory in awk, or has many output files open concurrently.

Answer 3

If your file format is consistent as you show, you can just do:
awk '
$4!="DEBUG" { next }
!f[$3] { f[$3]=++i }
{ print > ("out" f[$3] ".txt") }
' in.txt

Answer 4

1st solution: Using GNU awk try following single awk code. Where I am using PROCINFO["sorted_in"] method of GNU awk.
awk '
BEGIN{
PROCINFO["sorted_in"] = "#ind_num_asc"
}
!/DEBUG/{ next }
match($0,/uid [a-zA-Z0-9]+/){
ind=substr($0,RSTART,RLENGTH)
arr[ind]=(arr[ind]?arr[ind] ORS:"") $0
}
END{
for(i in arr){
outputFile=("out"++count".txt")
print arr[i] > (outputFile)
close(outputFile)
}
}
' Input_file
2nd solution: with any awk, with your shown samples please try following solution. Change Input_file name with your actual file's name here. Using GNU sort here with option -s to maintain the order while sorting values.
awk '
!/DEBUG/{ next }
match($0,/uid [0-9a-zA-Z]+/){
print substr($0,RSTART,RLENGTH)";"$0
}' Input_file |
sort -sk2n |
cut -d';' -f2- |
awk '
match($0,/uid [0-9a-zA-Z]+/){
if(prev!=substr($0,RSTART,RLENGTH)){
count++
close(outputFile)
}
outputFile="out"count".txt"
print > (outputFile)
prev=substr($0,RSTART,RLENGTH)
}
'
1st solution's Explanation: Adding detailed explanation for 1st solution:
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section from here.
PROCINFO["sorted_in"] = "#ind_num_asc" ##Setting PROCINFO["sorted_in"] to #ind_num_asc to sort any array with index.
}
!/DEBUG/{ next } ##If a line does not contain DEBUG then jump to next line.
match($0,/uid [a-zA-Z0-9]+/){ ##using match function to match uid space and alphanumeric values here.
ind=substr($0,RSTART,RLENGTH) ##Creating ind which contains sub string of matched sub string in match function.
arr[ind]=(arr[ind]?arr[ind] ORS:"") $0 ##Creating array arr with index of ind and keep adding current line value to same index.
}
END{ ##Starting END block of this program from here.
for(i in arr){ ##Traversing through array arr here.
outputFile=("out"++count".txt") ##Creating output file name here as per OP requirement.
print arr[i] > (outputFile) ##printing current array element into outputFile variable.
close(outputFile) ##Closing output file in backend to avoid too many files opened error.
}
}
' Input_file ##Mentioning Input_file name here.

Answer 5

A relatively portable awk-based solution with these highlights ::
output rows do not truncate leading edge double space
output filenames adhere to stabilized input row order without the need to pre-sort rows, post-sort rows, or utilize gnu gawk-specific features
tested and confirmed working on
gawk 5.1.1, including -ce flag,
mawk 1.3.4,
mawk 1.9.9.6, and
macOS nawk 20200816
————————————————————————————————
# gawk profile, created Thu May 19 12:10:56 2022
BEGIN {
____ = "test_72297811_" # opt. filename prefix
OFS = FS = "^ [[] uid "
_+=_ = gsub("\\^|[[][]]", _, OFS)
_*= _--
} NF *= / DEBUG / {
print >> (__[___ = substr($NF,_~_,_)] ?__[___]:\
__[___]= ____ "out" length(__) ".txt" )
} END {
for (_ in __) { close(__[_]) } }'
————————————————————————————————
==> test_72297811_out1.txt <==
[ uid 28fd4583833] DEBUG web.Action
==> test_72297811_out2.txt <==
[ uid 39fd5697944] DEBUG test.Action
[ uid 39fd5697944] DEBUG test7.Action
==> test_72297811_out3.txt <==
[ uid 56866969445] DEBUG test2.Action
==> test_72297811_out4.txt <==
[ uid 85483e10256] DEBUG testing.Action