parallelize awk script - files splitting - awk

Question

I have a small awk script which takes input from a stream and writes to the appropriate file based on the second column value. Here is how it goes:
cat mydir/*.csv | awk -F, '{if(NF==29)print $0 >> "output/"$2".csv"}'
How do I parallelize it, so that it can use multiple cores available in the machine? Right now, this is running on a single core.

Answer 1

you can try this.
I execute 1 awk per source file. Put content in temporary file (in each process it is a series of different one to avoid conflict in same final file and/or too much open/close handle on it). At the end of the awk, it put the content of temporary file into final one and remove temporary
you maybe have to use a batch limiter (a sleep or more smart grouping) if there are lot of file to treat to avoid to kill the machine with too much subprocess concurrent.
rm output/*.csv
for File in mydir/*.csv
do
# shell sub process
{
# ref for a series of temporary file
FileRef="${File##*/}"
awk -F ',' -v FR="${FileRef}" '
NF == 29 {
# put info in temporary file
ListFiles [ OutTemp = "output/"$2".csv_" FR ] = "output/"$2".csv"
print > OutTemp}
END {
# put temporary content into final file
for ( TempFile in ListFiles ) {
Command = sprintf( "cat \042%s\042 >> \042%s\042; rm \042%s\042" \
, TempFile, ListFiles[TempFile], TempFile )
printf "" | Command
}
' File
} &
done
wait
echo ls -l output/*.csv

Answer 2

Untested:
do_one() {
# Make a workdir only used by this process to ensure no files are added to in parallel
mkdir -p $1
cd $1
cat ../"$2" | awk -F, '{if(NF==29)print $0 >> $2".csv"}'
}
export -f do_one
parallel do_one workdir-{%} {} ::: mydir/*.csv
ls workdir-*/ | sort -u |
parallel 'cat workdir*/{} > output/{}'
rm -rf workdir-*
If you want to avoid the extra cat you can use this instead, though I find the cat version easier to read (performance is normally the same on modern systems http://oletange.blogspot.com/2013/10/useless-use-of-cat.html):
do_one() {
# Make a workdir only used by this process to ensure no files are added to in parallel
mkdir -p $1
cd $1
awk -F, <../"$2" '{if(NF==29)print $0 >> $2".csv"}'
}
export -f do_one
parallel do_one workdir-{%} {} ::: mydir/*.csv
ls workdir-*/ | sort -u |
parallel 'cat workdir*/{} > output/{}'
rm -rf workdir-*
But as #Thor writes, you are most likely I/O starved.