How to replace \n with a comma for certain lines on CLI - awk

Question

I have the following text file.
# This is a test 1
"watch"
"autoconf"
# This is another line 2
"coreutils"
"binutils"
"screen"
# This is another line 3
"bash"
"emacs"
"nano"
"bison"
# This is another line 4
"libressl"
"python"
"rsync"
"unzip"
"vim"
I want to change this to the following:
# This is a test 1
watch, autoconf
# This is another line 2
coreutils, binutils, screen
# This is another line 3
bash, emacs, nano, bison
# This is another line 4
libressl, python, rsync, unzip, vim
Remove the leading white spaces, remove quotes, replace a new line with a comma.
So far I got this:
$ cat in.txt | sed 's/"//g' | sed 's/^[[:space:]]*//'> out.txt
# This is a test 1
watch
autoconf
# This is another line 2
coreutils
binutils
screen
# This is another line 3
bash
emacs
nano
bison
...
I'm not sure how to replace a new line with a comma. I tried the following.
# no change
$ cat in.txt | sed 's/"//g' | sed 's/^[[:space:]]*//'| sed 's/\n/,/g'> out.txt
# changed all new lines
$ cat in.txt | sed 's/"//g' | sed 's/^[[:space:]]*//'| sed -z 's/\n/,/g'> out.txt
$ cat out.txt
# This is a test 1,watch,autoconf,,# This is another line 2,coreutils,binutils,screen,,# This is another line 3,bash,emacs,nano,bison,,# This is another line 4,libressl,python,rsync,unzip,vim
How can I achieve this?

Answer 1

This might work for you (GNU sed):
sed -E 's/^\s*//;/^"(\S*)"/{s//\1/;H;$!d};x;s/.//;s/\n/, /gp;$d;z;x' file
Strip off white space at the front of all lines.
Strip out double quotes and append those words to the hold space.
Otherwise, switch to the hold space, delete the first introduced newline, replace all other newlines by , , print the result and then switch back to the pattern space and print that.

Answer 2

Here's an awk version. Notice that we set the record separator RS to the empty string. This tells awk to treat each block separated by an empty line as a single record. Then by setting the field separator with -F to a newline, each line in the block becomes a single field in that record.
Then it's just a matter of brute-forcing our way through the fields of each record, using sub or gsub to remove leading spaces and quotation marks, and using printf to avoid a newline when we don't want one and printing a comma instead.
$ awk -v RS="" -F'\n' '{
sub(/^[[:space:]]*/, "", $1);
print $1;
sep="";
for (i=2; i<=NF; ++i) {
gsub(/[[:space:]]*"/, "", $i);
printf "%s%s", sep, $i;
sep=", "
}
print "\n"
}' file
Output:
# This is a test 1
watch, autoconf
# This is another line 2
coreutils, binutils, screen
# This is another line 3
bash, emacs, nano, bison
# This is another line 4
libressl, python, rsync, unzip, vim

Answer 3

A one-liner using GNU sed:
sed -Ez 's/\n[[:blank:]]*"?/\n/g; s/"\n([^\n])/, \1/g; s/"//g' file
or, using multiline techniques with standard sed:
sed '
s/^[[:blank:]]*//
/^".*"$/{
s/.//
s/.$//
:a
$b
N
s/\n[[:blank:]]*"\(.*\)"$/, \1/
ta
}' file

Answer 4

With your shown samples only, could you please try following. Written and tested in GNU awk.
awk '
BEGIN{
OFS=", "
}
NF{
gsub(/"|^ +| +$/,"")
}
/^#/ || !NF{
if(value){
print first ORS value
}
first=$0
value=""
if(!NF){ print }
next
}
{
value=(value?value OFS:"")$0
}
END{
if(value){
print first ORS value
}
}
' Input_file
Explanation: Adding detailed explanation for above.
awk ' ##Starting awk program from here.
BEGIN{ ##Starting BEGIN section of this program from here.
OFS=", " ##Setting OFS as comma space here.
}
NF{ ##Checking condition if line is NOT empty do following.
gsub(/"|^ +| +$/,"") ##Globally substituting " OR starting/ending spaces with NULL here.
}
/^#/ || !NF{ ##Checking condition if line starts from # OR line is NULL then do following.
if(value){ ##Checking condition if value is NOT NULL then do following.
print first ORS value ##Printing first ORS value values here.
}
first=$0 ##Setting first to current line here.
value="" ##Nullifying value here.
if(!NF){ print } ##Checking condition if line is empty then simply print it.
next ##next will skip all further statements from here.
}
{
value=(value?value OFS:"")$0 ##Creating value here and keep on adding current line value to it.
}
END{ ##Starting END block of this program from here.
if(value){ ##Checking condition if value is NOT NULL then do following.
print first ORS value ##Printing first ORS value values here.
}
}
' Input_file ##Mentioning Input_file name here.

Answer 5

Using any POSIX awk in any shell on every Unix box:
$ awk -v RS= -v ORS='\n\n' -F'[[:blank:]]*\n[[:blank:]]*' -v OFS=', ' '{
gsub(/^[[:blank:]]*|"/,"")
printf "%s\n", $1
for (i=2;i<=NF;i++) {
printf "%s%s", $i, (i<NF ? OFS : ORS)
}
}' file
# This is a test 1
watch, autoconf
# This is another line 2
coreutils, binutils, screen
# This is another line 3
bash, emacs, nano, bison
# This is another line 4
libressl, python, rsync, unzip, vim