How to use awk delimiters in a large csv with text fields with commas [closed] - awk

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
I have a .csv with 470 columns and tens of thousands of rows of products, many with text strings including commas, that cause my awk statements to blow out and write to the wrong columns thus corrupting my data. Here are the statements I'm using:
Input example:
LDB0010-300,"TIMELESS DESIGN: Classic, Elegant, Beautiful",Yes,1,Live,...
LDB0010-400,"CLASSIC DESIGN: Contemporary look",No,0,Not Live,...
LDB0010-500,"Everyone should wear this, almost!",Yes,0,Not Live,...
Code:
cat products.csv | sed -e 's/, /#/g' | awk -F, 'NR>1 {$308="LIVE" ; $310="LIVE" ; $467=0 ; print $0}' OFS=, | sed -e 's/#/, /g'
Current output, which is wrong with data written in the wrong columns:
LDB0010-300,"TIMELESS DESIGN: Classic",LIVE, Beautiful",Yes,1,Live,...
LDB0010-400,"CLASSIC DESIGN: Contemporary look",No,0,0,...
LDB0010-500,"Everyone should wear this",LIVE,Yes,0,Not Live,...
When studying the data closer, I noticed that in the cells with text descriptions, commas were always followed with a space, whereas commas used as delimiters had no space after them. So the approach I took was to substitute comma-space with '#', run my awk statement to set the values of those columns, then substitute back from '#' to comma-space. This all looked pretty good until I opened the spreadsheet and noticed that there were many rows with values written into the wrong columns. Does anyone know a better way to do this that will prevent these blow outs?

Answer 1

The sample data you posted does not reproduce the symptoms you report with the code you provided. The absolutely simplest explanation is that your observation that commas with a space are always field-internal and other commas are not is in fact incorrect. This should be easy enough to check;
sed 's/, /#/g' products.csv | awk -F, '{ a[NF]++ } END { for (n in a) print n, a[n] }'
If you don't get a single line of output with exactly the correct number of columns and rows, you can tell that your sed trick is not working correctly. (Notice also the fix for the useless cat.)
Anyway, here is a simple Python refactoring which should hopefully be obvious enough. The Python CSV library knows how to handle quoted fields so it will only split on commas which are outside double quotes.
#!/usr/bin/env python3
import csv
import sys
w = csv.writer(sys.stdout)
for file in sys.argv[1:]:
with open(file, newline='') as inputfile:
r = csv.reader(inputfile)
for row in r:
row[307] = "LIVE"
row[309] = "LIVE"
row[466] = 0
w.writerow(row)
Notice how Python's indexing is zero-based, whereas Awk counts fields starting at one.
You'd run it like
python3 this_script.py products.csv
See also the Python csv module documentation for various options you might want to use.
The above reads all the input files and writes the output to standard output. If you just want to read a single input file and write to a different file, that could be simplified to
#!/usr/bin/env python3
import csv
import sys
with open(sys.argv[1], newline='') as inputfile, open(sys.argv[2], 'w', newline='') as outputfile:
r = csv.reader(inputfile)
w = csv.writer(outputfile)
header = True
for row in r:
if not header: # don't muck with first line
row[307] = "LIVE"
row[309] = "LIVE"
row[466] = 0
w.writerow(row)
header = False
You'd run this as
python3 thisscript.py input.csv output.csv
I absolutely hate specifying the output file as a command-line argument (we should have an option instead) but for a quick one-off, I guess this is acceptable.