SQL: how to fix these errors? - sql

Question

So I have to loop through a folder of .dat files, extract the data and use INSERT INTO to insert the data into a database.
Here is a pastebin of one of the files to see the data I am working with:
http://pastebin.com/dn4wQjjE
To run the script I just call:
populate_database.sh directoryWithDatFiles
And the contents of the populate_database.sh script:
rm test.sql;
sqlite3 test.sql "CREATE TABLE HotelReviews (HotelID SMALLINT, ReviewID SMALLINT, Author CHAR, Content CHAR, Date CHAR, Readers SMALLINT, HelpfulReviews SMALLINT, Over$
IFS=$'\n'
for file in $1/*;
do
author=($(grep "<Author>" $file | sed 's/<Author>//g'));
content=($(grep "<Content>" $file | sed 's/<Content>//g'));
date=($(grep "<Date>" $file | sed 's/<Date>//g'));
readers=($(grep "<No. Reader>" $file | sed 's/<No. Reader>//g'));
helpful=($(grep "<No. Helpful>" $file | sed 's/<No. Helpful>//g'));
overall=($(grep "<Overall>" $file | sed 's/<Overall>//g'));
value=($(grep "<Values>" $file | sed 's/<Value>//g'));
rooms=($(grep "<Room>" $file | sed 's/<Room>//g'));
location=($(grep "<Location>" $file | sed 's/<Location>//g'));
cleanliness=($(grep "<Cleanliness>" $file | sed 's/<Cleanliness>//g'));
receptionarea=($(grep "<Check in / front desk>" $file | sed 's/<Check in \/ front desk>//g'));
service=($(grep "<Service>" $file | sed 's/<Service>//g'));
businessservice=($(grep "<Business service>" $file | sed 's/<Business service>//g'));
length=${#author[#]}
hotelID="$(echo $file | sed 's/.dat//g' | sed 's/[^0-9]*//g')";
for((i = 0; i < length; i++)); do
sqlite3 test.sql "INSERT INTO HotelReviews VALUES($hotelID, $i, 'author', 'content', 'date', ${readers[i]}, ${helpful[i]}, ${overall[i]}, 9, 10, ${location[i]}, ${cleanliness[i]}, ${receptionarea[i]}, ${service[i]}, ${businessservice[i]})";
done
done
sqlite3 test.sql "SELECT * FROM HotelReviews;"
The problem I have though, is that although much of the script is working, there are still 5 of the 15 columns that I can't get working. I'll just screenshot the errors I get when trying to change the code from:
'author' --> ${author[i]}: http://i.imgur.com/zKQLSqT.jpg
'content' --> ${content[i]}: http://i.imgur.com/pnirIo3.jpg
'date' --> ${date[i]}: http://i.imgur.com/urF5DTa.jpg
9 --> ${value[i]}: http://i.imgur.com/AnBFSWp.jpg
10 --> ${rooms[i]}: same errors as above
Anyway, if anyone could help me out on this, I'd be massively grateful.
Cheers!

Answer 1

If you deal with a lot of XML, I recommend getting to know a SAX parser, such as the one in the Python standard library. Anyone willing to write a shell script like that has the chops to learn it, and the result will be easier to read and at least have a prayer at being correct.
If you want to stick with regex hacking, turn to awk. Using ">" as your field separator, your script could be simplified with awk lines like
/<Author>/ { gsub(/'/, "''", $2); author=$2 }
/<Content>/ { gsub(/'/, "''", $2); content=$2 }
...
END { print author, content, ... }
The gsub takes care of your SQL quoting problem by doubling any single quotes in the data.