What is happening to my SQL connection in a for construct? - sql

Question

I'm working on a program that converts the USDA nutrition database files from ASCII into an sqlite database. Information about those files can be found at http://www.ars.usda.gov/Services/docs.htm?docid=8964 and amounts to something really awesome.
However, in the code below, I have my sql connection "disappearing" as soon as I enter a for construct in the function that does conversions.
(defn sanitize-field [field]
(cond
(= field "~~") ""
(= field "") 0
(and (.startsWith field "~") (.endsWith field "~"))
(.substring field 1 (- (.length field) 1))
:default (try
(Double. field)
(catch NumberFormatException n 0.0))))
(defn field-separator [line]
(.split line "\\^"))
(defn parse-file [file]
(map (fn [l] (map sanitize-field (field-separator l)))
(.split (slurp (.getCanonicalPath file)) "\r\n")))
(defn convert-food-description [source-dir]
(println (sql/connection))
(for [entry (parse-file (File. source-dir "FOOD_DES.txt"))]
(do
(println entry)
(println (sql/connection))
(sql/insert-rows "food_des" entry))))
(Class/forName "org.sqlite.JDBC")
(def db {:classname "com.sqlite.JDBC" :subprotocol "sqlite" :subname "nutrition.sqlite"})
(defn convert []
(sql/with-connection db
(convert-food-description (File. "/home/savanni/Downloads/nutrient_database"))))
When I run the convert operation, here is what I get:
user=> (convert)
(convert)
#<Conn org.sqlite.Conn#180e426>
((01001 0100 Butter, salted BUTTER,WITH SALT Y 0.0 6.38 4.27 8.79 3.87)
java.lang.Exception: no current database connection
So, my only println commands are in convert-food-description. It looks like the first one right after the function begins executes fine, printing out the connection that I created with the with-connection statement. But then the for loop begins, I print out the first line of data that I'm going to insert into the database, and then it throws an exception when trying to print the SQL connection again.
What could be causing the connection to disappear there? Is something else entirely happening?
UPDATE: the sql/ functions here all come from clojure.contrib.sql

Answer 1

for is lazy. Your connection handle is being opened and closed by with-connection before for in convert-food-description evaluates.
You have to force evaluation while the connection handle is still alive. Try changing for to doseq, for example.
user> (require '(clojure.contrib [sql :as sql]))
nil
user> (import 'org.sqlite.JDBC)
org.sqlite.JDBC
user> (def db {:classname "com.sqlite.JDBC"
:subprotocol "sqlite"
:subname "foo.sqlite"})
#'user/db
user> (sql/with-connection db (sql/create-table "foo" ["val" "int"]))
(0)
;; Bad
user> (defn foo []
(sql/with-connection db
(for [x (range 10)]
(sql/insert-records "foo" {:val x}))))
#'user/foo
user> (foo)
; Evaluation aborted.
;;java.lang.Exception: no current database connection
;; Good
user> (defn foo []
(sql/with-connection db
(doseq [x (range 10)]
(sql/insert-rows "foo" [x]))))
#'user/foo
user> (foo)
nil

Answer 2

First of all, your (println (sql/connection)) is not printing your current connection. Instead it creates new one and prints it.
Second, you have defined db as (def db {:classname "com.sqlite.JDBC" :subprotocol "sqlite" :subname "nutrition.sqlite"})
This doesn't open connection, this is just a connection description.
Third, you should pass db as a parameter to convert-food-description method, and use that for actual queries.