Can't explain results when testing clojure data structures - testing

Question

I'm fairly new to clojure and functional programming generally. Being curious about the speed of some basic operations with data structures (clojures defaults and some I may implement) I wrote something to automate testing operations such as adding to the structures.
My method that runs a test for 3 data structures consistently returns very different average run times depending on how its called even when its inputs remain the same.
Code with tests and results at the bottom
(import '(java.util Date))
(defrecord test-suite ;;holds the test results for 3 datastructures
[t-list
t-vector
t-set]
)
(defrecord test-series ;;holds list of test results (list of test-suite) and the list of functions used in the respective tests
[t-suites
t-functions])
;;;Runs the test, returns time it took
(defn time-test [func init-ds delta-list]
(def startTime (. (new Date) (getTime)))
(reduce func init-ds delta-list)
(def endTime (. (new Date) (getTime)))
(- endTime startTime)
)
;;;Runs the test x number of times returning the average run time
(defn test-struct ([iter func init-ds delta-list] (test-struct iter func init-ds delta-list ()))
([iter ;;number of times to run tests
func ;;function being tested (add remove etc)
init-ds ;;initial datastructure being tested
delta-list
addRes ;;test results
]
(println (first addRes));;print previous run time for debugging
;;test if done recursing
(if (> iter 0)
(test-struct
(- iter 1)
func
init-ds
delta-list
(conj addRes (time-test func init-ds delta-list)))
(/ (reduce + addRes) (count addRes)))
))
;;;Tests a function on a passed in data structure and a randomly generated list of numbers
(defn run-test
[iter ;;the number of times each test will be run
func ;;the function being tested
init-ds] ;;the initial datstructure being tested
(def delta-list (shuffle (range 1000000)));;the random values being added/removed/whatever from the ds
(println init-ds)
(println iter)
(test-suite.
;;the list test
(test-struct iter func (nth init-ds 0) delta-list)
;;the vector test
(test-struct iter func (nth init-ds 1) delta-list)
;;the set test
(test-struct iter func (nth init-ds 2) delta-list)
)
)
;;;Calls run-test a number of times storing the results as a list in a test-series data structure along with the list of functions tested.
(defn run-test-set
([iter func-list ds-list] (run-test-set iter (test-series. nil func-list) func-list ds-list))
([iter ;;the number of times each test is run before being averaged
series ;;data-structure that aggregates the list of test results, and ultimately is returned
func-list ;;the list of functions to be tested
ds-list] ;;the list of initial datastructures to be tested
(if (> (count func-list) 0)
(run-test-set ;;recursively run this aggregateing test-suites as we go
iter
(test-series. ;;create a new test series with all the functions and suites run so far
(conj (:t-suites series) ;;run a test suite and append it to those run so far
(run-test iter (first func-list) (first ds-list)))
(:t-functions series))
(rest func-list)
(rest ds-list)
)
series)) ;;finished with last run return results
)
Tests
All times in ms
;;;;;;;;;;;;;;;;;;EVALUATING 'run-test' directly
;;;get average speeds for adding 100000 random elements to list vector and set
;;;run the test 20 times and average the results
(run-test 20 conj '(() [] #{}))
;;;;;RESULT
#test.test-suite{:t-list 254/5, :t-vector 2249/20, :t-set 28641/20}
or about 51 112 and 1432 for list vector and set respectively
;;;;;;;;;;;;;;;;;;EVALUATING using 'run-test-set' which calls run-test
(run-test-set
20 ;;;times the test is run
'(conj) ;;;just run conj (adding to the ds for now)
'((() [] #{})) ;;;add random values to blank structures
)
;;;;RESULT
#test.test-series{
:t-suites (
#test.test-suite{
:t-list 1297/10,
:t-vector 1297/10,
:t-set 1289/10}) ;;;;;;;;;;;;Result of adding values
:t-functions (conj)}
or about 130 for list vector and set, this is roughly the same rate as vector above
Does anyone know why its returning such different results depending on how its run?
Is this clojure related or possibly an optimization Java is doing?

Answer 1

The right way to test performance of clojure code is criterium. Among other things, criterium reports statistical information about the distribution of times of code execution, and ensures that the jvm hotspot compiler is warmed up before taking the measurements. The jvm hotspot compiler is likely the reason you are seeing these performance discrepencies.
Don't use def inside defn, def is designed for top level global definitions. Use let for bindings that are only used inside one function.
Defining records that are only used once and only exist to hold a few variables is not idiomatic in clojure, the overhead of defining the classes is greater than any benefit they may give you (for increased difficulty understanding the code, if not the performance of your code). Save records for when you need to specialize a protocol or improve performance in a tight loop.
When your priority is human readability of numbers, rather than accuracy, you can use double to coerce to a more readable format for printing.
Here is how one would test the properties you are interested in idiomatically (transcript from a repl session, though this could be run from a -main function as well):
user> (require '[criterium.core :as crit])
nil
user> (def input (shuffle (range 1000000)))
#'user/input
user> (crit/bench (reduce conj [] input))
WARNING: Final GC required 3.501501258094413 % of runtime
WARNING: Final GC required 2.381979156956431 % of runtime
Evaluation count : 1680 in 60 samples of 28 calls.
Execution time mean : 36.435413 ms
Execution time std-deviation : 1.607847 ms
Execution time lower quantile : 35.764207 ms ( 2.5%)
Execution time upper quantile : 37.527755 ms (97.5%)
Overhead used : 2.222121 ns
Found 4 outliers in 60 samples (6.6667 %)
low-severe 1 (1.6667 %)
low-mild 3 (5.0000 %)
Variance from outliers : 30.3257 % Variance is moderately inflated by outliers
nil
user> (crit/bench (reduce conj () input))
WARNING: Final GC required 9.024275674955403 % of runtime
Evaluation count : 3540 in 60 samples of 59 calls.
Execution time mean : 19.623083 ms
Execution time std-deviation : 3.842658 ms
Execution time lower quantile : 17.891881 ms ( 2.5%)
Execution time upper quantile : 26.569738 ms (97.5%)
Overhead used : 2.222121 ns
Found 3 outliers in 60 samples (5.0000 %)
low-severe 3 (5.0000 %)
Variance from outliers : 91.0960 % Variance is severely inflated by outliers
nil
user> (crit/bench (reduce conj #{} input))
WARNING: Final GC required 12.0207279064623 % of runtime
Evaluation count : 120 in 60 samples of 2 calls.
Execution time mean : 965.389668 ms
Execution time std-deviation : 682.645918 ms
Execution time lower quantile : 674.958427 ms ( 2.5%)
Execution time upper quantile : 2.287535 sec (97.5%)
Overhead used : 2.222121 ns
Found 9 outliers in 60 samples (15.0000 %)
low-severe 9 (15.0000 %)
Variance from outliers : 98.2830 % Variance is severely inflated by outliers
nil
user>