Why my SQLite3 query takes time? - sql

Question

Context: I developed a Read-Only VFS for SQLite using the C API. The thing is what I need is both speed and small file size. I solved the size problem with a LZ4 based VFS. However I have some speed issues when I query my DB.
Specifications:
- I work on Linux (Ubuntu 12.10)
- DB files are 275MB compressed and about 700MB uncompressed
- I am doing queries on indexed fields.
- I evaluate the time taken for a given query after droping caches (echo 3 | sudo tee /proc/sys/vm/dropcaches)
Problem:
When I query the DB with the command time, I get the following output:
real 0m5.933s
user 0m0.124s
sys 0m0.096s
What is surprising is the difference between user+sys and real. This is why I decided to profile, with gprof, the code I have written as well as its dependencies (sqlite3,lz4). Hereafter, you will find few lines of the gprof flat and call-graph representation. After that, I have no idea about what to look out if I want to find a solution. Mainly because I do not understand why (and where) all this time is wasted. I hope you can help me.
Flat profile:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls ms/call ms/call name
93.75 0.15 0.15 2948 0.05 0.05 LZ4_decompress_fast
6.25 0.16 0.01 26068 0.00 0.01 sqlite3VdbeCursorMoveto
0.00 0.16 0.00 54088 0.00 0.00 sqlite3GetVarint
0.00 0.16 0.00 28459 0.00 0.00 sqlite3VdbeSerialGet
0.00 0.16 0.00 23708 0.00 0.00 sqlite3VdbeMemNulTerminate
0.00 0.16 0.00 23699 0.00 0.00 sqlite3_transfer_bindings
0.00 0.16 0.00 11910 0.00 0.00 sqlite3DbMallocSize
0.00 0.16 0.00 11883 0.00 0.00 sqlite3VdbeMemGrow
0.00 0.16 0.00 11851 0.00 0.00 sqlite3VdbeMemMakeWriteable
0.00 0.16 0.00 9480 0.00 0.00 fetchPayload
0.00 0.16 0.00 9478 0.00 0.00 sqlite3VdbeMemRelease
0.00 0.16 0.00 7170 0.00 0.02 btreeGetPage
0.00 0.16 0.00 7170 0.00 0.00 pcache1Fetch
0.00 0.16 0.00 7170 0.00 0.00 releasePage
0.00 0.16 0.00 7170 0.00 0.02 sqlite3PagerAcquire
0.00 0.16 0.00 7170 0.00 0.00 sqlite3PagerUnrefNotNull
0.00 0.16 0.00 7170 0.00 0.00 sqlite3PcacheFetch
0.00 0.16 0.00 7170 0.00 0.00 sqlite3PcacheRelease
0.00 0.16 0.00 7169 0.00 0.00 pcache1Unpin
0.00 0.16 0.00 7169 0.00 0.00 pcacheUnpin
0.00 0.16 0.00 7168 0.00 0.02 getAndInitPage
0.00 0.16 0.00 7165 0.00 0.02 moveToChild
granularity: each sample hit covers 2 byte(s) for 6.25% of 0.16 seconds
index % time self children called name
<spontaneous>
[1] 100.0 0.00 0.16 main [1]
0.00 0.16 1/1 sqlite3_exec <cycle 1> [5]
0.00 0.00 1/1 openDatabase [18]
0.00 0.00 2/3 sqlite3_vfs_find [249]
0.00 0.00 1/1 sqlite3_crodvfs [358]
0.00 0.00 1/14 sqlite3_vfs_register <cycle 5> [204]
0.00 0.00 1/1 sqlite3_open_v2 [359]
-----------------------------------------------
[2] 99.9 0.00 0.16 1+198 <cycle 1 as a whole> [2]
0.00 0.16 2 sqlite3_exec <cycle 1> [5]
0.00 0.00 2 sqlite3InitOne <cycle 1> [27]
0.00 0.00 124 sqlite3Parser <cycle 1> [82]
0.00 0.00 6+4 sqlite3WalkSelect <cycle 1> [186]
0.00 0.00 9 sqlite3ReadSchema <cycle 1> [149]
0.00 0.00 7 sqlite3LockAndPrepare <cycle 1> [168]
0.00 0.00 7 sqlite3Prepare <cycle 1> [171]
0.00 0.00 7 sqlite3RunParser <cycle 1> [172]
0.00 0.00 5 sqlite3InitCallback <cycle 1> [196]
0.00 0.00 5 sqlite3_prepare <cycle 1> [203]
0.00 0.00 5 sqlite3SelectPrep <cycle 1> [198]
0.00 0.00 4 sqlite3LocateTable <cycle 1> [213]
0.00 0.00 3 sqlite3StartTable <cycle 1> [243]
0.00 0.00 2 selectExpander <cycle 1> [277]
0.00 0.00 2 sqlite3Select <cycle 1> [299]
0.00 0.00 2 sqlite3CreateIndex <cycle 1> [282]
0.00 0.00 2 sqlite3_prepare_v2 <cycle 1> [312]
0.00 0.00 2 resolveSelectStep <cycle 1> [275]
0.00 0.00 2 resolveOrderGroupBy <cycle 1> [274]
0.00 0.00 1 sqlite3Init <cycle 1> [347]
-----------------------------------------------
0.00 0.16 2374/2374 sqlite3_step [4]
[3] 99.9 0.00 0.16 2374 sqlite3VdbeExec [3]
0.01 0.15 26068/26068 sqlite3VdbeCursorMoveto [6]
0.00 0.00 53/54 moveToLeftmost [17]
0.00 0.00 2/3 sqlite3BtreeBeginTrans [20]
0.00 0.00 1/2370 sqlite3BtreeMovetoUnpacked [16]
0.00 0.00 2372/2372 sqlite3BtreeNext [28]
0.00 0.00 1/2371 moveToRoot [23]
0.00 0.00 26068/28459 sqlite3VdbeSerialGet [34]
0.00 0.00 23699/23708 sqlite3VdbeMemNulTerminate [35]
0.00 0.00 23699/23699 sqlite3_transfer_bindings [36]
0.00 0.00 11851/11851 sqlite3VdbeMemMakeWriteable [39]
0.00 0.00 7108/7109 sqlite3BtreeKeySize [49]
0.00 0.00 4741/9480 fetchPayload [40]
0.00 0.00 4739/4739 sqlite3VdbeMemFromBtree [55]
0.00 0.00 4739/9478 sqlite3VdbeMemRelease [41]
0.00 0.00 2374/2376 sqlite3VdbeMemShallowCopy [65]
0.00 0.00 2372/2376 sqlite3VdbeCheckFk [64]
0.00 0.00 2372/2372 sqlite3VdbeCloseStatement [66]
0.00 0.00 2372/4742 btreeParseCellPtr [54]
0.00 0.00 2372/4742 btreeParseCell [53]
0.00 0.00 2370/2391 sqlite3VdbeRecordCompare [63]
0.00 0.00 2369/2369 sqlite3VdbeIntValue [67]
0.00 0.00 893/893 sqlite3VdbeRealValue [74]
0.00 0.00 3/3 sqlite3VdbeFreeCursor [245]
0.00 0.00 3/3 allocateCursor [226]
0.00 0.00 3/3 sqlite3BtreeCursor [237]
0.00 0.00 2/9 sqlite3VdbeHalt [152]
0.00 0.00 1/36 sqlite3BtreeLeave [98]
0.00 0.00 1/6 sqlite3BtreeGetMeta [185]
0.00 0.00 1/1 sqlite3GetVarint32 [345]