How do speed up date range select in sqlite? - sql

Question

Here is my query:
const tags = await db.queryEntries(
"SELECT tag, count(tag) AS count, created_at FROM tags WHERE DATE(created_at) >= DATE('now', '-1 days') GROUP BY tag ORDER BY count DESC LIMIT 100"
);
Here is my schema:
CREATE TABLE tags (
tag TEXT,
url STRING
, created_at TEXT);
CREATE UNIQUE INDEX tag_url ON tags (tag, url)
;
CREATE INDEX idx_tags_created_at ON tags(created_at);
It's still very slow (30+ seconds) when I run the query there are about 1.5 million records in the db for tags.
Here are the results of EXPLAIN:
addr opcode p1 p2 p3 p4 p5 comment
---- ------------- ---- ---- ---- ------------- -- -------------
0 Init 0 56 0 00 Start at 56
1 OpenEphemeral 1 5 0 k(1,-B) 00 nColumn=5
2 Integer 100 1 0 00 r[1]=100; LIMIT counter
3 Noop 2 2 0 00
4 Integer 0 6 0 00 r[6]=0; clear abort flag
5 Null 0 9 9 00 r[9..9]=NULL
6 Gosub 8 46 0 00
7 OpenRead 0 7 0 3 00 root=7 iDb=0; tags
8 OpenRead 3 3693502 0 k(3,,,) 00 root=3693502 iDb=0; tag_url
9 Rewind 3 28 11 0 00
10 DeferredSeek 3 0 0 00 Move 0 to 3.rowid if needed
11 Column 0 2 12 00 r[12]=tags.created_at
12 Function 0 12 11 date(-1) 00 r[11]=func(r[12])
13 Lt 13 27 11 50 if r[11]<r[13] goto 27
14 Column 3 0 10 00 r[10]=tags.tag
15 Compare 9 10 1 k(1,-B) 00 r[9] <-> r[10]
16 Jump 17 21 17 00
17 Move 10 9 1 00 r[9]=r[10]
18 Gosub 7 32 0 00 output one row
19 IfPos 6 49 0 00 if r[6]>0 then r[6]-=0, goto 49; check abort flag
20 Gosub 8 46 0 00 reset accumulator
21 Column 3 0 11 00 r[11]=tags.tag
22 AggStep 0 11 3 count(1) 01 accum=r[3] step(r[11])
23 If 5 26 0 00
24 Column 3 0 2 00 r[2]=tags.tag
25 Column 0 2 4 00 r[4]=tags.created_at
26 Integer 1 5 0 00 r[5]=1; indicate data in accumulator
27 Next 3 10 0 01
28 Gosub 7 32 0 00 output final row
29 Goto 0 49 0 00
30 Integer 1 6 0 00 r[6]=1; set abort flag
31 Return 7 0 0 00
32 IfPos 5 34 0 00 if r[5]>0 then r[5]-=0, goto 34; Groupby result generator entry point
33 Return 7 0 0 00
34 AggFinal 3 1 0 count(1) 00 accum=r[3] N=1
35 Copy 3 14 0 00 r[14]=r[3]
36 Sequence 1 15 0 00 r[15]=cursor[1].ctr++
37 IfNotZero 1 41 0 00 if r[1]!=0 then r[1]--, goto 41
38 Last 1 0 0 00
39 IdxLE 1 45 14 1 00 key=r[14]
40 Delete 1 0 0 00
41 Copy 2 16 0 00 r[16]=r[2]
42 Copy 4 17 0 00 r[17]=r[4]
43 MakeRecord 14 4 19 00 r[19]=mkrec(r[14..17])
44 IdxInsert 1 19 14 4 00 key=r[19]
45 Return 7 0 0 00 end groupby result generator
46 Null 0 2 4 00 r[2..4]=NULL
47 Integer 0 5 0 00 r[5]=0; indicate accumulator empty
48 Return 8 0 0 00
49 Sort 1 55 0 00
50 Column 1 3 18 00 r[18]=created_at
51 Column 1 0 17 00 r[17]=count
52 Column 1 2 16 00 r[16]=tag
53 ResultRow 16 3 0 00 output=r[16..18]
54 Next 1 50 0 00
55 Halt 0 0 0 00
56 Transaction 0 0 8 0 01 usesStmtJournal=0
57 String8 0 20 0 now 00 r[20]='now'
58 String8 0 21 0 -1 days 00 r[21]='-1 days'
59 Function 3 20 13 date(-1) 00 r[13]=func(r[20])
60 Goto 0 1 0 00

Answer 1

Rewrite the query so that it is sargable (i.e. it can take advantage of an index):
SELECT tag, COUNT(tag) AS count
FROM tags
WHERE created_at >= strftime('%Y-%m-%d', 'now', '-1 days')
GROUP BY tag
ORDER BY count DESC
LIMIT 100;
The above query should benefit from the following index:
CREATE INDEX idx ON tags (created_at, tag);