I'm currently trying to optimize a MYSQL statement that is taking quite some time. The table this is running on is 600k+ and the query is taking over 10 seconds.
SELECT DATE_FORMAT( timestamp, '%Y-%m-%d' ) AS date, COUNT( DISTINCT (
email
) ) AS count
FROM log
WHERE timestamp > '2009-02-23'
AND timestamp < '2020-01-01'
AND TYPE = 'play'
GROUP BY date
ORDER BY date DESC
I've just indexes on timestamp and type and also one on timestamp_type (type_2).
Here is the explain results, the problem seems to be a file sort but I don't know how to get around this...
id: 1
select_type: SIMPLE
table: log
type: ref
possible_keys: type,timestamp,type_2
key: type_2
key_len: 1
ref: const
rows: 226403
Extra: Using where; Using filesort
Thanks
Things to try:
Have a separate date column (indexed) and use that instead of your timestamp column
Add an index across type and date
Use BETWEEN (don't think it will affect the speed but it's easier to read)
So ideally you would
Create a date column and fill it using UPDATE table SET date = DATE(timestamp)
Index across type and date
Change your select to ... type = ? AND date BETWEEN ? AND ?
Try rewriting to filter on TYPE alone first. Then apply your date range and aggregates. Basically create an inline view that filters type down. I know it's likely that the optimizer is doing this already, but when trying to improve performance I find it's helpful to be very certain of what things are happening first.
DATE_FORMAT will not utilizing the indexes.
You can still use the below query to utilize the index on timestamp column
SELECT timestamp AS date, COUNT( DISTINCT (
email
) ) AS count
FROM log
WHERE timestamp > '2009-02-23 00:00:00'
AND timestamp < '2020-01-01 23:59:59'
AND TYPE = 'play'
GROUP BY date
ORDER BY date DESC
Format the datetime value to date while printing/using
Related
I have the following CODE
$thedate = "2018-06-04"; // YYYY-MM-DD
SELECT *
FROM (`leadactivity`)
WHERE statusdate = $thedate
ORDER BY id DESC
LIMIT 25
I am trying to only show the results where statusdate is = to the date provided. However because in sql the column autoupdates with Timestamp, it's including the time, and for some reason is just not giving any results.
Any ideas on what I am doing wrong?
Your code looks like MySQL. One simple way is:
SELECT la.*
FROM leadactivity la
WHERE DATE(la.statusdate) = $thedate
ORDER BY id DESC
LIMIT 25;
However, the use of the DATE() function prevents the use of an index. So, a better approach is:
SELECT la.*
FROM leadactivity la
WHERE la.statusdate >= $thedate AND la.statusdate < $thedate + interval 1 day
ORDER BY la.id DESC
LIMIT 25;
try this:
SELECT * FROM (`leadactivity`)
WHERE statusdate like '%"$thedate"%'
ORDER BY id DESC LIMIT 25
When comparing timestamps with date values, a time-of-day of midnight (ie 00:00:00) is assumed. Specify a range of timestamps based on the date from 00:00:00 to 23:59:59, like this:
SELECT * FROM (`leadactivity`)
WHERE statusdate between '$thedate 00:00:00' and '$thedate 23:59:59'
ORDER BY id DESC LIMIT 25
Depending on your language and database, the syntax to achieve this will vary.
Although converting the timestamp to a date before comparing is simpler, your database won’t use the index on the timestamp column (if one exists), resulting in the worst performance.
I have this data.
this is my query
SELECT transaction_date
FROM
tenant1.txn_transaction_record where '2015-04-14'
The said query renders empty result. Is it possible to query timestamp field using only date?
When you do:
transaction_date = '2015-04-14'
PG will convert string '2015-04-14' to timestamp '2015-04-14 00:00:00' value.
If you do:
transaction_date::date = '2015-04-14'
PG will convert both values to date (wich is only date part, without time part), and it'll work.
BUT... BE CAREFUL WHEN CASTING COLUMNS IN WHERE CLAUSE, because PG will not be able to take advantage of an index that contains that column, unless you've created the index with same cast on the column.
If you create only this index:
create index i_foo_1 on foo ( timestamp_field );
This query WILL NOT use that index:
select *
from foo
where timestamp_field::date = '2015-04-15';
So, or you'll need to create an aditional index:
create index i_foo_2 on foo ( timestamp_field::date );
Or you'll have to change your original "where clause":
select *
from foo
where timestamp_field >= ('2015-04-15'::timestamp)
and timestamp_field < (('2015-04-15'::date)+1)::timestamp;
No, If you do like transaction_date = '2015-04-14' It will automatically search for transaction_date = '2015-04-14T00:00:00' So you wont yield any result. Therefore if you want to search the date try transaction_date::date = '2015-04-14'
So the final query is,
SELECT transaction_date
FROM
tenant1.txn_transaction_record where transaction_date::date = '2015-04-14'
SELECT transaction_date
FROM
tenant1.txn_transaction_record where date_trunc('day', transaction_date) = '2015-04-14 00:00:00'
I don't have a postgres database up to try it :-)
Every record in my SQLite database contains a field which contains a Date stored as a string in the format 'yyyy-MM-dd HH:mm:ss'.
Is it possible to query the database to get the record which contains the most recent date please?
you can do it like this
SELECT * FROM Table ORDER BY date(dateColumn) DESC Limit 1
For me I had my query this way to solve my problem
select * from Table order by datetime(datetimeColumn) DESC LIMIT 1
Since I was storing it as datetime not date column
When you sure the format of text field is yyyy-MM-dd HH:mm:ss (ex.: 2017-01-02 16:02:55), So It works for me simply:
SELECT * FROM Table ORDER BY dateColumn DESC Limit 1
Without any extra date function!
You need to convert it to unix timestamp, and then compare them:
SELECT * FROM data ORDER BY strftime('%s', date_column) DESC
But this can be pretty slow, if there are lots of rows.
Better approach would be to store unix timestamp by default, and create an index for that column.
You can convert your column sent_date_time to yyyy-MM-dd format and then order by date:
1) substr(sent_date_time,7,4)||"-"||substr(sent_date_time,1,2)||"-"||substr(sent_date_time,4,2) as date
2) order by date desc
In my case everything works fine without casting column to type 'date'. Just by specifying column name with double quotes like that:
SELECT * FROM 'Repair' ORDER BY "Date" DESC;
I think SQLite makes casting by itself or something like that, but when I tried to 'cast' Date column by myself it's not worked. And there was no error messages.
You can also use the following query
"SELECT * FROM Table ORDER BY strftime('%Y-%m-%d %H:%M:%S'," + dateColumn + ") DESC Limit 1"
I found this ugly hack worked.
select *, substr(date_col_name,7,4)as yy,
substr(date_col_name,4,2) as mm,
substr(date_col_name,1,2) as dd
from my_table
order by yy desc,mm desc,dd desc
it would be better to convert the text column to date field type, but I found that did not work reliably for me.
If you do a lot of date sorting/comparison, you may get better results by storing time as ticks rather than strings, here is showing how to get 'now' in ticks with:
((strftime('%s', 'now') - strftime('%S', 'now') + strftime('%f', 'now')) * 1000)
(see https://stackoverflow.com/a/20478329/460084)
Then it's easy to sort, compare, etc ...
This will work for both date and time
SELECT *
FROM Table
ORDER BY
julianday(dateColumn)
DESC Limit 1
I have a table for matches. The table has a column named matchdate, which is a datetime field.
If I have 3 matches on 2011-12-01:
2011-12-01 12:00:00
2011-12-01 13:25:00
2011-12-01 16:00:00
How do I query that? How do I query all matches on 1 single date?
I have looked at date_trunc(), to_char(), etc.
Isn't there some "select * where datetime in date" function?
Cast your timestamp value to date if you want simple syntax. Like this:
SELECT *
FROM tbl
WHERE timestamp_col::date = '2011-12-01'; -- date literal
However, with big tables this will be faster:
SELECT *
FROM tbl
WHERE timestamp_col >= '2011-12-01 0:0' -- timestamp literal
AND timestamp_col < '2011-12-02 0:0';
Reason: the second query does not have to transform every single value in the table and can utilize a simple index on the timestamp column. The expression is sargable.
Note excluded the upper bound (< instead of <=) for a correct selection.
You can make up for that by creating an index on an expression like this:
CREATE INDEX tbl_ts_date_idx ON tbl (cast(timestamp_col AS date));
Then the first version of the query will be as fast as it gets.
not sure if i am missing something obvious here, but i think you can just
select * from table where date_trunc('day', ts) = '2011-12-01';
Just use the SQL BETWEEN function like so:
SELECT * FROM table WHERE date BETWEEN '2011-12-01' AND '2011-12-02'
You may need to include times in the date literals, but this should include the lover limit and exclude the upper.
From rails I believe you can do:
.where(:between => '2011-12-01'..'2011-12-02')
My table has records like these
23-MAY-11 11.40.39.000000 AM
The following query brings nothing
SELECT *
FROM my_table
WHERE tenant_pha = 'test'
AND create_date >= TO_DATE('05/10/2011','mm/dd/yyyy')
AND create_date <= TO_DATE('05/23/2011','mm/dd/yyyy')
However, the below query will bring data
SELECT *
FROM my_table
WHERE tenant_pha = 'test'
AND create_date >= TO_DATE('05/10/2011','mm/dd/yyyy')
AND create_date <= TO_DATE('05/24/2011','mm/dd/yyyy')
I think this is because create_date column is time stamp.
How can I change my query to bring the desired result ( I want to avoid doing functions on the left side columns because they will make the query long).
You are right about the timestamp. '05/23/2011' is the same as '05/23/2011 12:00 AM'.
To include the whole day I usually move my date up by a day. < '05/24/2011' will include all of 5/23.
or change to '05/23/2011 23:59:59'
You can use trunc() without problems, you only need to create a function based index.
If you create this index:
CREATE INDEX idx_trunc_date ON my_table (trunc(create_date));
then the following condition will make use of that index:
AND trunc(create_date) >= TO_DATE('05/10/2011','mm/dd/yyyy')