speeding up records fetch time - sql
I have a SQL sentence which takes around 44 Sec to fetch the data. how can I reduce the time ?
this is the sql
Select AVNR,replace(to_char((CAST(DBTM as timestamp)),'hh24:mi'),'00:00','24:00') as ptime,
Wert
From e_mw_60min_me
Where to_char((CAST ( DBTM as TIMESTAMP)), 'DD-MM-YYYY') = '13-03-1396' and
AVNR In (1141,1142,1144,10335,10336,12016,1146,1147,1149,1129,1130,1132,1134,1135,1137,5895,5896,5900,8906,8907,8909,8901,8902,8904,8940,8941,8943,8951,8952,8954,8972,8973,8975,8830,8831,8833,8835,8836,8838,1113,1982,1984,2314,2315,2317,3272,3273,3275,3267,3268,3270,3262,3263,3265,10231,10136,9066,8779,8780,8782,8774,8775,8777,8320,8321,8323,7696,7697,7699,10486,10487,10489,3329,3330,12018,3322,3328,10132,3320,3321,12017,3222,3223,3225,686,687,689,691,692,694,696,697,699,1,2,4,10527,10528,10529,4911,4912,4914,4917,4918,4920,5162,5163,5166,5157,5158,5160,5168,5169,5171,5449,5450,5452,10116,10117,10119,10120,10121,10123,2271,2272,2279,2266,2267,2269,2259,2260,2262,1292,1293,1295,5380,5381,5383,5374,5375,5377,10545,10546,10547,3281,10126,12031,3244,3245,12030,10983,10984,12033,10987,10989,12032,11073,11074,1125,11079,11080,3333,105,106,108,100,101,103,93,94,96,29,30,32,95,102,1197,124,123,126,12085,12086,12087,2520,2521,2523,2525,2526,2528,2515,2516,2518,2510,2511,2513,5444,5445,5447,2201,2202,12025,3336,3337,3339,3002,3003,12029,1643,1644,1646,1609,1610,1612,1596,1597,1599,9717,9718,9720,9722,9723,9725,2146,2147,2149,2141,2142,2144,2136,2137,2139,2131,2132,2134,2121,2122,2124,2126,2127,2129,2635,2636,2638,2641,2642,2644,7499,7500,7502,4610,4611,4613,4605,4606,4608,4600,4601,4603,9074,9075,9077,9079,9080,9082,9235,9236,9238,9240,9241,9243,9245,9246,9248,8468,8469,8471,5785,5786,5788,5790,5791,5793,5691,5692,5694,5685,5686,5688,8455,8456,8458,11312,11313,4588,7654,7655,7657,9376,9377,9379,9371,9372,9374,9382,9383,9385,5918,5919,5922,5934,5935,5938,5912,5913,5916,10963,7860,10964,135,136,138,10658,10659,10664,10660,10661,10662,5173,5458,5460);
//these AVNRs are completley Dynamic
the Table is like :
AVNR (Like ID Number)
DBTM( reocrds every time for every points)
Wert (the value)
I want to know any other sql command which makes it faster to fetch
If DBTM is a date you don't need to cast it to a timestamp; and if there is an index on that then you should not convert it to a string to compare with another string - leave it as a date and convert the fixed value to a date too, and compare with a range that covers the entire day:
select AVNR,
replace(to_char(DBTM,'hh24:mi'),'00:00','24:00') as ptime,
Wert
from e_mw_60min_me
where DBTM >= date '1396-03-13'
and DBTM < date '1396-03-14'
and AVNR In (1141, ...);
I've used date literals but you can use to_date() if you are starting from a variable string:
where DBTM >= to_date('13-03-1396', 'DD-MM-YYYY')
and DBTM < to_date('13-03-1396', 'DD-MM-YYYY') + 1
and AVNR In (1141, ...);
You should also consider creating a collection for all the AVNR values you are looking for and using member of instead if IN, or explode the collection and join to it. It depends where those values are coming from though.
You can add an index:
create index idx_e_mw_60min_me_2 on (to_char((CAST ( DBTM as TIMESTAMP)), 'DD-MM-YYYY'), AVNR)
This exactly matches the where clause so it should speed up the query.
Related
storing date in 'CCYYMMDD' format in Teradata
I would like to store dates in the format CCYYMMDD in Teradata, but I fail to do so. Find below what I tried so far: query 1: SEL CAST(CAST(CURRENT_DATE AS DATE FORMAT 'YYYYMMDD') AS VARCHAR(8)) -- Output: 20191230 ==> this works! query 2: SEL CAST(CAST(CURRENT_DATE AS DATE FORMAT 'CCYYMMDD') AS VARCHAR(8)) -- output: SELECT Failed. [3530] Invalid FORMAT string 'CCYYMMDD'. It seems that the CCYYMMDD is not available in Teradata right away. Is there a workaround? Tool used: Teradata SQL assistant
Internally, dates are stored as integers in Teradata. So when you say you want to store them in a different format, I don't think you can do that. But you can choose how to display / return the values. I'm sure there's a cleaner way to get the format you want, but here's one way: WITH cte (mydate) AS ( SELECT CAST(CAST(CURRENT_DATE AS DATE FORMAT 'YYYYMMDD') AS CHAR(8)) AS mydate ) SELECT CAST( (CAST(SUBSTRING(mydate FROM 1 FOR 2) AS INTEGER) + 1) -- generate "century" value AS CHAR(2) -- cast value as string ) || SUBSTRING(mydate FROM 3) AS new_date -- add remaining portion of date string FROM cte SQL Fiddle - Postgres You'd have to add some extra logic to handle years before 1000 and after 9999. I don't have a TD system to test, but give it a try and let me know.
Oracle use LIKE '%' on DATE
My table myTab has the column startDate, which has the datatype "DATE". The data in this column are stored like dd.mm.yyyy. Now I'm trying to get data with this query: SELECT * FROM myTab WHERE startDate like '%01.2015" Somehow it doesn't work and I don't know why. Hope someone can help.
To make a text search on the date you would have to convert the date to text. It's more efficient if you calculate the first and last date for what you want to find and get everything between them. That way it's done as numeric comparisons instead of a text pattern match, and it can make use of an index if there is one: SELECT * FROM myTab WHERE startDate >= DATE '2015-01-01' AND startDate < DATE '2015-02-01'
SELECT * FROM myTab WHERE TO_CHAR(startDate,'dd.mm.yyyy') LIKE '%01.2015'
If the field type is "DATE" then the value isn't stored as a string, it's a number managed by Oracle, so you have to convert it to a string:
SELECT * FROM myTab WHERE to_char(startDate, 'MM.YYYY') = '01.2015';
You can also use date ranges in SQL queries:
SELECT * FROM myTab
WHERE startDate
BETWEEN to_date('01.01.2015', 'DD.MM.YYYY')
AND to_date('31.01.2015', 'DD.MM.YYYY');
Regarding you actual question "Somehow it doesn't work and I don't know why." Oracle make an implicit conversion from DATE to VARHCAR2, however it uses the default NLS_DATE_FORMAT which is probably different to what you use in your query.
The data in this column are stored like dd.mm.yyyy. Oracle does not store date in the format you see. It stores it internally in proprietary format in 7 bytes with each byte storing different components of the datetime value. WHERE startDate like '%01.2015" You are comparing a DATE with a STRING, which is pointless. From performance point of view, you should use a date range condition so that if there is any regular INDEX on the date column, it would be used. SELECT * FROM table_name WHERE date_column BETWEEN DATE '2015-01-01' AND DATE '2015-02-01' To understand why a Date range condition is better in terms of performance, have a look at my answer here.
I solved my problem that way. Thank you for suggestions for improvements. Example in C#. string dd, mm, aa, trc, data; dd = nData.Text.Substring(0, 2); mm = nData.Text.Substring(3, 2); aa = nData.Text.Substring(6, 4); trc = "-"; data = aa + trc + mm + trc + dd; "Select * From bdPedidos Where Data Like '%" + data + "%'";
To provide a more detailed answer and address this https://stackoverflow.com/a/42429550/1267661 answer's issue.
In Oracle a column of type "date" is not a number nor a string, it's a "datetime" value with year, month, day, hour, minute and seconds.
The default time is always midnight "00:00:00"
The query:
Select * From bdPedidos Where Data Like '%" + data + "%'"
won't work in all circumstances because a date column is not a string, using "like" forces Oracle to do a conversion from date value to string value.
The string value may be year-month-day-time or month-day-year-time or day-month-year-time, that all depends how a particular Oracle instance has set the parameter NLS_DATE_FORMAT to show dates as strings.
The right way to cover all the possible times in a day is:
Select *
From bdPedidos
Where Data between to_date('" + data + " 00:00:00','yyyy-mm-dd hh24:mi:ss')
and to_date('" + data + " 23:59:59','yyyy-mm-dd hh24:mi:ss')
SELECT * FROM myTab WHERE startDate like '%-%-2015'; This will search for all dates in 2015. If this doesn't work, try: SELECT * FROM myTab WHERE startDate like '%-%-15';
SQL. Select Unixtime for whole day
I am looking for a way to select a whole days worth of data from a where statement. Timestamp is in unix time such as (1406045122). I want to select the today's date of unix time range and find all the food that has been added in today. Thank in advance. This is the code I wrote. I'm not sure what I should put in the ( ????? ) part. I know it has to do with 60*60*24=86400 secs per day but I'm not too sure how I can implement this. Select timestamp,food from table1 where timestamp = ( ????? );
Select timestamp,food
FROM table1
WHERE timestamp > :ts
AND timestamp <= (:ts + 86400);
replace :ts with the starting timstamp and you'll filter a whole day's worth of data
edit
This select query would give you the current timestamp (there may be more efficient ones, i don't work with sqlite often)
select strftime("%s", current_timestamp);
You can find more info about them here: http://www.tutorialspoint.com/sqlite/sqlite_date_time.htm
Using the strftime() function, combined with the date() function we can write this following query which will not need any manual editing. It will return the records filtered on timestamp > start of today & timestamp <= end of today.
Select timestamp,food
FROM table1
WHERE timestamp > strftime("%s", date(current_timestamp))
AND timestamp <= (strftime("%s", date(current_timestamp)) + 86400);
Your mileage will likely depend on your version of SQL but for example on MySQL you can specify a search as being BETWEEN two dates, which is taken conventionally to mean midnight on each. So SELECT * FROM FOO WHERE T BETWEEN '2014-07-01' AND '2014-07-02'; selects anything with a timestamp anywhere on 1st July 2014. If you want to make it readable you could even use the ADDDATE function. So you could do something like SET #mydate = DATE(T); SELECT * FROM FOO WHERE T BETWEEN #mydate AND ADDDATE(#mydate, 1); The first line should truncate your timestamp to be 00:00:00. The second line should SELECT only records from that date.
convert Excel Date Serial Number to Regular Date
I got a column called DateOfBirth in my csv file with Excel Date Serial Number Date Example: 36464 37104 35412 When i formatted cells in excel these are converted as 36464 => 1/11/1999 37104 => 1/08/2001 35412 => 13/12/1996 I need to do this transformation in SSIS or in SQL. How can this be achieved?
In SQL: select dateadd(d,36464,'1899-12-30') -- or thanks to rcdmk select CAST(36464 - 2 as SmallDateTime) In SSIS, see here http://msdn.microsoft.com/en-us/library/ms141719.aspx
The marked answer is not working fine, please change the date to "1899-12-30" instead of "1899-12-31". select dateadd(d,36464,'1899-12-30')
You can cast it to a SQL SMALLDATETIME: CAST(36464 - 2 as SMALLDATETIME) MS SQL Server counts its dates from 01/01/1900 and Excel from 12/30/1899 = 2 days less.
tldr:
select cast(#Input - 2e as datetime)
Explanation:
Excel stores datetimes as a floating point number that represents elapsed time since the beginning of the 20th century, and SQL Server can readily cast between floats and datetimes in the same manner. The difference between Excel and SQL server's conversion of this number to datetimes is 2 days (as of 1900-03-01, that is). Using a literal of 2e for this difference informs SQL Server to implicitly convert other datatypes to floats for very input-friendly and simple queries:
select
cast('43861.875433912' - 2e as datetime) as ExcelToSql, -- even varchar works!
cast(cast('2020-01-31 21:00:37.490' as datetime) + 2e as float) as SqlToExcel
-- Results:
-- ExcelToSql SqlToExcel
-- 2020-01-31 21:00:37.490 43861.875433912
this actually worked for me dateadd(mi,CONVERT(numeric(17,5),41869.166666666664)*1440,'1899-12-30') (minus 1 more day in the date) referring to the negative commented post
SSIS Solution The DT_DATE data type is implemented using an 8-byte floating-point number. Days are represented by whole number increments, starting with 30 December 1899, and midnight as time zero. Hour values are expressed as the absolute value of the fractional part of the number. However, a floating point value cannot represent all real values; therefore, there are limits on the range of dates that can be presented in DT_DATE. Read more From the description above you can see that you can convert these values implicitly when mapping them to a DT_DATE Column after converting it to a 8-byte floating-point number DT_R8. Use a derived column transformation to convert this column to 8-byte floating-point number: (DT_R8)[dateColumn] Then map it to a DT_DATE column Or cast it twice: (DT_DATE)(DT_R8)[dateColumn] You can check my full answer here: Is there a better way to parse [Integer].[Integer] style dates in SSIS?
Found this topic helpful so much so created a quick SQL UDF for it. CREATE FUNCTION dbo.ConvertExcelSerialDateToSQL ( #serial INT ) RETURNS DATETIME AS BEGIN DECLARE #dt AS DATETIME SELECT #dt = CASE WHEN #serial is not null THEN CAST(#serial - 2 AS DATETIME) ELSE NULL END RETURN #dt END GO
I had to take this to the next level because my Excel dates also had times, so I had values like this: 42039.46406 --> 02/04/2015 11:08 AM 42002.37709 --> 12/29/2014 09:03 AM 42032.61869 --> 01/28/2015 02:50 PM (also, to complicate it a little more, my numeric value with decimal was saved as an NVARCHAR) The SQL I used to make this conversion is: SELECT DATEADD(SECOND, ( CONVERT(FLOAT, t.ColumnName) - FLOOR(CONVERT(FLOAT, t.ColumnName)) ) * 86400, DATEADD(DAY, CONVERT(FLOAT, t.ColumnName), '1899-12-30') )
In postgresql, you can use the following syntax:
SELECT ((DATE('1899-12-30') + INTERVAL '1 day' * FLOOR(38242.7711805556)) + (INTERVAL '1 sec' * (38242.7711805556 - FLOOR(38242.7711805556)) * 3600 * 24)) as date
In this case, 38242.7711805556 represents 2004-09-12 18:30:30 in excel format
In addition of #Nick.McDermaid answer I would like to post this solution, which convert not only the day but also the hours, minutes and seconds: SELECT DATEADD(s, (42948.123 - FLOOR(42948.123))*3600*24, dateadd(d, FLOOR(42948.123),'1899-12-30')) For example 42948.123 to 2017-08-01 02:57:07.000 42818.7166666667 to 2017-03-24 17:12:00.000
You can do this if you just need to display the date in a view: CAST will be faster than CONVERT if you have a large amount of data, also remember to subtract (2) from the excel date: CAST(CAST(CAST([Column_With_Date]-2 AS INT)AS smalldatetime) AS DATE) If you need to update the column to show a date you can either update through a join (self join if necessary) or simply try the following: You may not need to cast the excel date as INT but since the table I was working with was a varchar I had to do that manipulation first. I also did not want the "time" element so I needed to remove that element with the final cast as "date." UPDATE [Table_with_Date] SET [Column_With_Excel_Date] = CAST(CAST(CAST([Column_With_Excel_Date]-2 AS INT)AS smalldatetime) AS DATE) If you are unsure of what you would like to do with this test and re-test! Make a copy of your table if you need. You can always create a view!
Google BigQuery solution Standard SQL Select Date, DATETIME_ADD(DATETIME(xy, xm, xd, 0, 0, 0), INTERVAL xonlyseconds SECOND) xaxsa from ( Select Date, EXTRACT(YEAR FROM xonlydate) xy, EXTRACT(MONTH FROM xonlydate) xm, EXTRACT(DAY FROM xonlydate) xd, xonlyseconds From ( Select Date , DATE_ADD(DATE '1899-12-30', INTERVAL cast(FLOOR(cast(Date as FLOAT64)) as INT64) DAY ) xonlydate , cast(FLOOR( ( cast(Date as FLOAT64) - cast(FLOOR( cast(Date as FLOAT64)) as INT64) ) * 86400 ) as INT64) xonlyseconds FROM (Select '43168.682974537034' Date) -- 09.03.2018 16:23:28 ) xx1 )
For those looking how to do this in excel (outside of formatting to a date field) you can do this by using the Text function https://exceljet.net/excel-functions/excel-text-function i.e. A1 = 132134 =Text(A1,"MM-DD-YYYY") will result in a date
This worked for me because sometimes the field was a numeric to get the time portion. Command: dateadd(mi,CONVERT(numeric(17,5),41869.166666666664)*1440,'1899-12-31')
How do I match an entire day to a datetime field?
I have a table for matches. The table has a column named matchdate, which is a datetime field. If I have 3 matches on 2011-12-01: 2011-12-01 12:00:00 2011-12-01 13:25:00 2011-12-01 16:00:00 How do I query that? How do I query all matches on 1 single date? I have looked at date_trunc(), to_char(), etc. Isn't there some "select * where datetime in date" function?
Cast your timestamp value to date if you want simple syntax. Like this: SELECT * FROM tbl WHERE timestamp_col::date = '2011-12-01'; -- date literal However, with big tables this will be faster: SELECT * FROM tbl WHERE timestamp_col >= '2011-12-01 0:0' -- timestamp literal AND timestamp_col < '2011-12-02 0:0'; Reason: the second query does not have to transform every single value in the table and can utilize a simple index on the timestamp column. The expression is sargable. Note excluded the upper bound (< instead of <=) for a correct selection. You can make up for that by creating an index on an expression like this: CREATE INDEX tbl_ts_date_idx ON tbl (cast(timestamp_col AS date)); Then the first version of the query will be as fast as it gets.
not sure if i am missing something obvious here, but i think you can just
select * from table where date_trunc('day', ts) = '2011-12-01';
Just use the SQL BETWEEN function like so: SELECT * FROM table WHERE date BETWEEN '2011-12-01' AND '2011-12-02' You may need to include times in the date literals, but this should include the lover limit and exclude the upper. From rails I believe you can do: .where(:between => '2011-12-01'..'2011-12-02')