Good day. I have a table that is collecting data in Universal Time Coordinated timestamps. However, the location is 9 hours back from this time. I am writing a query that gets the time-stamp and the value but 'casts' the timestamp 9 hours back since thats when it got recorded with respect to that location.
My issue is that I keep subtracting days not hours even though I specified hours in my 'datediff' and 'dateadd'. How do I select a timestamp and the value but represent that timestamp as 9 hours back? Thanks for any help.
select DATEADD(hour, DATEDIFF(hour,9,TimeUTC),0) as DateActual, Value
From TableData
Data
2015-12-15 00:00:00 45
2015-12-15 00:00:00 54
Current results
2015-12-06 00:00:00 45
2015-12-06 00:00:00 54
Desired results
2015-12-14 15:00:00 45
2015-12-14 15:00:00 54
Related
I have a table as:
Id start_timestamp end_timestamp
1 2021-07-12 03:00:00 2021-07-13 11:58:05
2 2021-07-13 04:00:00 2021-07-13 05:00:00
3 2021-07-13 04:00:00 2021-07-13 09:00:00
4 2021-07-13 04:00:00 NULL
5 2020-04-10 04:00:00 2020-04-10 04:01:00
....
I want to find all records that fall between two specific timestamps? Basically I'm looking to understand what process ran during a high pick time of the day (it doesn't matter if they have 1 sec in the window or hours.. just occurrence in the window is enough)
So if the timestamps are 2021-07-13 00:00:00 to 2021-07-13 04:30:00
The query will return
1
2
3
4
How can I do that with SQL? (Preferably Presto)
This is the overlapping range problem. You may use:
SELECT *
FROM yourTable
WHERE
(end_timestamp > '2021-07-13 00:00:00' OR end_timestamp IS NULL) AND
(start_timestamp < '2021-07-13 04:30:00' OR start_timestamp IS NULL);
My answer assumes that a missing start/end timestamp value in the table logically means that this value should not be considered. This seems to be the logic you want here.
I'm trying to create a timeseries in google bigquery SQL. My data is a series of time ranges covering the period of activity for that record. Here is an example:
Start End
2020-11-01 21:04:00 UTC 2020-11-02 07:15:00 UTC
2020-11-01 21:45:00 UTC 2020-11-02 04:00:00 UTC
2020-11-01 22:00:00 UTC 2020-11-02 09:48:00 UTC
2020-11-01 22:00:00 UTC 2020-11-02 06:00:00 UTC
I wish to create a new table to total the number of active records within a 15 minute block. "21:00:00" would for example be 21:00 to 21:14.59. My desired output for the above would be:
Period Active_Records
2020-11-01 21:00:00 1
2020-11-01 21:15:00 1
2020-11-01 21:30:00 1
2020-11-01 21:45:00 2
2020-11-01 22:00:00 4
2020-11-01 22:15:00 4
etc until the end of the last active range.
I would also like to be able to generate this on the fly by querying a date range and having it return every 15 minute block in the range and how many active records there was in that period.
Any assistance would be greatly appreciated.
Below is for BigQuery Standard SQL
#standardSQL
select ts as period, count(1) as Active_Records
from unnest((
select generate_timestamp_array(timestamp_trunc(min(start), hour), max(`end`), interval 15 minute)
from `project.dataset.table`
)) ts
join `project.dataset.table`
on not (`end` < ts or start > timestamp_add(ts, interval 15 * 60 - 1 second))
group by ts
if to apply to sample data from your question - output is
I have two data frames in csv files. First data described traffic incidents (df1) and second data has the traffic record data for each 15 minutes(df2). I want to merge between them based on the closest time. I used python pandas_merge_asof and I got the nearest match. but I want the 30 minutes records before and after the match from the traffic record data. And I want to join the closest incidents to the traffic data time. if the incidents occured 14:02:00, it will be mereged with the traffic date that recorded at 14:00:00
For example:
1- Incidents data
Date detector_id Inident_type
09/30/2015 8:00:00 1 crash
09/30/2015 8:02:00 1 congestion
04/22/2014 15:30:00 9 congestion
04/22/2014 15:33:00 9 Emergency vehicle
2 - Traffic data
Date detector_id traffic_volume
09/30/2015 7:30:00 1 55
09/30/2015 7:45:00 1 45
09/30/2015 8:00:00 1 60
09/30/2015 8:15:00 1 200
09/30/2015 8:30:00 1 70
04/22/2014 15:00:00 9 15
04/22/2014 15:15:00 9 7
04/22/2014 15:30:00 9 50
04/22/2014 15:45:00 9 11
04/22/2014 16:00:00 9 7
2- the desired table
Date detector_id traffic_volume Incident_type
09/30/2015 7:30:00 1 55 NA
09/30/2015 7:45:00 1 45 NA
09/30/2015 8:00:00 1 60 Crash
09/30/2015 8:00:00 1 60 congestion
09/30/2015 8:15:00 1 200 NA
09/30/2015 8:30:00 1 70 NA
04/22/2014 15:00:00 9 15 NA
04/22/2014 15:15:00 9 7 NA
04/22/2014 15:30:00 9 50 Congestion
04/22/2014 15:30:00 9 50 Emergency vehicle
04/22/2014 15:45:00 9 11 NA
04/22/2014 16:00:00 9 7 NA
The code that I used as follow
Merge = pd.merge_asof(df2, df1, left_index = True, right_index = True, allow_exact_maches = False,
on='Date', by='detector_id', direction='nearest')
but it gave me this table.
Date detector_id traffic_volume Incident_type
09/30/2015 8:00:00 1 60 Crash
04/22/2014 15:30:00 9 50 Congestion
and I want to know the situation after and before the incidents occur.
Any Idea?
Thank you.
*If I made mistake by asking like this way, please let me know.
For anyone has the same problem and want to do merge by using pandas.merge_asof, you have to use the Tolerance function. This function helps you adjust the time different between the two datasets.
But you may face two problems related to Timedelta and sorting index. so the solution of Timedelta is converting the time to datetime as follow:
df1.Date = pd.to_datetime(df1.Date)
df2.Date = pd.to_datetime(df2.Date)
and the sorting index you need apply sort in your main code as follow:
x = pd.merge_asof(df1.sort_values('Date'), #sort_values fix the error"left Key must be sorted"
df2.sort_values('Date'),
on = 'Date',
by = 'Detector_id',
direction = 'backward',
tolerance =pd.Timedelta('45 min'))
The direction could be nearest which in my case will select all the records accord before and after the match records within 45 minutes.
The direction could be backward will merge all records within 45 minutes after the exact or nearest match
and Forward will select all the records within 45 minutes before the exact or nearest match.
Thank you and hopefully this will help anyone in future.
I am using SQL Server 2014.
What I'm trying to do is add a new time to an old datetime.
I'm not even sure if it's possible but I thought I'd ask the experts.
So these are what my columns look like:
CurrentDate | Hours | NewDate
2017-03-10 08:00:00 | 25 | ??
2017-01-01 10:00:00 | 27 | ??
What I want is the Hours to be converted to days and hours so it can be added to the CurrentDate to create a NewDate.
So the NewDate would be: 2017-03-11 09:00:00 because 25 hours equates to 1 day and 1 hour. And the second NewDate would be: 2017-01-02 01:00:00 because 27 equates to 1 day and 3 hours.
I actually don't think this is possible and there's a chance I might have to put the hours already converted into days and times but if that's the case, how would I write 25 hours? Would it be 00-00-01 01:00:00? And would 27 hours be 00-00-01 03:00:00 and then just add those values into CurrentDate?
Thanks! Feel free to tell me this has been asked before (I looked, but couldn't find anything as unique as this or maybe I didn't look hard enough) or if this can't be done.
You can simply use DATEADD, no need to convert the hours to days first:
SELECT CurrentDate,
Hours,
DATEADD(HOUR,Hours,CurrentDate) NewDate
FROM dbo.YourTable;
You can try this:
select DATEADD(HOUR,25,'2017-03-10 08:00:00') -- 2017-03-11 09:00:00.000
select DATEADD(HOUR,27,'2017-01-01 10:00:00') -- 2017-01-02 13:00:00.000
I've got data in ten minutes intervals in my table:
2009-01-26 00:00:00 12
2009-01-26 00:10:00 1.1
2009-01-26 00:20:00 11
2009-01-26 00:30:00 0
2009-01-26 00:40:00 5
2009-01-26 00:50:00 3.4
2009-01-26 01:00:00 7
2009-01-26 01:10:00 7
2009-01-26 01:20:00 7.2
2009-01-26 01:30:00 3
2009-01-26 01:40:00 25
2009-01-26 01:50:00 4
2009-01-26 02:00:00 3
2009-01-26 02:10:00 4
etc.
Is it possible to formulate a single SQL-query for MySQL which will return a series of averages over each hour?
In this case it should return:
5.42
8.87
etc.
It's unclear whether you want the average to be aggregated over days or not.
If you want a different average for midnight on the 26th vs midnight on the 27th, then modify Mabwi's query thus:
SELECT AVG( value ) , thetime
FROM hourly_averages
GROUP BY DATE( thetime ), HOUR( thetime )
Note the additional DATE() in the GROUP BY clause. Without this, the query would average together all of the data from 00:00 to 00:59 without regard to the date on which it happened.
This should work:
SELECT AVG( value ) , thetime
FROM hourly_averages
GROUP BY HOUR( thetime )
Here's the result
AVG(value) thetime
5.4166666865349 2009-01-26 00:00:00
8.8666666348775 2009-01-26 01:00:00
3.5 2009-01-26 02:00:00
There is also another possibility considering the fact that dates have a string representation in the database:
You can use SUBSTRING(thetime, 1, [len]), extracting the common part of your group. For the example with hourly averages you have the SQL query
SELECT SUBSTRING(thetime, 1, 13) AS hours, AVG(value) FROM hourly_averages GROUP BY hours
By the len parameter you can specify the aggregated time interval considering the MySQL date format yyyy-MM-dd HH:mm:ss[.SS...]:
len = 4: group by years
len = 7: group by months
len = 10: group by days
len = 13: group by hours
len = 16: group by minutes
len = 19: group by seconds
We encountered a better performance of this method over using date and time function, especially when used in JOINs in MySQL 5.7. However in MySQL 8 at least for grouping both ways seem to take approximately the same time.