GROUP BY several hours - sql

I have a table where our product records its activity log. The product starts working at 23:00 every day and usually works one or two hours. This means that once a batch started at 23:00, it finishes about 1:00am next day.
Now, I need to take statistics on how many posts are registered per batch but cannot figure out a script that would allow me achiving this. So far I have following SQL code:
SELECT COUNT(*), DATEPART(DAY,registrationtime),DATEPART(HOUR,registrationtime)
FROM RegistrationMessageLogEntry
WHERE registrationtime > '2014-09-01 20:00'
GROUP BY DATEPART(DAY, registrationtime), DATEPART(HOUR,registrationtime)
ORDER BY DATEPART(DAY, registrationtime), DATEPART(HOUR,registrationtime)
which results in following
count day hour
....
1189 9 23
8611 10 0
2754 10 23
6462 11 0
1885 11 23
I.e. I want the number for 9th 23:00 grouped with the number for 10th 00:00, 10th 23:00 with 11th 00:00 and so on. How could I do it?

You can do it very easily. Use DATEADD to add an hour to the original registrationtime. If you do so, all the registrationtimes will be moved to the same day, and you can simply group by the day part.
You could also do it in a more complicated way using CASE WHEN, but it's overkill on the view of this easy solution.

I had to do something similar a few days ago. I had fixed timespans for work shifts to group by where one of them could start on one day at 10pm and end the next morning at 6am.
What I did was:
Define a "shift date", which was simply the day with zero timestamp when the shift started for every entry in the table. I was able to do so by checking whether the timestamp of the entry was between 0am and 6am. In that case I took only the date of this DATEADD(dd, -1, entryDate), which returned the previous day for all entries between 0am and 6am.
I also added an ID for the shift. 0 for the first one (6am to 2pm), 1 for the second one (2pm to 10pm) and 3 for the last one (10pm to 6am).
I was then able to group over the shift date and shift IDs.
Example:
Consider the following source entries:
Timestamp SomeData
=============================
2014-09-01 06:01:00 5
2014-09-01 14:01:00 6
2014-09-02 02:00:00 7
Step one extended the table as follows:
Timestamp SomeData ShiftDay
====================================================
2014-09-01 06:01:00 5 2014-09-01 00:00:00
2014-09-01 14:01:00 6 2014-09-01 00:00:00
2014-09-02 02:00:00 7 2014-09-01 00:00:00
Step two extended the table as follows:
Timestamp SomeData ShiftDay ShiftID
==============================================================
2014-09-01 06:01:00 5 2014-09-01 00:00:00 0
2014-09-01 14:01:00 6 2014-09-01 00:00:00 1
2014-09-02 02:00:00 7 2014-09-01 00:00:00 2

If you add one hour to registrationtime, you will be able to group by the date part:
GROUP BY
CAST(DATEADD(HOUR, 1, registrationtime) AS date)
If the starting hour must be reflected accurately in the output (as 9, 23, 10, 23 rather than as 10, 0, 11, 0), you could obtain it as MIN(registrationtime) in the SELECT clause:
SELECT
count = COUNT(*),
day = DATEPART(DAY, MIN(registrationtime)),
hour = DATEPART(HOUR, MIN(registrationtime))
Finally, in case you are not aware, you can reference columns by their aliases in ORDER BY:
ORDER BY
day,
hour
just so that you do not have to repeat the expressions.

The below query will give you what you are expecting..
;WITH CTE AS
(
SELECT COUNT(*) Count, DATEPART(DAY,registrationtime) Day,DATEPART(HOUR,registrationtime) Hour,
RANK() over (partition by DATEPART(HOUR,registrationtime) order by DATEPART(DAY,registrationtime),DATEPART(HOUR,registrationtime)) Batch_ID
FROM RegistrationMessageLogEntry
WHERE registrationtime > '2014-09-01 20:00'
GROUP BY DATEPART(DAY, registrationtime), DATEPART(HOUR,registrationtime)
)
SELECT SUM(COUNT) Count,Batch_ID
FROM CTE
GROUP BY Batch_ID
ORDER BY Batch_ID

You can write a CASE statement as below
CASE WHEN DATEPART(HOUR,registrationtime) = 23
THEN DATEPART(DAY,registrationtime)+1
END,
CASE WHEN DATEPART(HOUR,registrationtime) = 23
THEN 0
END

Related

Calculating difference (or deltas) between current and previous row with clickhouse

It would be awesome if there was a way to index rows during a query.
Is there a way to SELECT (compute) the difference of a single column between consecutive rows?
Let's say, something like the following query
SELECT
toStartOfDay(stamp) AS day,
count(day ) AS events ,
day[current] - day[previous] AS difference, -- how do I calculate this
day[current] / day[previous] as percent, -- and this
FROM records
GROUP BY day
ORDER BY day
I want to get the integer and percentage difference between the current row's 'events' column and the previous one for something similar to this:
day
events
difference
percent
2022-01-06 00:00:00
197
NULL
NULL
2022-01-07 00:00:00
656
459
3.32
2022-01-08 00:00:00
15
-641
0.02
2022-01-09 00:00:00
7
-8
0.46
2022-01-10 00:00:00
137
130
19.5
My version of Clickhouse doesn't support window-function but, on looking about the LAG() function mentioned in the comments, I found neighbor(), which works perfectly for what I'm trying to do
SELECT
toStartOfDay(stamp) AS day,
count(day ) AS events ,
(events - neighbor(events, -1)) as diff,
(events / neighbor(events, -1)) as perc
FROM records
GROUP BY day
ORDER BY day

how to return a specific set of data from multiple columns in a database in sql

I am new to sql and this is my first ever question. I am working with a sample database that I want to extract specific information from to display as a dashboard. The issue is that I can do this partially but I cannot seem to figure it out properly.
``SELECT
S_date as date,
p_time as time,
process_id as process,
sc_gun as scannumb,
sum(line_qty) as linetotal,
sum(area_qty) as areatotal
FROM dbfile6
WHERE
process_id in('0010','0020','0030')
and sc_gun in = ('10','20','30','40','50')
and s_date = curdate() - 1 and p_time between '22:00:00' and '23:59:59'
or s_date = curdate() and p_time between '00:00:00' and '06:00:00'
GROUP BY p_time, s_date, process_id, sc_gun
ORDER BY s_date, process_id
What do I want to display?
I can do partially where I want it to work to yesterdays date (s_date) recurring but I want this to only happen Monday to Friday, skipping the weekend so when we are on Monday, it looks at Fridays data from the database.
I want to show the time as a range, a night range. The range is 20:00:00 - 06:00:00. The range is tricky as it crosses over to the next day, this could work for Monday to Thursday but not Friday as there is no working weekend so what would I do here? In addition to this, I can sum up the values successfully and display it as averages for each process but then once I add the time in, it displays each result individually.
The table below is what it looks like in the database, however as mentioned earlier, the desired result is for each process to have the line_qty and area_qty summed up by time range and a day and night cycle.
s_date
p_time
process_id
sc_gun
line_qty
area_qty
04/05/2022
04:49:52
0010
10
2
12
03/05/2022
11:50:00
0010
10
5
14
03/05/2022
19:50:00
0010
10
7
16
03/05/2022
13:50:00
0020
20
4
6
03/05/2022
19:50:00
0010
10
7
16

Query help : Running Avg for last 15 days

please help me with a query to find running Avg for every 15 days. I have used below query but not sure how to display only 15 days Avg.
Select Date,
Avg(Qty) OVER (ORDER BY Date ROWS BETWEEN 15 PRECEDING AND CURRENT ROW) AS RunningAvg
FROM Sample
Sample Table : (Contains Qty for each Day)
Date Qty
2014-10-01 4
2014-10-02 5
..
..
2014-12-31 4
Expected Result.
Date RunningAvg
2014-10-01 4
2014-10-15 XX
2014-11-01 XX
2014-11-15 XX
2014-12-01 XX
.
.
.
I'm a bit baffled by the question. Your results seem to suggest that you want the values on the 1st and 15th of the month -- and that has nothing to do with 15-day moving averages. For such filtering you can use:
select t.*
from t
where day(date) in (1, 15);
As you know, some months have 28, 29, or 31 days so "15 days" has nothing to do with the day of the months. And the number of days between the 1st and 15th is 14 days, not 15.

SQL how to count but only count one instance if two columns match?

Wondering how to select from a table:
FIELDID personID purchaseID dateofPurchase
--------------------------------------------------
2 13 147 2014-03-21 00:00:00
3 15 165 2015-03-23 00:00:00
4 13 456 2018-03-24 00:00:00
5 1 133 2018-03-21 00:00:00
6 23 123 2013-03-22 00:00:00
7 25 456 2013-03-21 00:00:00
8 25 456 2013-03-23 00:00:00
9 22 456 2013-03-28 00:00:00
10 25 589 2013-03-21 00:00:00
11 82 147 1991-10-22 00:00:00
12 82 453 2003-03-22 00:00:00
I'd like to get a result table of two columns: weekday and the number of purchases of each weekday, but only count the distinct days of purchases if done by the same person on the same day - for example since personID 25 purchased two things on 2013-03-21, that should only count as one 'thursday' instead of 2.
Basically, if the personID and the dateofPurchase are the same for more than one row, only count it once is what I want.
Here is what I have currently: It does everything correctly except it will count the above scenario under the thursday twice, when I would only want to add one:
SELECT v.wkday as day, COUNT(*) as 'absences'
FROM dbo.AttendanceRecord pr CROSS APPLY
(VALUES (CASE WHEN DATEPART(WEEKDAY, date) IN (1, 7)
THEN 'Weekend'
ELSE DATENAME(WEEKDAY, date)
END)
) v(wkday)
GROUP BY v.wkday;
to clarify:
If an item is purchased for at least one puchaseID on a specific day they will be counted as purchased for that day, and do not need to be counted again for each new purchase ID on that day.
I think you want to count distinct persons, so that would be:
COUNT(DISTINCT personid) as absences
Note that single quotes are not appropriate around column aliases. If you need to escape them, use square braces.
EDIT:
If you want to count distinct person-days, then you can use:
COUNT(DISTINCT CONCAT(personid, ':', dateofpurchase) as absences

SQL - Normalising timestamps to business hours

My initial answer to this problem has been to script it. Instead of using SQL, I've dipped into Python and normalised them. I am curious whether anyone can come up with a solution using SQL though.
If a date occurs outside of business hours, I want to normalise the date to the next working day. I'll keep this really simple and say that business hours is 9am to 6pm Monday to Friday. Anything outside of those hours is outside of business hours.
What should happen the dates is that they are changed so that 2pm on Saturday becomes 9am on Monday morning (the first legitimate time in the business week). 7pm on a Wednesday becomes 9am Thursday morning. etc. etc. Let's ignore holidays.
Sample data:
mysql> select mydate from mytable ORDER by mydate;
+---------------------+
| mydate |
+---------------------+
| 2009-09-13 17:03:09 |
| 2009-09-14 09:45:49 |
| 2009-09-15 09:57:28 |
| 2009-09-16 21:55:01 |
+---------------------+
4 rows in set (0.00 sec)
The first date is a Sunday so it should be normalised to 2009-09-14 09:00:00
The second date is fine, it's at 9am on a Monday.
The third date is fine, it's at 9am on a Tuesday.
The fourth date is at 9pm (outside of our 9am to 6pm business hours) on a Wednesday and should be transformed to 9am Thursday morning.
I think you're better off with your Python solution ... but I like challenges :)
select mydate
, case dayadjust
-- BUG
-- when 0 then mydate
-- BUG
when 0 then case
when hour(mydate)<9
then date_add(from_days(to_days(mydate)),
INTERVAL 9 HOUR)
else mydate
end
-- BUG SQUASHED
else date_add(from_days(to_days(mydate) + dayadjust),
INTERVAL 9 HOUR)
end as mynewdate
from (
select mydate
, case
when addday>=moreday then addday
else moreday
end as dayadjust
from (
select mydate
, weekday(mydate) as w
, hour(mydate) as h
, case weekday(mydate)
when 6 then 1
when 5 then 2
when 4 then
case
when hour(mydate) >= 18 then 3
else 0
end
else 0
end as addday
, case when hour(mydate)>=18 then 1 else 0 end as moreday
from mytable
order by mydate
) alias1
) alias2
Tested on MySQL
$ mysql tmp < phil.sql
mydate mynewdate
2009-09-12 17:03:09 2009-09-14 09:00:00
2009-09-12 21:03:09 2009-09-14 09:00:00
2009-09-13 17:03:09 2009-09-14 09:00:00
2009-09-14 09:45:49 2009-09-14 09:45:49
2009-09-15 09:57:28 2009-09-15 09:57:28
2009-09-16 21:55:01 2009-09-17 09:00:00
2009-09-17 11:03:09 2009-09-17 11:03:09
2009-09-17 22:03:09 2009-09-18 09:00:00
2009-09-18 12:03:09 2009-09-18 12:03:09
2009-09-18 19:03:09 2009-09-21 09:00:00
2009-09-19 06:03:09 2009-09-21 09:00:00
2009-09-19 16:03:09 2009-09-21 09:00:00
2009-09-19 19:03:09 2009-09-21 09:00:00
Not sure why you want to do this, but if it needs to always be true of all data in your database, you need a trigger. I would set up a table to pull from that specifies the business hours and you can use that table to determine the next valid business hour day and time. (I might even consider making a table that tells you exactly what the next business day and hour is for each possibility, it's not like this changes a lot, might have to be updated once a year if you change holidays for the next year or if you change the overall business hours. By precalulating, you can probably save time in processing this.). I would also conmtinue to use your script becasue it's better to fix data before it gets entered, but you need the trigger to ensure that data from any source (and sooner or later there will be changes form sources other than your application) meets the data integrity rules.
I don't think you can do it in one query, but you can try this:
-- Mon-Thu, after 17:00
-- Set date = next day, 9:00
UPDATE
myTable
SET
mydate = DATE_ADD(DATE_ADD(DATE(date), INTERVAL 1 DAY), INTERVAL 9 HOUR)
WHERE
TIME(mydate) >= 17
AND DAYOFWEEK(mydate) IN (1,2,3,4)
-- Mon-Fri, before 9:00
-- Set date = the same day, 9:00
UPDATE
myTable
SET
mydate = DATE_ADD(DATE(date), INTERVAL 9 HOUR)
WHERE
TIME(mydate) < 9
AND DAYOFWEEK(mydate) IN (1,2,3,4,5)
-- Fri, after 17:00, Sat, Sun
-- Set date = monday, 9.00
UPDATE
myTable
SET
mydate = DATE_ADD(DATE_ADD(DATE(date), INTERVAL 3 DAY), INTERVAL 9 HOUR)
WHERE
(TIME(mydate) >= 17
AND DAYOFWEEK(mydate) = 5)
OR DAYOFWEEK(mydate) IN (6,7)