I'm trying to get several simple queries into one new table using Googe Big Query. In the final table is existing revenue data per day (that I can simply draw from another table). I then want to calculate the average revenue per day of the current month and continue this value until the end of the month. So the final table is updated every day and includes actual data and forecasted data.
So far, I came up with the following, which generates an error message in combination: Scalar subquery produced more than one element
#This gives me the date, the revenue per day and the info that it's actual data
SELECT
date, sum(revenue), 'ACTUAL' as type from `project.dataset.table` where date >"2020-01-01" and date < current_date() group by date
union distinct
# This shall provide the remaining dates of the current month
SELECT
(select calendar_date FROM `project.dataset.calendar_table` where calendar_date >= current_date() and calendar_date <=DATE_SUB(DATE_TRUNC(DATE_ADD(CURRENT_DATE(), INTERVAL 1 MONTH), MONTH), INTERVAL 1 DAY)),
#This shall provide the average revenue per day so far and write this value for each day of the remaining month
(SELECT avg(revenue_daily) FROM
(select sum(revenue) as revenue_daily from `project.dataset.table` WHERE date > "2020-01-01" and extract(month from date) = extract (month from current_date()) group by date) as average_daily_revenue where calendar >= current_date()),
'FORECAST'
How I wish the final data looks like:
+------------+------------+----------+
| date | revenue | type |
+------------+------------+----------+
| 01.04.2020 | 100 € | ACTUAL |
| … | 5.000 € | ACTUAL |
| 23.04.2020 | 200 € | ACTUAL |
| 24.04.2020 | 230,43 € | FORECAST |
| 25.04.2020 | 230,43 € | FORECAST |
| 26.04.2020 | 230,43 € | FORECAST |
| 27.04.2020 | 230,43 € | FORECAST |
| 28.04.2020 | 230,43 € | FORECAST |
| 29.04.2020 | 230,43 € | FORECAST |
| 30.04.2020 | 230,43 € | FORECAST |
+------------+------------+----------+
The forecast value is simply the sum of the actual revenue of the month divided by the number of days the month had so far.
Thanks for any hint on how to approach this.
I just figured something out, which creates the data I need. I'll still work on updating this every day automatically. But this is what I got so far:
select
date, 'actual' as type, sum(revenue) as revenue from `project.dataset.revenue` where date >="2020-01-01" and date < current_date() group by date
union distinct
select calendar_date, 'forecast',(SELECT avg(revenue_daily) FROM
(select sum(revenue) as revenue_daily from `project.dataset.revenue` WHERE extract(year from date) = extract (year from current_date()) and extract(month from date) = extract (month from current_date()) group by date order by date) as average_daily_revenue), FROM `project.dataset.calendar` where calendar_date >= current_date() and calendar_date <=DATE_SUB(DATE_TRUNC(DATE_ADD(CURRENT_DATE(), INTERVAL 1 MONTH), MONTH), INTERVAL 1 DAY) order by date
Related
I am trying to calculate a rolling average of data from incident reports. The exact quantity I'm looking for is the 30-day-mean-time-to-resolution (mttr) which means the average of the time it takes to resolve incidents in the last 30 days.
My incidents table looks something like this:
| incident_id | start_datetime | end_datetime |
|-------------|-----------------------|-----------------------|
| 1 | '2020-02-01T10:13:00' | '2020-02-01T10:59:33' |
| 2 | '2020-02-01T17:55:13' | '2020-02-02T00:35:28' |
| 3 | '2020-02-03T13:33:01' | '2020-02-03T15:54:01' |
What I want is something like this (the numbers are made up so don't try to actually calculate-- just note that the datetime intervals are every hour):
| datetime | mttr_last30days_in_hours |
|-----------------------|--------------------------|
| '2020-02-01T10:00:00' | 5.7 |
| '2020-02-01T11:00:00' | 5.6 |
| '2020-02-02T12:00:00' | 5.8 |
I can calculate the mttr in the last 30 days really easily if I'm doing it just for one point in time:
SELECT avg(end_datetime - start_datetime) mttr_last30days_in_hours
FROM incidents
WHERE datetime_diff(current_datetime(), start_datetime, DAY) <= 30
The problem is this just gives me ONE number. How do I create a time series spanning the range of say, the first incident's start_datetime (min(start_datetime)) to the current time, and then get a rolling 30 day average with evenly spaced, hourly intervals (like the example table above)?
If you have an unique field in your table, you can try doing that:
WITH
t_filter AS(
SELECT
*
FROM
incidents
WHERE datetime_diff(current_datetime(), start_datetime, DAY) <= 30
),
t_dates AS (
SELECT
unique_key,
GENERATE_DATE_ARRAY(DATE(start_datetime), CURRENT_DATE(), INTERVAL 1 DAY) AS date_array
FROM
t_filter
),
t_hour AS (
SELECT *
FROM
UNNEST(["00:00:00",
"01:00:00",
"02:00:00",
"03:00:00",
"04:00:00",
"05:00:00",
"06:00:00",
"07:00:00",
"08:00:00",
"09:00:00",
"10:00:00",
"11:00:00",
"12:00:00",
"13:00:00",
"14:00:00",
"15:00:00",
"16:00:00",
"17:00:00",
"18:00:00",
"19:00:00",
"20:00:00",
"21:00:00",
"22:00:00",
"23:00:00"]) h
),
sequence AS(
SELECT
unique_key,
CONCAT(CAST(arr AS string),"T", h) date_hour
FROM
t_dates,
UNNEST(date_array) arr,
t_hour
)
SELECT
date_hour,
AVG(end_datetime - start_datetime)
FROM
sequence
LEFT JOIN
t_filter
ON
incidents.unique_key = sequence.unique_key
GROUP BY
date_hour
I hope it helps
Given a table as such:
# SELECT * FROM payments ORDER BY payment_date DESC;
id | payment_type_id | payment_date | amount
----+-----------------+--------------+---------
4 | 1 | 2019-11-18 | 300.00
3 | 1 | 2019-11-17 | 1000.00
2 | 1 | 2019-11-16 | 250.00
1 | 1 | 2019-11-15 | 300.00
14 | 1 | 2019-10-18 | 130.00
13 | 1 | 2019-10-18 | 100.00
15 | 1 | 2019-09-18 | 1300.00
16 | 1 | 2019-09-17 | 1300.00
17 | 1 | 2019-09-01 | 400.00
18 | 1 | 2019-08-25 | 400.00
(10 rows)
How can I SUM the amount column based on an arbitrary date range, not simply a date truncation?
Taking the example of a date range beginning on the 15th of a month, and ending on the 14th of the following month, the output I would expect to see is:
payment_type_id | payment_date | amount
-----------------+--------------+---------
1 | 2019-11-15 | 1850.00
1 | 2019-10-15 | 230.00
1 | 2019-09-15 | 2600.00
1 | 2019-08-15 | 800.00
Can this be done in SQL, or is this something that's better handled in code? I would traditionally do this in code, but looking to extend my knowledge of SQL (which at this stage, isnt much!)
Click demo:db<>fiddle
You can use a combination of the CASE clause and the date_trunc() function:
SELECT
payment_type_id,
CASE
WHEN date_part('day', payment_date) < 15 THEN
date_trunc('month', payment_date) + interval '-1month 14 days'
ELSE date_trunc('month', payment_date) + interval '14 days'
END AS payment_date,
SUM(amount) AS amount
FROM
payments
GROUP BY 1,2
date_part('day', ...) gives out the current day of month
The CASE clause is for dividing the dates before the 15th of month and after.
The date_trunc('month', ...) converts all dates in a month to the first of this month
So, if date is before the 15th of the current month, it should be grouped to the 15th of the previous month (this is what +interval '-1month 14 days' calculates: +14, because the date_trunc() truncates to the 1st of month: 1 + 14 = 15). Otherwise it is group to the 15th of the current month.
After calculating these payment_days, you can use them for simple grouping.
I would simply subtract 14 days, truncate the month, and add 14 days back:
select payment_type_id,
date_trunc('month', payment_date - interval '14 day') + interval '14 day' as month_15,
sum(amount)
from payments
group by payment_type_id, month_15
order by payment_type_id, month_15;
No conditional logic is actually needed for this.
Here is a db<>fiddle.
You can use the generate_series() function and make a inner join comparing month and year, like this:
SELECT specific_date_on_month, SUM(amount)
FROM (SELECT generate_series('2015-01-15'::date, '2015-12-15'::date, '1 month'::interval) AS specific_date_on_month)
INNER JOIN payments
ON (TO_CHAR(payment_date, 'yyyymm')=TO_CHAR(specific_date_on_month, 'yyyymm'))
GROUP BY specific_date_on_month;
The generate_series(<begin>, <end>, <interval>) function generate a serie based on begin and end with an specific interval.
I have an Oracle Database I am trying to query multiple date fields by dates and get the totals by month and year as output.
This was my original query. This just gets what I want for the dates I want to input.
SELECT COUNT(*) as Total
FROM Some_Table s
WHERE (s.Start_DATE <= TO_Date ('2019/09/01', 'YYYY/MM/DD'))
AND (s.End_DATE IS NULL OR (s.End_DATE > TO_Date ('2019/08/31', 'YYYY/MM/DD')))
I would like to get an output where it gives me a count by Month and Year. The count would be the number between the Start_DATE (beginning of the month) and the End_DATE (end of the month).
I can't do
Edit: this was an example from another query and has no relation to the query above. I was just trying to provide an example of what I cannot do because I have two separate date fields. The example below was stating my knowledge of extracting month and year from a single date field. Sorry for the confusion.
SELECT extract(year from e.DATE_OCCURRED) as Year
,to_char(e.DATE_OCCURRED, 'MONTH') as Month
,count (*) as totals
because the Start_DATE and End_DATE are two separate fields.
Any help would be appreciated
Edit: Example would be
----------------------------------
| Name | Start_DATE | End_DATE |
----------------------------------
| John | 01/16/2018 | 07/09/2019 |
| Sue | 06/01/2015 | 09/01/2018 |
| Joe | 04/06/2016 | Null |
----------------------------------
I want to know my total number of workers that would have been working by month and year. Would want the output to look like.
------------------------
| Year | Month | Total |
------------------------
| 2016 | Aug | 2 |
| 2018 | May | 3 |
| 2019 | Aug | 2 |
------------------------
So I know I had two workers working in August 2016 and three in May 2018.
Do you want this?
SELECT count(*)
from some_table
where year(e.DATE_OCCURRED) > year(start_date)
and year(e.DATE_OCCURRED) < year(end_date)
and month(e.DATE_OCCURRED) > month(start_date)
and month(e.DATE_OCCURRED) < month(end_date)
note: using month and year functions is generally better when working with dates. If you convert to characters you might find that January comes after February (as an example) since J comes after F in the alphabet.
Are you looking for this?(Hoping that end_date > start_date)
select extract (year from end_dt2)- extract(YEAR from st_dt1) as YearDiff ,
extract (month from end_dt2)- extract (month from st_dt1) as monthDiff from tab;
I want to find the number of unique users active in the last 30 days. I want to calculate this for today, but also for days in the past. The dataset contains user ids, dates and events triggered by the user saved in BigQuery. A user is active by opening a mobile app triggering the event session_start. Example of the unnested dataset.
| resettable_device_id | date | event |
------------------------------------------------------
| xx | 2017-06-09 | session_start |
| yy | 2017-06-09 | session_start |
| xx | 2017-06-11 | session_start |
| zz | 2017-06-11 | session_start |
I found a solution which suits my problem:
BigQuery: how to group and count rows within rolling timestamp window?
My BigQuery script so far:
#standardSQL
WITH daily_aggregation AS (
SELECT
PARSE_DATE("%Y%m%d", event_dim.date) AS day,
COUNT(DISTINCT user_dim.device_info.resettable_device_id) AS unique_resettable_device_ids
FROM `ANDROID.app_events_*`,
UNNEST(event_dim) AS event_dim
WHERE event_dim.name = "session_start"
GROUP BY day
)
SELECT
day,
unique_resettable_device_ids,
SUM(unique_resettable_device_ids)
OVER(ORDER BY UNIX_SECONDS(TIMESTAMP(day)) DESC ROWS BETWEEN 2592000 PRECEDING AND CURRENT ROW) AS unique_ids_rolling_30_days
FROM daily_aggregation
ORDER BY day
This script results in the following table:
| day | unique_resettable_device_ids | unique_ids_rolling_30_days |
------------------------------------------------------------------------
| 2018-06-05 | 1807 | 2614 |
| 2018-06-06 | 711 | 807 |
| 2018-06-07 | 96 | 96 |
The problem is that the column unique_ids_rolling_30_days is just a cumulative sum of the column unique_resettable_device_ids. How can I fix the rolling window function in my script?
"The problem is that the column unique_ids_rolling_30_days is just a cumulative sum of the column unique_resettable_device_ids."
Of course, as that's exactly what the code
SUM(unique_resettable_device_ids)
OVER(ORDER BY UNIX_SECONDS(TIMESTAMP(day)) DESC ROWS BETWEEN 2592000 PRECEDING AND CURRENT ROW) AS unique_ids_rolling_30_days
is asking for.
Check out https://stackoverflow.com/a/49866033/132438 where the question asks about specifically counting uniques in a rolling window: Turns out it's a very slow operation given how much memory it requires.
The solution for this when you want a rolling count of uniques: Go for approximate results.
From the linked answer:
#standardSQL
SELECT DATE_SUB(date, INTERVAL i DAY) date_grp
, HLL_COUNT.MERGE(sketch) unique_90_day_users
, HLL_COUNT.MERGE(DISTINCT IF(i<31,sketch,null)) unique_30_day_users
, HLL_COUNT.MERGE(DISTINCT IF(i<8,sketch,null)) unique_7_day_users
, COUNT(*) window_days
FROM (
SELECT DATE(creation_date) date, HLL_COUNT.INIT(owner_user_id) sketch
FROM `bigquery-public-data.stackoverflow.posts_questions`
WHERE EXTRACT(YEAR FROM creation_date)=2017
GROUP BY 1
), UNNEST(GENERATE_ARRAY(1, 90)) i
GROUP BY 1
HAVING window_days=90
ORDER BY date_grp
Working solution for a weekly calculation of the number of active users in the last 30 days.
#standardSQL
WITH days AS (
SELECT day
FROM UNNEST(GENERATE_DATE_ARRAY('2018-01-01', CURRENT_DATE(), INTERVAL 1 WEEK)) AS day
), periods AS (
SELECT
DATE_SUB(days.day, INTERVAL 30 DAY) AS StartDate,
days.day AS EndDate FROM days
)
SELECT
periods.EndDate AS Day,
COUNT(DISTINCT user_dim.device_info.resettable_device_id) as resettable_device_ids
FROM `ANDROID.app_events_*`,
UNNEST(event_dim) AS event_dim
CROSS JOIN periods
WHERE
PARSE_DATE("%Y%m%d", event_dim.date) BETWEEN periods.StartDate AND periods.EndDate
AND event_dim.name = "session_start"
GROUP BY Day
ORDER BY Day DESC
I'm tasked with pulling the data for the four recent quarters. If I was dealing with dates this would be easy, but I'm not sure how to do so when I have a quarters table that looks like this:
| quarter | year |
+---------+------+
| 1 | 2016 |
| 2 | 2016 |
| 3 | 2016 |
...
I know that I can get the current quarter by doing something like this:
SELECT *
FROM quarters
WHERE quarter = (EXTRACT(QUARTER FROM CURRENT_DATE))
AND year = (EXTRACT(YEAR FROM CURRENT_DATE));
However, I'm not sure the best way to get the four most recent quarters. I thought about getting this quarter from last year, and selecting everything since then, but I don't know how to do that with tuples like this. My expected results would be:
| quarter | year |
+---------+------+
| 1 | 2017 |
| 2 | 2017 |
| 3 | 2017 |
| 4 | 2017 |
Keep in mind they won't always be the same year - in Q12018 this will change.
I've built a SQLFiddle that can be used to tinker with this - http://sqlfiddle.com/#!17/0561a/1
Here is one method:
select quarter, year
from quarters
order by year desc, quarter desc
fetch first 4 rows only;
This assumes that the quarters table only has quarters with data in it (as your sample data suggests). If the table has future quarters as well, then you need to compare the values to the current date:
select quarter, year
from quarters
where year < extract(year from current_date) or
(year = extract(year from current_date) and
quarter <= extract(quarter from current_date)
)
order by year desc, quarter desc
fetch first 4 rows only;
For the case that there can be gaps, like 2/2017 missing, and one would then want to return only three quarters instead of four, one can turn years and quarters into consecutive numbers by multiplying the year by four and adding the quarters.
select *
from quarters
where year * 4 + quarter
between extract(year from current_date) * 4 + extract(quarter from current_date) - 3
and extract(year from current_date) * 4 + extract(quarter from current_date)
order by year desc, quarter desc;