How does the group by function works in PostgreSQL? (beginner) - sql

I don't know much at all about SQL, I've just toyed with it here and there through the years but never really 'used' it.
I'm trying to get a list of prices / volumes and aggregate them:
CREATE TABLE IF NOT EXISTS test (
ts timestamp without time zone NOT NULL,
price decimal NOT NULL,
volume decimal NOT NULL
);
what I'd like is to extract:
min price
max price
sum volume
sum (price * volume) / sum (volume)
By 1h slices
If I forget about the last line for now, I have:
SELECT MIN(price) min_price, MAX(price) max_price, SUM(volume) sum_vol, date_trunc('hour', ts) ts_group FROM test
GROUP BY ts_group;
My understanding is that 'GROUP BY ts_group' will calculate ts_group, build groups of rows and then apply the MIN / MAX / SUM functions after. Since the syntax doesn't make any sense to me (entries on the select line would be treated differently while being declared together vs. building groups and then declaring operations on the groups), I could be dramatically wrong here.
But that will not return the min_price, max_price and sum_vol results after the grouping; I get ts, price and volume in the results.
If I remove the GROUP BY line to try to see all the output, I get the error:
column "test.ts" must appear in the GROUP BY clause or be used in an aggregate function
Which I don't really understand either...
I looked at:
must appear in the GROUP BY clause or be used in an aggregate function but I don't really get it
and I looked at the doc (https://www.postgresqltutorial.com/postgresql-group-by/) which shows working example, but doesn't really clarify what is wrong with what I'm trying to do here.
While I'd be happy to have a working solution, I'm more looking from an explanation, or pointers toward good resources, that would allow me to understand this.
I have this working solution:
SELECT MIN(price) min_price, MAX(price) max_price, SUM(volume) sum_vol, (SUM(price * volume)/SUM(volume)) vwap FROM test
GROUP BY date_trunc('hour', ts);
but I still don't understand the error message from my question

All of your expressions in SQL must use data elements and functions that are known to PostgreSQL. In your first example, ts_group is neither an element of your table, nor a defined function, so it complained that it did not know how to calculate it.
Your second example works because date_trunc is a known function and ts is defined as a data element of the test table.
It also gets you the correct grouping (by hour intervals) because date_trunc 'blurs' all of those unique timestamps that by themselves would not combine into groups.

Without a GROUP BY, then having any aggregates in your select list means it will aggregate everything down to just one row. But how does it aggregate date_trunc('hour', ts) down to one row, since there is no aggregating function specified for it? If you were using MySQL, it would just pick some arbitrary value for the column from all the seen values and report that as the "aggregate". But PostgreSQL is not so cavalier with your data. If your query is vague in this way, it refuses to run it. If you just want to see some value from the set without caring which one it is, you could use min or max to aggregate it.
Since the syntax doesn't make any sense to me (entries on the select line would be treated differently while being declared together vs. building groups and then declaring operations on the groups),
You are trying to understand SQL as if it were C. But it is very different. Just learn it for what it is, without trying to force it to be something else. The select list is where you define the columns you want to see in the output. They may be computed in different ways, but what they have in common is that you want each of them to show up in the output, so they are listed together in that spot.

Related

When can aggregate functions be nested in standard SQL?

I know it wasn't allowed in SQL-92. But since then it may have changed, particularly when there's a window applied. Can you explain the changes and give the version (or versions if there were more) in which they were introduced?
Examples
Is SUM(COUNT(votes.option_id)) OVER() valid syntax per standard SQL:2016 (or earlier)?
This is my comment (unanswered, an probably unlikely to in such an old question) in Why can you nest aggregate functions when using a window function in PostgreSQL?.
The Calculating Running Total (SQL) kata at Codewars has as its most upvoted solution (using PostgreSQL 13.0, a highly standard compliant engine, so the code is likely to be standard) this one:
SELECT
CREATED_AT::DATE AS DATE,
COUNT(CREATED_AT) AS COUNT,
SUM(COUNT(CREATED_AT)) OVER (ORDER BY CREATED_AT::DATE ROWS UNBOUNDED PRECEDING)::INT AS TOTAL
FROM
POSTS
GROUP BY
CREATED_AT::DATE
(Which could be simplified to:
SELECT
created_at::DATE date,
COUNT(*) COUNT,
SUM(COUNT(*)) OVER (ORDER BY created_at::DATE)::INT total
FROM posts
GROUP BY created_at::DATE
I assume the ::s are a new syntax for casting I didn't know of. And that casting from TIMESTAMP to DATE is now allowed (in SQL-92 it wasn't).)
As this SO answer explains, Oracle Database allows it even without a window, pulling in the GROUP BY from context. I don't know if the standard allows it.
You already noticed the difference yourself: It's all about the window. COUNT(*) without an OVER clause for instance is an aggregation function. COUNT(*) with an OVER clause is a window function.
By using aggregation functions you condense the original rows you get after the FROM clause and WHERE clause are applied to either the specified group in GROUP BY or to one row in the absence of a GROUP BY clause.
Window functions, aka analytic functions, are applied afterwards. They don't change the number of result rows, but merely add information by looking at all or some rows (the window) of the selected data.
In
SELECT
options.id,
options.option_text,
COUNT(votes.option_id) as vote_count,
COUNT(votes.option_id) / SUM(COUNT(votes.option_id)) OVER() * 100.0 as vote_percentage
FROM options
LEFT JOIN votes on options.id = votes.option_id
GROUP BY options.id;
we first join votes to options and then count the votes per option by aggregating the joined rows down to one result row per option (GROUP BY options.id). We count on a non-nullable column in the votes table (COUNT(votes.option_id), so we get a zero count in case there are no votes, because in an outer joined row this column is set to null.
After aggregating all rows and getting thus one row per option we apply a window function (SUM() OVER) on this result set. We apply the analytic SUM on the vote count (SUM(COUNT(votes.option_id)) by looking at the whole result set (empty OVER clause), thus getting the same total vote count in every row. We use this value for a calculation: option's vote count diveded by total vote count times 100, which is the option's percentage of total votes.
The PostgreSQL query is very similar. We select the number of posts per date (COUNT(created_at) is nothing else than a mere COUNT(*)) along with a running total of these counts (by using a window that looks at all rows up to the current row).
So, while this looks like we are nesting two aggregate functions, this is not really the case, because SUM OVER is not considered an agregation function but an analytic/window function.
Oracle does allow applying an aggregate function directly on another, thus invoking a final aggregation on a previous grouped by aggregation. This allows us to get one result row of, say, the average of sums without having to write a subquery for this. This is not compliant with the SQL standard, however, and very unpopular even among Oracle developers at that.

Configurable SQL GROUP BY variable

I am currently developing some SQL aggregation queries to calculate data from one source table. The goal is, to have different aggregation granularities in one executable query / function / etc. I am currently developing on PostgreSQL but the code should be ANSI SQL compliant as much as possible to be compatible to most DB variants.
Example:
SELECT
COUNT(a) as amount,
SUM(b) as sum,
c as static_grouping,
#vargr as variable_grouping,
#vardesc as variable_grouping_description
FROM whatever
GROUP BY c, #vargr, #vardesc
#vargr can be date driven like daily, weekly, monthly, ...
#vardesc is the identifier to see aggregation type as text
Having multiple queries with UNION is not an option, since there are multiple grouping statements changing (resulting in 60+ single queries per result set). Is there a way to do this with a function, while loop, etc.?
Thanks for a hint in the right direction, have a good day and stay safe!
Best regards
Christian
If you want to group by varying date granularity, then one option uses date_trunc(). You would typically use one of the supported precision (such as day, week, month and so on) as parameter. Assuming that your date or timestamp column is tscol, you would do:
SELECT
COUNT(a) as amount,
SUM(b) as sum,
c as static_grouping,
DATE_TRUNC($1, tscol) as variable_grouping,
$2 as variable_grouping_description
FROM whatever
GROUP BY c, DATE_TRUNC($1, tscol)

Which of these expressions is the more correct usage of the SUM() function?

I'm trying to get the sum of some daily volumes multiplied by their list price and a rate, but have noticed I am getting slightly different results depending on how I use the SUM() function.
I've tried multiplying the fields inside of SUM() as well as moving the multiplication outside of the aggregate function.
SELECT SUM(Vol * LP * Rate) FROM Table A
-- or
SELECT SUM(Vol) * LP * Rate FROM Table A GROUP BY LP, Rate
As these results will get aggregated to a customer, I didn't expect to be concerned about the grouping and still don't see how this could be an issue.
These do different things, so you should use the version appropriate for your problem.
For the second, you have to include LP and Rate in the GROUP BY.
This is not necessary -- or even desirable -- for the first.
So that is the answer. Use the version that corresponds to the GROUP BY that you want. I would imagine that this would typically be the first version.

Select and manipulate SQL data, DISTINCT and SUM?

Im trying to make a small report for myself to see how my much time I get inputed in my system every day.
The goal is to have my SQL to sum up the name, Total time worked and Total NG product found for one specific day.
In this order:
1.) Sort out my data for a specific 'date'. I.E 2016-06-03
2.) Present a DISTINCT value for 'operators'
3.) SUM() all time registered at this 'date' and by this 'operator' under 'total_working_time_h'
4.) SUM() all no_of_defects registered at this 'date' and by this 'operator' under 'no_of_defects'
date, operator, total_working_time_h, no_of_defects
Currently I get the data I want by using the Query below. But now I need both the DISTINCT value of the operator and the SUM of the information. Can I use sub-queries for this or should it be done by a loop? Any other hints where I can learn more about how to solve this?
If i run the DISTINCT function I don't get the opportunity to sum my data the way I try.
SELECT date, operator, total_working_time_h, no_of_defects FROM {$table_work_hours} WHERE date = '2016-06-03' "
Without knowing the table structure or contents, the following query is only a good guess. The bits to notice and work with are sum() and GROUP BY. Actually syntax will vary a bit depending on what RDBMS you are using.
SELECT
date
,operator
,SUM(total_working_time_h) AS total_working_time_h
,SUM(no_of_defects) AS no_of_defects
FROM {$table_work_hours}
WHERE date = '2016-06-03'
GROUP BY
date
,operator
(Take out the WHERE clause or replace it with a range of dates to get results per operator per date.)
I'm not sure why you are trying to do DISTINCT. You want to know the data, no of hours, etc for a specific date.
do this....
Select Date, Operator, 'SumWorkHrs'=sum(total_working_time_h),
'SumDefects'=sum(no_ofDefects) from {$table_work_hours}
Where date='2016-06-03'
Try this:
SELECT SUM(total_working_time) as total_working_time,
SUM(no_of_defects) as no_of_defects ,
DISTINCT(operator) AS operator FROM {$table_work_hours} WHERE
date = '2016-06-03'

BigQuery: Using threshold with COUNT DISTINCT in WINDOW function returns error

With COUNT DISTINCT, I often make use of a threshold to make it more precise. E.g. COUNT(DISTINCT users, 100000).
If I am using a WINDOW function though I get an error when trying to use a threshold COUNT_DISTINCT must have at most 1 argument(s), found 2. E.g. here's a made-up query that demonstrates the problem:
SELECT
day,
COUNT(DISTINCT state, 100000) OVER (PARTITION BY year, month, day)
FROM [publicdata:samples.natality]
LIMIT 1000
Is this by design? Is there a workaround?
COUNT(DISTINCT) is documented as approximation when used as aggregation function, but when it is used as analytic function - it is actually the exact implementation, so you don't need extra parameter - you will get the exact result without it.