PostgreSQL MAX and GROUP BY - sql

I have a table with id, year and count.
I want to get the MAX(count) for each id and keep the year when it happens, so I make this query:
SELECT id, year, MAX(count)
FROM table
GROUP BY id;
Unfortunately, it gives me an error:
ERROR: column "table.year" must appear in the GROUP BY clause or be
used in an aggregate function
So I try:
SELECT id, year, MAX(count)
FROM table
GROUP BY id, year;
But then, it doesn't do MAX(count), it just shows the table as it is. I suppose because when grouping by year and id, it gets the max for the id of that specific year.
So, how can I write that query? I want to get the id´s MAX(count) and the year when that happens.

The shortest (and possibly fastest) query would be with DISTINCT ON, a PostgreSQL extension of the SQL standard DISTINCT clause:
SELECT DISTINCT ON (1)
id, count, year
FROM tbl
ORDER BY 1, 2 DESC, 3;
The numbers refer to ordinal positions in the SELECT list. You can spell out column names for clarity:
SELECT DISTINCT ON (id)
id, count, year
FROM tbl
ORDER BY id, count DESC, year;
The result is ordered by id etc. which may or may not be welcome. It's better than "undefined" in any case.
It also breaks ties (when multiple years share the same maximum count) in a well defined way: pick the earliest year. If you don't care, drop year from the ORDER BY. Or pick the latest year with year DESC.
For many rows per id, other query techniques are (much) faster. See:
Select first row in each GROUP BY group?
Optimize GROUP BY query to retrieve latest row per user

select *
from (
select id,
year,
thing,
max(thing) over (partition by id) as max_thing
from the_table
) t
where thing = max_thing
or:
select t1.id,
t1.year,
t1.thing
from the_table t1
where t1.thing = (select max(t2.thing)
from the_table t2
where t2.id = t1.id);
or
select t1.id,
t1.year,
t1.thing
from the_table t1
join (
select id, max(t2.thing) as max_thing
from the_table t2
group by id
) t on t.id = t1.id and t.max_thing = t1.thing
or (same as the previous with a different notation)
with max_stuff as (
select id, max(t2.thing) as max_thing
from the_table t2
group by id
)
select t1.id,
t1.year,
t1.thing
from the_table t1
join max_stuff t2
on t1.id = t2.id
and t1.thing = t2.max_thing

Related

Alternative way of full outer join

I am running this query
select * from
(select name, count(distinct id) as ids, date
from table1
group by name, date ) as tt
full outer join
(select st_name as name,count(distinct id) as ids, date
from table2
group by st_name, date) as ts
on tt.name= ts.name
and tt.ids = ts.ids
It runs successfully but I want to ask if there is an alternative more efficient way to run this query.
I assume that you want to get days when the two numbers are not the same (it seems like the most reasonable thing you want from such a query). So, this addresses that question.
FULL OUTER JOIN should be fine. But an alternative is to try UNION ALL and aggregation:
select name, sum(ids_1), sum(ids_2), date
from ((select name, count(distinct id) as ids_1, NULL as ids_2, date
from table1
group by name, date
)
union all
(select st_name as name, NULL, count(distinct id) as ids_2, date
from table2
group by st_name, date
)
)
group by name, date
having sum(ids_1) = sum(ids_2)

Grouping values when only all values are equal

I'm trying to group some data. This is the situation:
I'm doing this select:
select
min(id) id
,week
,percentage
from table1
group by week,percentage
Could anyone help me out? I want that it only groups if all values are equal. If there is some different value in percentage it should not be grouping.. the id3 should not group for the week 3. I'm using SQL Server 2012.
Thanks.
You don't really want aggregation. You want to remove certain ids.
The following concatenates all the week/percentage values together to get a single identifier to identify duplicates. Despite what I just said, this then uses aggregation for "filtering" to get the first one:
select min(id) as id, week, percentage
from (select t1.*,
string_agg(concat(week, ':', percentage), ',') within group (order by week) over (partition by id) as id_week_percentages
from table1 t1
) t1
group by id_week_percentages, week, percentage;
The aggregation just allows this to be written without an extra subquery. You could do something similar as:
select top (1) with ties id, week, percentage
from (select t1.*,
string_agg(concat(week, ':', percentage), ',') within group (order by week) over (partition by id) as id_week_percentages
from table1 t1
) t1
order by row_number() over (partition by id_week_percentages, week, percentage order by id);
Or use a separate subquery to pull the first id for each week/percentage combination.
EDIT:
With for xml path:
select top (1) with ties id, week, percentage
from (select t1.*,
(select concat(week, ':', percentage, ',')
from table1 tt1
where tt1.id = t1.id
order by week, percentage
for xml path ('')
) as id_week_percentages
from table1 t1
) t1
order by row_number() over (partition by id_week_percentages, week, percentage order by id);

Show entire record from table with minimum timestamp in a group

I have been trying for about three hours to solve this problem but cannot find the solution.
How would I show the entire row (all 20 columns) for the first occurance (minimum time) of each name in my table?
For example, I would like to do something like this, which does not work:
SELECT name, MIN(time), col1, col2, col3, col4
FROM table
GROUP BY name;
You have to first get the minimum time for each name, and then join back to your original table where the name/time matches.
To get the minimum time:
SELECT name, MIN(time) AS minTime
FROM myTable
GROUP BY name;
Then, get all columns:
SELECT m.*
FROM myTable m
JOIN(
SELECT name, MIN(time) AS minTime
FROM myTable
GROUP BY name) tmp ON tmp.name = m.name AND tmp.minTime = m.time;
Most databases support ANSI standard window functions. With these, you can just do:
select t.*
from (select t.*, row_number() over (partition by name order by time) as seqnum
from table t
) t
where seqnum = 1;

How to find duplicate records in PostgreSQL

I have a PostgreSQL database table called "user_links" which currently allows the following duplicate fields:
year, user_id, sid, cid
The unique constraint is currently the first field called "id", however I am now looking to add a constraint to make sure the year, user_id, sid and cid are all unique but I cannot apply the constraint because duplicate values already exist which violate this constraint.
Is there a way to find all duplicates?
The basic idea will be using a nested query with count aggregation:
select * from yourTable ou
where (select count(*) from yourTable inr
where inr.sid = ou.sid) > 1
You can adjust the where clause in the inner query to narrow the search.
There is another good solution for that mentioned in the comments, (but not everyone reads them):
select Column1, Column2, count(*)
from yourTable
group by Column1, Column2
HAVING count(*) > 1
Or shorter:
SELECT (yourTable.*)::text, count(*)
FROM yourTable
GROUP BY yourTable.*
HAVING count(*) > 1
From "Find duplicate rows with PostgreSQL" here's smart solution:
select * from (
SELECT id,
ROW_NUMBER() OVER(PARTITION BY column1, column2 ORDER BY id asc) AS Row
FROM tbl
) dups
where
dups.Row > 1
In order to make it easier I assume that you wish to apply a unique constraint only for column year and the primary key is a column named id.
In order to find duplicate values you should run,
SELECT year, COUNT(id)
FROM YOUR_TABLE
GROUP BY year
HAVING COUNT(id) > 1
ORDER BY COUNT(id);
Using the sql statement above you get a table which contains all the duplicate years in your table. In order to delete all the duplicates except of the the latest duplicate entry you should use the above sql statement.
DELETE
FROM YOUR_TABLE A USING YOUR_TABLE_AGAIN B
WHERE A.year=B.year AND A.id<B.id;
You can join to the same table on the fields that would be duplicated and then anti-join on the id field. Select the id field from the first table alias (tn1) and then use the array_agg function on the id field of the second table alias. Finally, for the array_agg function to work properly, you will group the results by the tn1.id field. This will produce a result set that contains the the id of a record and an array of all the id's that fit the join conditions.
select tn1.id,
array_agg(tn2.id) as duplicate_entries,
from table_name tn1 join table_name tn2 on
tn1.year = tn2.year
and tn1.sid = tn2.sid
and tn1.user_id = tn2.user_id
and tn1.cid = tn2.cid
and tn1.id <> tn2.id
group by tn1.id;
Obviously, id's that will be in the duplicate_entries array for one id, will also have their own entries in the result set. You will have to use this result set to decide which id you want to become the source of 'truth.' The one record that shouldn't get deleted. Maybe you could do something like this:
with dupe_set as (
select tn1.id,
array_agg(tn2.id) as duplicate_entries,
from table_name tn1 join table_name tn2 on
tn1.year = tn2.year
and tn1.sid = tn2.sid
and tn1.user_id = tn2.user_id
and tn1.cid = tn2.cid
and tn1.id <> tn2.id
group by tn1.id
order by tn1.id asc)
select ds.id from dupe_set ds where not exists
(select de from unnest(ds.duplicate_entries) as de where de < ds.id)
Selects the lowest number ID's that have duplicates (assuming the ID is increasing int PK). These would be the ID's that you would keep around.
Inspired by Sandro Wiggers, I did something similiar to
WITH ordered AS (
SELECT id,year, user_id, sid, cid,
rank() OVER (PARTITION BY year, user_id, sid, cid ORDER BY id) AS rnk
FROM user_links
),
to_delete AS (
SELECT id
FROM ordered
WHERE rnk > 1
)
DELETE
FROM user_links
USING to_delete
WHERE user_link.id = to_delete.id;
If you want to test it, change it slightly:
WITH ordered AS (
SELECT id,year, user_id, sid, cid,
rank() OVER (PARTITION BY year, user_id, sid, cid ORDER BY id) AS rnk
FROM user_links
),
to_delete AS (
SELECT id,year,user_id,sid, cid
FROM ordered
WHERE rnk > 1
)
SELECT * FROM to_delete;
This will give an overview of what is going to be deleted (there is no problem to keep year,user_id,sid,cid in the to_delete query when running the deletion, but then they are not needed)
In your case, because of the constraint you need to delete the duplicated records.
Find the duplicated rows
Organize them by created_at date - in this case I'm keeping the oldest
Delete the records with USING to filter the right rows
WITH duplicated AS (
SELECT id,
count(*)
FROM products
GROUP BY id
HAVING count(*) > 1),
ordered AS (
SELECT p.id,
created_at,
rank() OVER (partition BY p.id ORDER BY p.created_at) AS rnk
FROM products o
JOIN duplicated d ON d.id = p.id ),
products_to_delete AS (
SELECT id,
created_at
FROM ordered
WHERE rnk = 2
)
DELETE
FROM products
USING products_to_delete
WHERE products.id = products_to_delete.id
AND products.created_at = products_to_delete.created_at;
Following SQL syntax provides better performance while checking for duplicate rows.
SELECT id, count(id)
FROM table1
GROUP BY id
HAVING count(id) > 1
begin;
create table user_links(id serial,year bigint, user_id bigint, sid bigint, cid bigint);
insert into user_links(year, user_id, sid, cid) values (null,null,null,null),
(null,null,null,null), (null,null,null,null),
(1,2,3,4), (1,2,3,4),
(1,2,3,4),(1,1,3,8),
(1,1,3,9),
(1,null,null,null),(1,null,null,null);
commit;
set operation with distinct and except.
(select id, year, user_id, sid, cid from user_links order by 1)
except
select distinct on (year, user_id, sid, cid) id, year, user_id, sid, cid
from user_links order by 1;
except all also works. Since id serial make all rows unique.
(select id, year, user_id, sid, cid from user_links order by 1)
except all
select distinct on (year, user_id, sid, cid)
id, year, user_id, sid, cid from user_links order by 1;
So far works nulls and non-nulls.
delete:
with a as(
(select id, year, user_id, sid, cid from user_links order by 1)
except all
select distinct on (year, user_id, sid, cid)
id, year, user_id, sid, cid from user_links order by 1)
delete from user_links using a where user_links.id = a.id returning *;

Select newest records that have distinct Name column

I did search around and I found this
SQL selecting rows by most recent date with two unique columns
Which is so close to what I want but I can't seem to make it work.
I get an error Column 'ID' is invalid in the select list because it is not contained in either an aggregate function or the GROUP BY clause.
I want the newest row by date for each Distinct Name
Select ID,Name,Price,Date
From table
Group By Name
Order By Date ASC
Here is an example of what I want
Table
ID
Name
Price
Date
0
A
10
2012-05-03
1
B
9
2012-05-02
2
A
8
2012-05-04
3
C
10
2012-05-03
4
B
8
2012-05-01
desired result
ID
Name
Price
Date
2
A
8
2012-05-04
3
C
10
2012-05-03
1
B
9
2012-05-02
I am using Microsoft SQL Server 2008
Select ID,Name, Price,Date
From temp t1
where date = (select max(date) from temp where t1.name =temp.name)
order by date desc
Here is a SQL Fiddle with a demo of the above
Or as Conrad points out you can use an INNER JOIN (another SQL Fiddle with a demo) :
SELECT t1.ID, t1.Name, t1.Price, t1.Date
FROM temp t1
INNER JOIN
(
SELECT Max(date) date, name
FROM temp
GROUP BY name
) AS t2
ON t1.name = t2.name
AND t1.date = t2.date
ORDER BY date DESC
There a couple ways to do this. This one uses ROW_NUMBER. Just partition by Name and then order by what you want to put the values you want in the first position.
WITH cte
AS (SELECT Row_number() OVER (partition BY NAME ORDER BY date DESC) RN,
id,
name,
price,
date
FROM table1)
SELECT id,
name,
price,
date
FROM cte
WHERE rn = 1
DEMO
Note you should probably add ID (partition BY NAME ORDER BY date DESC, ID DESC) in your actual query as a tie-breaker for date
select * from (
Select
ID, Name, Price, Date,
Rank() over (partition by Name order by Date) RankOrder
From table
) T
where RankOrder = 1
I have found another memory efficient way (but probably crude way)that has worked for me in postgress. Order the query by the date desc, then select the first record of each distinct field.
SELECT distinct on (Name) ID, Price, Date from
table
order by Date desc
Use Distinct instead of Group By
Select Distinct ID,Name,Price,Date
From table
Order By Date ASC
http://technet.microsoft.com/en-us/library/ms187831.aspx