I'm looking for a way to let me select all orders that have multiple distinct names within the same order-number, it looks like this:
order - name
111-Paul
112-Paula
113-John
113-John
113-Jessica
114-Eric
114-Eric
114-John
115-Zack
115-Zack
115-Zack
etc.
so that i would get all the orders that have 2 or more distinct names in it:
113-John
113-Jessica
114-Eric
114-John
with which I could do further queries but I'm stuck. Can anyone give me some hints on how to tackle this problem please? I've tried it with count(*) which looked like this:
select order, name, count(name) from dbo.orders
group by order, name
having count(name) > 1
which gave me all the orders which had more than 1 name in it but I don't know how to let it only show orders with the distinct names.
Here's one approach using exists:
select distinct [order], name
from orders o
where exists (
select 1
from orders o2
where o.[order] = o2.[order] and o.name != o2.name)
Fiddle Demo
I would use windows functions for this
For example:
select distinct order
from
(select
order,
row_number() over(partition by order, name order by order asc) as rn
) as t1
where rn > 1
you can do the same with count
count(*) over(partition by order,name order by order asc) as cnt
Here's a straight forward implementation in Sql Server:
select distinct *
from table1
where [order] in (
select [order]
from (select distinct * from table1) iq
group by [order]
having count(*) > 1)
It's essentially breaking down the problem into:
Finding the orders that have more than one distinct value.
Finding the pairs of distinct order - name that belong to the list previously calculated.
When you use HAVING COUNT(name) > 1, it is counting all of the rows in those groups, including duplicate rows (rows 113-John and 113-John are 2 rows for order 113). I would query distinct rows from your table, and then select from that:
SELECT [order], [name] FROM (
SELECT DISTINCT [order], [name] FROM dbo.orders
) A
GROUP BY [order], [name]
HAVING COUNT([name]) > 1
As a note, if a [name] is null, then it will not be counted with COUNT(name). If you want nulls to be counted, use COUNT(*) instead.
You can use count(distinct name) to get the number of unique names for each order:
select [order], count(distinct name)
from orders
group by [order]
To just get the order for those you can use having:
select [order]
from orders
group by [order]
having count(distinct name) > 1
To get the details for those orders you can put that in the where clause to just return the rows with order in that list:
select *
from orders
where [order] in (
select [order]
from orders
group by [order]
having count(distinct name) > 1
)
sqlfiddle
I would use RANK (or DENSE_RANK) for this as shown below.
SELECT [Order]
FROM (SELECT
[Order],
RANK() OVER(PARTITION BY [Order] ORDER BY Name) AS NameRank
FROM [StackOverflow].[dbo].[OrderAndName]) ranked
WHERE ranked.NameRank > 1
GROUP BY [Order]
The sub-query ranks (gives a seeding) to the names in an order according to their value. Names with the same value would have the same rank i.e. when an order has one name multiple times (like 115) the rank of all names would be 1.
The partition is important here as otherwise you would get the rank for all names for all orders which wouldn't give you the result you'd like.
It is then just a case of pulling out the orders that have a RANK greater than 1 and grouping (could use distinct if that's a preference).
You can then join to this table to get get the orders and names as follows;
SELECT oan.[Order], [Name]
FROM [StackOverflow].[dbo].[OrderAndName] oan
INNER JOIN (SELECT [Order]
FROM (SELECT [Order],
RANK() OVER(PARTITION BY [Order] ORDER BY Name) AS NameRank
FROM [StackOverflow].[dbo].[OrderAndName]) ranked
WHERE ranked.NameRank > 1
GROUP BY [Order]) twoOrMore ON oan.[Order] = twoOrMore.[Order]
The results are below. I need to get the records (seller and purchaser) with the max count- grouped by purchaser (marked with yellow)
You can use window functions:
with q as (
<your query here>
)
select q.*
from (select q.*,
row_number() over (order by seller desc) as seqnum_s,
row_number() over (order by purchaser desc) as seqnum_p
from q
) q
where seqnum_s = 1 or seqnum_p = 1;
Try this:
SELECT COUNT,seller,purchaser FROM YourTable ORDER BY seller,purchaser DESC
SELECT T2.MaxCount,T2.purchaser,T1.Seller FROM <Yourtable> T1
Inner JOIN
(
Select Max(Count) as MaxCount, purchaser
FROM <Yourtable>
GROUP BY Purchaser
)T2
On T2.Purchaser=T1.Purchaser AND T2.MaxCount=T1.Count
First you select the Seller from which will give you a list of all 5 sellers. Then you write another query where you select only the Purchaser and the Max(count) grouped by Purchaser which will give you the two yellow-marked lines. Join the two queries on fields Purchaser and Max(Count) and add the columns from the joined table to your first query.
I can't think of a faster way but this works pretty fast even with rather large queries. You can further-by order the fields as needed.
I have a PostgreSQL database table called "user_links" which currently allows the following duplicate fields:
year, user_id, sid, cid
The unique constraint is currently the first field called "id", however I am now looking to add a constraint to make sure the year, user_id, sid and cid are all unique but I cannot apply the constraint because duplicate values already exist which violate this constraint.
Is there a way to find all duplicates?
The basic idea will be using a nested query with count aggregation:
select * from yourTable ou
where (select count(*) from yourTable inr
where inr.sid = ou.sid) > 1
You can adjust the where clause in the inner query to narrow the search.
There is another good solution for that mentioned in the comments, (but not everyone reads them):
select Column1, Column2, count(*)
from yourTable
group by Column1, Column2
HAVING count(*) > 1
Or shorter:
SELECT (yourTable.*)::text, count(*)
FROM yourTable
GROUP BY yourTable.*
HAVING count(*) > 1
From "Find duplicate rows with PostgreSQL" here's smart solution:
select * from (
SELECT id,
ROW_NUMBER() OVER(PARTITION BY column1, column2 ORDER BY id asc) AS Row
FROM tbl
) dups
where
dups.Row > 1
In order to make it easier I assume that you wish to apply a unique constraint only for column year and the primary key is a column named id.
In order to find duplicate values you should run,
SELECT year, COUNT(id)
FROM YOUR_TABLE
GROUP BY year
HAVING COUNT(id) > 1
ORDER BY COUNT(id);
Using the sql statement above you get a table which contains all the duplicate years in your table. In order to delete all the duplicates except of the the latest duplicate entry you should use the above sql statement.
DELETE
FROM YOUR_TABLE A USING YOUR_TABLE_AGAIN B
WHERE A.year=B.year AND A.id<B.id;
You can join to the same table on the fields that would be duplicated and then anti-join on the id field. Select the id field from the first table alias (tn1) and then use the array_agg function on the id field of the second table alias. Finally, for the array_agg function to work properly, you will group the results by the tn1.id field. This will produce a result set that contains the the id of a record and an array of all the id's that fit the join conditions.
select tn1.id,
array_agg(tn2.id) as duplicate_entries,
from table_name tn1 join table_name tn2 on
tn1.year = tn2.year
and tn1.sid = tn2.sid
and tn1.user_id = tn2.user_id
and tn1.cid = tn2.cid
and tn1.id <> tn2.id
group by tn1.id;
Obviously, id's that will be in the duplicate_entries array for one id, will also have their own entries in the result set. You will have to use this result set to decide which id you want to become the source of 'truth.' The one record that shouldn't get deleted. Maybe you could do something like this:
with dupe_set as (
select tn1.id,
array_agg(tn2.id) as duplicate_entries,
from table_name tn1 join table_name tn2 on
tn1.year = tn2.year
and tn1.sid = tn2.sid
and tn1.user_id = tn2.user_id
and tn1.cid = tn2.cid
and tn1.id <> tn2.id
group by tn1.id
order by tn1.id asc)
select ds.id from dupe_set ds where not exists
(select de from unnest(ds.duplicate_entries) as de where de < ds.id)
Selects the lowest number ID's that have duplicates (assuming the ID is increasing int PK). These would be the ID's that you would keep around.
Inspired by Sandro Wiggers, I did something similiar to
WITH ordered AS (
SELECT id,year, user_id, sid, cid,
rank() OVER (PARTITION BY year, user_id, sid, cid ORDER BY id) AS rnk
FROM user_links
),
to_delete AS (
SELECT id
FROM ordered
WHERE rnk > 1
)
DELETE
FROM user_links
USING to_delete
WHERE user_link.id = to_delete.id;
If you want to test it, change it slightly:
WITH ordered AS (
SELECT id,year, user_id, sid, cid,
rank() OVER (PARTITION BY year, user_id, sid, cid ORDER BY id) AS rnk
FROM user_links
),
to_delete AS (
SELECT id,year,user_id,sid, cid
FROM ordered
WHERE rnk > 1
)
SELECT * FROM to_delete;
This will give an overview of what is going to be deleted (there is no problem to keep year,user_id,sid,cid in the to_delete query when running the deletion, but then they are not needed)
In your case, because of the constraint you need to delete the duplicated records.
Find the duplicated rows
Organize them by created_at date - in this case I'm keeping the oldest
Delete the records with USING to filter the right rows
WITH duplicated AS (
SELECT id,
count(*)
FROM products
GROUP BY id
HAVING count(*) > 1),
ordered AS (
SELECT p.id,
created_at,
rank() OVER (partition BY p.id ORDER BY p.created_at) AS rnk
FROM products o
JOIN duplicated d ON d.id = p.id ),
products_to_delete AS (
SELECT id,
created_at
FROM ordered
WHERE rnk = 2
)
DELETE
FROM products
USING products_to_delete
WHERE products.id = products_to_delete.id
AND products.created_at = products_to_delete.created_at;
Following SQL syntax provides better performance while checking for duplicate rows.
SELECT id, count(id)
FROM table1
GROUP BY id
HAVING count(id) > 1
begin;
create table user_links(id serial,year bigint, user_id bigint, sid bigint, cid bigint);
insert into user_links(year, user_id, sid, cid) values (null,null,null,null),
(null,null,null,null), (null,null,null,null),
(1,2,3,4), (1,2,3,4),
(1,2,3,4),(1,1,3,8),
(1,1,3,9),
(1,null,null,null),(1,null,null,null);
commit;
set operation with distinct and except.
(select id, year, user_id, sid, cid from user_links order by 1)
except
select distinct on (year, user_id, sid, cid) id, year, user_id, sid, cid
from user_links order by 1;
except all also works. Since id serial make all rows unique.
(select id, year, user_id, sid, cid from user_links order by 1)
except all
select distinct on (year, user_id, sid, cid)
id, year, user_id, sid, cid from user_links order by 1;
So far works nulls and non-nulls.
delete:
with a as(
(select id, year, user_id, sid, cid from user_links order by 1)
except all
select distinct on (year, user_id, sid, cid)
id, year, user_id, sid, cid from user_links order by 1)
delete from user_links using a where user_links.id = a.id returning *;
I have duplicate records in a table. I need to be able to identify only one unique identifier so I can delete it from the table.
The only way I know there are a duplicate is from columns subject and description so if there are at least 2 of the same subject and the same description, I need to delete one and leave one.
So I was able to get a list of the duplicate records but I am not able to get the unique identifier to be able to delete it.
This is what I have done to identify the duplicate records.
SELECT
p.accountid, p.subject, p.description, count(*) AS total
FROM
activities AS p
WHERE
(p.StateCode = 1) AND p.createdon >= getdate()-6
GROUP BY
p.accountid, p.subject, p.description
HAVING
count(*) > 1
ORDER BY
p.accountid
There is a column record_id which holds the unique identifier for each record. But if I added record_id to my select statement then I get no results because it is impossible to have a duplicate unique identifiers
How can I get the record_id using SQL Server?
NOTE: the record_id is not an integer it is something like "D32B275B-0B2F-4FF6-8089-00000FDA9E8E"
Thanks
One nice feature that I like about SQL Server is the use of CTEs with update and delete statements.
You are looking for duplicate records and presumably want to keep either the lowest or highest record_id. You can get the count and the id to keep using a CTE and window functions:
with todelete as (
SELECT p.accountid, p.subject, p.description,
COUNT(*) over (partition by p.accountid, p.subject, p.description) as total,
MIN(record_id) over (partition by p.accountid, p.subject, p.description) as IdToKeep
FROM activities AS p
WHERE (p.StateCode = 1) AND p.createdon >= getdate()-6
)
delete from todelete
where total > 1 and record_id <> IdToKeep;
The final where clause just uses the logic to select the right rows to delete.
I should add, if you just want the list that would be deleted, you can use the similar query:
with todelete as (
SELECT p.accountid, p.subject, p.description,
COUNT(*) over (partition by p.accountid, p.subject, p.description) as total,
MIN(record_id) over (partition by p.accountid, p.subject, p.description) as IdToKeep
FROM activities AS p
WHERE (p.StateCode = 1) AND p.createdon >= getdate()-6
)
select *
from todelete
where total > 1 and record_id <> IdToKeep;
The over function indicates that a function is being used as a window function. This idea is simple. Count(*) over returns the count for all records with the same values for the fields in the partition clause. It is a lot like the aggregation function, except you get the value on every row. This class of functions is quite powerful, and I'd recommend that you learn more about them.
Perhaps something like this?
SELECT max(p.record_id), p.accountid, p.subject, p.description, count(*) AS total
FROM activities AS p
WHERE (p.StateCode = 1) AND p.createdon >= getdate()-6
GROUP BY p.accountid, p.subject, p.description
HAVING count(*) > 1
ORDER BY p.accountid
Seems to me you need to do an inner query first, then join against the larger table to get what you want.
SELECT ALL
*
FROM (SELECT p.accountid
FROM activities AS p
WHERE p.statecode = 1 AND p.createdon >= getdate()-6
GROUP BY p.accountid
HAVING count(*) > 1) AS x
JOIN activities AS a ON x.accountid = a.accountid
ORDER BY p.accountid
Try this:
;with recordsToDelete as (
SELECT
recordId
,Row_Number() OVER(partition p.subject, p.description) as rowNum
FROM activities AS p
)
select
*
from recordsToDelete
where rowNum > 1
If that looks correct, you can replace the select with:
delete from recordsToDelete
where rowNum > 1