I'm looking to find a way to search a table for duplicate values and return those duplicates (or even just one of the set of duplicates) as the result set.
For instance, let's say I have these data:
uid | semi-unique id
1 | 12345
2 | 21345
3 | 54321
4 | 41235
5 | 12345
6 | 21345
I need to return either:
12345
12345
21345
21345
Or:
12345
21345
I've tried googling around and keep coming up short. Any help please?
To get each row, you can use window functions:
select t.*
from (select t.*, count(*) over (partition by [semi-unique id]) as totcnt
from t
) t
where totcnt > 1
To get just one instance, try this:
select t.*
from (select t.*, count(*) over (partition by [semi-unique id]) as totcnt,
row_number() over (partition by [semi-unique id] order by (select NULL)
) as seqnum
from t
) t
where totcnt > 1 and seqnum = 1
The advantage of this approach is that you get all the columns, instead of just the id (if that helps).
Sorry, I was short on time earlier so I couldn't explain my answer. The first query groups the semi_unique_ids that are the same and only returns the ones that have a duplicate.
SELECT semi_unique_id
FROM your_table
GROUP BY semi_unique_id
HAVING COUNT(semi_unique_id) > 1
If you wanted to get the uid in the query too you can easily add it like so.
SELECT uid,
semi_unique_uid
FROM your_table
GROUP BY
semi_unique_id,
uid
HAVING COUNT(semi_unique_id) > 1
Lastly if you would like to get an idea of how many duplicates per row returned you would do the following.
SELECT uid,
semi_unique_uid,
COUNT(semi_unique_uid) AS unique_id_count
FROM your_table
GROUP BY
semi_unique_id,
uid
HAVING COUNT(semi_unique_id) > 1
SELECT t.semi_unique_id AS i
FROM TABLE t
GROUP BY
t.semi_unique_id
HAVING (COUNT(t.semi_unique_id) > 1)
Try this for sql-server
Related
i have a table called "main" which has 4 columns, ID, name, DateID and Sign.
i want to create a query that will delete entries in this table if there is the same ID record in twice within a certain DateID.
i have my where clause that searches the previous 3 weeks
where DateID =((SELECT MAX( DateID)
WHERE DateID < ( SELECT MAX( DateID )-3))
e.g of my dataset im working with:
id
name
DateID
sign
12345
Paul
1915
Up
23658
Danny
1915
Down
37868
Jake
1916
Up
37542
Elle
1917
Up
12345
Paul
1917
Down
87456
John
1918
Up
78563
Luke
1919
Up
23658
Danny
1920
Up
in the case above, both entries for ID 12345 would need to be removed.
however the entries for ID 23658 would need to be kept as the DateID > 3
how would this be possible?
You can use window functions for this.
It's not quite clear, but it seems LAG and conditional COUNT should fit what you need.
DELETE t
FROM (
SELECT *,
CountWithinDate = COUNT(CASE WHEN t.PrevDate >= t.DateId - 3 THEN 1 END) OVER (PARTITION BY t.id)
FROM (
SELECT *,
PrevDate = LAG(t.DateID) OVER (PARTITION BY t.id ORDER BY t.DateID)
FROM YourTable t
) t
) t
WHERE CountWithinDate > 0;
db<>fiddle
Note that you do not need to re-join the table, you can delete directly from the t derived table.
Hope this works:
DELETE FROM test_tbl
WHERE id IN (
SELECT T1.id
FROM test_tbl T1
WHERE EXISTS (SELECT 1 FROM test_tbl T2 WHERE T1.id = T2.id AND ABS(T2.dateid - T1.dateid) < 3 AND T1.dateid <> T2.dateid)
)
In case you need more logic for data processing, I would suggest using Stored Procedure.
firstly let me describe you my problem. I need to ignore all repeated values in my select query. So for example if I have something like that:
| Other columns| THE COLUMN I'm working with |
| ............ | Value 1 |
| ............ | Value 2 |
| ............ | Value 2 |
I'd like to get the result containing only the row with "Value 1"
Now because of the specifics of my task I need to validate it with subquery.
So I've figured out something like this:
NOT EXISTS (SELECT 1 FROM TABLE fpd WHERE fpd.value = fp.value HAVING count(*) > 2)
It works like I want, but I'm aware of it being slow. Also I've tried putting 1 instead of 2 in HAVING comprassion, but it just returns zero results. Could you explain where does the 2 value come from?
I would suggest window functions:
select t.*
from (select t.*, count(*) over (partition by value) as cnt
from fpd t
) t
where cnt = 1;
Alternatively, you can use not exists with a primary key:
where not exists (select 1
from fpd fpd2
where fpd2.value = fp.value and
fpd2.primarykey <> fp.primarykey
)
SELECT DISTINCT myColumn FROM myTable
How can I delete rows where dateupdated was least updated ?
My table is
Name Dateupdated ID status
john 1/02/17 JHN1 A
john 1/03/17 JHN2 A
sally 1/02/17 SLLY1 A
sally 1/03/17 SLLY2 A
Mike 1/03/17 MK1 A
Mike 1/04/17 MK2 A
I want to be left with the following after the data removal:
Name Date ID status
john 1/03/17 JHN2 A
sally 1/03/17 SLLY2 A
Mike 1/04/17 MK2 A
If you really want to "delete rows where dateupdated was least updated" then a simple single-row subquery should do the trick.
DELETE MyTable
WHERE Date = (SELECT MIN(Date) From MyTable)
If on the other hand you just want to delete the row with the earliest Date per person (as identified by their ID) you could use:
DELETE MyTable
FROM MyTable a
JOIN (SELECT ID, MIN(Date) MinDate FROM MyTable GROUP BY ID) b
ON a.ID = b.ID AND a.Date = b.MinDate
The idea here is you create an aggregate query that returns rows containing the columns that would match the rows you want deleted, then join to it. Because it's an inner join, rows that do not match the criteria will be excluded.
If people are uniquely identified by something else (e.g. Name then you can just substitute that for the ID in my example above.
I am thinking though that you don't want either of these. I think you want to delete everything except for each person's latest row. If that is the case, try this:
DELETE MyTable
WHERE EXISTS (SELECT 0 FROM MyTable b WHERE b.ID = MyTable.ID AND b.Date > MyTable.Date)
The idea here is you check for existence of another data row with the same ID and a later date. If there is a later record, delete this one.
The nice thing about the last example is you can run it over and over and every person will still be left with exactly one row. The other two queries, if run over and over, will nibble away at the table until it is empty.
P.S. As these are significantly different solutions, I suggest you spend some effort learning how to articulate unambiguous requirements. This is an extremely important skill for any developer.
This deletes rows where the name is a duplicate, and deletes all but the latest row for each name. This is different from your stated question.
Using a common table expression (cte) and row_number():
;with cte as (
select *
, rn = row_number() over (
partition by Name
order by Dateupdated desc
)
from t
)
/* ------------------------------------------------
-- Remove duplicates by deleting rows
-- where the row number (rn) is greater than 1
-- leaving the first row for each partition
------------------------------------------------ */
delete
from cte
where cte.rn > 1
select * from t
rextester: http://rextester.com/HZBQ50469
returns:
+-------+-------------+-------+--------+
| Name | Dateupdated | ID | status |
+-------+-------------+-------+--------+
| john | 2017-01-03 | JHN2 | A |
| sally | 2017-01-03 | SLLY2 | A |
| Mike | 2017-01-04 | MK2 | A |
+-------+-------------+-------+--------+
Without using the cte it can be written as:
delete d
from (
select *
, rn = row_number() over (
partition by Name
order by Dateupdated desc
)
from t
) as d
where d.rn > 1
This should do the trick:
delete
from MyTable a
where not exists (
select top 1 1
from MyTable b
where b.name = a.name
and b.DateUpdated < a.DateUpdated
)
i.e. remove any entries from the table for which there is no record on the same name with a date earlier than the record to be deleted's.
Your Name column has Mike and Mik2 which is different for each other.
So, if you did not make a mistake, standard column to group by must be ID column without last digit.
I think following is more accurate if you did not mistaken.
delete a
from MyTable a
inner join
(select substring(ID, 1, len(ID) - 1) as ID, min(Dateupdated) as MinDate
from MyTable
group by substring(ID, 1, len(ID) - 1)
) b
on substring(a.ID, 1, len(a.ID) - 1) = b.ID and a.Dateupdated = b.MinDate
You can test it at SQLFiddle: http://sqlfiddle.com/#!6/9c440/1
I have a table like so
Id | Type | Value
--------------------
0 | Big | 2
1 | Big | 3
2 | Small | 3
3 | Small | 3
I would like to get a table like this
Type | Last Value
--------------------
Small | 3
Big | 3
How can I do this. I understand there is an SQL Server method called LAST_VALUE(...) OVER .(..) but I can't get this to work with GROUP BY.
I've also tried using SELECT MAX(ID) & SELECT TOP 1.. but this seems a bit inefficient since there would be a subquery for each value. The queries take too long when the table has a few million rows in it.
Is there a way to quickly get the last value for these, perhaps using LAST_VALUE?
You can do it using rownumber:
select
type,
value
from
(
select
type,
value,
rownumber() over (partition by type order by id desc) as RN
) TMP
where RN = 1
Can't test this now since SQL Fiddle doesn't seem to work, but hopefully that's ok.
The most efficient method might be not exists, which uses an anti-join for the underlying operator:
select type, value
from likeso l
where not exists (select 1 from likeso l2 where l2.type = l.type and l2.id > l.id)
For performance, you want an index on likeso(type, id).
I really wonder if there is more efficent solution but, I use following query on such needs;
Select Id, Type, Value
From ( Select *, Max (Id) Over (Partition By Type) As LastId
From #Table) T
Where Id = LastId
I've written a MySQL statment to select the next item using the items created timestamp, but I have a problem with it when two or more items have the exact same timestamp. To make things more confusing the item ID and created dates are not always in order, see the dummy data below.
ID | created
~~~~~~~~~~~~
10 | 1289780100
11 | 1289780100
12 | 1289780100 <- current item
13 | 1289780100
14 | 1289780050
15 | 1289780150
I need a SQL statement that can select the prev & next item (two queries are fine) by created and then ID. I've tired a few different queries but either gets stuck in a loop or both 'prev' & 'next' are the same.
Thanks,
Owen
SELECT *
FROM items
WHERE created >= current_item_created AND ID >= current_item_ID
ORDER BY created ASC, ID ASC LIMIT 1,1
SELECT *
FROM items
WHERE created <= current_item_created AND ID <= current_item_ID
ORDER BY created DESC, ID DESC LIMIT 1,1
To get prev_id:
SELECT
t1.id, t1.name, t1.prev as prev_id
FROM
(SELECT mt.id, mt.name, #prev_id as prev, #prev_id := id
FROM `main_table` mt, (SELECT #prev_id:=0) ni
ORDER BY name ASC) as t1;
To get next_id - just change ordering
Here is full example