I have a long table like the following. The table adds two similar rows after the id changes. E.g in the following table when ID changes from 1 to 2 a duplicate record is added. All I need is a SELECT query to skip this and all other duplicate records only if the ID changes.
# | name| id
--+-----+---
1 | abc | 1
2 | abc | 1
3 | abc | 1
4 | abc | 1
5 | abc | 1
5 | abc | 2
6 | abc | 2
7 | abc | 2
8 | abc | 2
9 | abc | 2
and so on
You could use NOT EXISTS to eliminate the duplicates:
SELECT *
FROM yourtable AS T
WHERE NOT EXISTS
( SELECT 1
FROM yourtable AS T2
WHERE T.[#] = T2.[#]
AND T2.ID > T.ID
);
This will return:
# name ID
------------------
. ... .
4 abc 1
5 abc 2
6 abc 2
. ... .
... (Some irrelevant rows have been removed from the start and the end)
If you wanted the first record to be retained, rather than the last, then just change the condition T2.ID > T.ID to T2.ID < T.ID.
You can use the following CTEs to simulate LAG window function not available in SQL Server 2008:
;WITH CTE_RN AS (
SELECT *, ROW_NUMBER() OVER (ORDER BY [#], id) AS rn
FROM #mytable
), CTE_LAG AS (
SELECT t1.[#], t1.name,
t1.id AS curId, t2.id AS prevId,
t1.[#] AS cur#, t2.[#] AS lag#
FROM CTE_RN t1
LEFT JOIN CTE_RN t2 ON t1.rn = t2.rn + 1 )
You can now filter out the 'duplicate' records using the above CTE_LAG and the following predicate in your WHERE clause:
;WITH (
... cte definitions here
) SELECT *
FROM CTE_LAG
WHERE (NOT ((prevId <> curId) AND (cur# = lag#))) OR (prevId IS NULL)
If prevId <> curId and cur# = lag#, then there is a change in the value of the id column and the following record has the same [#] value as the previous one, i.e. it is a duplicate.
Hence, using NOT on (prevId <> curId) AND (cur# = lag#), filters out all 'duplicate' records. This means record (5, abc, 2) will be eliminated.
SQL Fiddle Demo here
P.S. You can also add column name in the logical expression of the WHERE clause, depending on what defines a 'duplicate'.
So I achieved it by using the following query in SQL server.
select #, name, id
from table
group by #, name, id
having count(*) > 0
Related
I need to show all records for a specific value if ANY one of those records have another specific value. Essentially, if field3 = 'b', what is field1? Show all records with value of field1 regardless of their field3 value.
Record Number External Id Letter
1 123456 a
2 123456 b
3 123456 c
4 456852 t
5 456852 b
record 2 has a letter value of 'b' - so I want to look at externalid, which is 123456, now I want to pull all records for external id regardless if the other records have a letter value of 'b'
Use EXISTS and a correlated subquery:
SELECT *
FROM mytable t
WHERE
t.letter = 'b'
OR EXISTS (
SELECT 1
FROM mytable t1
WHERE
t1.record_number != t.record_number
AND t1.external_id = t.external_id
AND t1.letter = 'b'
)
Another option is to use a window function:
SELECT record_number, external_id, letter
FROM (
SELECT
t.*,
MAX(CASE WHEN letter = 'b' THEN 1 END) OVER(PARTITION BY external_id) mx
FROM mytable t
) x WHERE mx = 1
Demo on DB Fiddle:
record_number | external_id | letter
------------: | ----------: | :-----
1 | 123456 | a
2 | 123456 | b
3 | 123456 | c
4 | 456852 | t
5 | 456852 | b
Use exists, but don't worry about filtering in the outer query:
select t.*
from t
where exists (select 1
from t t2
where t2.external_id = t.external_id and t2.letter = 'b'
);
With an index on (external_id, letter), I would expect this to have very good performance.
Platform: Oracle 10g
I have a table (let's call it t1) like this:
ID | FK_ID | SOME_VALUE | SOME_DATE
----+-------+------------+-----------
1 | 101 | 10 | 1-JAN-2013
2 | 101 | 20 | 1-JAN-2014
3 | 101 | 30 | 1-JAN-2015
4 | 102 | 150 | 1-JAN-2013
5 | 102 | 250 | 1-JAN-2014
6 | 102 | 350 | 1-JAN-2015
For each FK_ID I wish to show a single result showing the two most recent SOME_VALUEs. That is:
FK_ID | CURRENT | PREVIOUS
------+---------+---------
101 | 30 | 20
102 | 350 | 250
There is another table (lets call it t2) for the FK_ID, and it is here that there is a reference
saying which is the 'CURRENT' record. So a table like:
ID | FK_CURRENT | OTHER_FIELDS
----+------------+-------------
101 | 3 | ...
102 | 6 | ...
I was attempting this with a flawed sub query join along the lines of:
SELECT id, curr.some_value as current, prev.some_value as previous FROM t2
JOIN t1 curr ON t2.fk_current = t1.id
JOIN t1 prev ON t1.id = (
SELECT * FROM (
SELECT id FROM (
SELECT id, ROW_NUMBER() OVER (ORDER BY SOME_DATE DESC) as rno FROM t1
WHERE t1.fk_id = t2.id
) WHERE rno = 2
)
)
However the t1.fk_id = t2.id is flawed (i.e. wont run), as (I now know) you can't pass a parent
field value into a sub query more than one level deep.
Then I started wondering if Common Table Expressions (CTE) are the tool for this, but then I've no
experience using these (so would like to know I'm not going down the wrong track attempting to use them - if that is the tool).
So I guess the key complexity that is tripping me up is:
Determining the previous value by ordering, but while limiting it to the first record (and not the whole table). (Hence the somewhat convoluted sub query attempt.)
Otherwise, I can just write some code to first execute a query to get the 'current' value, and then
execute a second query to get the 'previous' - but I'd love to know how to solve this with a single
SQL query as it seems this would be a common enough thing to do (sure is with the DB I need to work
with).
Thanks!
Try an approach with LAG function:
SELECT FK_ID ,
SOME_VALUE as "CURRENT",
PREV_VALUE as Previous
FROM (
SELECT t1.*,
lag( some_value ) over (partition by fk_id order by some_date ) prev_value
FROM t1
) x
JOIN t2 on t2.id = x.fk_id
and t2.fk_current = x.id
Demo: http://sqlfiddle.com/#!4/d3e640/15
Try out this:
select t1.FK_ID ,t1.SOME_VALUE as CURRENT,
(select SOME_VALUE from t1 where p1.id2=t1.id and t1.fk_id=p1.fk_id) as PREVIOUS
from t1 inner join
(
select t1.fk_id, max(t1.id) as id1,max(t1.id)-1 as id2 from t1 group by t1.FK_ID
) as p1 on t1.id=p1.id1
I'll try to explain the type of the query that I want:
Assume I have a table like this:
| ID | someID | Number |
|----|--------|--------|
| 1 | 1 | 10 |
| 2 | 1 | 11 |
| 3 | 1 | 14 |
| 4 | 2 | 10 |
| 5 | 2 | 13 |
Now, I want to find the someID that have a specific numbers (For example query for numbers 10, 11, 14 will return someID 1 and query for numbers 10, 13 will return 2). But, if someID contains all the query numbers but also more numbers, it will not return by the query. (For example query for 10, 11 will return nothing).
Is it possible?
SELECT t1.someId
FROM yourTable t1
WHERE t1.number IN (10,14,11)
GROUP BY t1.someID
HAVING COUNT(DISTINCT t1.ID) = (SELECT COUNT(DISTINCT t2.ID) FROM yourTable t2 WHERE t1.someID=t2.someID)
Example Fiddle
select someID
from yourtable
where number in (10,11,14)
and not exists (select * from yourtable t2 where number not in(10,11,14)
and t2.someid=yourtable.someid)
group by someID
having count(distinct ID) = 3
Where 3 is the number of items you are querying for
Yes, once you get the query numbers into a table variable (say it's called #QNums, with one column named QNum)) try
Select distinct someId
From table t
Where exists (Select * from #QNums
where QNum = t.Number)
And not Exists (Select * From table t2
Where someId = t.someId
And not exists(Select * From #QNums
where QNum = t3.Number))
I have table with data something like this:
ID | RowNumber | Data
------------------------------
1 | 1 | Data
2 | 2 | Data
3 | 3 | Data
4 | 1 | Data
5 | 2 | Data
6 | 1 | Data
7 | 2 | Data
8 | 3 | Data
9 | 4 | Data
I want to group each set of RowNumbers So that my result is something like this:
ID | RowNumber | Group | Data
--------------------------------------
1 | 1 | a | Data
2 | 2 | a | Data
3 | 3 | a | Data
4 | 1 | b | Data
5 | 2 | b | Data
6 | 1 | c | Data
7 | 2 | c | Data
8 | 3 | c | Data
9 | 4 | c | Data
The only way I know where each group starts and stops is when the RowNumber starts over. How can I accomplish this? It also needs to be fairly efficient since the table I need to do this on has 52 Million Rows.
Additional Info
ID is truly sequential, but RowNumber may not be. I think RowNumber will always begin with 1 but for example the RowNumbers for group1 could be "1,1,2,2,3,4" and for group2 they could be "1,2,4,6", etc.
For the clarified requirements in the comments
The rownumbers for group1 could be "1,1,2,2,3,4" and for group2 they
could be "1,2,4,6" ... a higher number followed by a lower would be a
new group.
A SQL Server 2012 solution could be as follows.
Use LAG to access the previous row and set a flag to 1 if that row is the start of a new group or 0 otherwise.
Calculate a running sum of these flags to use as the grouping value.
Code
WITH T1 AS
(
SELECT *,
LAG(RowNumber) OVER (ORDER BY ID) AS PrevRowNumber
FROM YourTable
), T2 AS
(
SELECT *,
IIF(PrevRowNumber IS NULL OR PrevRowNumber > RowNumber, 1, 0) AS NewGroup
FROM T1
)
SELECT ID,
RowNumber,
Data,
SUM(NewGroup) OVER (ORDER BY ID
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS Grp
FROM T2
SQL Fiddle
Assuming ID is the clustered index the plan for this has one scan against YourTable and avoids any sort operations.
If the ids are truly sequential, you can do:
select t.*,
(id - rowNumber) as grp
from t
Also you can use recursive CTE
;WITH cte AS
(
SELECT ID, RowNumber, Data, 1 AS [Group]
FROM dbo.test1
WHERE ID = 1
UNION ALL
SELECT t.ID, t.RowNumber, t.Data,
CASE WHEN t.RowNumber != 1 THEN c.[Group] ELSE c.[Group] + 1 END
FROM dbo.test1 t JOIN cte c ON t.ID = c.ID + 1
)
SELECT *
FROM cte
Demo on SQLFiddle
How about:
select ID, RowNumber, Data, dense_rank() over (order by grp) as Grp
from (
select *, (select min(ID) from [Your Table] where ID > t.ID and RowNumber = 1) as grp
from [Your Table] t
) t
order by ID
This should work on SQL 2005. You could also use rank() instead if you don't care about consecutive numbers.
I'm trying to group data in sequence order. Say I have the following table:
| 1 | A |
| 1 | A |
| 1 | B |
| 1 | B |
| 1 | C |
| 1 | B |
I need the SQL query to output the following:
| 1 | A | 1 |
| 1 | A | 1 |
| 1 | B | 2 |
| 1 | B | 2 |
| 1 | C | 3 |
| 1 | B | 4 |
The last column is a group number that is incremented in each group. The important thing to note is that rows 3, 4 and 5 contain the same data which should be grouped into 2 groups not 1.
For MSSQL2008:
Suppose you have a SampleStatuses table:
Status Date
A 2014-06-11
A 2014-06-14
B 2014-06-25
B 2014-07-01
A 2014-07-06
A 2014-07-19
B 2014-07-21
B 2014-08-13
C 2014-08-19
you write the following:
;with
cte as (
select top 1 RowNumber, 1 as GroupNumber, [Status], [Date] from SampleStatuses order by RowNumber
union all
select c1.RowNumber,
case when c2.Status <> c1.Status then c2.GroupNumber + 1 else c2.GroupNumber end as GroupNumber, c1.[Status], c1.[Date]
from cte c2 join SampleStatuses c1 on c1.RowNumber = c2.RowNumber + 1
)
select * from cte;
you get this result:
RowNumber GroupNumber Status Date
1 1 A 2014-06-11
2 1 A 2014-06-14
3 2 B 2014-06-25
4 2 B 2014-07-01
5 3 A 2014-07-06
6 3 A 2014-07-19
7 4 B 2014-07-21
8 4 B 2014-08-13
9 5 C 2014-08-19
The normal way you would do what you want is the dense_rank function:
select key, val,
dense_rank() over (order by key, val)
from t
However, this does not address the problem of separating the last groups.
To handle this, I have to assume there is an "id" column. Tables, in SQL, do not have an ordering, so I need the ordering. If you are using SQL Server 2012, then you can use the lag() function to get what you need. Use the lag to see if the key, val pair is the same on consecutive rows:
with t1 as (
select id, key, val,
(case when key = lead(key, 1) over (order by id) and
val = lead(val, 1) over (order by id)
then 1
else 0
end) as SameAsNext
from t
)
select id, key, val,
sum(SameAsNext) over (order by id) as GroupNum
from t
Without SQL Server 2012 (which has cumulative sums), you have to do a self-join to identify the beginning of each group:
select t.*,
from t left outer join
t tprev
on t.id = t2.id + 1 and t.key = t2.key and t.val = t2.val
where t2.id is null
With this, assign the group as the minimum id using a join:
select t.id, t.key, t.val,
min(tgrp.id) as GroupId
from t left outer join
(select t.*,
from t left outer join
t tprev
on t.id = t2.id + 1 and t.key = t2.key and t.val = t2.val
where t2.id is null
) tgrp
on t.id >= tgrp.id
If you want these to be consecutive numbers, then put them in a subquery and use dense_rank().
This will give you rankings on your columns.
It will not give you 1,2,3 however.
It will give you 1,3,6 etc based on how many in each grouping
select
a,
b,
rank() over (order by a,b)
from
table1
See this SQLFiddle for a clearer idea of what I mean: http://sqlfiddle.com/#!3/0f201/2/0