SQL Select only once of each id, chosen by earliest datetime - sql

I'm currently having to run a query like the below for a one off report process at work.
However for each item in the table there are multiple associated "messages" that are all saved, this means each item is returned multiple times. I'd like to only show each item once, as per the examples and further explanation below.
I realize this is (at least in my opinion) a poor structure, but the report needs to be done and this is how the data is stored :-(
SQL Fiddle: http://sqlfiddle.com/#!6/76fce/8
Query:
SELECT messageId, receiver, createdDate, itemId from messages_0,items WHERE
itemId IN (1, 2, 3)
AND (receiver = '100' OR receiver = '200')
AND messages_0.description LIKE '%'+items.name+'%'
union all
SELECT messageId, receiver, createdDate, itemId from messages_1,items WHERE
itemId IN (1, 2, 3)
AND (receiver = '100' OR receiver = '200')
AND messages_1.description LIKE '%'+items.name+'%'
Note: there are two message tables, hence the union all
Example messages:
messageId | receiver | createdDate | description
--------------
1 | 100 | 2012/11/27 12:00 | The Dog is awesome
2 | 100 | 2012/11/27 13:00 | Now the Dog is boring
4 | 200 | 2012/11/27 11:30 | I have Wood :-)
Example items:
itemID | name
--------------
1 | Dave
2 | Dog
3 | Wood
Result:
messageId | receiver | createdDate | itemId
1 | 100 | 2012/11/27 12:00 | 2
2 | 100 | 2012/11/27 13:00 | 2
4 | 200 | 2012/11/27 11:00 | 3
However, I need to only show each item once. Where only the oldest row (by the createdDate) is shown.
Target Result:
messageId | receiver | createdDate | itemId
1 | 100 | 2012/11/27 12:00 | 2
4 | 200 | 2012/11/27 11:00 | 3
How can I do this in SQL (Sybase)?
So far I have been looking at both group by (which would only return an id) and some sort of sub query, but have been unable to get anything to work!
SQL Fiddle: http://sqlfiddle.com/#!6/76fce/8

If I understood you right, something like this could be a start.
SELECT
t, messageId, receiver, createdDate, itemId
FROM
(
SELECT
m.messageId, m.receiver, m.createdDate, m.t,
i.itemId
FROM
items i
INNER JOIN (
SELECT description, messageId, receiver, createdDate, 0 t FROM messages_0
UNION
SELECT description, messageId, receiver, createdDate, 1 t FROM messages_1
) m ON m.description LIKE '%' + i.name + '%'
AND m.receiver IN ('100', '200')
WHERE
i.itemId IN (1, 2, 3)
) data
WHERE
createdDate = (
SELECT MIN(createdDate) FROM (
SELECT createdDate FROM messages_0 WHERE messageId = data.messageId AND data.t = 0
UNION
SELECT createdDate FROM messages_1 WHERE messageId = data.messageId AND data.t = 1
)
)
I would put indexes on
messages_0 / messages_1
(messageId, createdDate)
(receiver, messageId, createdDate, description)
items
(itemId, name)

Related

Big query query is too complex after pivot

Assume I have the following table table and a list of interests (cat, dog, music, soccer, coding)
| userId | user_interest | label |
| -------- | -------------- |----------|
| 12345 | cat | 1 |
| 12345 | dog | 1 |
| 6789 | music | 1 |
| 6789 | soccer | 1 |
I want to transform the user interest into a binary array (i.e. binarization), and the resulting table will be something like
| userId | labels |
| -------- | -------------- |
| 12345 | [1,1,0,0,0] |
| 6789 | [0,0,1,1,0] |
I am able to do it with PIVOT and ARRAY, e.g.
WITH user_interest_pivot AS (
SELECT
*
FROM (
SELECT userId, user_interest, label FROM table
) AS T
PIVOT
(
MAX(label) FOR user_interestc IN ('cat', 'dog', 'music', 'soccer', 'coding')
) AS P
)
SELECT
userId,
ARRAY[IFNULL(cat,0), IFNULL(dog,0), IFNULL(music,0), IFNULL(soccer,0), IFNULL(coding,0)] AS labels,
FROM user_interea_pivot
HOWEVER, in reality I have a very long list of interests, and the above method in bigquery seems to not work due to
Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too comple
Please help to let me know if there is anything I can do to deal with this situation. Thanks!
Still it's likely to face resource problem depending on your real data, but it is worth trying the following approach without PIVOT.
Create interests table with additional index column first
+----------+-----+-----------------+
| interest | idx | total_interests |
+----------+-----+-----------------+
| cat | 0 | 5 |
| dog | 1 | 5 |
| music | 2 | 5 |
| soccer | 3 | 5 |
| coding | 4 | 5 |
+----------+-----+-----------------+
find idx of each user interest and aggreage them like below. (assuming that user intererest is sparse over overall interests)
SELECT userId, ARRAY_AGG(idx) user_interests
FROM sample_table t JOIN interests i ON t.user_interest = i.interest
GROUP BY 1
Lastly, create labels vector using a sparse user interest array and dimension of interest space (i.e. total_interests) like below
ARRAY(SELECT IF(ui IS NULL, 0, 1)
FROM UNNEST(GENERATE_ARRAY(0, total_interests - 1)) i
LEFT JOIN t.user_interests ui ON i = ui
ORDER BY i
) AS labels
Query
CREATE TEMP TABLE sample_table AS
SELECT '12345' AS userId, 'cat' AS user_interest, 1 AS label UNION ALL
SELECT '12345' AS userId, 'dog' AS user_interest, 1 AS label UNION ALL
SELECT '6789' AS userId, 'music' AS user_interest, 1 AS label UNION ALL
SELECT '6789' AS userId, 'soccer' AS user_interest, 1 AS label;
CREATE TEMP TABLE interests AS
SELECT *, COUNT(1) OVER () AS total_interests
FROM UNNEST(['cat', 'dog', 'music', 'soccer', 'coding']) interest
WITH OFFSET idx
;
SELECT userId,
ARRAY(SELECT IF(ui IS NULL, 0, 1)
FROM UNNEST(GENERATE_ARRAY(0, total_interests - 1)) i
LEFT JOIN t.user_interests ui ON i = ui
ORDER BY i
) AS labels
FROM (
SELECT userId, total_interests, ARRAY_AGG(idx) user_interests
FROM sample_table t JOIN interests i ON t.user_interest = i.interest
GROUP BY 1, 2
) t;
Query results
I think below approach will "survive" any [reasonable] data
create temp function base10to2(x float64) returns string
language js as r'return x.toString(2);';
with your_table as (
select '12345' as userid, 'cat' as user_interest, 1 as label union all
select '12345' as userid, 'dog' as user_interest, 1 as label union all
select '6789' as userid, 'music' as user_interest, 1 as label union all
select '6789' as userid, 'soccer' as user_interest, 1 as label
), interests as (
select *, pow(2, offset) weight, max(offset + 1) over() as len
from unnest(['cat', 'dog', 'music', 'soccer', 'coding']) user_interest
with offset
)
select userid,
split(rpad(reverse(base10to2(sum(weight))), any_value(len), '0'), '') labels,
from your_table
join interests
using(user_interest)
group by userid
with output

Finding created on dates for duplicates in SQL

I have one table of contact records and I'm trying to get the count of duplicate records that were created on each date. I'm not looking to include the original instance in the count. I'm using SQL Server.
Here's an example table
| email | created_on |
| ------------- | ---------- |
| aaa#email.com | 08-16-22 |
| bbb#email.com | 08-16-22 |
| zzz#email.com | 08-16-22 |
| bbb#email.com | 07-12-22 |
| aaa#email.com | 07-12-22 |
| zzz#email.com | 06-08-22 |
| aaa#email.com | 06-08-22 |
| bbb#email.com | 04-21-22 |
And I'm expecting to return
| created_on | dupe_count |
| ---------- | ---------- |
| 08-16-22 | 3 |
| 07-12-22 | 2 |
| 06-08-22 | 0 |
| 04-21-22 | 0 |
Edited to add error message:
error message
I created a sub table based on email and created date row number. Then, you query that, and ignore the date when the email first was created (row number 1). Works perfectly fine in this case.
Entire code:
Create table #Temp
(
email varchar(50),
dateCreated date
)
insert into #Temp
(email, dateCreated) values
('aaa#email.com', '08-16-22'),
('bbb#email.com', '08-16-22'),
('zzz#email.com', '08-16-22'),
('bbb#email.com', '07-12-22'),
('aaa#email.com', '07-12-22'),
('zzz#email.com', '06-08-22'),
('aaa#email.com', '06-08-22'),
('bbb#email.com', '04-21-22')
select datecreated, sum(case when r = 1 then 0 else 1 end) as duplicates
from
(
Select email, datecreated, ROW_NUMBER() over(partition by email
order by datecreated) as r from #Temp
) b
group by dateCreated
drop table #Temp
Output:
datecreated duplicates
2022-04-21 0
2022-06-08 0
2022-07-12 2
2022-08-16 3
You can calculate the difference between total count of emails for every day and the count of unique emails for the day:
select created_on,
count(email) - count(distinct email) as dupe_count
from cte
group by created_on
It seems I have misunderstood your request, and you wanted to consider previous created_on dates' too:
ct as (
select created_on,
(select case when (select count(*)
from cte t2
where t1.email = t2.email and t1.created_on > t2.created_on
) > 0 then email end) as c
from cte t1)
select created_on,
count(distinct c) as dupe_count
from ct
group by created_on
order by 1
It seems that in oracle it is also possible to aggregate it using one query:
select created_on,
count(distinct case when (select count(*)
from cte t2
where t1.email = t2.email and t1.created_on > t2.created_on
) > 0 then email end) as c
from cte t1
group by created_on
order by 1

Returning most recent row SQL Server

I have this table
CREATE TABLE Test (
OrderID int,
Person varchar(10),
LastModified Date
);
INSERT INTO Test (OrderID, Person, LastModified)
VALUES (1, 'Sam', '2018-05-15'),
(1, 'Tim','2018-05-14'),
(1, 'Kim','2018-05-05'),
(1, 'Dave','2018-05-13'),
(1, 'James','2018-05-11'),
(1, 'Fred','2018-05-05');
select * result:
| OrderID | Person | LastModified |
|---------|--------|--------------|
| 1 | Sam | 2018-05-15 |
| 1 | Tim | 2018-05-14 |
| 1 | Kim | 2018-05-05 |
| 1 | Dave | 2018-05-13 |
| 1 | James | 2018-05-11 |
| 1 | Fred | 2018-05-05 |
I am looking to return the most recent modified row which is the first row with 'Sam'.
Now i now i can use max to return the most recent date but how can i aggregate the person column to return sam?
Looking for a result set like
| OrderID | Person | LastModified |
|---------|--------|--------------|
| 1 | Sam | 2018-05-15 |
I ran this:
SELECT
OrderID,
max(Person) AS [Person],
max(LastModified) AS [LastModified]
FROM Test
GROUP BY
OrderID
but this returns:
| OrderID | Person | LastModified |
|---------|--------|--------------|
| 1 | Tim | 2018-05-15 |
Can someone advice me further please? thanks
*** UPDATE
INSERT INTO Test (OrderID, Person, LastModified)
VALUES (1, 'Sam', '2018-05-15'),
(1, 'Tim','2018-05-14'),
(1, 'Kim','2018-05-05'),
(1, 'Dave','2018-05-13'),
(1, 'James','2018-05-11'),
(1, 'Fred','2018-05-05'),
(2, 'Dave','2018-05-13'),
(2, 'James','2018-05-11'),
(2, 'Fred','2018-05-05');
So i would be looking for this result to be:
| OrderID | Person | LastModified |
|---------|--------|--------------|
| 1 | Sam | 2018-05-15 |
| 2 | Dave | 2018-05-13 |
If you always want just one record (the latest modified one) per OrderID then this would do it:
SELECT
t2.OrderID
, t2.Person
, t2.LastModified
FROM (
SELECT
MAX( LastModified ) AS LastModified
, OrderID
FROM
Test
GROUP BY
OrderID
) t
INNER JOIN Test t2
ON t2.LastModified = t.LastModified
AND t2.OrderID = t.OrderID
Expanding on your comment ("thanks very much, is there a way i can do this if there is more than one orderID e.g. multiple people and lastmodified for multiple orderID's?"), in xcvd's answer, I assume what you therefore want is this:
WITH CTE AS(
SELECT OrderId,
Person,
LastModifed,
ROW_NUMBER() OVER (PARTITION BY OrderID ORDER BY LastModified DESC) AS RN
FROM YourTable)
SELECT OrderID,
Person,
LastModified
FROM CTE
WHERE RN = 1;
How about just using TOP (1) and ORDER BY?
SELECT TOP (1) t.*
FROM Test t
ORDER BY LastModified DESC;
If you want this for each orderid, then this is a handy method in SQL Server:
SELECT TOP (1) WITH TIES t.*
FROM Test t
ORDER BY ROW_NUMBER() OVER (PARTITION BY OrderId ORDER BY LastModified DESC);
"xcvd's" answer is perfect for this, I would just like to add another solution that can be used here for the sake of showing you a method that can be used in more complex situations than this. This solution uses a nested query (sub-query) to find the MAX(LastModified) regardless of any other field and it will use the result in the original query's WHERE clause to find any results that meet the new criteria. Cheers.
SELECT OrderID
, Person
, LastModified
FROM Test
WHERE LastModified IN (SELECT MAX(LastModified)
FROM Test)
Here is one other method :
select t.*
from Test t
where LastModified = (select max(t1.LastModified) from Test t1 where t1.OrderID = t.OrderID);

Self join next timestamp

I am looking to merge timestamp from 2 different row based on Employee and punch card but the max or limit does not work with the from statement, if I only use > then i get every subsequent for everyday... I want the next higher value on a self join, also I have to mention that i have to use SQL 2008! so the lag and Lead does not work!
please help me.
SELECT , Det.name
,Det.[time]
,Det2.[time]
,Det.[type]
,det2.type
,Det.[detail]
FROM [detail] Det
join [detail] Det2 on
Det2.name = Det.name
and
Det2.time > Det.time Max 1
where det.type <>3
Table detail
NAME | Time | Type | detail
john | 10:30| 1 | On
steve| 10:32| 1 | On
john | 10:34| 2 | break
paul | 10:35| 1 | On
steve| 10:45| 3 | Off
john | 10:49| 2 | on
paul | 10:55| 3 | Off
john | 11:12| 3 | Off
Wanted result
John | 10:30 | 10:34 | 1 | 2 | On
John | 10:34 | 10:49 | 2 | 1 | Break
John | 10:49 | 11:12 | 1 | 3 | on
Steve| 10:32 | 10:45 | 1 | 3 | on
Paul | 10:35 | 10:55 | 1 | 3 | On
Thank you in advance!
You can do it with cross apply:
SELECT Det.name
,Det.[time]
,ca.[time]
,Det.[type]
,ca.type
,Det.[detail]
FROM [detail] Det
Cross Apply(Select Top 1 * From detail det2 where det.Name = det2.Name Order By det2.Time) ca
Where det.Type <> 3
As you said LAG or LEAD functions won't work for you, but you could use ROW_NUMBER() OVER (PARTITION BY name ORDER BY time DESC) on both tables and then do a JOIN on RN1 = RN2 + 1
This is just a idea, but I don't see an issue why it shouldn't work.
Query:
;WITH Data (NAME, TIME, type, detail)
AS (
SELECT 'john', CAST('10:30' AS DATETIME2), 1, 'On'
UNION ALL
SELECT 'steve', '10:32', 1, 'On'
UNION ALL
SELECT 'john', '10:34', 2, 'break'
UNION ALL
SELECT 'paul', '10:35', 1, 'On'
UNION ALL
SELECT 'steve', '10:45', 3, 'Off'
UNION ALL
SELECT 'john', '10:49', 2, 'on'
UNION ALL
SELECT 'paul', '10:55', 3, 'Off'
UNION ALL
SELECT 'john', '11:12', 3, 'Off'
)
SELECT t.NAME, LTRIM(RIGHT(CONVERT(VARCHAR(25), t.TIME, 100), 7)) AS time, LTRIM(RIGHT(CONVERT(VARCHAR(25), t2.TIME, 100), 7)) AS time, t.type, t2.type, t.detail
FROM (
SELECT ROW_NUMBER() OVER (PARTITION BY NAME ORDER BY TIME) rn, *
FROM Data
) AS t
INNER JOIN (
SELECT ROW_NUMBER() OVER (PARTITION BY NAME ORDER BY TIME) rn, *
FROM Data
) AS t2
ON t2.NAME = t.NAME
AND t2.rn = t.rn + 1;
Result:
NAME time time type type detail
----------------------------------------------
john 10:30AM 10:34AM 1 2 On
john 10:34AM 10:49AM 2 2 break
john 10:49AM 11:12AM 2 3 on
paul 10:35AM 10:55AM 1 3 On
steve 10:32AM 10:45AM 1 3 On
Any comments, concerns - let me know. :)
As #evaldas-buinauskas said,
The OVER and LAG statements in SQL will work for you.
Here is a similar example:
http://www.databasejournal.com/features/mssql/lead-and-lag-functions-in-sql-server-2012.html

How to limit SQL result with recurrance

I have a SQL Table with recurring tasks. For instance:
+------------------+----+--------------+
| Task | ID | RecurranceID |
+------------------+----+--------------+
| Take Out Garbage | 1 | 0 |
| Order Pizza | 2 | 0 |
| Eat Breakfast | 3 | 1 |
| Eat Breakfast | 4 | 1 |
| Eat Breakfast | 5 | 1 |
| Order Pizza | 6 | 0 |
+------------------+----+--------------+
Anything with a RecurranceID of 0 is not a recurring task, but otherwise it is a recurring task.
How can I show all tasks with a limit of one row on a recurring task?
I would just like the resulting set to show:
+------------------+----+
| Task | ID |
+------------------+----+
| Take Out Garbage | 1 |
| Order Pizza | 2 |
| Eat Breakfast | 3 |
| Order Pizza | 6 |
+------------------+----+
Using SQL Server 2012
Thank you!
;WITH MyCTE AS
(
SELECT Task,
ID,
ROW_NUMBER() OVER (PARTITION BY TASK ORDER BY ID) AS rn
FROM Tasks
)
SELECT *
FROM MyCTE
WHERE rn = 1
It is not clear by your sample data, but you may need to also apply RecurranceID in the PARTITION BY clause, as bellow:
;WITH MyCTE AS
(
SELECT Task,
ID,
RecurranceID,
ROW_NUMBER() OVER (PARTITION BY TASK,RecurranceID ORDER BY ID) AS rn
FROM Tasks
)
SELECT *
FROM MyCTE
WHERE rn = 1
OR RecurranceID = 0
You seem to want all non-recurring tasks returned, along with a single row for each recurring task (whether or not it shares names with a non-recurring task). Are you looking for:
SELECT Task, ID
FROM RecurringTaskTable
WHERE RecurrenceID = 0
UNION ALL
SELECT Task, MIN(ID) AS ID
FROM RecurringTaskTable
WHERE RecurrenceID <> 0
GROUP BY Task
Try this:
select distinct task
, DENSE_RANK () OVER (order by task) dr
from tasks
Hope This Helps,
DECLARE #test TABLE(Task VARCHAR(30),
ID INT,
RecurranceID INT)
INSERT INTO #test
VALUES('Take Out Garbage', '1', '0'),
('Order Pizza', '2', '0'),
('Eat Breakfast', '3', '1'),
('Eat Breakfast', '4', '1'),
('Eat Breakfast', '5', '1'),
('Order Pizza', '6', '0')
;WITH cte_OneofEach AS
(
SELECT *,
ROW_NUMBER() OVER (PARTITION BY Task ORDER BY ID DESC) AS rn
FROM #test
WHERE RecurranceID = '1'
)
SELECT Task,ID,RecurranceID
FROM #test
WHERE RecurranceID = '0'
UNION
SELECT Task,ID,RecurranceID
FROM cte_OneofEach
WHERE rn = 1