I have a table with 4 columns:
ID, GUID, Binary, Timestamp.
My goal is to save last 10 modifications of the binary into database. If the 11th modification is inserted, the oldest one should be removed.
My current approach is to do it in two steps (pseudo mssql):
1) DELETE FROM mytable WHERE GUID = 'XXX' AND
ID NOT IN (SELECT TOP 9 ID FROM mytable WHERE GUID = 'XXX' ORDER BY Timestamp)
2) INSERT new binary ...
Is there a way to do it more efficient, maybe with one statement? Is there a way to make it both, mssql and postgresql compatible (without TOP / Limit)?
You could use cte for compatibility:
with cte as (
select
row_number() over(order by Timestamp desc) as row_num
from mytable
where GUID = 'XXX'
)
delete from cte
where row_num > 10
editsee Gordon Linoff answer, my syntax is not working in PostgreSQL, just tested it in sqlfiddle. I'm working too much with SQL Server...
edit2
About delete and insert in one query, PostgreSQL allows that:
with cte_del as (
select
id,
row_number() over(order by id desc) as row_num
from tbl
where GUID = 'XXX'
), cte_d as (
delete from tbl where id in (select id from cte_del where row_num > 10)
)
insert into ...
select id from cte_del where row_num <= 10;
sql fiddle demo
I think the following will work in both SQL Server and Postgres:
with todelete as (
select id, row_number() over (partition by GUID order by timestamp) as seqnum
from mytable
where GUID = 'xxx'
)
delete from mytable
where id in (select id from todelete where seqnum > 10);
Related
I have one table. From that I need to get latest "Date" for each "id". I wrote query for One id. But I don't know how to apply for multiple ids.(I mean for each id)
My query for one id is (say table name is tt):
select * from (
SELECT DISTINCT id ,date FROM tt
WHERE Trim(id) ='1000082'
ORDER BY date desc
) where rownum<=1;
If you have just two columns, aggregation is good enough:
select id, max(date) max_date
from mytable
group by id
If you have more columns and you want the entire row that has the latest date for each id, then one option uses a correlated subquery for filtering:
select t.*
from mytable t
where t.date = (select max(t1.date) from mytable t1 where t1.id = t.id)
Or you can use window functions, if your database supports them:
select *
from (select t.*, row_number() over(partition by id order by date desc) rn from mytable t) t
where rn = 1
I want to drop duplicates in mytable if there are identical value in col1.
WITH CTE AS
(
SELECT
*, ROW_NUMBER() OVER (PARTITION BY col1 ORDER BY col1) AS RN
FROM
mytable
)
DELETE FROM CTE
WHERE RN <> 1
I got error:
Cannot recognize input near 'DELETE' 'FROM' 'CTE' in statement
I don't think Hive supports that syntax for DELETE. Try this:
DELETE FROM mytable t
WHERE t.id > (SELECT MIN(t2.id) -- some sort of unique id
FROM t t2
WHERE t2.id = t.id
);
If you have complete duplicates, then the above won't work. In the most recent versions of Hive you can use MERGE. In older versions:
create table temp_t as
select distinct t.*
from t;
truncate table t;
insert into t
select * from temp_t;
Of course, backup the table before trying this!
Alternative way: assuming you have UNIQUE ID Column.
Delete from MyTable where ID in
(SELECT ID FROM (SELECT *, ROW_NUMBER() OVER (PARTITION BY col1 ORDER BY col1) AS RN
FROM mytable) a where RN <> 1)
I have a quite large table with a field ID and another field as collection_time. I want to select latest record for each ID. Unfortunately combination of (ID, collection_time) time is not unique together in my data. I want just one of records with the maximum collection time. I have tried two solutions but none of them has worked for me:
First: using query
SELECT * FROM
(SELECT *, ROW_NUMBER() OVER (PARTITION BY ID ORDER BY collection_time) as rn
FROM mytable) where rn=1
This results in Resources exceeded error that I guess is because of ORDER BY in the query.
Second
Using join between table and latest time:
(SELECT tab1.*
FROM mytable AS tab1
INNER JOIN EACH
(SELECT ID, MAX(collection_time) AS second_time
FROM mytable GROUP EACH BY ID) AS tab2
ON tab1.ID=tab2.ID AND tab1.collection_time=tab2.second_time)
this solution does not work for me because (ID, collection_time) are not unique together so in JOIN result there would be multiple rows for each ID.
I am wondering if there is a workaround for the resourcesExceeded error, or a different query that would work in my case?
SELECT
agg.table.*
FROM (
SELECT
id,
ARRAY_AGG(STRUCT(table)
ORDER BY
collection_time DESC)[SAFE_OFFSET(0)] agg
FROM
`dataset.table` table
GROUP BY
id)
This will do the job for you and is scalable considering the fact that the schema keeps changing, you won't have to change this
Short and scalable version:
select array_agg(t order by collection_time desc limit 1)[offset(0)].*
from mytable t
group by t.id;
Quick and dirty option - combine your both queries into one - first get all records with latest collection_time (using your second query) and then dedup them using your first query:
SELECT * FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY tab1.ID) AS rn
FROM (
SELECT tab1.*
FROM mytable AS tab1
INNER JOIN (
SELECT ID, MAX(collection_time) AS second_time
FROM mytable GROUP BY ID
) AS tab2
ON tab1.ID=tab2.ID AND tab1.collection_time=tab2.second_time
)
)
WHERE rn = 1
And with Standard SQL (proposed by S.Mohsen sh)
WITH myTable AS (
SELECT 1 AS ID, 1 AS collection_time
),
tab1 AS (
SELECT ID,
MAX(collection_time) AS second_time
FROM myTable GROUP BY ID
),
tab2 AS (
SELECT * FROM myTable
),
joint AS (
SELECT tab2.*
FROM tab2 INNER JOIN tab1
ON tab2.ID=tab1.ID AND tab2.collection_time=tab1.second_time
)
SELECT * EXCEPT(rn)
FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY ID) AS rn
FROM joint
)
WHERE rn=1
If you don't care about writing a piece of code for every column:
SELECT ID,
ARRAY_AGG(col1 ORDER BY collection_time DESC)[OFFSET(0)] AS col1,
ARRAY_AGG(col2 ORDER BY collection_time DESC)[OFFSET(0)] AS col2
FROM myTable
GROUP BY ID
I see no one has mentioned window functions with QUALIFY:
SELECT *, MAX(collection_time) OVER (PARTITION BY id) AS max_timestamp
FROM my_table
QUALIFY collection_time = max_timestamp
The window function adds a column max_timestamp that is accessible in the QUALIFY clause to filter on.
As per your comment, Considering you have a table with unique ID's for which you need to find latest collection_time. Here is another way to do it using Correlated Sub-Query. Give it a try.
SELECT id,
(SELECT Max(collection_time)
FROM mytable B
WHERE A.id = B.id) AS Max_collection_time
FROM id_table A
Another solution, which could be more scalable since it avoids multiple scans of the same table (which will happen with both self-join and correlated subquery in above answers). This solution only works with standard SQL (uncheck "Use Legacy SQL" option):
SELECT
ID,
(SELECT srow.*
FROM UNNEST(t.srows) srow
WHERE srow.collection_time = MAX(srow.collection_time))
FROM
(SELECT ID, ARRAY_AGG(STRUCT(col1, col2, col3, ...)) srows
FROM id_table
GROUP BY ID) t
Need to find in databse max value, but then i need read other values in columns.
Can this be done with one SQL command or I have to use this two commands?
SELECT MAX(id) FROM Table;
SELECT * FROM Table WHERE id = $value;
where $value is variable from 1st command
select * from your_table
where id = (select max(id) from your_table)
or
select t1.* from your_table t1
inner join
(
select max(id) as mid
from your_table
)
t2 on t1.id = t2.mid
Probably the simplest way is:
select *
from t
order by id
limit 1
Or use top 1 or where rownum = 1 or whatever is the right logic for your database.
Note: this only returns one row. If you have duplicate such rows, then comparison to the maximum will give you all of them.
Also, if you are using a database that supports window functions:
select *
from (select t.*, row_number() over (order by id desc) as seqnum
from t
) t
where seqnum = 1;
I'm using SQL Server 2005 and I'm trying to achieve something like this:
I want to get the first x rows and the last x rows in the same select statement.
SELECT TOP(5) BOTTOM(5)
Of course BOTTOM does not exist, so I need another solution. I believe there is an easy and elegant solution that I'm not getting. Doing the select again with GROUP BY DESC is not an option.
Using a union is the only thing I can think of to accomplish this
select * from (select top(5) * from logins order by USERNAME ASC) a
union
select * from (select top(5) * from logins order by USERNAME DESC) b
Check the link
SQL SERVER – How to Retrieve TOP and BOTTOM Rows Together using T-SQL
Did you try to using rownumber?
SELECT *
FROM
(SELECT *, ROW_NUMBER() OVER (Order BY columnName) as TopFive
,ROW_NUMBER() OVER (Order BY columnName Desc) as BottomFive
FROM Table
)
WHERE TopFive <=5 or BottomFive <=5
http://www.sqlservercurry.com/2009/02/select-top-n-and-bottom-n-rows-using.html
I think you've two main options:
SELECT TOP 5 ...
FROM ...
ORDER BY ... ASC
UNION
SELECT TOP 5 ...
FROM ...
ORDER BY ... DESC
Or, if you know how many items there are in the table:
SELECT ...
FROM (
SELECT ..., ROW_NUMBER() OVER (ORDER BY ... ASC) AS intRow
FROM ...
) AS T
WHERE intRow BETWEEN 1 AND 5 OR intRow BETWEEN #Number - 5 AND #Number
Is it an option for you to use a union?
E.g.
select top 5 ... order by {specify columns asc}
union
select top 5 ... order by {specify columns desc}
i guess you have to do it using subquery only
select * from table where id in (
(SELECT id ORDER BY columnName LIMIT 5) OR
(SELECT id ORDER BY columnName DESC LIMIT 5)
)
select * from table where id in (
(SELECT TOP(5) id ORDER BY columnName) OR
(SELECT TOP(5) id ORDER BY columnName DESC)
)
EDITED
select * from table where id in (
(SELECT TOP 5 id ORDER BY columnName) OR
(SELECT TOP 5 id ORDER BY columnName DESC)
)
No real difference between this and the union that I'm aware of, but technically it is a single query.
select t.*
from table t
where t.id in (select top 5 t2.id from table t2 order by MyColumn)
or
t.id in (select top 5 t2.id from table t2 order by MyColumn desc);
SELECT *
FROM (
SELECT x, rank() over (order by x asc) as rown
FROM table
) temp
where temp.rown = 1
or temp.rown = (select count(x) from table)
Then you are out - doing the select again IS the only option, unless you want to pull in the complete result set and then throwing away everything in between.
ANY sql I cna think of is the same way - for the bottom you need to know first either how many items you have (materialize everything or use count(*)) or a reverse sort order.
Sorry if that does not suit you, but at the end.... reality does not care, and I do not see any other way to do that.
I had to do this recently for a very large stored procedure; if your query is quite large, and you want to minimize the amount of queries you could declare a #tempTable, insert into that #tempTable then query from that #tempTable,
DECLARE #tempTable TABLE ( columns.. )
INSERT INTO #tempTable
VALUES ( SELECT.. your query here ..)
SELECT TOP(5) columns FROM #tempTable ORDER BY column ASC -- returns first to last
SELECT TOP(5) columns FROM #tempTable ORDER BY column DESC -- returns last to first