This is a simplified extract for a more complex algorithm.
The problem is I have a simple table C_HASH like this:
CREATE TABLE C_HASH
(
HASH CHAR (48),
RANK INTEGER
);
First I fill the table with all the hash values. But because I can have duplicates in HASH, to identify the duplicates one by one I need to set the RANK by HASH.
I do this SQL statement but it is way to long, I have indexed the HASH column, with no effect:
UPDATE C_HASH a set RANK = ( select temp.rank from ( select rowid, rank() over ( PARTITION BY HASH ORDER BY ROWID ) rank from C_HASH ) temp where temp.rowid = a.rowid);
I need to optimize this! A clue?
You could use the merge syntax:
merge into c_hash c
using (
select rowid, row_number() over(partition by hash order by rowid) rank
from c_hash
) c1
on (c1.rowid = c.rowid)
when matched then update set c.rank = c1.rank
Demo on DB Fiddle
Sample data:
HASH | RANK
:----------------------------------------------- | ---:
foo | null
foo | null
foo | null
bar | null
Results:
HASH | RANK
:----------------------------------------------- | ---:
foo | 1
foo | 2
foo | 3
bar | 1
If you are going to update a lot of rows, it might be more efficient to create a new table, using the insert ... select syntax:
create table c_hash2 as
select hash, row_number() over(partition by hash order by rowid) as rank
from c_hash
This is going to take a long time, because you are updating all rows. But you can simplify the logic to:
update c_hash h
set rank = (select count(*)
from c_hash h2
where h2.hash = h.hash and h2.rowid <= h.rowid
);
This should be table to take advantage of your existing index.
Related
I have a table that contains these columns:
ID (varchar)
SETUP_ID (varchar)
MENU (varchar)
LABEL (varchar)
The thing I want to achieve is to remove all duplicates from the table based on two columns (SETUP_ID, MENU).
Table I have:
id | setup_id | menu | label |
-------------------------------------
1 | 10 | main | txt |
2 | 10 | main | txt |
3 | 11 | second | txt |
4 | 11 | second | txt |
5 | 12 | third | txt |
Table I want:
id | setup_id | menu | label |
-------------------------------------
1 | 10 | main | txt |
3 | 11 | second | txt |
5 | 12 | third | txt |
You can achieve this with a common table expression (cte)
with cte as (
select id, setup_id, menu,
row_number () over (partition by setup_id, menu, label) rownum
from atable )
delete from atable a
where id in (select id from cte where rownum >= 2)
This will give you your desired output.
Common Table Expression docs
Assuming a table named tbl where both setup_id and menu are defined NOT NULL and id is the PRIMARY KEY.
EXISTS will do nicely:
DELETE FROM tbl t0
WHERE EXISTS (
SELECT FROM tbl t1
WHERE t1.setup_id = t0.setup_id
AND t1.menu = t0.menu
AND t1.id < t0.id
);
This deletes every row where a dupe with lower id is found, effectively only keeping the row with the smallest id from each set of dupes. An index on (setup_id, menu) or even (setup_id, menu, id) will help performance with big tables a lot.
If there is no PK and no reliable UNIQUE (combination of) column(s), you can fall back to using the ctid. If NULL values can be involved, you need to specify how to deal with those.
Consider:
Delete duplicate rows from small table
How to delete duplicate rows without unique identifier
How do I (or can I) SELECT DISTINCT on multiple columns?
After cleaning up duplicates, add a UNIQUE constraint to prevent new dupes:
ALTER TABLE tbl ADD CONSTRAINT tbl_setup_id_menu_uni UNIQUE (setup_id, menu);
If you had an index on (setup_id, menu), drop that now. It's superseded by the UNIQUE constraint.
I have found a solution that fits me the best.
Here it is if anyone needs it:
DELETE FROM table_name
WHERE id IN
(SELECT id
FROM
(SELECT id,
ROW_NUMBER() OVER( PARTITION BY setup_id,
menu
ORDER BY id ) AS row_num
FROM table_name ) t
WHERE t.row_num > 1 );
link: https://www.postgresql.org/docs/current/queries-union.html
https://www.postgresql.org/docs/current/sql-select.html#SQL-DISTINCT
let's sat table name is a
select distinct on (setup_id,menu ) a.* from a;
Key point: The DISTINCT ON expression(s) must match the leftmost ORDER BY expression(s). The ORDER BY clause will normally contain additional expression(s) that determine the desired precedence of rows within each DISTINCT ON group.
Which means you can only order by setup_id,menu in this distinct on query scope.
Want the opposite:
EXCEPT returns all rows that are in the result of query1 but not in the result of query2. (This is sometimes called the difference between two queries.) Again, duplicates are eliminated unless EXCEPT ALL is used.
SELECT * FROM a
EXCEPT
select distinct on (setup_id,menu ) a.* from a;
You can try something along these lines to delete all but the first row in case of duplicates (please note that this is not tested in any way!):
DELETE FROM your_table WHERE id IN (
SELECT unnest(duplicate_ids[2:]) FROM (
SELECT array_agg(id) AS duplicate_ids FROM your_table
GROUP BY SETUP_ID, MENU
HAVING COUNT(*) > 1
)
)
)
The above collects the ids of the duplicate rows (COUNT(*) > 1) in an array (array_agg), then takes all but the first element in that array ([2:]) and "explodes" the id values into rows (unnest).
The outer query just deletes every id that ends up in that result.
For mysql the similar question is already answered here Find and remove duplicate rows by two columns
Try if any of the approach helps in this matter.
I like the below one for MySql:
ALTER IGNORE TABLE your_table ADD UNIQUE (SETUP_ID, MENU);
DELETE t1
FROM table_name t1
join table_name t2 on
(t2.setup_id = t1.setup_id or t2.menu = t1.menu) and t2.id < t1.id
There are many ways to find and delete all duplicate row(s) based on conditions. But I like inner join method, which works very fast even in a large amount of Data. Please check follows :
DELETE T1 FROM <TableName> T1
INNER JOIN <TableName> T2
WHERE
T1.id > T2.id AND
T1.<ColumnName1> = T2.<ColumnName1> AND T1.<ColumnName2> = T2.<ColumnName2>;
In your case you can write as follows :
DELETE T1 FROM <TableName> T1
INNER JOIN <TableName> T2
WHERE
T1.id > T2.id AND
T1.setup_id = T2. setup_id;
Let me know if you face any issue or need more help.
I have the following table tbl:
column1 | column2 | column 3
-----------------------------------
1 | 'value1' | 3
2 | 'value2' | 4
How to do "pivot" with column names to produce output like:
column1 | 1 | 2
column2 | 'value1' |'value2'
column3 | 3 | 4
As has been commented, the issue of data types is undefined in the question.
If you are OK with all result columns being type text (every data type can be converted to text), you can use one of these:
Plain SQL
WITH cte AS (
SELECT nu.*
FROM tbl t
, LATERAL (
VALUES
(1, t.column1::text)
, (2, t.column2)
, (3, t.column3::text)
) nu(rn, c)
)
SELECT *
FROM (TABLE cte OFFSET 0 LIMIT 3) c1
JOIN (TABLE cte OFFSET 3 LIMIT 3) c2 USING (rn);
The same with useful column names:
WITH cte AS (
SELECT nu.*
FROM tbl t
, LATERAL (
VALUES
('column1', t.column1::text)
, ('column2', t.column2)
, ('column3', t.column3::text)
) nu(rn, c)
)
SELECT * FROM (
SELECT *
FROM (TABLE cte OFFSET 0 LIMIT 3) c1
JOIN (TABLE cte OFFSET 3 LIMIT 3) c2 USING (rn)
) t (key, row1, row2);
Works in any modern version of Postgres.
The SQL string has to be adapted to the number of rows and columns. See fiddles below!
Using a document type as stepping stone
Makes for shorter code.
With many rows and many columns, performance of the SQL solution may scale better because the intermediate derived table is smaller.
(The thread is limited as you can't have more than ~ 1600 table columns in Postgres.)
Since everything is converted to text anyway, hstore seems most efficient. See:
Key value pair in PostgreSQL
SELECT key
, arr[1] AS row1
, arr[2] AS row2
FROM (
SELECT x.key, array_agg(x.value) AS arr
FROM tbl t, each(hstore(t)) x
GROUP BY 1
) sub
ORDER BY 1;
Technically speaking we would have to enforce the right sort order when in array_agg(), but that should work without explicit ORDER BY. To be absolutely sure you can add one: array_agg(x.value ORDER BY t.ctid) Using ctid for lack of information.
You can do the same with JSON functions in (Postgres 9.3+). Just replace each(hstore(t) with json_each_text(row_to_json(t). The rest is identical.
These fiddles demonstrate how to scale each query:
Original example with 2 rows of 3 columns:
db<>fiddle here
Scaled up to 3 rows of 4 columns:
db<>fiddle here
In SQLIte I would like to renumber the values in a specific column with a sequence of numbers.
For example the relevance-column in these rows:
relevance | value
-------------------
3 | value1
5 | valueb
8 | valuex
9 | valueaa
must be updated starting from 1 with increment 1:
relevance | value
-------------------
1 | value1
2 | valueb
3 | valuex
4 | valueaa
What I'm looking for, is something like this:
-- first set all to startvalue
UPDATE MyTable SET relevance = 0;
-- then renumber:
UPDATE MyTable SET relevance = (some function to increase by 1 to the previous row);
I tried this, but its not increasing, seems like Max is not evaluating on each row:
UPDATE MyTable SET relevance = (SELECT Max(relevance ))+1;
First create a temporary table where you will insert the column relevance from your table and with ROW_NUMBER() window function another column with the new sequence and then update from this temporary table:
drop table if exists temp.tmp;
create temporary table tmp as
select relevance, row_number() over (order by relevance) rn
from MyTable;
update MyTable
set relevance = (
select rn from temp.tmp
where temp.tmp.relevance = MyTable.relevance
);
drop table temp.tmp;
See the demo.
i have created generate_letters & generate_num table using below query
select chr(i) as letter from generate_series(65,90) i;
select lpad(i::text,6,'0') as num from generate_series(1,100000) i;
after doing cross join with two tables using below query
select concat_ws('','2018',gl1.letter,gl2.letter,d.num) as seq
from generate_letters gl1
cross join generate_letters gl2
cross join generate_num d limit 10;
i am getting the out put result (
2018AA000001
2018AA000002
2018AA000003
2018AA000004
2018AA000005
)
but how i need to use sequence for column(bill_id) increment using above query.
please suggest me.
Create a sequence and then use the DEFAULT clause for the required expression.
SQL Fiddle
PostgreSQL 9.6 Schema Setup:
CREATE SEQUENCE yourseq INCREMENT 1 START 1 MINVALUE 1;
CREATE TABLE yourtable
(
bill_id TEXT DEFAULT '2018AA'||lpad(NEXTVAL('yourseq'::regclass)::text, 6
, '0' ),
bill_desc TEXT
);
INSERT INTO yourtable(bill_desc) VALUES ('Telephone Bill');
INSERT INTO yourtable(bill_desc) VALUES ('Water Bill');
Query 1:
select * FROM yourtable
Results:
| bill_id | bill_desc |
|--------------|----------------|
| 2018AA000001 | Telephone Bill |
| 2018AA000002 | Water Bill |
i need Add Row Numbers To a SELECT Query without using Row_Number() function.
and without using user defined functions or stored procedures.
Select (obtain the row number) as [Row], field1, field2, fieldn from aTable
UPDATE
i am using SAP B1 DIAPI, to make a query , this system does not allow the use of rownumber() function in the select statement.
Bye.
I'm not sure if this will work for your particular situation or not, but can you execute this query with a stored procedure? If so, you can:
A) Create a temp table with all your normal result columns, plus a Row column as an auto-incremented identity.
B) Select-Insert your original query, sans the row column (SQL will fill this in automatically for you)
C) Select * on the temp table for your result set.
Not the most elegant solution, but will accomplish the row numbering you are wanting.
This query will give you the row_number,
SELECT
(SELECT COUNT(*) FROM #table t2 WHERE t2.field <= t1.field) AS row_number,
field,
otherField
FROM #table t1
but there are some restrictions when you want to use it. You have to have one column in your table (in the example it is field) which is unique and numeric and you can use it as a reference. For example:
DECLARE #table TABLE
(
field INT,
otherField VARCHAR(10)
)
INSERT INTO #table(field,otherField) VALUES (1,'a')
INSERT INTO #table(field,otherField) VALUES (4,'b')
INSERT INTO #table(field,otherField) VALUES (6,'c')
INSERT INTO #table(field,otherField) VALUES (7,'d')
SELECT * FROM #table
returns
field | otherField
------------------
1 | a
4 | b
6 | c
7 | d
and
SELECT
(SELECT COUNT(*) FROM #table t2 WHERE t2.field <= t1.field) AS row_number,
field,
otherField
FROM #table t1
returns
row_number | field | otherField
-------------------------------
1 | 1 | a
2 | 4 | b
3 | 6 | c
4 | 7 | d
This is the solution without functions and stored procedures, but as I said there are the restrictions. But anyway, maybe it is enough for you.
RRUZ, you might be able to hide the use of a function by wrapping your query in a View. It would be transparent to the caller. I don't see any other options, besides the ones already mentioned.