I have a large number of rows that i want to insert simultaneously into a PostgreSQL database. I need to track what id is assigned for each row that is inserted. For example say we have the table:
CREATE TABLE example
(
id serial,
name text,
CONSTRAINT example_pkey PRIMARY KEY (id),
);
Now i have some data with ids that i dont want inserted (as the serial id column will assign a new id), but i need to keep track of the mapping between the old id and new id:
old id | name
-------------
-1 | foo
-2 | bar
-3 | baz
So i wrote this query
WITH data(oldid,name) AS ( VALUES
(-1,'foo'),
(-2,'bar'),
(-3,'baz')
)
INSERT INTO example(name)
SELECT name FROM data d
RETURNING id, d.oldid
Expecting to get something back like:
id | oldid
-----------
1 | -1
2 | -2
3 | -3
However this doesn't work, as i don't believe you can return a column that wasn't inserted. Is there any alternative way to do this?
I ended up creating a function that wrapped the inserting of a single row:
CREATE OR REPLACE FUNCTION add_example(
in_name text)
RETURNS integer AS
$BODY$
DECLARE
new_id integer;
BEGIN
INSERT INTO example(name)
VALUES (in_name) RETURNING id INTO new_id;
RETURN new_id;
END;
$BODY$
LANGUAGE plpgsql;
Then i can do:
WITH data(oldid, name) AS (VALUES
(-1,'foo'),
(-2,'bar'),
(-3,'baz')
)
SELECT oldid, add_example(name) AS id
FROM data
Which returns what i expect. I'd like to see if this can be done without the function though.
CREATE SEQUENCE data_id_seq;
CREATE TABLE DATA (
id integer default nextval('data_id_seq') NOT NULL PRIMARY KEY,
oldid integer,
name text,
);
INSERT INTO DATA(oldid,name) values (-1,'foo'),(-2,'bar'),(-3,'baz') returning id,oldid;
The optional RETURNING clause causes INSERT to compute and return
value(s) based on each row actually inserted
from https://www.postgresql.org/docs/current/static/sql-insert.html
so column parasite is unavoidable for such solution:
alter table example add column old bigint;
WITH d(oldid,name) AS ( VALUES
(-1,'foo'),
(-2,'bar'),
(-3,'baz')
)
INSERT INTO example(name,old)
SELECT "name", oldid FROM d
RETURNING id, old
Related
I have a table
CREATE TABLE items(
id SERIAL PRIMARY KEY,
group_id INT NOT NULL,
item_id INT NOT NULL,
name TEXT,
.....
.....
);
I am creating a function that
takes set of row values for a single group_id, fail if multiple group_ids present in in input rows
compares it with matching values in the table (only for that group_id
updates changed values (only for the input group_id)
inserts new values
deletes table rows that are absent in the row input (compare rows with group_id and item_id)(only for the input group_id)
this is my function definition
CREATE OR REPLACE FUNCTION update_items(rows_input items[]) RETURNS boolean as $$
DECLARE
rows items[];
group_id_input integer;
BEGIN
-- get single group_id from input rows, fail if multiple group_id's present in input
-- read items of that group_id in table
-- compare input rows and table rows (of the same group_id)
-- create transaction
-- delete absent rows
-- upsert
-- return success of transaction (boolean)
END;
$$ LANGUAGE plpgsql;
I am trying to call the function in a query
select update_items(
(38,1,1283,"Name1"),
(39,1,1471,"Name2"),
(40,1,1333,"Name3")
);
I get the following error
Failed to run sql query: column "Name1" does not exist
I tried removing the id column values: that gives me the same error
What is the correct way to pass row values to a function that accepts table type array as arguments?
updates changed values
inserts new values deletes table rows that are
absent in the row input (compare rows with group_id and item_id)
If you want do upsert, you must upsert with unique constraint.
So there is two unique constraints. primary key(id), (group_id, item_id).
insert on conflict need consider these two unique constraint.
Since You want pass items[] type to the functions. So it also means that any id that is not in the input function arguments will also be deleted.
drop table if exists items cascade;
begin;
CREATE TABLE items(
id bigint GENERATED BY DEFAULT as identity PRIMARY KEY,
group_id INT NOT NULL,
item_id INT NOT NULL,
name TEXT
,unique(group_id,item_id)
);
insert into items values
(38,1,1283,'original_38'),
(39,1,1471,'original_39'),
(40,1,1333,'original_40'),
(42,1,1332,'original_42');
end;
main function:
CREATE OR REPLACE FUNCTION update_items (in_items items[])
RETURNS boolean
AS $FUNC$
DECLARE
iter items;
saved_ids bigint[];
BEGIN
saved_ids := (SELECT ARRAY (SELECT (unnest(in_items)).id));
DELETE FROM items
WHERE NOT (id = ANY (saved_ids));
FOREACH iter IN ARRAY in_items LOOP
INSERT INTO items
SELECT
iter.*
ON CONFLICT (id)
DO NOTHING;
INSERT INTO items
SELECT
iter.*
ON CONFLICT (group_id,
item_id)
DO UPDATE SET
name = EXCLUDED.name;
RAISE NOTICE 'rec.groupid: %, rec.items_id:%', iter.group_id, iter.item_id;
END LOOP;
RETURN TRUE;
END
$FUNC$
LANGUAGE plpgsql;
call it:
SELECT
*
FROM
update_items ('{"(38, 1, 1283, Name1) "," (39, 1, 1471, Name2) "," (40, 1, 1333, Name3)"}'::items[]);
references:
Iterating over integer[] in PL/pgSQL
How to match elements in an array of composite type?
IN vs ANY operator in PostgreSQL
Here's how I achieved UPSERT with DELETE missing rows, if anyone is looking to do the same.
CREATE OR REPLACE FUNCTION update_items(in_rows items[]) RETURNS INT AS $$
DECLARE
in_groups INTEGER[];
in_group_id INTEGER;
in_item_ids INTEGER[];
BEGIN
-- get single group id from input rows, fail if multiple group ids present in input
in_groups = (SELECT ARRAY (SELECT distinct(group_id) FROM UNNEST(in_rows)));
IF ARRAY_LENGTH(in_groups,1)>1 THEN
RAISE EXCEPTION 'Multiple group_ids found in input items: %', in_groups;
END IF;
in_group_id = in_groups[1];
-- delete items of this group that are absent in in_rows
in_item_ids := (SELECT ARRAY (SELECT (UNNEST(in_rows)).item_id));
DELETE FROM items
WHERE
master_code <> ANY (in_item_ids)
AND group_id = in_group_id;
-- upsert in_rows
INSERT INTO items
SELECT * FROM UNNEST(in_rows)
ON CONFLICT (group_id,item_d)
DO UPDATE SET
parent_group_id = EXCLUDED.parent_group_id,
mat_centre_id = EXCLUDED.mat_centre_id,
NAME = EXCLUDED.NAME,
opening_date = EXCLUDED.opening_date;
RETURN in_group_id;
-- return success of transaction (boolean)
END;
$$ LANGUAGE plpgsql;
This function removes rows that are missing from your in_rows
If I have a table like this:
CREATE TABLE mytable
(
id SERIAL,
content TEXT,
copyofid INTEGER
);
Is there a way to copy id into copyofid in a single insert statement?
I tried:
INSERT INTO mytable(content, copyofid) VALUES("test", id);
But that doesn't seem to work.
You can find the sequence behind your serial column using pg_get_serial_sequence() and access it using currval() to get what serial column just got as a result of your INSERT.
CREATE TABLE mytable
( id SERIAL,
content TEXT,
copyofid INTEGER
);
--this works for a single-record insert
INSERT INTO mytable
(content, copyofid)
VALUES
('test', currval(pg_get_serial_sequence('mytable','id')));
--inserting more, you'll have to handle both columns relying on the sequence
INSERT INTO mytable
( id,
content,
copyofid)
VALUES
( nextval(pg_get_serial_sequence('mytable','id')),
'test3',
currval(pg_get_serial_sequence('mytable','id'))),
( nextval(pg_get_serial_sequence('mytable','id')),
'test4',
currval(pg_get_serial_sequence('mytable','id')));
table mytable;
-- id | content | copyofid
------+---------+----------
-- 1 | test | 1
-- 2 | test3 | 2
-- 3 | test4 | 3
--(3 rows)
Fiddle
Edouard makes makes a fair point that if you can specify the conditions when you want this behaviour, you can add them to the definition:
CREATE TABLE mytable
( id SERIAL,
content TEXT,
copyofid integer
generated always as (
case when content ilike '%requires copying ID%' then id end)
stored
);
insert into mytable (content) values ('abc') returning *;
-- id | content | copyofid
------+---------+----------
-- 1 | abc |
--(1 row)
insert into mytable (content) values ('abc, but requires copying ID') returning *;
-- id | content | copyofid
------+------------------------------+----------
-- 2 | abc, but requires copying ID | 2
--(1 row)
If they vary between inserts
CREATE TABLE mytable
( id SERIAL,
content TEXT,
copyofid integer
generated always as (
case when should_copy_id then id end)
stored,
should_copy_id boolean default false
);
insert into mytable (content) values ('efg') returning *;
-- id | content | copyofid | should_copy_id
------+---------+----------+----------------
-- 1 | efg | | f
--(1 row)
insert into mytable (content,should_copy_id) values ('klm','today'::date<>'2022-10-28'::date) returning *;
-- id | content | copyofid | should_copy_id
------+---------+----------+----------------
-- 2 | klm | 2 | t
--(1 row)
The trigger will be better if
the check is fairly complex - generated columns are pretty limited in terms of the definition complexity. For example, you can't use mutable functions in them - not even STABLE are accepted
you want to save the logic and change it later without having to drop the column each time, then re-add it with a new definition (only way to alter a generated column definition)
as a part of the insert you'll want to do more than just copy the id column
The solution is to create a trigger function which is fired before inserting a new row in table mytable and which copy NEW.id into NEW.copyofid if a condition is true :
CREATE OR REPLACE FUNCTION before_insert_mytable() RETURN trigger LANGUAGE plpgsql AS $$
BEGIN
IF condition
THEN NEW.copyofid = NEW.id ;
END IF ;
RETURN NEW ;
END ; $$
CREATE OR REPLACE TRIGGER before_insert_mytable BEFORE INSERT ON mytable
FOR EACH ROW EXECUTE FUNCTION before_insert_mytable () ;
The condition can also be stated directly in the WHEN clause of the trigger instead of in the function :
CREATE OR REPLACE FUNCTION before_insert_mytable() RETURN trigger LANGUAGE plpgsql AS $$
BEGIN
NEW.copyofid = NEW.id ;
RETURN NEW ;
END ; $$
CREATE OR REPLACE TRIGGER before_insert_mytable BEFORE INSERT ON mytable
WHEN condition
FOR EACH ROW EXECUTE FUNCTION before_insert_mytable () ;
see the manual
create type recordObj as object(
rID varchar(5),
rName varchar(5));
create type recordsArr as varray(2) of recordObj;
create table RecordsTable(
ID varchar(3),
records recordsArr);
insert into RecordsTable values ('RE1', recordsArr(recordObj('RE-12','CONF')));
After successfully inserting this row, I want to add an additional record into the 'recordsArr' varray where RecordsTable ID is also 'RE1', since varray is of size 2, but I don't want to do it in the query I have just written. If I run this query again, but with two 'recordObj' inserts, it just duplicates the data and create a new row. I want add a new varray record but in the existing record where ID is 'RE1'. Any help?
You can do it via PL/SQL:
DECLARE
p_records recordsArr;
BEGIN
SELECT records
INTO p_records
FROM recordsTable
WHERE id = 'RE1'
FOR UPDATE;
p_records.EXTEND;
p_records(2) := recordObj( 'RE-13', 'ABC' );
UPDATE RecordsTable
SET records = p_records
WHERE id = 'RE1';
END;
/
Then:
SELECT id,
r.*
FROM recordsTable rt
CROSS JOIN TABLE( rt.records ) r;
Outputs:
ID
RID
RNAME
RE1
RE-12
CONF
RE1
RE-13
ABC
db<>fiddle here
If, instead, you created the data-type as a collection using:
create type recordObj as object(
rID varchar(5),
rName varchar(5)
);
create type recordsArr as table of recordObj; -- TABLE not VARRAY(2)
create table RecordsTable(
ID varchar(3),
records recordsArr
) NESTED TABLE records STORE AS recordstable__records; -- Add storage for the collection
insert into RecordsTable values ('RE1', recordsArr(recordObj('RE-12','CONF')));
Then you could use:
INSERT INTO TABLE( SELECT records FROM RecordsTable WHERE id = 'RE1' )
VALUES ( 'RE-14', 'DEF' );
But, that will not work for a VARRAY.
db<>fiddle here
I have a view I need to allow my users to update and insert on. Importantly, when they Insert, they need to be able to return the new value from the Inserted row, however right now they get NULL. It must be a view, as in the SELECT of the view, it needs to be able to return values that are the result of a join.
My underlying table definitions:
CREATE TABLE my_assets (
asset_id bigserial not null primary key,
asset_price NUMERIC(32,10) -- This will vary constantly via an independent process
);
CREATE TABLE my_transactions (
id bigserial not null primary key,
asset_id bigint not null REFERENCES my_assets(asset_id),
some_text varchar(100)
);
INSERT INTO my_assets(asset_price) SELECT 100 as asset_price;
My view that shows the result of the table:
CREATE VIEW my_transactions_view AS
SELECT tx.id, tx.asset_id, tx.some_text, a.asset_price
FROM my_transactions tx
JOIN my_assets a ON tx.asset_id = a.asset_id
My trigger that allows insertion into my_transactions_view:
CREATE OR REPLACE FUNCTION trigfx_insert_to_my_transactions_view()
RETURNS trigger AS
$BODY$
BEGIN
INSERT INTO my_transactions(asset_id, some_text)
SELECT NEW.asset_id, NEW.some_text;
RETURN NEW;
END
$BODY$
LANGUAGE 'plpgsql';
CREATE TRIGGER trig_my_transactions_view INSTEAD OF INSERT on my_transactions_view
FOR EACH ROW EXECUTE PROCEDURE trigfx_insert_to_my_transactions_view();
All good so far. However, the problem arises from trying to run the below SQL:
INSERT INTO my_transactions_view(asset_id, some_text)
SELECT 1 as asset_id, 'Hello World' as some_text
RETURNING id, asset_id, some_text;
The returned table returns NULL for ID, but I want to return the newly updated ID from the my_transactions table:
|---------------------|------------------|------------------|
| ID | asset_id | some_text |
|---------------------|------------------|------------------|
| null | 1 | Hello World |
|---------------------|------------------|------------------|
Running a subsequent SELECT * FROM my_transactions_view DOES produce the updated result:
|------------------|------------------|------------------|------------------|
| ID | asset_id | some_text | asset_price |
|------------------|------------------|------------------|------------------|
| 1 | 1 | Hello World | 100.0000000 |
|------------------|------------------|------------------|------------------|
but I need it produced during the RETURNING of the INSERT statement.
Thank you!!!
You can populate the new record with the generated ID:
CREATE OR REPLACE FUNCTION trigfx_insert_to_my_transactions_view()
RETURNS trigger AS
$BODY$
BEGIN
INSERT INTO my_transactions(asset_id, some_text)
values (NEW.asset_id, NEW.some_text);
new.id := lastval(); --<< this gets the generated id from the transactions table
RETURN NEW;
END
$BODY$
LANGUAGE plpgsql;
Online example
Alternatively you could use currval(pg_get_serial_sequence('my_transactions','id')) instead of lastval()
Turns out we can avoid the extra function call via SELECT INTO from a CTE:
CREATE OR REPLACE FUNCTION trigfx_insert_to_my_transactions_view()
RETURNS trigger AS
$BODY$
BEGIN
WITH ins_q as (INSERT INTO my_transactions(asset_id, some_text)
values (NEW.asset_id, NEW.some_text)
RETURNING id, asset_id, some_text)
SELECT ins_q.id, ins_q.asset_id, ins_q.some_text
INTO NEW.id, NEW.asset_id, NEW.some_text
FROM ins_q;
RETURN NEW;
END
$BODY$
LANGUAGE plpgsql;
See online example here.
I was running into initialisation errors (lastval is not yet defined in this session) with the new.id := lastval(); approach.
I'd like to add a unique key to column value but I must ignore rows that have the same values in columns value and header_id. For example, consider this table:
id | header_id | value
1 | 1 | a
2 | 1 | a
3 | 2 | a
So rows 1 and 2 point to same object and the unique key should accept them, but row 3 has a different header_id (pointing to another object) and, because it has the same value as object 1, it should violate unique constraint and raise an error.
Edit 16.2:1327:
I'm using a core framework that generates columns to handle history so I cannot normalize the table. My class has lots of columns but for this example I'm only considering the value column.
You could do it if you can change your table structure slightly:
your_table
id header_value
1 1
2 1
3 2
header_value
id header_id value
1 1 a
2 2 a
Add a foreign key constraint from your_table.header_value to header_value.id.
Now you can add a unique constraint on header_value.value.
You could use a trigger to simulate a unique constraint with your desired properties. Something like this would do the trick:
create or replace function sort_of_unique() returns trigger as $$
declare
got_one boolean;
begin
select exists(
select 1
from your_table
where header_id != new.header_id
and value = new.value
) into got_one;
if got_one then
raise exception 'Uniqueness violation in your_table';
end if;
return new;
end;
$$ language plpgsql;
create trigger sort_of_unique_trigger
before insert or update on your_table
for each row execute procedure sort_of_unique();
Then you'd get things like this happening:
=> insert into your_table (id, header_id, value) values (1, 1, 'a');
=> insert into your_table (id, header_id, value) values (2, 1, 'a');
=> insert into your_table (id, header_id, value) values (3, 2, 'a');
ERROR: Uniqueness violation in your_table
=> insert into your_table (id, header_id, value) values (3, 2, 'b');
=> update your_table set value = 'a' where id = 3;
ERROR: Uniqueness violation in your_table
You can create partial unique indexes by attaching a WHERE clause to the index. This allows you to apply your uniqueness constraint to slices of the table; however, I can't think of a way to get the WHERE clause to specify an "anti-slice" so I don't see a way to make this work with a partial index. I could be missing something obvious though.
After a while I found something. Using constrain CHECK with function to determine if exist (Cannot use SELECT in CHECK statement but you can use function with desired select)
CREATE OR REPLACE FUNCTION is_value_free(_header_id integer, _value varchar) RETURNS BOOLEAN AS
$$
BEGIN
RETURN NOT EXISTS (SELECT header_id,value FROM myschema.mytalbe WHERE value LIKE _value AND header_id != _header_id LIMIT 1);
END;
$$ LANGUAGE plpgsql;
ALTER TABLE mytable ADD CONSTRAINT uniq_value CHECK (is_value_free(header_id,value))