I have an airflow job upserting the columns of my table on daily basis via INSERT ON CONFLICT statement. The table contains a field updated_mode of type enum which is set to automatic when the row is inserted/updated by job or manual if's done manually.
Now, I want my job to update rows only if updated_mode is set to automatic. How can I do that?
Basically, I want to do something like:
Insert into table (data) values (data) on conflict (fields) if updated_mode=automatic set data=excluded.data else do nothing
You need WHERE clause in ON CONFLICT.
INSERT INTO table_data
VALUES (data)
ON CONFLICT (fields)
DO UPDATE SET data=excluded.data
WHERE EXCLUDED.updated_mode='automatic'
Take a look at db fiddle: https://www.db-fiddle.com/f/qwRzRSFaJx4KDMYn2GEXTe/1
You should use the regular WHERE condition. The magic EXCLUDEED RECORD will contain existing confliting record. Something like that :
Insert into table (data) values (data)
on conflict (fields) do update
set data=excluded.data
WHERE updated_mode=automatic
and fields = EXCLUDEED.fields
I assume that fields is the conflicting field and table name is data
https://www.postgresql.org/docs/release/14.0/
Allow column names in the WHERE clause of ON CONFLICT to be table-qualified (Tom Lane)
Now You can more easily reference excluded column names and the original table columns.
setup the table.
create table test101(id bigint primary key, field1 text, update_mode boolean);
insert into test101 values (1,'a', TRUE);
insert into test101 values (2 ,'b', false);
excluded refer to the part you want to insert.
--this one will insert.
insert into test101(id, field1,update_mode)
values(1,'asdf', TRUE)
on conflict (id)
do update set
field1 = excluded.field1
WHERE
excluded.update_mode= test101.update_mode;
--this will not insert
insert into test101(id, field1,update_mode)
values(2,'asdf', TRUE)
on conflict (id)
do update set
field1 = excluded.field1
WHERE
excluded.update_mode= test101.update_mode;
Related
In my SQL Server table, I have a table whose PK is GUID with lots of records already.
Now I want to add records which only needs to change the COMMON_ID and COMMON_ASSET_TYPE column of some existing records.
select * from My_Table where COMMON_ASSET_TYPE = "ASSET"
I am writing sql to copy above query result, changing COMMON_ID value to new GUID value and COMMON_ASSET_TYPE value from "ASSET" to "USER", then insert the new result into My_Table.
I do not know how to write it since now I feel it is a trouble to insert records manually.
Update:
I have far more columns in table and most of them are not nullable, I want to keep all these columns' data for new records except above two columns.Is there any way if I do not have to write all these column names in sql?
Try to use NEWID if you want to create new guid:
INSERT INTO dbo.YourTable
(
COMMON_ID,
COMMON_ASSET_TYPE
)
select NEWID(), 'User' as Common_Asset_Type
from My_Table
where COMMON_ASSET_TYPE = "ASSET"
UPDATE:
As a good practice I would suggest to write all column names explicitly to have a clean and clear insert statement. However, you can use the following construction, but it is not advisable in my opinion:
insert into table_One
select
id
, isnull(name,'Jon')
from table_Two
INSERT INTO My_Table (COMMON_ID,COMMON_LIMIT_IDENTITY, COMMON_CLASS_ID,COMMON_ASSET_TYPE)
SELECT NEWID(), COMMON_LIMIT_IDENTITY, COMMON_CLASS_ID,'USER'
FROM My_Table
WHERE COMMON_ASSET_TYPE = 'ASSET'
If I've understood correctly you want to take existing records in your table, modify them, and insert them as new records in the same table.
I'll assume ID column contains the the GUID?
I'd first create a temporary table
CREATE TABLE #myTempTable(
ID UNIQUEIDENTIFIER,
Name varchar(max),
... etc
);
Fill this temp table with the records to change with your SELECT statement.
Change the records in the temp table using UPDATE statement.
Finally, Insert those "new" records back into the primary table. with INSERT INTO SELECT statement.
You will probably have to sandwitch the INSERT INTO SELECT with IDENTITY_INSERT (on/off):
SET IDENTITY_INSERT schema_name.table_name ON
SET IDENTITY_INSERT schema_name.table_name OFF
IDENTITY_INSERT "Allows explicit values to be inserted into the identity column of a table."
Hi I'm running into the following problem on SQlite3
I have a simple table
CREATE TABLE TestTable (id INT, cnt INT);
There are some rows already in the table.
I have some data I want to be inserted into the table: {(id0, cnt0), (id1, cnt1)...}
I want to insert data into the table, on id conflict, update TestTable.cnt = TestTable.cnt + value.cnt
(values.cnt is cnt0, cnt1 ... basically my data to be inserted)
*** But the problem is, there is no primary or unique constraint on id, and I am not allowed to change it!
What I currently have :
In my program I loop through all the values
UPDATE TestTABLE SET count = count + value.cnt WHERE id = value.id;
if (sqlite3_changes() == 0)
INSERT INTO MyTable (id, cnt) values (value.id, value.cnt);
But the problem is, with a very large dataset, doing 2 queries for each data entry takes too long. I'm trying to bundle multiple entries together into one call.
Please let me know if you have questions about my description, thank you for helping!
If you are able to create temporary tables, then do the following. Although I don't show it here, I suggest wrapping all this in a transaction. This technique will likely increase efficiency even if you are also able to add a temporary unique index. (In that case you could use an UPSERT with source data in the temporary table.)
CREATE TEMP TABLE data(id INT, cnt INT);
Now insert the new data into the temporary table, whether by using the host-language data libraries or crafting an insert statement similar to
INSERT INTO data (id, cnt)
VALUES (1, 100),
(2, 200),
(5, 400),
(7, 500);
Now update all existing rows using the single UPDATE statement. SQLite does not have a convenient syntax for joining tables and/or providing a source query for an UPDATE statement. However, one can use nested statement to provide similar convenience:
UPDATE TestTable AS tt
SET cnt = cnt + ifnull((SELECT cnt FROM data WHERE data.id == tt.id), 0)
WHERE tt.id IN (SELECT id FROM data);
Note that the two nested queries are independent of each other. In fact, one could eliminate the WHERE clause altogether and get the same results for this simple case. The WHERE clause is simply to make it more efficient, only attempting to update matching id's. The other subquery in the SET clause also specifies a match on id, but alone it would still allow updates of rows that don't have a match, defaulting to a null value and being converted to 0 (by isnull() function) for a no-op. By the way, without the isnull() function, the sum would result in null and would overwrite non-null values.
Finally, insert only rows with non-existing id values:
INSERT INTO TestTable (id, cnt)
SELECT data.id, data.cnt
FROM data LEFT JOIN TestTable
ON data.id == TestTable.id
WHERE TestTable.id IS NULL;
I am trying to move bytea data from one table to another, updating references in one query.
Therefore I would like to return data from the query used for the insert that is not used for the insert.
INSERT INTO file_data (data)
select image from task_log where image is not null
RETURNING id as file_data_id, task_log.id as task_log_id
But I get an error for that query:
[42P01] ERROR: missing FROM-clause entry for table "task_log"
I want to do something like:
WITH inserted AS (
INSERT INTO file_data (data)
SELECT image FROM task_log WHERE image IS NOT NULL
RETURNING id AS file_data_id, task_log.id AS task_log_id
)
UPDATE task_log
SET task_log.attachment_id = inserted.file_data_id,
task_log.attachment_type = 'INLINE_IMAGE'
FROM inserted
WHERE inserted.task_log_id = task_log.id;
But I fail to get all data used for the insert, I can't return the id from the subselect.
I was inspired by this answer on how to do that with Common Table Expressions but I can't find a way to make it work.
You need to get your table names and aliases right. Plus, the connection between the two tables is the column image (datain the new table file_data):
WITH inserted AS (
INSERT INTO file_data (data)
SELECT image
FROM task_log
WHERE image IS NOT NULL
RETURNING id, data -- can only reference target row
)
UPDATE task_log t
SET attachment_id = i.id
, attachment_type = 'INLINE_IMAGE'
FROM inserted i
WHERE t.image = i.data;
Like explained in my old answer you referenced, image must be unique in task_log for this to work:
Insert data and set foreign keys with Postgres
I added a technique how to disambiguate non-unique values in the referenced answer. Not sure if you'd want duplicate images in file_data, though.
In the RETURNING clause of an INSERT you can only reference columns from the inserted row. The manual:
The optional RETURNING clause causes INSERT to compute and return
value(s) based on each row actually inserted (...) However, any
expression using the table's columns is allowed.
Bold emphasis mine.
Fold duplicate source values
If you want distinct entries in the target table of the INSERT (task_log), all you need in this case is DISTINCT in the initial SELECT:
WITH inserted AS (
INSERT INTO file_data (data)
SELECT DISTINCT image -- fold duplicates
FROM task_log
WHERE image IS NOT NULL
RETURNING id, data -- can only reference target row
)
UPDATE task_log t
SET attachment_id = i.id
, attachment_type = 'INLINE_IMAGE'
FROM inserted i
WHERE t.image = i.data;
The resulting file_data.id is used multiple times in task_log. Be aware that multiple rows in task_log now point to the same image in file_data. Careful with updates and deletes ...
I needed to replicate duplicates so I ended up adding a temp column for the id of the used data row.
alter table file_data add column task_log_id bigint;
-- insert & update data
alter table file_data drop column task_log_id;
The full move script was
-- A new table for any file data
CREATE TABLE file_data (
id BIGSERIAL PRIMARY KEY,
data bytea
);
-- Move data from task_log to bytes
-- Create new columns to reference file_data
alter table task_log add column attachment_type VARCHAR(50);
alter table task_log add column attachment_id bigint REFERENCES file_data;
-- add a temp column for the task_id used for the insert
alter table file_data add column task_log_id bigint;
-- insert data into file_data and set references
with inserted as (
INSERT INTO file_data (data, task_log_id)
select image, id from task_log where image is not null
RETURNING id, task_log_id
)
UPDATE task_log
SET attachment_id = inserted.id,
attachment_type = 'INLINE_IMAGE'
FROM inserted
where inserted.task_log_id = task_log.id;
-- delete the temp column
alter table file_data drop column task_log_id;
-- delete task_log images
alter table task_log drop column image;
As this produces some dead data I ran a vacuum full afterwards to clean up.
But please let me repeat the warning from #ErwinBrandstetter:
Performance is much worse than for the method using a serial number I proposed in the linked answer. Adding & removing a column require's owner's privileges, a full table rewrite and exclusive locks on the table, which is poison for concurrent access.
I would like to add a BOOLEAN column to a MySQL table which will be named is_default. In this column, only one record can have is_default set to true.
How can I add this constraint to my column with MySQL?
Thanks!
UPDATE
If it is not a constraint that I should add. How are we dealing with this type of problem on DBs?
I think this is not the best way to model the situation of a single default value.
Instead, I would leave the IsDefault column out and create a separate table with one row and only the column(s) that make(s) up the primary key of your main table. In this table you place the PK value(s) that identify the default record.
This results in considerably less storage and avoids the update issue of temporarily not having a default value (or, alternatively, temporarily having two default values) when you update.
You have numerous options for ensuring that there is one-and-only-one row in the default table.
You can't have such a constraint in MySQL.
However if instead of TRUE and FALSE you use the values TRUE and NULL then it will work because a UNIQUE column can have multiple NULL values. Note that this doesn't apply to all databases, but it will work in MySQL.
CREATE TABLE table1(b BOOLEAN UNIQUE);
INSERT INTO table1 (b) VALUES (TRUE); // Succeeds
INSERT INTO table1 (b) VALUES (TRUE); // Fails: duplicate entry '1' for key 'b'
INSERT INTO table1 (b) VALUES (FALSE); // Succeeds
INSERT INTO table1 (b) VALUES (FALSE); // Fails: duplicate entry '0' for key 'b'
INSERT INTO table1 (b) VALUES (NULL); // Succeeds
INSERT INTO table1 (b) VALUES (NULL); // Succeeds!
How are we dealing with this type of problem on DBs?
In some DBMS you can create a partial index.
In PostgreSQL this would look like this:
CREATE UNIQUE INDEX only_one_true
ON the_table (is_default)
WHERE is_default
SQL Server 2008 has a very similar syntax.
On Oracle it's a bit more complicated but doable as well:
CREATE UNIQUE INDEX only_one_true
ON the_table (CASE
WHEN is_default = 1 THEN 1
ELSE null
END)
The Oracle solution might work on any DBMS that supports expression for an index definition.
Check out triggers. They were introduced in version 5.0.2, I believe. You want a "before insert" trigger. If there is already a row with is_default=true, raise an error. I don't know what problems you might with concurrency and so on, but hopefully this is enough to you started.
I don't think it is a problem with the database as much as it is a problem with your model. It is hard for me to come up with a good example of how to solve it since you haven't mentioned what type of data you are representing, but a XXXType or XXXConfiguration table would be able to hold a defaultXXXId column.
Think of it like this: Should the color blue know that it is default or should something else know that the color blue is default when used in a given context?
Changing the way you model your data is often a much better approach to cross-database compatibility than trying to use specific features of one database flavor to represent data in a way that is not necessarily natural to your problem domain if you think about it.
Check constraints are not supported in MySQL, this is the solution using trigger:
create table if not exists myTable (
id int not null auto_increment primary key,
is_default bit not null
) engine=innodb;
select 'create trigger tbi_myTable';
drop trigger if exists tbi_myTable;
delimiter //
create trigger tbi_myTable
before insert on myTable
for each row
begin
if (select count(1) from myTable where is_default=true) > 0 && NEW.is_default then
-- Signal is only in 5.6 and above use another way to raise an error: if less than 5.6
SIGNAL SQLSTATE '50000' SET MESSAGE_TEXT = 'Cannot insert into myTable only one row with is_default true is allowed!';
end if;
END //
delimiter ;
insert into myTable (is_default) values (false);
insert into myTable (is_default) values (true);
insert into myTable (is_default) values (false);
insert into myTable (is_default) values (false);
-- This will generate an error
insert into myTable (is_default) values (true);
insert into myTable (is_default) values (false);
select * from myTable;
-- will give
/*
id is_default
1 false
2 true
3 false
4 false
*/
how to
compare the values of same table(say for eg: Order table) each and every time the record get inserted ,
if the record with same values get inserted already in same table i should not insert the new record with same values. how to do that exactly in sql server 2008
If exists(select * from Order where key_column=#some_value)
print 'data already exists'
else
Insert into Order(columns) values (#some_value,...)
I'd suggest adding a unique index on the key columns...
ALTER TABLE mytable ADD UNIQUE INDEX myindex (keycolumn1, keycolumn2, ...);
That'd make it impossible to insert a duplicate by accident.