I'm trying to figure out how to insert data from Table1 into Table2, then use the newly-created ID from Table2 to update the corresponding row in Table1.
I'm using Postgres 12.4 for what it's worth
Example:
I've got two tables, e.g. users and metadata
The users tables has the following columns
| id | info | metadata_id |
The metadata table has the following columns
| id | data |
I want to migrate all of my info values from the users table into the data column of the metadata table, and update my users.metadata_id (currently blank) with the corresponding metadata.id values, essentially backfilling foreign keys.
Is there any way to accomplish this gracefully? I've got a working query which locks both tables and creates a temporary sequence to insert into the metadata.id and users.metadata_id but this seems brittle and I would need to start the sequence after the highest-existing ID in the metadata table, which isn't ideal.
I've also tried to use a data-modifying CTE with a RETURNING clause to update the users table, but couldn't get that to work.
You can't use returning here, since you need to keep track of the association of users and metadata while inserting.
I think it is simpler to first pre-generate the metadata serial of each user in a CTE, using nextval(). You can then use that information to insert into metadata and update the users table:
with
candidates as (
select u.*, nextval(pg_get_serial_sequence('metadata', 'id')) new_metadata_id
from users u
),
inserted as (
insert into metadata (id, data) overriding system value
select new_metadata_id, info from candidates
)
update users u
set metadata_id = c.new_metadata_id
from candidates c
where c.id = u.id
We need the overriding system value clause in the insert statement so Postgres allows us to write to a serial column.
Demo on DB Fiddle
Related
I am building a tool that will display all the tables in a given PostgreSQL database (client's legacy app), then the user would dig in and can see all the data in given table. It is essentially a database viewer.
Next step will be to allow user to update each row, in a similar manner to how one updates data in Airtable.
While for most columns I will have the primary keys so I can use to build appropriate Update ... where ID=? statements, I realized that may not be the case always. For some join tables, for example, I do not have the ID or any other primary key.
I still would like to have the functionality where the user looks at the grid of data displayed from such columns, selects a row with click of mouse and provides new values.
PostgreSQL used to use OIDs to uniquelly identify rows for such cases, but this is no longer the case even for the legacy database I am dealing with.
The only solution I can think of is using the offset/sort order to figure out which row is to be updated, but this leads to race conditions if sort changes in the meantime or the user deletes/adds some rows.
Any ideas how I can update such "anonymous" rows?
Each table in Postgres has a system column ctid which unambiguously identifies a row. Example:
drop table if exists my_table;
create table my_table(id int, str text);
insert into my_table values
(1, 'one'),
(1, 'two'),
(2, 'one');
select ctid, *
from my_table;
ctid | id | str
-------+----+-----
(0,1) | 1 | one
(0,2) | 1 | two
(0,3) | 2 | one
(3 rows)
You can use the column in delete or update:
delete from my_table
where ctid = '(0,2)'
returning *
id | str
----+-----
1 | two
(1 row)
DELETE 1
Note however, that there is no guarantee that a row has always the same ctid, per the documentation:
ctid
The physical location of the row version within its table. Note that although the ctid can be used to locate the row version very quickly, a row's ctid will change if it is updated or moved by VACUUM FULL. Therefore ctid is useless as a long-term row identifier. The OID, or even better a user-defined serial number, should be used to identify logical rows.
This is my situation:
Hint: COLUMN (NUMBER_IDENTITY) on 2 tables have the same value but those 2 table have no relationship with each other (this is my choice for my situation)
Early database state
I had table CONTRACTS (ID, NUMBER_IDENTITY, USER_ID)
I had table SUB_CONTRACTS (ID, NUMBER_IDENTITY)
NOW
I have table CONTRACTS (ID, NUMBER_IDENTITY, USER_ID)
I add new column in SUB_CONTRACTS (ID, NUMBER_IDENTITY, USER_ID)
My problem
I should update USER_ID column in SUB_CONTRACTS with the same value that exists in CONTRACTS. I can do this update because NUMBER_IDENTITY column of the two tables is the same. So SUB_CONTRACT is complementary of CONTRACT.
Question
How to build a function, procedure or query in SQL Server that will solve my problem? Can I make a function that update in the same time all the column that meets this condition? I can solve in mini solution with query but I can't join all this mini query to solve my problem.
update SUB_CONTRACTS
set USER_ID = CONTRACTS.USER_ID
from SUB_CONTRACTS
inner join CONTRACTS on (SUB_CONTRACTS.NUMBER_IDENTITY = CONTRACTS.NUMBER_IDENTITY)
This syntax will work for SQL Server. It will update all USER_ID values in SUB_CONTRACTS to match the USER_ID values in CONTRACTS where the NUMBER_IDENTITY value is the same in the two tables.
I managed to delete 4,000 rows from a table in my 129,000-row production database (Postgres 9.4 on Heroku), but only identified the problem a few days later.
I have a backup from before the loss, but only want to selectively restore the missing rows back to the table, preserving their id's. (A complete restore is not an option as new data has since been added to the table.)
Into a local testing database I have imported the backed-up table as articles_backup, alongside the actual articles table. I want to find all the rows in articles_backups that are missing from articles and then copy these to a new table articles_restores that I will then restore to the production database, back into the articles table (preserving record id's).
This query successfully returns all the id's of the deleted records:
select articles_backups.id
from articles_backups
left outer join articles on (articles_backups.id = articles.id)
where articles.id is null
But I have not been able to copy the result to a new table. I have unsuccessfully tried:
select *
into articles_restores
from articles_backups
left outer join articles on (articles_backups.id = articles.id)
where articles.id is null;
Which gives:
ERROR: column "id" specified more than once
Basically your query with LEFT JOIN / IS NULL does what you are after:
Select rows which are not present in other table
You get the error because you select all columns from both tables, and there is an id column in both. It's not possible to create a new table with duplicate column names, and it's not what you want to begin with. Only select columns from articles_backups:
CREATE TABLE articles_restores AS
SELECT ab.*
FROM articles_backups ab
LEFT JOIN articles a USING (id)
WHERE a.id IS NULL;
While being at it I simplified your query syntax with table aliases. The USING clause is just for the convenience of shorter code. It folds the two id columns into one, but all other columns are still in there twice if you SELECT *.
Use CREATE TABLE AS. SELECT INTO is also defined by the SQL standard and implemented in Postgres, but its use is discouraged. It's used in PL/pgSQL functions for a different purpose. Details:
Creating temporary tables in SQL
You could use an except to retrieve all the rows from articles_backup that are different from articles:
(assuming both tables have the same columns in the same order)
you could also create a temp table with this info to make it easy on your repairing statements:
create table temp_articles as
select * from articles_backup
except
select * from articles
step 1 - update rows from 'articles_backup' present in articles.
This step needs attention... you will have to establish a rule to choose between the data present in articles and the one present in temp_articles.
UPDATE articles a
SET a.col1=b.col1,
a.col2=b.col2,
(... other columns ...)
FROM (SELECT * FROM temp_articles) AS b
WHERE a.id = b.id and /* your rule for data to be (or not) updated goes here */
step 2 - insert rows from 'articles_backup' not present in articles (your deleted records):
insert into articles
select * from temp_articles where id not in (select id from articles)
Let us know if you need more help.
I have to import data from an old schema to a new one, where a column 'career_name' (in table 'users') that used to be a VARCHAR now should be an INTEGER which is a foreign key to another table careers. This way I intend to tipify the data that was stored in the VARCHAR column in order to keep integrity in my database.
Actually, I do the following:
Create both new tables in the new schema
Use SELECT DISTINCT 'career_name' FROM old_table in order to obtain all possible values
INSERT into my new table careers the rows obtained above
INSERT data into my new table users using a CASE clause in order to obtain the ID from table careers of the corresponding former career_name so the relation is created.
Now the problem is the following, the table careers is big, so writing one CASE clause for each row in the table in order to import users is nearly impossible.
Is there a way to avoid this? I wish I could use my table careers as an associative array, where the key was the former career_name and the value the row's ID... Thanks!
INSERT into new_Users
SELECT Users.Name, careers.id
FROM Users inner join careers ON Users.career_name = careers.career_name
Because the schema isnt known to me, I am assuming that new_users table has 2 columns which is name and career_id.
Is there a way to avoid this? I wish I could use my table 'careers' as an associative array, where the key was the former 'career_name' and the value the row's ID... Thanks!
Er... That's what a table is, isn't it? Just join to your new careers table to use it as a lookup when doing the insert:
INSERT INTO users (blah, whatever, career_id)
SELECT
old_users_table.blah,
old_users_table.whatever,
careers.career_id
FROM
old_users_table INNER JOIN careers ON old_users_table.career_name = careers.career_name
... where users is your new users table, old_users_table is wherever you're getting the data you want to migrate, and careers is your new careers table.
Heres the situation. I have 2 tables here of the schema:
ID | COMPANY_NAME | DESC | CONTACT
ID | COMPANY_ID | X_COORDINATE | Y_COORDINATE
The first tabel contains a list of companies and the second contacts coordinates of the companies as mentioned.
The thing is that I want to merge the data in this table with the data in another set of tables which already have data. The other tables have similar structure but are already propopulated with data. The IDs are autoincremental.
SO if we have lets say companies marked 1-1000 in table1 and companies marked 1-500 in table 2. We need it merged such that ID number 1 in table 2 becomes ID 1001 when migrated to the other table. And side by side we would also want to migrated the entries in the coordinates table as well in such a way that they map with the new ids of the table. Can this be done in SQL or do I need to resort to using a script here for this kind of work.
i`m not sure i understand how many tables are there and who is table 1 ,2, but the problem is pretty clear. i think the easy way is:
back up all your database before you start this process
add a column to the destination table that will contain the original id.
insert all the records you want to merge (source) into the destination table, putting the original id in the column you added.
now you can update the geo X,Y data using the old ID
after all is done and good you can remove the original id column.
EDIT: in reply to your comment , i`ll add teh code here, since its more readable.
adapted from SQL Books Online: insert rows from another table
INSERT INTO MyNewTable (TheOriginalID, Desc)
SELECT ID, Desc
FROM OldTable;
Then you can do an update to the new table based on values from the old table like so:
UPDATE MyNewTable SET X = oldTable.X , Y = oldTable.Y where
FROM MYNewTable inner JOIN OldTable ON MYNewTable.TheOriginalID = OldTable.ID