I have a text file containing the row numbers of the rows that should be deleted in my table like this:
3
32
40
55
[...]
How can I get a PostgreSQL compatible SQL statement which deletes each of these rows from my table using the text file?
Doing it once could look like this:
CREATE TEMP TABLE tmp_x (nr int);
COPY tmp_x FROM '/absolute/path/to/file';
DELETE FROM mytable d
USING tmp_x
WHERE d.mycol = tmp_x.nr;
DROP TABLE tmp_x; -- optional
Or use the psql meta-command \copy. The manual:
COPY naming a file or command is only allowed to database superusers
or users who are granted one of the roles pg_read_server_files,
pg_write_server_files, or pg_execute_server_program, since it
allows reading or writing any file or running a program that the
server has privileges to access.
Do not confuse COPY with the psql instruction \copy. \copy
invokes COPY FROM STDIN or COPY TO STDOUT, and then fetches/stores
the data in a file accessible to the psql client. Thus, file
accessibility and access rights depend on the client rather than the
server when \copy is used.
For repeated use, wrap it into a PL/pgSQL function with file-path / table name / column name as parameters. If any identifiers are dynamic you must use EXECUTE for the DELETE.
If you work with \copy, you have to do that in psql in the same session before executing SQL commands (possibly wrapped in a server-side function).
I have a slightly different solution than Erwin's. I would use IN because performing a JOIN (USING) it would increase the number of rows that the query will process.
CREATE TEMP TABLE tmp_x (nr int);
COPY tmp_x FROM '/absolute/path/to/file';
DELETE FROM mytable d
WHERE d.mycol IN (SELECT nr FROM tmp_x);
DROP TABLE tmp_x;
Related
I have an SQL script that does a bunch of stuff to some existing tables in my DB, but part of that scripts need to operate with data present on a non-existing table in order to update some values.
That table is only temporary; I have an .sql file that contains:
CREATE TABLE big_table
POPULATION of that same big_table
So the idea is, my SQL script contains the import of that .sql file, that should execute it (it will create the big_table in my DB and populate it), then my already existing script will execute all it's operations and then I'll just run a DROP TABLE big_table since I'll no longer need it.
Being an POSTGRESQL DDBB, I've thought about:
db_name < file.sql
, also:
mysql -u username -p db_name < file.sql
, but that I cannot include into an SQL script, I can only run that trough a shell which is not the idea.
Suppose, We have three files in our release package.
insertTable.sql contains all insert statement for various database table
deleteTable.sql contains all delete statement for various database table
updateTable.sql contains all update statement for various database table
Now, I want to create one single file that executes all these files. So users need not to execute all file statements one by one.
Database Environment - Oracle 12c
Not too complicated; name all those scripts in the "master" script, with their names preceded by the # sign.
For example, I have created 3 scripts: a.sql, b.sql and c.sql. All of them look similar, i.e. they just display where I am:
select 'this is script A' what from dual;
run_all.sql looks like this:
#a
#b
#c
Let's test it:
SQL> #run_all
WHAT
----------------
this is script A
WHAT
----------------
this is script B
WHAT
----------------
this is script C
SQL>
See if it helps.
I created external table with following command
db2 "
CREATE EXTERNAL TABLE TEST(a int) using
(dataobject '/home/db2inst2/test.tbl' )
)
"
db2 "insert into TEST values(1)"
db2 "insert into TEST values(2)"
But looks like it is replacing value. Is there any option to append files & do basic DML operation on external table. Please let me know if any other option available in db2 V11.5
It's not possible.
CREATE EXTERNAL TABLE statement
Restrictions
External tables cannot be used by a Db2 instance running on a Windows system.
Data being loaded must be properly formatted.
You cannot delete, truncate, or update an external table.
For remote external tables (that is, for external tables are not located in a Swift or S3 object store and for which the REMOTESOURCE option is set to a value other than LOCAL):
A single query or subquery cannot select from more than one external table at a time, and cannot reference the same external table
more than once. If necessary, combine data from several external
tables into a single table and use that table in the query.
A union operation cannot involve more than one external table.
In addition:
For an unload operation, the following conditions apply:
If the file exists, it is overwritten.
Given a set of complex, consecutive (Postgres) SQL select statements, each stored in a single sql-file, how can I write a bash-script dropping (if exists) and creating a table with the results of each statement having the same table name as file name.
background: we share these sql-files in a git repo where different users want to use the statements in a different way. I want automatically create tables, the others use temp tables, thus I dont want to write the 'create table...' in the sql-file's headers.
A skeleton for your shell script could look like this:
set -e # stop immediately on any error
for script in s1 s2 s3
do
echo "processing $script"
select=`cat $script`
psql -d dbname -U user <<EOF
DROP TABLE IF EXISTS "$script";
CREATE TABLE "$script" AS $select ;
EOF
done
Note however that any SELECT is not necessarily suitable as the source for a CREATE TABLE .. AS SELECT...
As the simplest example, consider the case when two different columns share the same name. This is legal in a SELECT, but an error condition when creating a table from it.
I'm using Postgres and would like to make a big update query that would pick up from a CSV file, lets say I got a table that's got (id, banana, apple).
I'd like to run an update that changes the Bananas and not the Apples, each new Banana and their ID would be in a CSV file.
I tried looking at the Postgres site but the examples are killing me.
COPY the file to a temporary staging table and update the actual table from there. Like:
CREATE TEMP TABLE tmp_x (id int, apple text, banana text); -- but see below
COPY tmp_x FROM '/absolute/path/to/file' (FORMAT csv);
UPDATE tbl
SET banana = tmp_x.banana
FROM tmp_x
WHERE tbl.id = tmp_x.id;
DROP TABLE tmp_x; -- else it is dropped at end of session automatically
If the imported table matches the table to be updated exactly, this may be convenient:
CREATE TEMP TABLE tmp_x AS SELECT * FROM tbl LIMIT 0;
Creates an empty temporary table matching the structure of the existing table, without constraints.
Privileges
Up to Postgres 10, SQL COPY requires superuser privileges for this.
In Postgres 11 or later, there are also some predefined roles (formerly "default roles") to allow it. The manual:
COPY naming a file or command is only allowed to database superusers
or users who are granted one of the roles pg_read_server_files,
pg_write_server_files, or pg_execute_server_program [...]
The psql meta-command \copy works for any db role. The manual:
Performs a frontend (client) copy. This is an operation that runs an
SQL COPY command, but instead of the server reading or writing the
specified file, psql reads or writes the file and routes the data
between the server and the local file system. This means that file
accessibility and privileges are those of the local user, not the
server, and no SQL superuser privileges are required.
The scope of temporary tables is limited to a single session of a single role, so the above has to be executed in the same psql session:
CREATE TEMP TABLE ...;
\copy tmp_x FROM '/absolute/path/to/file' (FORMAT csv);
UPDATE ...;
If you are scripting this in a bash command, be sure to wrap it all in a single psql call. Like:
echo 'CREATE TEMP TABLE tmp_x ...; \copy tmp_x FROM ...; UPDATE ...;' | psql
Normally, you need the meta-command \\ to switch between psql meta commands and SQL commands in psql, but \copy is an exception to this rule. The manual again:
special parsing rules apply to the \copy meta-command. Unlike most other meta-commands, the entire remainder of the line is always taken to be the arguments of \copy, and neither variable interpolation nor backquote expansion are performed in the arguments.
Big tables
If the import-table is big it may pay to increase temp_buffers temporarily for the session (first thing in the session):
SET temp_buffers = '500MB'; -- example value
Add an index to the temporary table:
CREATE INDEX tmp_x_id_idx ON tmp_x(id);
And run ANALYZE manually, since temporary tables are not covered by autovacuum / auto-analyze.
ANALYZE tmp_x;
Related answers:
Best way to delete millions of rows by ID
How can I insert common data into a temp table from disparate schemas?
How to delete duplicate entries?
I was having the same problem. But in this solution I have found some difficulties. Since I was not superuser using copy gives error. So I have found an alternative solution for my problem.
I am using postgresql and pgadmin4. here is the solution I came with.
Create a new table and copy the fruits table to the new table.
CREATE TABLE fruits_copy AS TABLE fruits WITH NO DATA;
Import the CSV file data to the new table(fruits_copy). I am using pgadmin4,so here is the how to import details. (It may differ).
Update the fruits table from fruits_copy table.
UPDATE fruits SET banana = fruits_copy.banana FROM fruits_copy WHERE fruits.id = fruits_copy.id;
After that if you want to delete the new table, you can just drop it.
DROP TABLE fruits_copy;
You can try the below code written in python, the input file is the csv file whose contents you want to update into the table. Each row is split based on comma so for each row, row[0]is the value under first column, row[1] is value under second column etc.
import csv
import xlrd
import os
import psycopg2
import django
from yourapp import settings
django.setup()
from yourapp import models
try:
conn = psycopg2.connect("host=localhost dbname=prodmealsdb
user=postgres password=blank")
cur = conn.cursor()
filepath = '/path/to/your/data_to_be_updated.csv'
ext = os.path.splitext(filepath)[-1].lower()
if (ext == '.csv'):
with open(filepath) as csvfile:
next(csvfile)
readCSV = csv.reader(csvfile, delimiter=',')
for row in readCSV:
print(row[3],row[5])
cur.execute("UPDATE your_table SET column_to_be_updated = %s where
id = %s", (row[5], row[3]))
conn.commit()
conn.close()
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()