I have the following SQL script -
--This is a function
SELECT * FROM import ('test');
TRUNCATE table some_table;
cat <<SQL
\COPY (
SELECT * from large_query
)
TO '/tmp/dash.csv' WITH DELIMITER ',' CSV HEADER
SQL
;
I am getting a parse error when I run this script like this -
psql -h host -p port -U user db -f my_file.sql
Can regular SQL statements not be combined with the \COPY command?
I'm trying to write a script in R to export a query from a Postgresql database to a csv file.
If I do this from cmd it works fine :
psql -U postgres
\copy (select * from clickthru.train limit 10) to 'c:\\me\\psql_test.csv' with csv header;
However when I try this in R it looks like it executes (no errors) but no file is generated:
system('psql -U postgres COPY (select * from clickthru.train limit 10;) TO "C:\\me\\psql_test.csv" with CSV')
Any suggestions?
use;
\copy (select * from clickthru.train limit 10) to 'c:\\me\\psql_test.csv' DELIMITER ',' CSV HEADER;
There are several issues here
Query should be passed via -c argument: psql -U postgres -c '... query ...'
COPY and \copy work differently. COPY operates on servers disk space, \copy operates on client disk space. If you run both on one computer, then the difference is that COPY will work as postgres user, and \copy as user who runs the script.
In your example you have extra semicolon (;) in query ...COPY (select * from clickthru.train limit 10;)...
I'm trying to insert data from excel sheet to sql database. The query is stored in a text file as follows:
insert into [demo].[dbo].[relative]
select *
from openrowset('Microsoft.Jet.OLEDB.4.0','Excel 8.0;Database=D:\relative.xls','select * from [sheet1$]');
When I am executing the following command:
sqlcmd -S ADMIN-PC/SEXPRESS -i d:\demo.txt.
it is showing this error:
Msg 7357, Level 16, State 2, Server ADMIN-PC\SEXPRESS, Line 1
Can anyone please help in rectifying my problem.
Try using the sql server import vizard to create a new table from the xls file and then insert that data to the existing table from there. The problem you are having is maybe due to the non-compatibility between 64bit sql instance and 32 bit excel.
Or try using bcp
bcp demo.dbo.relative in "D:\relative.xls" -c -T
There is another way to get the same result..
create a temp table.
declare #sometable table (value varchar(50), translation varchar(max))
select * into #sometable from YOUR_DATABASE_TABLE (nolock)
Then, do your OPENROWSET, BCP, etc. from here..
You can create a shell script which will automatically read the insert commands from the .csv file and then write it to the database. If you want I can help you up with it. What you just need to do is to write all the insert statements in the .csv file.
#!/bin/ksh
sqlplus -silent /nolog << EOF > /dev/null
username/pwd#"Connection String"
set linesize 0;
set pagesize 0;
set echo off;
while read line; do
A=`echo "$line" | awk -F" " {print 1}`
and so on depends on the number of words in the insert statements.
$A $B
done < your_insert_statements.csv
It will read the .csv file and automatically insert the records in the database.
I'm trying to dump the contents of a particular schema in one IBM DB2 UDB server into an sql text file (much like the mysqldump functionality of mysql).
I came across db2look, but it only dumps the structure of the schema (only ddl, no dml).
So how can I get my thing done?
jrh.
What you're looking for is the db2move command. For a particular schema you should use the "sn" switch.
So for example to export the data:
db2move [your_db_name] EXPORT -sn [your_schema_name]
There are many options and switches available for db2move depending on exactly what you want to do.
If db2move is not exactly what you need, you can review the table of Data Movement Options available in DB2.
You could use SQquirreL, an SQL Client implemented in Java, to accomplish this. In its "Objects"-Tree you would select all desired tables and select "Scripts > Create Data Script" from the context menu.
You could use the EXPORT, and related IMPORT or LOAD commands if the goal is to transfer data back into another DB2 database.
In fact, you can generate the the statements based on metadata from SYSCAT.TABLES
EXPORT
SELECT 'EXPORT TO /usr/data/SCHEMA/' || TABNAME || '.ixf OF IXF LOBS TO /usr/data/SCHEMA/lbos/ MODIFIED BY LOBSINFILE SELECT * FROM SCHEMA.' || TABNAME || ';'
FROM SYSCAT.TABLES
WHERE TABSCHEMA = 'SCHEMA'
ORDER BY TABNAME
IMPORT
SELECT 'IMPORT FROM /usr/data/SCHEMA/' || TABNAME || '.ixf OF IXF LOBS FROM /usr/data/SCHEMA/lobs/ MODIFIED BY LOBSINFILE INSERT INTO SCHEMA.' || TABNAME || ';'
FROM SYSCAT.TABLES
WHERE TABSCHEMA = 'SCHEMA'
ORDER BY TABNAME
If you want the actual insert scripts, then you may need to go with a third-party tool (I'm not aware of one provided by DB2, though I could be wrong.)
Db2 Schema with all DDL Backup :
I have use below command it worked for me to export all DDL.
db2look -d CusDb -x -e -z CusSchema -o OutputFile
Syntax : db2look -d DbName -x -e -z SchemaName -o OutputFile_name
For importation, adjusting slightly to use the load helps avoid rows being rejected.
db2 -x "SELECT 'load FROM /usr/data/SCHEMA/' || TABNAME || '.ixf OF IXF LOBS FROM /usr/data/SCHEMA/ MODIFIED BY identityoverride INSERT INTO CFEXT.' || TABNAME || ';'FROM SYSCAT.TABLES WHERE TABSCHEMA = 'CFEXT' ORDER BY TABNAME" > /tmp/db2cfeimport.sql
How do I dump the data, and only the data, not the schema, of some SQLite3 tables of a database (not all the tables)?
The dump should be in SQL format, as it should be easily re-entered into the database later and should be done from the command line. Something like
sqlite3 db .dump
but without dumping the schema and selecting which tables to dump.
You're not saying what you wish to do with the dumped file.
To get a CSV file (which can be imported into almost everything)
.mode csv
-- use '.separator SOME_STRING' for something other than a comma.
.headers on
.out file.csv
select * from MyTable;
To get an SQL file (which can be reinserted into a different SQLite database)
.mode insert <target_table_name>
.out file.sql
select * from MyTable;
You can do this getting difference of .schema and .dump commands. for example with grep:
sqlite3 some.db .schema > schema.sql
sqlite3 some.db .dump > dump.sql
grep -vx -f schema.sql dump.sql > data.sql
data.sql file will contain only data without schema, something like this:
BEGIN TRANSACTION;
INSERT INTO "table1" VALUES ...;
...
INSERT INTO "table2" VALUES ...;
...
COMMIT;
You can specify one or more table arguments to the .dump special command, e.g.sqlite3 db ".dump 'table1' 'table2'".
Not the best way, but at lease does not need external tools (except grep, which is standard on *nix boxes anyway)
sqlite3 database.db3 .dump | grep '^INSERT INTO "tablename"'
but you do need to do this command for each table you are looking for though.
Note that this does not include schema.
Any answer which suggests using grep to exclude the CREATE lines or just grab the INSERT lines from the sqlite3 $DB .dump output will fail badly. The CREATE TABLE commands list one column per line (so excluding CREATE won't get all of it), and values on the INSERT lines can have embedded newlines (so you can't grab just the INSERT lines).
for t in $(sqlite3 $DB .tables); do
echo -e ".mode insert $t\nselect * from $t;"
done | sqlite3 $DB > backup.sql
Tested on sqlite3 version 3.6.20.
If you want to exclude certain tables you can filter them with $(sqlite $DB .tables | grep -v -e one -e two -e three), or if you want to get a specific subset replace that with one two three.
As an improvement to Paul Egan's answer, this can be accomplished as follows:
sqlite3 database.db3 '.dump "table1" "table2"' | grep '^INSERT'
--or--
sqlite3 database.db3 '.dump "table1" "table2"' | grep -v '^CREATE'
The caveat, of course, is that you have to have grep installed.
In Python or Java or any high level language the .dump does not work. We need to code the conversion to CSV by hand. I give an Python example. Others, examples would be appreciated:
from os import path
import csv
def convert_to_csv(directory, db_name):
conn = sqlite3.connect(path.join(directory, db_name + '.db'))
cursor = conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
tables = cursor.fetchall()
for table in tables:
table = table[0]
cursor.execute('SELECT * FROM ' + table)
column_names = [column_name[0] for column_name in cursor.description]
with open(path.join(directory, table + '.csv'), 'w') as csv_file:
csv_writer = csv.writer(csv_file)
csv_writer.writerow(column_names)
while True:
try:
csv_writer.writerow(cursor.fetchone())
except csv.Error:
break
If you have 'panel data, in other words many individual entries with id's add this to the with look and it also dumps summary statistics:
if 'id' in column_names:
with open(path.join(directory, table + '_aggregate.csv'), 'w') as csv_file:
csv_writer = csv.writer(csv_file)
column_names.remove('id')
column_names.remove('round')
sum_string = ','.join('sum(%s)' % item for item in column_names)
cursor.execute('SELECT round, ' + sum_string +' FROM ' + table + ' GROUP BY round;')
csv_writer.writerow(['round'] + column_names)
while True:
try:
csv_writer.writerow(cursor.fetchone())
except csv.Error:
break
Review of other possible solutions
Include only INSERTs
sqlite3 database.db3 .dump | grep '^INSERT INTO "tablename"'
Easy to implement but it will fail if any of your columns include new lines
SQLite insert mode
for t in $(sqlite3 $DB .tables); do
echo -e ".mode insert $t\nselect * from $t;"
done | sqlite3 $DB > backup.sql
This is a nice and customizable solution, but it doesn't work if your columns have blob objects like 'Geometry' type in spatialite
Diff the dump with the schema
sqlite3 some.db .schema > schema.sql
sqlite3 some.db .dump > dump.sql
grep -v -f schema.sql dump > data.sql
Not sure why, but is not working for me
Another (new) possible solution
Probably there is not a best answer to this question, but one that is working for me is grep the inserts taking into account that be new lines in the column values with an expression like this
grep -Pzo "(?s)^INSERT.*\);[ \t]*$"
To select the tables do be dumped .dump admits a LIKE argument to match the table names, but if this is not enough probably a simple script is better option
TABLES='table1 table2 table3'
echo '' > /tmp/backup.sql
for t in $TABLES ; do
echo -e ".dump ${t}" | sqlite3 database.db3 | grep -Pzo "(?s)^INSERT.*?\);$" >> /tmp/backup.sql
done
or, something more elaborated to respect foreign keys and encapsulate all the dump in only one transaction
TABLES='table1 table2 table3'
echo 'BEGIN TRANSACTION;' > /tmp/backup.sql
echo '' >> /tmp/backup.sql
for t in $TABLES ; do
echo -e ".dump ${t}" | sqlite3 $1 | grep -Pzo "(?s)^INSERT.*?\);$" | grep -v -e 'PRAGMA foreign_keys=OFF;' -e 'BEGIN TRANSACTION;' -e 'COMMIT;' >> /tmp/backup.sql
done
echo '' >> /tmp/backup.sql
echo 'COMMIT;' >> /tmp/backup.sql
Take into account that the grep expression will fail if ); is a string present in any of the columns
To restore it (in a database with the tables already created)
sqlite3 -bail database.db3 < /tmp/backup.sql
According to the SQLite documentation for the Command Line Shell For SQLite you can export an SQLite table (or part of a table) as CSV, simply by setting the "mode" to "csv" and then run a query to extract the desired rows of the table:
sqlite> .header on
sqlite> .mode csv
sqlite> .once c:/work/dataout.csv
sqlite> SELECT * FROM tab1;
sqlite> .exit
Then use the ".import" command to import CSV (comma separated value) data into an SQLite table:
sqlite> .mode csv
sqlite> .import C:/work/dataout.csv tab1
sqlite> .exit
Please read the further documentation about the two cases to consider: (1) Table "tab1" does not previously exist and (2) table "tab1" does already exist.
The best method would be to take the code the sqlite3 db dump would do, excluding schema parts.
Example pseudo code:
SELECT 'INSERT INTO ' || tableName || ' VALUES( ' ||
{for each value} ' quote(' || value || ')' (+ commas until final)
|| ')' FROM 'tableName' ORDER BY rowid DESC
See: src/shell.c:838 (for sqlite-3.5.9) for actual code
You might even just take that shell and comment out the schema parts and use that.
This version works well with newlines inside inserts:
sqlite3 database.sqlite3 .dump | grep -v '^CREATE'
In practice excludes all the lines starting with CREATE which is less likely to contain newlines
The answer by retracile should be the closest one, yet it does not work for my case. One insert query just broke in the middle and the export just stopped. Not sure what is the reason. However It works fine during .dump.
Finally I wrote a tool for the split up the SQL generated from .dump:
https://github.com/motherapp/sqlite_sql_parser/
You could do a select on the tables inserting commas after each field to produce a csv, or use a GUI tool to return all the data and save it to a csv.