I'm importing a legacy db to a new version of our program, and I'm wondering if there's a way to not import some columns/tables from the dump, and rename other tables/columns as i import them? I'm aware I could edit the dump file in theory, but that seems like a hack, and so far none of my editors can handle opening the 1.3 gb file (Yes, I've read the question about that on here. No, none of the answers worked for me so far.).
Suggestions?
It's possible to not import some tables by denying permissions to do so, and using --force as a command line option.
Not importing some columns, or renaming them is not possible (at least without editing the dump file, or making modifications once imported).
My recommendation would be:
Import the tables into another database (1.3G should still be very quick).
Do your dropping/renaming.
Export the data to create yourself a new dump file.
If you're worried the dump contains multiple databases, the mysql command line tool has a -o flag to only import the one.
I'd say import it into a temporary database and do the changes live - possibly applying a pre-built script that does the necessary operations:
DROP TABLE ....
DROP TABLE ....
DROP TABLE ....
ALTER TABLE ..... DROP column ....
Then copy the finished result into the production database.
This can be very nicely automated as well.
It's likely to work out faster and with less problems than finding a tool that edits dumps (or, as so often with these things, trying out five different tools and finding out none works well).
Assuming you have both databases, you could rename all tables in OldDB (just make sure the prefix isn't used already in any table name, because renaming back has a string-replace) …
USE olddb;
DROP PROCEDURE IF EXISTS rename_tables;
DELIMITER ||
CREATE PROCEDURE rename_tables(
IN plz_remove BOOLEAN
)
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE tab VARCHAR(64);
DECLARE mycursor CURSOR FOR
SELECT table_name FROM information_schema.tables
WHERE table_schema = (SELECT DATABASE() FROM DUAL)
;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN mycursor;
myloop: LOOP
FETCH mycursor INTO tab;
IF done THEN
LEAVE myloop;
END IF;
IF plz_remove THEN
SET #sql = CONCAT(
'RENAME TABLE ', tab, ' TO ', REPLACE(tab, 'olddb_', '')
);
ELSE
SET #sql = CONCAT('RENAME TABLE ', tab, ' TO olddb_', tab);
END IF;
-- construct due to RENAME × CONCAT / variables.
PREPARE s FROM #sql;
EXECUTE s;
END LOOP;
CLOSE mycursor;
END ||
DELIMITER ;
-- append 'olddb_'.
CALL rename_tables(false);
-- […]
-- rename back after dump.
CALL rename_tables(true);
… then dump and import into NewDB.
$ mysqldump -hlocalhost -uroot -p --complete-insert --routines --default-character-set=utf8 olddb > olddb.sql
$ mysql -hlocalhost -uroot -p --default-character-set=utf8 newdb < olddb.sql
This would give you (for example):
USE newdb;
SHOW TABLES;
+------------------+
| Tables_in_newdb |
+------------------+
| bar |
| foo |
| olddb_company |
| olddb_department |
| olddb_user |
| user |
+------------------+
Further reading / based on:
MySQL Manual: Cursors
MySQL Manual: Loop
MySQL Manual: Prepare
SO answer for "MySQL foreach alternative for procedure"
SO answer for "MySQL foreach loop"
SO answer for "How do you mysqldump specific table(s)?"
Related
This may be a futile excersise, but one of my client totally insisted that he needs whole database dump to perform analytics in Excel.
There are many answers to how to dump single table to csv (like this: Export to CSV and Compress with GZIP in postgres, Save PL/pgSQL output from PostgreSQL to a CSV file, Export Postgres table to CSV file with headings). There is even a closed question on this subject: https://stackoverflow.com/questions/9226229/how-to-take-whole-database-dump-in-csv-format-for-postgres. But there are no answers on how to dump whole database in single command.
Anyways, here is my script:
DO $DO$
DECLARE
r record;
BEGIN
FOR r IN select tablename from pg_tables where NOT (tablename LIKE 'pg%' OR tablename LIKE 'sql%') LOOP
EXECUTE 'copy (select * from "'|| r.tablename || '" ) to ''/tmp/dump/' || r.tablename || '.csv'' with csv header';
END LOOP;
END;
$DO$;
Some fine points:
It can be pasted into psql command and it will dump all tables in current schema to /tmp/dump directory (please create this directory first)
Query in the for loop (that is: select tablename from pg_tables where NOT (tablename LIKE 'pg%' OR tablename LIKE 'sql%') select all table names in current schema except for ones starting with pg and sql that will most likely be reserved names for postgres and SQL stuff. There most probably is a better way, but hell, who cares?
I was wondering how / if it's possible to have a query which drops all temporary tables?
I've been trying to work something out using the tempdb.sys.tables, but am struggling to format the name column to make it something that can then be dropped - another factor making things a bit trickier is that often the temp table names contain a '_' which means doing a replace becomes a bit more fiddly (for me at least!)
Is there anything I can use that will drop all temp tables (local or global) without having to drop them all individually on a named basis?
Thanks!
The point of temporary tables is that they are.. temporary. As soon as they go out of scope
#temp create in stored proc : stored proc exits
#temp created in session : session disconnects
##temp : session that created it disconnects
The query disappears. If you find that you need to remove temporary tables manually, you need to revisit how you are using them.
For the global ones, this will generate and execute the statement to drop them all.
declare #sql nvarchar(max)
select #sql = isnull(#sql+';', '') + 'drop table ' + quotename(name)
from tempdb..sysobjects
where name like '##%'
exec (#sql)
It is a bad idea to drop other sessions' [global] temp tables though.
For the local (to this session) temp tables, just disconnect and reconnect again.
The version below avoids all of the hassles of dealing with the '_'s. I just wanted to get rid of non-global temp tables, hence the '#[^#]%' in my WHERE clause, drop the [^#] if you want to drop global temp tables as well, or use a '##%' if you only want to drop global temp tables.
The DROP statement seems happy to take the full name with the '_', etc., so we don't need to manipulate and edit these. The OBJECT_ID(...) NOT NULL allows me to avoid tables that were not created by my session, presumably since these tables should not be 'visible' to me, they come back with NULL from this call. The QUOTENAME is needed to make sure the name is correctly quoted / escaped. If you have no temp tables, #d_sql will be the empty string still, so we check for that before printing / executing.
DECLARE #d_sql NVARCHAR(MAX)
SET #d_sql = ''
SELECT #d_sql = #d_sql + 'DROP TABLE ' + QUOTENAME(name) + ';
'
FROM tempdb..sysobjects
WHERE name like '#[^#]%'
AND OBJECT_ID('tempdb..'+QUOTENAME(name)) IS NOT NULL
IF #d_sql <> ''
BEGIN
PRINT #d_sql
-- EXEC( #d_sql )
END
In a stored procedure they are dropped automatically when the execution of the proc completes.
I normally come across the desire for this when I copy code out of a stored procedure to debug part of it and the stored proc does not contain the drop table commands.
Closing and reopening the connection works as stated in the accepted answer. Rather than doing this manually after each execution you can enable SQLCMD mode on the Query menu in SSMS
And then use the :connect command (adjust to your server/instance name)
:connect (local)\SQL2014
create table #foo(x int)
create table #bar(x int)
select *
from #foo
Can be run multiple times without problems. The messages tab shows
Connecting to (local)\SQL2014...
(0 row(s) affected)
Disconnecting connection from (local)\SQL2014...
I have a table with image data stored in a blob field in a MySQL database. Is there a way to export those images to files on the filesystem by using only SQL? The images should be named {imageId}.jpg
I know that it is easy to do this with Java or whatever but is it possible with just a SQL script?
Using INTO, and assuming you have write permission as the mysql user in the location you wish to store the files, you can do:
SELECT id, blob INTO DUMPFILE '/tmp/path' FROM table;
Unfortunately, in MySQL it is not possible to specify the dumpfile as an expression/variable. However, you could achieve this if you wrapped it in a stored procedure and use variables.
I don't like the idea ...
drop procedure if exists dump_image;
delimiter //
create procedure dump_image()
begin
declare this_id int;
declare cur1 cursor for select imageId from image;
open cur1;
read_loop: loop
fetch cur1 into this_id;
set #query = concat('select blob_field from image where imageId=',
this_id, ' into outfile "/tmp/xyz-', this_id,'.jpg"');
prepare write_file from #query;
execute write_file;
end loop;
close cur1;
end //
delimiter ;
Despite the error
mysql> call dump_image();
ERROR 1329 (02000): No data - zero rows fetched, selected, or processed
ls -1 /tmp/xyz*
In SQL Server I could copy sql code out of an application and paste it into SSMS, declare & assign vars that exist in the sql and run. yay great debugging scenario.
E.g. (please note I am rusty and syntax may be incorrect):
declare #x as varchar(10)
set #x = 'abc'
select * from sometable where somefield = #x
I want to do something similar with Postgres in pgAdmin (or another postgres tool, any recommendations?) where I can just drop my SQL (params & all) into something that will run against Postgres DB.
I realise you can create pgscript, but it doesn't appear to be very good, for example, if I do the equivalent of above, it doesn't put the single quotes around the value in #x, nor does it let me by doubling them up and you don't get a table out after - only text...
Currently I have a piece of SQL someone has written that has 3 unique variables in it which are used around 6 times each...
So the question is how do other people debug SQL efficiently, preferably in a similar fashion to my SQL Server days.
You can achieve this using the PREPARE, EXECUTE, DEALLOCATE commands for handling statements, which is really what we are talking about here.
For example:
PREPARE test AS SELECT * FROM users WHERE first_name = $1;
EXECUTE test ('paul');
DEALLOCATE test;
Perhaps not as graphical as some may like, but certainly workable.
I would give a shot at writing a SQL function that wraps your query. It can be something as simple as
CREATE OR REPLACE FUNCTION my_function(integer, integer)
RETURNS integer
AS
$$
SELECT $1 + $2;
$$
LANGUAGE SQL;
SELECT my_function(1, 2);
I would do this instead of a PREPARE since it will be simpler to update it. Depending on how complex the function is, you might want to also look at some of the other PL's in Postgres.
SQL procs are notoriously hard to debug. My lame but practical solution has been to write log messages to a log table, like this (please excuse syntax issues):
create table log_message (
log_timestamp timestamp not null default current_timestamp,
message varchar(1000)
);
then add lines to your stored proc like:
insert into log_message (message) values ("The value of x is " || #x);
Then after a run:
select * from log_message order by 1;
It's not pretty, but works in every DB.
How do I dump the data, and only the data, not the schema, of some SQLite3 tables of a database (not all the tables)?
The dump should be in SQL format, as it should be easily re-entered into the database later and should be done from the command line. Something like
sqlite3 db .dump
but without dumping the schema and selecting which tables to dump.
You're not saying what you wish to do with the dumped file.
To get a CSV file (which can be imported into almost everything)
.mode csv
-- use '.separator SOME_STRING' for something other than a comma.
.headers on
.out file.csv
select * from MyTable;
To get an SQL file (which can be reinserted into a different SQLite database)
.mode insert <target_table_name>
.out file.sql
select * from MyTable;
You can do this getting difference of .schema and .dump commands. for example with grep:
sqlite3 some.db .schema > schema.sql
sqlite3 some.db .dump > dump.sql
grep -vx -f schema.sql dump.sql > data.sql
data.sql file will contain only data without schema, something like this:
BEGIN TRANSACTION;
INSERT INTO "table1" VALUES ...;
...
INSERT INTO "table2" VALUES ...;
...
COMMIT;
You can specify one or more table arguments to the .dump special command, e.g.sqlite3 db ".dump 'table1' 'table2'".
Not the best way, but at lease does not need external tools (except grep, which is standard on *nix boxes anyway)
sqlite3 database.db3 .dump | grep '^INSERT INTO "tablename"'
but you do need to do this command for each table you are looking for though.
Note that this does not include schema.
Any answer which suggests using grep to exclude the CREATE lines or just grab the INSERT lines from the sqlite3 $DB .dump output will fail badly. The CREATE TABLE commands list one column per line (so excluding CREATE won't get all of it), and values on the INSERT lines can have embedded newlines (so you can't grab just the INSERT lines).
for t in $(sqlite3 $DB .tables); do
echo -e ".mode insert $t\nselect * from $t;"
done | sqlite3 $DB > backup.sql
Tested on sqlite3 version 3.6.20.
If you want to exclude certain tables you can filter them with $(sqlite $DB .tables | grep -v -e one -e two -e three), or if you want to get a specific subset replace that with one two three.
As an improvement to Paul Egan's answer, this can be accomplished as follows:
sqlite3 database.db3 '.dump "table1" "table2"' | grep '^INSERT'
--or--
sqlite3 database.db3 '.dump "table1" "table2"' | grep -v '^CREATE'
The caveat, of course, is that you have to have grep installed.
In Python or Java or any high level language the .dump does not work. We need to code the conversion to CSV by hand. I give an Python example. Others, examples would be appreciated:
from os import path
import csv
def convert_to_csv(directory, db_name):
conn = sqlite3.connect(path.join(directory, db_name + '.db'))
cursor = conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
tables = cursor.fetchall()
for table in tables:
table = table[0]
cursor.execute('SELECT * FROM ' + table)
column_names = [column_name[0] for column_name in cursor.description]
with open(path.join(directory, table + '.csv'), 'w') as csv_file:
csv_writer = csv.writer(csv_file)
csv_writer.writerow(column_names)
while True:
try:
csv_writer.writerow(cursor.fetchone())
except csv.Error:
break
If you have 'panel data, in other words many individual entries with id's add this to the with look and it also dumps summary statistics:
if 'id' in column_names:
with open(path.join(directory, table + '_aggregate.csv'), 'w') as csv_file:
csv_writer = csv.writer(csv_file)
column_names.remove('id')
column_names.remove('round')
sum_string = ','.join('sum(%s)' % item for item in column_names)
cursor.execute('SELECT round, ' + sum_string +' FROM ' + table + ' GROUP BY round;')
csv_writer.writerow(['round'] + column_names)
while True:
try:
csv_writer.writerow(cursor.fetchone())
except csv.Error:
break
Review of other possible solutions
Include only INSERTs
sqlite3 database.db3 .dump | grep '^INSERT INTO "tablename"'
Easy to implement but it will fail if any of your columns include new lines
SQLite insert mode
for t in $(sqlite3 $DB .tables); do
echo -e ".mode insert $t\nselect * from $t;"
done | sqlite3 $DB > backup.sql
This is a nice and customizable solution, but it doesn't work if your columns have blob objects like 'Geometry' type in spatialite
Diff the dump with the schema
sqlite3 some.db .schema > schema.sql
sqlite3 some.db .dump > dump.sql
grep -v -f schema.sql dump > data.sql
Not sure why, but is not working for me
Another (new) possible solution
Probably there is not a best answer to this question, but one that is working for me is grep the inserts taking into account that be new lines in the column values with an expression like this
grep -Pzo "(?s)^INSERT.*\);[ \t]*$"
To select the tables do be dumped .dump admits a LIKE argument to match the table names, but if this is not enough probably a simple script is better option
TABLES='table1 table2 table3'
echo '' > /tmp/backup.sql
for t in $TABLES ; do
echo -e ".dump ${t}" | sqlite3 database.db3 | grep -Pzo "(?s)^INSERT.*?\);$" >> /tmp/backup.sql
done
or, something more elaborated to respect foreign keys and encapsulate all the dump in only one transaction
TABLES='table1 table2 table3'
echo 'BEGIN TRANSACTION;' > /tmp/backup.sql
echo '' >> /tmp/backup.sql
for t in $TABLES ; do
echo -e ".dump ${t}" | sqlite3 $1 | grep -Pzo "(?s)^INSERT.*?\);$" | grep -v -e 'PRAGMA foreign_keys=OFF;' -e 'BEGIN TRANSACTION;' -e 'COMMIT;' >> /tmp/backup.sql
done
echo '' >> /tmp/backup.sql
echo 'COMMIT;' >> /tmp/backup.sql
Take into account that the grep expression will fail if ); is a string present in any of the columns
To restore it (in a database with the tables already created)
sqlite3 -bail database.db3 < /tmp/backup.sql
According to the SQLite documentation for the Command Line Shell For SQLite you can export an SQLite table (or part of a table) as CSV, simply by setting the "mode" to "csv" and then run a query to extract the desired rows of the table:
sqlite> .header on
sqlite> .mode csv
sqlite> .once c:/work/dataout.csv
sqlite> SELECT * FROM tab1;
sqlite> .exit
Then use the ".import" command to import CSV (comma separated value) data into an SQLite table:
sqlite> .mode csv
sqlite> .import C:/work/dataout.csv tab1
sqlite> .exit
Please read the further documentation about the two cases to consider: (1) Table "tab1" does not previously exist and (2) table "tab1" does already exist.
The best method would be to take the code the sqlite3 db dump would do, excluding schema parts.
Example pseudo code:
SELECT 'INSERT INTO ' || tableName || ' VALUES( ' ||
{for each value} ' quote(' || value || ')' (+ commas until final)
|| ')' FROM 'tableName' ORDER BY rowid DESC
See: src/shell.c:838 (for sqlite-3.5.9) for actual code
You might even just take that shell and comment out the schema parts and use that.
This version works well with newlines inside inserts:
sqlite3 database.sqlite3 .dump | grep -v '^CREATE'
In practice excludes all the lines starting with CREATE which is less likely to contain newlines
The answer by retracile should be the closest one, yet it does not work for my case. One insert query just broke in the middle and the export just stopped. Not sure what is the reason. However It works fine during .dump.
Finally I wrote a tool for the split up the SQL generated from .dump:
https://github.com/motherapp/sqlite_sql_parser/
You could do a select on the tables inserting commas after each field to produce a csv, or use a GUI tool to return all the data and save it to a csv.