This may be a futile excersise, but one of my client totally insisted that he needs whole database dump to perform analytics in Excel.
There are many answers to how to dump single table to csv (like this: Export to CSV and Compress with GZIP in postgres, Save PL/pgSQL output from PostgreSQL to a CSV file, Export Postgres table to CSV file with headings). There is even a closed question on this subject: https://stackoverflow.com/questions/9226229/how-to-take-whole-database-dump-in-csv-format-for-postgres. But there are no answers on how to dump whole database in single command.
Anyways, here is my script:
DO $DO$
DECLARE
r record;
BEGIN
FOR r IN select tablename from pg_tables where NOT (tablename LIKE 'pg%' OR tablename LIKE 'sql%') LOOP
EXECUTE 'copy (select * from "'|| r.tablename || '" ) to ''/tmp/dump/' || r.tablename || '.csv'' with csv header';
END LOOP;
END;
$DO$;
Some fine points:
It can be pasted into psql command and it will dump all tables in current schema to /tmp/dump directory (please create this directory first)
Query in the for loop (that is: select tablename from pg_tables where NOT (tablename LIKE 'pg%' OR tablename LIKE 'sql%') select all table names in current schema except for ones starting with pg and sql that will most likely be reserved names for postgres and SQL stuff. There most probably is a better way, but hell, who cares?
Related
How can rows be inserted into a table from a table in a remote database?
We currently have a stored procedure which does this using a database link. However, we are having to delete the link because our company policy doesn't allow their usage.
begin
...
execute immediate 'insert into '|| table_name
|| ' (select * from schema_name.'|| table_name ||'#link)';
...
end;
I'm unable to use the COPY command as it appears to be not recognized.
COPY FROM username/pwd#SID
insert into table_name using (select *
from table_name);
Error report:
SQL Error: ORA-00926: missing VALUES keyword
00926. 00000 - "missing VALUES keyword"
*Cause:
*Action:
According to this SQL Plus page, COPY command is now obsolete.
Your Query syntax is slightly wrong, and you just need to specify INSERT/REPLACE/CREATE .. and INTO is NOT needed.
COPY is not obsolete, but it ends up with some encoding issues.
You would use the line continuation character to get around that.
COPY
FROM username/pwd#SID
TO username/pass#SID2
insert
table_name
using
select * from schema_name.table_name;
You can also, download the table data into a text file, and use SQL*Loader to load the data into the another Database.
I prefer the SQL*Loader option! Since maintenance is easy!
For download,
- you can either use SPOOL with delimiter option as mentioned Here
- Write a ProC/PLSQL to output the data into a File (Method 4 in ProC, OR select column names from dba_columns))
I have a table with image data stored in a blob field in a MySQL database. Is there a way to export those images to files on the filesystem by using only SQL? The images should be named {imageId}.jpg
I know that it is easy to do this with Java or whatever but is it possible with just a SQL script?
Using INTO, and assuming you have write permission as the mysql user in the location you wish to store the files, you can do:
SELECT id, blob INTO DUMPFILE '/tmp/path' FROM table;
Unfortunately, in MySQL it is not possible to specify the dumpfile as an expression/variable. However, you could achieve this if you wrapped it in a stored procedure and use variables.
I don't like the idea ...
drop procedure if exists dump_image;
delimiter //
create procedure dump_image()
begin
declare this_id int;
declare cur1 cursor for select imageId from image;
open cur1;
read_loop: loop
fetch cur1 into this_id;
set #query = concat('select blob_field from image where imageId=',
this_id, ' into outfile "/tmp/xyz-', this_id,'.jpg"');
prepare write_file from #query;
execute write_file;
end loop;
close cur1;
end //
delimiter ;
Despite the error
mysql> call dump_image();
ERROR 1329 (02000): No data - zero rows fetched, selected, or processed
ls -1 /tmp/xyz*
I'm importing a legacy db to a new version of our program, and I'm wondering if there's a way to not import some columns/tables from the dump, and rename other tables/columns as i import them? I'm aware I could edit the dump file in theory, but that seems like a hack, and so far none of my editors can handle opening the 1.3 gb file (Yes, I've read the question about that on here. No, none of the answers worked for me so far.).
Suggestions?
It's possible to not import some tables by denying permissions to do so, and using --force as a command line option.
Not importing some columns, or renaming them is not possible (at least without editing the dump file, or making modifications once imported).
My recommendation would be:
Import the tables into another database (1.3G should still be very quick).
Do your dropping/renaming.
Export the data to create yourself a new dump file.
If you're worried the dump contains multiple databases, the mysql command line tool has a -o flag to only import the one.
I'd say import it into a temporary database and do the changes live - possibly applying a pre-built script that does the necessary operations:
DROP TABLE ....
DROP TABLE ....
DROP TABLE ....
ALTER TABLE ..... DROP column ....
Then copy the finished result into the production database.
This can be very nicely automated as well.
It's likely to work out faster and with less problems than finding a tool that edits dumps (or, as so often with these things, trying out five different tools and finding out none works well).
Assuming you have both databases, you could rename all tables in OldDB (just make sure the prefix isn't used already in any table name, because renaming back has a string-replace) …
USE olddb;
DROP PROCEDURE IF EXISTS rename_tables;
DELIMITER ||
CREATE PROCEDURE rename_tables(
IN plz_remove BOOLEAN
)
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE tab VARCHAR(64);
DECLARE mycursor CURSOR FOR
SELECT table_name FROM information_schema.tables
WHERE table_schema = (SELECT DATABASE() FROM DUAL)
;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN mycursor;
myloop: LOOP
FETCH mycursor INTO tab;
IF done THEN
LEAVE myloop;
END IF;
IF plz_remove THEN
SET #sql = CONCAT(
'RENAME TABLE ', tab, ' TO ', REPLACE(tab, 'olddb_', '')
);
ELSE
SET #sql = CONCAT('RENAME TABLE ', tab, ' TO olddb_', tab);
END IF;
-- construct due to RENAME × CONCAT / variables.
PREPARE s FROM #sql;
EXECUTE s;
END LOOP;
CLOSE mycursor;
END ||
DELIMITER ;
-- append 'olddb_'.
CALL rename_tables(false);
-- […]
-- rename back after dump.
CALL rename_tables(true);
… then dump and import into NewDB.
$ mysqldump -hlocalhost -uroot -p --complete-insert --routines --default-character-set=utf8 olddb > olddb.sql
$ mysql -hlocalhost -uroot -p --default-character-set=utf8 newdb < olddb.sql
This would give you (for example):
USE newdb;
SHOW TABLES;
+------------------+
| Tables_in_newdb |
+------------------+
| bar |
| foo |
| olddb_company |
| olddb_department |
| olddb_user |
| user |
+------------------+
Further reading / based on:
MySQL Manual: Cursors
MySQL Manual: Loop
MySQL Manual: Prepare
SO answer for "MySQL foreach alternative for procedure"
SO answer for "MySQL foreach loop"
SO answer for "How do you mysqldump specific table(s)?"
I'm playing with some code from an article written by Peter Brawley found here on page 6 of the pdf. I'm trying to figure out how to automate it so that the result of the procedure is automatically placed in the select query. Right now what I am doing is calling the procedure, exporting the result into a text file, going to the text file manually (point click with mouse), copying the result and pasting it into a select statement. I haven't been able to figure out how to either insert the select statement into the procedure, or put the procedure into a table in my database or variable that I can call from the select statement. Any ideas?
Here is the sample code from Peter Brawley, that I've been trying to automate:
use database;
DROP PROCEDURE IF EXISTS writesumpivot;
DELIMITER |
CREATE PROCEDURE writesumpivot(
db CHAR(64), tbl CHAR(64), pivotcol CHAR(64), sumcol CHAR(64)
)
BEGIN
DECLARE datadelim CHAR(1) DEFAULT '"';
DECLARE comma CHAR(1) DEFAULT ',';
DECLARE singlequote CHAR(1) DEFAULT CHAR(39);
SET #sqlmode = (SELECT ##sql_mode);
SET ##sql_mode='';
SET #pivotstr = CONCAT( 'SELECT DISTINCT CONCAT(', singlequote,
',SUM(IF(', pivotcol, ' = ', datadelim, singlequote,
comma, pivotcol, comma, singlequote, datadelim,
comma, sumcol, ',0)) AS `',
singlequote, comma, pivotcol, comma, singlequote, '`',
singlequote, ') AS sumpivotarg FROM ', db, '.', tbl,
' WHERE ', pivotcol, ' IS NOT NULL' );
-- UNCOMMENT TO SEE THET MIDLEVEL SQL:
-- SELECT #pivotstr;
PREPARE stmt FROM #pivotstr;
EXECUTE stmt;
drop prepare stmt;
SET ##sql_mode=#sqlmode;
END
|
DELIMITER ;
call writesumpivot('database', 'table', 'pivotcol','sumcol');
Then the Select statement is as follows:
SELECT
infoField
[results of the call]
FROM
database.table
GROUP BY infoField;
Assuming I've ran the call, exported the results, copied them and pasted them into the select statement, my personal results of the call in the SELECT query would look something like this:
SELECT
infoField
,SUM(IF(pivotcol = "Yellow",sumcol,0)) AS `Yellow`
,SUM(IF(pivotcol = "Red",sumcol,0)) AS `Red`
,SUM(IF(pivotcol = "Purple",sumcol,0)) AS `Purple`
,SUM(IF(pivotcol = "Orange",sumcol,0)) AS `Orange`
,SUM(IF(pivotcol = "Green",sumcol,0)) AS `Green`
,SUM(IF(pivotcol = "Blue",sumcol,0)) AS `Blue`
,SUM(IF(pivotcol = "White",sumcol,0)) AS `White`
FROM database.table
GROUP BY infoField;
Running the above select statement gives me the pivot table that I need. I'm tryig to figure out how to incorporate this into a website, which is why it needs to be automated.
I tried inserting a create table, and then reference the table, but didn't get desired results.
Edited the last section of the PROCEDURE as follows:
--SELECT #pivotstr;
DROP TABLE IF EXISTS temp2;
CREATE TABLE IF NOT EXISTS temp2(sumpivotarg varchar(8000));
PREPARE stmt FROM #pivotstr;
...
changed call and select as follows:
call writesumpivot('database','table','pivotcol','sumcol');
insert into temp2(sumpivotarg) values(#pivotstr);
SELECT
table.infoField, temp2.sumpivotarg
FROM table, temp2
GROUP BY infoField
Results from this were the generic code rather than summing the contents of the cells in the database. it looks something like this:
infoField | sumpivotarg <-- Col Headings
123 | SELECT DISTINCT CONCAT('Sum(if(pivotcol=",pivotcol",sumcol,0)) AS'pivotcol,'')..
124 | SELECT DISTINCT CONCAT('Sum(if(pivotcol=",pivotcol",sumcol,0)) AS'pivotcol,'')..
125 | select DISTINCT CONCAT('Sum(if(pivotcol=",pivotcol",sumcol,0)) AS'pivotcol,'')..
I do not mean any disrespect towards mySQL, but this whole writing to a temp table solution for passing tablular data between stored procedures is suboptimal and dangerous (in real world transaction processing). I truly hope that the mySQL team will build in some enterprise level stored procedure functionality. Also, mySQL Functions not being able to return tables is a distinct disadvantage.
I have been slowly moving process over to Linux and mySQL from MSSQL. The short comings of mySQL in the procedure and function department is forcing some major kludgey type rewrites (ala temp tables and globals, etc).
I have been writing SPs for about 20 years (Sybase before SQL Server) and feel strongly that using dynamic SQL does not take advantage of the server side database. Many folks try to implement a Data layer at the client level, but the sever is better suited to this task. It is a natural division of functionality and data. Also, simultaneously running multiple precompiled calls at the server is quite a bit more optimal than repeated calls to the server, for the same processes.
Come on mySQL team, I am keeping my fingers crossed....
You could create a temp table in your DB. Use SQL insert to insert data into temp table as the result of the stored procedure execution. Afterwards you could use that temp table inside your select statement.
Here's an answer that shows how to do that:
Use result set of mysql stored procedure in another stored procedure
Just to mention a similar question:
MySQL How to INSERT INTO temp table FROM Stored Procedure
Is there an Oracle SQL tool that builds insert statements from a result set? We are currently only allowed to use a tool called SQL Station. I'd like to either suggest a tool, like Rapid SQL or CrazySQuirrell, or build my own re-usable chunk of sql.
Where is this result set coming from? If you mean that you want to execute a SELECT, then insert the resulting data into another table, you can do that in a single SQL statement:
INSERT INTO table2 (columnA, columnB)
SELECT columnA, columnB
FROM table1;
PL/SQL Developer will do this as well. I've used both PL/SQL Developer as well as Oracle's SQL Developer, and in my opinion PL/SQL Developer has a smoother and more consistent interface. Not sure about SQL Developer, but PL/SQL Dev. also lets you export result sets as CSV,XML, and HTML.
It also behaves OK under WINE if you're running Linux.
If you want command line tools, the free cx_OracleTools will do this, and some other nice things as well.
http://cx-oracletools.sourceforge.net/
CompileSource - execute statements in a file, checking for errors
CopyData - copy data from one table or view to another
DbDebugger - allows simple debugging of PL/SQL
DescribeObject - describe objects as SQL statements for recreation
DescribeSchema - describe multiple objects as SQL statements for recreation
DumpCSV - dump the results of a select statement as comma separated values
DumpData - dump the results of a select statement as insert statements
ExportColumn - dump the data from a column into a file
ExportData - dump the data from a database into a portable dump file
ExportObjects - describe object as SQL statements for recreation in files
ExportXML - export data from a table into a simple XML file
GeneratePatch - generate SQL script to go from one set of objects to another
GenerateView - generate a view statement for a table
ImportColumn - import the contents of a file into a column in the database
ImportData - import the data dumped with ExportData
ImportXML - import data from an XML file (such as those created by ExportXML)
RebuildTable - generate SQL script to rebuild the table
RecompileSource - recompile all invalid objects in the database
Yes look at Oracle sql developer.Its free can be downloaded from otn.oracle.com
I found this solution, which is what I'm using now. Thanks for all of the help.
It turns out we can use SQL+ too. For some reason I can't run it in SQL Station.
COPY FROM userid/password#from_DB TO userid/password>#to_DB INSERT toDB_tablename USING SELECT * FROM fromDB_tablename where ....;
commit;
In a pinch, using string contatenation works great for smaller statements you want to build:
Select
'Insert Into MyOtherTableTable Values(''' || MyMainTableColumn1 || ''' and ''' || MyMainTableColumn2 || ''')'
From MyMainTable
Right click on the result set of the query, you will get a pop up. select export data and insert. it will ask you for the location to save the file in which insert statements are generated. give file name and the path to save it.
I know it is too late but It could be helpfull for somebody.
If you go to the table, you can "export" the data. The second step is "Specify Data" where you can add some filters.
This only works for a table data.
Cheers
With Oracle SQL-Developer type and execute as script (F5):
select /*insert*/
* from dual;
output:
Insert into "dual" (DUMMY) values ('X');
you can try also /*csv*/" or /*html*/
source: http://www.thatjeffsmith.com/archive/2012/05/formatting-query-results-to-csv-in-oracle-sql-developer/
SELECT /*csv*/ * FROM scott.emp;
SELECT /*xml*/ * FROM scott.emp;
SELECT /*html*/ * FROM scott.emp;
SELECT /*delimited*/ * FROM scott.emp;
SELECT /*insert*/ * FROM scott.emp;
SELECT /*loader*/ * FROM scott.emp;
SELECT /*fixed*/ * FROM scott.emp;
SELECT /*text*/ * FROM scott.emp;