I have a function like so -
CREATE
OR REPLACE FUNCTION ind (bucket text) RETURNS table (
middle character varying (100),
last character varying (100)
) AS $body$ BEGIN return query
select
fname as first,
lname as last
from all_records
; END;
$body$ LANGUAGE PLPGSQL;
How do I output the results of select ind ('Mob') into a tsv file?
I want the output to look like this -
first last
MARY KATHERINE
You can use the COPY command
example:
COPY (select * from ind('Mob')) TO '/tmp/ind.tsv' CSV HEADER DELIMITER E'\t';
the file '/tmp/ind.tsv' will contain you data
Postgres doesn't allow copy with header for tsv for some reason.
If you're using a linux based system you can do it with a script like this:
#create file with tab delimited column list (use \t between each column name)
echo -e "user_id\temail" > user_output.tsv
#now you can append the results of your query to that file by copying to STDOUT
psql -h your_host_name -d your_database_name -c "\copy (SELECT user_id, email FROM my_user_table) to STDOUT;" >> user_output.tsv
Alternatively, if your script is long and you don't want to pass it in with -c command you can use the same approach from a .sql file, use "--quiet" to avoid notices being passed into your file
psql --quiet -h your_host_name -d your_database_name -f your_sql_file.sql >> user_output.tsv
Related
This is an image of what I'm asking for
I am using the following -echo- in a script and after I execute, the output format is as shown below:
`echo -e "UPDATE table1 SET table1_f1='$Fname' ,table1_f2='$Lname' where table1_f3='$id';\ncommit;" >> $OutputFile`
output: UPDATE table1 SET table1_f1='Fname' ,table1_f2='Lname' where table1_f3='id ';
the '; is appearing on a new line, why is that happening?
The variable $id in your shell script actually contains that newline (\n or \r\n) at the end; so there isn't really anything wrong in the part of the script you've shown here.
This effect is pretty common if the variable is created based on external commands (update:) or by reading external files as you are here.
For simple values, one way to strip the newline off the end of the value, prior to using it in your echo is:
id=$( echo "${id}" | tr -s '\r' '' | tr -s '\n' '' );
or for scripts that already rely on a particular bash IFS value:
OLDIFS="${IFS}";
IFS=$'\n\t ';
id=$( echo "${id}" | tr -s '\r' '' | tr -s '\n' '' );
IFS="${OLDIFS}";
I have PostgreSQL database dump by pg_dump version 9.5.2, which contains DDLs and also INSERT INTO statements for each table in given database. Dump looks like this:
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE unimportant_table (
id integer NOT NULL,
col1 character varying
);
CREATE TABLE important_table (
id integer NOT NULL,
col2 character varying NOT NULL,
unimportant_col character varying NOT NULL
);
INSERT INTO unimportant_table VALUES (123456, 'some data split into
- multiple
- lines
just for fun');
INSERT INTO important_table VALUES (987654321, 'some important data', 'another crap split into
- lines');
...
-- thousands of inserts into both tables
The dump file is really large and it is produced by another company, so I am not able to influence the export process. I need create 2 files from this dump:
All DDL statements (all statements that doesn't start with INSERT INTO)
All INSERT INTO important_table statements (I want restore only some tables from dump)
If all statements would be on single line without new line character in the data, it will be very easy to create 2 SQL script by grep, for example:
grep -v '^INSERT INTO .*;$' my_dump.sql > ddl.sql
grep -o '^INSERT INTO important_table .*;$' my_dump.sql > important_table.sql
# Create empty structures
psql < ddl.sql
# Import only one table for now
psql < important_table.sql
Firstly I was thinking about using grep but I did not find, how to process multiple lines at once, then I tried sed but it is returning only single line inserts. I also used https://regex101.com/ to find out right regular expression but I don't know how to combine it with grep or sed:
^(?!(INSERT INTO)).*$ -- for ddl
^INSERT INTO important_table(\s|[[:alnum:]])*;$ -- for inserts
I found similar question pcregrep multiline SQL match but there is no answer. Also, I don't mind if the solution will work with grep, sed or whatever you suggest, but it should work on Ubuntu 18.04.4 TLS.
Here is a bash based solution that uses perl one-liners to prepare your SQL dump data for the subsequent grep statements.
In my approach, the goal is to get one SQL statement on one line through a script that I called prepare.sh. It got a little more complicated because I wanted to accomodate for semicolons and quotes within your insert data strings (these, along with the line breaks, are represented by their hex codes in the intermediate output):
EDIT: In response to #32cupo's comment, below is a modified set of scripts that avoids xargs with large data sets (although I don't have huge dump files to test it with):
#!/bin/bash
perl -pne 's/;(?=\s*$)/__ENDOFSTATEMENT__/g' \
| perl -pne 's/\\/\\\\x5c/g' \
| perl -pne 's/\n/\\\\x0a/g' \
| perl -pne 's/"/\\\\x22/g' \
| perl -pne 's/'\''/\\\\x27/g' \
| perl -pne 's/__ENDOFSTATEMENT__/;\n/g' \
Then, a separate script (called ddl.sh) includes your grep statement for the DDL (and, with the help of the loop, only feeds smaller chunks (lines) into xargs):
#!/bin/bash
while read -r line; do
<<<"$line" xargs -I{} echo -e "{}"
done < <(grep -viE '^(\\\\x0a)*insert into')
Another separate script (called important_table.sh) includes your grep statement for the inserts into important-table:
#!/bin/bash
while read -r line; do
<<<"$line" xargs -I{} echo -e "{}"
done < <(grep -iE '^(\\\\x0a)*insert into important_table')
Here is the set of scripts in action (please also note that I spiced up your insert data with some semicolons and quotes):
~/$ cat dump.sql
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE unimportant_table (
id integer NOT NULL,
col1 character varying
);
CREATE TABLE important_table (
id integer NOT NULL,
col2 character varying NOT NULL,
unimportant_col character varying NOT NULL
);
INSERT INTO unimportant_table VALUES (123456, 'some data split into
- multiple
- lines
;just for fun');
INSERT INTO important_table VALUES (987654321, 'some important ";data"', 'another crap split into
- lines;');
...
-- thousands of inserts into both tables
~/$ cat dump.sql | ./prepare.sh | ./ddl.sh >ddl.sql
~/$ cat ddl.sql
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE unimportant_table (
id integer NOT NULL,
col1 character varying
);
CREATE TABLE important_table (
id integer NOT NULL,
col2 character varying NOT NULL,
unimportant_col character varying NOT NULL
);
...
-- thousands of inserts into both tables
~/$ cat dump.sql | ./prepare.sh | ./important_table.sh > important_table.sql
~/$ cat important_table.sql
INSERT INTO important_table VALUES (987654321, 'some important ";data"', 'another crap split into
- lines;');
I am having trouble writing up a batch script. I've narrowed it down to what seems to break and I am providing an example. The following code, when pasted into a console returns 10 rows:
set TESTRUNID=111222
set QUERY="select distinct col1 from Table where col2='%TESTRUNID%' and col3 LIKE '%es'"
start /B /wait sqlcmd -S dahost -U usr -P pwd -Q %QUERY% -o resfile.txt
When I put it in a batch script, it returns 0 rows!
#echo off
setlocal EnableDelayedExpansion
REM remark
REM remark
set TESTRUNID=111222
set QUERY="select distinct col1 from Table where col2='%TESTRUNID%' and col3 LIKE '%es'"
start /B /wait sqlcmd -S dahost -U usr -P pwd -Q %QUERY% -o resfile.txt
I think you are mixing up the use of the percent sign to mean (1) batch variable expansion, and (2) SQL wildcard. Inside a batch file, use a double %% sign for an SQL wildcard:
set QUERY="select distinct col1 from Table where col2='%TESTRUNID%' and col3 LIKE '%%es'"
The double % sign gets translated to a single % sign before it's passed to SQLCMD.
Is there any way to pass a query to SQL*Plus through a variable?
I'm aware SQL*Plus has the capability to execute a file like:
sqlplus user/pass#db #filename
Within a kornshell script I'm trying to do:
query="select * from dual;"
sqlplus user/pass#db $query
There might have solution to do that BUT I can achieve the same goal using the following method.
[oracle#myserver Desktop]$ $ORACLE_HOME/bin/sqlplus -s jay/passsword#db <<!
select * from dual;
exit
!
D
-
X
Update, you can store the returned result in a variable as shown below.
query="select * from dual;"
var=$($ORACLE_HOME/bin/sqlplus -s jay/pass#db <<!
set pages 0
set head off
set feed off
$query
!
);
DB: Oracle
I'm accepting filesystem name from user input and would like to use that value in sql query to generate dynamic script to restore datafile.
I can generate dynamic script if I know what filesystems name will be but trying to figure out on how to take user input value and put that in sql query?
Runtime: ./gen_query.ksh -oldDB db1 -newDB db2 -mt /u01, /u02, /u03, /u04
values for -oldDB -newDB -mt are entered by users.
-mt values will be separated by comma ","
-mt values can differ from user and can be only two mounts or can be three or four and can start with anything (e.g. /u01 or /u06 etc) so I can't really hardcode them.
I think what I need is to chop -mt values and store them into another variable and use those variable to generate query.
Is such thing possible?
Dynamic query:
#!/bin/ksh
LOG_FILE = test_gen_query.log
exec >> $LOG_FILE 2>&1
gen_query=
SELECT 'set newname for datafile '''
|| file_name
|| ''' to ''/u01 or /u02 or /u03 or /u04'
|| replace to_char(mod(rownum, 4)
|| replace(substr(file_name, 5, length (file_name)),'$2','$4')
|| ''';'
FROM (
SELECT file_name, bytes from dba_data_files
ORDER BY bytes desc
)
/
Sample output required:
set newname for datafile '/u40/oradata/db1/test1.dbf' to '/u01/oradata/db2/test1.dbf';
set newname for datafile '/u40/oradata/db1/test2.dbf' to '/u02/oradata/db2/test2.dbf';
set newname for datafile '/u41/oradata/db1/test3.dbf' to '/u03/oradata/db2/test3.dbf';
set newname for datafile '/u40/oradata/db1/test4.dbf' to '/u04/oradata/db2/test4.dbf';
set newname for datafile '/u40/oradata/db1/test5.dbf' to '/u01/oradata/db2/test5.dbf';
set newname for datafile '/u40/oradata/db1/test6.dbf' to '/u02/oradata/db2/test6.dbf';
...
...
When you want to use arguments with a minus-sign, see
learn about getopts. When you are lazy, you can use fixed positions:
./gen_query.ksh db1 db2 /u01 /u02 /u03 /u04
and get the vars with:
if [ $# -lt 3 ]; then
echo "Usage: $0 olddb newdb mount(s)"
exit 1
fi
# assign the first 2 vars
olddb=$1
newdb=$2
# Move "pointer" in the arglist
shift 2
# Split remaining args on a space
for mnt in $*; do
echo "my sql for dbold ${olddb} to dbnew ${newdb} for mount ${mnt}"
done