This is an image of what I'm asking for
I am using the following -echo- in a script and after I execute, the output format is as shown below:
`echo -e "UPDATE table1 SET table1_f1='$Fname' ,table1_f2='$Lname' where table1_f3='$id';\ncommit;" >> $OutputFile`
output: UPDATE table1 SET table1_f1='Fname' ,table1_f2='Lname' where table1_f3='id ';
the '; is appearing on a new line, why is that happening?
The variable $id in your shell script actually contains that newline (\n or \r\n) at the end; so there isn't really anything wrong in the part of the script you've shown here.
This effect is pretty common if the variable is created based on external commands (update:) or by reading external files as you are here.
For simple values, one way to strip the newline off the end of the value, prior to using it in your echo is:
id=$( echo "${id}" | tr -s '\r' '' | tr -s '\n' '' );
or for scripts that already rely on a particular bash IFS value:
OLDIFS="${IFS}";
IFS=$'\n\t ';
id=$( echo "${id}" | tr -s '\r' '' | tr -s '\n' '' );
IFS="${OLDIFS}";
Related
I have PostgreSQL database dump by pg_dump version 9.5.2, which contains DDLs and also INSERT INTO statements for each table in given database. Dump looks like this:
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE unimportant_table (
id integer NOT NULL,
col1 character varying
);
CREATE TABLE important_table (
id integer NOT NULL,
col2 character varying NOT NULL,
unimportant_col character varying NOT NULL
);
INSERT INTO unimportant_table VALUES (123456, 'some data split into
- multiple
- lines
just for fun');
INSERT INTO important_table VALUES (987654321, 'some important data', 'another crap split into
- lines');
...
-- thousands of inserts into both tables
The dump file is really large and it is produced by another company, so I am not able to influence the export process. I need create 2 files from this dump:
All DDL statements (all statements that doesn't start with INSERT INTO)
All INSERT INTO important_table statements (I want restore only some tables from dump)
If all statements would be on single line without new line character in the data, it will be very easy to create 2 SQL script by grep, for example:
grep -v '^INSERT INTO .*;$' my_dump.sql > ddl.sql
grep -o '^INSERT INTO important_table .*;$' my_dump.sql > important_table.sql
# Create empty structures
psql < ddl.sql
# Import only one table for now
psql < important_table.sql
Firstly I was thinking about using grep but I did not find, how to process multiple lines at once, then I tried sed but it is returning only single line inserts. I also used https://regex101.com/ to find out right regular expression but I don't know how to combine it with grep or sed:
^(?!(INSERT INTO)).*$ -- for ddl
^INSERT INTO important_table(\s|[[:alnum:]])*;$ -- for inserts
I found similar question pcregrep multiline SQL match but there is no answer. Also, I don't mind if the solution will work with grep, sed or whatever you suggest, but it should work on Ubuntu 18.04.4 TLS.
Here is a bash based solution that uses perl one-liners to prepare your SQL dump data for the subsequent grep statements.
In my approach, the goal is to get one SQL statement on one line through a script that I called prepare.sh. It got a little more complicated because I wanted to accomodate for semicolons and quotes within your insert data strings (these, along with the line breaks, are represented by their hex codes in the intermediate output):
EDIT: In response to #32cupo's comment, below is a modified set of scripts that avoids xargs with large data sets (although I don't have huge dump files to test it with):
#!/bin/bash
perl -pne 's/;(?=\s*$)/__ENDOFSTATEMENT__/g' \
| perl -pne 's/\\/\\\\x5c/g' \
| perl -pne 's/\n/\\\\x0a/g' \
| perl -pne 's/"/\\\\x22/g' \
| perl -pne 's/'\''/\\\\x27/g' \
| perl -pne 's/__ENDOFSTATEMENT__/;\n/g' \
Then, a separate script (called ddl.sh) includes your grep statement for the DDL (and, with the help of the loop, only feeds smaller chunks (lines) into xargs):
#!/bin/bash
while read -r line; do
<<<"$line" xargs -I{} echo -e "{}"
done < <(grep -viE '^(\\\\x0a)*insert into')
Another separate script (called important_table.sh) includes your grep statement for the inserts into important-table:
#!/bin/bash
while read -r line; do
<<<"$line" xargs -I{} echo -e "{}"
done < <(grep -iE '^(\\\\x0a)*insert into important_table')
Here is the set of scripts in action (please also note that I spiced up your insert data with some semicolons and quotes):
~/$ cat dump.sql
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE unimportant_table (
id integer NOT NULL,
col1 character varying
);
CREATE TABLE important_table (
id integer NOT NULL,
col2 character varying NOT NULL,
unimportant_col character varying NOT NULL
);
INSERT INTO unimportant_table VALUES (123456, 'some data split into
- multiple
- lines
;just for fun');
INSERT INTO important_table VALUES (987654321, 'some important ";data"', 'another crap split into
- lines;');
...
-- thousands of inserts into both tables
~/$ cat dump.sql | ./prepare.sh | ./ddl.sh >ddl.sql
~/$ cat ddl.sql
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE unimportant_table (
id integer NOT NULL,
col1 character varying
);
CREATE TABLE important_table (
id integer NOT NULL,
col2 character varying NOT NULL,
unimportant_col character varying NOT NULL
);
...
-- thousands of inserts into both tables
~/$ cat dump.sql | ./prepare.sh | ./important_table.sh > important_table.sql
~/$ cat important_table.sql
INSERT INTO important_table VALUES (987654321, 'some important ";data"', 'another crap split into
- lines;');
I have a function like so -
CREATE
OR REPLACE FUNCTION ind (bucket text) RETURNS table (
middle character varying (100),
last character varying (100)
) AS $body$ BEGIN return query
select
fname as first,
lname as last
from all_records
; END;
$body$ LANGUAGE PLPGSQL;
How do I output the results of select ind ('Mob') into a tsv file?
I want the output to look like this -
first last
MARY KATHERINE
You can use the COPY command
example:
COPY (select * from ind('Mob')) TO '/tmp/ind.tsv' CSV HEADER DELIMITER E'\t';
the file '/tmp/ind.tsv' will contain you data
Postgres doesn't allow copy with header for tsv for some reason.
If you're using a linux based system you can do it with a script like this:
#create file with tab delimited column list (use \t between each column name)
echo -e "user_id\temail" > user_output.tsv
#now you can append the results of your query to that file by copying to STDOUT
psql -h your_host_name -d your_database_name -c "\copy (SELECT user_id, email FROM my_user_table) to STDOUT;" >> user_output.tsv
Alternatively, if your script is long and you don't want to pass it in with -c command you can use the same approach from a .sql file, use "--quiet" to avoid notices being passed into your file
psql --quiet -h your_host_name -d your_database_name -f your_sql_file.sql >> user_output.tsv
My code is:
#!/bin/sh
cat tmp_ts.log | awk ' {print $8}'
lookup=$8
sqlplus -s "sys/Orcl1234 as sysdba" << EOF
SELECT tablespace_name FROM dba_tablespaces WHERE tablespace_name='$lookup';
exit;
EOF
and my output is:
IAM_OIM
no rows selected
In this variable lookup I have passed to select statement but it's not working.
My end result should be with select statement. See below the output of select query:
See below:
My end result should be this but that variable is not working in select statement.
#!/bin/sh
lookup="$(awk '/tablespace/{print $8;exit}' tmp_ts.log)"
echo "Querying database with lookup = $lookup"
sqlplus -s "sys/Orcl1234 as sysdba" <<EOF
SELECT tablespace_name FROM dba_tablespaces WHERE tablespace_name='$lookup';
exit;
EOF
You have to use awk's output to set lookup. The shell knows nothing about the $8 which was set in awk. Also, I have ensured that awk exits after the first matching line, so that there is no risk of returning multiple values, or simply empty lines as it did in your version.
You can fill lookup with a command like awk, sed or cut.
lookup=$(cut -d" " -f8 tmp_ts.log)
You should add some checks, like #Dario did (with an exit after the first match and only converting lines with tablespace, but what to do when no lines match?).
When you don't add the checks you can skip setting the $lookup:
sqlplus -s "sys/Orcl1234 as sysdba" << EOF
SELECT tablespace_name FROM dba_tablespaces
WHERE tablespace_name='$(sed 's/.*tablespace- //' tmp_ts.log)';
exit;
EOF
I`m working with Vertica. I try to export data from SELECT query into csv. I tried making it with sql query:
SELECT * FROM table_name INTO OUTFILE '/tmp/fileName.csv' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
I got an error:
[Vertica][VJDBC](4856) ERROR: Syntax error at or near "INTO"
Is there a way to export a query result to a csv file? I prefer not to use vsql, but if there no other way, I will use it. I tried the following:
vsql -c "select * from table_name;" > /tmp/export_data.txt
Here is how you do it:
vsql -U dbadmin -F ',' -A -P footer=off -o dumpfile.txt -c "select ... from ... where ...;"
Reference: Exporting Data Using vsql
Accordingly to https://my.vertica.com/docs/7.1.x/HTML/Content/Authoring/ConnectingToHPVertica/vsql/ExportingDataUsingVsql.htm
=> SELECT * FROM my_table;
a | b | c
---+-------+---
a | one | 1
b | two | 2
c | three | 3
d | four | 4
e | five | 5
(5 rows)
=> \a
Output format is unaligned.
=> \t
Showing only tuples.
=> \pset fieldsep ','
Field separator is ",".
=> \o dumpfile.txt
=> select * from my_table;
=> \o
=> \! cat dumpfile.txt
a,one,1
b,two,2
c,three,3
d,four,4
e,five,5
By following way you can write to CSV file as comma separated and no footer.
vsql -h $HOST -U $USER -d $DATABASE -w $PASSWORD -f $SQL_PATH/SQL_FILE -A -o $FILE -F ',' -P footer=off -q
I have a text file with three columns of text (strings) per line. I want to create an SQL insert command by substituting each of the three strings into a skeleton SQL command. I have put place markers in the skeleton script and used SED s/placemarker1/first string/ but with no success. Is there an easier way to accomplish this task. I used pipes to repeat the process for 'second string' etc. I actually used awk to get the fields but could not convert to the actual values.
enter code here
for i in [ *x100* ]; do
if [ -f "$i" ]; then {
grep -e "You received a payment" -e "Transaction ID:" -e "Receipt No: " $i >> ../temp
cat ../temp | awk 'NR == 1 {printf("%s\t",$9)} NR == 2 {printf("%s\t",$9)} NR == 3 {printf("%s\n",$3)}' | awk '{print $2,$1,$3}' | sed 's/(/ /' | sed 's/)./ /' >> ../temp1
cat temp1 | awk 'email="$1"; transaction="$2"; ccreceipt="$3";'
cat /home/linux014/opt/skeleton.sql | sed 's/EMAIL/"$email"/' | sed 's/TRANSACTION/"$transaction"/' | sed 's/CCRECEIPT/"$ccreceipt"/' > /home/linux014/opt/new-member.sql
rm -f ../temp
} fi
done
I cannot figure out how to get the values instead of the names of the variables inserted into my string.
Sample input (one line only):
catdog#gmail.com 2w4e5r6t7y8u9i8u7 1111-2222-3333-4444
Sample actual output:
INSERT INTO users (email,paypal_tran,CCReceipt) VALUES ('"$email"','"$transaction"','"$ccreceipt"');
Preferred output:
INSERT INTO users (email,paypal_tran,CCReceipt) VALUES ('catdog#gmail.com','2w4e5r6t7y8u9i8u7','1111-2222-3333-4444');
awk '{print "INSERT INTO users (email,paypal_tran,CCReceipt) VALUES"; print "(\x27"$1"\x27,\x27"$2"\x27,\x27"$3"\x27);"}' input.txt
Converts your sample input to preferred output. It should work for multi line input.
EDIT
The variables you are using in this line:
cat temp1 | awk 'email="$1"; transaction="$2"; ccreceipt="$3";'
are only visible to awk and in this command. They are not shell variables.
Also in your sed commands remove those single quotes then you can get the values:
sed "s/EMAIL/$email/"
You can try this bash,
while read email transaction ccreceipt; do echo "INSERT INTO users (email,paypal_tran,CCReceipt) VALUES ('$email','$transaction','$ccreceipt');"; done<inputfile
inputfile:
catdog#gmail.com 2w4e5r6t7y8u9i8u7 1111-2222-3333-4444
dog#gmail.com 2dsdsda53563u9i8u7 3333-4444-5555-6666
Test:
sat:~$ while read email transaction ccreceipt; do echo "INSERT INTO users (email,paypal_tran,CCReceipt) VALUES ('$email','$transaction','$ccreceipt')"; done<inputfile
INSERT INTO users (email,paypal_tran,CCReceipt) VALUES ('catdog#gmail.com','2w4e5r6t7y8u9i8u7','1111-2222-3333-4444')
INSERT INTO users (email,paypal_tran,CCReceipt) VALUES ('dog#gmail.com','2dsdsda53563u9i8u7','3333-4444-5555-6666')
You can write a small procedure for this
CREATE PROCEDURE [dbo].[appInsert]--
#string VARCHAR(500)
AS
BEGIN
DECLARE #I INT
DECLARE #SubString VARCHAR(500)
SET #String = 'catdog#gmail.com 2w4e5r6t7y8u9i8u7 1111-2222-3333-4444'
SET #I = 1
SET #String = REPLACE(#String, ' ', '`~`')
WHILE #I > 0
BEGIN
SET #SubString = SUBSTRING (REVERSE(#String), 1, ( CHARINDEX( '`~`', REVERSE(#String)) - 1))
SET #String = SUBSTRING(#String, 1, LEN(#String) - CHARINDEX( '`~`', REVERSE(#String)) - 2 )
print REVERSE(#SubString) + ' === ' + #String
SET #i = CHARINDEX( '`~`', #String)
END
END