i am trying to export table from redshift to s3 using unload command
command :
unload ('SELECT * FROM table where id = 4') TO 's3://path/temp/table1'
credentials 'aws_access_key_id="ahsvdgsfadhsagdffjh;aws_secret_access_key=ahgsdasdhgsahdgsahdgsahdgsahgsa' delimiter '|' NULL
AS
'\\N' escape;
one of the field in my table contains new line so the csv generated breaks into two lines
is there any way to replace new line to \n or to add a end of line character
You should be removing escape, and just try ADDQUOTE Option, it will produce correct CSV. As "" will instruct your CSV reader to treat \n as verbiage rather than newline.
UNLOAD ('SELECT * FROM table where id = 4')
TO 's3://path/temp/table1'
credentials aws_access_key_id="ahsvdgsfadhsagdffjh;
aws_secret_access_key=ahgsdasdhgsahdgsahdgsahdgsahgsa'
delimiter '|'
NULL AS '\\N' ADDQUOTE;
Related
I have an oracle Table that holds all the Project Details, This table (HR.AA_PROJECT_D) has a column project_name which has both white space and the new line character "/n" present in it, Is there a way to remove them both and update the Column? Since I have data security issues I will not be able to post the output here for reference.
Replace: \n = CHR(10) and blank ' ' with '':
UPDATE hr.aa_project_d SET project_name=REPLACE(REPLACE(project_name,CHR(10),''),' ','');
I have that kind of file
info1;info2;info3";info4;info5
And after parsing I have that error
Error: [42636] ETL-2106: Error while parsing row=0 (starting from 0) [CSV Parser found at byte 5 (starting with 0 at the beginning of the row) of 5 a field delimiter after an quoted field (with an additional whitespace) in file '~/path'. Please check for correct enclosed fields, valid field separators and e.g. unescaped field delimiters that are contained in the data (these have to be escaped)]
I'm sure that the reason is here info3"; but how can I solve this problem I have no idea
Also I can't rid of quotes, because it should be in report
The main part of python code is
# Transform data to valid CSV format: remove BOM, remove '=' sign, remove repeating quotes in Size column
decoded_csv = r.content.decode('utf-8').replace(u'\ufeff', '').replace('=', '')
print(decoded_csv)
cr = csv.reader(decoded_csv.splitlines(), delimiter=';')
lst = list(cr)[1:]
f = csv.writer(open(base_folder + 'txt/' + shop, "w+"), delimiter=';')
for row in lst:
f.writerow(row[:-2])
After this code I get that kind of file
info1;info2;"info3""";info4;info5
And it is not what I need
But when I change code a little by adding "quoting=csv.QUOTE_NONE, quotechar='')"
# Transform data to valid CSV format: remove BOM, remove '=' sign, remove repeating quotes in Size column
decoded_csv = r.content.decode('utf-8').replace(u'\ufeff', '').replace('=', '')
print(decoded_csv)
cr = csv.reader(decoded_csv.splitlines(), delimiter=';')
lst = list(cr)[1:]
f = csv.writer(open(base_folder + 'txt/' + shop, "w+"), delimiter=';' quoting=csv.QUOTE_NONE, quotechar='')
for row in lst:
f.writerow(row[:-2])
I get what I need
info1;info2;info3";info4;info5
It is a 2nd step (exasol) and code returned the error
MERGE INTO hst AS dst
USING (
SELECT DISTINCT
ar,
ar_na,
FROM (
IMPORT INTO
(
ar VARCHAR(100) UTF8 COMMENT IS 'ar',
ar_na VARCHAR(100) UTF8 COMMENT IS 'ar na',
)
FROM CSV /*SS:R*/
AT '&1'
USER '&2'
IDENTIFIED BY '&3'
FILE '~/path'
SKIP = 0
ROW SEPARATOR = 'CRLF'
COLUMN SEPARATOR = ';'
TRIM
)
GROUP BY
ar,
ar_na,
) src ON src.ar = dst.ar
WHEN MATCHED THEN UPDATE SET
dst.ar_na = src.ar_na,
WHEN NOT MATCHED THEN
INSERT (
ar
ar_na,
)
VALUES (
src.ar,
src.ar_na,
);
If file looks like info1;info2;info3;info4;info5 everything works fine, all scripts work
By default, Exaosl treats double quotes (") as column delimiter. This enables you to specify values that contain the column separator (in your case that's the semicolon). See the entry "Special characters" in the documentation.
You have two options here:
Disable the column delimiter by passing COLUMN DELIMITER = '' to the import statement.
Duplicate all double quotes in the csv file. Exasol ignores the column delimiter if it occurs twice consecutively.
i have a table:
id | detail
1 | ddsffdfdf ;df, deef,"dgfgf",/dfdf/
when I did: insert into details values(1,'ddsffdfdf ;df, deef'); => got inserted properly
When I copied that inserted value from database to a file,the file had: 1 ddsffdfdf ;df, deef
Then I loaded the whole csv file to pgsql database,with values in the format: 1 ddsffdfdf ;df, deef
ERROR: invalid input syntax for integer: "1 ddsffdfdf ;df, deef is obtained. How to solve the problem?
CSVs need a delimiter that Postgres will recognize to break the text into respective fields. Your delimiter is a space, which is insufficient. Your CSV file should look more like:
1,"ddsffdfdf df, deef"
And your SQL should look like:
COPY details FROM 'filename' WITH CSV;
The WITH CSV is important because it tells Postgres to use a comma as the delimiter and parses your values based on that. Because your second field contains a comma, you want to enclose its value in quotes so that its comma is not mistaken for a delimiter.
To look at a good example of a properly formatted CSV file, you can output your current table:
COPY details TO '/your/filename.csv' WITH CSV;
I am using command line Hive. For example hive -e "SELECT * FROM my_db.my_table;"
It is currently returning what looks like tab separated values. Is it possible to specify which delimiter it should use? For example, can I make it return pipe separated values?
what i am done in my case, i fired a query like below.
INSERT OVERWRITE LOCAL DIRECTORY '/home/Desktop/test3'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
select * from stud_02
other solution would be
hive -e 'select * from stud_01 limit 10' | sed 's/[[:space:]]\+/,/g' >> /home/Desktop/test.csv
I am using the following command to output the result of an SQL query to a text file:
$sqlite3 my_db.sqlite "SELECT text FROM message;" > out.txt
This gives me output like this:
text for entry 1
text for entry 2
Unfortunately, this breaks down when the text contains a newline:
text for entry 1
text for
entry 2
How can I specify an output delimiter (which I know doesn't exist in the text) for SQLite to use when outputting the data so I can more easily parse the result? E.g.:
text for entry 1
=%=
text for
entry 2
=%=
text for entry 3
Try -separator option for this.
$sqlite3 -separator '=%=' my_db.sqlite "SELECT text FROM message;" > out.txt
Update 1
I quess this is because of '-list' default option. In order to turn this option off you need to change current mode.
This is a list of modes
.mode MODE ?TABLE? Set output mode where MODE is one of:
csv Comma-separated values
column Left-aligned columns. (See .width)
html HTML <table> code
insert SQL insert statements for TABLE
line One value per line
list Values delimited by .separator string
tabs Tab-separated values
tcl TCL list elements
-list Query results will be displayed with the separator (|, by
default) character between each field value. The default.
-separator separator
Set output field separator. Default is '|'.
Found this info here
I had the same question and there is a simpler solution. I found this at https://sqlite.org/cli.html :
.separator COL ?ROW? Change the column and row separators
For example:
sqlite> .separator | ,
sqlite> select * from example_table;
1|3,1|4,1|15,1|21,1|33,2|13,2|16,2|32,
Or with no column separator:
sqlite> .separator '' ,
sqlite> select * from example_table;
13,14,115,121,133,213,216,232,
Or, to answer the specific question posed above, this is all that is needed:
sqlite> .separator '' \r\n=%=\r\n
sqlite> select * from message;
text for entry 1
=%=
text for
entry 2
=%=
text for entry 3
=%=
In order to seperate columns, you would have to work with group_concat and a seperator.
Query evolution:
SELECT text FROM messages;
SELECT GROUP_CONCAT(text, "=%=") FROM messages;
SELECT GROUP_CONCAT(text, "\r\n=%=\r\n") FROM messages;
// to get rid of the concat comma, use replace OR change seperator
SELECT REPLACE(GROUP_CONCAT(text, "\r\n=%="), ',', '\r\n') FROM messages;
SQLFiddle
Alternative: Sqlite to CSV export (with custom seperator), then work with that.