i have a table:
id | detail
1 | ddsffdfdf ;df, deef,"dgfgf",/dfdf/
when I did: insert into details values(1,'ddsffdfdf ;df, deef'); => got inserted properly
When I copied that inserted value from database to a file,the file had: 1 ddsffdfdf ;df, deef
Then I loaded the whole csv file to pgsql database,with values in the format: 1 ddsffdfdf ;df, deef
ERROR: invalid input syntax for integer: "1 ddsffdfdf ;df, deef is obtained. How to solve the problem?
CSVs need a delimiter that Postgres will recognize to break the text into respective fields. Your delimiter is a space, which is insufficient. Your CSV file should look more like:
1,"ddsffdfdf df, deef"
And your SQL should look like:
COPY details FROM 'filename' WITH CSV;
The WITH CSV is important because it tells Postgres to use a comma as the delimiter and parses your values based on that. Because your second field contains a comma, you want to enclose its value in quotes so that its comma is not mistaken for a delimiter.
To look at a good example of a properly formatted CSV file, you can output your current table:
COPY details TO '/your/filename.csv' WITH CSV;
Related
Trying to import a csv from S3 into RDS postgres:
The data in some columns contains ",". for example the address column.
The data is enclosed in "".
Here is our select:
SELECT aws_s3.table_import_from_s3(
'Change_National_IDs',
'',
'DELIMITER '',''',
aws_commons.create_s3_uri('data-migration-s3-bucket', 'Change_IDs_WqlResults_20xxxxx4_xxxxxx.csv', 'us-west-2')
);
Tried many different combinations. How can we handle the "," in the data?
Current error:
ERROR: extra data after last expected column
you should change your column delimiter to another different than comma. Something like | and then use code like this.
SELECT aws_s3.table_import_from_s3(
'Change_National_IDs',
'',
'DELIMITER ''|''',
aws_commons.create_s3_uri('data-migration-s3-bucket',
'Change_IDs_WqlResults_20xxxxx4_xxxxxx.csv', 'us-west-2')
);
More Information
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html#USER_PostgreSQL.S3Import.FileFormats.CustomDelimiter
I have a table that has some rows with normal JSON and some with escaped values in the JSON field (backslashes)
id
obj
1
{"is_from_shopping_bag":true,"products":[{"price":{"amount":"18.00","currency":"USD","offset":100,"amount_with_offset":"1800"},"product_id":"1234","quantity":1}],"source":"cart"}
2
{"is_from_shopping_bag":"","products":"[{\ "product_id\ ":\ "2345\ ",\ "price\ ":{\ "currency\ ":\ "USD\ ",\ "amount\ ":\ "140.00\ ",\ "offset\ ":100},\ "quantity\ ":1}]"}
(Note: I needed to include a space after the backslashes in the above table so that they would show up in the github generated markdown table -- my actual table does not include those spaces between the backslash and the quote character)
I am doing a sql query in Hive to get the 'currency' field.
Currently I can run
SELECT
id,
JSON_EXTRACT(obj, '$.products[0].price.currency')
FROM my_table
Which will give me the correct output for the first row, but gives me a NULL in the second row
id
obj
1
"USD"
2
NULL
What is the best way to get currency field from the second row? Is there a way to clean up the field and remove the backslashes before trying to JSON_EXTRACT the relevant data?
I could use REPLACE to swap the '\ ' for '', but is that the most efficient method?
Replace \" with " using regexp_replace like this:
regexp_replace(obj,'\\\\"','"')
In my CSV file there is:
prenom; nom; age
prenom1; nom1; age1
prenom2; nom2; age2
...
When I import my CSV file using this command:
CREATE TABLE TEST AS SELECT * FROM CSVREAD('C:\Users\anonymous\Desktop\test.csv');
The main problem is that my database has 1 column with my CSV file..
I would like 3 columsn with prenom, nom and age with the data in each column.
Thanks for your help!
As #jdv stated, you must specify the field separator if it is not the default ,. The null specifies that the column names will be parsed from the first row.
CREATE TABLE TEST AS SELECT * FROM CSVREAD('C:\Users\anonymous\Desktop\test.csv',null,'fieldSeparator=;');
Keep in mind you may have to specify charset=Cp1252 as well, if the CSV file was generated with Excel. If you see something like prénom you have the wrong encoding.
I am using the following command to output the result of an SQL query to a text file:
$sqlite3 my_db.sqlite "SELECT text FROM message;" > out.txt
This gives me output like this:
text for entry 1
text for entry 2
Unfortunately, this breaks down when the text contains a newline:
text for entry 1
text for
entry 2
How can I specify an output delimiter (which I know doesn't exist in the text) for SQLite to use when outputting the data so I can more easily parse the result? E.g.:
text for entry 1
=%=
text for
entry 2
=%=
text for entry 3
Try -separator option for this.
$sqlite3 -separator '=%=' my_db.sqlite "SELECT text FROM message;" > out.txt
Update 1
I quess this is because of '-list' default option. In order to turn this option off you need to change current mode.
This is a list of modes
.mode MODE ?TABLE? Set output mode where MODE is one of:
csv Comma-separated values
column Left-aligned columns. (See .width)
html HTML <table> code
insert SQL insert statements for TABLE
line One value per line
list Values delimited by .separator string
tabs Tab-separated values
tcl TCL list elements
-list Query results will be displayed with the separator (|, by
default) character between each field value. The default.
-separator separator
Set output field separator. Default is '|'.
Found this info here
I had the same question and there is a simpler solution. I found this at https://sqlite.org/cli.html :
.separator COL ?ROW? Change the column and row separators
For example:
sqlite> .separator | ,
sqlite> select * from example_table;
1|3,1|4,1|15,1|21,1|33,2|13,2|16,2|32,
Or with no column separator:
sqlite> .separator '' ,
sqlite> select * from example_table;
13,14,115,121,133,213,216,232,
Or, to answer the specific question posed above, this is all that is needed:
sqlite> .separator '' \r\n=%=\r\n
sqlite> select * from message;
text for entry 1
=%=
text for
entry 2
=%=
text for entry 3
=%=
In order to seperate columns, you would have to work with group_concat and a seperator.
Query evolution:
SELECT text FROM messages;
SELECT GROUP_CONCAT(text, "=%=") FROM messages;
SELECT GROUP_CONCAT(text, "\r\n=%=\r\n") FROM messages;
// to get rid of the concat comma, use replace OR change seperator
SELECT REPLACE(GROUP_CONCAT(text, "\r\n=%="), ',', '\r\n') FROM messages;
SQLFiddle
Alternative: Sqlite to CSV export (with custom seperator), then work with that.
I am running a job in MS SQL Server that outputs a text file with white space in between columns. What I'd like to do is specify a specific character sequence between each column as a delimiter.
For example, I'd like the output to look like:
Apple%%%red%%%fruit
banana%%%yellow%%%fruit
onion%%%White%%%veggie
In this example, %%% is the delimiter.
How can I do this?
Assuming that you are using the output file of the job step, and the output file is currently structured something like this:
---- --------
row1 somedata
row2 somedata
You could just concatenate the columns using '+' and fit the percent signs in as appropriate. So the job step definition would contain:
select column1 + '%%%' + column2 from table1;
And the output would look like:
---------------
row1%%%somedata
row2%%%somedata
This assumes that you are OK with concatenating each row of results into a single column. You will need to cast/convert non-character column values for this to work.
My guess is that you are looking for the T-SQL equivalent of Oracle's P/SQL command "set colsep" command. This command lets you alter the delimter of the output. TO make it semicolon, for example, you would call:
set colsep ";"
But in SQL Server... I see the way to do it.
Use the "bcp" utility and you can specify the delimiter and write to your file. He are instructions:
http://msdn.microsoft.com/en-us/library/ms162802.aspx
Look at the -t option to change the separator.