Error during \COPY command in PostgreSQL to import CSV data - sql

Subject: Error during \COPY command in PostgreSQL
Hello,
I am trying to copy data from multiple CSV files into corresponding tables in PostgreSQL database using the \COPY command. However, I am getting a syntax error at the end of the line and near "FROM". I need help. Thanks in advance. Here's my code:
tablequery.sql
-- copy data from budgets.csv file into the budgets table
\COPY budgets (budget_id, dt_create, dt_update, dt_value, category, amount)
FROM 'setup/psql/budgets.csv'
WITH (FORMAT CSV, HEADER true, DELIMITER ';');
-- copy data from expenses.csv file into the expenses table
\COPY expenses (ulid_id, dt_create, dt_update, dt_value, amount, description, budget_id, category)
FROM 'setup/psql/expenses.csv'
WITH (FORMAT CSV, HEADER true, DELIMITER ';');
-- copy data from revenues.csv file into the revenues table
\COPY revenues (ulid_id, dt_create, dt_update, dt_value, amount, description)
FROM 'setup/psql/revenues.csv'
WITH (FORMAT CSV, HEADER true, DELIMITER ';');
Terminal output :
psql:setup/psql/tablequery.sql:61: error: \copy: parse error at end of line
psql:setup/psql/tablequery.sql:63: ERROR: syntax error at or near "FROM"
LINE 1: FROM 'setup/psql/budgets.csv'
^
psql:setup/psql/tablequery.sql:66: error: \copy: parse error at end of line
psql:setup/psql/tablequery.sql:68: ERROR: syntax error at or near "FROM"
LINE 1: FROM 'setup/psql/expenses.csv'
^
psql:setup/psql/tablequery.sql:71: error: \copy: parse error at end of line
psql:setup/psql/tablequery.sql:73: ERROR: syntax error at or near "FROM"
LINE 1: FROM 'setup/psql/revenues.csv'
```
└── setup
└── psql
├── budgets.csv
├── dbquery.sql
├── expenses.csv
├── revenues.csv
└── tablequery.sql

Related

trying to import csv file to table in sql

I have 4 csv files each having 500,000 rows. I am trying to import the csv data into my Exasol databse, but there is an error with the date column and I have a problem with the first unwanted column in the files.
Here is an example CSV file:
unnamed:0 , time, lat, lon, nobs_cloud_day
0, 2006-03-30, 24.125, -119.375, 22.0
1, 2006-03-30, 24.125, -119.125, 25.0
The table I created to import csv to is
CREATE TABLE cloud_coverage_CONUS (
index_cloud DECIMAL(10,0)
,"time" DATE -- PRIMARY KEY
,lat DECIMAL(10,6)
,lon DECIMAL(10,6)
,nobs_cloud_day DECIMAL (3,1)
)
The command to import is
IMPORT INTO cloud_coverage_CONUS FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv';
But I get this error:
SQL Error [42636]: java.sql.SQLException: ETL-3050: [Column=0 Row=0] [Transformation of value='Unnamed: 0' failed - invalid character value for cast; Value: 'Unnamed: 0'] (Session: 1750854753345597339) while executing '/* add path to the 4 csv files, that are in the cloud database folder*/ IMPORT INTO cloud_coverage_CONUS FROM CSV AT 'https://27.1.0.10:59205' FILE 'e12a96a6-a98f-4c0a-963a-e5dad7319fd5' ;'; 04509 java.sql.SQLException: java.net.SocketException: Connection reset by peer: socket write error
Alternatively I use this table (without the first column):
CREATE TABLE cloud_coverage_CONUS (
"time" DATE -- PRIMARY KEY
,lat DECIMAL(10,6)
,lon DECIMAL(10,6)
,nobs_cloud_day DECIMAL (3,1)
)
And use this import code:
IMPORT INTO cloud_coverage_CONUS FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv'(2 FORMAT='YYYY-MM-DD', 3 .. 5);
But I still get this error:
SQL Error [42636]: java.sql.SQLException: ETL-3052: [Column=0 Row=0] [Transformation of value='time' failed - invalid value for YYYY format token; Value: 'time' Format: 'YYYY-MM-DD'] (Session: 1750854753345597339) while executing '/* add path to the 4 csv files, that are in the cloud database folder*/ IMPORT INTO cloud_coverage_CONUS FROM CSV AT 'https://27.1.0.10:60350' FILE '22c64219-cd10-4c35-9e81-018d20146222' (2 FORMAT='YYYY-MM-DD', 3 .. 5);'; 04509 java.sql.SQLException: java.net.SocketException: Connection reset by peer: socket write error
(I actually do want to ignore the first column in the files.)
How can I solve this issue?
Solution:
IMPORT INTO cloud_coverage_CONUS FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv' (2 .. 5) ROW SEPARATOR = 'CRLF' COLUMN SEPARATOR = ',' SKIP = 1;
I did not realise that mysql is different from exasol
Looking at the first error message, a few things stand out. First we see this:
[Column=0 Row=0]
This tells us the problem is with the very first value in the file. This brings us to the next thing, where the message even tells us what value was read:
Transformation of value='Unnamed: 0' failed
So it's failing to convert Unnamed: 0. You also provided the table definition, where we see the first column in the table is a decimal type.
This makes sense. Unnamed: 0 is not a decimal. For this to work, the CSV data MUST align with the data types for the columns in the table.
But we also see this looks like a header row. Assuming everything else matches we can fix it by telling the database to skip this first row. I'm not familiar with Exasol, but according to the documentation I believe the correct code will look like this:
IMPORT INTO cloud_coverage_CONUS
FROM LOCAL CSV FILE 'D:\uni\BI\project 1\AOL_DB_ANALYSIS_TASK1\datasets\cloud\cfc_us_part0.csv'
(2 FORMAT='YYYY-MM-DD', 3 .. 5)
ROW SEPARATOR = 'CRLF'
COLUMN SEPARATOR = ','
SKIP = 1;

Problem calling a .SQL file via a batch file due to use of single and double quotes in SQL Query

I have a script that executes fine inside SQL Server 2017 query window. I want to run the script using a batch file so I can export the results to a file.
I am getting errors when I call the query using a batch file. I know it has to do with how Windows DOS handles single and double quotes. I have played around with adding single and or double quotes around the data and around the existing single and double quotes but with no luck. I have included below my batch file and the query I need to run.
BATCH CONTENTS (Line 9 is the line that executes the SQL statement by calling itemexport.sql file)
cd %UserProfile%\Documents\Tend\Reports
set bd=%UserProfile%\Documents\Tend\Reports\Export
echo *****%date%#%time% - Exporting Item List >> %bd%\item_export.log
echo off
set hh=%time:~0,2%
if "%time:~0,1%"==" " set hh=0%hh:~1,1%
set dt=%date:~4,2%-%date:~7,2%-%date:~10,4%_%hh:~0,2%_%time:~3,2%_%time:~6,2%
echo on
sqlcmd -S%1 -E -i"itemexport.sql" >> %bd%\item_export-%dt%.csv
echo *****%date%#%time% - Item Export complete >> %bd%\item_export.log
This is the SQL QUERY I am Executing. This file is named itemexport.sql
SELECT
ITEM.NAME, UPC, ITEM_DEPARTMENT.NAME AS DEPARTMENT,
CAST(ON_HAND_QTY AS decimal(18, 0)) AS "ON HAND QTY",
CAST(ITEM.PRICING AS XML).value(N'(/prices[#type="Level"]/item[#level="A"])[1]/#price', N'DECIMAL(14,2)') AS "RETAIL PRICE"
FROM
ITEM
LEFT OUTER JOIN
ITEM_DEPARTMENT ON ITEM_DEPARTMENT.RECORD_KEY = ITEM.ITEM_DEPARTMENT_RECORD_KEY
WHERE
(ITEM_DEPARTMENT.NAME IN ('BEER','CIDER','WINE'))
AND (ITEM.CAN_SELL = 1)
AND (ON_HAND_QTY > 0)
GROUP BY
ITEM.NAME, SKU, UPC, ITEM_DEPARTMENT_RECORD_KEY,
ITEM_DEPARTMENT.NAME, ITEM.ON_HAND_QTY, ITEM.PRICING
ORDER BY
ITEM_DEPARTMENT.NAME
This is the part of the select Statement that I am getting errors when using the batch file
CAST(ITEM.PRICING AS XML).value(N'(/prices[#type="Level"]/item[#level="A"])[1]/#price', N'DECIMAL(14,2)')
This is the error that I receive....
"Msg 102, Level 15, State 1, Server BYND5CG8470J1H, Line 1
Incorrect syntax near 'Level'."
If I comment out that cast statement the items are exported but that "Retail Price" column is Null for each item because the line I commented out is supposed to parse the price out of a line like <prices type="Level"><item level="A" price="1.25" /></prices>
Here is a sample of the output when running the query in SQL.
Any help formatting the cast statement would be greatly appreciated.
You can see if disabling variable substitution with the parameter -x helps with your sqlcmd (see this) but if the issue is limited to your CAST(parse) statement, can you simply make that a UDF and bypass the error altogether?
CREATE FUNCTION parsePrice(#in varchar(max))
RETURNS decimal(14,2) AS
BEGIN
RETURN CAST(#in AS XML).value(N'(/prices[#type="Level"]/item[#level="A"])[1]/#price', N'DECIMAL(14,2)') ;
END
And then your query becomes
SELECT
ITEM.NAME, UPC, ITEM_DEPARTMENT.NAME AS DEPARTMENT,
CAST(ON_HAND_QTY AS decimal(18, 0)) AS "ON HAND QTY",
dbo.parsePrice(ITEM.PRICING) AS "RETAIL PRICE"
FROM
ITEM
...

Exporting data containing line feeds as CSV from PostgreSQL

I'm trying to export data From postgresql to csv.
First i created the query and tried exporting From pgadmin with the File -> Export to CSV. The CSV is wrong, as it contains for example :
The header : Field1;Field2;Field3;Field4
Now, the rows begin well, except for the last field that it puts it on another line:
Example :
Data1;Data2;Data3;
Data4;
The problem is i get error when trying to import the data to another server.
The data is From a view i created.
I also tried
COPY view(field1,field2...) TO 'C:\test.csv' DELIMITER ',' CSV HEADER;
It exports the same file.
I just want to export the data to another server.
Edit:
When trying to import the csv i get the error :
ERROR : Extra data after the last expected column. Context Copy
actions, line 3: <<"Data1, data2 etc.">>
So the first line is the header, the second line is the first row with data minus the last field, which is on the 3rd line, alone.
In order for you to export the file in another server you have two options:
Creating a shared folder between the two servers, so that the
database also has access to this directory.
COPY (SELECT field1,field2 FROM your_table) TO '[shared directory]' DELIMITER ',' CSV HEADER;
Triggering the export from the target server using the STDOUT of
COPY. Using psql you can achieve this running the following
command:
psql yourdb -c "COPY (SELECT * FROM your_table) TO STDOUT" > output.csv
EDIT: Addressing the issue of fields containing line feeds (\n)
In case you wanna get rid of the line feeds, use the REPLACE function.
Example:
SELECT E'foo\nbar';
?column?
----------
foo +
bar
(1 Zeile)
Removing the line feed:
SELECT REPLACE(E'foo\nbaar',E'\n','');
replace
---------
foobaar
(1 Zeile)
So your COPY should look like this:
COPY (SELECT field1,REPLACE(field2,E'\n','') AS field2 FROM your_table) TO '[shared directory]' DELIMITER ',' CSV HEADER;
the described above export procedure is OK, e.g:
t=# create table so(i int, t text);
CREATE TABLE
t=# insert into so select 1,chr(10)||'aaa';
INSERT 0 1
t=# copy so to stdout csv header;
i,t
1,"
aaa"
t=# create table so1(i int, t text);
CREATE TABLE
t=# copy so1 from stdout csv header;
Enter data to be copied followed by a newline.
End with a backslash and a period on a line by itself, or an EOF signal.
>> i,t
1,"
aaa"
>> >> >> \.
COPY 1
t=# select * from so1;
i | t
---+-----
1 | +
| aaa
(1 row)

Unexpected END OF FILE while processing row

Getting the following error when copying an input file into an empty db table. The input file only has 56732 rows, however I am getting an error on row 56733:
continue
* * * * * * * * * *
copy table temptable
(
abc = c(3),
bcao = c(1),
cba = c(10),
test = c(1)nl
)
from 'tempfile'
Executing . . .
E_CO0024 COPY: Unexpected END OF FILE while processing row 56733.
E_CO002A COPY: Copy has been aborted.
Anyone have any ideas why its trying to process an extra row? I have four other files the exact same format with different data and it processes fine.
Have no idea why this is happening...
The most likely cause is that you have some spaces or similar after your final row of data. You have set a new line as a delimiter on test, so the file needs to end with a new line. Delete anything after your data which isn't a blank new line.
As an example. Using the code below:
DECLARE GLOBAL TEMPORARY TABLE test (
v int
) ON COMMIT PRESERVE ROWS WITH NORECOVERY;
COPY test (
v = c(5)nl
) FROM 'J:\test.csv';
Will result in an error on line 4 for the following data:
34565
37457
35764
45685
and error on line 5 for this data (punctuation used to show issue, but it is probably a space or tab in your own file):
34565
37457
35764
45685
.

Import date dd/mm from txt to table with SQL

I'm a total beginner and already searched all over the place, so please bear with me.
I have a txt file with this kind of data (DD/MM) and ; as delimiters:
01/10;10/06;15/11;10/07
01/10;10/06;15/11;10/07
01/11;20/06;10/11;30/07
01/11;20/06;10/11;30/07
10/11;20/06;20/01;30/07
01/10;01/06;15/11;30/06
Firstly, I set datestyle to European;
So I have DateStyle - "ISO, DMY".
After, I tried to import this data into some of the columns of the pheno table (see code below), using postgresql:
COPY pheno(planting_onset, harvesting_onset, planting_end, harvesting_end)
FROM '/home/user/Documents/worldcrops/algeria_times.txt' DELIMITERS ';';
And gave the following error:
ERROR: invalid input syntax for type date: "01/10"
CONTEXT: COPY pheno, line 1, column planting_onset: "01/10"
********** Error **********
ERROR: invalid input syntax for type date: "01/10"
SQL state: 22007
Context: COPY pheno, line 1, column planting_onset: "01/10"
Questions: How do I copy this data type DD/MM into a table which columns have date as "data type"? Should I change the columns "data type"?
Thanks in advance.
It's expecting DMY but you're only giving it days and months. This is kind of hacky but i think it should work:
ALTER TABLE pheno
ADD planting_onset_temp VARCHAR(16),
harvesting_onset_temp VARCHAR(16),
planting_end_temp VARCHAR(16),
harvesting_end_temp VARCHAR(16);
COPY pheno(planting_onset_temp, harvesting_onset_temp, planting_end_temp, harvesting_end_temp) FROM '/home/user/Documents/worldcrops/algeria_times.txt' DELIMITERS ';';
UPDATE pheno
SET planting_onset = CONCAT(planting_onset_temp, '/2016'),
harvesting_onset = CONCAT(harvesting_onset_temp, '/2016'),
planting_end = CONCAT(planting_end_temp, '/2016'),
harvesting_end = CONCAT(harvesting_end_temp, '/2016');
ALTER TABLE pheno DROP COLUMN planting_onset_temp, harvesting_onset_temp, planting_end_temp, harvesting_end_temp;
Replace '/2016' with whatever year is relevant.