I make table:
CREATE TABLE data (
date date,
time time,
val3 float,
val4 float);
And trying to load csv to it with next command:
copy data from 'G:\test\1.txt' DELIMETERS ' ' CSV;
the CSV have same structure:
date time val3 val4
2012.08.10 06:53:18 695.417 773.29
But I am getting next error:
ERROR: invalid input syntax for type date: "date"
Could you help me to find the reason of error?
I'm not very used to postgres, but I think you should set your datestyle before importing your file :
set datestyle German, YMD;
Look at these links : How do I alter the date format in Postgres? and DateTime Output
Sorry if I'm wrong, but I think you have to correctly set your datestyle (you can also do in the postgres.cnf file).
Do you have a header line in your CSV? If so, try
copy data from 'G:\test\1.txt' DELIMETERS ' ' CSV HEADER;
Related
I'm trying to load some data from stage to relational environment and something is happening I can't figure out.
I'm trying to run the following query:
SELECT
CAST(SPLIT_PART(some_field,'_',2) AS BIGINT) cmt_par
FROM
public.some_table;
The some_field is a column that has data with two numbers joined by an underscore like this:
some_field -> 38972691802309_48937927428392
And I'm trying to get the second part.
That said, here is the error I'm getting:
[Amazon](500310) Invalid operation: Invalid digit, Value '1', Pos 0,
Type: Long
Details:
-----------------------------------------------
error: Invalid digit, Value '1', Pos 0, Type: Long
code: 1207
context:
query: 1097254
location: :0
process: query0_99 [pid=0]
-----------------------------------------------;
Execution time: 2.61s
Statement 1 of 1 finished
1 statement failed.
It's literally saying some numbers are not valid digits. I've already tried to get the exactly data which is throwing the error and it appears to be a normal field like I was expecting. It happens even if I throw out NULL fields.
I thought it would be an encoding error, but I've not found any references to solve that.
Anyone has any idea?
Thanks everybody.
I just ran into this problem and did some digging. Seems like the error Value '1' is the misleading part, and the problem is actually that these fields are just not valid as numeric.
In my case they were empty strings. I found the solution to my problem in this blogpost, which is essentially to find any fields that aren't numeric, and fill them with null before casting.
select cast(colname as integer) from
(select
case when colname ~ '^[0-9]+$' then colname
else null
end as colname
from tablename);
Bottom line: this Redshift error is completely confusing and really needs to be fixed.
When you are using a Glue job to upsert data from any data source to Redshift:
Glue will rearrange the data then copy which can cause this issue. This happened to me even after using apply-mapping.
In my case, the datatype was not an issue at all. In the source they were typecast to exactly match the fields in Redshift.
Glue was rearranging the columns by the alphabetical order of column names then copying the data into Redshift table (which will
obviously throw an error because my first column is an ID Key, not
like the other string column).
To fix the issue, I used a SQL query within Glue to run a select command with the correct order of the columns in the table..
It's weird why Glue did that even after using apply-mapping, but the work-around I used helped.
For example: source table has fields ID|EMAIL|NAME with values 1|abcd#gmail.com|abcd and target table has fields ID|EMAIL|NAME But when Glue is upserting the data, it is rearranging the data by their column names before writing. Glue is trying to write abcd#gmail.com|1|abcd in ID|EMAIL|NAME. This is throwing an error because ID is expecting a int value, EMAIL is expecting a string. I did a SQL query transform using the query "SELECT ID, EMAIL, NAME FROM data" to rearrange the columns before writing the data.
Hmmm. I would start by investigating the problem. Are there any non-digit characters?
SELECT some_field
FROM public.some_table
WHERE SPLIT_PART(some_field, '_', 2) ~ '[^0-9]';
Is the value too long for a bigint?
SELECT some_field
FROM public.some_table
WHERE LEN(SPLIT_PART(some_field, '_', 2)) > 27
If you need more than 27 digits of precision, consider a decimal rather than bigint.
If you get error message like “Invalid digit, Value ‘O’, Pos 0, Type: Integer” try executing your copy command by eliminating the header row. Use IGNOREHEADER parameter in your copy command to ignore the first line of the data file.
So the COPY command will look like below:
COPY orders FROM 's3://sourcedatainorig/order.txt' credentials 'aws_access_key_id=<your access key id>;aws_secret_access_key=<your secret key>' delimiter '\t' IGNOREHEADER 1;
For my Redshift SQL, I had to wrap my columns with Cast(col As Datatype) to make this error go away.
For example, setting my columns datatype to Char with a specific length worked:
Cast(COLUMN1 As Char(xx)) = Cast(COLUMN2 As Char(xxx))
I am having trouble exporting a file in proper csv in UTF8 from excel. I got the error : "the code page on input column is 1252 and is required to be 65001"
I looked on the error and some solution was to pass the column type to varchar.
I am trying to get my request to cast every columns (I have 191) at the same time and export this from SQL server to csv file, but i can't figure out how to make it work
My request is :
SELECT top 100000 CAST( * AS varchar(200)) from dbo.AccountingData ORDER BY NEWID();
The file (date.sql) has the below commands
date=2014-12-03
SET VAR vDate $date
insert into (Some_table) (date,location)
select $vDate , location from (let say from table "Location")
while executing this sql script i am getting the below error as its expecting values like '2014-12-03' but here its coming without quotes
ERROR:Insert data type mismatch for column date
Is there anyway to concat ' before and after date or is there any simpler way to achieve it ???
Thanks in advance.
I've never had to do this before but I'm looking to do a SQL replace on varbinary data. I'm trying this but it's not successfully replacing, I think because it's treating the data as varchar and then cast back to varbinary. This stemmed from blank spaces at the end of these values but are not actual 'spaces' so RTRIM doesn't work in this scenario (bold below).
Example of Data trying to change:
0x457874656368203430304120414320636C616D70206D657465722C2041432063757272656E74206D6F64656C20746F206D65657420796F7572206170706C69636174696F6E206E656564732E20203230303020636F756E74204C434420646973706C61792E20204869676820616363757261637920666F722063757272656E74206D6561737572656D656E74732E2020302E3922202832336D6D29206A61772073697A65206163636F6D6D6F646174657320636F6E647563746F727320757020746F203330304D434D2E2020436F6E74696E756974792062656570657220616E642064696F646520746573742E20204461746120686F6C6420616E64206D617820686F6C642E20204F7665726C6F61642070726F74656374696F6E20666F7220616C6C2072616E6765732E20204F76657272616E676520616E64206C6F77206261747465727920696E64696361746F72732E20204175746F72616E67696E672077697468206175746F20706F776572206F66662E0D0A090909090909090909090D0A090909090909090909090D0A090909090909090909090D0A09090909090909090909
Script:
update digitalassetcontent
set content = (CAST(REPLACE(content, '0D0A09090909090909090909', '') as varbinary(MAX)))
update digitalassetcontent
set content = REPLACE(content,0x0D0A09090909090909090909,0x)
During a bulk insert from a csv file a row in the file has 00000100008 value, both source (from which csv file is created) and the destination temptable has same field (char(11)).
When I try to insert I got the following error:
Bulk load data conversion error (truncation) for row 1, column 1 (fieldname)
If I remove the leading 0s and change this value to 100008 in csv file and then bulk insert, the destination table temptable shows '++ 100008 as inserted value. Why is that? How I can cope with value without leading double plus signs?
Here is the script:
BULK
INSERT temptable
FROM 'c:\TestFile.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
Edit: Some sample records from csv file.
100008,111122223333,Mr,ForeName1,SurName1,1 Test Lane,London,NULL,NULL,NULL,wd25 123,test#email.com.com,NULL
322,910315715845,Ms,G,Dally,17 Elsie Street,NULL,NULL,GOOLE,NULL,DN146DU,test1#email1.com,
323,910517288401,Mrs,G,Tom,2 White Mead,NULL,NULL,YEOVIL,NULL,BA213RS,test3#tmail2.com,
My first thought is that the file was saved on a Unix system and that you may have incompatibilities with the different style line breaks.
My first advice would be to analyze the text file using a hex editor to try to determine what character is getting put there.
++ 100008 basically means - Row format is inconsistent with page header. To solve this problem Run dbcc checktable.
I hope that this is going to help you.
Regards,