I am having trouble exporting a file in proper csv in UTF8 from excel. I got the error : "the code page on input column is 1252 and is required to be 65001"
I looked on the error and some solution was to pass the column type to varchar.
I am trying to get my request to cast every columns (I have 191) at the same time and export this from SQL server to csv file, but i can't figure out how to make it work
My request is :
SELECT top 100000 CAST( * AS varchar(200)) from dbo.AccountingData ORDER BY NEWID();
Related
I'm trying to load some data from stage to relational environment and something is happening I can't figure out.
I'm trying to run the following query:
SELECT
CAST(SPLIT_PART(some_field,'_',2) AS BIGINT) cmt_par
FROM
public.some_table;
The some_field is a column that has data with two numbers joined by an underscore like this:
some_field -> 38972691802309_48937927428392
And I'm trying to get the second part.
That said, here is the error I'm getting:
[Amazon](500310) Invalid operation: Invalid digit, Value '1', Pos 0,
Type: Long
Details:
-----------------------------------------------
error: Invalid digit, Value '1', Pos 0, Type: Long
code: 1207
context:
query: 1097254
location: :0
process: query0_99 [pid=0]
-----------------------------------------------;
Execution time: 2.61s
Statement 1 of 1 finished
1 statement failed.
It's literally saying some numbers are not valid digits. I've already tried to get the exactly data which is throwing the error and it appears to be a normal field like I was expecting. It happens even if I throw out NULL fields.
I thought it would be an encoding error, but I've not found any references to solve that.
Anyone has any idea?
Thanks everybody.
I just ran into this problem and did some digging. Seems like the error Value '1' is the misleading part, and the problem is actually that these fields are just not valid as numeric.
In my case they were empty strings. I found the solution to my problem in this blogpost, which is essentially to find any fields that aren't numeric, and fill them with null before casting.
select cast(colname as integer) from
(select
case when colname ~ '^[0-9]+$' then colname
else null
end as colname
from tablename);
Bottom line: this Redshift error is completely confusing and really needs to be fixed.
When you are using a Glue job to upsert data from any data source to Redshift:
Glue will rearrange the data then copy which can cause this issue. This happened to me even after using apply-mapping.
In my case, the datatype was not an issue at all. In the source they were typecast to exactly match the fields in Redshift.
Glue was rearranging the columns by the alphabetical order of column names then copying the data into Redshift table (which will
obviously throw an error because my first column is an ID Key, not
like the other string column).
To fix the issue, I used a SQL query within Glue to run a select command with the correct order of the columns in the table..
It's weird why Glue did that even after using apply-mapping, but the work-around I used helped.
For example: source table has fields ID|EMAIL|NAME with values 1|abcd#gmail.com|abcd and target table has fields ID|EMAIL|NAME But when Glue is upserting the data, it is rearranging the data by their column names before writing. Glue is trying to write abcd#gmail.com|1|abcd in ID|EMAIL|NAME. This is throwing an error because ID is expecting a int value, EMAIL is expecting a string. I did a SQL query transform using the query "SELECT ID, EMAIL, NAME FROM data" to rearrange the columns before writing the data.
Hmmm. I would start by investigating the problem. Are there any non-digit characters?
SELECT some_field
FROM public.some_table
WHERE SPLIT_PART(some_field, '_', 2) ~ '[^0-9]';
Is the value too long for a bigint?
SELECT some_field
FROM public.some_table
WHERE LEN(SPLIT_PART(some_field, '_', 2)) > 27
If you need more than 27 digits of precision, consider a decimal rather than bigint.
If you get error message like “Invalid digit, Value ‘O’, Pos 0, Type: Integer” try executing your copy command by eliminating the header row. Use IGNOREHEADER parameter in your copy command to ignore the first line of the data file.
So the COPY command will look like below:
COPY orders FROM 's3://sourcedatainorig/order.txt' credentials 'aws_access_key_id=<your access key id>;aws_secret_access_key=<your secret key>' delimiter '\t' IGNOREHEADER 1;
For my Redshift SQL, I had to wrap my columns with Cast(col As Datatype) to make this error go away.
For example, setting my columns datatype to Char with a specific length worked:
Cast(COLUMN1 As Char(xx)) = Cast(COLUMN2 As Char(xxx))
I have this code
SELECT
TRY_CONVERT(varchar(150), "PCR_Fecha") AS "PCR_Fecha"
FROM OPENQUERY(EXTRACCION, 'SELECT * FROM EXTRACCION.Extraccion')
But i get the error
Error converting data type DBTYPE_DBTIMESTAMP to datetime2.
I know there are wrong values in PCR_Fecha (like 40218:00:00 or 14mayo09) that's why I'm trying to convert them to varchar.
I can see the data using
SELECT * FROM OPENQUERY(EXTRACCION, 'SELECT CAST(PCR_Fecha AS varchar(26)) FROM EXTRACCION.Extraccion');
The linked server is from Filemaker and there PCR_Fecha is set as a date.
Looking for an answer I found that I could define "dbtimestamp_rules=2" in connection string but I don't know how to add the rule.
Any other suggestions?
I found that I could change the data type from Filemaker and set it to text (it was date before) and that solves the problem.
With sql it was
SELECT
Convert(NVARCHAR(26),PCR_Fecha)
FROM OPENQUERY(EXTRACCION, 'SELECT CAST(PCR_Fecha AS VARCHAR(26)) as PCR_Fecha FROM EXTRACCION.Extraccion')
A cast and then convert, anyways thx :)
I had data in Excel like
7540006
7540447
But when I import the data into SQL Server, it is saved as
7.54001e+006
7.54045e+006
So now when I try to convert it to back to original state, it's not ending up with the correct value.
I have tried following queries for conversion
declare #a varchar(40)
set #a = '7.54001e+006'
declare #b decimal(27, 12)
SELECT #b = CONVERT(REAL, #a, 2)
SELECT LTRIM(RTRIM(str(#a))), LTRIM(STR(#a))
SELECT CAST('7.54001e+006' as REAL)
and the output I am getting is addition of 3 to original value for all methods
i.e.
7540010
7540050
How do I convert it back to original state ??
Try the following query which gives the exact value
select CAST(CAST('7.54001e+006' AS FLOAT) AS bigint)
All data is stored as Unicode string 255 (DT_WSTR) in excel.
Read excel data as Unicode form. then do conversion in ssis or database using.
SELECT CAST('7.54001e+006' as REAL)
In excel data source >> connection manager >>Data access mode
== QUERY
SELECT * FROM [SheetName$]
when you will save in database that time convert value into toString() like
Convert.toString(7540006)
then it will save original value in database.
I'm trying to get a percentage to display as a decimal in my database.
I have the following set up to convert the percentage columns into decimals:
---------------- ---------------- ------------
excel source ---------> data conversion ----------> db output
---------------- ---------------- ------------
I've tried to strictly convert the input to decimal and numeric.
Neither of these have changed my results.
In my columns in the database I'm getting just 0's and 1's.
Forgive my crude drawing; I do not have enough rep to post pictures yet.
Hope this is what you are looking for
Excel sheet like this is the source.
I just tested it in my system.It is working fine. This is what I did.
Created an SSIS package with just 1 DFT.
Data flow is given below. Please note that the value which appeared as 40% in Excel sheet is visible as 0.40. So I added two derived columns. One converting as such and the next which multiplies with 100.
the derived column structure is shown below.
The destination table structure be
Create table Destination
(
id int,
name varchar(15),
hike decimal(8,2)
)
I am getting the result as expected.
Select * from Destination
There are many ways to accomplish this. Here's one:
1) Save your excel file as a tab delimited text file.
2) Create a New Flat File Connection in SSIS
a) Set File Name = .txt file
b) Go to Advanced tab and click on the column with the percentages
c) Set the Data Type to match the target field in your database (e.g., numeric(10,5)
3) In the SSIS workflow, create a derived column of your percent field to convert from percent to decimal(e.g., newfield = oldfield/100). Make sure to check the data type has not changed in the Derived Column Transformation Editor.
SQL Server 2008 - Table contains nvarchar(max) datatype and store hindi & english data without N' prefix. like - "मांगलिक welcome" but in table store as "×梻çÜ·¤ welcome".
Please guide how to display the data from SQL server in .net.
The N prefix only denotes the string is NVARCHAR as opposed to VARCHAR
See this for more info
C# is Unicode by default so your data will be ok.
In fact re-reading your question I'm not sure what you are asking.
Are you saying you store the data in the database WITHOUT the N prefix ? Is this done via .net ?
Can you please make your question clearer ?
** EDIT
I'm not sure you can. The data outside of the non Unicode code page will be lost.
Check this page here for further details
First try to create a table as shown:
Create table TestLang (strText nvarchar(max))
Next try to insert values
insert into TestLang values ( N'मांगलिक')
insert into TestLang values ( N'Welcome')
Now try to search the name as shown:
SELECT * FROM TestLang WHERE strText LIKE N'मां%'
UPDATE:
If you want to display the data try this way:
string input = "0928;0940;0932;092E;";
Regex rx = new Regex(#"([0-9A-Fa-f]{4});");
string output = rx.Replace(input, match => ((char)Int32.Parse(match.Groups[1].Value, NumberStyles.HexNumber)).ToString());
Output: "नीलम"
Took from here