I am loading data from excel sheet to sql using SSIS package. When I load data into table1 using the datatype varchar(255) for all fields, I had no problem. But, when I tried to load data from table1 to table2 it showed an error : can not convert datatype varchar to numeric.
All of the fields in table2 have valid datatypes. Now when I look at the data in table1 for the field (its datatype is decimal(5,2) in table2) which was giving me that error, I saw one of the record had a value of "2.9999999999999999E-2" in table1. The same record in the excel sheet is 0.03.
In the same column there is a record with a value of 0.01. Why did it change the value for 0.03? Do I have to convert the data in excel sheet? I want to load records from excel sheet the way they are.
I am using sql server 2005.
Thanks
Odds are, that the 0.01 value is in a text cell in excel, while the 0.03 is in a numeric cell. Your library sees that it has a number, and tries to import it numerically. Floating point numbers can't represent certain numbers perfectly, which leads to your error.
One way to solve this is to mark the cells in the original spreadsheet as text, or if your library has the capability, mark them as text when importing the values.
Have you considered checking the connection string, particularly IMEX=1? This worked for me when I was facing similar issues in the past.
http://www.connectionstrings.com/excel
Sorry for the digging up of a 5 year old post, but it looks like a lot of people have looked at it.
I ran into a similar issue with numbers from excel coming over in the Scientific Notation format
Try:
CONVERT(FLOAT, [Your Column Here], 2)
Then wrap it into whatever type you need
Apparently when you get the Scientific Notation format don't work as expected.
This is how I found the answer see Quassnoi's post
Related
I am using MS query to pull data from sql server and all is good.
Problem starts when data comes from the server I am stuck with data type general for everything, and no way to change the data type in excel.
Main issue is numbers, where in database datatype is decimal yet i can do no calculations on it in excel. Any help would be appreciated.
I am using excel to execute a stored procedure on server
This pulls the data into the following table
Even though the data in the sql server for column price is formatted as decimal it becomes a general data type after getting to excel.
Changing it to number/currency etc. does not change anything.
Also no errors appear. Simply data comes down and no matter what changes in excel I apply nothing changes it all is treated as text.
You can do these things.
Select Column
Click Data-> Text to Columns
Follow the wizard
Set the format
Use this official support ticket from Microsoft
Problem in this case was created by myself.
But I suppose it could easily happen to others who are just starting on their path with sql and excel.
Here is what happened as I established after few days of going in circles.
as there was load of trailing spaces in the data coming down from the server I have decided to tidy things up.
Without considerring implications I have stuck an RTRIM() on everything.
This caused excel to treat everything as strings as string RTRIM is a built in string function.
What made things worse is the fact that when using power query I was able to transform the data to the desired, formats.
Unfortunately MS query does not seem to be quite as clever as power query hence the issues.
So it's been a long time since I've done anything fancy with SQL, so I'm going to do my best to explain. Please be nice, I'm trying my best here.
Basically, I'm pulling information from a database in Snowflake and putting it into a new XML file, and that data is input exactly as-written into a form email.
One of the values is an ID number that's 14 characters long (example: 12345678912345), which is stored in the database as an integer (or so I'm told), but Excel keeps automatically converting it into scientific notation. Since it's an ID number, it needs to look like an ID number, not scientific notation.
Right now, my query just selects & inputs the regular ol' value, and then we manually change it in the Excel sheet. Like literally just SELECT ID_Number from TheThing
One thing I thought might work is:
SELECT CAST(ID_Number as bigint) as ID_Number
... But it doesn't work. Most other solutions I've found don't seem to address my specific scenario of unwanted integer-to-string conversion & I'm distraught.
I'm just an intern and this might have a very obvious answer, but my fellow interns have given up on it and I need to find the answer for my own sanity. It's been a minute since I did anything fancy with SQL so please be nice to me and sorry if this is a dumb question.
In Snowflake, BIGINT and INT(EGER) are the same thing, what you want is VARCHAR. As Ross mentioned in his comment, this is likely just a formatting issue within Excel. In Excel any value can be cast as a string by including a single quote ' at the beginning of the value, or by using the Text-to-Column feature.
If you wanted to try to format it out of Snowflake as a string, casting it might not do the trick unless you include some kind of additional string character.
To get this type of formatting out of Snowflake, you can try:
SELECT '\'' || CAST(ID_Number AS VARCHAR) as ID_Number;
I am having to create a second header line and am using the first record of the Query to do this. I am using a UNION All to create this header record and the second part of the UNION to extract the Data required.
I have one issue on one column.
,'Active Energy kWh'
UNION ALL
,SUM(cast(invc.UNITS as Decimal (15,0)))
Each side are 11 lines before and after the Union and I have tried all sorts of combinations but it always results in an error message.
The above gives me "Error converting data type varchar to numeric."
Any help would be much appreciated.
The error message indicates that one of your values in the INVC table UNITS column is non-numeric. I would hazard a guess that it's either a string (VARCHAR or similar) column or something else - and one of the values has ended up in a state where it cannot be parsed.
Unfortunately there is no way other than checking small ranges of the table to gradually locate the 'bad' row (i.e. Try running the query for a few million rows at a time, then reducing the number until you home in on the bad data). SQL 2014 if you can get a database restored to it has the TRY_CONVERT function which will permit conversions to fail, enabling a more direct check - but you'll need to play with this on another system
(I'm assuming that an upgrade to 2014 for this feature is out of the question - your best bet is likely just looking for the bad row).
The problem is that you are trying to mix header information with data information in a single query.
Obviously, all your header columns will be strings. But not all your data columns will be strings, and SQL Server is unhappy when you mix data types this way.
What you are doing is equivalent to this:
select 'header1' as col1 -- string
union all
select 123.5 -- decimal
The above query produces the following error:
Error converting data type varchar to numeric.
...which makes sense, because you are trying to mix both a string (the header) with a decimal field.
So you have 2 options:
Remove the header columns from your query, and deal with header information outside your query.
Accept the fact that you'll need to convert the data type of every column to a string type. So when you have numeric data, you'll need to cast the column to varchar(n) explicitly.
In your case, it would mean adding the cast like this:
,'Active Energy kWh'
UNION ALL
,CAST(SUM(cast(invc.UNITS as Decimal (15,0))) AS VARCHAR(50)) -- Change 50 to appropriate value for your case
EDIT: Based on comment feedback, changed the cast to varchar to have an explicit length (varchar(n)) to avoid relying on the default length, which may or may not be long enough. OP knows the data, so OP needs to pick the right length.
I'm trying to import a flat file into an oledb target sql server database.
here's the field that's giving me trouble:
here are the properties of that flat file connection, specifically the field:
here's the error message:
[Source - 18942979103_txt [424]] Error: Data conversion failed. The
data conversion for column "recipient-name" returned status value 4
and status text "Text was truncated or one or more characters had no
match in the target code page.".
What am I doing wrong?
Here is what fixed the problem for me. I did not have to convert to Excel. Just modified the DataType when choosing the data source to "text stream" (Figure 1). You can also check the "Edit Mappings" dialog to verify the change to the size (Figure 2).
Figure 1
Figure 2
After failing by increasing the length or even changing to data type text, I solved this by creating an XLSX file and importing. It accurately detected the data type instead of setting all columns as varchar(50). Turns out nvarchar(255) for that column would have done it too.
I solved this problem by ORDERING my source data (xls, csv, whatever) such that the longest text values on at the top of the file. Excel is great. use the LEN() function on your challenging column. Order by that length value with the longest value on top of your dataset. Save. Try the import again.
SQL Server may be able to suggest the right data type for you (even when it does not choose the right type by default) - clicking the "Suggest Types" button (shown in your screenshot above) allows you to have SQL Server scan the source and suggest a data type for the field that's throwing an error. In my case, choosing to scan 20000 rows to generate the suggestions, and using the resulting suggested data type, fixed the issue.
While an approach proposed above (#chookoos, here in this q&a convert to Excel workbook) and import resolves those kinds of issues, this solution this solution in another q&a is excellent because you can stay with your csv or tsv or txt file, and perfom the necessary fine tuning without creating a Microsoft product related solution
I've resolved it by checking the 'UNICODE'checkbox. Click on below Image link:
You need to go increase the column length while importing the data for particular column.
Choose a data source >> Advanced >> increase the column from default 50 to 200 or more.
Not really a technical solution, but SQL Server 2017 flat file import is totally revamped, and imported my large-ish file with 5 clicks, handled encoding / field length issues without any input from me
SQl Management Studio data import looks at the first few rows to determine source data specs..
shift your records around so that the longest text is at top.
None of the above worked for me. I SOLVED my problem by saving my source data (save as) Excel file as a single xls Worksheet Excel 5.0/95 and imported without column headings. Also, I created the table in advance and mapped manually instead of letting SQL create the table.
I had similar problem against 2 different databases (DB2 and SQL), finally I solved it by using CAST in the source query from DB2. I also take advantage of using a query by adapting the source column to varchar and avoiding the useless blank spaces:
CAST(RTRIM(LTRIM(COLUMN_NAME)) AS VARCHAR(60) CCSID UNICODE
FOR SBCS DATA) COLUMN_NAME
The important issue here is the CCSID conversion.
It usually because in connection manager it may be still of 50 char , hence I have resolved the problem by going to Connection Manager--> Advanced and then change to 100 or may be 1000 if its big enough
I live in Brasil and decimal separators are commas. For a bunch of reasons, I use dots as decimal separators in SQL Server, which is different from Excel.
With that being said, I would like to know why the following query
select 1.0*5
is understood as text in Excel (if so), when copying and pasting, and dots are not converted to commas, while
select cast(1.0*5 as float)
is understood as float in Excel.
What is the type of result in the first query?
UPDATE
If the query were
select 1.1*5
the result of copy and paste in Excel cell would be 5.5. It is not possible to convert this to value in Excel.
While the second query would result in 5,5. I can use the use this value in Excel in an addition operation, for example.
If you're doing it directly IN Excel, it seems that your regional settings are not seeing that as an operation with a decimal, but rather text. If you change your regional settings to US, it would probably resolve it correctly.
The difference between the two is that you are literally telling the value to be cast differently than the default. So your regional setting is overridden.
Excel, as smart as it is, tends to make many assumptions that could be tied to any number of things. Sometimes you just have to deal with it.
In the end, your 2nd query is likely to produce better results.