I am using a bulk insert to import data from a CSV file. One of the fields is a number, and I import it as a DECIMAL(4,3). However, this data file has a few values where the number is "3.2342e-05". This obviously throws an error since this is a char. How can I convert this number to zero? For my purposes, I plan to consider any number that small as a zero anyway.
I figure that I will be importing the data into a temp table (staging table) first, so that I can clean it up in there, and then I will be inserting it from there into my final table.
SQL Server 2008
EDIT: One thing I am considering is importing the data as a char and then converting the column type, and then using a CASE statement to set anything greater than 1 to a zero. This field should never be greater than 1, which is why I am able to do this.
This is recognised as float, so you can double CAST via float
Related
I am trying to import a csv file into SQL Server 2017 using bulk insert. Most of the int columns, except the primary key have NULLs (as blanks in the csv file) but when I import I get an error that it can't convert a string to type int for the specified target column. Since I don't know which column it is affected, I went and replaced all the blanks with 0s, but I still get the game error.
Here is what I get after I import:
I don't know what to do to make this work, but it doesn't seem to make sense to me.
The issue is not the NULLs. There is a column in your CSV that your are expecting to contain only integer values, but it in fact contains string values that cannot be converted to integers.
Here would be an example:
my_str_to_int_col
1.0
2.0
3.O
4.0
5.0
...
Notice that 3.O should actually be 3.0
So you need to determine which column you are converting to an integer contains non-integer values.
I have a flat CSV file and some fields contain a value like "1.8e-5, 8.139717345049093e-39" (exponent or scientific numbers). I need to store this value in a SQL real data type field (not float). But the maximum exponent supported by real is e-38.
But I need a mechanism to convert this string field to a real number through SSIS. Basically the e-39 or smaller values should be replaced as 0. and the rest should be stored properly.
I tried setting the data type to DT_R4 in flat file connection field mapping and that didn't help. I tried casting it to DT_R4 through a derived column and that didn't help too. When I check through Data Viewer still the value has the unsupported exponent value and it fails when I insert it to the SQL table.
I am having to create a second header line and am using the first record of the Query to do this. I am using a UNION All to create this header record and the second part of the UNION to extract the Data required.
I have one issue on one column.
,'Active Energy kWh'
UNION ALL
,SUM(cast(invc.UNITS as Decimal (15,0)))
Each side are 11 lines before and after the Union and I have tried all sorts of combinations but it always results in an error message.
The above gives me "Error converting data type varchar to numeric."
Any help would be much appreciated.
The error message indicates that one of your values in the INVC table UNITS column is non-numeric. I would hazard a guess that it's either a string (VARCHAR or similar) column or something else - and one of the values has ended up in a state where it cannot be parsed.
Unfortunately there is no way other than checking small ranges of the table to gradually locate the 'bad' row (i.e. Try running the query for a few million rows at a time, then reducing the number until you home in on the bad data). SQL 2014 if you can get a database restored to it has the TRY_CONVERT function which will permit conversions to fail, enabling a more direct check - but you'll need to play with this on another system
(I'm assuming that an upgrade to 2014 for this feature is out of the question - your best bet is likely just looking for the bad row).
The problem is that you are trying to mix header information with data information in a single query.
Obviously, all your header columns will be strings. But not all your data columns will be strings, and SQL Server is unhappy when you mix data types this way.
What you are doing is equivalent to this:
select 'header1' as col1 -- string
union all
select 123.5 -- decimal
The above query produces the following error:
Error converting data type varchar to numeric.
...which makes sense, because you are trying to mix both a string (the header) with a decimal field.
So you have 2 options:
Remove the header columns from your query, and deal with header information outside your query.
Accept the fact that you'll need to convert the data type of every column to a string type. So when you have numeric data, you'll need to cast the column to varchar(n) explicitly.
In your case, it would mean adding the cast like this:
,'Active Energy kWh'
UNION ALL
,CAST(SUM(cast(invc.UNITS as Decimal (15,0))) AS VARCHAR(50)) -- Change 50 to appropriate value for your case
EDIT: Based on comment feedback, changed the cast to varchar to have an explicit length (varchar(n)) to avoid relying on the default length, which may or may not be long enough. OP knows the data, so OP needs to pick the right length.
What data type in SQL Server will insert the correct number of decimal places for the number entered into a table. When I try to enter a number like so 6.828.678 into a deciaml or numeric field it says input value was not in the correct format. What data type do i use so that it allows this?
If you are trying to store the data in format 6.828.678 (as in your post) then consider using VARCHAR data type.
You can't store 6.828.678 format data into either deciaml or numeric type field cause they are not valid number format and that's why you are getting the error.
Not sure, but per your comment if you want to SUM and compare the value then store it as INT datatype but before inserting the data make sure to remove the ..
You can use REPLACE() function like CAST(REPLACE('6.828.678','.','') as INT). Hope this helps.
I am importing data from a text file and have hit a snag. I have a numeric field which occasionally has very large values (10 billion+) and some of these values which are being converted to NULLs.
Upon further testing I have isolated the problem as follows - the first 25 rows of data are used to determine the field size, and if none of the first 25 values are large then it throws out any value >= 2,147,483,648 (2^31) which comes after.
I'm using ADO and the following connection string:
Provider=Microsoft.Jet.OLEDB.4.0;Data Source=FILE_ADDRESS;Extended Properties=""text;HDR=YES;FMT=Delimited""
Therefore, can anyone suggest how I can get round this problem without having to get the source data sorted descending on the large value column? Is there some way I could define the data types of the recordset prior to importing rather than let it decide for itself?
Many thanks!
You can use an INI file placed in the directory you are connecting to which describes the column types.
See here for details:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms709353(v=vs.85).aspx