I'm looking at an inherited SSIS package, and a stored procedure is sending records to a recordset called USER:NEW_RECORDS. It's of type Object, and the value is System.Object. It is then used for inputting that data to a SQL table. We're getting an error, because it seems that the numeric results of the stored procedure are being put in a DT_WSTR field, and then failing when it is then put into a decimal field in the database.
Most of the records are working, but one, which happens to have a longer number of decimal digits, is failing.
I want to see exactly what my SSIS recordset field types are, and probably change them, so I can force the data to be truncated properly and copied. Or, perhaps, I'm not even looking at this correctly. The data is put into the recordset using a SQL Task that executes the stored procedure.
Edit: It appears that this particular recordset is used twice, and this is the second use of it. I'm thinking that perhaps it has the data types of the first use. But I can't put a Data Viewer on a SQL Task, can I?
I am having the same trouble, so I directed the flow behind the record set into a flat file.
I did make a new recordset to use, so that the other one was not used. And while I never did figure out how to see the data, I could change the data types of the types in the parameter mapping, which was apparently what was needed. I changed a type from NUMERIC to FLOAT, and it quit complaining about some of the data.
This question may be too specific to my own problem to be of use to others. I may delete it.
Related
I am using MS query to pull data from sql server and all is good.
Problem starts when data comes from the server I am stuck with data type general for everything, and no way to change the data type in excel.
Main issue is numbers, where in database datatype is decimal yet i can do no calculations on it in excel. Any help would be appreciated.
I am using excel to execute a stored procedure on server
This pulls the data into the following table
Even though the data in the sql server for column price is formatted as decimal it becomes a general data type after getting to excel.
Changing it to number/currency etc. does not change anything.
Also no errors appear. Simply data comes down and no matter what changes in excel I apply nothing changes it all is treated as text.
You can do these things.
Select Column
Click Data-> Text to Columns
Follow the wizard
Set the format
Use this official support ticket from Microsoft
Problem in this case was created by myself.
But I suppose it could easily happen to others who are just starting on their path with sql and excel.
Here is what happened as I established after few days of going in circles.
as there was load of trailing spaces in the data coming down from the server I have decided to tidy things up.
Without considerring implications I have stuck an RTRIM() on everything.
This caused excel to treat everything as strings as string RTRIM is a built in string function.
What made things worse is the fact that when using power query I was able to transform the data to the desired, formats.
Unfortunately MS query does not seem to be quite as clever as power query hence the issues.
I have some data which was dumped from a PostgreSQL database (allegedly, using pg_dump) which needs to get imported into SQL Server.
While the data types are ok, I am running into an issue where there seems to be a placeholder for a NULL. I see a backslash followed by an uppercase N in many fields. Below is a snippet of the data, as viewed from within Excel. Left column has a Boolean data type, and the right one has an integer as the data type
Some of these are supposed to be of the Boolean datatype, and having two characters in there is most certainly not going to fly.
Here's what I tried so far:
Import via dirty read - keeping whatever datatypes SSIS decided each field had; to no avail. There were error messages about truncation on all of the boolean fields.
Creating a table for the data based on the correct data types, though this was more fun... I needed to do the same as in the dirty read, as the source would otherwise not load properly. There was also a need to transform the data into the correct data type for insertion into the destination data source; yet, I am getting truncation issues, when it most certainly shouldn't be.
Here is a sample expression in my derived column transformation editor:
(DT_BOOL)REPLACE(observation,"\\N","")
The data type should be Boolean.
Any suggestion would be really helpful!
Thanks!
Since I was unable to circumvent the SSIS rules in order to get my data into my tables without an error, I took the quick-and-dirty approach.
The solution which worked for me was to have the source data read each column as if it were a string, and the destination table had all fields be of the datatype VARCHAR. This destination table will be used as a staging table, once in SS, I can manipulate as needed.
Thank you #cha for your input.
When I was going through all the tables in my database, I could see a table called Measbinary and an attribute attracted me was RawData. Which is Image type and Allow null. I have attached a screenshot of the table Could someone help me understand what is that? and how could I understand How it has been processed ?
Update : I checked the stored procedures and could find that the image parameter is passed to it like
SP_StoreBinary #rawspectra image
and then the value is inserted to the table mentioned above.
This is the raw data of a binary field. It has "no meaning" except being a way for SSMS (Management Studio) to show SOMETHING for a binary field. Remember - SSMS (and the database) have no clue what is in that field (image, word document, whatever) and how to show it. A hex coded string is "as good as it gets" as a generic approach, as it allows a programmer to compare the first bytes.
I have a table with a column1 nvarchar(50) null. I want to insert this into a more 'tight' table with a nvarchar(30) not null. My idea was to insert a derived column task between source and destination task with this expression: Replace column1 = (DT_WSTR,30)Column1
I get the "truncation may occur error" and I am not allowed to insert the data into the new tighter table.
Also I am 100% sure that no values are over 30 characters in the column. Moreover I do not have the possibility to change the column data type in the source.
What is the best way to create the ETL process?
JotaBe recommended using a data conversion transformation. Yes, that is another way to achieve the same thing, but it will also error out if truncation occurs. Your way should work (I tried it), provided the input data really is less than 30 characters.
You could modify your derived column expression to
(DT_WSTR,30)Substring([Column1], 1, 30)
Consider changing the truncation error disposition of the Derived Column component within your Data Flow. By default, a truncation will cause the Derived Column component to fail. You can configure the component to ignore or redirect rows which are causing a truncation error.
To do this, open the Derived Column Transformation editor and click the 'Configure Error Output...' button in the bottom-left of the dialog. From here, change the setting in the 'Truncation' column for any applicable columns as required.
Be aware that any data which is truncated for columns ignoring failure will not be reported by SSIS during execution. It sounds like you've already done this, but it's important to be sure you've analysed your data as it currently stands and taken into consideration any possible future changes to the nature of the data before disabling truncation reporting.
To do so you must use a Data Conversion Transformation, which allows to change the data type from the original nvarchar(50) to the desired nvarchar(30).
You'll get a new column with the required data type.
Of course, you can decide what to do in case of error: truncation, by configuring this component.
UPDATE
As there are people who have downvoted this answer, let's add 3 more comments:
this solution is checked and works. Create a table with a nvarchar(50) column, a new table with a nvarchar(30) column, add a data flow that uses a data conversion transform and it works witout a glitch. Please, chek it, I guarantee. Besides, as the OP states "Also I am 100% sure that no values are over 30 characters in the column" in his case there will be no truncation problems. However, I recommend treating the possible errors, just in case they happen.
from MSDN: "a package can perform the following types of data conversions: ... Set the column length of string data"
from MSDN: "If the length of an output column of string data is shorter than the length of its corresponding input column, the output data is truncated."
Suppose I have a bunch of varchar(6000) fields in a table and want to change those to text fields. What are the ramifications of the stored procedures whose arguments are of type varchar(6000). Does each stored procedure also need those argument data types changed?
Text fields are deprecated in SQL Server 2005 and above. You should use varchar(MAX), if possible. If you expect to have more than 6000 characters passed in the arguments to your stored procedures, you will need to change them as well.
Text fields are rough to work with in SQL Server. You can't actually declare local variables of type text (except as parameters to a stored procedure) and most of the string manipulation functions no longer work on text fields.
Also if you have triggers the text fields will not appear on the INSERTED or DELETED tables.
Basically if the field is just holding data from a program and you aren't manipulating it then no big deal. But if you have stored procedures to manipulate the string then your task will be way more difficult.
As tvanfosson mentioned if you have SQL Server 2005 use VARCHAR(MAX) then you get the length of a text field with the ability to manipulate it like it is a VARCHAR.
The other answers are right, but they don't answer your question. Varchar(max) is the way to go. If you made the feilds varchar(max)/text, but kept the stored proc arguments the same, any field that came in through the stored proc would be truncated to 6000 characters. Since you say that it will never exceed that, you will be fine, until, of course, that isn't the case. It doesn't throw an error. It just truncate.
I'm not sure of the exact behavior of varchar(max) verses text, but I'm pretty sure that once you start putting a lot of them in one table, you can get some crazy performance hits. Why so many big fields in one table?
The reason for text field usage is that all of the varchar(6000) fields in one row exceed the max row length. Text fields just store a pointer in the row thus not exceeding the SQL Server max row length of 8000 something. ATM the database cannot be normalized. The data is not manipulated by the stored procedures it's just inserted, updated and deleted.
Does VARCHAR(MAX) behave like a text field and only store a pointer to the data in the row?