I want to use the foreach container to iterate through a folder matching something like: "Filename_MMYYYY.xls". That's easy enough to do; but I can't seem to find a way to parse the MMYYYY from the filename and add it to a variable (or something) that i can use as a lookup field for my DimDate table. It seems possible with a flat file data source, but not an excel connection. I'm using Visual Studio 2005. Please help!
Do I understand correctly that you want to take your filename, deconstruct it, and get a date-typed variable out of it? If so, then you need to start with the filename variable that you get from the Foreach Loop - I'll call that variable #FileName.
First, make a new variable - #FileDate - as a DateTime type. Go to its properties window (F4), and set the EvaluateAsExpression property to True. Edit the expression, and type in something like this (you may need to tweak):
(DT_DBTIMESTAMP)(SUBSTRING(#FileName, 12, 4) + "-" + SUBSTRING(#FileName, 10, 2) + "-01")
Now, if you want to take that date value and use it in your Data Flow, you can just use it straight in a Derived Column transform, or in an expression on your Lookup SQL statement, or wherever.
Related
I am creating an SSIS package which has an execute SQL task and it passes result set variable to a for each loop container.
My Sql Query is:
Select distinct code from house where active=1 and campus='W'
I want the execute sql task to run this query and assign its results to a variable which is passed to a for each loop container which should loop through all the values in the result set.
But my execute sql task fails with error:
The type of the value (DBNull) being assigned to variable
"User::house" differs from the current variable type (String)
Now i have done my research and i have tried assigning the variable datatype Object but did not work. I tried using cast in my sql query and that also did not work.
Since my query returns multiple rows and one column, i am not sure how i can assign a datatype to the whole query?
Sample:
Code
AR
BN
CN
It sounds like you have a variety of issues in here.
Result Set
The first is in your Execute SQL Task and the need for agreement between the Result Set specification and the data type of the Variable(s) specified in the Result Set tab. If you specify Full Resultset, then the receiving object must be of type System::Object and you will only have 1 result set. The type of Connection Manager (ODBC/OLE/ADO) used will determine how you specify it but it's infinitely searchable on these fine forums.
The other two options are Single Row and XML. In 13 years of working with SSIS, I've never had cause to specify XML. That leaves us with Single Row. For a Single Row Result Set, you need to provide a variable for each column returned and it needs to be correctly typed.
To correct your issue, you need to declare a second variable. I usually call my rsObject (record set object) and then specify the data type as System.Object.
For Each Loop Container
Your For Each Loop Container will then be set with an Enumerator of "Foreach ADO Enumerator" and then the ADO object source variable will become "User::rsObject"
In the Variable Mappings, you'll specify your variable User::house to index 0.
Testing
Given a sample set of source source data, you can verify that you have your Execute SQL Task correctly assigning a result set to our object and the Foreach Loop Container is properly populating our variable.
SELECT DISTINCT
code
FROM
(
VALUES
('ABC', 1, 'w')
, ('BCD', 1, 'w')
, ('CDE', 0, 'w')
, ('DEF', 1, 'w')
, ('EFG', 1, 'x')
) house(code, active, campus)
WHERE
active = 1
AND campus = 'w';
If you change the value of campus from w to something that doesn't exist, like f then things will continue to work.
However, the error you're receiving can only be generated if the code is a NULL
Add one more entry to the VALUES collection like
, (NULL, 1, 'w')
and when the For Each Loop Container hits that value, you will encounter the error you indicate
The type of the value (DBNull) being assigned to variable "User::house" differs from the current variable type (String)
Now what?
SSIS variables cannot change their data type, unless they're of type Object (but that's not the solution here). The "problem" is that you cannot store a NULL value in an SSIS variable (unless it's of type object). Therefore you need to either exclude the rows that return a NULL (AND code IS NOT NULL) or you need to cast the NULL into sentinel/placeholder value as a substitute (SELECT DISTINCT ISNULL(code, '') AS code). If an empty string is a valid value, then you need to find something that isn't - "billinkcisthegreatestever10123432" is unlikely to exist in your set of codes but that might be a bit excessive.
Finally, think about renaming your SSIS variable from house to code. You might be able to keep things straight but some day you'll hand this code over to someone else for maintenance and you don't want to confuse them.
A picturesque answer https://stackoverflow.com/a/13976990/181965
the variable "User::house" is string , so , did you use it in result set?
you need declare son "object" var for result set
result set
then declare a string variable for every single Code from your result
For Each Loop Container
good luck
I have a stored procedures which receives a list of ItemId's. If the list only contain one item this works:
AND (#OrgItemIds = -1 OR ...)
AND (#OrgItemIds = -1 AND...)
but if the list contains more than one item it crashes.
Anybody knows how to fix this? Can I check the size of the list somehow?
Or can I check just the first element of the list like #OrgItemIds[0] or something like that?
There is no "comma-separated set of values" or "array" datatypes in SQL SERVER. You handle the parameter as a scalar variable in your code. Thus when you provide a single value in that list, server implicitly converts it to int and your sp succeeds.When you provide string with commas - it becomes impossible to convert it to int.You have to manually parse your argument, put it into a table variable and then use this table in WHERE clause. Or change handling of this argument to support a string of values instead of scalar value:
... where #OrgItemIds like '%,' + cast(t.OrgItemIds as varchar(10)) + ',%'
which is much worse for performance than filtering by id or list of ids.
Trying to use this:
substring(#[User::v_Filename],37,3)
However, it seems substring can only handle 20 characaters ?
The file name looks like this:
D:\Projects\OTS\MYSSA Dashboard\Data\ATL_20150725Text.csv
All I want is the ATL Portion
But when the ssis moves to the next file, it may change to NYC or DAL, there are about 26 files to be processed all from different regions.
Test the substring without your file variable, that uses the substring function in an expression.
Example - create a variable with the expression, then click "evaluate":
substring("ABCDEFGHIJKLMNOPQRSTUVWXYZ", 25, 2)
It will work fine.
Given that the substring function works fine, there must be something wrong with your [User::v_Filename] variable. Are you sure it is being set correctly? Perhaps you should try running BIDS with the debugger on and a breakpoint set to right after you assign the filename, and verify that it indeed is being set correctly.
myRs=myStmt.executeQuery("select i_col,col_name from tab_col")
i=0
while (myRs.next()):
list= myRs.getString("I_COL")+','+myRs.getString("COL_NAME")
i have a jython code to run a sql statement i want to store all the row of the sql into a single variable. I used to list to store the value but its always storing only the single line , so is there is way to append all the rows and keep adding to single variable.
Thanks so much for your help.
With that code you overwrite the "list" variable in every iteration of the while loop (= is an assignment), try something like this (I used rs rather than list to avoid a name clash with the builtin function list()):
myRs=myStmt.executeQuery("select i_col,col_name from tab_col")
rs=[]
i=0
while (myRs.next()):
rs.append(myRs.getString("I_COL")+','+myRs.getString("COL_NAME"))
It seems like it should be simple but as of yet I havent found a way to save the value stored in an SSIS string variable to a text file. I've looked at using the flat file destination inside of a data flow but that requires a data flow source.
Any ideas on how to do this?
Use a script task.
I just tried this. I created a File connection manager, with the connection string pointing to the file I wanted to write to. I then created a string variable containing the text to write.
I added a Script Task, specified my string variable in the Read Only Variables list, then clicked Edit Script. The script was as follows:
public void Main()
{
ConnectionManager cm = Dts.Connections["File.tmp"];
var path = cm.ConnectionString;
var textToWrite = (string)Dts.Variables["User::StringVariable"].Value;
System.IO.File.WriteAllText(path, textToWrite);
Dts.TaskResult = (int)ScriptResults.Success;
}
This worked with no problems.
Here's a little sample of some code that worked in a SQL CLR in C#. You'll need to use VB if you're on 2005 I believe. The script task also needs the read variable property set to MyVariable to make the value of your variable available to it.
// create a writer and open the file
TextWriter tw = new StreamWriter("\\\\server\\share$\\myfile.txt");
// write a line of text to the file
tw.WriteLine(Dts.Variables["MyVariable"].Value);
// close the stream
tw.Close();
All it takes is one line of code in a simple Script task. No other dependencies, such as a connection manager, are needed.
Here's what it would look like in C#:
public void Main()
{
string variableValue = Dts.Variables["TheVariable"].Value.ToString();
string outputFile = Dts.Variables["Path"].Value.ToString();
System.IO.File.WriteAllText(outputFile, variableValue);
Dts.TaskResult = (int)ScriptResults.Success;
}
Obviously the most important line here is the one containing the WriteAllText function call.
The Path variable should contain a full path + filename for the output file.
Ok, I have an answer that doesn't involve use of script task. Pick some oledb sql source you have that's simple and you have a lot of control over. Make a query that returns only one row. Then put this query in a string variable:
"select vara, ' var =: " + #[User:varIWantToSee] + "' as myvar from tablea where vara = 1"
Then in OLEDB source pick "SQL command from a variable"
For varIWantToSee make sure you initialize it with a lot characters or ssis makes a very small length for that column that it doesn't let you override. At run time varIWantToSee will get set and you can see it. Pump this all into a flat file destination and you are in business. Why do some people have to do this? Because some people need to know the value of the variables in the runtime environment, their laptop development doesn't show the variable values they need. In my case I was running this on an Azure environment that had the database accesses I needed to test. If I were microsoft I would create a task that shows the runtime variable value at that stage of the job by writing it to the ssis log file created when the package runs. If someone knows how to do that, please enlighten us.
It's possible to use a Derived Column transformation to write the value of a variable into a column. The problem is that it needs a source to drive it, and there's no stock data source you can use that just spits out a null row onto the pipeline.
So, either you repurpose a single-row source to drive the derived column transformation, or you do what another answer suggests, and do it with a Script source.
I did it the way you described. I already had a oledb connection manager defined so I used an OLE DB Source and used the SQL Command data access mode. I used a simple query:
select getdate() as dt
...just to get it out of the way. Now I know the date of my variable pull. Then I used a Derived Column Transform to make my package variables available and wrote it out to a flat file.
Elegant? No, but it gets the job done.
Lets say you don't want to mess with Script tasks and you don't have a database you can connect to just to issue a data source command like:
SELECT 'Some arbitrary text'
There are still several ways to use a Process task for something as simple as writing a line of text to a file. For example you can use PowerShell with an input variable built using the following expression:
"'"+REPLACE(#[User::Text],"'","''")+"' > '"+REPLACE(#[User::Filename],"'","''")+"'"
Notice I escaped the filename because single quotes are legal there. Also note I used '>' for redirecting which overwrites the file if it exists. If I wanted to append I'd use '>>'.
Initially I had trouble with this method when User::Text contained multiple lines. It turns out you need some extra EOL characters after your filename when a command spans lines. Like this:
"'"+REPLACE(#[User::Text],"'","''")+"' > '"+REPLACE(#[User::Filename],"'","''")+"'\r\n\r\n"
Using cmd.exe with echo is a bit more precarious but can also work in certain circumstances and has much less overhead.
P.S. I've noticed with some versions of PowerShell that StandardInputVariable content is ignored without this:
-Command -
in the Arguments box. A lone minus sign as a Command argument is 'magic' and documented at https://learn.microsoft.com/en-us/powershell/scripting/powershell.exe-command-line-help. I believe all versions of PowerShell accept this param so even if it's not required for your version you may want to include it since it shouldn't break anything and may keep your code from breaking if PowerShell is updated to a version that requires it.