It seems like it should be simple but as of yet I havent found a way to save the value stored in an SSIS string variable to a text file. I've looked at using the flat file destination inside of a data flow but that requires a data flow source.
Any ideas on how to do this?
Use a script task.
I just tried this. I created a File connection manager, with the connection string pointing to the file I wanted to write to. I then created a string variable containing the text to write.
I added a Script Task, specified my string variable in the Read Only Variables list, then clicked Edit Script. The script was as follows:
public void Main()
{
ConnectionManager cm = Dts.Connections["File.tmp"];
var path = cm.ConnectionString;
var textToWrite = (string)Dts.Variables["User::StringVariable"].Value;
System.IO.File.WriteAllText(path, textToWrite);
Dts.TaskResult = (int)ScriptResults.Success;
}
This worked with no problems.
Here's a little sample of some code that worked in a SQL CLR in C#. You'll need to use VB if you're on 2005 I believe. The script task also needs the read variable property set to MyVariable to make the value of your variable available to it.
// create a writer and open the file
TextWriter tw = new StreamWriter("\\\\server\\share$\\myfile.txt");
// write a line of text to the file
tw.WriteLine(Dts.Variables["MyVariable"].Value);
// close the stream
tw.Close();
All it takes is one line of code in a simple Script task. No other dependencies, such as a connection manager, are needed.
Here's what it would look like in C#:
public void Main()
{
string variableValue = Dts.Variables["TheVariable"].Value.ToString();
string outputFile = Dts.Variables["Path"].Value.ToString();
System.IO.File.WriteAllText(outputFile, variableValue);
Dts.TaskResult = (int)ScriptResults.Success;
}
Obviously the most important line here is the one containing the WriteAllText function call.
The Path variable should contain a full path + filename for the output file.
Ok, I have an answer that doesn't involve use of script task. Pick some oledb sql source you have that's simple and you have a lot of control over. Make a query that returns only one row. Then put this query in a string variable:
"select vara, ' var =: " + #[User:varIWantToSee] + "' as myvar from tablea where vara = 1"
Then in OLEDB source pick "SQL command from a variable"
For varIWantToSee make sure you initialize it with a lot characters or ssis makes a very small length for that column that it doesn't let you override. At run time varIWantToSee will get set and you can see it. Pump this all into a flat file destination and you are in business. Why do some people have to do this? Because some people need to know the value of the variables in the runtime environment, their laptop development doesn't show the variable values they need. In my case I was running this on an Azure environment that had the database accesses I needed to test. If I were microsoft I would create a task that shows the runtime variable value at that stage of the job by writing it to the ssis log file created when the package runs. If someone knows how to do that, please enlighten us.
It's possible to use a Derived Column transformation to write the value of a variable into a column. The problem is that it needs a source to drive it, and there's no stock data source you can use that just spits out a null row onto the pipeline.
So, either you repurpose a single-row source to drive the derived column transformation, or you do what another answer suggests, and do it with a Script source.
I did it the way you described. I already had a oledb connection manager defined so I used an OLE DB Source and used the SQL Command data access mode. I used a simple query:
select getdate() as dt
...just to get it out of the way. Now I know the date of my variable pull. Then I used a Derived Column Transform to make my package variables available and wrote it out to a flat file.
Elegant? No, but it gets the job done.
Lets say you don't want to mess with Script tasks and you don't have a database you can connect to just to issue a data source command like:
SELECT 'Some arbitrary text'
There are still several ways to use a Process task for something as simple as writing a line of text to a file. For example you can use PowerShell with an input variable built using the following expression:
"'"+REPLACE(#[User::Text],"'","''")+"' > '"+REPLACE(#[User::Filename],"'","''")+"'"
Notice I escaped the filename because single quotes are legal there. Also note I used '>' for redirecting which overwrites the file if it exists. If I wanted to append I'd use '>>'.
Initially I had trouble with this method when User::Text contained multiple lines. It turns out you need some extra EOL characters after your filename when a command spans lines. Like this:
"'"+REPLACE(#[User::Text],"'","''")+"' > '"+REPLACE(#[User::Filename],"'","''")+"'\r\n\r\n"
Using cmd.exe with echo is a bit more precarious but can also work in certain circumstances and has much less overhead.
P.S. I've noticed with some versions of PowerShell that StandardInputVariable content is ignored without this:
-Command -
in the Arguments box. A lone minus sign as a Command argument is 'magic' and documented at https://learn.microsoft.com/en-us/powershell/scripting/powershell.exe-command-line-help. I believe all versions of PowerShell accept this param so even if it's not required for your version you may want to include it since it shouldn't break anything and may keep your code from breaking if PowerShell is updated to a version that requires it.
Related
I am trying to dynamically name an excel file after processing it for archiving purposes.
If I process Logistics.xlsx I want to save it as U:\Archive\${varDP}.xlsx
Resulting file name U:\Archive\20190709.xlsx
I have tried Get system variable to get the date, This works fine. I have created the field (DateProcessed). However, I am unable to Set variables varDP to DateProcessed.
Thank you
You cannot set and use a variable in the same transformation. If you want to use a variable you should have a job with two transformations: first transformation gets the date and sets the variable; second transformation can then use the variable.
The main reason for that is that all steps initialise at the same time. Therefore, when the variable is read by the step that is using it, it's probably not set yet.
For these cases of Variables usage and passing parameters, i've been forwarding this previous answer, it has a link to another answer of mine where i go step by step of how to pass parameters to another Transformation without 'Set Variables', and in the linked answerm i have sent a downloadable example.
Trying to use this:
substring(#[User::v_Filename],37,3)
However, it seems substring can only handle 20 characaters ?
The file name looks like this:
D:\Projects\OTS\MYSSA Dashboard\Data\ATL_20150725Text.csv
All I want is the ATL Portion
But when the ssis moves to the next file, it may change to NYC or DAL, there are about 26 files to be processed all from different regions.
Test the substring without your file variable, that uses the substring function in an expression.
Example - create a variable with the expression, then click "evaluate":
substring("ABCDEFGHIJKLMNOPQRSTUVWXYZ", 25, 2)
It will work fine.
Given that the substring function works fine, there must be something wrong with your [User::v_Filename] variable. Are you sure it is being set correctly? Perhaps you should try running BIDS with the debugger on and a breakpoint set to right after you assign the filename, and verify that it indeed is being set correctly.
I have a file variable in d3 pick basic and I am trying to figure out what file it corresponds to.
I tried the obvious thing which was to say:
print f *suppose the file variable's name is f in this case
but that didn't work, because:
SELECTION: 58[B34] in program "FILEPRINTER", Line 7: File variable used
where string expression expected.
I also tried things like:
list f *didn't compile
execute list dict f *same error
execute list f *same error
but those also did not work.
In case any one is wondering, the reason I am trying to do this in the first place is that there is a global variable that is passed up and down in the code base I am working with, but I can't find where the global variable gets its value from.
That file pointer variable is called a "file descriptor". You can't get any information from it.
You can use the file-of-files to log Write events, and after a Write is performed by the code, check to see what file was updated. The details for doing this would be a bit cumbersome. You really should rely on the Value-Add Reseller or contract with competent assistance for this.
If this is not a live end-user system, you can also modify an item getting written with some very unique text like "WHAT!FILE!IS!THIS?". Then you can do a Search-System command to search the entire account (or system) to find that text. See docs for proper use of that command.
This is probably the best option... Inject the following:
IF #USER = "CRISZ" THEN ; * substitute your user ID
READU FOO FROM F,"BLAH" ELSE
DEBUG
RELEASE F,"BLAH"
END
END
That code will stop only for one person - for everyone else it will flow as normal. When it does stop, use the LIST-LOCKS command to see which file has a read lock for item "BLAH". That's your file! Don't forget to remove and recompile the code. Note that recompiling code while users are actively using it results in aborts. It's best to do this kind of thing after hours or on a test system.
If you can't modify the code like that, diagnostics like this can be difficult. If the above suggestions don't help, I think this challenge might be beyond your personal level of experience yet and recommend you get some help.
If suggestion here Does help, please flag this as the answer. :)
In kettle what is output field in java script and how to use setVariable in it.I tried to set variable in it but it gave me error
The javascript step takes the input from the previous steps and can be accessed from the input field. If you want to pass the same field to the output, you need to us the output field.
Also if you want to set the variable in javascript step, you can use
setVariable("variablename","value","type");
they are two different things.
the javascript if connected in a stream , gets as input all the fields (columns)
and can manipulate them with regular javascript.
if you want a new variable that will be a part of the stream all you need to do is:
var X;
then you can write this X as output at the bottom of the step.
give it a name and use it
so if you use something like
x = fieldA + fieldB
you can use the x on the stream.
the set variables used for setting a variable in one job to use in another job
its more like global / public in programming.
if you want to learn more about it you can take my course
just click pentaho kettle tutorial there is a lesson (video) on both steps
I have an email body stored as a string in a database, something like this:
This is an email body containing lots of different variables. Dear #name#, <br/> Please contact #representativeName# for further details.
I pull this field from the database using a stored proc, and then I want to evaluate it on the coldfusion side, so that instead of "#name#", it will insert the value of the name variable.
I've tried using evaluate, but that only seems to work if there's just a variable name. It throws an error because of the other text.
(I can't just use placeholders and a find/replace like this - Resolving variables inside a Coldfusion string, because the whole point of storing this in a database is that the variables used to build the string are dynamic. For example, in one case the name field can be called "name" and in another it could be "firstName", etc.)
I would loop over each #variableName# reference and replace it with the evaluated version.
A regex will be able to find them all and then a loop to go over them all and just evaluate them one by one.
You need to write it to a file and CFINCLUDE it. This will incur a compilation overhead, but that's unavoidable.
Can you not save the code to the file system and just store a reference to where it is in the DB? That way it'll only get recompiled when it changes, rather than every time you come to use it?
<!--- pseudo code --->
<cfquery name="q">
SELECT fileContent // [etc]
</cfquery>
<cfset fileWrite(expandPath("/path/to/file/to/write/code.cfm"), q.fileContent)>
<cfinclude template="/path/to/file/to/write/code.cfm">
<cfset fileDelete(expandPath("/path/to/file/to/write/code.cfm"))>
That's the basic idea: get the code, write the code, include the code, delete the code. Although you'll want to make sure the file that gets created doesn't collide with any other file (use a UUID as the file name, or something, as per someone else's suggestion).
You're also gonna want to load test this. I doubt it'll perform very well. Others have suggested using the virtual file system, but I am not so sure there'll be much of a gain there: it's the compilation process that takes the time, not the actual file ops. But it's worth investigating.
Are you using ColdFusion 9 or Railo? If yes, writing to and including in-memory files may be quick and simple solution. Just generate file name with something like CreateUUID() to avoid collisions.
So basically, after researching and reading the answers, it seems that these are my options:
Have separate fields in the table for each variable and evaluate them individually. e.g. nameVariable, reprNameVariable, and then I can build the body with code like this:
This is an email body containing lots of different variables. Dear #evaluate(nameVariable)#, <br/> Please contact #evaluate(reprNameVariable)# for further details.
Have an "emailBody" field with all the text and applicable field names, and write it to a temporary file and cfinclude it. (As suggested by Adam Cameron, and I think that's what Sergii's getting at)
Have an "emailBody" field and write code to loop over it, find all coldfusion variables, and replace them with their "evaluate"d version. (As suggested by Dale Fraser)
Have small template files, one for each report type, with the email body, and have a field "emailBodyTemplate" that indicates which template to include. (As suggested by Adam Cameron)
Now I just have to decide which to use :) When I do, I'll accept the answer of the person who suggested that method (unless it's one that wasn't suggested, in which case I'll probably accept this, or if someone comes up with another method that makes more sense)
It's been a while since you posted this - but it's EXACTLY what I do. I found your question while looking for something else.
I've simply created my own simple syntax for variables when I write my emails into the database:
Hello ~FirstName~ ~LastName~,
Then, in my sending cfm file, I pull the email text from the database, and save it to a variable:
<cfset EmailBody = mydatabasequery.HTMLBody>
Then I quickly strip my own syntax with my variables (from another query called RecipientList):
<cfset EmailBody = ReplaceNoCase(EmailBody, "~FirstName~", "#RecipientList.First#", "ALL")>
<cfset EmailBody = ReplaceNoCase(EmailBody, "~LastName~", "#RecipientList.Last#", "ALL")>
Then I simply send my email:
<cfmail ....>#EmailBody#</cfmail>
I hope you manage to see this. If you control the authoring of the email messages, which I suspect you do, this should work well.
Russell