I am trying to dynamically name an excel file after processing it for archiving purposes.
If I process Logistics.xlsx I want to save it as U:\Archive\${varDP}.xlsx
Resulting file name U:\Archive\20190709.xlsx
I have tried Get system variable to get the date, This works fine. I have created the field (DateProcessed). However, I am unable to Set variables varDP to DateProcessed.
Thank you
You cannot set and use a variable in the same transformation. If you want to use a variable you should have a job with two transformations: first transformation gets the date and sets the variable; second transformation can then use the variable.
The main reason for that is that all steps initialise at the same time. Therefore, when the variable is read by the step that is using it, it's probably not set yet.
For these cases of Variables usage and passing parameters, i've been forwarding this previous answer, it has a link to another answer of mine where i go step by step of how to pass parameters to another Transformation without 'Set Variables', and in the linked answerm i have sent a downloadable example.
Related
Enterprise Architect 13.5.
I made MDG technology extending Object metatype. I have a shape script for my stereotype working well. I need to print several predefined run-state parameters for element. Is it possible to access to run-state params within Shape ?
As Geert already commented there is no direct way to get the runstate variables from an object. You might send a feature request to Sparx. But I'm pretty sure you can't hold your breath long enough to see it in time (if at all).
So if you really need the runstate in the script the only way is to use an add-in. It's actually not too difficult to create one and Geert has a nice intro how to create it in 10 minutes. In your shape script you can print a string restult returned from an operation like
print("#addin:myAddIn,pFunc1#")
where myAddIn is the name of the registered operation and pFunc1 is a parameter you pass to it. In order to control the script flow you can use
hasproperty('addin:myAddIn,pFunc2','1')
which evaluates the returned string to match or not match the string 1.
I once got that to work with no too much hassle. But until now I never had the real need to use it somewhere in production. Know that the addin is called from the interpreted script for each shaped element on the diagram and might (dramatically) affect rendering times.
Using Pentaho PDI 8.3.0
I am unable to use a parameter in a REST call within a transformation. What I've done is:
Create a transformation and given it a parameter called PAGE_NR with default value 1
Create a job
Call the transformation with parameter PAGE_NR = 1
In the transformation, set up a GET request to a REST API.
In the URL field, setup the call like http://myurl.com/foo/bar?page=${PAGE_NR}
When I call this from either SoapUI or a browser it works but it always breaks when running the job. It does not seem to translate this parameter into the value, but instead passes it exactly like mentioned above.
I need this parameter because of calling the same URL but with different results. I don't know the amount of pages up front but take care of that logic later in said transformation.
Working on Linux btw. I have tried different variations of calling the parameter but nothing seems to work.
With the information given in the comments, I am willing to make an educated guess:
The REST Client step does not perform variable substitution on the URL if it comes from a field in the stream. What you can do is insert a Calculator step before the REST step with the operation "Variable substitution on string A" with your URL field as Field A.
This should give you the desired URL with page number.
in PDI I've got the following structure
0_Metajob
1_Load_1
1_Load_2
1_SimpleEvaluation
1_Mail
As of now
1_Load_1 and 1_Load_2 are independent of each other. The second one will run, irrespective of the success of the first one. That is okay, I want it that way!
Issue
I want to have a counter that is incremented by one every time one of the single loads fails, i.e. in my example the counter can take the values 0, 1 or 2.
What do I need it for? Customer will receive a mail at the end of the metajob. The aforementioned value determines the subject of the mail, i.e. 0=everything fine, 1=so-so, 2=load totally failed!
Why not mailing within every single the Load-Job? I do that but without attaching the log-file because it is usually non-finished. Therefore the log-file is mailed with the mail that is sent when the Metajob is finished.
Tried
"Set a variable". Thought I can simply increment it with adding a one in the value field, i.e. "${VariableName}+1". Of course, this step is implementened within a fail path of each Load-Job.
However, it didn't work.
Would anyone mind helping me? I would appreciate that!
Set Variable doesn't do calculations, you'll need a Javascript step for that.
Fortunately, variables can be also be set within the Javascript step. This bit of code should go into each of the steps you put in place of the Set Variable steps:
var i = parseInt(parent_job.getVariable("Counter"),0);
i = i + 1;
parent_job.setVariable("Counter",i);
true;
This bit of code gets the variable "Counter" from the parent job and converts it to int, since all Pentaho variables are strings. Then it increments it and sets the job variable again. The "true" at the end is to ensure that the javascript step reports success to the main job.
IMPORTANT: This works roughly as you would expect in a Job. It will NOT in a transformation!
In kettle what is output field in java script and how to use setVariable in it.I tried to set variable in it but it gave me error
The javascript step takes the input from the previous steps and can be accessed from the input field. If you want to pass the same field to the output, you need to us the output field.
Also if you want to set the variable in javascript step, you can use
setVariable("variablename","value","type");
they are two different things.
the javascript if connected in a stream , gets as input all the fields (columns)
and can manipulate them with regular javascript.
if you want a new variable that will be a part of the stream all you need to do is:
var X;
then you can write this X as output at the bottom of the step.
give it a name and use it
so if you use something like
x = fieldA + fieldB
you can use the x on the stream.
the set variables used for setting a variable in one job to use in another job
its more like global / public in programming.
if you want to learn more about it you can take my course
just click pentaho kettle tutorial there is a lesson (video) on both steps
It seems like it should be simple but as of yet I havent found a way to save the value stored in an SSIS string variable to a text file. I've looked at using the flat file destination inside of a data flow but that requires a data flow source.
Any ideas on how to do this?
Use a script task.
I just tried this. I created a File connection manager, with the connection string pointing to the file I wanted to write to. I then created a string variable containing the text to write.
I added a Script Task, specified my string variable in the Read Only Variables list, then clicked Edit Script. The script was as follows:
public void Main()
{
ConnectionManager cm = Dts.Connections["File.tmp"];
var path = cm.ConnectionString;
var textToWrite = (string)Dts.Variables["User::StringVariable"].Value;
System.IO.File.WriteAllText(path, textToWrite);
Dts.TaskResult = (int)ScriptResults.Success;
}
This worked with no problems.
Here's a little sample of some code that worked in a SQL CLR in C#. You'll need to use VB if you're on 2005 I believe. The script task also needs the read variable property set to MyVariable to make the value of your variable available to it.
// create a writer and open the file
TextWriter tw = new StreamWriter("\\\\server\\share$\\myfile.txt");
// write a line of text to the file
tw.WriteLine(Dts.Variables["MyVariable"].Value);
// close the stream
tw.Close();
All it takes is one line of code in a simple Script task. No other dependencies, such as a connection manager, are needed.
Here's what it would look like in C#:
public void Main()
{
string variableValue = Dts.Variables["TheVariable"].Value.ToString();
string outputFile = Dts.Variables["Path"].Value.ToString();
System.IO.File.WriteAllText(outputFile, variableValue);
Dts.TaskResult = (int)ScriptResults.Success;
}
Obviously the most important line here is the one containing the WriteAllText function call.
The Path variable should contain a full path + filename for the output file.
Ok, I have an answer that doesn't involve use of script task. Pick some oledb sql source you have that's simple and you have a lot of control over. Make a query that returns only one row. Then put this query in a string variable:
"select vara, ' var =: " + #[User:varIWantToSee] + "' as myvar from tablea where vara = 1"
Then in OLEDB source pick "SQL command from a variable"
For varIWantToSee make sure you initialize it with a lot characters or ssis makes a very small length for that column that it doesn't let you override. At run time varIWantToSee will get set and you can see it. Pump this all into a flat file destination and you are in business. Why do some people have to do this? Because some people need to know the value of the variables in the runtime environment, their laptop development doesn't show the variable values they need. In my case I was running this on an Azure environment that had the database accesses I needed to test. If I were microsoft I would create a task that shows the runtime variable value at that stage of the job by writing it to the ssis log file created when the package runs. If someone knows how to do that, please enlighten us.
It's possible to use a Derived Column transformation to write the value of a variable into a column. The problem is that it needs a source to drive it, and there's no stock data source you can use that just spits out a null row onto the pipeline.
So, either you repurpose a single-row source to drive the derived column transformation, or you do what another answer suggests, and do it with a Script source.
I did it the way you described. I already had a oledb connection manager defined so I used an OLE DB Source and used the SQL Command data access mode. I used a simple query:
select getdate() as dt
...just to get it out of the way. Now I know the date of my variable pull. Then I used a Derived Column Transform to make my package variables available and wrote it out to a flat file.
Elegant? No, but it gets the job done.
Lets say you don't want to mess with Script tasks and you don't have a database you can connect to just to issue a data source command like:
SELECT 'Some arbitrary text'
There are still several ways to use a Process task for something as simple as writing a line of text to a file. For example you can use PowerShell with an input variable built using the following expression:
"'"+REPLACE(#[User::Text],"'","''")+"' > '"+REPLACE(#[User::Filename],"'","''")+"'"
Notice I escaped the filename because single quotes are legal there. Also note I used '>' for redirecting which overwrites the file if it exists. If I wanted to append I'd use '>>'.
Initially I had trouble with this method when User::Text contained multiple lines. It turns out you need some extra EOL characters after your filename when a command spans lines. Like this:
"'"+REPLACE(#[User::Text],"'","''")+"' > '"+REPLACE(#[User::Filename],"'","''")+"'\r\n\r\n"
Using cmd.exe with echo is a bit more precarious but can also work in certain circumstances and has much less overhead.
P.S. I've noticed with some versions of PowerShell that StandardInputVariable content is ignored without this:
-Command -
in the Arguments box. A lone minus sign as a Command argument is 'magic' and documented at https://learn.microsoft.com/en-us/powershell/scripting/powershell.exe-command-line-help. I believe all versions of PowerShell accept this param so even if it's not required for your version you may want to include it since it shouldn't break anything and may keep your code from breaking if PowerShell is updated to a version that requires it.