While developing transformations on local I set my transformation path to the target folders that are presented in Local PC and Once testing is completed on local I am moving our transformation to server repository to schedule it from server environments but every time I require to change the path set to the server folders. I believe it can be done by creating dynamic path or creating any variable but I am unable to resolve it. Is this option available in Pentaho? if Yes, Can you please help me for setting the dynamic path?
In This answer there is a link to a described solution, and in the answer i have a sample KTR that should help.
You can also use the pentaho properties file in different environments, meaning, you can utilize the same variable in both environments, say ${path}, but in each environment this has a different value.
kettle.properties can be found in your user folder .. C:\Users\YourUser.kettle
The standard way to handle environments in Kettleis with variables.
In the home directory there is a (hidden) folder named .kettle which contains every thing that should be local : your preferences, your shared connections, your cache, and, most of all, THE kettle.property file.
You can define variables in it, like a ${myPath}. To do this, use the menu Edit/Edit the Kettle.properties and add a variable named myPath and give it for value your prefered path, with an optional description.
Then, when you see a blue diamond with a $ on the right of a field in a step window (which means almost any field you'll need), you can press Crtl+Enter in the field and choose any variable defined in your kettle.properties. Alternatively, you may type or copy/paste ${your-variable-name} in the field.
Then, when launching spoon, it will not use the hard-coded path, but the content of the variable in the kettle.properties.
And nothing prevent you from having a different kettle.properties on your dev PC and on the prod server.
While we are there, three usefull tricks.
There is a predefined ${Internal.Job.Filename.Directory} variable contaning the path of the current transformation which by used for relative path. For example, ${Internal.Job.Filename.Directory}/../myDir/myFile.ext.
If you right-click anywhere on the screen, and go to the Properties/Parameters, you may also define your variable here.
You may also redefine these variables in the Run Option window that anoys you each time you rune a transformation (yes, there was a reason).
Finally, you can send these variables from job to jobs and transfos.
Related
I hope this message finds everyone well!
I'm stucked on a situation on Pentaho PDI Tool and I'm looking for an answer (or at least a light in the end of the cave) to solve it!
I have to import, every month, a bunch of xls's files of differents clients. Every file has a different name (witch is given aleatory) and this files are on a folder named with the name of the client. However, I use the same process for all clients and situations.
Is there a way to pass the name of the directory as a variable and change this variable on every process? How can I read this files on differents paths?
The answer you're looking for requires a flow with variables as you stated. In a JOB you will start with a KTR with the client's name and their respective folder. In the same JOB you are going to pass these results and use them as variables, to another JOB if needed, or to a KTR, and you are going to use the options "Copy previous results to parameters" and "Execute for every input row" (Advanced Tab), and in the parameters tab you will name the variables and stream column name (where the data is coming from in the previous KTR, ie.: Clients name and directory).
If you have trouble with creating this flow i can spare some more time and share a sample if you need.
EDIT:
Sample Here
You have an example of this in the sample directory which is shipped with your PDI distribution.
Your case is covered by the samples/jobs/run_all.
Intro:
Every QVW has their own variables that manually defined where to store/load the records they pulled in their SQL scripts.
ex:
SET vLoadPath = \\dev_server\Extract QVD\;
SET vStorePath = \\dev_server\Transform QVD\;
Scenario:
As Qlikview Admin, we promote tested QVWs from DEV to PROD environment.
Variable path is always hardcoded in DEV (see example code above)
When we promote the QVW in PROD, we will change the defined path variable to have a 'prod_server'.
Question:
Is there a way to make the sample variables above, be a Global variable? These variables will change its value when we promote the QVW from DEV to PROD environment.
Reason:
I think the manual changing of hardcoded path variable is a poor practice.
What if the server should change? Or the subdirectory?
Then you’ll have to go back into every single QVW script and change the hardcoded subdirectory path.
Not only is that a high cost, it also introduces the possibility of error.
The way I've generally handled such things is via a master config file. You $include (or probably $must_include) this file in all QVWs and it sets your environment variables. Make the path to the config file relative so when you deploy between environments the config file in the new environment is picked up.
How can I set the dynamic file path or folder directory for kettle jobs?
Please check the attached screenshot.
Goal: Read the path from a config file as a variable[so that we can change the path dynamically as per the other parameters.]
Details: Say, we want to use the /web/test directory for test environment and we want to fetch file repository from the normal path when the parameter is not test! I assume, there must be a way to keep a config/ini file from where we can read the path and use the variable inside the "File/Directory" section of pentaho.
I am gone through variable reference option but which is mainly for database configurations parameter ,some people suggested which is not good option instead of you can specified the database configuration in xml.
Please suggest any idea or solution.
Sounds like you want to set a parameter/variable in the .kettle file and reference it in the File or directory text box. Note the red dollar sign next to the box. That means this field accepts variables. Here's the wiki entry for variables:
PDI Variables
You can also read from a config file directly (from a transform) and set it dynamically with the Set variables step if you can only have one .kettle file. Also check out the Check if connected to repository (from the Repository branch) step as well and see if that will suit your needs.
If none of these suite your needs, please add detail to your question to describe exactly what you're trying to do and how you're trying to do it.
I am using the Intellij 11.1.5. We are a large team, and have a pretty complex project setup. so we've made a template and when someone needs a new project set up, we just clone it and she is pretty much ready to go. One other thing i would like to automate is the creation of run configurations. One such configuration starts a custom bat file that requires a parameter representing a path that is user specific. I wanted to know if can store that value as a path variable specific to each project. Maybe somewhere in the .idea folder in my project. I know that Intellij stores it in its .IntelliJIdea11\config\options\path.macros.xml file, but is there a way to tweak that?
Any other idea that would allow me to locally store a parameter passed to the run config script would be usefull.
Thanks
I'm afraid you can't do it in IDEA, but you can use some environment variable directly in the .bat file instead of using the parameter (or rewrite the batch script to detect this value automatically, if possible). Instruct your users to define this environment variable.
IDEA Path variables are global and cannot be made project specific.
We have a standard MSBuild project file that is used for our different deployment stages (pre-stage, stage, live, etc). Since each deployment stage is performed on a different server we introduced a server parameter called $SourceDatabaseServer and used this extensively in each of the targets inside the project file. Note: This database server name could be different from the server name on which the build is run.
To assist us with the customization of this parameter, we created a response file for each deployment stage and subsequently defined a value for this parameter in the response file, e.g. /p:SourceDatabaseServer=SRC_DB_NAME.
This worked fine, until we created a new deployment stage in which this value had to be the current computer name. So we thought by using the $(COMPUTERNAME) reserved property in the response file (/p:SourceDatabaseServer=$(COMPUTERNAME)), this would do the trick, but it seems like this value is interpreted literally by MSBuild, and we consequently get an error that server $(ComputerName) could not be found.
Interestingly, when the $(COMPUTERNAME) property is used directly in the proj file it works, but as stated above, we do not necessarily want to use the computer name in all the cases.
Is there a way to still use the $(COMPUTERNAME) property in the response file and get MSBuild to interpret this correctly?
What if you use %COMPUTERNAME%?
$(VAR) is the syntax for variable expansion when you're "inside" the MSBuild system, but coming from the outside, I believe you'd have to use the shell environment variable expansion syntax, %VAR%.