How can I set the dynamic file path or folder directory for kettle jobs?
Please check the attached screenshot.
Goal: Read the path from a config file as a variable[so that we can change the path dynamically as per the other parameters.]
Details: Say, we want to use the /web/test directory for test environment and we want to fetch file repository from the normal path when the parameter is not test! I assume, there must be a way to keep a config/ini file from where we can read the path and use the variable inside the "File/Directory" section of pentaho.
I am gone through variable reference option but which is mainly for database configurations parameter ,some people suggested which is not good option instead of you can specified the database configuration in xml.
Please suggest any idea or solution.
Sounds like you want to set a parameter/variable in the .kettle file and reference it in the File or directory text box. Note the red dollar sign next to the box. That means this field accepts variables. Here's the wiki entry for variables:
PDI Variables
You can also read from a config file directly (from a transform) and set it dynamically with the Set variables step if you can only have one .kettle file. Also check out the Check if connected to repository (from the Repository branch) step as well and see if that will suit your needs.
If none of these suite your needs, please add detail to your question to describe exactly what you're trying to do and how you're trying to do it.
Related
I am in process of migrating Pentaho from Database repository to file repository.
I have exported the database repository into xml file and then created a file repository and imported the repository...
The first issue that I saw after importing is all my database connections are being stored in .ktr and .kjb files, This is going to be a big issue If I update a connection string like updating a password, I have more than a hundreds of sub transformations and jobs, do I have to update this in all those files?
Is there any way to ignore the password and other connection settings that is stored in the .ktr and .kjb files and instead use the repository connection or specify it in the .kettle property?
The other issue that I face is When I try to run the master job via kitchen in cmd it does not recognize the sub transformation and jobs. However when I change the Transformation root to ${Internal.Entry.Current.Directory} - the sub transformation is being recognized and processed- As I mentioned I have more than 100 sub transformation and jobs - is there any way to update this root for all jobs and transformation at once.
Kitchen.bat /file:"C:\pentaho-8-1\Dev_Repo\home\jobs\MainProcess\MasterJob.kjb" /level:Basic /logfile:"C:\pentaho-8-1\logs\my-job.txt"
This fails with error (.ktr is not a file or the repository is not defined)
However when I change the root directory to ${Internal.Entry.Current.Directory} it works!
For the database connections, you can make .kdbs in the repository and enter variables for all the properties (Host, Port, Schema, User, etc) and define them in kettle.properties or another properties file.
This works like a more convenient version of JNDI files, with one properties file per environment. You can easily inspect current values by opening the kettle properties from within the Spoon client (don't edit them or it will mess up the layout!) and you can also put kettle "encrypted" passwords in the properties file.
PDI will still save copies of the connections into all the .kjb and ktr files (and should in theory update them from .kdb or shared.xml when opening them) but since the contents are just generic variable names (${STAGING_DB_HOST} etc) you will almost never run into problems with this.
For the transformation filenames, a good text search and replace tool should fix most of your transformations in one go. Include some of the XML tag to prevent replacing too much.
While developing transformations on local I set my transformation path to the target folders that are presented in Local PC and Once testing is completed on local I am moving our transformation to server repository to schedule it from server environments but every time I require to change the path set to the server folders. I believe it can be done by creating dynamic path or creating any variable but I am unable to resolve it. Is this option available in Pentaho? if Yes, Can you please help me for setting the dynamic path?
In This answer there is a link to a described solution, and in the answer i have a sample KTR that should help.
You can also use the pentaho properties file in different environments, meaning, you can utilize the same variable in both environments, say ${path}, but in each environment this has a different value.
kettle.properties can be found in your user folder .. C:\Users\YourUser.kettle
The standard way to handle environments in Kettleis with variables.
In the home directory there is a (hidden) folder named .kettle which contains every thing that should be local : your preferences, your shared connections, your cache, and, most of all, THE kettle.property file.
You can define variables in it, like a ${myPath}. To do this, use the menu Edit/Edit the Kettle.properties and add a variable named myPath and give it for value your prefered path, with an optional description.
Then, when you see a blue diamond with a $ on the right of a field in a step window (which means almost any field you'll need), you can press Crtl+Enter in the field and choose any variable defined in your kettle.properties. Alternatively, you may type or copy/paste ${your-variable-name} in the field.
Then, when launching spoon, it will not use the hard-coded path, but the content of the variable in the kettle.properties.
And nothing prevent you from having a different kettle.properties on your dev PC and on the prod server.
While we are there, three usefull tricks.
There is a predefined ${Internal.Job.Filename.Directory} variable contaning the path of the current transformation which by used for relative path. For example, ${Internal.Job.Filename.Directory}/../myDir/myFile.ext.
If you right-click anywhere on the screen, and go to the Properties/Parameters, you may also define your variable here.
You may also redefine these variables in the Run Option window that anoys you each time you rune a transformation (yes, there was a reason).
Finally, you can send these variables from job to jobs and transfos.
Is it possible to get the actual file, or the file that gets copied from version control to a location?
This sounds confusing. Basically I have the file path of the version controlled file, but I need an actual path to the file because I need to make a cconsole command using powershell.exe. The file will look something like this
$/MyTeamProject/MyProject/Development/MyPowershellScript.ps1
Now, I am looking for a vb expression to see if I can get the actual file and make call the powershell.exe command from console. Any thoughts?
You may use VersionControlServer.GetItem(String path) to obtain a reference to the Item. Then use Item.DownloadFile() or Item.DownloadFile(String localPath) to copy the file locally. I have a variation of this that creates a shipment based on multiple changesets.
Is it possible to load values for infinispan-config.xml file from some property file so that we can get rid of hard coded values. If possible then can somebody show me the way how i load property file in infinispan-config.xml file because there is no Pre defined tag for configuration.
This is possible by setting respective system properties.
For example here is one specific Infinispan configuration file which is using this approach: https://github.com/infinispan/infinispan/blob/master/core/src/test/resources/configs/string-property-replaced.xml
and here is a test which is working with that file: https://github.com/infinispan/infinispan/blob/master/core/src/test/java/org/infinispan/config/StringPropertyReplacementTest.java
This looks to be the most straightforward way how to achieve this.
The last thing which needs to be done is to simply read all lines in your configuration file and put them correctly to system properties.
I am using the Intellij 11.1.5. We are a large team, and have a pretty complex project setup. so we've made a template and when someone needs a new project set up, we just clone it and she is pretty much ready to go. One other thing i would like to automate is the creation of run configurations. One such configuration starts a custom bat file that requires a parameter representing a path that is user specific. I wanted to know if can store that value as a path variable specific to each project. Maybe somewhere in the .idea folder in my project. I know that Intellij stores it in its .IntelliJIdea11\config\options\path.macros.xml file, but is there a way to tweak that?
Any other idea that would allow me to locally store a parameter passed to the run config script would be usefull.
Thanks
I'm afraid you can't do it in IDEA, but you can use some environment variable directly in the .bat file instead of using the parameter (or rewrite the batch script to detect this value automatically, if possible). Instruct your users to define this environment variable.
IDEA Path variables are global and cannot be made project specific.