am using Pentaho community edition 5.4.0 ,I explain My requirement very Simply,
1) I have my jobs and transformation in my local windows machine and i like to execute those in my client machine ,So that i installed same Pentaho community version 5.4.0 on his machine. For Remote Execution i heard about Carte.bat service,I searched the installation procedure and configuration settings for remote execution,but i didn't get a clear idea about that,Please help me a clear step by step procedure for how to run remotely in my client machine .
2) Is there possible for Schedule those jobs and transformation in Pentaho Community edition 5..4.0 ? Is it possible please explain the same.
Thanks and Regards
Dhamodharan.
Install jenkins
https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins
At least read what variables are available in Jenkins. It is pretty handy to know them.
Download PDI KETTLE from http://pentaho.com unzip in any suitable directory.
Configure executables and PDI variables as in here
How to configure Database connection for production environment in Pentaho data integration Kettle transformation
Start jenkins and login into admin panel. Create an new job,
in paragraph Build add Execute shell inside input text area add lines:
cd $WORKSPACE
kitchen.sh -file=main.kjb
Done.
There are a lot of jenkins plugins.
You can add post-build actions:
notice by email
archive publish result
.... so on
Worth to use Jenkins if it is used for some other functionalities, means it is already exists in infrastructure, otherwise carte will be enought.
Variable configured in .bashrc and .bash_profile (User should be same as used for Jenkins)
#.bashrc
export KETTLE_HOME=/opt/R1/data-integration
export KETTLE_JNDI_ROOT=$KETTLE_HOME/simple-jndi
export PATH=$PATH:$KETTLE_HOME
To force evaluate .bashrc on ssh login add to .bash_profile
#.bash_profile
if [ -f .bashrc ]; then
. ~/.bashrc
fi
Then
source .bashrc
After restart Jenkins (not from admin panel)
Related
I get an "not recognised as an internal or external command " error message when I run the following command:
C:\pdi-ce>Pan.bat /file=c:\pdi_labs\matches.ktr usa_201210.txt
pentaho 4.4.0 community edition is installed in :
C:\pdi-ce
transformation and files are saved in :
C:\pdi_labs\
Any hint to run the transformation from Pan since I am able to run it from Spoon.
regards
Generally Pentaho kettle creates another wrapper folder "data-integration" in installation directory. Check whether Pan.bat exists in the directory. If not cd to the directory which has it.
Pan.bat -file="c:\pdi_labs\matches.ktr" > usa_201210.txt
Assuming usa_201210.txt is the log file.
I would recommend starting pan.bat in a different shell and make it wait until it completes as below.
start /wait cmd "c:\pdi-ce\Pan.bat -file=c:\pdi_labs\matches.ktr > usa_201210.txt"
cd /C:/pdi-ce/
Pan.bat /file:"c:\pdi_labs\matches.ktr" /usa_201210.txt
When I try to run the external script (R Script) from kognitio console. I'm getting the below error message.
Error:external script vfork child: No such file or directory
Can someone please help me to understand what it is!
This will be because you have not replicated the script environment to all the DB nodes which are eligible to run the script.
Chapter 10 of the Kognitio Guide (downloadable from http://www.kognitio.com/forums/viewtopic.php?t=3), explains in section 10.2 how the script environment myst be identically installed in the same location on all nodes which will be used in processing, and section 10.6 explains how you can contrain this to a subset of nodes if for some reason you do not want the script environment to be on all nodes (e.g. if it has an expensive per-node licence).
You can use the wxsync tool to synchronise files across all nodes, or a remote deployment tool, such as HP's RDP, to ensure that the script environment is installed identically on all nodes.
Hello guys I'm using kettle spoon 5.0.1 and I can't find the Schedule Perspective, or any other except Data Integration Perspective, can you help me, please?
If you need scheduling in CE, you have to do it yourself. Fortunately this isn't terribly hard. Just use an external scheduler to launch transforms and jobs via Pan and Kitchen.
For example, I use pgagent since I have PostgreSQL. So I set up my schedules and launch my job as follows:
set KETTLE_PASSWORD=Encrypted (Some pwd)
"C:\Program Files\pentaho\Daily_Jobs.bat" > "C:\Program Files\pentaho\Daily_Jobs.log"
And then Daily_Jobs.bat is this:
cd /d "C:\Program Files\pentaho\"
Kitchen.bat /rep:"ETL_PROD" /job:"Daily Jobs" /dir:"/Finished" /user:Brian /level:Basic
This runs the job "Daily Jobs" in the "/Finished" directory of my PDI repository as use "Brian" with basic logging and using the password stored in KETTLE_PASSWORD. These are Windows batch files, but .sh scripts in Linux work just as well. No they don't have to be in two separate batch files (I forget why I did that originally).
While I use pgagent, any scheduler that can launch batch files/shell scripts should work.
Please note that this is an AIX related question.
I have a jenkins server running on Redhat which is running a node via SSH on an AIX server.
The commands are run non-interactively using SSH to a user on the AIX machine who has ksh as its standard shell.
The problem is that this build needs a number of environment variables, and i can't seem to get it to work.
I have tried:
Jenkins allows me to set some environment variables for the session. So i tried:
ENV="$HOME/.profile"
I tried creating a .kshrc file containing
. .profile
But none of these approaches seems to make KSH run the .profile script.
The .profile script contains the environment setup for the user i need.
How do i get an AIX implementation of KSH to run my .profile script before executing commands?
You need to specifically tell Jenkins that you want to execute them in ksh shell.
By default, Jenkins runs as sh <commands>.
Add a shebang in your shell command as first line,
#!/bin/ksh
Most shells don't source their .profile files on non-interactive sessions. A simple solution is to source the .profile yourself as part of the command you are sending.
So instead of
yourcommand1; yourcommand2
you should send
. ~/.profile; yourcommand1; yourcommand2
over ssh
UPDATE after reading the comment about Jenkins controlling the ssh command
In the case your ssh command is performed by Jenkins you should have a look at https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin, especially the 'Login profile files' paragraph.
I'd say one of these solutions is best
Set all environment variables from Jenkins using the node's configure page. Install the EnvInject plugin to do this.
Write a wrapper around the java command on the slave that sources your profile script and adjust the JavaPath (also on the node's configure page) to point to that wrapper.
The only way I know of for setting environment variables that will apply for non-interactive shells on AIX is via /etc/environment. I believe this is the correct place, but it will of course then apply to all users and all shells.
I am new to WinSCP and am attempting to create a script file that will eventually be used with SSIS to download files from an SFTP site. A lot of the literature WinSCP includes explains the file downloading or uploading portions. For the time being, I just want to create a script to test the connection first and will build from there.
So far I saved the connection in WinSCP and have the following. The below code does not seem to function at all and I am not sure where else to go as I am still reading about the scripting for WinSCP. Is there a way or can someone point me in a direction to see if I am in fact connecting via through the script?
option batch on
option confirm off
open username#address
exit
Not sure what SSIS is (sorry) but I can tell you how I'd set it up from a windows batch file if that helps:
If you are open to using a different software, consider using cygwin. It mimics a linux shell so linux users on windows have a lot of linux utilities handy. That being said, there are some commands which can run on windows straight from command prompt (and thus batchable). What you'd need to do:
1) install cygwin
2) Create a "passwordless" login (using ssh-rsa authentication). To do this start your cygwin terminal and use the commands "ssh-keygen" and "ssh-copy-id" (more on that later)
3) Now you can run "sftp" from the DOS command prompt (does not require cygwin terminal) and sftp to your account. No password required because of step 2).
A few follow up info:
What can run from dos command prompt and what must be run from cygwin terminal?
If you go to the "bin" directory of cygwin (for me it's in c:\cygwin\bin) you can see all the cygwin utilities. Anything with "exe" extension can be run from dos command prompt. If no "exe" extension, must start cygwin terminal first
How to set up ssh-rsa authentication?
You can pretty much google "ssh login without password" and pull up a lot of results. This is common for setting up login from one linux system to another. You would be using the same steps using cygwin on windows. My instructions are here:
http://geekswing.com/geek/unix/how-to-ssh-login-without-a-password-using-ssh-keygen-quick-tutorial/
Storing session settings in WinSCP GUI and trying to access them from WinSCP script running in SSIS is generally a bad idea. I believe there's no example or guide on WinSCP site that would suggest doing that.
WinSCP stores its configuration in registry in HKEY_CURRENT_USER hive. The SSIS typically runs under a dedicated system account, that have its own HKEY_CURRENT_USER hive, and won't see the GUI configuration.
For details see WinSCP FAQ about your problem:
https://winscp.net/eng/docs/faq_scheduler
The best you can do is isolate your your script from configuration by using the session URL with the open command, instead of the stored site name.
See also https://winscp.net/eng/docs/scripting#configuration
Your actual problem can be completely different though. But that's hard to guess as you have not shared any details, such as error message, log file, etc.