Hello guys I'm using kettle spoon 5.0.1 and I can't find the Schedule Perspective, or any other except Data Integration Perspective, can you help me, please?
If you need scheduling in CE, you have to do it yourself. Fortunately this isn't terribly hard. Just use an external scheduler to launch transforms and jobs via Pan and Kitchen.
For example, I use pgagent since I have PostgreSQL. So I set up my schedules and launch my job as follows:
set KETTLE_PASSWORD=Encrypted (Some pwd)
"C:\Program Files\pentaho\Daily_Jobs.bat" > "C:\Program Files\pentaho\Daily_Jobs.log"
And then Daily_Jobs.bat is this:
cd /d "C:\Program Files\pentaho\"
Kitchen.bat /rep:"ETL_PROD" /job:"Daily Jobs" /dir:"/Finished" /user:Brian /level:Basic
This runs the job "Daily Jobs" in the "/Finished" directory of my PDI repository as use "Brian" with basic logging and using the password stored in KETTLE_PASSWORD. These are Windows batch files, but .sh scripts in Linux work just as well. No they don't have to be in two separate batch files (I forget why I did that originally).
While I use pgagent, any scheduler that can launch batch files/shell scripts should work.
Related
I am trying to run a .sql script on a schedule. I have created a batch file to run the script. The script runs fine in sql server management studio and also when I run the batch file content through cmd.
Contents of the batch file:
sqlcmd -S omfmesql -U OMESRV -P orat -i "\\pvsrv-
fsr14\data\Projects\Stat_Table_Creation_unique.sql"
The sql script is supposed to update a stat table. When I run it though cmd and refresh the stat table, the numbers are updated. But when I run this batch file through Task Scheduler, the only action that seems to be performed is running C:\Windows\SYSTEM32\cmd.exe
The task is stated to be completed successfully but the sql query is just not run.
I am not too experienced with Task Scheduler. Any help here would be very much appreciated. Thanks!
Note: I am not intending to use SQL Server Agent
If you have not done so, you need to set the location in Task Scheduler (TS). In at least some versions of TS, this can only be done when you create a basic task, not from the more general "Create Task..." option. Ensure that all the paths in the batch file are absolute or are based in this location.
I get an "not recognised as an internal or external command " error message when I run the following command:
C:\pdi-ce>Pan.bat /file=c:\pdi_labs\matches.ktr usa_201210.txt
pentaho 4.4.0 community edition is installed in :
C:\pdi-ce
transformation and files are saved in :
C:\pdi_labs\
Any hint to run the transformation from Pan since I am able to run it from Spoon.
regards
Generally Pentaho kettle creates another wrapper folder "data-integration" in installation directory. Check whether Pan.bat exists in the directory. If not cd to the directory which has it.
Pan.bat -file="c:\pdi_labs\matches.ktr" > usa_201210.txt
Assuming usa_201210.txt is the log file.
I would recommend starting pan.bat in a different shell and make it wait until it completes as below.
start /wait cmd "c:\pdi-ce\Pan.bat -file=c:\pdi_labs\matches.ktr > usa_201210.txt"
cd /C:/pdi-ce/
Pan.bat /file:"c:\pdi_labs\matches.ktr" /usa_201210.txt
am using Pentaho community edition 5.4.0 ,I explain My requirement very Simply,
1) I have my jobs and transformation in my local windows machine and i like to execute those in my client machine ,So that i installed same Pentaho community version 5.4.0 on his machine. For Remote Execution i heard about Carte.bat service,I searched the installation procedure and configuration settings for remote execution,but i didn't get a clear idea about that,Please help me a clear step by step procedure for how to run remotely in my client machine .
2) Is there possible for Schedule those jobs and transformation in Pentaho Community edition 5..4.0 ? Is it possible please explain the same.
Thanks and Regards
Dhamodharan.
Install jenkins
https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins
At least read what variables are available in Jenkins. It is pretty handy to know them.
Download PDI KETTLE from http://pentaho.com unzip in any suitable directory.
Configure executables and PDI variables as in here
How to configure Database connection for production environment in Pentaho data integration Kettle transformation
Start jenkins and login into admin panel. Create an new job,
in paragraph Build add Execute shell inside input text area add lines:
cd $WORKSPACE
kitchen.sh -file=main.kjb
Done.
There are a lot of jenkins plugins.
You can add post-build actions:
notice by email
archive publish result
.... so on
Worth to use Jenkins if it is used for some other functionalities, means it is already exists in infrastructure, otherwise carte will be enought.
Variable configured in .bashrc and .bash_profile (User should be same as used for Jenkins)
#.bashrc
export KETTLE_HOME=/opt/R1/data-integration
export KETTLE_JNDI_ROOT=$KETTLE_HOME/simple-jndi
export PATH=$PATH:$KETTLE_HOME
To force evaluate .bashrc on ssh login add to .bash_profile
#.bash_profile
if [ -f .bashrc ]; then
. ~/.bashrc
fi
Then
source .bashrc
After restart Jenkins (not from admin panel)
I have an application where I manually go in and back up certain files(1), and manually copy those files via WinSCP(2).
I would like to be able to use two scripts/ cron jobs to automate these tasks.
Here are the steps/commands I use to back up the files:
sudo as admin
cd to directory
export PATH=$PATH:/<filePath>/
./export-<appData.sh Backups/ /<filePath>/
Once that is complete, I will login to WINSCP and look in the directory where the data is backed up. I copy them over to the Backups shared directory manually.
Any help with creating a script/cron job to automate this process would be greatly appreciated.
Thank You!
You can create a shared drive in windows that can be accessible through sambha in linux ; then make a script that can copy from your linux server to windows server.
I hope this will be useful.
I'm looking for some help/direction and am generally baffled as this must be a common issue that people face (i.e. there should be an straight forward solution!).
I'm deploying a web site remotely using TeamCity, MSBuild and WebDeploy/MSDeploy. I also wish to run some scripts on the remote server that I'm deploying to. ... How do people generally do this?! I just want to automate some everyday tasks remotely on ther server e.g. backing up files or similar.
Is it a case of:
Using MSDeploy to remotely run some scripts like here, but then how do I transfer the scripts to the remote server as part of the automated deploy??
Starting some sort of PowerShell session and remoting into the server this way?
Some sort of ssh-ing or another way?
FYI these are the params I use for MSBuild and MSDeploy to the remote server (they work fine):
/p:Configuration=TeamCityLiveRelease
/p:OutputPath=bin
/p:DeployOnBuild=True
/p:DeployTarget=MSDeployPublish
/p:ContinueOnError="false"
/p:MSDeployPublishMethod=WMSVC
/p:MsDeployServiceUrl=https://MY.SITE:PORT/msdeploy.axd
/p:DeployIisAppPath=my.iis.site.here
/p:AllowUntrustedCertificate=True
/p:Username=FOO
/p:Password=baaa
Ideally I'd like to write a nice command line script and do something like (about as pseudo as it comes sorry!):
Log in to "http://some.server.ip" -username "user" -password "something"
run script "echo this is a remote script running"
I think psexec is the tool you're looking for: http://technet.microsoft.com/en-us/sysinternals/bb897553
In case anyone is interested: it is possible to use MSDeploy to carry out scripts on a remote computer (a script called MYSCRIPT.bat here). If you have an admin account on the remote server then it can be done using the following arguments:
"c:\program files\iis\microsoft web deploy\msdeploy.exe"
-verb:sync
-source:runCommand="C:\MYSCRIPT.bat",
waitInterval=5000,waitAttempts=1
-dest:auto,
computerName=https://IP.ADDRESS.HERE:PORT/msdeploy.axd?site=IIS.SITE.NAME.HERE,userName=MYUSERNAME,password=MYPASSWORD
-verbose
-allowUntrusted