I have a directory in remote Linux machine where files are being archived and kept for a certain period of time. I want to delete a file from remote (Linux) machine using kettle transformation based on some condition.
If file does not exists then job should not throw any error but if file exists at remote location, then job should delete file or raise an error in case some other reason, i.e., permission issue.
Here, the file name will be retrieved as a variable from previous steps of transformation and directory path of archived files will be fixed one.
How can I achieve this in Pentaho Kettle transformation?
Make use of "Run SSH commands" utility to pass commands to your remote server.
Assuming you do a rm -f /path/file it won't error for a non-existent file.
You can capture the output and perform an error handling as well (Filter rows and trigger the course of action).
Or you can mount remote directory to machine where kettle is, and try to delete file as regular.
Using ssh, i think, non trivial. It needs a lots of experiments to find out error types, to find way to distinguish errors. It might be and error with ssh connection or error to delete file.
Related
We are executing SQL loader command from shell script on RH Linux OS. The command passes both control file and data file as command line parameters. The syntax used is given below (masked sensitive data):
sqlldr userid=$connstring control=/local/abc-1.2.3/instances/www.abc.com/apps/int/script/bin/ABC_CONTROL.ctl data=$f log=/local/abc-1.2.3/instances/www.abc.com/apps/int/script/logs/ABC.log bad=/local/abc-1.2.3/instances/www.abc.com/apps/int/script/logs/ABC.bad
The data file name is passed as dynamic variable in a FOR loop to process multiple files. The data file extension is *.app and path is /local/abc-1.2.3/instances/www.abc.com/apps/int/script/input.
We have verified that $f variable is able to correctly point to data file. Also verified file permissions. We tried changing the directory paths as well.
Still script fails with below error: **SQL Loader-500 Unable to open file (ABC_CTL.dat)**, SQL Loader-553 File not found, SQL Loader-509 System error: No such file or directory
The same script runs with exact same syntax on another server. Please suggest any solutions.
A process from a host joined a container's namepsace and is trying to write ns_last_pid file present in /proc/sys/kernel/ns_last_pid. but it is giving an error of Read only file system.
whereas i'm able to do this on the host's ns_last_pid file via this solution
unable to write /proc/sys/kernel/ns_last_pid file
How to solve this issue?
You have to run your container in --privileged mode to be able to write to /proc/sys/kernel/ns_last_pid.
A similar question has been asked before but this one is different.
I am doing a rsync of a single file on remote server and destination directory does not exist. I would like the destination directory to be created if it does not exist. I am using :: syntax which uses modules and I could not find a similar case in the forums.
Here is the syntax. remote_dir2 does not exist and i want it to be created.
rsync -avz --password-file=<file> <source-file> remote-user#remote-server::remote_dir1/remote_dir2
Note: there is a module named remote-user in /etc/rsyncd.conf in the remote server and connection and everything else works, except that source file ends up in the remote_dir1 with the name of remote_dir2
Is there any solution that is different from what mentioned below ?
I do not want to open an ssh to remote server to 'mkdir'
I do not want to use -R, --relative because directory structure names in source and destination are very different.
I also know that there is a trick mentioned
here but it does not work when you specify a module. There is no error or anything in the logs, apparently it gets ignored.
I am using the 7zip standalone .exe to unzip a file. I am using the Execute Process task for this. I have tested this over and over again on multiple machines and I know it works (at least in debug mode/visual studio). I have uploaded this package the server. I have created a job that calls said package from the Package Store. The package is not able to find the .exe no matter where I put it.
My first thought was to put the .exe on the C:\ drive, which failed. I have also failed in my attempts to place the .exe on a network location that the account the package is running under has full control over.
Basically, has anybody else had issues getting the Execute Process Task to find an executable when the package is uploaded to the server?
The error message is
Can't find 7za.exe in directory C:\7zip
I'll risk a downvote for being wrong, but I believe you have a permission issue.
You say it runs fine on other servers from BIDS, try it without BIDS. Call it from a command-line on a box that it works on.
dtexec.exe /file C:\HereComesTheUnzipper.dtsx
If that works, then repeat the step on the troublesome server. RDC into the box and try again
dtexec.exe /ser localhost /sq HereComesTheUnzipper
If that still works, then you are looking at an issue with the job. What account is the SQL Agent service running as? Is the SSIS job step running as a particular set of credentials? If so, is it a SQL Server login (which wouldn't map to anything on the physical box)? Regardless of what your answer is, the resolution will be to ensure the account has access to
7z.exe
whatever scratch area 7zip may use while unpacking files (I assume %temp%)
the output folder (C:\bin\7z.exe -e e:\data\MyThing.7z)
I am running Cloudera's distribution of Hadoop and everything is working perfectly.The hdfs contains a large number of .seq files.I need to merge the contents of all the .seq files into one large .seq file.However, the getmerge command did nothing for me.I then used cat and piped the data of some .seq files onto a local file.When i want to "put" this file into hdfs it does nothing.No error message shows up,and no file is created.
I am able to "touchz" files in the hdfs and user permissions are not a problem here.The put command simply does not work.What am I doing wrong?
Write a job that merges the all sequence files into a single one. It's just the standard mapper and reducer with only one reduce task.
if the "hadoop" commands fails silently you should have a look at it.
Just type: 'which hadoop', this will give you the location of the "hadoop" executable. It is a shell script, just edit it and add logging to see what's going on.
If the hadoop bash script fails at the beginning it is no surprise that the hadoop dfs -put command does not work.