As the title says, I'd like to know how to create a folder with current date(dd/mm/yyyy) (on Job). I've seen two answers but they seemed quite long, for example this one:
pentaho create archive folder with MM-YYYY
Anyone has an idea? Thanks :)
That answer is also from 2014, better and newer versions of PDI are out, and this answer is overcomplicating the task.
Though you do need to create the folder name in a KTR, you can just use the 'Create folder' Job step. I'm pretty sure you cannot set AND use a variable in the same JOB/KTR that you have set it to begin with. So you need a KTR that generates a Date and sends it as string to a JOB.
Your KTR should like this, to create a String date with the format YYYY-MM-DD
And your Job like this
Also remenber, you can't create folders with '/', so you have to use another pattern.
This is a very simplistic approach, there are other ways to automate folder creation through use of variables.
EDIT: Corrected the JOB image, the first KTR needs to actually run for the variable to be set, sorry for the confusion.
Related
apologies if I've phrased this terribly. I only started using SSIS today.
I've written a FOREACH which loops through all the files in a folder, and updates my table f_actuals together with the filename without the extension - this filename is a combination of a PeriodKey and Business Unit. It works well.
However, this is intended to be a daily upload from our system for the entire month for each business unit (so the month-to-date refreshes daily until we close that period), so what I really need is the FOREACH to include something which does the following: -
Checks the filenames due for import in the designated folder against the filenames already in the f_actuals table
Removes all the matches from the f_actuals table
Continues with the FOREACH I've already built
I know this is probably a massively inefficient way to do this (preference would be daily incremental uploads), but the files need to be month-to-date, as our system cannot provide anything else easily.
Hope this makes sense.
Any help greatly appreciated.
You can use an Execute SQL Task within the For Each Loop to do this.
You can either use an SQL statement:
DELETE
FROM f_actuals
WHERE filename = ?
Or perhaps a stored procedure (accepting your filename as a parameter and doing the same thing as the statement above), e.g.:
EXEC DeleteFromActuals ?
For each filename in your loop, you would store this in a variable, and pass the variable as a parameter in the Execute SQL Task (this is what the ? is).
To map the parameter in the Execute SQL Task, go to 'Parameter Mapping', and add a new parameter. Select the variable containing the filename from the dropdown list, choose a data type of VARCHAR, and set the 'Parameter Name' to 0. The 'Direction' should be 'Input', which is the default.
I'm trying to use oracle external tables to load flat files into a database but I'm having a bit of an issue with the location clause. The files we receive are appended with several pieces of information including the date so I was hoping to use wildcards in the location clause but it doesn't look like I'm able to.
I think I'm right in assuming I'm unable to use wildcards, does anyone have a suggestion on how I can accomplish this without writing large amounts of code per external table?
Current thoughts:
The only way I can think of doing it at the moment is to have a shell watcher script and parameter table. User can specify: input directory, file mask, external table etc. Then when a file is found in the directory, the shell script generates a list of files found with the file mask. For each file found issue a alter table command to change the location on the given external table to that file and launch the rest of the pl/sql associated with that file. This can be repeated for each file found with the file mask. I guess the benefit to this is I could also add the date to the end of the log and bad files after each run.
I'll post the solution I went with in the end which appears to be the only way.
I have a file watcher than looks for files in a given input dir with a certain file mask. The lookup table also includes the name of the external table. I then simply issue an alter table on the external table with the list of new file names.
For me this wasn't much of an issue as I'm already using shell for most of the file watching and file manipulation. Hopefully this saves someone searching for ages for a solution.
Is there any way where I can use Batch files to get backup of the selected scripts from the SQL database...?
Say - I have one stored procedure, one function and one view in a folder.
sp1.sql
vie1.sql
fn1.sql
Before run the batch file I want to take the backup of these files.
Kindly note: I do not want to take entire database backup. Just the provided scripts alone.
Help me to achieve this one pls...
The specific answer depends entirely on the flavor of your database engine. But the general answer is you need to SELECT the definition from your database's data catalog (meta data). The function and procedure definition will probably come out intact. But the view definition may come out as just the SELECT statement - you might have to prefix it with the CREATE VIEW XXXXXXX AS part.
Im new to the pentaho suite and its automation functionality. i have files that come in on a daily basis and two columns need to be put in place. I have figured out how to add the columns but now i am stuck on the automation side of things. The filename is constant but it has a datestamp at the end. EG: LEAVER_REPORT_NEW_20110623.csv. The file will always be in the same directory. How do i go about using Pentaho data integration to solve this issue? ive tried get files but that doesnt seem to work.
create a variable in a previous transform which contains 20110623 (easy with a get system info step to get the date, and then a select values step to format to string, then a set variables step)
then change the filename of the textfile input to use:
LEAVER_REPORT_NEW_${variablename}.csv
I am starting to write an installer which will use the SqlScript-element.
That takes a reference to the Binary-table what script to run.
I would like to dynamically generate the script during the installation.
I can see three possibilities:
Somehow to get SqlScript to read it data from a file rather then a Binary entry.
Inject my generated script into the Binary table
Using SqlString
Which will cause the need to place some rather long strings into Properties, but I guess that shouldn't really be a prolem.
Any advice?
Regards
Leif
(My reason, should anyone be interested is that the database should have a job set up, that calls on an installed exe-file. I prefer to create the job using sqlscript. And the path of that file is not known until InstallDir has been choosen.)
The way this is typically handled is to have the static stuff in SqlScript and use SqlString (which can contain formatted Properties) to execute the dynamic stuff. You can interleave the two with careful use of the Sequence attribute.