New to PDI here. Need to output data from a view in a postgresql database to a file daily. The output file will be like xxxx_20160427.txt, so need to append the dynamic date in the file name. How to do it?
EDIT-----------------
I was not clear here by asking how to add dynamic date, I was trying to add not just date but optional other parts to the file name. E.g adding a serial no (01) at the end: xxxx_2016042701.txt etc. So my real question is how to make a dynamic file name? In other ETL tool e.g. SSIS it will be a simple expression. Not sure how it is done in PDI?
In your Text file output step, simply check "Include date in filename?" under the files tab.
You can create a dynamic filename variable with a Modified Java Script value STEP.
and then in the Text File Output STEP click on "Accept file name from field", and select your variable declared from previous step (filename_var on this example).
Related
I need to archive the txt file using Pentaho PDI by giving it a dynamic timestamp and append the variable to the output filename. I used get system info which automatically assigns variable as well as value. So my job was Start__ get system info___zip file. In the zip file component, I tried called the variable while giving the output filename along with ${Variable} but the output filename is not coming properly. It should be off filename__timestamp__variable. Can someone please help me with this?
continuation to this question
Pentaho Spoon - Output to multiple files based on field content
how to do it for "xml output" since "xml output" doesn't have
"accept filename from the field" input in its form
Since the step doesn't accept the filename from previous fields, you can use variables to set the value.
It's a longer path, but at least it allows you to resolve it:
You need to create a transformation to set the filenames in the result (using the "Copy rows to result" step)
You need to create another transformation with "Execute for every input row" option marked. This transformation has to read from the result ("Get rows from result" step) and set the values with "Set Variables" step.
Finally, a third transformation should create the xml file, using the previously created variable as filename (variables are called using this structure: ${variable_name}).
You need to connect those 3 transformations with a job.
I'm attempting to create a process to import data. I created the entire process and it works, but I'm having trouble creating the variable to find the file name of the csv i want to import automatically. Each time a new csv is uploaded to me it has a timestamp on it. I want to be able to grab that file no matter what the name is and do work to it.
So for example this week the file name would be
filename_4-14-2014.csv
And next week
filename_4_21_2014.csv
And so on into eternity. . .
Is there a way to create a variable that picks up the full file name even though its changing?
After doing some poking around, I've discovered the following...
You can use a file system task to perform the copy operation I was referring to. You can set the input file and the output file as variables. This way you can always know that the file you use for import is always named the same, and has the right data.
You just need to add the variables and a File System Task to your package.
Ok so to accomplish what I wanted I created a Foreach Loop Container. Using the foreach loop container I had it look for any files ending with .csv in my specified folder by using a wildcard [denoted by asterisk: *.csv] .
Within the Foreach Loop container is as follows.
Step 1: File System Task - rename file.
Step 2: Data Flow Task - Import data to sql
Step 3: File System Task - Copy the file to another folder, append datetime to filename
Step 4: File System Task - Delete source file.
I used variables to get all the file and folder names plus datetimes.
I'm using JMeter for performance testing and generate an CSV file for data analysis.
How should I add JMeter Variable names to the CSV output? It seems that only pre-defined columns can be added (see screenshot).
This is my log-file:
14/03/2014,Login,OK,982
14/03/2014,Search,OK,2182
14/03/2014,Login,OK,3982
I would like to add a column with the ${username} to generate something like:
14/03/2014,Login,OK,982,user1
14/03/2014,Search,OK,2182,user7
14/03/2014,Login,OK,3982,user9
Try to pass sample_variables property, for example as command-line parameter -Jsample_variables=username,some_other_var
Also see FlexibleFileWriter plugin if you need custom formatting http://jmeter-plugins.org/wiki/FlexibleFileWriter/?utm_source=jmeter&utm_medium=helplink&utm_campaign=FlexibleFileWriter
anyone know how to set variable for file name in 'Text File Input'?
I want the file name depends on when I execute the transformation, example:
D:\input_file_<variable>.txt
today = D:\input_file_20131128.txt
tomorrow = D:\input_file_20131129.txt
FYI, I'm using Kettle Spoon - 4.2.0
In set form, you can use variable as ${Variable_Name} in file name.
You should notice the system information:
Please remember that the variables you define with this step can't be used in this transformation. This is simply because all steps in a transformation run in parallel without a certain order of execution.
As alternative correct usage, you can set variables you want to use in the first transformation of a job