Can I use a parameter from Carte's URL in a Job? Something like this:
http://localhost:8080/kettle/startJob/?name=myjob&xml=Y&testvar=filename.txt
I want to do this because I have a job to transform an input file but I want to change that filename dynamically, and creating a new XML file for each file is a bit nonsense.
I've tried many things and I couldn't find a solution :-(
All named parameters must be declared in the parameters tab of said JOB and KTR to be executed and receive the information passed from the URL parameters.
Related
I need to create a Azure Data Factory pipeline which has to first format the source file and then call another pipeline. The pipeline would be triggered every time a new file is uploaded in the source blob storage. I want to re-use this pipeline for different source file formats.
For this I intend to use a Switch activity and based on the source file name, call corresponding Copy activity to create a formatted sink file. The issue is that the source files have standard prefixes but then have a timestamp, which means that file name would be different every time, something like:
File 1:
ABCDEF_1233
ABCDEF_2244
File 2:
UVWXYZ_1222
UVWXYX_2345
Can anyone help me understand how to do this?
I was thinking of using a Switch activity, and in the expression, use the #startsWith(triggerBody().fileName, ) and then in the CASE statements, I would like to provide the file name prefixes like ABCDEF, UVWXYZ etc. and then call a copy activity for each of the CASE statements.
But I am not sure how to specify the second argument in the startsWith() function.
suppose you have the filename in a variable called filename. write expression like this to find out which file we are going to load.
Have a set variable activity and assign file prefix to another variable called prefix
#if(greater(indexof(filename),'ABCDEF'),0),'ABCDEF',if(greater(indexof(filename),'UVWXYZ'),0),'UVWXYZ'))
At the end of this set variable, your prefix will have either ABCDEF or UVWXYZ
Then, you can use a switch activity based on prefix variable and mention the cases as
ABCDEF
UVWXYZ
for each case, you can have a copy activity for doing related transforamtions.
I'm trying to call a http endpoint. For that I need to specify a url that uses a query string to filter data.
Sample URL: http://example.com?date=2017-10-04T22:18.007Z
I need to use the current system time as a value for date query string.
I created a script and assigned the generated url with the current datetime to a variable. However, when I assigned that variable for the url field in the source HTTP definition, it did not resolve the variable.
Is there a way to solve this issue?
I do this all the time. As long as your script is running properly (you can test that with the test feature on the script), you are writing the URL value to a global variable (something like $URL), and you are writing that global variable out in your target (something like [URL]), it should work.
If you want to show your script (just where you are creating the URL), and your target URL field that could help narrow down the problem.
I have two transformations in the job.
In the first trasnformation - I get details about the file.
Now I would like to pass this information to the second transformation, I have set variable in the settings parameters of the trasnformation #2 and use Get Variables inside - but the values are not passed.
See attached sample: https://www.hightail.com/download/bXBiV28wMVhLVlZWeHNUQw
LOG_1 - displays file time, however LOG_2 and LOG_3 are not producing any value.
How can I pass variable across transformations and to parent job.
Try checking the box in the second transformation as below image:
Remove all the logs from the Job file you shared.
You may try reading this blog.
Hope this will resolve your issue :)
I'm trying to pass a parameter into the File List component through input port 0. All of my attempts thus far have been met with an error,
Input edge has no effect. Disconnect edge or use metadata fields as parameters in Target URL, Source path or Target path.
Ideally, I would like the Target URL to be something along the lines of http://${S3_ACCESS_KEY}:${S3_SECRET_KEY}#${MY_BUCKET}.s3.amazonaws.com/reports/${port:0.value}/*_interestingReport.csv where ${port:0.value} is the value passed in from the input port.
What is the correct way to use data coming in on input port 0?
The way how passing parameters from input edge for File List (but other file components as well) works, is that you use the name of the metadata column from an input edge and enclose it between ${ and }.
So if the metadata on the edge have a field called directory, which contains the dir you want to use, this is the way how to do it.
http://${S3_ACCESS_KEY}:${S3_SECRET_KEY}#${MY_BUCKET}.s3.amazonaws.com/reports/${directory}/*_interestingReport.csv
Let me show you an example of a very simple graph which uses a 'Data Generator' that creates the flow and sends it as the input of a 'File List' component'.
http://www.filedropper.com/inputportfilelist_1
As you can see the way the input field is referenced is '${DATA_SOURCE_DIR}/${fileDir}/', being 'fileDir' the only field contained in the metadata of the link that connects both components. It'll basically list the files located in ${PROJECT}\data\source\manifests.
I hope this helps.
I have a SoapUI project which contains around 60 plus services. Each service requires some input which will be changed for every execution. So I have created certain Global Properties and assign some values to that properties.
I have to use these properties values in my SoapUI request ( i.e. JSON Format request ).
If it is groovy script means, I will use like this.
String HTiC_Username = com.eviware.soapui.model.propertyexpansion.PropertyExpansionUtils.globalProperties['HTiC_Username'].value;
But, how to get the value of the Global Property in the request?
Hope you understand my question. Please provide proper guidance.
Thanks
To dynamically "expand" (i.e. substitute) the value of a property into a test step, the following syntax is used: ${#scope#propertyName}
...where 'scope' refers to the level at which the property has been defined (e.g. Global, Project, TestSuite, TestCase).
So to expand a property named username defined as a Global property, for example, the following code can be used directly within a Request Test Step (e.g within a JSON body, or header value, etc):
${#Global#username}
To access the same property value within a Groovy Test Step, you can use the following syntax:
context.expand('${#scope#propertyName}')
...as in the following example:
context.expand('${#Global#username}')
What we did was the following:
created a test data file to store all the specific input data for the different services (testdata.properties)
Example content of testdata.properties:
Billing_customerID=1234567
OtherService_paymentid=12121212
....
create a SoupUi global parameter (File/Preferences/Global properties): testdata_filepath=C:\...
For specific services we added a Properties test step. You can specify the "Load from" field to our new global parameter: ${#Global#testdata_filepath} Now you can use the Load button to load parameters.
Finally you can reference the parameter in your xml in the following format: ${Properties#Billing_customerID}
Example content of a service with parameter:
...
<BillingCustomerIdentification>
<BillingCustomerID>${#Properties#Billing_customerID}</BillingCustomerID>
</BillingCustomerIdentification>
...
To set up your projects in this manner also helps to automate service tests eg. using Hudson (see my previous SO answer).
If it is too heavy and automation is not a target, you can simply use ${#Global#someinputvariable} format in your xml ;-)