I have two transformations in the job.
In the first trasnformation - I get details about the file.
Now I would like to pass this information to the second transformation, I have set variable in the settings parameters of the trasnformation #2 and use Get Variables inside - but the values are not passed.
See attached sample: https://www.hightail.com/download/bXBiV28wMVhLVlZWeHNUQw
LOG_1 - displays file time, however LOG_2 and LOG_3 are not producing any value.
How can I pass variable across transformations and to parent job.
Try checking the box in the second transformation as below image:
Remove all the logs from the Job file you shared.
You may try reading this blog.
Hope this will resolve your issue :)
Related
I'm running a simple penguin pipeline in interactive mode with a split train/eval, the transform step run but i can't get post_transform_statistics artifacts.
Inside the dedicated artifacts folder /tmp/tfx-penguin_custom_INTERACTIVE-nq5dn56x/Transform/post_transform_stats/5, i have just one FeaturesStats.pb inside, but not subfolders Split-train and Split-eval with a FeaturesStats.pb inside each.
However, I have the subfolders inside artifacts dedicated to transformed examples (/tmp/tfx-penguin_custom_INTERACTIVE-nq5dn56x/Transform/transformed_examples/5/).
Here is how i define the transform components by explicitly providing splits and also disable_statistics=False:
transform = tfx.components.Transform(
examples=example_gen.outputs['examples'],
schema=schema_gen.outputs['schema'],
disable_statistics=False,
splits_config= transform_pb2.SplitsConfig(
analyze=['train'], transform=['train', 'eval']),
module_file=_transformer_module_file)
I went to the docstring and even the __init__ of the component https://github.com/tensorflow/tfx/blob/master/tfx/components/transform/component.py, it seems there is nothing i would have forgotten or mistaken but i was very disturbed to read following comment with an untraceable location for stats....
disable_statistics: If True, do not invoke TFDV to compute pre-transform
and post-transform statistics. When statistics are computed, they will
will be stored in the `pre_transform_feature_stats/` and
`post_transform_feature_stats/` subfolders of the `transform_graph`
export.
For now, the workaround is to explicitly disable stats in the transform component and define next to it, a dedicated statistics components to work on transformed features splits but it would have been great to have the splits statistics inside transform component directly.
Thanks for any help
This is expected as StatisticsGen in Transform is currently working on the entire transform dataset regardless of split/span.
To generate separate statistics for different splits, please use StatisticsGen component.
Thank you!
Trying to add my own table to Jmeter dashboard report. In Jmeter it's not clear enough. Tried as in https://jmeter.apache.org/usermanual/generating-dashboard.html#customs_graphs
but the custom part doesn't appear in dashboard. Would appreciate your help and examples.
It looks like the documentation you're referring is not comprehensive enough, the chapter assumes that you can plot the numeric value of a Sample Variable
So if you put the next line to user.properties file (lives in "bin" folder of your JMeter installation)
sample_variables=ts-hit
and copy and paste the example configuration from the documentation:
jmeter.reportgenerator.graph.custom_testGraph.classname=org.apache.jmeter.report.processor.graph.impl.CustomGraphConsumer
jmeter.reportgenerator.graph.custom_testGraph.title=Chunk Hit
jmeter.reportgenerator.graph.custom_testGraph.property.set_Y_Axis=Number of Hits
jmeter.reportgenerator.graph.custom_testGraph.set_X_Axis=Over Time
jmeter.reportgenerator.graph.custom_testGraph.property.set_granularity=60000
jmeter.reportgenerator.graph.custom_testGraph.property.set_Sample_Variable_Name=ts-hit
jmeter.reportgenerator.graph.custom_testGraph.property.set_Content_Message=Number of Hits :
Create some fake values for the ts-hit variable using i.e. __Random() function:
As the result you should see something like this:
With regards to custom tables - as of JMeter 5.5 it's not possible, however if you're comfortable with FreeMarker you can amend the template and add whatever you want there
Can I use a parameter from Carte's URL in a Job? Something like this:
http://localhost:8080/kettle/startJob/?name=myjob&xml=Y&testvar=filename.txt
I want to do this because I have a job to transform an input file but I want to change that filename dynamically, and creating a new XML file for each file is a bit nonsense.
I've tried many things and I couldn't find a solution :-(
All named parameters must be declared in the parameters tab of said JOB and KTR to be executed and receive the information passed from the URL parameters.
When I do an extract from multiple files and include part of the filename in the fields list and in the FROM clause (e.g. FROM "/input/filename-{filedate:*}.nc"), the resulting output file only contains a header row. If I remove "filedate" from the fields list and the FROM clause, I get the correct output.
I noticed in the job graph that when including "filedate", an "Empty Input" and an "Extract Cross" step is added before the "PodAggregate" step, and in the "Extract Cross" no data is written. What is this step?
Also, if I run the original extract including "filedate" locally, I get the correct output, so it's only in ADLA this error occurs.
I use a custom extractor and I don't know if this has anything to do with it. I haven't tested with a built-in extractor.
We released the new "fast file set" option by default. Unfortunately, it introduced a regression for some plans. Until we fix it, please add the following statement to your script:
SET ##InternalDebug = "FileSetV2:off";
Our apologies for any inconvenience this may have caused.
I have a SoapUI project which contains around 60 plus services. Each service requires some input which will be changed for every execution. So I have created certain Global Properties and assign some values to that properties.
I have to use these properties values in my SoapUI request ( i.e. JSON Format request ).
If it is groovy script means, I will use like this.
String HTiC_Username = com.eviware.soapui.model.propertyexpansion.PropertyExpansionUtils.globalProperties['HTiC_Username'].value;
But, how to get the value of the Global Property in the request?
Hope you understand my question. Please provide proper guidance.
Thanks
To dynamically "expand" (i.e. substitute) the value of a property into a test step, the following syntax is used: ${#scope#propertyName}
...where 'scope' refers to the level at which the property has been defined (e.g. Global, Project, TestSuite, TestCase).
So to expand a property named username defined as a Global property, for example, the following code can be used directly within a Request Test Step (e.g within a JSON body, or header value, etc):
${#Global#username}
To access the same property value within a Groovy Test Step, you can use the following syntax:
context.expand('${#scope#propertyName}')
...as in the following example:
context.expand('${#Global#username}')
What we did was the following:
created a test data file to store all the specific input data for the different services (testdata.properties)
Example content of testdata.properties:
Billing_customerID=1234567
OtherService_paymentid=12121212
....
create a SoupUi global parameter (File/Preferences/Global properties): testdata_filepath=C:\...
For specific services we added a Properties test step. You can specify the "Load from" field to our new global parameter: ${#Global#testdata_filepath} Now you can use the Load button to load parameters.
Finally you can reference the parameter in your xml in the following format: ${Properties#Billing_customerID}
Example content of a service with parameter:
...
<BillingCustomerIdentification>
<BillingCustomerID>${#Properties#Billing_customerID}</BillingCustomerID>
</BillingCustomerIdentification>
...
To set up your projects in this manner also helps to automate service tests eg. using Hudson (see my previous SO answer).
If it is too heavy and automation is not a target, you can simply use ${#Global#someinputvariable} format in your xml ;-)