In a dataflow job written in Kotlin
using a PubSub subscription as input i receive a Proto object (Event) and map this object to Strings.
My pipeline has type:
PCollection<KV<Event, String>>
These strings are the lines of a file that must be written in GCS.
The Event Object has a "Id" that must be used to set the filename, and a "name" to set the folder.
Is it possible using FileIO ?
pipeline.apply(
FileIO.writeDynamic<String, String>()
.to("gs://my-bucket")
// withNaming?
)
My goal is to write the right lines in the right files, based on the information in the Event object
File-names can be customized by providing a FileNaming implementation to the withNaming() API.
However this currently does not support mapping input elements directly to final file names. Input elements can be mapped to groups using the dynamic destinations API and for each group you can provide a file-naming strategy.
To fully customize naming using input element values you might need to implement a new sink transform.
Related
I have a simple flow consisting of
GenerateFlowFile ----> PutS3Object ----> Wait
And the generated flow files are getting stored in the bucket correctly.
Now I want to add Metadata to my flow file.
If I add a property "Test1" to PutS3Object, it shows up as "X-Amz-Meta-Test1" in the metadata of the object.
But if I add a property "Test2" in GenerateFlowFile it doesn't show up in metadata.
I tried adding "Test2" as s3.usermetadata.Test2 but it still didn't work.
Is there a way to pass all the flow files attributes as metadata without explicitly adding properties in the PutS3Object.
PutS3Object only inserts metadata values that you have set as Dynamic Properties on the PutS3Object processor itself. Please see the docs link and look at the Dynamic Properties section.
PutS3Object does not just stick any Attribute you set as metadata, otherwise you would end up with potentially hundreds of metadata entries that you aren't interested in. The only Attribute it reads by default is filename - please see the Reads Attributes section of the docs.
If you have an existing Attibute, and you want to push the value of this Attribute into the metadata, you must add a Dynamic Property to PutS3Object and reference the value of the Attribute.
E.g. you have an Attribute called file_author with a value Steve and you want the S3 object to have the metadata field author with the value Steve:
You would add a Dynamic Property to PutS3Object with a name of author and a value of ${file_author}.
Edit:
You could fork PutS3Object into a custom processor to add the dynamic functionality you want, but I would recommend just using the standard PutS3Object config and manually configuring the Attributes you want.
I am working on a project that integrates Imebra inside an android application. The application is supposed to extract all the data from a given DICOM file and put them into a .xml file. I need a little bit of help with it. For example, I don't know how to get all the VR tags that the given DICOM has, instead of getting them one by one using tag ids.
Thank you for your help.
Load the file using CodecFactory.load(filename).
Then you can use DataSet.getTags() to retrieve a list of tags stored into the DICOM structure.
The returned class TagsIds is a list containing all the TagId: scan each tag ID and retrieve it via DataSet.getString() (to retrieve the value as string) and DataSet.getDataType() to retrieve its VR.
When DataSet.getString() fails then you are dealing with a sequence (an embedded DICOM structure) which can be retrieved with DataSet.getSequenceItem().
You can use the static method DicomDictionary.getTagName() to get a description of a particular tag.
I have a simple conversion of JSON to XML using MuleSoft. In "Transform Message" component, I provided JSON Schema as Input and XML Schema as Output. When I run the app, the conversion happens if the file matches with both schema but it generates an empty XML file if it doesn't match.
I want below conditions:
1) If the file matches with schema, the converted output file should be sent to converted folder and the original file should move to Success folder.
2) If the file doesn't match with schema, the original file should move to the Failure folder instead of conversion.
Hope, I explained it comprehensively as I am new to MuleSoft. Here is a sample diagram which may simplify my requirement. Provide me with a new one if I badly designed the process.
First thing you need to create a flowVar that will hold your original payload.
When your doing your evaluation, if its XML then use a simple XPath expression like //elementName[not(node())]
Lastly, on your success use scatter-gather for multi-threading write. Pull your original payload from flowVar and write to Success and Write your regular payload to your Converted folder
I have a SoapUI project which contains around 60 plus services. Each service requires some input which will be changed for every execution. So I have created certain Global Properties and assign some values to that properties.
I have to use these properties values in my SoapUI request ( i.e. JSON Format request ).
If it is groovy script means, I will use like this.
String HTiC_Username = com.eviware.soapui.model.propertyexpansion.PropertyExpansionUtils.globalProperties['HTiC_Username'].value;
But, how to get the value of the Global Property in the request?
Hope you understand my question. Please provide proper guidance.
Thanks
To dynamically "expand" (i.e. substitute) the value of a property into a test step, the following syntax is used: ${#scope#propertyName}
...where 'scope' refers to the level at which the property has been defined (e.g. Global, Project, TestSuite, TestCase).
So to expand a property named username defined as a Global property, for example, the following code can be used directly within a Request Test Step (e.g within a JSON body, or header value, etc):
${#Global#username}
To access the same property value within a Groovy Test Step, you can use the following syntax:
context.expand('${#scope#propertyName}')
...as in the following example:
context.expand('${#Global#username}')
What we did was the following:
created a test data file to store all the specific input data for the different services (testdata.properties)
Example content of testdata.properties:
Billing_customerID=1234567
OtherService_paymentid=12121212
....
create a SoupUi global parameter (File/Preferences/Global properties): testdata_filepath=C:\...
For specific services we added a Properties test step. You can specify the "Load from" field to our new global parameter: ${#Global#testdata_filepath} Now you can use the Load button to load parameters.
Finally you can reference the parameter in your xml in the following format: ${Properties#Billing_customerID}
Example content of a service with parameter:
...
<BillingCustomerIdentification>
<BillingCustomerID>${#Properties#Billing_customerID}</BillingCustomerID>
</BillingCustomerIdentification>
...
To set up your projects in this manner also helps to automate service tests eg. using Hudson (see my previous SO answer).
If it is too heavy and automation is not a target, you can simply use ${#Global#someinputvariable} format in your xml ;-)
I have an application where in I am trying to create a velocity template repository which will help me centralise all my email templates and will allow me to create a communication hub. All templates will be called at runtime and populates with data via services.
My problem is that I need to provide users with optional and compulsory params list when they define the template inputs for the velocity template.
Is there a way to read the tokens/tags from the velocity template file and extract them??
Like I want a list of tokens $name.address.streetName to be available to me from .vm file.
I do not want to go for Regex .
I do not have to cache or reuse them , its just going to be a one time read and store the default,compulsory & optional params in the database.
I am following these patterns : http://kickjava.com/src/org/apache/velocity/test/view/TemplateNodeView.java.htm
How to use String as Velocity Template?
Please advice.
I got it working like this
RuntimeServices runtimeServices = RuntimeSingleton.getRuntimeServices();
StringReader reader = new StringReader(String velocityTemplateBodu);
SimpleNode node = runtimeServices.parse(reader, "dummyOne.vm");
for(int i=0; i<node.jjtGetNumChildren();i++){
if(node.jjtGetChild(i) instanceof org.apache.velocity.runtime.parser.node.ASTReference ){
System.out.println("Node -----------------"+i +"---"+node.jjtGetChild(i).literal());
}
}
Using SimpleNode class you get all the nodes on the .vm file.
The Nodes are read using javaCC as ASTReference and ASTText (both extend SimpleNode). To get the tokens you need to get the ASTReference and to get HTML text use the ASTText.