I have known good code (it's been implemented for days).
My runtime is 3.9.5
My Anypoint Studio version is 6.6.11
The error shows up on all transforms and is simply "Error in DW script:" with no further explanation. None of the dw scripts have been altered so it's clearly environmental, but I can't identify the issue.
Here is an example of one:
<dw:transform-message metadata:id="4bbf0f0a-d0fb-4018-972c-fa62138b3f60" doc:name="fail message">
<dw:input-variable mimeType="application/java" variableName="responseMessage"/>
<dw:input-variable mimeType="application/java" variableName="postMessage"/>
<dw:set-payload><![CDATA[%dw 1.0% output application/json
---
{
statusCode: "403",
statusMessage: flowVars.postMessage
}]]></dw:set-payload>
</dw:transform-message>
That looks like a known issue in Studio 6. It should have been fixed in Studio 6.6.11 but it seems not fully fixed: https://docs.mulesoft.com/release-notes/studio/anypoint-studio-6.6-with-3.9.5-runtime-update-site-11-release-notes#fixed-issues
Related
I am using the Mule 4.4 community edition on premise.
Thanks to help, I have been able to read a large file without consuming memory and processing it, which is all good (here).
Now building on this further - my use case is to read all .csv files from within a directory.
And then process them one by one:
\opt\out\
students.csv
teachers.csv
collesges.csv
....
So my plan was to list the files in the directory:
<sftp:list doc:name="List" config-ref="SFTP_Config" directoryPath="/opt/out">
<non-repeatable-iterable />
<sftp:matcher filenamePattern="#['*.csv' ]"
directories="EXCLUDE" symLinks="EXCLUDE" />
</sftp:list>
And then I wanted to only read file names from directory and not read payload.
As per this early access article we are advised to use <non-repeatable-iterable />. However, after the list file operation as per article when I try to extract attributes:
<set-payload doc:name="Set Payload" value="#[output application/json --- payload map $.attributes]"/>
No attributes are available... (my plan is to extract the file names and then run a for loop for each file name and then a choice condition to determine if file name has student, use student transformer, if teacher use teacher transformer, etc.)
However, as attributes are not available, I am not able to pass file names to the for loop (yet to be written).
So I changed from <non-repeatable-iterable /> to <repeatable-in-memory-iterable />
Code below:
<sftp:list doc:name="List" config-ref="SFTP_Config" directoryPath="/opt/out">
<repeatable-in-memory-iterable />
<sftp:matcher filenamePattern="#['*.csv' ]"
directories="EXCLUDE" symLinks="EXCLUDE" />
</sftp:list>
Using the above, I can extract the attributes of file names.
I am confused about the following:
The files to be processed in the above directory will be large (each file 700 MB), so while iterating the directory by using repeatable-in-memory-iterable, will it cause any memory issues? (I do not want to read file content, simply get file names at this stage)
Here is the complete payload till now (note - it does not contain any for loop to iterate over files, which I will plug in...)
<flow name="employee-process-flow">
<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="/processFiles"/>
<set-variable value='#[now() as String { format: "ddMMuu" }]' doc:name="Set todays date as ddmmyy" doc:id="c6a91a41-65b1-46df-a720-9c13fe360b6b" variableName="today"/>
<sftp:list doc:name="List" config-ref="SFTP_Config" directoryPath="/opt/out">
<repeatable-in-memory-iterable />
<sftp:matcher filenamePattern="#['*.csv' ]"
directories="EXCLUDE" symLinks="EXCLUDE" />
</sftp:list>
<set-payload doc:name="Set Payload" value="#[output application/json --- payload map $.attributes]"/>
<foreach doc:name="For Each" >
<logger level="INFO" doc:name="Logger" message="we are here"/>
</foreach>
</flow>
The List operation returns a list of messages, and each has a payload and attributes. The content of the files is returned as the payload, in a lazy way, meaning that the file's content is read only if you try to access that element's payload.
It makes sense that if you a non-repeatable-iterator and don't access the payload of each item in the <foreach> then you should not have any memory issues, because the contents are not read.
By using in memory repeatable streaming it is possible that the entire payload is being read into memory. Try reading a file a few gigabytes in size and see what happens there.
I'm not sure what the problem is with the attributes. It should work the same in any streaming mode.
Note that if you plan on doing something with the attributes—other than printing them—then you should output to application/java instead of JSON, to avoid unneeded conversions to and from JSON. For example, in your flow the output is used as input for the <foreach>, so it would be better for it to be Java.
Example:
output application/java --- payload map $.attributes
I'm trying to find solution for deleting request headers in MULE 4 by more effective way then actual.
In request config is possible to have <http:default-headers />, but I can't find way to delete headers from here. So I'm deleting headers in flow ->
like this:
<http:headers>#[attributes.headers -- ["host", "http.headers"]]</http:headers>
Full version of flow is here:
<flow name="WS_name">
<http:listener path="someway/*" config-ref="External_Listener" doc:name="HTTP">
<http:response statusCode="#[vars.httpStatus default 200]">
<http:headers>#[vars.outboundHeaders default {}]</http:headers>
</http:response>
<http:error-response statusCode="#[vars.httpStatus default 500]">
<http:body>#[payload]</http:body>
<http:headers>#[vars.outboundHeaders default {}]</http:headers>
</http:error-response>
</http:listener>
<http:request method="#[attributes.method]" sendBodyMode="AUTO" config-ref="HTTPS_Request_configuration_proxy" url="#['https://${endpoint.WS_name.host}:${endpoint.WS_name.port}' ++ '${endpoint.WS_name.path}/']">
<http:headers>#[attributes.headers -- ["host", "http.headers"]]</http:headers>
</http:request>
If anyone have better solution, please, share it.
What you are doing is fine. If you want to do that in the flow outside of the http:request, then you can. All the http:headers expects is a map. So here is an example of using a var:
<set-variable variableName="myAttributes" value="#[attributes.headers -- ['host', 'http.headers']]" />
<http:request method="#[attributes.method]" sendBodyMode="AUTO" config-ref="HTTPS_Request_configuration_proxy"
url="#['https://${endpoint.WS_name.host}:${endpoint.WS_name.port}' +++ '${endpoint.WS_name.path}/']">
<http:headers>#[vars.myAttributes]</http:headers>
</http:request>
I am new to Mule and I am using Mule version 3.x. We can set values of variable using Set Variable Component. And using expression component, we can set multiple variables at a time. But expression component does not allow to Declare a flow variable. So I have to first use 3 Set Variable component and then use expression to set them at one go(if it needs to be changed).
In case of multiple variables, is there any way to declare and initialize variable in one component instead of having a separate Set Variable component to declare it?
The Message Properties Transformer
If you want to set multiple flow vars in a single component in Mule 3, and you don't need to do any transformation beforehand, the Message Properties transformer is probably the most succinct ways to do this. I like it because in addition to being able to set multiple flow vars in a single component, the XML is clear, and when you click on the component in AP Studio, the UI makes it immediately obvious that you're setting multiple variables with a single component. Just make sure to use scope="invocation" so that you're setting flow vars:
<message-properties-transformer scope="invocation" doc:name="Set flowVars">
<add-message-property key="varName1" value="1"/>
<add-message-property key="varName2" value="2"/>
<add-message-property key="varName3" value="2"/>
</message-properties-transformer>
If you need to do small transformations, you can always call DataWeave from MEL. Here's an example:
...
<add-message-property key="varName1" value="dw('payload map $.id')"/>
...
The Transform Message Component (DataWeave)
You can also do this in DataWeave with the Transform Message component as well. I do feel it has a couple disadvantages that I should point out, though. The first is the XML is more verbose and difficult to read:
<dw:transform-message doc:name="Transform Message">
<dw:set-variable variableName="varName1"><![CDATA[%dw 1.0
%output application/java
---
1]]></dw:set-variable>
<dw:set-variable variableName="varName2"><![CDATA[%dw 1.0
%output application/java
---
2]]></dw:set-variable>
<dw:set-variable variableName="varName3"><![CDATA[%dw 1.0
%output application/java
---
3]]></dw:set-variable>
</dw:transform-message>
In addition, there is no obvious way to tell from the AP Studio UI that this particular transform message component is setting multiple variables:
If you're using enterprise edition, please use dataweave to set multiple variables in one component.
Runtime 4.1
https://docs.mulesoft.com/mule-runtime/4.1/dataweave-variables
Runtime 3.8
https://docs.mulesoft.com/mule-runtime/3.8/dataweave
I would like to know why do you need to just declare variables. Anyways if in case you just need it blank while declaring, you can simply set it as below.
<expression-component doc:name="Expression">
<![CDATA[flowVars.var1 ="" ;
flowVars.var2="";]]>
</expression-component>
Hope this help.
I am facing an issue while running product2 SOQL long query in Mule salesforce Connector. I am able to run and got successful json message with certain fields like Id, Name, ProductCode, Description etc . But some other fields are not working like... Units__c, Stock_Item__c, Sub_Category__c etc. Non working fields are not available as well while I was selecting with "Query builder" in "DataSense Query Language". I guess those fields are customs made from my organization. But All kinds of queries are working perfectly on Salesforce developer console. I am looking for solution so that i can invoke all the fields with result from salesforce product object.
My GET URL :
http://localhost:8081/xxxxxxx/product2?lastModifiedDate=20180629215028
AnyPoint Studio Version: 6.4.2
MuleRuntime 3.9.0
Salesforce Connector - Mule 3 - version : 8.6.0
salesforce API version 40
salesfroce url
:https://xxxxxxxxx.cs96.my.salesforce.com/services/Soap/u/40.0
Tried Also :
with Native Query Language. but unsuccessful.
Thanks in advance !
Mule Configuration XML
<flow name="salesforce_invoking_all_product2_information_flow">
<logger level="INFO" doc:name="Get product2 -Start" message="Get product2- Start"/>
<dw:transform-message doc:name="Transform Message">
<dw:input-payload doc:sample="sample_data\json.json"/>
<dw:input-inbound-property doc:sample="sample_data\Map_1.dwl" propertyName="http.query.params"/>
<dw:set-variable variableName="lastModifiedDateTime"><![CDATA[%dw 1.0
%output application/java
---
(((inboundProperties."http.query.params".lastModifiedDate as :string)
as :localdatetime {format: "yyyyMMddHHmmss"})
as :string {format: "yyyy-MM-dd'T'HH:mm:ss'.000Z'"})
]]></dw:set-variable>
</dw:transform-message>
<sfdc:query-all config-ref="Salesforce_Basic_Authentication" query="dsql:SELECT Id, Name, ProductCode, Description, IsActive, CreatedDate, CreatedById, LastModifiedDate, LastModifiedById, SystemModstamp, RecordTypeId, Family, ExternalDataSourceId, ExternalId, DisplayUrl, QuantityUnitOfMeasure, IsDeleted, LastViewedDate, LastReferencedDate, Account_Number__c, Color__c, Cost_PO__c, Inventoried_Item__c, Length__c, Paint_Stain_Misc_Job_and_Unit_Charges__c, Style_Code__c, SalesforceID__c, Units__c, Number_Service_Products__c, Master_Legacy_Product__c, Has_Service_Products__c, Products_Joined__c, Number_of_Units__c, Mull_Sequence__c, Mull_Number__c, Mull_Material__c, Mull_Assembly_Type__c, isStandard__c, Unit_of_Measure__c, Stock_Item__c, Sub_Category__c, Cross_Reference__c, Width__c, Category__c, Pivotal_Id__c, Height__c, Unit_Short_Description__c, UI_Minimum__c, UI_Maximum__c, Service_PO__c, Report_Grouping__c, Master_Product__c, Name_Part_Number__c, Part_Number__c, Product_Configuration__c, Product_Image__c, Vendor__c, Product_PO__c, Product_Type__c FROM Product2 limit 10" doc:name="get product 2 details"/>
<json:object-to-json-transformer doc:name="Object to JSON"/>
<logger message="Get Contact- Finish" level="INFO" doc:name="Get Product 2 Finish"/>
</flow>
ERROR Console
Message : org.mule.modules.salesforce.exception.SalesforceException (org.codehaus.jackson.map.JsonMappingException)
Payload : org.mule.streaming.ConsumerIterator#6b1280fb
Transformer : ObjectToJson{this=599abd04, name='ObjectToString', ignoreBadInput=false, returnClass=SimpleDataType{type=java.lang.String, mimeType='application/json', encoding='null'}, sourceTypes=[]}
Payload Type : org.mule.streaming.ConsumerIterator
Element : /salesforce_invoking_all_product2_information_flow/processors/3 # rba-salesforce-product-system:implementation.xml:26 (Object to JSON)
Element XML : <json:object-to-json-transformer doc:name="Object to JSON"></json:object-to-json-transformer>
--------------------------------------------------------------------------------
Root Exception stack trace:
[InvalidFieldFault [ApiQueryFault [ApiFault exceptionCode='INVALID_FIELD'
exceptionMessage='
Style_Code__c,SalesforceID__c,Units__c,Number_Service_Products__c
^
ERROR at Row:1:Column:390
No such column 'Units__c' on entity 'Product2'. If you are attempting to use a custom field, be sure to append the '__c' after the custom field name. Please reference your WSDL or the describe call for the appropriate names.'
extendedErrorDetails='{[0]}'
]
row='1'
column='390'
]
]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.sforce.ws.bind.TypeMapper.readSingle(TypeMapper.java:673)
at com.sforce.ws.bind.TypeMapper.readObject(TypeMapper.java:556)
at com.sforce.ws.transport.SoapConnection.parseDetail(SoapConnection.java:236)
at com.sforce.ws.transport.SoapConnection.createException(SoapConnection.java:210)
at com.sforce.ws.transport.SoapConnection.receive(SoapConnection.java:156)
at com.sforce.ws.transport.SoapConnection.send(SoapConnection.java:99)
at com.sforce.soap.partner.PartnerConnection.queryAll(PartnerConnection.java:836)
at org.mule.modules.salesforce.SalesforceConnector$2.doQuery(SalesforceConnector.java:877)
Still learning Mulesoft's Anypoint Studio... I am confused as how will I be able to access raw JSON POST data via the HTTP Listener then use the Choice flow control to execute conditions based on a value from a given JSON index. Anyone can show/tell me how to do this?
The JSON HTTP body will automatically become the payload of your message in Mule probably represented as Stream.
Just for demo purposes, try logging the payload after your http:listener using:
<object-to-string-transformer />
<logger level="INFO" message="#[payload]" />
There best way to query JSON is to transform it to a Map suing the JSON module transformers.
<json:json-to-object-transformer returnClass="java.util.HashMap" />
And then query it using MEL like standard MVEL or Java syntax.
For a JSON document like: {"person" : {"name" : "bob"}}
<logger message="#[payload.person.name]" level="INFO" />
You can use these expressions in your choic router also:
<choice>
<when expression="#[payload.person.name == 'bob']">
do something ...
</when>
</choice>