WSO2 ESB Smooks - XML to CSV Conversion - smooks

I have a simple proxy service which is required to convert an XML payload into a CSV message. For this purpose I'm using smooks mediator. Following is my proxy service.
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="TestProxy" startOnLoad="true" trace="disable"
transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log level="custom">
<property name="STATUS" value="TESTING PROXY SERVICE"/>
</log>
<payloadFactory media-type="xml">
<format>
<csv-set>
<csv-record number="2">
<Name>Jhone</Name>
<City>Colombo</City>
<Age>32</Age>
</csv-record>
<csv-record number="3">
<Name>Doe</Name>
<City>Jaffna</City>
<Age>32</Age>
</csv-record>
</csv-set>
</format>
<args/>
</payloadFactory>
<log level="full"/>
<smooks config-key="gov:Test/smooks/TestSmooksConfig.xml">
<input type="xml"/>
<output type="text"/>
</smooks>
<log level="custom">
<property name="STATUS" value="PROCESSED MSG******"/>
</log>
<log level="full"/>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
</proxy>
Following is my Smooks configuration.
<?xml version="1.0" encoding="UTF-8"?><smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd" xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd">
<params>
<param name="inputType">input.xml</param>
<param name="stream.filter.type">SAX</param>
<param name="input.xml" type="input.type.actived">File:/F:\Practicals\WSO2\vfs_file_listening\input-csv\b.xml</param>
</params>
<ftl:freemarker applyOnElement="#document">
<ftl:template><![CDATA["Name","City","Age"
<#list .vars["csv-set"]["csv-record"] as csv_record>
"${.vars["csv_record"]["Name"]}","${.vars["csv_record"]["City"]}","${.vars["csv_record"]["Age"]}"
</#list>]]></ftl:template>
<param name="includeFieldNames">true</param>
<param name="seperator">,</param>
<param name="quote">"</param>
<param name="csvFields">Name,City,Age</param>
<param name="messageType">CSV</param>
<param name="templateDataProvider">input</param>
</ftl:freemarker>
<resource-config selector="#document">
<resource>org.milyn.delivery.DomModelCreator</resource>
</resource-config>
</smooks-resource-list>
The output message only prints the following.
<?xml version='1.0' encoding='utf-8'?>
<soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope">
<soapenv:Body>
<ax:text xmlns:ax="http://ws.apache.org/commons/ns/payload">"Name","City","Age"</ax:text>
</soapenv:Body>
</soapenv:Envelope>
Not with the data in the input XML tags, Can anyone point out what I'm doing wrong?
Thanks in advance.

I was able to achieve this by the following Smooks configuration. (with the help of wso2 dev support, thanks Evanthika).
<?xml version="1.0" encoding="UTF-8"?>
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
xmlns:ftl="http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd">
<params>
<param name="stream.filter.type">SAX</param>
<param name="inputType">input.xml</param>
<param name="input.xml" type="input.type.actived">File:/C:\Work\2016\09_ESB_Auto_Mail_rpts\TestFiles\TestFileXMLCSVMap.xml
</param>
<param name="default.serialization.on">true</param>
</params>
<resource-config selector="csv-set,csv-record">
<resource>org.milyn.delivery.DomModelCreator</resource>
</resource-config>
<ftl:freemarker applyOnElement="csv-record">
<ftl:template>
<!-- <#ftl ns_prefixes={"D":"http://ws.apache.org/ns/synapse"}> ${.vars["csv-record"].Name},${.vars["csv-record"].City},${.vars["csv-record"].Age} -->
</ftl:template>
<param name="quote">"</param>
<param name="includeFieldNames">true</param>
<param name="csvFields">Name,City,Age</param>
<param name="seperator">,</param>
<param name="messageType">CSV</param>
<param name="templateDataProvider">input</param>
</ftl:freemarker>
</smooks-resource-list>
Changes that have been done are,
<ftl:template> with xml namespace. (In my case it comes from a dataservice, hence the http://ws.apache.org/ns/synapse namespace in the <#ftl ns_prefixes> tag).
<resource-config selector="csv-set,csv-record"> and
<ftl:freemarker applyOnElement="csv-record"> to point out the
elements for smooks to work on.
Hope this helps as reference to anyone.

Related

How to add HTTP Headers Logback Logstash

I read through the following documentation but cannot figure out what other configuration I need: https://github.com/logstash/logstash-logback-encoder#header-fields
My config file:
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<include resource="org/springframework/boot/logging/logback/console-appender.xml" />
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/spring.log}"/>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"app_name":"${APP_NAME:-N/A}","app_version":"${APP_VERSION:-N/A}","hostname":"${HOST:-N/A}","environment":"${environment:-${ENVIRONMENT:-N/A}}"}</customFields>
<includeContext>false</includeContext>
<timeZone>UTC</timeZone>
</encoder>
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<fieldNames>
<requestHeaders>request_headers</requestHeaders>
</fieldNames>
</encoder>
<file>${LOG_FILE}</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE_ROTATION:-${LOG_FILE}%d{yyyy-MM-dd}.%i}</fileNamePattern>
<maxHistory>${LOG_FILE_ROTATION_MAX_HISTORY:-1}</maxHistory>
<totalSizeCap>${LOG_FILE_ROTATION_TOTAL_SIZE_CAP:-3GB}</totalSizeCap>
<maxFileSize>${LOG_FILE_ROTATION_MAX_FILE_SIZE:-1GB}</maxFileSize>
</rollingPolicy>
</appender>
<root level="${LOG_LEVEL:-INFO}">
<appender-ref ref="CONSOLE" />
<appender-ref ref="FILE" />
</root>
</configuration>
Error:
java.lang.IllegalStateException: Logback configuration error detected:
ERROR in ch.qos.logback.core.joran.spi.Interpreter#19:29 - no applicable action for [requestHeaders], current ElementPath is [[configuration][appender][encoder][fieldNames][requestHeaders]]
The xml configuration file you provided appears to be for logback-classic (since it includes <root level=...).
Automatic logging of request headers by logstash-logback-encoder is only available for IAccessEvents logged via logback-access.
In other words, automatic logging of request headers is not available for ILoggingEvents logged via a Logger from logback-classic. However, you can manually include them when using logback-classic as described below.
Logging of request headers with logback-access
To log request headers for IAccessEvents logged via logback-access, follow the instructions for setting up logback-access for tomcat or jetty, and add the following to your logback-access.xml. (Note this is not the logback.xml file used by logback-classic)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashAccessEncoder">
<fieldNames>
<requestHeaders>request_headers</requestHeaders>
</fieldNames>
</encoder>
</appender>
<appender-ref ref="console"/>
</configuration>
This configuration was tested using logstash-logback-encoder 5.3. Note that the xml element names are different in logstash-logback-encoder versions prior to 5.0, so ensure you are using 5.0+ with that configuration.
Logging of request headers with logback-classic
logstash-logback-encoder does not provide first-class support for logging request headers via logback-classic. However, you include them in a log event logged via a Logger by using event-specific custom fields.
For example, in a class that has access to the http request (such as a servlet filter), you could do something like this:
Map<String, String> httpHeadersMap = ...; // get http request headers as a map
LOGGER.info("request", StructuredArguments.entries(httpHeadersMap));
and configure a LogbackEncoder in your logback.xml like this:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
</appender>
<root level="info">
<appender-ref ref="console" />
</root>
</configuration>

WSO2 ESB iterate mediator joining

i'm facing some problems with this process.. let me explain:
i want to call two services and join the responses in this order:
i have the first service's response:
<locations xmlns="http://ssocial.com/traceit/services/location">
<location>
<latitude>-33.45209980</latitude>
<longitude>-70.66241990</longitude>
<altitude>0.0</altitude>
<creation>2016-09-04T14:59:48.000-03:00</creation>
<updated>2016-09-04T14:59:48.000-03:00</updated>
<deviceId>25</deviceId>
</location>
<location>
<latitude>-33.45149521</latitude>
<longitude>-70.66146224</longitude>
<altitude>817.0</altitude>
<creation>2016-09-04T13:18:43.000-03:00</creation>
<updated>2016-09-04T13:18:43.000-03:00</updated>
<deviceId>25</deviceId>
</location>
</locations>
then i have the secon service's response :
<device xmlns="http://ssocial.com/traceit/services/device">
<id>25</id>
<code>mobile_02</code>
<type>
<id>1</id>
<name>MOBILE</name>
</type>
And then, the final response that i would like to have is :
<locations xmlns="http://ssocial.com/traceit/services/location">
<location>
<latitude>-33.45209980</latitude>
<longitude>-70.66241990</longitude>
<altitude>0.0</altitude>
<creation>2016-09-04T14:59:48.000-03:00</creation>
<updated>2016-09-04T14:59:48.000-03:00</updated>
<deviceId>25</deviceId>
<device xmlns="http://ssocial.com/traceit/services/device">
<id>25</id>
<code>mobile_02</code>
<type>
<id>1</id>
<name>MOBILE</name>
</type>
</device>
</location>
<location>
<latitude>-33.45149521</latitude>
<longitude>-70.66146224</longitude>
<altitude>817.0</altitude>
<creation>2016-09-04T13:18:43.000-03:00</creation>
<updated>2016-09-04T13:18:43.000-03:00</updated>
<deviceId>25</deviceId>
<device xmlns="http://ssocial.com/traceit/services/device">
<id>25</id>
<code>mobile_02</code>
<type>
<id>1</id>
<name>MOBILE</name>
</type>
</device>
</location>
</locations>
i'm really having lot of troubles trying to join both responses like that, and this is my sequence in wso2:
<sequence name="getDeviceInfoByLocationProcess" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<property expression="count(//ns1:locations/ns1:location)" name="LOCATIONS_COUNT" scope="default" type="STRING" xmlns:ns1="http://ssocial.com/traceit/services/location"/>
<iterate description="" expression="//ns1:locations/ns1:location/ns1:deviceId" id="LOCATIONS_ITERATOR" xmlns:ns1="http://ssocial.com/traceit/services/location">
<target>
<sequence>
<property description="device_id" expression="//ns1:deviceId" name="device_id" scope="default" type="STRING"/>
<sequence key="getDeviceInfoProcess"/>
</sequence>
</target>
</iterate>
<log level="custom">
<property expression="//ns1:locations" name="LOCATIONS" xmlns:ns1="http://ssocial.com/traceit/services/location"/>
<property expression="//ns2:device" name="DEVICE_INFO" xmlns:ns2="http://ssocial.com/traceit/services/device"/>
</log>
<property name="ECNCLOSING_ELEMENT" scope="default">
<wrapper xmlns=""/>
</property>
<aggregate id="AGG_01">
<completeCondition>
<messageCount max="-1" min="{get-property('LOCATIONS_COUNT')}"/>
</completeCondition>
<onComplete expression="//ns2:device" xmlns:ns2="http://ssocial.com/traceit/services/device">
<log level="full">
<property expression="$ctx:LOCATIONS_COUNT" name="COUNTER"/>
<property expression="$ctx:ECNCLOSING_ELEMENT" name="ENCLOSING"/>
</log>
<call/>
</onComplete>
</aggregate>
and actually with this sequence, i'm just getting this:
<device xmlns="http://ssocial.com/traceit/services/device">
<id>25</id>
<code>mobile_02</code>
<type>
<id>1</id>
<name>MOBILE</name>
</type>
In advance , thank you very much for the help
I can't see the content of "getDeviceInfoProcess" sequence, but I guess your are using call mediator. So, as a result of those requests (multiple requests because of iterate mediator), you receive some responses and that what you have to deal with in the rest of the initial mediation : aggregate offers you a way to wait for all the responses, but the initial message is lost : you should save it in a property (type OM) and use something like payloadMediator inside aggregate/onComplete to compose the final message.

Simple WSOESB Proxy doen't response

Very simple example. Request:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header/>
<soapenv:Body>
<T>
<m1>bysy</m1>
<m1>momo</m1>
<m1>gigi</m1>
</T>
</soapenv:Body>
</soapenv:Envelope>
I want to iterate and log every element in tag m1.
Proxy code:
<proxy xmlns="http://ws.apache.org/ns/synapse" name="Loopback" transports="https,http"
<target>
<inSequence>
<property name="OUT_ONLY" value="true" scope="axis2"/>
<iterate expression="//T/m1">
<target>
<sequence>
<property name="OUT_ONLY" value="true" scope="axis2"/>
<log level="custom">
<property name="M1" expression="//text()"/>
</log>
<drop/>
</sequence>
</target>
</iterate>
</inSequence>
</target>
</proxy>
It work as expected and logs all values. But WSO2ESB server doesn't close connection, until it falls with timeout. (I use SOAPUI for testing)
I tried to put properties OUT_ONLY RESPOND NO_KEEPALIVE in different part of proxy but it didn't help, as well I tried to use , with same result.
It looks like problem in iterate mediator which doesn't let to inSequence to terminate and send response to client. Once i remove it, proxy disconnect immediately with status code '202' message accepted.
This is because, by default, attribute "continueParent" is set to false with iterate mediator : change to <iterate continueParent="true" expression="//T/m1">
And you should explicitly send a 202 inside your inSequence, set this property : <property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>

Error at server side while executing dss query

I'm using wso2dss 3.1.0 and wso2esb 4.7.0.I wish to fire select from esb.I have write select query in dss and dbs for this is like :
<query id="Capp_select_emercontactid" useConfig="default">
<sql>select userid,mailid,phonenumber from muser where phonenumber = ? or mailid = ?</sql>
<result element="Entries" rowName="Entry">
<element column="userid" name="userid" xsdType="string"/>
<element column="mailid" name="mailid" xsdType="string"/>
<element column="phonenumber" name="phonenumber" xsdType="string"/>
</result>
<param name="phonenumber" ordinal="1" sqlType="STRING"/>
<param name="mailid" ordinal="2" sqlType="STRING"/>
</query>
<operation disableStreaming="true" name="Capp_select_emercontactid_op">
<call-query href="Capp_select_emercontactid">
<with-param name="phonenumber" query-param="phonenumber"/>
<with-param name="mailid" query-param="mailid"/>
</call-query>
</operation>
It's working fine in dss.
Now i have write a payload in esb like :
<payloadFactory>
<format>
<p:Capp_select_emercontactid_op xmlns:p="http://ws.wso2.org/dataservice">
<p:phonenumber>$1</p:phonenumber>
<p:mailid>$2</p:mailid>
</p:Capp_select_emercontactid_op>
</format>
<args>
<arg expression="get-property('phoneno1')" evaluator="xml"/>
<arg expression="get-property('mailid1')" evaluator="xml"/>
</args>
</payloadFactory>
<log level="full"/>
<send>
<endpoint>
<address uri="http://localhost:9764/services/Capp_MuserDataservice/" format="soap11"/>
</endpoint>
</send>
When i'm hiting this proxy it didn't give responce..Why so?Is it correct configuration?Please let me know
I never had any luck using the send mediator with the DSS. Could you consider using the CalloutMediator instead (it's a synchronous send/response call so you may need to use a respond mediator after if you're wanting an immediate response to your client).
It'd look something like:
<callout xmlns="http://ws.apache.org/ns/synapse" action="Capp_select_emercontactid_op">
<source xmlns:p="http://ws.wso2.org/dataservice" xpath="//p:Capp_select_emercontactid_op"/>
<target xmlns:p="http://ws.wso2.org/dataservice" xpath="//p:Capp_select_emercontactid_op"/>
</callout>

VFS file processing in WSO2 Esb

I am executing a sample given for VFS file processing, but during execution getting the error as "Cannot find the object for smooks config key: smooks".
Could you please let me know what I have missed out?
Configurations made are as below.
Local entry for smooke is created with below configuration in smooke-key.xml
<localEntry key="smooke" src="file:/development/Dev/wso2esb-4.8.0/repository/resources/smooks-config.xml"><description/></localEntry>
==================================================================================
Smooke-config.xml is as below
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.0.xsd">
<!--Configure the CSVParser to parse the message into a stream of SAX events. -->
<resource-config selector="org.xml.sax.driver">
<resource>org.milyn.csv.CSVParser</resource>
<param name="fields" type="string-list">name,value</param>
</resource-config>
Proxy service is as below
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="FileProcessor"
transports="vfs"
startOnLoad="true"
trace="disable">
<description/>
<target>
<inSequence>
<log level="full"/>
<smooks config-key="smooke">
<input type="text"/>
<output type="xml"/>
</smooks>
<clone>
<target sequence="fileWriteSequence"/>
<target sequence="databaseSequence"/>
</clone>
</inSequence>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///development/Dev/fileProcessing/sourcefiles</parameter>
<parameter name="transport.vfs.FileURI">file:///development/Dev/fileProcessing/in</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///development/Dev/fileProcessing/error</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>
create the smooks configuration at
wso2esb-4.8.0/repository/samples/resources/smooks/Smooke-config.xml
and restart the esb
change the localentry file like this
<localEntry key="smooke" src="file:repository/resources/smooks-config.xml"><description/></localEntry>