I have cucumber data tables with headers, and I am wondering how to parse them with Jackson when they have headers. I have the following configuration. This seems to work as long as everything is a String. But if my object contains for instance a Double, then when it tries to parse the header for that column it will choke on the column name. How can I tell it to skip the header?
private final ObjectMapper csvMapper = new CsvMapper()//
.registerModule(new JavaTimeModule())//
.enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT)//
.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)//
.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);
#DefaultParameterTransformer
#DefaultDataTableEntryTransformer
#DefaultDataTableCellTransformer
public Object defaultTransformer(Object fromValue, Type toValueType) {
JavaType javaType = csvMapper.constructType(toValueType);
return csvMapper.convertValue(fromValue, javaType);
}
Related
Version used: Kafka 3.1.1, Confluent 7.1.0, Avro 1.11.0
I’m creating a REST controller which is “searching” for AVRO objects in a topic. The objects in the topic are serialized using SpecificAvroSerde<>. Each topic has assigned two AVRO schemas. One for the key (with several fields of various types) and one for the value (multiple fields and types).
I’ve done this several times whereby I’m consuming the topic in a KTable and then materialize it. There is only one pair of serdes involved and the serialized format is the same for both the topic and the materialized view (RocksaltDb). The REST controller then can look up the store and either perform a get with a key or do a range scan between two keys. This all works as expected.
private final static String TOPIC_NAME = "input-topic";
private final static String VIEW_NAME = "materialized-view";
private final SpecificAvroSerde<ProductXrefKey> productXrefKeySerde = new SpecificAvroSerde<>();
private final SpecificAvroSerde<ProductXref> productXrefSerde = new SpecificAvroSerde<>();
final Map<String, Object> props = this.kafkaProperties.buildStreamsProperties();
productXrefKeySerde.configure(props, true);
productXrefSerde.configure(props, false);
KTable<ProductXrefKey, ProductXref> productXrefTable = builder
.table(TOPIC_NAME, Consumed.with(productXrefKeySerde, productXrefSerde),
Materialized.<ProductXrefKey, ProductXref, KeyValueStore<Bytes, byte[]>>as(VIEW_NAME)
.withKeySerde(productXrefKeySerde)
.withValueSerde(productXrefSerde));
<…>
final ReadOnlyKeyValueStore<ProductXrefKey, ProductXref> store =
streamsBuilderFactoryBean.getKafkaStreams().store(fromNameAndType(VIEW_NAME, keyValueStore()));
try (KeyValueIterator<ProductXrefKey, ProductXref> range = store.range(fromKey, toKey)) {
if (range != null) {
range.forEachRemaining(kv -> {
<…>
});
} else {
log.info("Could not find {} in local ReadOnlyKeyValueStore {}", fromKey, viewName);
}
}
I now want to change this using a prefix scan instead. Since the key contains multiple fields there is no way to only serialize first part (i.e. first few fields) of the key I need a specialized serializer. This also means I have to use a different serializer for the materialized view itself (SpecificAvroSerde puts the magic number and schema ID at the beginning of the byte array) as otherwise the serialized output for the prefix and the key in the materialized view can’t be compared. Hence I created a specialised Serde which serializes the key using the same logic as when used for serializing the prefix but omitting the fields not required for the scan (i.e. omitting the last field). Above code now looks
private final static String TOPIC_NAME = "input-topic";
private final static String VIEW_NAME = "materialized-view";
private final SpecificAvroSerde<ProductXrefKey> productXrefKeySerde = new SpecificAvroSerde<>();
private final SpecificAvroSerde<ProductXref> productXrefSerde = new SpecificAvroSerde<>();
private final SpecificAvroSerde<ProductXrefKey> materializedProductXrefKeySerde = new ProductXrefKeySerde();
// for the value part we can still used standard serde as no change in serialization logic needed
private final SpecificAvroSerde<ProductXref> materializedProductXrefSerde = new SpecificAvroSerde<>();
// telling the serializer to cut off last field
private final SpecificAvroSerde<ProductXref> prefixScanProductXrefSerde = new ProductXrefKeySerde(true);
final Map<String, Object> props = this.kafkaProperties.buildStreamsProperties();
productXrefKeySerde.configure(props, true);
productXrefSerde.configure(props, false);
KTable<ProductXrefKey, ProductXref> productXrefTable = builder
.table(TOPIC_NAME, Consumed.with(productXrefKeySerde, productXrefSerde),
Materialized.<ProductXrefKey, ProductXref, KeyValueStore<Bytes, byte[]>>as(VIEW_NAME)
.withKeySerde(materializedProductXrefKeySerde)
.withValueSerde(materializedProductXrefSerde));
<…>
final ReadOnlyKeyValueStore<ProductXrefKey, ProductXref> store =
streamsBuilderFactoryBean.getKafkaStreams().store(fromNameAndType(VIEW_NAME, keyValueStore()));
try (KeyValueIterator<ProductXrefKey, ProductXref> range = store.prefixScan(prefixKey, prefixScanProductXrefSerde)){
if (range != null) {
range.forEachRemaining(kv -> {
<…>
});
} else {
log.info("Could not find {} in local ReadOnlyKeyValueStore {}", prefixKey, viewName);
}
}
My assumption was, that the topic gets deserialized using the SpecificAvroSerde and then gets serialized for the view using my ProductXrefKeySerde. The problem is, that the content in the materialized view is still serialized using the same logic as in the original topic. It appears that the serializer is never used during the topic being processed and stored in the materialized view. I can verify that also on the file system and see that the keys in the RocksaltDb files are serialized with the magic byte and schema ID and hence prefixScan wont be able to fine anything.
How can I change the serialization format for the materialized view?
Or is there a better way for serializing a prefix AVRO object?
It appears that there is some optimization happening which avoids deserialization/serialization if KTable is directly materialized. I've changed the logic such that it consumes it as a KStream and then creates the KTable (toTable(...))
KTable<ProductXrefKey, ProductXref> productXrefStream = builder
.stream(TOPIC_NAME, Consumed.with(productXrefKeySerde, productXrefSerde))
.toTable(Materialized.<ProductXrefKey, ProductXref, KeyValueStore<Bytes, byte[]>>as(VIEW_NAME)
.withKeySerde(productXrefKeySerde)
.withValueSerde(productXrefSerde));
With this small change, data now gets deserialized (using SpecificAvroSerde<>) and serialized again using the provided ProductXrefKeySerde. Now also the prefix scan works and returns the records as expected.
Cannot create Dynamic Scenario Outline via Java call in Karate.
I can create a Dynamic Scenario Outline with "hard-coded" Json Array for example:
* def foobar = [{ 'type': 'app' }]
But when I attempt to generate the same Json Array from a Java class, I always get the following Karate warning(s) and the Dynamic Scenario Outline never executes:
WARN com.intuit.karate - ignoring dynamic expression, did not evaluate to list
-- OR --
WARN com.intuit.karate - ignoring dynamic expression list item 0, not map-like
I've tried using the Karate key-words 'def', 'string', 'json' as the var type but no luck. I've even tried hard-coding the same string shown above in the Java method with no luck.
I declare/call my Java class in 'Background:' and print what is given back and it "looks" correct.
Background:
* def IdaDataApiUtil = Java.type('data.IdaDataApiUtil')
* def foobar = IdaDataApiUtil.getClientExample('ida-sp')
* print foobar
I then try to utilize the JsonArray in my 'Example:' like such:
Examples:
| foobar |
At this point I get the above mentioned error(s), depending on what I've tried to return (JsonArray, JsonObject, Map, List).
If I simply use the hard-coded 'def':
* def foobar = [{ 'type': 'app' }]
It works as expected.
In my Java code I've tried various things:
Hard-coded Json string:
public static String getClientExample() {
return "[{ 'type': 'app' }]";
}
List:
public static List<String> getClientExample() {
List<String> list = new ArrayList<>();
list.add("'type': 'app'");
return list
}
Map:
public static Map<String, Object> getClientExample() {
Map<String, Object> map = new HashMap<>();
map.put("type", "app");
return map;
}
I've played with variations of key/values in both list/map with no luck. I've also tried with JSONObject/JSONArray but no luck there as well.
I feel like I'm missing something vary obvious but I can't see the forest through the trees at the moment...
I attempt to generate the same Json Array from a Java class,
What you return from the Java code should be a List<Map<String, Object>> and it should work fine. You have tried List<String> and that is the problem.
Read this section of the docs: https://github.com/intuit/karate#type-conversion
Another tip, you can try a "cast" to ensure it is in the JSON array form you need, so if you are too lazy to form properly nested Map-s, just return a raw JSON string from Java, and the below line will convert it correctly.
* json foobar = foobar
I am preparing Java object from json using ObjectMapper. Here is the json data
"dateTimeSent" : "LongValue"
Source and target both java classes have field is java.util.Date type.
I tried mapping of like this
<jb:value property="dtSent" data="dateTimeSent" decoder="Date">
<jb:decodeParam name="format">EEE MMM dd HH:mm:ss z yyyy</jb:decodeParam>
</jb:value>
In documentation it is mentioned that this decoder used to encode/decode from String to java.util.Date. Is that i need to write custom decoder for that. If yes please let me know how to write. I am new to smooks.
As smooks encode/decode from String to java.util.Date/java.sql.Date/ java.util.Calendar/java.sql.Time/java.sql.Timestamp. My use case, i have to decode from date to date. So i had created one more variable in source class with setter and getter like - private String modeifiedDateTimeSent
I am mapping with modifiedDateTimeSent variable in smooks-config.xml
<jb:value property="dtSent" data="modeifiedDateTimeSent" decoder="Date">
<jb:decodeParam name="format">yyyy-MM-dd HH:mm:ss</jb:decodeParam>
</jb:value>
Next, I have to set the value in the variable before mapping java class A to Class B.
Date modifiedDtTimeSent = order.getLr().getAdminSection().getDateTimeSent();
String modifiedDtTimeSentString = getDateAsString(modifiedDtTimeSent,"yyyy-MM-dd HH:mm:ss");
object.setModifieddatetimesent(modifiedDtTimeSentString);
Then finally, do your smooks java to java convertion -
Smooks smooks = new Smooks("smooks-config.xml");
ExecutionContext executionContext = smooks.createExecutionContext();
JavaSource source = new JavaSource(object);
JavaResult result = new JavaResult();
smooks.filterSource(executionContext, source, result);
ConvertedClass cc = (IimLocalResponse) result.getBean("xyz");
Hope this will help.
As part of a SSIS (2005) package I am calling a RESTful API through a script component (VB.NET). I am getting the records back OK. The problem I have though is processing the format it comes back in so I can output it (split into the separate output columns) within the data flow and write into a SQL table. Don't know where to start - anyone got any ideas? I have no control over the API so am stuck with this format.
Here is the schema as per the API documentation:
{StatusMessage : <string>, Unit : <string> [ { VehicleID : <int>, RegistrationNumber : <string>, DistanceTravelled : <double>} ] }
And here is a sample of the data returned (it comes back as a single line of text):
{"VehicleDistance":
[
{
"VehicleId":508767,
"RegistrationNumber":"BJ63 NYO",
"DistanceTravelled":0.09322560578584671
},
{
"VehicleId":508788,
"RegistrationNumber":"BJ63 NYL",
"DistanceTravelled":6.1591048240661621
},
{
"VehicleId":508977,
"RegistrationNumber":"PE12 LLC",
"DistanceTravelled":60.975761413574219
},
{
"VehicleId":510092,
"RegistrationNumber":"BJ64 FCY",
"DistanceTravelled":14.369173049926758
},
{
"VehicleId":510456,
"RegistrationNumber":"BJ63 NYY",
"DistanceTravelled":4.04599142074585
},
{
"VehicleId":513574,
"RegistrationNumber":"BL64 AEM",
"DistanceTravelled":302.150390625
}
],
"StatusMessage":null,
"Unit":"imperial",
"HttpStatus":200
}
This is Javscript Object Notation AKA JSON and you need to deserialise it. The problem is that using SSIS is tricky with third party tools that are popular (and fast) but VB.Net actually has a built in class to serialise and deserialise JSON called JavaScriptSerializer
First add a Reference to your project called System.Web.Extensions and then you can use the JavaScriptSerializer to deserialise your JSON.
I've put your JSON in a file for easier handling so first I have to...
Dim sJSON As String = ""
Using swReadFile As New System.IO.StreamReader("E:\JSON.txt")
sJSON = swReadFile.ReadToEnd()
End Using
The rest is the pertinent bit, so first add 2 Imports...
Imports System.Collections.Generic
Imports System.Web.Script.Serialization
Then for example we can...
Dim lvSerializer As JavaScriptSerializer = New JavaScriptSerializer()
lvSerializer.MaxJsonLength = 2147483644
Dim dictParsedJSONPairs As Dictionary(Of String, Object) = lvSerializer.Deserialize(Of Dictionary(Of String, Object))(sJSON)
If dictParsedJSONPairs.ContainsKey("VehicleDistance") AndAlso _
TypeOf dictParsedJSONPairs("VehicleDistance") Is ArrayList Then
Dim ArrayEntries As ArrayList = DirectCast(dictParsedJSONPairs("VehicleDistance"), ArrayList)
For Each ArrayEntry As Object In ArrayEntries
Dim DictEntry As Dictionary(Of String, Object) = DirectCast(ArrayEntry, Dictionary(Of String, Object))
If DictEntry.ContainsKey("VehicleId") Then Console.WriteLine("VehichleId:" & DictEntry("VehicleId"))
Next
End If
If dictParsedJSONPairs.ContainsKey("Unit") Then
Console.WriteLine("Unit is " & dictParsedJSONPairs.Item("Unit"))
End If
Clearly you should research JSON before launching into serious use. The object can be nested JSON (i.e. Dictionary(Of String, Object)), a number of some sort, a string or an ArrayList
It could be a bit late but you may want to have a look at json.net from Newtonsoft (website). The component provided contains .Net 2.0 version.
Using it in a SSIS script task is quite simple. I did it in SSIS2008 based on .Net 3.5 to parse JSON string like below. And according to the document, it should work for .Net 2.0 version as well.
//I assume you had obtained JSON in string format
string JasonBuffer;
//define Json object
JObject jObject = JObject.Parse(JasonBuffer);
//In my example the Json object starts with result
JArray Results = (JArray)jObject["result"];
//loop the result
foreach (JObject Result in Results)
{
//for simple object
JObject joCustomer = (JObject)Result["Customer"];
//do something...
//for complex object continue drill down
foreach (JObject joSection in Result["Sections"])
{
foreach (JObject joDepartment in joSection["Departments"])
{
foreach (JObject joItem in joDepartment["Items"])
{
}
You can find some actual code here: link
I would like to pass a complete JSON object to a java adapter in worklight. This adapter will call multiple other remote resources to fulfill the request. I would like to pass the json structure instead of listing out all of the parameters for a number of reasons. Invoking the worklight procedure works well. I pass the following as the parameter:
{ "parm1": 1, "parm2" : "hello" }
Which the tool is fine with. When it calls my java code, I see a object type of JSObjectConverter$1 being passed. In java debug, I can see the values in the object, but I do not see any documentation on how to do this. If memory serves me, the $1 says that it is an anonymous inner class that is being passed. Is there a better way to pass a json object/structure in adapters?
Lets assume you have this in adapter code
function test(){
var jsonObject = { "param1": 1, "param2" : "hello" };
var param2value = com.mycode.MyClass.parseJsonObject(jsonObject);
return {
result: param2value
};
}
Doesn't really matter where you're getting jsonObject from, it may come as a param from client. Worklight uses Rhino JS engine, therefore com.mycode.MyClass.parseJsonObject() function will get jsonObject as a org.mozilla.javascript.NativeObject. You can easily get obj properties like this
package com.mycode;
import org.mozilla.javascript.NativeObject;
public class MyClass {
public static String parseJsonObject(NativeObject obj){
String param2 = (String) NativeObject.getProperty(obj, "param2");
return param2;
}
}
To better explain what I'm doing here, I wanted to be able to pass a javascript object into an adapter and have it return an updated javascript object. Looks like there are two ways. The first it what I answered above a few days ago with serializing and unserializing the javascript object. The other is using the ScriptableObject class. What I wanted in the end was to use the adapter framework as described to pass in the javascript object. In doing so, this is what the Adapter JS-impl code looks like:
function add2(a) {
return {
result: com.ibm.us.roberso.Calculator.add2(a)
};
The javascript code in the client application calling the above adapter. Note that I have a function to test passing the javascript object as a parameter to the adapter framework. See the invocationData.parameters below:
function runAdapterCode2() {
// x+y=z
var jsonObject = { "x": 1, "y" : 2, "z" : "?" };
var invocationData = {
adapter : "CalculatorAdapter",
procedure : 'add2',
parameters : [jsonObject]
};
var options = {
onSuccess : success2,
onFailure : failure,
invocationContext : { 'action' : 'add2 test' }
};
WL.Client.invokeProcedure(invocationData, options);
}
In runAdapterCode2(), the javascript object is passed as you would pass any parameter into the adapter. When worklight tries to execute the java method it will look for a method signature of either taking an Object or ScriptableObject (not a NativeObject). I used the java reflection api to verify the class and hierarchy being passed in. Using the static methods on ScriptableObject you can query and modify the value in the object. At the end of the method, you can have it return a Scriptable object. Doing this will give you a javascript object back in the invocationResults.result field. Below is the java code supporting this. Please note that a good chunk of the code is there as part of the investigation on what object type is really being passed. At the bottom of the method are the few lines really needed to work with the javascript object.
#SuppressWarnings({ "unused", "rawtypes" })
public static ScriptableObject add2(ScriptableObject obj) {
// code to determine object class being passed in and its heirarchy
String result = "";
Class objClass = obj.getClass();
result = "objClass = " + objClass.getName() + "\r\n";
result += "implements=";
Class[] interfaces = objClass.getInterfaces();
for (Class classInterface : interfaces) {
result += " " + classInterface.getName() ;
}
result += "\r\nsuperclasses=";
Class superClass = objClass.getSuperclass();
while(superClass != null) {
result += " " + superClass.getName();
superClass = superClass.getSuperclass();
}
// actual code working with the javascript object
String a = (String) ScriptableObject.getProperty((ScriptableObject)obj, "z");
ScriptableObject.putProperty((ScriptableObject)obj, "z", new Long(3));
return obj;
}
Note that for javascript object, a numeric value is a Long and not int. Strings are still Strings.
Summary
There are two ways to pass in a javascript object that I've found so far.
Convert to a string in javascript, pass string to java, and have it reconstitute into a JSONObject.
Pass the javascript object and use the ScriptableObject classes to manipulate on the java side.