Retrieving data from CBOR ByteArray - kotlin

I am trying to serialize a map into CBOR in Kotlin with the Jackson CBOR Dataformats Library, this works fine if the key is a String , I can retrieve the value of that key easily but when the key in an Int, it returns null to me for every get I do, If I print out the output from values(), it gives me all values from all keys.
Code looks like this :
val mapper = CBORMapper()
val map = HashMap<Any,Any>()
map[123] = intArrayOf(22,67,2)
map[456] = intArrayOf(34,12,1)
val cborData = mapper.writeValueAsBytes(map)
println(cborData.toHex())
val deserialized = mapper.readValue(cborData, HashMap<Any,Any>().javaClass)
println(deserialized.get(123)) // returns null
println(values()) // returns all values

Try to iterate over keys and check the type:
deserialized.keys.iterator().next().javaClass
Above code, in your case should print:
123 - class java.lang.String
456 - class java.lang.String
And:
println(deserialized.get("123"))
prints:
[22, 67, 2]
Take a look on documentation:
Module extends standard Jackson streaming API (JsonFactory,
JsonParser, JsonGenerator), and as such works seamlessly with all the
higher level data abstractions (data binding, tree model, and
pluggable extensions).
You can force type using Kotlin's readValue method:
import com.fasterxml.jackson.module.kotlin.readValue
and use it like this:
val deserialized = mapper.readValue<Map<Int, IntArray>>(cborData)
deserialized.keys.forEach { key -> println("$key - ${key.javaClass}") }
println(Arrays.toString(deserialized[123]))
Above code prints:
456 - int
123 - int
[22, 67, 2]
See also:
How to use jackson to deserialize to Kotlin collections

Related

How to deserialize JSON from Kafka Consumer Record

I'm looking to access some fields on a Kafka Consumer record. I'm able to receive the event data which is a Java object i.e ConsumerRecord(topic = test.topic, partition = 0, leaderEpoch = 0, offset = 0, CreateTime = 1660933724665, serialized key size = 32, serialized value size = 394, headers = RecordHeaders(headers = [], isReadOnly = false), key = db166cbf1e9e438ab4eae15093f89c34, value = {"eventInfo":...}).
I'm able to access the eventInfo values which comes back as a json string. I'm fairly new to Kotlin and using Kafka so I'm not entirely sure if this is correct but I'm looking to basically access the fields in value but I can't get rid of an error that appears when trying to use mapper.readValue which is:
None of the following functions can be called with the arguments supplied.
import com.afterpay.shop.favorites.model.Product
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import org.apache.avro.generic.GenericData.Record
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.springframework.kafka.annotation.KafkaListener
import org.springframework.kafka.support.Acknowledgment
import org.springframework.stereotype.Component
#Component
class KafkaConsumer {
#KafkaListener(topics = ["test.topic"], groupId = "group-id")
fun consume(consumerRecord: ConsumerRecord<String, Any>, ack: Acknowledgment) {
val mapper = jacksonObjectMapper()
val value = consumerRecord.value()
val record = mapper.readValue(value, Product::class.java)
println(value)
ack.acknowledge()
}
}
Is this the correct way to accomplish this?
First, change ConsumerRecord<String, Any> to ConsumerRecord<String, Product>, then change value.deserializer in your consumer config/factory to use JSONDeserializer
Then your consumerRecord.value() will already be a Product instance, and you don't need an ObjectMapper
https://docs.spring.io/spring-kafka/docs/current/reference/html/#json-serde
Otherwise, if you use StringDeserializer, change Any to String so that the mapper.readValue argument types are correct.

Using map function to transform an item in a list

Kotlin 1.4.21
I have a list of Products and I want to transform the price to a formatted string. However, the price doesn't change to the formatted one.
i.e.
orginal price: 1658
expected: "1,658.00"
actual: 1658
This is the class structure
data class Product(
val productName: String,
val price: Double)
Here I am using my list of projects and I want to transform the price.
listOfProjects.map {
it.price.toAmount(2) // extension function that formats to 2 decimal places and converts to a string
}
if (listOfProjects.isNotEmpty()) {
// Do something with the list. However, the price hasn't changed
}
You probably are expecting it to perform in-place formatting. Maybe if you assign an output variable you would be able to get it:
val result = listOfProjects.map { it.price.toAmount(2) }
println(result)
Or if you add a val formattedPrice: String property on the Project class you can define it like this:
val formattedPrice: String = price.toAmount(2)
and then you can use this property in a toString() method for example.
Have you ever tried :
import java.text.DecimalFormat
val dec = DecimalFormat("###.###,##")

Why can't Jackson deserialize this JSON?

I'm using Jackson to parse an ElasticSearch document into following data class
data class ElasticCity(
val id: Long,
val regionId: Long,
val countryIso: String,
val isFeatured: Boolean?
) {
// For now Jackson does not support this for constructor parameters https://github.com/FasterXML/jackson-databind/issues/562
#JsonAnySetter
val names: MutableMap<String, String> = mutableMapOf()
}
However I'm getting following error (formatting mine)
com.fasterxml.jackson.module.kotlin.MissingKotlinParameterException:
Instantiation of [simple type, class net.goout.locations.model.ElasticCity] value failed
for JSON property country_iso due to missing (therefore NULL) value for creator parameter countryIso which is a non-nullable type
at [Source: (byte[])
"{
"name.cs":"Brno",
"countryIso":"CZ",
"regionId":85682423,
"timezone":"Europe/Prague",
"name.de":"Brünn",
"name.sk":"Brno",
"id":101748109,
"isFeatured":true,
"name.pl":"Brno",
"name.en":"Brno"
}";
line: 1, column: 186] (through reference chain: net.goout.locations.model.ElasticCity["country_iso"])
Clearly the key countryIso is present in the JSON but for some reason Jackson complains a key country_iso is missing. Why? How can I fix this?
try adding
data class ElasticCity(
val id: Long,
val regionId: Long,
#JsonProperty(value = "countryIso") val countryIso: String,
val isFeatured: Boolean?
)
Jackson mapper implicitly converts non start caps characters _
If you want to fix this at multiple places then take a look at
#JsonNaming(PropertyNamingStrategy..
https://www.programcreek.com/java-api-examples/?api=com.fasterxml.jackson.databind.PropertyNamingStrategy

Groovy maps with Integer keys - 'getAt' in DefaultGroovyMethods cannot be applied to (java.lang.Integer)

I've just started to practice Groovy and I have a question related to maps and IDEA IDE.
Why IDEA shows me the notification below when I try to use Integer as a key for a map? This simple Groovy script works fine and print correct result.
list = [4, 7, 3, 7, 7, 1, 4, 2, 4, 2, 7, 5]
map = [:]
list.each {
t = map[(it)]
map[(it)] = t != null ? t + 1 : 1
}
map.each {key, value -> if (value == 1) println key}
It is caused because IntelliJ IDEA sees map variable as Object - it seems like IDEA does not follow type inference if static type or keyword def is missing in front of the variable. If you take a look at DefaultGroovyMethods you will see that there is only one method getAt implemented for Object type:
public static Object getAt(Object self, String property) {
return InvokerHelper.getProperty(self, property);
}
This is why IDEA warns you about missing method getAt(Object self, Integer property) because it is not aware that map is actually a Map and not an Object.
Please follow the official Groovy's guideline that says:
Variables can be defined using either their type (like String) or by using the keyword def:
String x
def o
Source: http://docs.groovy-lang.org/latest/html/documentation/core-semantics.html#_variable_definition
If you define your variable as
def map = [:]
IntelliJ wont complain anymore.

Avro specific vs generic record types - which is best or can I convert between?

We’re trying to decide between providing generic vs specific record formats for consumption by our clients
with an eye to providing an online schema registry clients can access when the schemas are updated.
We expect to send out serialized blobs prefixed with a few bytes denoting the version number so schema
retrieval from our registry can be automated.
Now, we’ve come across code examples illustrating the relative adaptability of the generic format for
schema changes but we’re reluctant to give up the type safety and ease-of-use provided by the specific
format.
Is there a way to obtain the best of both worlds? I.e. could we work with and manipulate the specific generated
classes internally and then have them converted them to generic records automatically just before serialization?
Clients would then deserialize the generic records (after looking up the schema).
Also, could clients convert these generic records they received to specific ones at a later time? Some small code examples would be helpful!
Or are we looking at this all the wrong way?
What you are looking for is Confluent Schema registry service and libs which helps to integrate with this.
Providing a sample to write Serialize De-serialize avro data with a evolving schema. Please note providing sample from Kafka.
import io.confluent.kafka.serializers.KafkaAvroDeserializer;
import io.confluent.kafka.serializers.KafkaAvroSerializer;
import org.apache.avro.generic.GenericRecord;
import org.apache.commons.codec.DecoderException;
import org.apache.commons.codec.binary.Hex;
import java.util.HashMap; import java.util.Map;
public class ConfluentSchemaService {
public static final String TOPIC = "DUMMYTOPIC";
private KafkaAvroSerializer avroSerializer;
private KafkaAvroDeserializer avroDeserializer;
public ConfluentSchemaService(String conFluentSchemaRigistryURL) {
//PropertiesMap
Map<String, String> propMap = new HashMap<>();
propMap.put("schema.registry.url", conFluentSchemaRigistryURL);
// Output afterDeserialize should be a specific Record and not Generic Record
propMap.put("specific.avro.reader", "true");
avroSerializer = new KafkaAvroSerializer();
avroSerializer.configure(propMap, true);
avroDeserializer = new KafkaAvroDeserializer();
avroDeserializer.configure(propMap, true);
}
public String hexBytesToString(byte[] inputBytes) {
return Hex.encodeHexString(inputBytes);
}
public byte[] hexStringToBytes(String hexEncodedString) throws DecoderException {
return Hex.decodeHex(hexEncodedString.toCharArray());
}
public byte[] serializeAvroPOJOToBytes(GenericRecord avroRecord) {
return avroSerializer.serialize(TOPIC, avroRecord);
}
public Object deserializeBytesToAvroPOJO(byte[] avroBytearray) {
return avroDeserializer.deserialize(TOPIC, avroBytearray);
} }
Following classes have all the code you are looking for.
io.confluent.kafka.serializers.KafkaAvroDeserializer;
io.confluent.kafka.serializers.KafkaAvroSerializer;
Please follow the link for more details :
http://bytepadding.com/big-data/spark/avro/avro-serialization-de-serialization-using-confluent-schema-registry/
Can I convert between them?
I wrote the following kotlin code to convert from a SpecificRecord to GenericRecord and back - via JSON.
PositionReport is an object generated off of avro with the avro plugin for gradle - it is:
#org.apache.avro.specific.AvroGenerated
public class PositionReport extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
...
The functions used are below
/**
* Encodes a record in AVRO Compatible JSON, meaning union types
* are wrapped. For prettier JSON just use the Object Mapper
* #param pos PositionReport
* #return String
*/
private fun PositionReport.toAvroJson() : String {
val writer = SpecificDatumWriter(PositionReport::class.java)
val baos = ByteArrayOutputStream()
val jsonEncoder = EncoderFactory.get().jsonEncoder(this.schema, baos)
writer.write(this, jsonEncoder)
jsonEncoder.flush()
return baos.toString("UTF-8")
}
/**
* Converts from Genreic Record into JSON - Seems smarter, however,
* to unify this function and the one above but whatevs
* #param record GenericRecord
* #param schema Schema
*/
private fun GenericRecord.toAvroJson(): String {
val writer = GenericDatumWriter<Any>(this.schema)
val baos = ByteArrayOutputStream()
val jsonEncoder = EncoderFactory.get().jsonEncoder(this.schema, baos)
writer.write(this, jsonEncoder)
jsonEncoder.flush()
return baos.toString("UTF-8")
}
/**
* Takes a Generic Record of a position report and hopefully turns
* it into a position report... maybe it will work
* #param gen GenericRecord
* #return PositionReport
*/
private fun toPosition(gen: GenericRecord) : PositionReport {
if (gen.schema != PositionReport.getClassSchema()) {
throw Exception("Cannot convert GenericRecord to PositionReport as the Schemas do not match")
}
// We will convert into JSON - and use that to then convert back to the SpecificRecord
// Probalby there is a better way
val json = gen.toAvroJson()
val reader: DatumReader<PositionReport> = SpecificDatumReader(PositionReport::class.java)
val decoder: Decoder = DecoderFactory.get().jsonDecoder(PositionReport.getClassSchema(), json)
val pos = reader.read(null, decoder)
return pos
}
/**
* Converts a Specific Record to a Generic Record (I think)
* #param pos PositionReport
* #return GenericData.Record
*/
private fun toGenericRecord(pos: PositionReport): GenericData.Record {
val json = pos.toAvroJson()
val reader : DatumReader<GenericData.Record> = GenericDatumReader(pos.schema)
val decoder: Decoder = DecoderFactory.get().jsonDecoder(pos.schema, json)
val datum = reader.read(null, decoder)
return datum
}
There are a couple difference however between the two:
Fields in the SpecificRecord that are of Instant type will be encoded in the GenericRecord as long and Enums are slightly different
So for example in my unit test of this function time fields are tested like this:
val gen = toGenericRecord(basePosition)
assertEquals(basePosition.getIgtd().toEpochMilli(), gen.get("igtd"))
And enums are validated by string
val gen = toGenericRecord(basePosition)
assertEquals(basePosition.getSource().toString(), gen.get("source").toString())
So to convert between you can do:
val gen = toGenericRecord(basePosition)
val newPos = toPosition(gen)
assertEquals(newPos, basePosition)