I'm writing a microservice that takes a json object as input. This json object is only partially known and therefore the struct to which I map it looks like this:
#[derive(Serialize, Deserialize)]
pub struct Incoming {
uri: String,
payload: serde_json::Value,
id: String
}
I then want to publish this to a RabbitMQ queue so I serialize it using bincode:
let incoming = serde_json::from_slice::<Incoming>(&incoming).expect("Fail to serialize");
// This line fails:
bincode::serialize(&incoming).expect("Failed to deserialize to binary");
The receiving service (the consumer) fails to deserialize this (even though it has the exact same model) and results in Err(DeserializeAnyNotSupported).
From my understanding this comed from the serde_json::Value-part of the struct.
So how do I serialize a partially unknown JSON-object to binary in order to deserialize it on the receiving service?
Related
I have a class that gets serialized for network traffic.
#Serializable
data class Packet(val dataType: String, val payload: Any)
I've used Java serialization to send it over the wire. The receiver can't know the type of the payload but Java deserializes it just fine, and then I use when(dataType) as a lookup to correctly cast the Any object to its correct type. Easy breazy.
But Kotlinx Serialization (with ProtoBuf) is a stickler about this Any type for reasons that aren't obvious to me. I can't register a serializer for Any. In the docs they recommend a polymorphic approach, which sorta works but you have to make the packet typed:
data class Packet<out T : Any>(val dataType: String, val payload: T) : SomeBaseClass<T>
but this kinda sucks because it weighs down a lot of code paths with inline reified typing, plus this doesn't solve that the receiving end won't know what type to try to deserialize the payload as without being to look at the dataType field first.
This is the worst catch-22. The framework won't ignore the payload: Any field (gives a compile error) and I can't even write a custom serializer because defining an element of type Any in a customer serializer (for the descriptor) gives the same run-time error of "no serializer registered for Any."
I've used Java serialization to send it over the wire. The receiver can't know the type of the payload but Java deserializes it just fine, and then I use when(dataType) as a lookup to correctly cast the Any object to its correct type. Easy breazy.
This is because java serialization is rather primitive - there is only one way to serialize (and hence to deserialize) an object. In kotlinx.serialization each class can have its own serialization strategy (or even several ones). And this flexibility comes with a price.
Serialization of Any could be handled (for declared list of its subclasses), but dynamic determintaion of deserialization strategy based on dataType field of partly deseriazed object is impossible in general case, because there is no guarantee that dataType field will be deserialized first. Some serialization formats (like JSON or Protobuf) have unordered schema. It could happen that payload is about to be deserialized before dataType, and Decoder interface doesn't allow to go back/make several passes.
If you're sure about the order of properties in your serialization format/message (or just feel lucky) you may go with the following custom serializer:
import kotlinx.serialization.*
import kotlinx.serialization.descriptors.*
import kotlinx.serialization.encoding.*
#Serializable(with = PacketSerializer::class)
data class Packet(val dataType: String, val payload: Any)
object PacketSerializer : KSerializer<Packet> {
override val descriptor: SerialDescriptor = buildClassSerialDescriptor("Packet") {
element("dataType", serialDescriptor<String>())
element("payload", buildClassSerialDescriptor("Any"))
}
#Suppress("UNCHECKED_CAST")
private val dataTypeSerializers: Map<String, KSerializer<Any>> =
mapOf(
"String" to serializer<String>(),
"Int" to serializer<Int>(),
//list them all
).mapValues { (_, v) -> v as KSerializer<Any> }
private fun getPayloadSerializer(dataType: String): KSerializer<Any> = dataTypeSerializers[dataType]
?: throw SerializationException("Serializer for class $dataType is not registered in PacketSerializer")
override fun serialize(encoder: Encoder, value: Packet) {
encoder.encodeStructure(descriptor) {
encodeStringElement(descriptor, 0, value.dataType)
encodeSerializableElement(descriptor, 1, getPayloadSerializer(value.dataType), value.payload)
}
}
#ExperimentalSerializationApi
override fun deserialize(decoder: Decoder): Packet = decoder.decodeStructure(descriptor) {
if (decodeSequentially()) {
val dataType = decodeStringElement(descriptor, 0)
val payload = decodeSerializableElement(descriptor, 1, getPayloadSerializer(dataType))
Packet(dataType, payload)
} else {
require(decodeElementIndex(descriptor) == 0) { "dataType field should precede payload field" }
val dataType = decodeStringElement(descriptor, 0)
val payload = when (val index = decodeElementIndex(descriptor)) {
1 -> decodeSerializableElement(descriptor, 1, getPayloadSerializer(dataType))
CompositeDecoder.DECODE_DONE -> throw SerializationException("payload field is missing")
else -> error("Unexpected index: $index")
}
Packet(dataType, payload)
}
}
}
I have a question related to Jackson and polymorphism: is there a way to deserialize a JSON string without specifying a type?
Assuming I don't own this message (e.g., external API) and I have two separate messages that come in at separate times:
{
"responseCode": 200
"responseMessage": "You did something successfully"
}
{
"errorCode": 401
"errorDescription": "Permission denied"
}
And I want to deserialize this message with some data classes that I created based on these messages through polymorphism (see abstract class in next code block):
data class MyDataClass(
val responseCode: Int,
val responseMessage: String
): MyAbstractClass()
data class MyOtherDataClass(
val errorCode: Int,
val errorDescription: String
): MyAbstractClass()
And I am resolving these messages through a function that will use the Jackson Object Mapper to deserialize the stringified JSON payload:
#JsonSubTypes(
JsonSubTypes.Type(value = MyDataClass::class),
JsonSubTypes.Type(value = MyOtherDataClass::class)
)
#JsonIgnoreProperties(ignoreProperties = true)
abstract class MyAbstractClass
fun receiveMessage(message: String) {
val convertedMessage = jacksonObjectMapper().readValue<MyAbstractClass>(message)
log.info(convertedMessage)
/* prints either:
MyDataClass(responseCode=200, responseMessage=You did something successfully)
OR
MyOtherDataClass(errorCode=401, errorDescription=Permission denied)
*/
}
But since I haven't described how to identify the data class (using #JsonTypeInfo), it fails.
To repeat, I am curious if there is a way that I can deserialize the incoming message to one of my polymorphic types without having to specify the #JsonTypeInfo. Or if I must describe the #JsonTypeInfo, how would I do this with no similarities between the two child classes of MyAbstractClass?
I would write a custom deserializer which takes it as a generic JSONObject or the like. Then I'd check if a differentiating key exists. For example:
// pseudocode
when (json: JSONObject) {
hasKey("responseCode") -> // deserialize as MyDataClass
hasKey("errorCode") -> // deserialize as MyOtherDataClass
}
I'm trying to separate my code into models and serializers with the idea that there be defined serializers that handles all json responsibilities, i.e. separation of concerns. I also want to be able to call a model object obj.Serialize() to get the serializer struct obj that I can then marshal. Therefore, I've come up with the following design. To avoid circular import I had to use interfaces in my serializers which leads to using getters in my models. I've read that getters/setters aren't idiomatic go code and I would prefer not to have "boilerplate" getter code all over my models. Is there a better solution to what I want to accomplish, keeping in mind I want separation of concerns and obj.Serialize()?
src/
models/
a.go
serializers/
a.go
models/a.go
import "../serializers"
type A struct {
name string
age int // do not marshal me
}
func (a *A) Name() string {
return a.name
}
// Serialize converts A to ASerializer
func (a *A) Serialize() interface{} {
s := serializers.ASerializer{}
s.SetAttrs(a)
return s
}
serializers/a.go
// AInterface used to get Post attributes
type AInterface interface {
Name() string
}
// ASerializer holds json fields and values
type ASerializer struct {
Name `json:"full_name"`
}
// SetAttrs sets attributes for PostSerializer
func (s *ASerializer) SetAttrs(a AInterface) {
s.Name = a.Name()
}
It looks like you are actually trying to translate between your internal structs and json. We can start by taking advantage of the json library.
If you want certain libraries to handle your struct fields in certain ways, there are tags. This example shows how json tags tell json to never marshal the field age into json, and to only add the field jobTitle if it is not empty, and that the field jobTitle is actually called title in json. This renaming feature is very useful when structs in go contain capitalized (exported) fields, but the json api you're connecting to uses lowercase keys.
type A struct {
Name string
Age int `json:"-"`// do not marshal me
location string // unexported (private) fields are not included in the json marshal output
JobTitle string `json:"title,omitempty"` // in our json, this field is called "title", but we only want to write the key if the field is not empty.
}
If you need to precompute a field, or simply add a field in your json output of a struct that isn't a member of that struct, we can do that with some magic. When json objects are decoded again into golang structs, fields that don't fit (after checking renamed fields and capitalization differences) are simply ignored.
// AntiRecursionMyStruct avoids infinite recursion in MashalJSON. Only intended for the json package to use.
type AntiRecursionMyStruct MyStruct
// MarshalJSON implements the json.Marshaller interface. This lets us marshal this struct into json however we want. In this case, we add a field and then cast it to another type that doesn't implement the json.Marshaller interface, and thereby letting the json library marshal it for us.
func (t MyStruct) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
AntiRecursionMyStruct
Kind string // the field we want to add, in this case a text representation of the golang type used to generate the struct
}{
AntiRecursionMyStruct: AntiRecursionMyStruct(t),
Kind: fmt.Sprintf("%T", MyStruct{}),
})
}
Keep in mind that json will only include your exported (capitalized) struct members. I've made this misstake multiple times.
As a general rule, if something seems too complicated, there's probably a better way to do it.
I want to deserialize a configuration structure like this:
#[deriving(Clone, PartialEq, Decodable, Show)]
pub struct Config {
network1: Network,
network2: Network
}
#[deriving(Clone, PartialEq, Decodable, Show)]
pub struct Network {
interface: Option<String>,
prefer: Option<AddressPreference>,
address: Option<String>,
port: u16
}
#[deriving(Clone, PartialEq, Show)]
pub enum AddressPreference {
IPv4, IPv6, Any
}
It forms a tree of structs and enums. But the decoder I want to use does not support enums, so I thought that I will be able to represent the enum as a string by implementing Decodable for the enum which matches on strings. But I don't know how to do it properly.
The main problem is that I don't know how to write Decodable implementation which handles errors correctly. If I write
impl<E, D: Decoder<E>> Decodable<D, E> for AddressPreference { ... }
the only way I can indicate parsing error is fail!() - I don't know actual error type, so I can't create its instance. This is unacceptable.
Buf if I write
impl Decodable<toml::Decoder, toml::Error> for AddressPreference { ... }
that is, binding Decodable instance to concrete decoder type (which is fine for me), I lose automatic deriving of outer structures, because they are implemented for generic decoder and error:
src/sfc/config.rs:11:30: 11:39 error: expected serialize::serialize::Decodable<__D,__E>, but found serialize::serialize::Decodable<rust-toml::Decoder,rust-toml::Error> (expected type parameter but found struct rust-toml::Decoder)
src/sfc/config.rs:11 #[deriving(Clone, PartialEq, Decodable, Show)]
What should I do in this case? I guess, the proper answer in this particular case is submitting a pull request to the library enabling support for enums serialization, but I think that the question is more broad. What it the library supported enums, but I wanted to customize decoding nonetheless?
When trying to deserialize a record member of type Header option returned from a JSON string, I get the following exception:
The data contract type
'Microsoft.FSharp.Core.FSharpOption`1[[MyWeb.Controllers.Header,
MyWeb.Controllers, Version=0.0.0.0, Culture=neutral,
PublicKeyToken=null]]' cannot be deserialized because the required
data member 'value' was not found.
I'm serializing/deserializing a Message record:
[<DataContract>]
type Header =
{ [<DataMember>] mutable ID : int
[<DataMember>] mutable Description : string }
[<DataContract>]
type Message =
{ [<DataMember>] mutable ID : int
[<DataMember>] mutable Header : Header option
[<DataMember>] mutable SenderID : string
[<DataMember>] mutable ReceiverID : string }
The code I use to deserialize the JSON:
let deserializeJson<'a> (s:string) =
use ms = new MemoryStream(ASCIIEncoding.ASCII.GetBytes s)
let serialize = DataContractJsonSerializer(typeof<'a>)
serialize.ReadObject ms :?> 'a
And the actual raw JSON result:
"Message":
{
"ID":13,
"Header": { "Value":{"ID":21,"Description":"some"}},
"SenderID":"312345332423",
"ReceiverID":"16564543423"
}
The question: how do I deserialize a 'a option?
Update
ASP.NET MVC uses JavaScriptSerializer by default to serialize objects and I'm using DataContractJsonSerializer to deserialize.
For some reason it seems DataContractJsonSerializer can't read the JSON string unless the Value property for the option is in lowercase (as pointed out by #svick). A dirty fix would be to replace "Value" with "value" in the returned JSON string, but I've chosen to go with Roberts' suggestion.
If you were to hop over to using json.net (a.k.a Newtonsoft.Json) instead of the json serializer that comes with the .NET framework, then you could use the option serializer I built to allow me to work more effectively with ravendb. Should just be a matter of registering the convert with the serializer and calling Deserialize.