I need to set up a serialization/deserialization mechanism for a polymorphic class hierarchy that also includes primitives and nulls. There are container classes containing collections with polymorphic objects, primitives, and nulls. And, the subclasses for these objects are spread across modules (therefore sealed is not an option).
I have been reading through the kotlinx.serialization polymorphism docs trying to come up with a solution. I've been able to make some incremental progress by working through that tutorial but I seem to still be hitting a wall when I try to put everything together.
The code I am posting here is a minimal example that brings together everything I need. If I can get this example to work, that should cover everything I need for my real project. This example does run without error but introduces some unnecessary readability and efficiency issues.
All classes in my custom class hierarchy are serializable data classes. The outermost container object that needs to be serialized/deserialized is a map wrapper. This map has keys which are each an instance of one of these data classes. And the values of this map can be primitives, nulls, or instances of one of my data classes. I think my main challenge here is to include those primitives and nulls in my polymorphic serialization in a clean way.
The goal of my code below is to represent this problem in the simplest way possible and to serialize and deserialize one container object successfully.
There are two main issues in the code:
I've had to replace null with FakeNull. Without this, I get null cannot be cast to non-null type kotlin.Any. This will reduce the readability and simplicity of my code and I suspect it could decrease efficiency as well.
I've had to add StringClassSerializer and DoubleClassSerializer and wrapper classes. I would also need to add serializers like these for every primitive class. If I don't register these primitives as subclasses of Any, I get Class 'String' is not registered for polymorphic serialization in the scope of 'Any'.. And if I try to register them with their default serializers (like subclass(String::class, String.serializer())) I get Serializer for String of kind STRING cannot be serialized polymorphically with class discriminator.. The problem with using serializers like StringClassSerializer and wrappers like StringWrapper is that it removes the efficiency and readability benefits of using primitives.
The json comes out looking like:
{"type":"MapContainer","map":[{"type":"SubA","data":1.0},{"type":"StringWrapper","s":"valueA"},{"type":"SubB","data":2.0},{"type":"DoubleWrapper","d":2.0},{"type":"SubB","data":3.0},{"type":"SubA","data":1.0},{"type":"SubB","data":4.0},{"type":"matt.play.FakeNull"}]}
I don't like the way this looks. I want the nulls to simply be null and the primitives to simply be primitives.
import kotlinx.serialization.KSerializer
import kotlinx.serialization.PolymorphicSerializer
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.descriptors.buildClassSerialDescriptor
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder
import kotlinx.serialization.json.Json
import kotlinx.serialization.modules.SerializersModule
import kotlinx.serialization.modules.polymorphic
import kotlinx.serialization.modules.subclass
import kotlin.collections.set
#Serializable
abstract class SuperClass
#Serializable
#SerialName("SubA")
data class SubA(val data: Double): SuperClass()
#Serializable
#SerialName("SubB")
data class SubB(val data: Double): SuperClass()
#Serializable
#SerialName("MapContainer")
data class MapContainer<K: SuperClass, V>(val map: Map<K, V>): Map<K, V> by map
#Serializable
#SerialName("StringWrapper")
data class StringWrapper(val s: String)
#Serializable
#SerialName("DoubleWrapper")
data class DoubleWrapper(val d: Double)
object StringClassSerializer: KSerializer<String> {
override val descriptor = buildClassSerialDescriptor("string")
override fun deserialize(decoder: Decoder) = decoder.decodeSerializableValue(StringWrapper.serializer()).s
override fun serialize(encoder: Encoder, value: String) =
encoder.encodeSerializableValue(StringWrapper.serializer(), StringWrapper(value))
}
object DoubleClassSerializer: KSerializer<Double> {
override val descriptor = buildClassSerialDescriptor("double")
override fun deserialize(decoder: Decoder) = decoder.decodeSerializableValue(DoubleWrapper.serializer()).d
override fun serialize(encoder: Encoder, value: Double) =
encoder.encodeSerializableValue(DoubleWrapper.serializer(), DoubleWrapper(value))
}
#Serializable
object FakeNull
fun main() {
val theMap = mutableMapOf<SuperClass, Any?>()
theMap[SubA(1.0)] = "valueA"
theMap[SubB(2.0)] = 2.0
theMap[SubB(3.0)] = SubA(1.0)
theMap[SubB(4.0)] = FakeNull /*wish I could make this just `null`*/
val theMapContainer = MapContainer(theMap)
val format = Json {
allowStructuredMapKeys = true
ignoreUnknownKeys = true
serializersModule = SerializersModule {
polymorphic(SuperClass::class) {
subclass(SubA::class)
subclass(SubB::class)
}
polymorphic(Any::class) {
/*I wish I could remove all of this primitive wrapper stuff*/
default {
when (it) {
StringWrapper::class.simpleName -> StringClassSerializer
DoubleWrapper::class.simpleName -> DoubleClassSerializer
else -> throw RuntimeException("unknown type: ${it}?")
}
}
subclass(String::class, StringClassSerializer)
subclass(Double::class, DoubleClassSerializer)
subclass(SubA::class)
subclass(SubB::class)
subclass(FakeNull::class)
}
polymorphic(
MapContainer::class, MapContainer::class, actualSerializer = MapContainer.serializer(
PolymorphicSerializer(SuperClass::class),
PolymorphicSerializer(Any::class)
) as KSerializer<MapContainer<*, *>>
)
}
}
val encoded = format.encodeToString(PolymorphicSerializer(MapContainer::class), theMapContainer)
println("\n\n${encoded}\n\n")
val decoded = format.decodeFromString(PolymorphicSerializer(MapContainer::class), encoded)
if (theMapContainer != decoded) {
throw RuntimeException("the decoded object is not the same as the original")
} else {
println("success")
}
}
Primitives (such as strings, numbers, and enums) by default are serialized as JSON primitives (e.g., "answer" or 42), not JSON objects ({ ... }). This is why they don't support polymorphic serialization; there is no "space" to place the type information in (the class discriminator).
There is no JSON object to place the class discriminator in, e.g., {"type": "fully.qualified.Name"} by default.
But, kotlinx serialization does allow you to write custom serializers, which allows you to work around this. I wrote a custom serializer for enums since I wanted to register enums as concrete types in polymophic serialization. It sounds like you should be able to do something similar. (Disclosure: I only read your problem description in detail; not your ongoing attempts/solution.)
A serializer which supports registering [Enum]s as subclasses in polymorphic serialization when class discriminators are used.
When class discriminators are used, an enum is not encoded as a structure which the class discriminator can be added to.
An exception is thrown when initializing [Json]: " "Serializer for of kind ENUM cannot be serialized polymorphically with class discriminator."
This serializer encodes the enum as a structure with a single value holding the enum value.
Use this serializer to register the enum in the serializers module, e.g.:
subclass( <enum>::class, PolymorphicEnumSerializer( <enum>.serializer() )
This custom serializer can possibly be generalized to any primitive type and thus support your use case.
Related
In Kotlin how is possible to convert a data class from its string representation back to the actual data class without having to manually parse the string?
for example, I have the next data class where both Interest and EmploymentType are enums, being the second element a List.
data class DataFilter(val mainInterest: Interest, val employments: List<EmploymentType>)
with toString I can get the contents its string representation, but if I want to get it back to a data class, how is it done?
For this purpose, you need to use a serialization library. I recommend you use Kotlinx.serialization library. This will help you convert the data object into a JSON String and convert it back to the data object easily.
You can follow the guideline to setup Kotlinx.serialization library.
After the setup is done, look at the following example:
I will define both enum classes Interest and EmploymentType, and the DataFilter class:
import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
enum class Interest {
SPORTS,
BOOKS,
TRAVEL,
FOOD,
TECHNOLOGY,
ART,
OTHER
}
enum class EmploymentType {
FULL_TIME,
PART_TIME,
CONTRACT,
FREELANCE,
OTHER
}
#Serializable //You have to annotate your class as Serializable to make it works.
data class DataFilter(
val mainInterest: Interest,
val employment: List<EmploymentType>
)
Notice that Interest and EmploymentType are not annotated as #Serializable because enums are Serializable by default, but if you defined any other normal classes to use with DataFilter, then you have to annotate them as well.
Next, I will show you how conversions work:
fun main() {
val filter = DataFilter(
mainInterest = Interest.BOOKS,
employment = listOf(EmploymentType.FULL_TIME, EmploymentType.FREELANCE)
)
// Converting filter object into JSON String:
val filterAsJsonString = Json.encodeToString(filter)
println(filterAsJsonString) //prints {"mainInterest":"BOOKS","employment":["FULL_TIME","FREELANCE"]}
//Convert JSON String back to data object:
val filterAsDataObject: DataFilter = Json.decodeFromString(filterAsJsonString)
println(filterAsDataObject) //prints DataFilter(mainInterest=BOOKS, employment=[FULL_TIME, FREELANCE])
}
I'm trying to create a serializer using kotlinx.serialization for Compose Desktop classes, I have this :
#Serializer(forClass = MutableState::class)
class MutableStateSerializer<T>(private val dataSerializer: KSerializer<T>) : KSerializer<MutableState<T>> {
override fun deserialize(decoder: Decoder) = mutableStateOf(decoder.decodeSerializableValue(dataSerializer))
override val descriptor: SerialDescriptor = dataSerializer.descriptor
override fun serialize(encoder: Encoder, value: MutableState<T>) = encoder.encodeSerializableValue(dataSerializer, value.value)
}
That should be used for instances of MutableState class (as the #Serializer annotation says), but I have to put an explicit serializer for each properties otherwise I get this error :
xception in thread "main" kotlinx.serialization.SerializationException: Class 'SnapshotMutableStateImpl' is not registered for polymorphic serialization in the scope of 'MutableState'.
Mark the base class as 'sealed' or register the serializer explicitly
Code used :
#Serializable
class Test {
var number = mutableStateOf(0)
}
fun main() {
val json = Json { prettyPrint = true }
val serialized = json.encodeToString(Test())
println(serialized)
}
I have to put this annotation on my property :
#Serializable(with = MutableStateSerializer::class)
Isn't there a way to automatically link my serializer to the MutableState interface ? As the SnapshotMutableStateImpl is internal I can't set it to this class.
What you want is currently not possible. Other people seem to have requested a feature similar to what you need on GitHub: Global Custom Serializers.
Currently, for 3rd party classes, you need to specify the serializer in one of three ways:
Pass the custom serializer to the encode/decode method in case you are serializing it as the root object.
Specify the serializer on the property using #Serializable, as you do now.
Specify the serializer to be used by a full file using #file:UseSerializers.
Note that due to type inference, number will be attempted to be serialized as the return type of mutableStateOf. If you specify the type as an interface instead (does it have a supertype?), using polymorphic serialization, you could try to register the concrete type and pass your custom serializer there for the concrete type. Not really what this feature is designed for, but I believe it may work if you don't want to specify your serializer in multiple places. However, the serialized form will then include a type discriminator everywhere.
Kotlinx serialization documentation
According to Kotlinx.serialization user-defined annotations doc:
"Inside a process of serialization/deserialization, your own annotation class are available in SerialDescriptor object" :
override fun encodeElement(desc: SerialDescriptor, index: Int): Boolean {
val annotations = desc.getElementAnnotations(index)
...
}
What I want to do
I need a #Transient equivalent, but conditional:
classic way where : Json.stringify(serializer, myClass) works as usual.
custom way where : Json.stringify(customSerializer, myClass) would return usual json but exculding all #MyAnnotation-tagged values.
Here is my code
#SerialInfo
#Target(AnnotationTarget.PROPERTY)
annotation class CustomAnnotation
#Serializable
data class MyClass(val a: String, #CustomAnnotation val b: Int = -1)
And I would like to build a custom Serializer and achieve something like
override fun encodeElement(desc: SerialDescriptor, index: Int): Boolean {
val isTaggedAsCustomAnnotation = desc.getElementAnnotations(index).any{ it is CustomAnnotation }
val myCondition = mySerializer.getMyConditionBlablabla
if(myCondition && isTaggedAsCustomAnnotation) {
encode()
}
...
}
What I found
abstract class ElementValueEncoder : Encoder, CompositeEncoder {
...
open fun encodeElement(desc: SerialDescriptor, index: Int): Boolean = true
}
But I don't know how I can build a custom Serializer so that I can override that function Encoder.encodeElement. Where can I access to ElementValueEncoder in a custom Serializer ?
I also found this sample demo in kotlinx.serialization github repo. It's using TaggedEncoder & TaggedDecoder where I'm able to override encodeTaggedValue. But here again I don't know how I can use those encoder/decoder in a process of serialization/deserialization.
Finally
Where can I override fun encodeElement(desc: SerialDescriptor, index: Int): Boolean, and how I can handle my own-defined serialization annotation ?
Thanks !!
First of all, you need to grasp the difference between Serializer and Encoder. Serializer (represented by KSerializer) defines how your class looks like, and Encoder (represented by e.g. JsonOutput) defines how data will be recorded. You can find more info on that topic here: https://github.com/Kotlin/KEEP/blob/master/proposals/extensions/serialization.md#core-api-overview-and-mental-model .
So, custom annotations feature is mainly used for providing format-specific information to Encoder. Typical usage of such an annotation is ProtoId – property id, specific to protobuf format, that should be recognized by ProtobufEncoder. Such annotations are usually defined by format authors alongside their encoders.
What you want to do here, as I can see, is to use already existing encoder (JSON format), so overriding encodeElement is impossible since Json encoders can not be subclassed. I'd advise you to use custom json transofrming serializer to achieve your goal. Unfortunately, currently kotlinx.serialization does not have mechanism to generalize such a transformation, so you need to write such serializer for each class.
I have a Firestore collection that holds different data objects with no common key or values.
In Kotlin, this is represented by something like
sealed class Task()
data class WorkTask(val id: String): Task()
data class ReductionTask(val time: Date): Task()
I would like to deserialize the data from the Firestore collection in a way like:
val tasks = result.toObjects(Task::class.java)
val workTasks = result.filterInstance(WorkTask::class.java)
val reductionTasks= result.filterInstance(ReductionTask::class.java)
In summary, I would like to retrieve a union from Firestore WorkTask | ReductionTask | OtherTask that I would be able to hold in one list and later either filter or patternmatch by instance.
EDIT:
Currently, my workaround is to have 1 common key (type) that holds the type of the object:
inline fun <reifed T: Any> QuerySnapshot.deserializeByType(
crossinline selector: (type:String) -> Class<out T>
): List<T> {
return this.documents.map({ document ->
val type = firestoreDoc.getString("type")
document.toObject(selector(type))
})
}
querySnapshot.deserializeByType<Task> { type ->
when (type) {
"WORK" -> WorkTask::class.java
"REDUCE" -> ReductionTask::class.java
...
}
}
And in theory, I could just provide a list of classes and let it try/catch to deserialize. But that seems to be hacky as hell.
Since the Firestore SDK is implemented in Java, it doesn't know anything about Kotlin sealed classes. It is working purely off what understands about JavaBean style POJO objects that use conventions for names of getter and setter methods on classes. toObject() is simply mapping the names of Firestore document fields to the getters and setters (obtained by reflection) on the provided class instance. That's all.
Your workaround (or some variation of it) is currently the only viable option, since your code needs to make a judgement about which actual class is actually being represented by the document in question. There's really no way for the Firestore SDK to know which class the document should populate - you have to tell it that.
I now understand that scala #serializable objects can be used the same as a Java Serializable object. In a Java Serializable object there are methods you can override to change how the object streams: writeObject(ObjectOutputStream) / readObject(ObjectOutputStream).
Can you override or inject methods into a scala #serializable object allowing you to change how the object serializes?
Yes, you can use the same methods in Scala as in Java:
#throws(classOf[IOException])
private def writeObject(out: ObjectOutputStream): Unit = // ...
#throws(classOf[IOException])
private def readObject(in: ObjectInputStream): Unit = // ...
As already stated, you can define your own writeObject and readObject methods.
#throws(classOf[java.io.IOException])
private def writeObject(out : java.io.ObjectOutputStream) : Unit = /* your definition here */
However be careful when performing this on nested classes, objects or traits.
#serializable
class Foo(x : Int) {
#serializable object X { def y = x }
}
If I serialize object X, it will actually serialize the containing Foo class, so this must also be serializable. This can be a PITA to deal with in custom serialization methods, so here's fair warning.
Another pain-point can be closure serialization. Try to keep a mental model of what variables are being captured in serialized closures. Ensure that these variables are something you'd want sent over IO!