I have a question about specific avro serialization with kryo. Sorry if this question seems redundant but I didn't find a simple answer.
I've configured Spark to run with Kryo and it works well : Avro generated classes are well serialized using a custom KryoRegistrator class. I use twitter.chill-avro package too.
class AvroClassRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[MyAvroClass], AvroSerializer.SpecificRecordSerializer[MyAvroClass])
}
Nevertheless, when I try to serialize a Map[String, MyAvroClass], the only way to make the serialization work is to extends MyAvroClass with java.io.Serializable.
Could somebody explain me what really happens, and tell me how to register this Map[..., AvroClass] with kryo ?
Thank you
Related
I am writing a custom spring boot starter that provides a uniform error response class for all repositories that will add this starter. It also provides the corresponding exception handler.
The problem is, that this error response needs an error code which might differ between all those repositories, using my starter. So the solution would be to create an error response with an error code that is an interface. Other repositories can then create enums implementing this interface to achieve the desired behavior.
It is written in kotlin
interface BaseErrorCode {
val message: String
}
class ErrorResponse {
val customMessage: String,
val errorCode: BaseErrorCode,
val timestamp: OffsetDateTime
}
Now, in another repository, I use this starter, get access to the classes above and create my error code enum:
enum class MyCustomErrorCodes : BaseErrorCode {
FOO
}
Now, I can throw an exception and through the handler, this json will be produced:
{
"customMessage": "My message",
"errorCode": "FOO",
"timestamp": "2023-01-27T12:15:31.7730645+01:00"
}
So serializing works absolutely fine.
However, when deserializing the ErrorResponse in my integration-tests, I get the following exception:
com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `my.package.BaseErrorCode` (no Creators, like default constructor, exist): abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
at [Source: (String)"{"customMessage":"My message","errorCode":"FOO","timestamp":"2023-01-27T12:12:06.4932227+01:00"}"; line: 1, column: 54] (through reference chain: my.package.ErrorResponse["errorCode"])
Approach 1
I know a solution where you write a custom deserializer. But this could not be placed in the spring boot starter because I need to know the implementing classes of that ´BaseErrorCode`-interface.
Approach 2
Using #JsonSubType, I can also tell Jackson how to handle this interface. But I do need some information about implementing enums as well, which I do not have in my starter
Is there any way to deserialize this error response
only by modifying the starter, not each repository
without knowing the classes that implement BaseErrorCode, BUT knowing that it will ALWAYS be an enum?
I think the answer lies in using intentionally dissimilar objects. I have not tested, but I think:
follow what you have with serialization, and note that you are not serializing the whole enum rather an instance of enum which conforms to BaseErrorCode.
on deserialization you make a concrete implementation of BaseErrorCode (this can be private to the Starter) that you deserialize the JSON in to. (Adding #JsonIgnoreProperties(ignoreUnknown = true) to guard against an unknown subclass sending more data than you can deal with).
Trying to use Kotlin for an app utilizing Apache Beam, and I get the warning that:
#ProcessElement processElement(String, OutputReceiver), parameter of type DoFn.OutputReceiver<Map<String, String>> at index 1: OutputReceiver should be parameterized by java.util.Map<java.lang.String, ? extends java.lang.String>
I'm assuming somewhere in the implementation they use reflection to determine the specific type being used; is there a way to work with this cleanly in Kotlin? Or would I need to create a wrapper around such functions to do the conversion?
It's hard to respond without seeing the entire class but I think you can avoid wrapper by importing java map with the alias as
import java.util.Map as JavaMap
and then return a JavaMap
The internal apache beam reflection considers only java collection types so a case like this one can happen also with Iterables or Collection etc..
To force spark to use kryo serialization the following option can be set spark.kryo.registrationRequired.
How can I register org.apache.spark.sql.types.DataType[]?
Class is not registered: org.apache.spark.sql.types.DataType[]
which is throwing an exception. But trying to register like classOf[org.apache.spark.sql.types.DataType[]] will not compile
I was confused with java and scala array notation.
classOf[Array[org.apache.spark.sql.types.DataType]]
is the correct registration.
Nonetheless, this is a spark internal class which should already be registered by spark.
I am currently using Guava's ForwardingMap as a base class and have numerous types that extend it. I need to maintain the Map type because instances need to be treated as such in consumers. So, even though internally the ForwardingMap using composition the external interface still has to be a map.
As a map, deserializing just key-value properties using #JsonAnyGetter and #JsonAnySetter work fine but, I also need to take into account custom properties, using #JsonProperty, which may also be a part of the instance as well.
So, when serializing or deserializing I want all of the entries and any custom properties which may be a part of the extended class.
I have looked at numerous types of solutions, such as using the Shape.OBJECT and apply interfaces, but none of them seem to work properly for me. I believe I need to create a custom deserializer/serializer to handle the bean + map processing in Jackson but cannot find any examples as to how to do this.
These links help to explain what I am trying to do with no luck:
http://www.cowtowncoder.com/blog/archives/2013/10/entry_482.html
How to serialize with Jackson a java.util.Map based class (cannot change base of ForwardingMap)
Jackson - ignore Map superclass when serializing (cannot change base because it needs to remain a Map)
Ideally, I would like an example or pointer of how to serialize and deserialize an instance that extends ForwardingMap using #JsonAnySetter and #JsonAnyGetter and has custom properties using #JsonProperty as well.
I would want my output to look like
"modules": {
"MyModel": { <-- extends ForwardingMap<>
"domain": "typeinfo",
"property":"hello", <-- comes from #JsonProperty
"another": "GoodBye", <-- comes from #JsonAnyGetter
"another2": 50 <-- comes from #JsonAnyGetter
}
}
Background: Classes that inherit from Monobehaviour can't be serialized.
Premise: A way to save the data (variables/fields and their values) of a MonoBehaviour script so it can be serialized, and deserialize it again and use this data to "fill in" a corresponding MonoBehaviour script's variable/field values.
Tried so far:
Having a serializable "wrapper/container" class that has the same fields as the MB script, but does not inherit from MB. Works nicely but every MV script needs it's own wrapper class and it's own wrapping function.
Serializing a List<FieldInfo> and fill it with the MB's fields... Works 30%;
The FieldInfos get added but are of the wrong Type, and
When deserialzing their values can't be accessed because an instance of a class is needed, but only a list is available
I feel like it can't be that hard but my Reflection skills and related are limited but seeing as saving/loading is a rather common feature I hope there is either someone who did it or someone who can point me in the right direction.
There is no easy way to serialize a MonoBehaviour using a BinaryFormatter built in .NET. There are a few options you can consider:
Using a Memento Patter. That is (more or less) what you have tried to achieve using a wrapper. Momento assumes a saving and restoring internal state of objects, so serialization is one of techniques.
Using Unity Serialization, by declaring the methods:
void Serialize(){}
void Deserialize(){}
In your MonoBehaviour script, so within the methods you will choose the properties/fields you want to serialize/deserialize.
There is an interesting framework, source code is on GitHub. It has a custom serialization framework that lets you serialize almost anything (not only monobehaviors). I have never used it, here is the forum page on Unity3d forum, I believe it's worth a look.
The answer to the question is: ScriptableObject. That's what they're for.
Put your variables in a ScriptableObject and Unity will handle the serialisation and give you a custom editor and other nice features. Recommended.