I'm currently struggling with the experimental KXS-properties serialization backend, mainly because of two reasons:
I can't find any documentation for it (I think there is none)
KXS-properties only includes a serializer / deserializer, but no encoder / decoder
The endpoint provided by the framework is essentially Map<String, Any>, but the map is flat and the keys already have the usual dot-separated properties syntax. So the step that I have to take is to encode the map to a single string that is printable to a .properties file AND decode a single string from a .properties file into the map. I'm generally following the Properties Format Spec from https://docs.oracle.com/javase/10/docs/api/java/util/Properties.html#load(java.io.Reader), it's not as easy as one might think.
The problem is that I can't use java.util.Properties right away because KXS is multiplatform and it would kinda kill the purpose of it when I'd restrict it to JVM because I use java.util.Properties. If I were to use it, the solution would be pretty simple, like this: https://gist.github.com/RaphaelTarita/748e02c06574b20c25ab96c87235096d
So I'm trying to implement my own encoder / decoder, following the rough structure of kotlinx.serialization.json.Json.kt. Although it's pretty tedious, it went well so far, but now I've stumbled upon a new problem:
As far as I know (I am not sure because there is no documentation), the map only contains primitives (or primitive-equivalents, as Kotlin does not really have primitives). I suspect this because when you write your own KSerializers for the KXS frontend, you can specify to encode to any primitive by invoking the encodeXXX() functions of the Encoder interface. Now the problem is: When I try to decode to the map that should contain primitives, how do I even know which primitives are expected by the model class?
I've once written my own serializer / deserializer in Java to learn about the topic, but in that implementation, the backend was a lot more tightly coupled to the frontend, so that I could query the expected primitive type from the model class in the backend. But in my situation, I don't have access to the model class and I have no clue how to retrieve the expected types.
As you can see, I've tried multiple approaches, but none of them worked right away. If you can help me to get any of these to work, that would be very much appreciated
Thank you!
The way it works in kotlinx.serialization is that there are serializers that describe classes and structures etc. as well as code that writes/read properties as well as the struct. It is then the job of the format to map those operations to/from a data format.
The intended purpose of kotlinx.serialization.Properties is to support serializing a Kotlin class to/from a java.util.Properties like structure. It is fairly simple in setup in that every nested property is serialized by prepending the property name to the name (the dotted properties syntax).
Unfortunately it is indeed the case that this deserializing from this format requires knowing the types expected. It doesn't just read from string. However, it is possible to determine the structure. You can use the descriptor property of the serializer to introspect the expectations.
From my perspective this format is a bit more simple than it should be. It is a good example of a custom format though. A key distinction between formats is whether they are intended to just provide a storage format, or whether the output is intended (be able to) to represent a well designed api. The latter ones need to be more complex.
Related
I'm interested to know what data is held in the MetaData annotation added to each Kotlin class.
But most fields give no more detail than
"Metadata in a custom format. The format may be different (or even absent) for different kinds."
https://github.com/JetBrains/kotlin/blob/master/libraries/stdlib/jvm/runtime/kotlin/Metadata.kt
Is there are reference somewhere that explains how to interpret this data?
kotlin.Metadata contains information about Kotlin symbols, such as their names, signatures, relations between types, etc. Some of this information is already present in the JVM signatures in the class files, but a lot is not, since there's quite a few Kotlin-specific things which JVM class files cannot represent properly: type nullability, mutable/read-only collection interfaces, declaration-site variance, and others.
No specific actions were taken to make the schema of the data encoded in this annotation public, because for most users such data is needed to introspect a program at runtime, and the Kotlin reflection library provides a nice API for that.
If you need to inspect Kotlin-specific stuff which is not exposed via the reflection API, or you're just generally curious what else is stored in that annotation, you can take a look at the implementation of kotlinx.reflect.lite. It's a light-weight library, the core of which is the protobuf-generated schema parser. There's not much supported there at the moment, but there are schemas available
which you can use to read any other data you need.
UPD (August 2018): since this was answered, we've published a new (experimental and unstable) library, which is designed to be the intended way for reading and modifying the metadata: https://discuss.kotlinlang.org/t/announcing-kotlinx-metadata-jvm-library-for-reading-modifying-metadata-of-kotlin-jvm-class-files/7980
For now, all I am looking for is simple serialization / deserialization. I am not looking for transport layers or other network stack elements.
I have found that building a simple serialize/deserialize scenario is easy in Thrift and Protocol Buffers. I would like to try to do the same using Ice's Slice.
The main benefit I see is that Slice seems to support "classes" in addition to "structs". these classes support inheritance, which seems nice. I'd like to try serializing these in a simple fashion, while ignoring the rest of the transport layers, etc which Ice provides.
I've gotten as far as running slice2java on a .ice file containing a simple class (not even with inheritance yet), but am unsure how to proceed. The generated class doesn't seem to provide a direct way to serialize itself, and I can't find documentation on how to do it using Ice libraries.
As an example, here is the PB code to do what I want:
Person p = Person.newBuilder().
setEmail("John#doe.com").
setId(1234).
setName("John Doe").build();
//write the buffer to a file.
p.writeTo(new FileOutputStream("JohnDoe.pb"));
//read it back in!
Person IsItJohnDoe = Person.parseFrom(new FileInputStream("JohnDoe.pb"));
System.out.println(IsItJohnDoe);
If anyone has encountered a similar problem, thank you in advance. I unfortunately don't have the time to investigate ice / slice as comprehensively as I would like.
According to ZeroC documentation, all Java classes, generated from Slice definitions implement java.io.Serializable interface, so you can serialize them using that interface. More details you can find in documentation: Serializable Objects in Java
I'm trying to write/read a class object from/to a file.
I'm new to D and I just want to play a little bit around with it.
Is there a Class/Function to write/read an object to/from a file?
I'm looking for something similar to the ObjectOutputStream сlass in Java.
Or do I have to serialize (concatenate) the object's variables as strings in the file?
I have a Movie class and a MovieManager class, which contains a dynamic movie-array.
A Movie object contains just a few strings and integer values.
Extending answer, provided in comment, it is worth explicitly stating, that D does not provide "one true way" of reading/writing objects to/from files, as there can't be a single optimal one. Different considerations about speed, resulting file format, handling references and similar corner cases may results in different serialization strategies.
That being said, most likely proper serialization library is needed, and, by lucky chance, one of most mature D solutions ("Orange" by Jacob Carlborg https://github.com/jacob-carlborg/orange) is being reviewed right now as a candidate for inclusion into standard library as a std.serialization: newsgroup thread. It may be your best bet.
The library Unmanaged provides a serialization system. You also have Orange
which is less restrictive, as Unmanaged serialization only works if the object to serialize is an ancestor of one of the framework base class.But...Unmanaged works on the "accessor" principle. The data serialized are get via a method and the data deserialized are set via a method, which allows to update some stuffs when the deserializer recall for example...
I need to serialize some complex interface (template.Template). It has many unexported fields, and gob don't want to work with them. Any suggestions?
P.S. Actualy, I trying to put a parsed template to memcache on App Engine.
The short answer is that there's usually a reason for unexported fields--template.Template, for instance, contains information which changes during parsing--so I'd advise against serializing them yourself with reflect. However, the GobEncoder and GobDecoder interfaces were recently added to gob; if you need to serialize a complex struct with unexported fields, encourage the author of the package to implement these interfaces. Even better, implement them yourself (shouldn't be hard for template.Template) and contribute your patch.
If the type is from another package (such as template) this can't be done with any of the current serialization libs for Go (gob, json, bson, etc.). Nor should it be done, because the fields are unexported.
However, if you really need to, you can write your own serializer using package reflect, specifically Value.Field() and friends to get the unexported fields. Then you just need to store them in a way that you can decode later.
how does Serialization of objects works? How object got deserialized and a instance is created from serialized date without a call to any constructor?
I've kept this answer language agnostic since a language wasn't given.
When the object is serialized, all the require information to rebuild it is encoded in way which can be retrieved. This typically includes the type of the object, as well as the value of all the instance variables.
When the object is deserialized, an area in memory of the correct size is allocated and is populated using the serialized information such that the new object is identical to the serialized one.
The running program can then refer to this new object in memory without having to actually call the constructor.
There are lots of little details which this doesn't explain, but this is the general idea of serialization/deserialization.
Are you talking about Java? If so, serialization is an extralingual object creation mechanism. It's a backdoor that uses native code to create the object without calling any constructors. Therefore, when designing a class for serializability, you need to make sure that a class created through deserialization maintains the same invariants (key fields being initialized) as you would through the constructor path. A third way to create objects in Java is through cloning, and similar issues apply.
Cloning and serialization don't interact well with the use of final fields if you need to set the value of that field to something different than what is returned by clone or the deserialization process.
Josh Bloch's "Effective Java" has some chapters that explain these issues in more depth.
(this answer may apply to other languages too, but I've only used serialization in Java)
Regarding .NET: this isn't a definitive or textbook answer, and I might be all-out wrong...
.NET Serialization needs to be seperated out into Binary vs. others (XML or an XML derivitave typically). Binary serialization is mostly a black-box to me, but it allows the object to be serialized and restored in their current state. XML serialization typically only serialized the public fields/properties of an object, unless overriden by adding a custom ISerializable implementation.
In the case of XML serialization I believe .NET uses Reflection to determine which fields and properties get converted to their equivalent Elements. Adding an [XMLSerializable] attribute will implement a default behavior which can be adjusted by applying other attributes at the field level (such as [XMLAttribute]).
The metadata (which Reflection depends on) stores all the object members as well as their attributes and addresses, which allows the serializer to determine how it should build the output.