I am interested in making a variable not automatically serialized (e.g., by Serde) with the use of a keyword like Java's transient, but in Rust. I need this to store passwords. Of course, I could manually clear the data upon serialization to the disk, but I would like to know if there are better, more automatic, options.
You can use the #[serde(skip)] attribute:
use serde::{Deserialize, Serialize}; // 1.0.88
#[derive(Deserialize, Serialize)]
struct MyStruct {
field1: i32, // this will be (de)serialized
#[serde(skip)]
field2: i32, // this will be skipped
}
If the type needs to be deserialized, it's a good idea to accompany a #[serde(skip)] with a #[serde(default)] (or a #[serde(default = "fn_name")]) attribute. Serde deserializes skipped fields as if a #[serde(default)] was implicitly added, but it's clearer to someone reading your code if you make it explicit where this value will come from.
See:
List of all serde field attributes
All serde attributes
Related
I'd like to encode a given class of type T: EventData with Kotlinx Serialization encodeToString.
This is my code:
class EventDispatcher<T: EventData>(
val pubSubTemplate: PubSubTemplate
) {
/**
* Dispatch an event to the game engine event manager pipeline
*/
fun dispatchEvent(event: T, initiator: String) {
val eventData: String = Json.encodeToString(event)
}
The compiler tells me:
Cannot use `T` as reified type parameter. Use a class instead
Is there a way to make this still work?
For Json.encodeToString(event) to work, it needs the type information for T. But, this type information is lost at runtime due to the way how generics work in Kotlin/Java.
One way to retain the type information would be by making dispatchEvent an inline function with T as a reified type parameter.
However, this also raises the question how you want to serialize event. You could also use polymorphic serialization of EventData, rather than trying to serialize T. This will include an additional class discriminator in your serialized output (it necessarily has to for polymorphic serialization/deserialization to work).
If you serialize the concrete type T, this class discriminator wouldn't be included, which is questionable; how would whoever will deserialize this know what type it is?
In short, I think you need polymorphic serialization.
I stuck with some simple thing) Let's say I have following:
interface IMessagePayload // marker interface
data class IdPayload(
val id: Long
) : IMessagePayload
data class StringPayload(
val id: String,
) : IMessagePayload
Then I have a class:
data class Message<T : IMessagePayload>(
val id: String,
val payload: T,
)
Also I have some interface describing processor of this message:
interface IMessageProcessor<T : IMessagePayload> {
fun process(message: Message<T>)
}
And some implementation:
class ProcessorImpl : IMessageProcessor<IdPayload> {
override fun process(message: Message<IdPayload>) {
}
}
Now I wanna have a map of such processors. Lets use some enum type as a keys of this map:
enum class ActionType {
UPDATE,
DELETE,
ADD
}
private var map = mutableMapOf<ActionType, IMessageProcessor<IMessagePayload>>()
map[ActionType.ADD] = ProcessorImpl() // <-- error here
And that's where the problem occurs. I cannot put my ProcessorImpl into this map. The compiler says that there is an error: Type mismatch. Required: IMessageProcessor. Found: ProcessorImpl().
I could declare the map in the following way (using star projection):
private var map = mutableMapOf<ActionType, IMessageProcessor<*>>()
But in this case I cannot call processors's process method fetching it from the map by key first:
map[ActionType.ADD]?.process(Message("message-id", IdPayload(1))) // <-- error here
Compiler complains: Type mismatch. Required nothing. Found Message<IdPayload>
What am I doing wrong? Any help is appreciated.
This is about variance.
IMessageProcessor is defined as interface IMessageProcessor<T : IMessagePayload>; it has one type parameter, which must be IMessagePayload or a subtype.
But it is invariant in that type parameter; an IMessageProcessor< IdPayload> is not related to an IMessageProcessor<IMessagePayload>. In particular, it's not a subtype.
And your map is defined with a value type IMessageProcessor<IMessagePayload>. So its value cannot be an IMessageProcessor< IdPayload>, because that's neither the value type, nor a subtype. Hence the compile error.
In this case, the simplest way to get it to compile is to change your map:
private var map = mutableMapOf<ActionType, IMessageProcessor<out IMessagePayload>>()
The only difference there is the out; that tells the compiler that the value IMessageProcessor is covariant in its type parameter. (It may help to think of out as meaning ‘…or any subtype’. Similarly, you could make it contravariant by using in, which you might think of as ‘…or any supertype’.)
This lets you store in the map an IMessageProcessor for any subtype of IMessagePayload.
However, if you do that, you'll find that you can't use any value you pull out of your map — because it can't tell which messages the processor can handle, i.e. which subtype of IMessagePayload it works for! (The compiler expresses this as expecting a type parameter of Nothing.)
In general, it's often better to specify variance on the interface or superclass itself (declaration-site variance) rather than the use-site variance shown above. But I can't see a good way to do that here, because you have multiple generic classes, and they interact in a complicated way…)
Think for a moment what IMessageProcessor's type parameter means: it's the type of message that the processor can consume. So an IMessageProcessor<A> can handle messages of type Message<A>.
Now, a subtype must be able to do everything its supertype can do (and usually more) — otherwise you can't drop that subtype anywhere that's expecting to use the supertype. (That has the grand name of the Liskov substitution principle — but it's really just common sense.)
So an IMessageProcessor<B> is a subtype of IMessageProcessor<A> only if it can handle at least all the messages that an IMessageProcessor<A> can. This means it must accept all messages of type Message<A>.
But Message is invariant in its type parameter: a Message<B> is not directly related to a Message<A>. So you can't write a processor that handles them both.
The most natural solution I can find is to specify variance on both Message and IMessageProcessor:
data class Message<out T : IMessagePayload>( /*…*/ )
interface IMessageProcessor<in T : IMessagePayload> { /*…*/ }
And then use a wildcard in your map to make it explicit that you don't know anything about the type parameters of its values:
private var map = mutableMapOf<ActionType, IMessageProcessor<*>>()
That lets you safely store a ProcessorImpl() in the map.
But you still have to use an (unchecked) cast on the values you pull out of the map before you can use them:
(map[ActionType.ADD] as IMessageProcessor<IdPayload>)
.process(Message("4", IdPayload(4L)))
I don't think there's any easy way around that, because the problem is inherent in having values which are processors that can handle only some (unknown) types of message.
I'm afraid the best thing would be to have a rethink about what these classes mean and how they should interact, and redesign accordingly.
I'm trying to serialize a rust HashMap with a custom hash function applying bincode::serialize_into(writer, cache)?; where cache is a struct-object defined as following:
use std::hash::{BuildHasherDefault, Hasher};
use serde::{Deserialize, Serialize};
#[derive(Deserialize, Serialize)]
pub(crate) struct Cache<S>
where
S: Default + Hasher,
{
records: HashMap<DNA, Mapping, BuildHasherDefault<S>>,
}
When compiling I receive an error the trait 'cli::_::_serde::Serialize' is not implemented for 'S' referring to the cache variable.
As described in this issue serialization for custom hash functions (in my case FnvHasher) should exist requiring only to implement the BuildHasher trait.
That trait is already implemented by BuildHasherDefault which is used to create a BuildHasher instance.
Edit:
To illustrate the issue I've create a playground example (as bincode is not available in playground I used serde_json instead).
As the example shows it is possible to create and serialize a HashMap custom_hashmap with custom hash-function (in this case FnvHasher).
But this does not work for a Cache-object cache wrapping a HashMap with a custom hash-function. In that case the compiler asks for adding the Serialize trait which does not exist for the FnvHasher
Is there way to declare a Map that would accept both String and Int in Kotlin? Like:
var map: Map<String, String | Int>
There is no union type in Kotlin, so you can't directly express this.
A few options:
use TreeMap<String, Any>, which is the simplest of all, but it allows more types than just Int and String, and requires additional conversion boilerplate on the consumer side when accessing values.
use a custom sealed class to encapsulate your "Int or String" type, and use it in a regular TreeMap<String, IntOrString>. This will have some runtime overhead for wrapping/unwrapping the int/string objects for each insertion/access. Also, this puts the burden of (un)wrapping on the consumer side. That being said, maybe your domain can be better modeled with clearer names than Int or String, so it might actually improve the consumer's experience to define such types.
use a custom map type, with independent accessors for ints and strings, but backed by a single TreeMap<String, Any> behind the scenes. This has the advantage of encapsulating the logic and exposing a neat API for the consumers. If you extend TreeMap you will save some overhead as opposed to composition, but it shouldn't matter much.
I am wondering about the actual use case at hand, though. Usually the need for union types stems from using plain generic primitive types to represent meaningful domain data that could maybe better be expressed using more specific custom domain types.
Option 2 would therefore usually be my choice, but I would use a sealed class that actually represents my domain data instead of the generic "string or int".
You can use Any here, see this example:
import java.util.TreeMap
fun main() {
var map = TreeMap<String, Any>()
map.put("One", 1)
map.put("Two", "two")
println(map)
}
which outputs
{One=1, Two=two}
Please note:
This will allow more types than just the desired String and Int, because the Kotlin Docs state (...). Every Kotlin class has Any as a superclass..
I'm hoping someone could illustrate a common use case for the Microsoft Bond runtime schemas (SchemaDef). I understand these are used when schema definitions are not known at compile time, but if the shape of an object is fluid and changes frequently, what benefits might a runtime generated schema provide?
My use case is that the business user is in control of the shape of an object (via a rules engine). They could conceivably do all sorts of things that could break our backward compatibility (for example, invert the order of fields on the object). If we plan on persisting all the object versions that the user created, is there any way to manage backward/forward compatibility using Bond runtime schemas? I presume no, as if they invert from this:
0: int64 myInt;
1: string myString;
to this
0: string myString;
1: int64 myInt;
I'd expect a runtime error. Which implies managing the object with runtime schemas wouldn't provide much help to me.
What would be a usecase where a runtime schema would in fact be useful?
Thank you!
Some of the uses for runtime schemas are:
with the Simple Binary protocol to handle schema changes
schema validation/evoluton
rendering a struct in a GUI
custom mapping from one struct to another
Your case feels like schema validation, if you can pro-actively reject a schema that would no be compatible. I worked on a system that used Bond under the hood and took this approach. There was an explicit "change the schema of this entity" operation that validated whether the two schemas were compatible with each other.
I don't know the data flow in your system, so such validation might not be possible. In that case, you could use the runtime schemas, along with some rules provided by the business users, to convert between different shapes.
Simple Binary
When deserializing from Simple Binary, the reader must know the exact schema that the writer used, otherwise it has no way to interpret the bytes, resulting in potentially silent data corruption.
Such corruption can happen if the schema undergoes the following change:
// starting struct
struct Foo
{
0: uint8 f1;
1: uint16 f2;
}
The Simple Binary serialized representation of Foo { f1: 1, f2: 2} is 0x01 0x02 0x00.
Let's now change the schema to this:
// changed struct
struct Foo
{
0: uint8 f1;
// It's OK to remove an optional field.
// 1: uint16 f2;
2: uint8 f3;
3: uint8 f4;
}
If we deserialize 0x01 0x02 0x00 with this schema, we'll get Foo { f1: 1, f3: 2, f4: 0}. Notice that f3 is 2, which is not correct: it should be 0. With the runtime schema for the old Foo, the reader will know that the second and third bytes correspond to a field that has since been deleted and can skip them, resulting in the expected Foo { f1:1, f3: 0, f4: 0 }.
Schema Validation and Evolution
Some systems that use Bond have different rules for schema evolution that the normal Bond rules. Runtime schemas can be used to enforce such rules (e.g., checking a type to enforce a rule that no collections are used) before accepting structs of a given type or before registering such a schema in, say, a repository of known schemas.
You could also walk two schemas to determine with they are compatible with each other. It would be nice if Bond provided such an API itself, so that it doesn't have to be reimplemented again and again. I've opened a GitHub issue for such an API.
GUI
With a runtime schema, you have extra information about the struct, including things like the names of the fields. (The binary encoding protocols omit field names, relying, instead, on field IDs.) You can use this additional information to do things like create GUI controls specific to each field.
There's an example showing inspection of a runtime schema in both C# and C++.
Custom Mapping
In C++, the MapTo transform can be used to convert one struct to another, which incompatible shapes, given a set of rules. There's an example of this, that makes use of a runtime schema to derive the rules.