How can we write a generic function for checking Serde serialization and deserialization? - serialization

Question

In a project where custom Serde (1.0) serialization and deserialization methods are involved, I have relied on this test routine to check whether serializing an object and back would yield an equivalent object.
// let o: T = ...;
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: T = from_slice(&buf).unwrap();
assert_eq!(o, o2);
Doing this inline works pretty well. My next step towards reusability was to make a function check_serde for this purpose.
pub fn check_serde<T>(o: T)
where
T: Debug + PartialEq<T> + Serialize + DeserializeOwned,
{
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: T = from_slice(&buf).unwrap();
assert_eq!(o, o2);
}
This works well for owning types, but not for types with lifetime bounds (Playground):
check_serde(5);
check_serde(vec![1, 2, 5]);
check_serde("five".to_string());
check_serde("wait"); // [E0279]
The error:
error[E0279]: the requirement `for<'de> 'de : ` is not satisfied (`expected bound lifetime parameter 'de, found concrete lifetime`)
--> src/main.rs:24:5
|
24 | check_serde("wait"); // [E0277]
| ^^^^^^^^^^^
|
= note: required because of the requirements on the impl of `for<'de> serde::Deserialize<'de>` for `&str`
= note: required because of the requirements on the impl of `serde::de::DeserializeOwned` for `&str`
= note: required by `check_serde`
As I wish to make the function work with these cases (including structs with string slices), I attempted a new version with an explicit object deserialization lifetime:
pub fn check_serde<'a, T>(o: &'a T)
where
T: Debug + PartialEq<T> + Serialize + Deserialize<'a>,
{
let buf: Vec<u8> = to_vec(o).unwrap();
let o2: T = from_slice(&buf).unwrap();
assert_eq!(o, &o2);
}
check_serde(&5);
check_serde(&vec![1, 2, 5]);
check_serde(&"five".to_string());
check_serde(&"wait"); // [E0405]
This implementation leads to another issue, and it won't compile (Playground).
error[E0597]: `buf` does not live long enough
--> src/main.rs:14:29
|
14 | let o2: T = from_slice(&buf).unwrap();
| ^^^ does not live long enough
15 | assert_eq!(o, &o2);
16 | }
| - borrowed value only lives until here
|
note: borrowed value must be valid for the lifetime 'a as defined on the function body at 10:1...
--> src/main.rs:10:1
|
10 | / pub fn check_serde<'a, T>(o: &'a T)
11 | | where T: Debug + PartialEq<T> + Serialize + Deserialize<'a>
12 | | {
13 | | let buf: Vec<u8> = to_vec(o).unwrap();
14 | | let o2: T = from_slice(&buf).unwrap();
15 | | assert_eq!(o, &o2);
16 | | }
| |_^
I have already expected this one: this version implies that the serialized content (and so the deserialized object) lives as long as the input object, which is not true. The buffer is only meant to live as long as the function's scope.
My third attempt seeks to build owned versions of the original input, thus evading the issue of having a deserialized object with different lifetime boundaries. The ToOwned trait appears to suit this use case.
pub fn check_serde<'a, T: ?Sized>(o: &'a T)
where
T: Debug + ToOwned + PartialEq<<T as ToOwned>::Owned> + Serialize,
<T as ToOwned>::Owned: Debug + DeserializeOwned,
{
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: T::Owned = from_slice(&buf).unwrap();
assert_eq!(o, &o2);
}
This makes the function work for plain string slices now, but not for composite objects containing them (Playground):
check_serde(&5);
check_serde(&vec![1, 2, 5]);
check_serde(&"five".to_string());
check_serde("wait");
check_serde(&("There's more!", 36)); // [E0279]
Again, we stumble upon the same error kind as the first version:
error[E0279]: the requirement `for<'de> 'de : ` is not satisfied (`expected bound lifetime parameter 'de, found concrete lifetime`)
--> src/main.rs:25:5
|
25 | check_serde(&("There's more!", 36)); // [E0279]
| ^^^^^^^^^^^
|
= note: required because of the requirements on the impl of `for<'de> serde::Deserialize<'de>` for `&str`
= note: required because of the requirements on the impl of `for<'de> serde::Deserialize<'de>` for `(&str, {integer})`
= note: required because of the requirements on the impl of `serde::de::DeserializeOwned` for `(&str, {integer})`
= note: required by `check_serde`
Granted, I'm at a loss. How can we build a generic function that, using Serde, serializes an object and deserializes it back into a new object? In particular, can this function be made in Rust (stable or nightly), and if so, what adjustments to my implementation are missing?

Answer 1

Unfortunately, what you need is a feature that is not yet implemented in Rust: generic associated types.
Let's look at a different variant of check_serde:
pub fn check_serde<T>(o: T)
where
for<'a> T: Debug + PartialEq<T> + Serialize + Deserialize<'a>,
{
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: T = from_slice(&buf).unwrap();
assert_eq!(o, o2);
}
fn main() {
check_serde("wait"); // [E0279]
}
The problem here is that o2 cannot be of type T: o2 refers to buf, which is a local variable, but type parameters cannot be inferred to types constrained by a lifetime that is restricted to the function's body. We'd like for T to be something like &str without a specific lifetime attached to it.
With generic associated types, this could be solved with something like this (obviously I can't test it, since it's not implemented yet):
trait SerdeFamily {
type Member<'a>: Debug + for<'b> PartialEq<Self::Member<'b>> + Serialize + Deserialize<'a>;
}
struct I32Family;
struct StrFamily;
impl SerdeFamily for I32Family {
type Member<'a> = i32; // ignoring a parameter is allowed
}
impl SerdeFamily for StrFamily {
type Member<'a> = &'a str;
}
pub fn check_serde<'a, Family>(o: Family::Member<'a>)
where
Family: SerdeFamily,
{
let buf: Vec<u8> = to_vec(&o).unwrap();
// `o2` is of type `Family::Member<'b>`
// with a lifetime 'b different from 'a
let o2: Family::Member = from_slice(&buf).unwrap();
assert_eq!(o, o2);
}
fn main() {
check_serde::<I32Family>(5);
check_serde::<StrFamily>("wait");
}

Answer 2

The answer from Francis Gagné has shown that we cannot do this efficiently without generic associated types. Establishing deep ownership of the deserialized object is a possible work-around which I describe here.
The third attempt is very close to a flexible solution, but it falls short due to how std::borrow::ToOwned works. The trait is not suitable for retrieving a deeply owned version of an object. Attempting to use the implementation of ToOwned for &str, for instance, gives you another string slice.
let a: &str = "hello";
let b: String = (&a).to_owned(); // expected String, got &str
Likewise, the Owned type for a struct containing string slices cannot be a struct containing Strings. In code:
#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct Foo<'a>(&str, i32);
#[derive(Debug, PartialEq, Serialize, Deserialize)]
struct FooOwned(String, i32);
We cannot impl ToOwned for Foo to provide FooOwned because:
If we derive Clone, the implementation of ToOwned for T: Clone is only applicable to Owned = Self.
Even with a custom implementation of ToOwned, the trait requires that the owned type can be borrowed into the original type (due to the constraint Owned: Borrow<Self>). That is, we are supposed to be able to retrieve a &Foo(&str, i32) out of a FooOwned, but their internal structure is different, and so this is not attainable.
This means that, in order to follow the third approach, we need a different trait. Let's have a new trait ToDeeplyOwned which turns an object into a fully owned one, with no slices or references involved.
pub trait ToDeeplyOwned {
type Owned;
fn to_deeply_owned(&self) -> Self::Owned;
}
The intent here is to produce a deep copy out of anything. There doesn't seem to be an easy catch-all implementation, but some tricks are possible. First, we can implement it to all reference types where T: ToDeeplyOwned.
impl<'a, T: ?Sized + ToDeeplyOwned> ToDeeplyOwned for &'a T {
type Owned = T::Owned;
fn to_deeply_owned(&self) -> Self::Owned {
(**self).to_deeply_owned()
}
}
At this point we would have to selectively implement it to non-reference types where we know it's ok. I wrote a macro for making this process less verbose, which uses to_owned() internally.
macro_rules! impl_deeply_owned {
($t: ty, $t2: ty) => { // turn $t into $t2
impl ToDeeplyOwned for $t {
type Owned = $t2;
fn to_deeply_owned(&self) -> Self::Owned {
self.to_owned()
}
}
};
($t: ty) => { // turn $t into itself, self-contained type
impl ToDeeplyOwned for $t {
type Owned = $t;
fn to_deeply_owned(&self) -> Self::Owned {
self.to_owned()
}
}
};
}
For the examples in the question to work, we need at least these:
impl_deeply_owned!(i32);
impl_deeply_owned!(String);
impl_deeply_owned!(Vec<i32>);
impl_deeply_owned!(str, String);
Once we implement the necessary traits on Foo/FooOwned and adapt serde_check to use the new trait, the code now compiles and runs successfully (Playground):
#[derive(Debug, PartialEq, Serialize)]
struct Foo<'a>(&'a str, i32);
#[derive(Debug, PartialEq, Clone, Deserialize)]
struct FooOwned(String, i32);
impl<'a> ToDeeplyOwned for Foo<'a> {
type Owned = FooOwned;
fn to_deeply_owned(&self) -> FooOwned {
FooOwned(self.0.to_string(), self.1)
}
}
impl<'a> PartialEq<FooOwned> for Foo<'a> {
fn eq(&self, o: &FooOwned) -> bool {
self.0 == o.0 && self.1 == o.1
}
}
pub fn check_serde<'a, T: ?Sized>(o: &'a T)
where
T: Debug + ToDeeplyOwned + PartialEq<<T as ToDeeplyOwned>::Owned> + Serialize,
<T as ToDeeplyOwned>::Owned: Debug + DeserializeOwned,
{
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: T::Owned = from_slice(&buf).unwrap();
assert_eq!(o, &o2);
}
// all of these are ok
check_serde(&5);
check_serde(&vec![1, 2, 5]);
check_serde(&"five".to_string());
check_serde("wait");
check_serde(&"wait");
check_serde(&Foo("There's more!", 36));

Answer 3

Update (04.09.2021):
The latest nightly has some fixes around GATs which basically allows the original example:
#![feature(generic_associated_types)]
use serde::{Deserialize, Serialize};
use serde_json::{from_slice, to_vec};
use std::fmt::Debug;
trait SerdeFamily {
type Member<'a>:
Debug +
for<'b> PartialEq<Self::Member<'b>> +
Serialize +
Deserialize<'a>;
}
struct I32Family;
struct StrFamily;
impl SerdeFamily for I32Family {
type Member<'a> = i32;
}
impl SerdeFamily for StrFamily {
type Member<'a> = &'a str;
}
fn check_serde<F: SerdeFamily>(o: F::Member<'_>) {
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: F::Member<'_> = from_slice(&buf).unwrap();
assert_eq!(o, o2);
}
fn main() {
check_serde::<I32Family>(5);
check_serde::<StrFamily>("wait");
}
The example above compiles now: playground.
As of now it's possible to implement this on rust nightly (with an explicit variance workaround):
#![feature(generic_associated_types)]
use serde::{Deserialize, Serialize};
use serde_json::{from_slice, to_vec};
use std::fmt::Debug;
trait SerdeFamily {
type Member<'a>: Debug + PartialEq + Serialize + Deserialize<'a>;
// https://internals.rust-lang.org/t/variance-of-lifetime-arguments-in-gats/14769/19
fn upcast_gat<'short, 'long: 'short>(long: Self::Member<'long>) -> Self::Member<'short>;
}
struct I32Family;
struct StrFamily;
impl SerdeFamily for I32Family {
type Member<'a> = i32; // we can ignore parameters
fn upcast_gat<'short, 'long: 'short>(long: Self::Member<'long>) -> Self::Member<'short> {
long
}
}
impl SerdeFamily for StrFamily {
type Member<'a> = &'a str;
fn upcast_gat<'short, 'long: 'short>(long: Self::Member<'long>) -> Self::Member<'short> {
long
}
}
fn check_serde<F: SerdeFamily>(o: F::Member<'_>) {
let buf: Vec<u8> = to_vec(&o).unwrap();
let o2: F::Member<'_> = from_slice(&buf).unwrap();
assert_eq!(F::upcast_gat(o), o2);
}
fn main() {
check_serde::<I32Family>(5);
check_serde::<StrFamily>("wait");
}
Playground

Answer 4

Simple (but a little awkward) solution: Provide buf from outside of the function.
pub fn check_serde<'a, T>(o: &'a T, buf: &'a mut Vec<u8>)
where
T: Debug + PartialEq<T> + Serialize + Deserialize<'a>,
{
*buf = to_vec(o).unwrap();
let o2: T = from_slice(buf).unwrap();
assert_eq!(o, &o2);
}
buf can be reused with Cursor
pub fn check_serde_with_cursor<'a, T>(o: &'a T, buf: &'a mut Vec<u8>)
where
T: Debug + PartialEq<T> + Serialize + Deserialize<'a>,
{
buf.clear();
let mut cursor = Cursor::new(buf);
to_writer(&mut cursor, o).unwrap();
let o2: T = from_slice(cursor.into_inner()).unwrap();
assert_eq!(o, &o2);
}