Can you serialize a vector of stuct's to TOML in rust? - serialization

Summary
I'm writing a program in rust and I would prefer use a TOML file to store a vector of struct's. However I can't figure out how to store a vector of struct's in a TOML file. I am able to do this using JSON but was hoping for TOML so just confirming that I'm not overlooking something (not even sure if the TOML format would be able to support what I want). Therefore, I'm trying to find out if anyone knows of a way use rust to serialize a vector of struct's to TOML and more importantly to deserialize it back into a vector.
Error message (on attempt to deserialize)
thread 'main' panicked at 'called Result::unwrap() on an Err value: Error { inner: ErrorInner { kind: Wanted { expected: "a table key", found: "a right bracket" }, line: Some(0), col: 2, at: Some(2), message: "", key: [] } }', src/main.rs:22:55
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Excerpt from Cargo.toml
[dependencies]
serde = { version = "1", features = ["derive"] }
serde_json = "1.0.86"
toml = "0.5.9"
Example code
Link to code on Playground
use serde::{Deserialize, Serialize};
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Point {
x: i32,
}
/// An ordered list of points (This is what I want to store in the TOML file)
type Points = Vec<Point>;
fn main(){
// Create sample data to test on
let original: Points = vec![Point { x: 1 }];
// Working code that converts it to json and back
let json = serde_json::to_string(&original).unwrap();
let reconstructed: Points = serde_json::from_str(&json).unwrap();
assert_eq!(original, reconstructed); // No panic
// "Desired" code that converts it to toml but unable to deserialize
let toml = toml::to_string(&original).unwrap();
let reconstructed: Points = toml::from_str(&toml).unwrap(); // Panics!
assert_eq!(original, reconstructed);
}
Output of toml::to_string(&original).unwrap()
[[]]
x = 1
Explanation of example code
In the example code I create some sample data then convert it to JSON and back with no issue. I then try to convert it to TOML, which doesn't give an error but the output doesn't make sense to me. Then I try to convert it back into a rust vector and that triggers the error. My biggest problem is I'm not even sure how I would expect the TOML file to look for a valid representation of a vector with multiple struct's.
Related Questions / Research
I wasn't able to find any information for creating a vector with multiple struct's the closest I could find is this question, and while the question looks like it should solve my problem the actual problem was serializing enums and the solution hence refers to that and doesn't solve my problem.

It seems that to represent an array of tables in Toml the syntax is
[[points]]
x = 1
[[points]]
x = 2
So backtracking from Toml syntax and original panic error Error { inner: ErrorInner { kind: Wanted { expected: "a table key", found: "a right bracket" }: Introducing a wrapper struct to represent table key fixes the issue.
use serde::{Deserialize, Serialize};
#[derive(PartialEq, Debug, Serialize, Deserialize)]
struct Point {
x: i32,
}
#[derive(PartialEq, Debug,Serialize, Deserialize)]
struct Points {
points: Vec<Point>
}
impl From<Vec<Point>> for Points {
fn from(points: Vec<Point>) -> Self {
Points {
points
}
}
}
fn main(){
let original: Points = vec![Point { x: 1 }, Point {x : 2}].into();
let json = serde_json::to_string(&original).unwrap();
let reconstructed: Points = serde_json::from_str(&json).unwrap();
assert_eq!(original, reconstructed);
let toml = toml::to_string(&original).unwrap();
let reconstructed: Points = toml::from_str(&toml).unwrap();
assert_eq!(original, reconstructed);
}

Related

Deserializing an enum using a combination of #[serde(untagged)] and #[serde(with)]

I'm trying to use an actix-web server as a gateway to a small stack to guarantee a strict data format inside of the stack while allowing some freedoms for the user.
To do that, I want to deserialize a JSON string to the struct, then validate it, serialize it again and publish it on a message broker. The main part of the data is an array of arrays that contain integers, floats and datetimes. I'm using serde for deserialization and chrono to deal with datetimes.
I tried using a struct combined with an enum to allow the different types:
#[derive(Serialize, Deserialize)]
pub struct Data {
pub column_names: Option<Vec<String>>,
pub values: Vec<Vec<ValueType>>,
}
#[derive(Serialize, Deserialize)]
#[serde(untagged)]
pub enum ValueType {
I32(i32),
F64(f64),
#[serde(with = "datetime_handler")]
Dt(DateTime<Utc>),
}
Since chrono::DateTime<T> does not implement Serialize, I added a custom module for that similar to how it is described in the serde docs.
mod datetime_handler {
use chrono::{DateTime, TimeZone, Utc};
use serde::{self, Deserialize, Deserializer, Serializer};
pub fn serialize<S>(dt: &DateTime<Utc>, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
let s = dt.to_rfc3339();
serializer.serialize_str(&s)
}
pub fn deserialize<'de, D>(deserializer: D) -> Result<DateTime<Utc>, D::Error>
where
D: Deserializer<'de>,
{
println!("Checkpoint 1");
let s = String::deserialize(deserializer)?;
println!("{}", s);
println!("Checkpoint 2");
let err1 = match DateTime::parse_from_rfc3339(&s) {
Ok(dt) => return Ok(dt.with_timezone(&Utc)),
Err(e) => Err(e),
};
println!("Checkpoint 3");
const FORMAT1: &'static str = "%Y-%m-%d %H:%M:%S";
match Utc.datetime_from_str(&s, FORMAT1) {
Ok(dt) => return Ok(dt.with_timezone(&Utc)),
Err(e) => println!("{}", e), // return first error not second if both fail
};
println!("Checkpoint 4");
return err1.map_err(serde::de::Error::custom);
}
}
This tries 2 different time formats one after the other and works for DateTime strings.
The Problem
It seems like the combination of `#[derive(Serialize, Deserialize)]`, `#[serde(untagged)]` and `#[serde(with)]` does something unexpected. `serde:from_str(...)` tries to deserialize every entry in the array with my custom `deserialize` function.
I would expect it to either try to deserialize into `ValueType::I32` first, succeed and continue with the next entry, as [the docs](https://serde.rs/enum-representations.html) say:
Serde will try to match the data against each variant in order and the first one that deserializes successfully is the one returned.
What happens is that the custom deserializeis applied to e.g. "0" fails and the deserialization stops.
What's going on? How do I solve it?
My ideas are that I either fail to deserialize in the wrong way or that I somehow "overwrite" the derived deserialize with my own.
#jonasbb helped me realize the code works when using [0,16.9,"2020-12-23 00:23:14"] but it does not when trying to deserialize ["0","16.9","2020-12-23 00:23:14"]. Serde does not serialize numbers from strings by default, the attempts for I32 and F64 just fail silently. This is discussed in this serde-issue and can be solved using the inofficial serde-aux crate.
Many crates will implement serde and other common utility crates, but will leave them as optional features. This can help save time when compiling. You can check a crate by viewing the Cargo.toml file to see if there is a feature for it or the dependency is included but marked as optional.
In your case, I can go to chrono on crates.io and select the Repository link to view the source code for the crate. In the Cargo.toml file, I can see that serde is used, but is not enabled by default.
[features]
default = ["clock", "std", "oldtime"]
alloc = []
std = []
clock = ["libc", "std", "winapi"]
oldtime = ["time"]
wasmbind = ["wasm-bindgen", "js-sys"]
unstable-locales = ["pure-rust-locales", "alloc"]
__internal_bench = []
__doctest = []
[depenencies]
...
serde = { version = "1.0.99", default-features = false, optional = true }
To enable it you can go into the Cargo.toml for your project and add it as a feature to chrono.
[depenencies]
chrono = { version: "0.4.19", features = ["serde"] }
Alternatively, chrono lists some (but not all?) of their optional features in their documentation. However, not all crates do this and docs can sometimes be out of date so I usually prefer the manual method.
As for the issue between the interaction of deserialize_with and untagged on enums, I don't see any issue with your code. It may be a bug in serde so I suggest you create an issue on the serde Repository so they can further look into why this error occurs.

Rust deserialize JSON into custom HashMap<String, google_firestore1::Value>

I just started with Rust and I have some trouble with deserialization.
I'm actually trying to use the function ProjectDatabaseDocumentCreateDocumentCall from the following crate google_firestore1. I want to populate the field fields of the struct Document. The documentation of the struct is clear, it's expecting a HashMap<String, google_firestore1::Value> as a value.
The question is, how can I deserialize a JSON string to a HashMap<String, google_firestore1::Value> ?
Here is the code I wrote for the moment:
extern crate google_firestore1 as firestore1;
use google_firestore1::Document;
use std::collections::HashMap;
use serde_json;
pub fn go() {
let _my_doc = Document::default();
let test = "{\"test\":\"test\", \"myarray\": [1]}";
// Working perfectly fine
let _working: HashMap<String, serde_json::Value> = serde_json::from_str(test).unwrap();
// Not working
let _not_working: HashMap<String, firestore1::Value> = serde_json::from_str(test).unwrap();
// Later I want to do the following
// _my_doc.fields = _not_working
}
Obvsiouly this is not working, and it crashes with the following error.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("invalid type: string \"test\", expected struct Value", line: 1, column: 14)', src/firestore.rs:17:85
stack backtrace:
Of course, I noticed that serde_json::Value and firestore1::Value are not the same Struct.
But I gave a look at the source code and it seems that firestore1::Value is implementing the Deserialize trait.
So why is it not working ? In this case, do I need to iterate over the first HashMap and deserialize serde_json::Value to firestore1::Value again ? Is there a cleaner way to do what I want ?
Thanks for your answer !
The definition of the firestore1::Value is:
/// A message that can hold any of the supported value types.
///
/// This type is not used in any activity, and only used as *part* of another schema.
///
#[derive(Default, Clone, Debug, Serialize, Deserialize)]
pub struct Value {
/// A bytes value.
///
/// Must not exceed 1 MiB - 89 bytes.
/// Only the first 1,500 bytes are considered by queries.
#[serde(rename="bytesValue")]
pub bytes_value: Option<String>,
/// A timestamp value.
///
/// Precise only to microseconds. When stored, any additional precision is
/// rounded down.
#[serde(rename="timestampValue")]
pub timestamp_value: Option<String>,
...
}
This means each entry for a firestore1::Value must be an object.
I suspect that only one of the fields would actually be set, corresponding
to the actual type of the value (as they're all optional).
So your json would need to be something like:
let test = r#"{
"test":{"stringValue":"test"},
"myarray": {
"arrayValue":{"values":[{"integerValue":1}]}
}
}"#;
This is pretty ugly, so if you're doing a lot of your own JSON to firestore conversations, I'd probably write some helpers to convert from the serde_json::Value to firestore1::Value.
It would probably look something like this:
fn my_firestore_from_json(v:serde_json::Value) -> firestore1::Value {
match v {
serde_json::Value::Null => firestore::Value {
// I don't know why this is a Option<String>
null_value: Some("".to_string),
..Default::default(),
},
serde_json::Value::Bool(b) => firestore::Value {
bool_value: Some(b),
..Default::default(),
},
// Implement this
serde_json::Value::Number(n) => my_firestore_number(n),
serde_json::Value::String(s) => firestore::Value {
string_value: Some(s),
..Default::default(),
},
serde_json::Value::Array(v) => firestore::Value {
array_value:
Some(firestore1::ArrayValue{
values:v.into_iter().map(my_firestore_from_json)
}),
..Default::default(),
},
// Implement this
serde_json::Value::Object(d) => my_firststore_object(/* something */)
}
}
This would be a bit neater if there were various implementations of From<T> for the firestore1::Value, but using the implementation of
Default makes this not too ugly.
It is also worth noting that not all firebase types are created here,
since the types expressed in serde_json are different from those supported by firebase.
Anyway this allows you to use your JSON as written by doing something like:
let test = "{\"test\":\"test\", \"myarray\": [1]}";
let working: HashMap<String, serde_json::Value> = serde_json::from_str(test).unwrap();
let value_map: HashMap<String, firestore1::Value> = working.iter().map(|(k,v)| (k, my_firestore_from_json(v)).collect();

Rust: Read and map lines from stdin and handling different error types

I'm learning Rust and trying to solve some basic algorithm problems with it. In many cases, I want to read lines from stdin, perform some transformation on each line and return a vector of resulting items. One way I did this was like this:
// Fully working Rust code
let my_values: Vec<u32> = stdin
.lock()
.lines()
.filter_map(Result::ok)
.map(|line| line.parse::<u32>())
.filter_map(Result::ok)
.map(|x|x*2) // For example
.collect();
This works but of course silently ignores any errors that may occur. Now what I woud like to do is something along the lines of:
// Pseudo-ish code
let my_values: Result<Vec<u32>, X> = stdin
.lock()
.lines() // Can cause std::io::Error
.map(|line| line.parse::<u32>()) // Can cause std::num::ParseIntError
.map(|x| x*2)
.collect();
Where X is some kind of error type that I can match on afterwards. Preferably I want to perform the whole operation on one line at a time and immediately discard the string data after it has been parsed to an int.
I think I need to create some kind of Enum type to hold the various possible errors, possibly like this:
#[derive(Debug)]
enum InputError {
Io(std::io::Error),
Parse(std::num::ParseIntError),
}
However, I don't quite understand how to put everything together to make it clean and avoid having to explicitly match and cast everywhere. Also, is there some way to automatically create these enum error types or do I have to explicilty enumerate them every time I do this?
You're on the right track.
The way I'd approach this is by using the enum you've defined,
then add implementations of From for the error types you're interested in.
That will allow you to use the ? operator on your maps to get the kind of behaviour you want.
#[derive(Debug)]
enum MyError {
IOError(std::io::Error),
ParseIntError(std::num::ParseIntError),
}
impl From<std::io::Error> for MyError {
fn from(e:std::io::Error) -> MyError {
return MyError::IOError(e)
}
}
impl From<std::num::ParseIntError> for MyError {
fn from(e:std::num::ParseIntError) -> MyError {
return MyError::ParseIntError(e)
}
}
Then you can implement the actual transform as either
let my_values: Vec<_> = stdin
.lock()
.lines()
.map(|line| -> Result<u32,MyError> { Ok(line?.parse::<u32>()?*2) } )
.collect();
which will give you one entry for each input, like: {Ok(x), Err(MyError(x)), Ok(x)}.
or you can do:
let my_values: Result<Vec<_>,MyError> = stdin
.lock()
.lines()
.map(|line| -> Result<u32,MyError> { Ok(line?.parse::<u32>()?*2) } )
.collect();
Which will give you either Err(MyError(...)) or Ok([1,2,3])
Note that you can further reduce some of the error boilerplate by using an error handling crate like snafu, but in this case it's not too much.

Is there a way to get SNAFU's `.backtrace()` on arbitrary `&dyn std::error::Error` trait objects?

RFC 2504 will add a required fn backtrace(&self) -> Option<&Backtrace> to all std::error::Error. This is not ready yet, so for now, SNAFU, an error helper macro, polyfills this by tying an ErrorCompat trait to all types generated by the macro. This allows for backtrace support before it lands in Rust nightly.
However, this ErrorCompat trait is not implemented for all implementors of std::error::Error. I want to — in some generic error printing code — be able to display the chain of causes along with the stacktrace associated with where the SNAFU error was instantiated. Unfortunately, the source() function returns &(dyn Error + 'static).
use std::error::Error as StdError;
use snafu::{ResultExt, ErrorCompat};
fn main() {
let err: Result<(), _> = Err(std::io::Error::new(std::io::ErrorKind::Other, "oh no!"));
let err = err.with_context(|| parse_error::ReadInput {
filename: "hello"
});
let err = err.with_context(|| compile_error::ParseStage);
// some generic error handling code
if let Err(err) = err {
// `cause` is type &(dyn std::error::Error + 'static)
let cause = err.source().unwrap();
if let Some(err) = /* attempt to downcast cause into &dyn snafu::ErrorCompat trait object */ {
println!("{}", err.backtrace().unwrap());
}
}
}
pub mod compile_error {
use snafu::{Snafu, Backtrace};
#[derive(Debug, Snafu)]
#[snafu(visibility(pub(super)))]
pub enum Error {
#[snafu(display("Error parsing code: {}", source))]
ParseStage {
source: crate::parse_error::Error,
backtrace: Backtrace
},
}
}
pub mod parse_error {
use snafu::{Snafu, Backtrace};
#[derive(Debug, Snafu)]
#[snafu(visibility(pub(super)))]
pub enum Error {
#[snafu(display("Could not read input {:?}: {}", filename, source))]
ReadInput {
filename: std::path::PathBuf,
source: std::io::Error,
backtrace: Backtrace
},
}
}
I've looked at std::any::Any::downcast_ref but this is for downcasting to a struct, not downcasting a trait object to another trait object. I'd like to avoid having to list out all possible concrete-typed SNAFU errors in my error-handling code.
I could cryo-freeze myself until RFC 2504 is (fully) implemented but surely there's some way to do this.
A dyn Error has the methods of Error and nothing else. If the backtrace cannot be deduced from those methods then where else could that information come from?
Unfortunately RFC 2504 is not yet stabilised, so you will need to be cryogenically frozen until at least Rust 1.39 if you want to wait for it.
It seems I missed this because nightly std docs weren't recompiled, but #![feature(backtrace)] is in nightly right now. SNAFU still needs to add support for it, so I'm still stuck on getting this all working.

Iterating over a slice's values instead of references in Rust?

When looping over a slice of structs, the value I get is a reference (which is fine), however in some cases it's annoying to have to write var as (*var) in many places.
Is there a better way to avoid re-declaring the variable?
fn my_fn(slice: &[MyStruct]) {
for var in slice {
let var = *var; // <-- how to avoid this?
// Without the line above, errors in comments occur:
other_fn(var); // <-- expected struct `MyStruct`, found reference
if var != var.other {
// ^^ trait `&MyStruct: std::cmp::PartialEq<MyStruct>>` not satisfied
foo();
}
}
}
See: actual error output (more cryptic).
You can remove the reference by destructuring in the pattern:
// |
// v
for &var in slice {
other_fn(var);
}
However, this only works for Copy-types! If you have a type that doesn't implement Copy but does implement Clone, you could use the cloned() iterator adapter; see Chris Emerson's answer for more information.
In some cases you can iterate directly on values if you can consume the iterable, e.g. using Vec::into_iter().
With slices, you can use cloned or copied on the iterator:
fn main() {
let v = vec![1, 2, 3];
let slice = &v[..];
for u in slice.iter().cloned() {
let u: usize = u; // prove it's really usize, not &usize
println!("{}", u);
}
}
This relies on the item implementing Clone or Copy, but if it doesn't you probably do want references after all.