Rust deserialize JSON into custom HashMap<String, google_firestore1::Value> - serialization

I just started with Rust and I have some trouble with deserialization.
I'm actually trying to use the function ProjectDatabaseDocumentCreateDocumentCall from the following crate google_firestore1. I want to populate the field fields of the struct Document. The documentation of the struct is clear, it's expecting a HashMap<String, google_firestore1::Value> as a value.
The question is, how can I deserialize a JSON string to a HashMap<String, google_firestore1::Value> ?
Here is the code I wrote for the moment:
extern crate google_firestore1 as firestore1;
use google_firestore1::Document;
use std::collections::HashMap;
use serde_json;
pub fn go() {
let _my_doc = Document::default();
let test = "{\"test\":\"test\", \"myarray\": [1]}";
// Working perfectly fine
let _working: HashMap<String, serde_json::Value> = serde_json::from_str(test).unwrap();
// Not working
let _not_working: HashMap<String, firestore1::Value> = serde_json::from_str(test).unwrap();
// Later I want to do the following
// _my_doc.fields = _not_working
}
Obvsiouly this is not working, and it crashes with the following error.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("invalid type: string \"test\", expected struct Value", line: 1, column: 14)', src/firestore.rs:17:85
stack backtrace:
Of course, I noticed that serde_json::Value and firestore1::Value are not the same Struct.
But I gave a look at the source code and it seems that firestore1::Value is implementing the Deserialize trait.
So why is it not working ? In this case, do I need to iterate over the first HashMap and deserialize serde_json::Value to firestore1::Value again ? Is there a cleaner way to do what I want ?
Thanks for your answer !

The definition of the firestore1::Value is:
/// A message that can hold any of the supported value types.
///
/// This type is not used in any activity, and only used as *part* of another schema.
///
#[derive(Default, Clone, Debug, Serialize, Deserialize)]
pub struct Value {
/// A bytes value.
///
/// Must not exceed 1 MiB - 89 bytes.
/// Only the first 1,500 bytes are considered by queries.
#[serde(rename="bytesValue")]
pub bytes_value: Option<String>,
/// A timestamp value.
///
/// Precise only to microseconds. When stored, any additional precision is
/// rounded down.
#[serde(rename="timestampValue")]
pub timestamp_value: Option<String>,
...
}
This means each entry for a firestore1::Value must be an object.
I suspect that only one of the fields would actually be set, corresponding
to the actual type of the value (as they're all optional).
So your json would need to be something like:
let test = r#"{
"test":{"stringValue":"test"},
"myarray": {
"arrayValue":{"values":[{"integerValue":1}]}
}
}"#;
This is pretty ugly, so if you're doing a lot of your own JSON to firestore conversations, I'd probably write some helpers to convert from the serde_json::Value to firestore1::Value.
It would probably look something like this:
fn my_firestore_from_json(v:serde_json::Value) -> firestore1::Value {
match v {
serde_json::Value::Null => firestore::Value {
// I don't know why this is a Option<String>
null_value: Some("".to_string),
..Default::default(),
},
serde_json::Value::Bool(b) => firestore::Value {
bool_value: Some(b),
..Default::default(),
},
// Implement this
serde_json::Value::Number(n) => my_firestore_number(n),
serde_json::Value::String(s) => firestore::Value {
string_value: Some(s),
..Default::default(),
},
serde_json::Value::Array(v) => firestore::Value {
array_value:
Some(firestore1::ArrayValue{
values:v.into_iter().map(my_firestore_from_json)
}),
..Default::default(),
},
// Implement this
serde_json::Value::Object(d) => my_firststore_object(/* something */)
}
}
This would be a bit neater if there were various implementations of From<T> for the firestore1::Value, but using the implementation of
Default makes this not too ugly.
It is also worth noting that not all firebase types are created here,
since the types expressed in serde_json are different from those supported by firebase.
Anyway this allows you to use your JSON as written by doing something like:
let test = "{\"test\":\"test\", \"myarray\": [1]}";
let working: HashMap<String, serde_json::Value> = serde_json::from_str(test).unwrap();
let value_map: HashMap<String, firestore1::Value> = working.iter().map(|(k,v)| (k, my_firestore_from_json(v)).collect();

Related

How do I return raw bytes in a HTTP Response in Tower Web?

I am developing an API using the tower_web framework, and for some API calls, it would be best to just return bytes with a content type like "application/octet-stream". This data is not directly read from a file, but generated dynamically, so I can't refer to the example code for serving a static file.
For implementing the desired functionality, I tried using a http::Response<Vec<u8>> with a body using a u8 array. Here are the relevant snippets of my code, for testing purposes I used a u8 array filled with zeroes, that would later be the data I want to send to the client:
extern crate tower_web;
use tower_web::ServiceBuilder;
use http::Response as HttpResponse;
const SERVER_ADDRESS : &str = "127.0.0.1:8080";
[...]
#[derive(Clone, Debug)]
pub struct HttpApi;
impl_web! {
impl HttpApi {
[...]
#[get("/test")]
#[content_type("application/octet-stream")]
fn test(&self) -> http::Response<Vec<u8>> {
let response = HttpResponse::builder().status(200).body([0u8;16]);
println!("{:?}", response);
response
}
}
}
pub fn main() {
let addr = SERVER_ADDRESS.parse().expect("Invalid address");
println!("Listening on http://{}", addr);
ServiceBuilder::new()
.resource(HttpApi)
.run(&addr)
.unwrap();
}
However, when I try to build it using cargo, I get this error:
the trait `tower_web::codegen::futures::Future` is not implemented for `http::Response<Vec<u8>>`
I also tried setting the type returned by the function to a Result wrapping the http::Response<Vec<u8>> and (), as it would work that way when I try returning strings or json. This gave me the error
the trait `BufStream` is not implemented for `Vec<u8>`
Is there any simple way to solve this, or any other approach for returning bytes as the body of a HTTP Response in Tower Web?

Deserializing an enum using a combination of #[serde(untagged)] and #[serde(with)]

I'm trying to use an actix-web server as a gateway to a small stack to guarantee a strict data format inside of the stack while allowing some freedoms for the user.
To do that, I want to deserialize a JSON string to the struct, then validate it, serialize it again and publish it on a message broker. The main part of the data is an array of arrays that contain integers, floats and datetimes. I'm using serde for deserialization and chrono to deal with datetimes.
I tried using a struct combined with an enum to allow the different types:
#[derive(Serialize, Deserialize)]
pub struct Data {
pub column_names: Option<Vec<String>>,
pub values: Vec<Vec<ValueType>>,
}
#[derive(Serialize, Deserialize)]
#[serde(untagged)]
pub enum ValueType {
I32(i32),
F64(f64),
#[serde(with = "datetime_handler")]
Dt(DateTime<Utc>),
}
Since chrono::DateTime<T> does not implement Serialize, I added a custom module for that similar to how it is described in the serde docs.
mod datetime_handler {
use chrono::{DateTime, TimeZone, Utc};
use serde::{self, Deserialize, Deserializer, Serializer};
pub fn serialize<S>(dt: &DateTime<Utc>, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
let s = dt.to_rfc3339();
serializer.serialize_str(&s)
}
pub fn deserialize<'de, D>(deserializer: D) -> Result<DateTime<Utc>, D::Error>
where
D: Deserializer<'de>,
{
println!("Checkpoint 1");
let s = String::deserialize(deserializer)?;
println!("{}", s);
println!("Checkpoint 2");
let err1 = match DateTime::parse_from_rfc3339(&s) {
Ok(dt) => return Ok(dt.with_timezone(&Utc)),
Err(e) => Err(e),
};
println!("Checkpoint 3");
const FORMAT1: &'static str = "%Y-%m-%d %H:%M:%S";
match Utc.datetime_from_str(&s, FORMAT1) {
Ok(dt) => return Ok(dt.with_timezone(&Utc)),
Err(e) => println!("{}", e), // return first error not second if both fail
};
println!("Checkpoint 4");
return err1.map_err(serde::de::Error::custom);
}
}
This tries 2 different time formats one after the other and works for DateTime strings.
The Problem
It seems like the combination of `#[derive(Serialize, Deserialize)]`, `#[serde(untagged)]` and `#[serde(with)]` does something unexpected. `serde:from_str(...)` tries to deserialize every entry in the array with my custom `deserialize` function.
I would expect it to either try to deserialize into `ValueType::I32` first, succeed and continue with the next entry, as [the docs](https://serde.rs/enum-representations.html) say:
Serde will try to match the data against each variant in order and the first one that deserializes successfully is the one returned.
What happens is that the custom deserializeis applied to e.g. "0" fails and the deserialization stops.
What's going on? How do I solve it?
My ideas are that I either fail to deserialize in the wrong way or that I somehow "overwrite" the derived deserialize with my own.
#jonasbb helped me realize the code works when using [0,16.9,"2020-12-23 00:23:14"] but it does not when trying to deserialize ["0","16.9","2020-12-23 00:23:14"]. Serde does not serialize numbers from strings by default, the attempts for I32 and F64 just fail silently. This is discussed in this serde-issue and can be solved using the inofficial serde-aux crate.
Many crates will implement serde and other common utility crates, but will leave them as optional features. This can help save time when compiling. You can check a crate by viewing the Cargo.toml file to see if there is a feature for it or the dependency is included but marked as optional.
In your case, I can go to chrono on crates.io and select the Repository link to view the source code for the crate. In the Cargo.toml file, I can see that serde is used, but is not enabled by default.
[features]
default = ["clock", "std", "oldtime"]
alloc = []
std = []
clock = ["libc", "std", "winapi"]
oldtime = ["time"]
wasmbind = ["wasm-bindgen", "js-sys"]
unstable-locales = ["pure-rust-locales", "alloc"]
__internal_bench = []
__doctest = []
[depenencies]
...
serde = { version = "1.0.99", default-features = false, optional = true }
To enable it you can go into the Cargo.toml for your project and add it as a feature to chrono.
[depenencies]
chrono = { version: "0.4.19", features = ["serde"] }
Alternatively, chrono lists some (but not all?) of their optional features in their documentation. However, not all crates do this and docs can sometimes be out of date so I usually prefer the manual method.
As for the issue between the interaction of deserialize_with and untagged on enums, I don't see any issue with your code. It may be a bug in serde so I suggest you create an issue on the serde Repository so they can further look into why this error occurs.

Rust: Read and map lines from stdin and handling different error types

I'm learning Rust and trying to solve some basic algorithm problems with it. In many cases, I want to read lines from stdin, perform some transformation on each line and return a vector of resulting items. One way I did this was like this:
// Fully working Rust code
let my_values: Vec<u32> = stdin
.lock()
.lines()
.filter_map(Result::ok)
.map(|line| line.parse::<u32>())
.filter_map(Result::ok)
.map(|x|x*2) // For example
.collect();
This works but of course silently ignores any errors that may occur. Now what I woud like to do is something along the lines of:
// Pseudo-ish code
let my_values: Result<Vec<u32>, X> = stdin
.lock()
.lines() // Can cause std::io::Error
.map(|line| line.parse::<u32>()) // Can cause std::num::ParseIntError
.map(|x| x*2)
.collect();
Where X is some kind of error type that I can match on afterwards. Preferably I want to perform the whole operation on one line at a time and immediately discard the string data after it has been parsed to an int.
I think I need to create some kind of Enum type to hold the various possible errors, possibly like this:
#[derive(Debug)]
enum InputError {
Io(std::io::Error),
Parse(std::num::ParseIntError),
}
However, I don't quite understand how to put everything together to make it clean and avoid having to explicitly match and cast everywhere. Also, is there some way to automatically create these enum error types or do I have to explicilty enumerate them every time I do this?
You're on the right track.
The way I'd approach this is by using the enum you've defined,
then add implementations of From for the error types you're interested in.
That will allow you to use the ? operator on your maps to get the kind of behaviour you want.
#[derive(Debug)]
enum MyError {
IOError(std::io::Error),
ParseIntError(std::num::ParseIntError),
}
impl From<std::io::Error> for MyError {
fn from(e:std::io::Error) -> MyError {
return MyError::IOError(e)
}
}
impl From<std::num::ParseIntError> for MyError {
fn from(e:std::num::ParseIntError) -> MyError {
return MyError::ParseIntError(e)
}
}
Then you can implement the actual transform as either
let my_values: Vec<_> = stdin
.lock()
.lines()
.map(|line| -> Result<u32,MyError> { Ok(line?.parse::<u32>()?*2) } )
.collect();
which will give you one entry for each input, like: {Ok(x), Err(MyError(x)), Ok(x)}.
or you can do:
let my_values: Result<Vec<_>,MyError> = stdin
.lock()
.lines()
.map(|line| -> Result<u32,MyError> { Ok(line?.parse::<u32>()?*2) } )
.collect();
Which will give you either Err(MyError(...)) or Ok([1,2,3])
Note that you can further reduce some of the error boilerplate by using an error handling crate like snafu, but in this case it's not too much.

How to match custom Fails with the failure crate

I'm trying to understand how to use the failure crate. It works splendidly as a unification of different types of standard errors, but when creating custom errors (Fails), I do not understand how to match for custom errors. For example:
use failure::{Fail, Error};
#[derive(Debug, Fail)]
pub enum Badness {
#[fail(display = "Ze badness")]
Level(String)
}
pub fn do_badly() -> Result<(), Error> {
Err(Badness::Level("much".to_owned()).into())
}
#[test]
pub fn get_badness() {
match do_badly() {
Err(Badness::Level(level)) => panic!("{:?} badness!", level),
_ => (),
};
}
fails with
error[E0308]: mismatched types
--> barsa-nagios-forwarder/src/main.rs:74:9
|
73 | match do_badly() {
| ---------- this match expression has type `failure::Error`
74 | Err(Badness::Level(level)) => panic!("{:?} badness!", level),
| ^^^^^^^^^^^^^^^^^^^^^ expected struct `failure::Error`, found enum `Badness`
|
= note: expected type `failure::Error`
found type `Badness`
How can I formulate a pattern which matches a specific custom error?
You need to downcast the Error
When you create a failure::Error from some type that implements the Fail trait (via from or into, as you do), you temporarily hide the information about the type you're wrapping from the compiler. It doesn't know that Error is a Badness - because it can also be any other Fail type, that's the point. You need to remind the compiler of this, the action is called downcasting. The failure::Error has three methods for this: downcast, downcast_ref and downcast_mut. After you've downcast it, you can pattern match on the result as normal - but you need to take into account the possibility that downcasting itself may fail (if you try to downcast to a wrong type).
Here's how it'd look with downcast:
pub fn get_badness() {
if let Err(wrapped_error) = do_badly() {
if let Ok(bad) = wrapped_error.downcast::<Badness>() {
panic!("{:?} badness!", bad);
}
}
}
(two if lets can be combined in this case).
This quickly gets very unpleasant if more than one error type needs to be tested, since downcast consumes the failure::Error it was called on (so you can't try another downcast on the same variable if the first one fails). I sadly couldn't figure out an elegant way to do this. Here's a variant one shouldn't really use (panic! in map is questionable, and doing anything else there would be plenty awkward, and I don't even want to think about more cases than two):
#[derive(Debug, Fail)]
pub enum JustSoSo {
#[fail(display = "meh")]
Average,
}
pub fn get_badness() {
if let Err(wrapped_error) = do_badly() {
let e = wrapped_error.downcast::<Badness>()
.map(|bad| panic!("{:?} badness!", bad))
.or_else(|original| original.downcast::<JustSoSo>());
if let Ok(so) = e {
println!("{}", so);
}
}
}
or_else chain should work OK if you actually want to produce some value of the same type from all of the possible\relevant errors. Consider also using non-consuming methods if a reference to the original error is fine for you, as this would allow you to just make a series of if let blocks , one for each downcast attempt.
An alternative
Don't put your errors into failure::Error, put them in a custom enum as variants. It's more boilerplate, but you get painless pattern matching, which the compiler also will be able to check for sanity. If you choose to do this, I'd recommend derive_more crate which is capable of deriving From for such enums; snafu looks very interesting as well, but I have yet to try it. In its most basic form this approach looks like this:
pub enum SomeError {
Bad(Badness),
NotTooBad(JustSoSo),
}
pub fn do_badly_alt() -> Result<(), SomeError> {
Err(SomeError::Bad(Badness::Level("much".to_owned())))
}
pub fn get_badness_alt() {
if let Err(wrapper) = do_badly_alt() {
match wrapper {
SomeError::Bad(bad) => panic!("{:?} badness!", bad),
SomeError::NotTooBad(so) => println!("{}", so),
}
}
}

How do I execute Dynamically (like Eval) in Dart?

Since getting started in Dart I've been watching for a way to execute Dart (Text) Source (that the same program may well be generating dynamically) as Code. Like the infamous "eval()" function.
Recently I have caught a few hints that the communication port between Isolates support some sort of "Spawn" that seems like it could allow this "trick". In Ruby there is also the possibility to load a module dynamically as a language feature, perhaps there is some way to do this in Dart?
Any clues or a simple example will be greatly appreciated.
Thanks in advance!
Ladislav Thon provided this answer on the Dart forum:
I believe it's very safe to say that Dart will never have eval. But it will have other, more structured ways of dynamically generating code (code name mirror builders). There is nothing like that right now, though.
There are two ways of spawning an isolate: spawnFunction, which runs an existing function from the existing code in a new isolate, so nothing you are looking for, and spawnUri, which downloads code from given URI and runs it in new isolate. That is essentially dynamic code loading -- but the dynamically loaded code is isolated from the existing code. It runs in a new isolate, so the only means of communicating with it is via message passing (through ports).
You can run a string as Dart code by first constructing a data URI from it and then passing it into Isolate.spawnUri.
import 'dart:isolate';
void main() async {
final uri = Uri.dataFromString(
'''
void main() {
print("Hellooooooo from the other side!");
}
''',
mimeType: 'application/dart',
);
await Isolate.spawnUri(uri, [], null);
}
Note that you can only do this in JIT mode, which means that the only place you might benefit from it is Dart VM command line apps / package:build scripts. It will not work in Flutter release builds.
To get a result back from it, you can use ports:
import 'dart:isolate';
void main() async {
final name = 'Eval Knievel';
final uri = Uri.dataFromString(
'''
import "dart:isolate";
void main(_, SendPort port) {
port.send("Nice to meet you, $name!");
}
''',
mimeType: 'application/dart',
);
final port = ReceivePort();
await Isolate.spawnUri(uri, [], port.sendPort);
final String response = await port.first;
print(response);
}
I wrote about it on my blog.
Eval(), in Ruby at least, can execute anything from a single statement (like an assignment) to complete involved programs. There is a substantial time penalty for executing many small snippets over most any other form of execution that is possible.
Looking at the problem closer, there are at least three different functions that were at the base of the various schemes where eval might be used. Dart handles at least 2 of these in at least minimal ways.
Dart does not, nor does it look like there is any plan to support "general" script execution.
However, the NoSuchMethod method can be used to effectively implement the dynamic "injection" of variables into your local class environment. It replaces an eval() with a string that would look like this: eval( "String text = 'your first name here';" );
The second function that Dart readily supports now is the invocation of a method, that would look like this: eval( "Map map = SomeClass.some_method()" );
After messing about with this it finally dawned on me that a single simple class can be used to store the information needed to invoke a method, for a class, as a string which seems to have general utility. I can replace a big maintenance prone switch statement that might otherwise be necessary to invoke a series of methods. In Ruby this was almost trivial, however in Dart there are some fairly less than intuitive calls so I wanted to get this "trick" in one place, which fits will with doing ordering and filtering on the strings such as you may need.
Here's the code to "accumulate" as many classes (a whole library?) into a map using reflection such that the class.methodName() can be called with nothing more than a key (as a string).
Note: I used a few "helper methods" to do Map & List functions, you will probably want to replace them with straight Dart. However this code is used and tested only using the functions..
Here's the code:
//The used "Helpers" here..
MAP_add(var map, var key, var value){ if(key != null){map[key] = value;}return(map);}
Object MAP_fetch(var map, var key, [var dflt = null]) {var value = map[key];if (value==null) {value = dflt;}return( value );}
class ClassMethodMapper {
Map _helperMirrorsMap, _methodMap;
void accum_class_map(Object myClass){
InstanceMirror helperMirror = reflect(myClass);
List methodsAr = helperMirror.type.methods.values;
String classNm = myClass.toString().split("'")[1]; ///#FRAGILE
MAP_add(_helperMirrorsMap, classNm, helperMirror);
methodsAr.forEach(( method) {
String key = method.simpleName;
if (key.charCodeAt(0) != 95) { //Ignore private methods
MAP_add(_methodMap, "${classNm}.${key}()", method);
}
});
}
Map invoker( String methodNm ) {
var method = MAP_fetch(_methodMap, methodNm, null);
if (method != null) {
String classNm = methodNm.split('.')[0];
InstanceMirror helperMirror = MAP_fetch(_helperMirrorsMap, classNm);
helperMirror.invoke(method.simpleName, []);
}
}
ClassMethodMapper() {
_methodMap = {};
_helperMirrorsMap = {};
}
}//END_OF_CLASS( ClassMethodMapper );
============
main() {
ClassMethodMapper cMM = new ClassMethodMapper();
cMM.accum_class_map(MyFirstExampleClass);
cMM.accum_class_map(MySecondExampleClass);
//Now you're ready to execute any method (not private as per a special line of code above)
//by simply doing this:
cMM.invoker( MyFirstExampleClass.my_example_method() );
}
Actually there some libraries in pub.dev/packages but has some limitations because are young versions, so that I can recommend you this library expressions to dart and flutter.
A library to parse and evaluate simple expressions.
This library can handle simple expressions, but no blocks of code, control flow statements and so on. It supports a syntax that is common to most programming languages.
There I create an example of code to evaluate arithmetic operations and comparations of data.
import 'package:expressions/expressions.dart';
import 'dart:math';
#override
Widget build(BuildContext context) {
final parsing = FormulaMath();
// Expression example
String condition = "(cos(x)*cos(x)+sin(x)*sin(x)==1) && respuesta_texto == 'si'";
Expression expression = Expression.parse(condition);
var context = {
"x": pi / 5,
"cos": cos,
"sin": sin,
"respuesta_texto" : 'si'
};
// Evaluate expression
final evaluator = const ExpressionEvaluator();
var r = evaluator.eval(expression, context);
print(r);
return Scaffold(
body: Container(
margin: EdgeInsets.only(top: 50.0),
child: Column(
children: [
Text(condition),
Text(r.toString())
],
),
),
);
}
I/flutter (27188): true