Is there a way to remove unwrapping of items in an iterator chain that filters and partitions based on a Result? - iterator

I created this example code:
fn main() {
let books = vec![
Book {
data: Ok("type1".to_owned()),
metadata: "meta1".to_owned(),
},
Book {
data: Err("-".to_owned()),
metadata: "meta2".to_owned(),
},
Book {
data: Ok("type2".to_owned()),
metadata: "meta2".to_owned(),
},
];
// metadata without data being error
let (book_type_1, book_type_2): &(Vec<_>, Vec<_>) = &books
.iter()
.filter(|f| f.data.is_ok())
.partition(|p| p.data.as_ref().unwrap() == "type1");
println!("Books {:?}", books);
println!("Type 1 {:?}", book_type_1); // Should be the original Vec<Book> with Type 1 filtered.
println!("Type 2 {:?}", book_type_2); // Should be the original Vec<Book> with Type 2 filtered.
}
#[derive(Debug)]
struct Book {
data: Result<String, String>,
metadata: String,
}
On the let (book_type_1, book_type_2) expression, I need to use Book::data twice, but I already filtered it so I know it can't be Err. Is there a way to restructure to remove the use of unwrap here?

I'm not sure exactly what you mean, but it seems like you want to use Iterator::filter_map(). It lets you filter for values that are Some(T), which then get passed on as unwrapped Ts.
So what you can do is convert your Results to Options with Result::ok(), so a Result::Ok(T) will become Some(T) which means it passes the filter as T.
fn main() {
let books = vec![
Book {
data: Ok("type1".to_owned()),
metadata: "meta1".to_owned(),
},
Book {
data: Err("-".to_owned()),
metadata: "meta2".to_owned(),
},
Book {
data: Ok("type2".to_owned()),
metadata: "meta2".to_owned(),
},
];
// metadata without data being error
let (book_type_1, book_type_2): (Vec<_>, Vec<_>) = books
.iter()
.filter_map(|f| {
match &f.data {
Ok(data) => Some((f, data)),
Err(_) => None,
}
})
.partition(|(book, data)| *data == "type1");
println!("Books {:?}", books);
println!("Type 1 {:?}", book_type_1);
println!("Type 2 {:?}", book_type_2);
}
#[derive(Debug)]
struct Book {
data: Result<String, String>,
metadata: String,
}
playground
I removed the unnecessary reference to the returned partitioned tuple.
Also note that None pertains to Option<T>, but you're using Result<T, E>. I think you knew that but just making sure.

I would use flat_map as described in the other answer, but I'd leave the yielded values as a Result. Result implements IntoIterator, so you can use it directly in flat_map.
I'd also use Result::as_ref instead of writing out the match explicitly.
I'd then use Itertools::partition_map to simultaneously select between the types and remove the extra property:
extern crate itertools;
use itertools::{Either, Itertools};
// ...
let (book_type_1, book_type_2): (Vec<_>, Vec<_>) = books
.iter()
.flat_map(|b| b.data.as_ref().map(|data| (b, data)))
.partition_map(|(b, data)| {
if data == "type1" {
Either::Left(b)
} else {
Either::Right(b)
}
});
Note:
There's no reason to take a reference to the result tuple.
This only works because you are iterating on references to books.
If you needed to operate on the owned Books, I'd move the comparison into the flat_map call and pass it along:
let (book_type_1, book_type_2): (Vec<_>, Vec<_>) = books
.into_iter()
.flat_map(|book| {
book.data
.as_ref()
.ok()
.map(|data| data == "type1")
.map(|is_type1| (is_type1, book))
})
.partition_map(|(is_type1, book)| {
if is_type1 {
Either::Left(book)
} else {
Either::Right(book)
}
});
There's also the pragmatic solution of sorting all errors into the same bin as one of the types. Since you know that there won't be any errors, this has no effect:
let (book_type_1, book_type_2): (Vec<_>, Vec<_>) = books
.iter()
.filter(|f| f.data.is_ok())
.partition(|p| p.data.as_ref().ok().map_or(false, |d| d == "type1"));
See also:
What's the most idiomatic way of working with an Iterator of Results?

Related

Idiomatic way to collect all errors from an iterator

Let's say I have a attrs: Vec<Attribute> of some function attributes and a function fn map_attribute(attr: &Attribute) -> Result<TokenStream, Error> that maps the attributes to some code.
I know that I could write something like this:
attrs.into_iter()
.map(map_attribute)
.collect::<Result<Vec<_>, _>()?
However, this is not what I want. What I want is spit out all errors at once, not stop with the first Error. Currently I do something like this:
let mut codes : Vec<TokenStream> = Vec::new();
let mut errors: Vec<Error> = Vec::new();
for attr in attrs {
match map_attribute(attr) {
Ok(code) => codes.push(code),
Err(err) => errors.push(err)
}
}
let mut error_iter = errors.into_iter();
if let Some(first) = error_iter.nth(0) {
return Err(iter.fold(first, |mut e0, e1| { e0.combine(e1); e0 }));
}
This second version does what I want, but is considerably more verbose than the first version. Is there a better / more idiomatic way to acchieve this, if possible without creating my own iterator?
The standard library does not have a convenient one-liner for this as far as I know, however the excellent itertools library does:
use itertools::Itertools; // 0.9.0
fn main() {
let foo = vec![Ok(42), Err(":("), Ok(321), Err("oh noes")];
let (codes, errors): (Vec<_>, Vec<_>)
= foo.into_iter().partition_map(From::from);
println!("codes={:?}", codes);
println!("errors={:?}", errors);
}
(Permalink to the playground)
I ended up writing my own extension for Iterator, which allows me to stop collecting codes when I encounter my first error. This is in my use case probably a bit more efficient than the answer by mcarton, since I only need the first partition bucket if the second one is empty. Also, I need to fold the errors anyways if I want to combine them into a single error.
pub trait CollectToResult
{
type Item;
fn collect_to_result(self) -> Result<Vec<Self::Item>, Error>;
}
impl<Item, I> CollectToResult for I
where
I : Iterator<Item = Result<Item, Error>>
{
type Item = Item;
fn collect_to_result(self) -> Result<Vec<Item>, Error>
{
self.fold(<Result<Vec<Item>, Error>>::Ok(Vec::new()), |res, code| {
match (code, res) {
(Ok(code), Ok(mut codes)) => { codes.push(code); Ok(codes) },
(Ok(_), Err(errors)) => Err(errors),
(Err(err), Ok(_)) => Err(err),
(Err(err), Err(mut errors)) => { errors.combine(err); Err(errors) }
}
})
}
}

How to use Serde to parse a field that might fail to be deserialized without failing the entire deserialization?

I am deserializing some JSON objects which come in as requests. The input body is nested, but a certain field is sometimes misformatted for a variety of reasons. In that situation I still want the rest of the object. This doesn't all have to be done through serde; but what is happening now, is that if a single subfield is messed up, the whole request is trashed. I want to somehow still deserialize that result and just mark the field as errored out. How can this be done?
E.g. the data schema might look like:
struct BigNested {
a: Vec<A>,
b: B, // definition omitted
}
struct A {
keep_this: Foo,
trouble: SometimesBad,
}
trouble is the field that's frequently coming in messed up. I would be happy to (e.g.) turn trouble into a Result<SometimesBad, Whatever> and process it from there, but I don't know how to get serde to let me do that.
certain field is sometimes misformatted
You didn't say how malformed the incoming JSON was. Assuming it's still valid JSON, you can pull this off with Serde's struct flatten and customized deserialization:
The customized deserialization is done in a way that never fails for valid JSON input, although it may not return value of expected type if the input has unexpected format.
But these unexpected fields still need to go somewhere. Serde's struct flatten comes in handy here to catch them since any JSON snippet can be deserialized to a HashMap<String, Value>.
//# serde = { version = "1.0.103", features = ["derive"] }
//# serde_json = "1.0.44"
use serde::{Deserialize, Deserializer, de::DeserializeOwned};
use serde_json::Value;
use std::collections::HashMap;
#[derive(Deserialize, Debug)]
struct A {
keep_this: Foo,
trouble: SometimesBad,
}
#[derive(Deserialize, Debug)]
struct Foo {
foo: i32,
}
#[derive(Deserialize, Debug)]
struct SometimesBad {
inner: TryParse<Bar>,
#[serde(flatten)]
blackhole: HashMap<String, Value>,
}
#[derive(Deserialize, Debug)]
struct Bar {
bar: String,
}
#[derive(Debug)]
enum TryParse<T> {
Parsed(T),
Unparsed(Value),
NotPresent
}
impl<'de, T: DeserializeOwned> Deserialize<'de> for TryParse<T> {
fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
match Option::<Value>::deserialize(deserializer)? {
None => Ok(TryParse::NotPresent),
Some(value) => match T::deserialize(&value) {
Ok(t) => Ok(TryParse::Parsed(t)),
Err(_) => Ok(TryParse::Unparsed(value)),
},
}
}
}
fn main() {
let valid = r#"{ "keep_this": { "foo": 1 }, "trouble": { "inner": { "bar": "one"}}}"#;
println!("{:#?}", serde_json::from_str::<A>(valid));
let extra_field = r#"{ "keep_this": { "foo": 1 }, "trouble": { "inner": { "bar": "one"}, "extra": 2019}}"#;
println!("{:#?}", serde_json::from_str::<A>(extra_field));
let wrong_type = r#"{ "keep_this": { "foo": 1 }, "trouble": { "inner": { "bar": 1}}}"#;
println!("{:#?}", serde_json::from_str::<A>(wrong_type));
let missing_field = r#"{ "keep_this": { "foo": 1 }, "trouble": { "inner": { "baz": "one"}}}"#;
println!("{:#?}", serde_json::from_str::<A>(missing_field));
let missing_inner = r#"{ "keep_this": { "foo": 1 }, "trouble": { "whatever": { "bar": "one"}}}"#;
println!("{:#?}", serde_json::from_str::<A>(missing_inner));
}
(The credit isn't all mine. Serde's issue 1583 basically has everything.)

GraphQL, Dataloader, [ORM or not], hasMany relationship understanding

I'm using for the first time Facebook's dataloader (https://github.com/facebook/dataloader).
What I don't understand is how to use it when I have 1 to many relationships.
Here it is a reproduction of my problem: https://enshrined-hydrant.glitch.me.
If you use this query in the Playground:
query {
persons {
name
bestFriend {
name
}
opponents {
name
}
}
}
you get values.
But if you open the console log here: https://glitch.com/edit/#!/enshrined-hydrant you can see these database calls I want to avoid:
My Person type is:
type Person {
id: ID!
name: String!
bestFriend: Person
opponents: [Person]
}
I can use dataloader good for bestFriend: Person but I don't understand how to use it with opponents: [Person].
As you can see the resolver has to return an array of values.
Have you any hint about this?
You need to create batched endpoints to work with dataloader - it can't do batching by itself.
For example, you probably want the following endpoints:
GET /persons - returns all people
POST /bestFriends, Array<personId>` - returns an array of best friends matchin the corresponding array of `personId`s
Then, your dataloaders can look like:
function batchedBestFriends(personIds) {
return fetch('/bestFriends', {
method: 'POST',
body: JSON.stringify(personIds))
}).then(response => response.json());
// We assume above that the API returns a straight array of the data.
// If the data was keyed, you could add another accessor such as
// .then(data => data.bestFriends)
}
// The `keys` here will just be the accumulated list of `personId`s from the `load` call in the resolver
const bestFriendLoader = new DataLoader(keys => batchedBestFriends(keys));
Now, your resolver will look something like:
const PersonType = new GraphQLObjectType({
...
bestFriend: {
type: BestFriendType,
resolve: (person, args, context) => {
return bestFriendLoader.load(person.id);
}
}
});

Using map to reduce in Gun

I am new to Gun. I have existing code that very effectively reduces an array of objects based on a pattern. I am thinking I should tweak this to run in the context of Gun's .map and return undefined for non-matches. I think I will also have to provide two arguments, one of which is the where clause and the other the properties I want shown on returned objects. I also presume that if I use .on future matches will automagically get spit out! Am I on the right path?
const match = (object,key,value) => {
const type = typeof(value);
if(value && type==="object") {
return Object.keys(value).every(childkey =>
match(object[key],childkey,value[childkey]));
if(type==="function") return value(object[key]);
return object[key]===value;
}
const reduce = (objects,where) => {
const keys = Object.keys(where);
return objects.reduce((accumulator,current) => {
if(keys.every(key => match(current,key,where[key]))) {
accumulator.push(current);
}
return accumulator;
},[]);
}
let rows = reduce([{name: "Joe",address:{city: "Seattle"},age:25},
{name: "Mary",address:{city: "Seattle"},age:16},
{name: "Joe",address:{city: "New York"},age:20}],
{name: () => true,
address: {city: "Seattle"},
age: (age) => age > 10});
// results in
[{name: "Joe",address:{city: "Seattle"},age:25},
{name: "Mary",address:{city: "Seattle"},age:16}]
Further exploration of this resulted in the code below, which is stylistically different, but conforms to the immediate responsive nature of Gun. However, it is unclear how to deal with nested objects. The code below only works for primitives.
const match = (object,key,value) => {
const type = typeof(value);
if(!object || typeof(object)!=="object") return false;
if(value && type==="object") {
const child = gun.get(object[key]["#"]);
for(let key in value) {
const value = {};
child.get(key).val(v => value[key] = v,{wait:0});
if(!match(value,key,value[key])) return;
}
}
if(type==="function") return value(object[key]);
return object[key]===value;
}
const gun = Gun(["http://localhost:8080/gun"]),
users = [{name: "Joe",address:{city: "Seattle"},age:25},
{address:{city: "Seattle"},age:25},
{name: "Mary",address:{city: "Seattle"},age:16},
{name: "Joe",address:{city: "New York"},age:20}];
//gun.get("users").map().put(null);
for(let user of users) {
const object = gun.get(user.name).put(user);
gun.get("users").set(object);
}
gun.get("users").map(user => {
const pattern = {name: (value) => value!=null, age: (age) => age > 20}; //, address: {city: "Seattle"}
for(let key in pattern) {
if(!match(user,key,pattern[key])) return;
}
return user;
}).on(data => console.log(data));
Yes. GUN's .map method does more than what it seems.
Say we have var users = gun.get('users'). We can do:
users.map() with no callback acts like a forEach because the default callback is to return the data as-is.
users.map(user => user.age * 2) with a callback, it lets you transform the data like you would expect from a map, except where:
users.map(function(){ return }) if you return undefined, it will filter out that record.
WARNING: As of the current time, .map(transform) function is currently experimental and my have bugs with it. Please try it and report any you find.
Now we can combine it with some other methods, to get some cool behavior:
users.map().on(cb) will get current and future users as they are added to the table, and gets notified for updates on each of those users.
users.map().val(cb) will get current and future users as they are added to the table, but only gets each one once.
users.val().map().on(cb) gets only the current users (not future), but gets the updates to those users.
users.val().map().val(cb) gets only the current users (not future), and only gets them once.
So yes, you are on the right track. For instance, I have a test in gun core that does this:
list.map(user => user.age === 27? user.name + "thezombie" : u).on(function(data){
// verify
});
list.set({name: 'alice', age: 27});
list.set({name: 'bob', age: 27});
list.set({name: 'carl', age: 29});
list.set({name: 'dave', age: 25});
This creates a live map that filters the results and locally (view only) transforms the data.
In the future, this is how the SQL and MongoDB Mango query extensions will work for gun.
Note: GUN only loads the property you request on an object/node, so it is bandwidth efficient. If we do users.map().get('age') it will only load the age value on every user, nothing else.
So internally, you can do some efficient checks, and if all your conditionals match, only /then/ load the entire object. Additionally, there are two other options: (1) you can use an in-memory version of gun to create server-side request-response patterns, so you can have server-side filtering/querying that is efficient. (2) if you become an adapter developer and learn the simple wire spec and then write your own custom query language extensions!
Anything else? Hit me up! More than happy to answer.
Edit: My reply in the comments, comments apparently can't have code. Here is pseudo-code of how to "build up" more complex queries, which will be similar to how SQL/Mango query extensions will work:
mutli-value & nested value matching can be "built up" from this as the base, but yes, you are right, until we have SQL/Mango query examples, there isn't a simple/immediate "out of the box" example. This is pseudo code, but should get the idea across:
```
Gun.chain.match = function(query, cb){
var gun = this;
var fields = Object.keys(query);
var check = {};
fields.forEach(function(field){
check[field] = true;
gun.get(field).val(function(val){
if(val !== query[field]){ return }
check[field] = false;
//all checks done?
cb(results)
});
});
return gun;
}
```
Solution, the trick is to use map and not val:
Gun.chain.match = function(pattern,cb) {
let node = this,
passed = true,
keys = Object.keys(pattern);
keys.every(key => {
const test = pattern[key],
type = typeof(test);
if(test && type==="object") {
node.get(key).match(test);
} else if(type==="function") {
node.get(key).map(value => {
if(test(value[key])) {
return value;
} else {
passed = false;
}
});
} else {
node.get(key).map(value => {
if(value[key]===test) {
return value;
} else {
passed = false;
}
});
}
return passed;
});
if(passed && cb) this.val(value => cb(value))
return this;
}
const gun = new Gun();
gun.get("Joe").put({name:"Joe",address:{city:"Seattle"},age:20});
gun.get("Joe").match({age: value => value > 15,address:{ city: "Seattle"}},value => console.log("cb1",value));

Cannot format or transform data before save, bound too tightly to the view

I have some data in vuejs that I want to format before sending it off through an ajax call but it changes the view its bound to. For example I have a birthday field that is formatted like this on the view 01/11/1981 but I need to format that to YYYY-MM-DD HH:mm:ss for the db and I don't want to do this on the backend.
Where and when would I do this on the frontend? I have tried doing this before the ajax request and it changes the view, so I made a copy of the data and modified it and that also changed the view. It seems no matter what I do it affects the view.
Here is my methods block:
methods: {
/**
* Update the user's contact information.
*/
update() {
/*Attempt to copy and format*/
var formattedForm = this.form;
formattedForm.birthday = moment(formattedForm.birthday).format('YYYY-MM-DD HH:mm:ss');```
Spark.put('/settings/contact', formattedForm)
.then(() => {
Bus.$emit('updateUser');
});
},
}
Here is my data block as well:
data() {
return {
form: $.extend(true, new SparkForm({
gender: '',
height: '',
weight: '',
birthday: '',
age: '',
}), Spark.forms.updateContactInformation),
};
},
The easiest way is to make a clone using Object.assign, like so:
let form = Object.assign({}, this.form);
form.age = 21;
Here's the JSFiddle: https://jsfiddle.net/y51yuf05/
Objects are passed by reference in javascript, which means:
let a = {
"apple": 6
}
let b = a
then, b and a are pointing to the same location in the memory, it is essentially copying the address of the object in a to the variable b.
You need to therefore clone the object, there are many ways to do it like:
b = Object.assign({}, a)
MDN: Object.assign()
this would not be deeply cloned, which means if your object is nested then the nested objects would still be linked between the original and the copy.
for which I use:
function isObject(obj) {
return typeof obj === 'object' && !Array.isArray(obj)
}
function clone(obj) {
let result = {}
for (let key in obj) {
if (isObject(obj[key])) {
result[key] = clone(obj[key])
} else {
result[key] = obj[key]
}
}
return result
}
function logger () {
console.log("p.a.b.c: ", p.a.b.c)
console.log("q.a.b.c:", q.a.b.c)
console.log("r.a.b.c:", r.a.b.c)
}
let p = {a: {b: {c: 5}}}
let q = clone(p)
let r = Object.assign({}, p)
logger()
p.a.b.c = 11
logger()