I want to achieve the following flow logic in Kotlin (Android):
Collectors listen to a List<Data> across several screens of my app.
The source-of-truth is a database, that exposes data and all changes to it as a flow.
On the first initialization the data should be initialized or updated via a remote API
If any API exception occurs, the collectors must be made aware of it
In my first attempt, the flow was of the type Flow<List<Data>>, with the following logic:
val dataFlow = combine(localDataSource.dataFlow, flow {
emit(emptyList()) //do not wait for API on first combination
emit(remoteDataSource.suspendGetDataMightThrow())
}) { (local, remote) ->
remote.takeUnless { it.isEmpty() }?.let { localDataSource.updateIfChanged(it) }
local
}.shareIn(externalScope, SharingStarted.Lazily, 1)
This worked fine, except when suspendGetDataMightThrow() throws an exception. Because shareIn stops propagating the exception through the flow, and instead breaks execution of the externalScope, my collectors are not notified about the exception.
My solution was to wrap the data with a Result<>, resulting of a flow type of Flow<Result<List<Data>>>, and the code:
val dataFlow = combine(localDataSource.dataFlow, flow {
emit(Result.success(emptyList())) //do not wait for API on first combination
emit(runCatching { remoteDataSource.suspendGetDataMightThrow() })
}) { (local, remote) ->
remote.onSuccess {
data -> data.takeUnless { it.isEmpty() }?.let { localDataSource.updateIfChanged(it) }
}
if (remote.isFailure) remote else local
}.shareIn(externalScope, SharingStarted.Lazily, 1)
I can now collect it as follows, and the exception is passed to the collectors:
dataRepository.dataFlow
.map { it.getOrThrow() }
.catch {
// ...
}
.collect {
// ...
}
Is there a less verbose solution to obtain the exception, than to wrap the whole thing in a Result?
I am aware that there are other issues with the code (1 API failure is emitted forever). This is only a proof-of-concept to get the error-handling working.
Could you please explain what exactly happens in Flux/Mono returned by HttpClient.response() ? I thought value generated by http client will NOT be passed downstream until Mono completes but I see that tons of requests are generated which ends up with reactor.netty.internal.shaded.reactor.pool.PoolAcquirePendingLimitException: Pending acquire queue has reached its maximum size of 8 exception. It works as expected (items being processed one by one) if I replace call to testRequest() with Mono.fromCallable { }.
What am I missing ?
Test code:
import org.asynchttpclient.netty.util.ByteBufUtils
import reactor.core.publisher.Flux
import reactor.core.publisher.Mono
import reactor.netty.http.client.HttpClient
import reactor.netty.resources.ConnectionProvider
class Test {
private val client = HttpClient.create(ConnectionProvider.create("meh", 4))
fun main() {
Flux.fromIterable(0..99)
.flatMap { obj ->
println("Creating request for: $obj")
testRequest()
.doOnError { ex ->
println("Failed request for: $obj")
ex.printStackTrace()
}
.map { res ->
obj to res
}
}
.doOnNext { (obj, res) ->
println("Created request for: $obj ${res.length} characters")
}
.collectList().block()!!
}
fun testRequest(): Mono<String> {
return client.get()
.uri("https://projectreactor.io/docs/netty/release/reference/index.html#_connection_pool")
.responseContent()
.reduce(StringBuilder(), { sb, buf ->
val str= ByteBufUtils.byteBuf2String(Charsets.UTF_8, buf)
sb.append(str)
})
.map { it.toString() }
}
}
When you create the ConnectionProvider like this ConnectionProvider.create("meh", 4), this means connection pool with max connections 4 and max pending requests 8. See here more about this.
When you use flatMap this means Transform the elements emitted by this Flux asynchronously into Publishers, then flatten these inner publishers into a single Flux through merging, which allow them to interleave See here more about this.
So what happens is that you are trying to run all requests simultaneously.
So you have two options:
If you want to use flatMap then increase the number of the pending requests.
If you want to keep the number of the pending requests you may consider for example using concatMap instead of flatMap, which means Transform the elements emitted by this Flux asynchronously into Publishers, then flatten these inner publishers into a single Flux, sequentially and preserving order using concatenation. See more here about this.
my question is similar to this one How to throw error from RxJS map operator (angular), but I'm on angular6 with rxjs6 and I guess all is changed ;)
I want to know, how I could propagate an Error-Object within map of an observable to the subscribe OnError Part. I always end up in the OnNext Part.
Here is what I have so far:
Within a ng-component I have maybe the following method call
[...]
this.dataStreamService.execCall({ method : 'order_list',params : {}})
.subscribe( r => {
// here r provides the result data from http call
console.log("execCall result", r);
}, err => {
// HERE the "MAP ERROR OCCURED" Error should be occured as well,
// but in doesn't
console.log("execCall error",err);
});
[...]
The called service method looks like:
execCall(dataStreamCall: DataStreamCall): Observable<DataStreamResult> {
let apiURL = '<some API-URL>';
let params = dataStreamCall.params;
// do HTTP request (this.http calls an extra service handler which wraps
// the angular httpClient and the API errors there
// There is NO Problem with that part :)
let apiResult = this.http.post(apiURL, params);
// Build a new Observable from type "DataStreamResult"
let dsr : Observable<DataStreamResult> = apiResult
.pipe(
map( httpresult => {
if (httpresult['status'] == false){
// the http call was basically successful,
// but state in data is false
// *** THIS IS NOT PROPAGATE TO SUBSCRIBE OnERROR ***
throwError({'msg' : 'MAP ERROR OCCURED'});
// also tried as alternative
return throwError({'msg' : 'MAP ERROR OCCURED'});
} else {
// here the http call was successful
let d = new DataStreamResult();
d.result = httpresult;
return d;
}
}),
catchError( err => {
// error is bubble up from http request handler
return throwError(err);
})
);
return dsr;
}
Finally the Question:
How could manage, that the "throwError" within the piped "map" is propagated to subscribe "err => { ... }".
The actual behavior for:
throwError({..})
I ended up in the subscribe OnNext Part with r = undefined
If I use:
return throwError({..})
I also ended up in the subscribe OnNext Part where r is the throwError-Observable
Thx in Advance
Best Regards
throwError({'msg' : 'MAP ERROR OCCURED'}) will return an observable that, when subscribed to, will effect an error notification. That is, it will call the subscriber's error method.
In your snippet, you either call throwError and ignore the value. Or you return its return value from a project function passed to the map operator.
Neither will effect an error.
There is no subscriber in the first situation, because the return value is ignored. And, in the second situation, there is no subscriber because the map operator doesn't subscribe to what it receives from the project function - the map operator's project function can return anything; it doesn't have to return an observable.
To throw an error within map, use:
throw {'msg' : 'MAP ERROR OCCURED'};
I had a look at the bluebird promise FAQ, in which it mentions that .then(success, fail) is an antipattern. I don't quite understand its explanation as for the try and catch.
What's wrong with the following?
some_promise_call()
.then(function(res) { logger.log(res) }, function(err) { logger.log(err) })
It seems that the example is suggesting the following to be the correct way.
some_promise_call()
.then(function(res) { logger.log(res) })
.catch(function(err) { logger.log(err) })
What's the difference?
What's the difference?
The .then() call will return a promise that will be rejected in case the callback throws an error. This means, when your success logger fails, the error would be passed to the following .catch() callback, but not to the fail callback that goes alongside success.
Here's a control flow diagram:
To express it in synchronous code:
// some_promise_call().then(logger.log, logger.log)
then: {
try {
var results = some_call();
} catch(e) {
logger.log(e);
break then;
} // else
logger.log(results);
}
The second log (which is like the first argument to .then()) will only be executed in the case that no exception happened. The labelled block and the break statement feel a bit odd, this is actually what python has try-except-else for (recommended reading!).
// some_promise_call().then(logger.log).catch(logger.log)
try {
var results = some_call();
logger.log(results);
} catch(e) {
logger.log(e);
}
The catch logger will also handle exceptions from the success logger call.
So much for the difference.
I don't quite understand its explanation as for the try and catch
The argument is that usually, you want to catch errors in every step of the processing and that you shouldn't use it in chains. The expectation is that you only have one final handler which handles all errors - while, when you use the "antipattern", errors in some of the then-callbacks are not handled.
However, this pattern is actually very useful: When you want to handle errors that happened in exactly this step, and you want to do something entirely different when no error happened - i.e. when the error is unrecoverable. Be aware that this is branching your control flow. Of course, this is sometimes desired.
What's wrong with the following?
some_promise_call()
.then(function(res) { logger.log(res) }, function(err) { logger.log(err) })
That you had to repeat your callback. You rather want
some_promise_call()
.catch(function(e) {
return e; // it's OK, we'll just log it
})
.done(function(res) {
logger.log(res);
});
You also might consider using .finally() for this.
The two aren't quite identical. The difference is that the first example won't catch an exception that's thrown in your success handler. So if your method should only ever return resolved promises, as is often the case, you need a trailing catch handler (or yet another then with an empty success parameter). Sure, it may be that your then handler doesn't do anything that might potentially fail, in which case using one 2-parameter then could be fine.
But I believe the point of the text you linked to is that then is mostly useful versus callbacks in its ability to chain a bunch of asynchronous steps, and when you actually do this, the 2-parameter form of then subtly doesn't behave quite as expected, for the above reason. It's particularly counterintuitive when used mid-chain.
As someone who's done a lot of complex async stuff and bumped into corners like this more than I care to admit, I really recommend avoiding this anti-pattern and going with the separate handler approach.
By looking at advantages and disadvantages of both we can make a calculated guess as to which is appropriate for the situation.
These are the two main approaches to implementing promises. Both have it's pluses and minus
Catch Approach
some_promise_call()
.then(function(res) { logger.log(res) })
.catch(function(err) { logger.log(err) })
Advantages
All errors are handled by one catch block.
Even catches any exception in the then block.
Chaining of multiple success callbacks
Disadvantages
In case of chaining it becomes difficult to show different error messages.
Success/Error Approach
some_promise_call()
.then(function success(res) { logger.log(res) },
function error(err) { logger.log(err) })
Advantages
You get fine grained error control.
You can have common error handling function for various categories of errors like db error, 500 error etc.
Disavantages
You will still need another catch if you wish to handler errors thrown by the success callback
Simple explain:
In ES2018
When the catch method is called with argument onRejected, the
following steps are taken:
Let promise be the this value.
Return ? Invoke(promise, "then", « undefined, onRejected »).
that means:
promise.then(f1).catch(f2)
equals
promise.then(f1).then(undefiend, f2)
Using .then().catch() lets you enable Promise Chaining which is required to fulfil a workflow. You may need to read some information from database then you want to pass it to an async API then you want to manipulate the response. You may want to push the response back into the database. Handling all these workflows with your concept is doable but very hard to manage. The better solution will be then().then().then().then().catch() which receives all errors in just once catch and lets you keep the maintainability of the code.
Using then() and catch() helps chain success and failure handler on the promise.catch() works on promise returned by then(). It handles,
If promise was rejected. See #3 in the picture
If error occurred in success handler of then(), between line numbers 4 to 7 below. See #2.a in the picture
(Failure callback on then() does not handle this.)
If error occurred in failure handler of then(), line number 8 below. See #3.b in the picture.
1. let promiseRef: Promise = this. aTimetakingTask (false);
2. promiseRef
3. .then(
4. (result) => {
5. /* successfully, resolved promise.
6. Work on data here */
7. },
8. (error) => console.log(error)
9. )
10. .catch( (e) => {
11. /* successfully, resolved promise.
12. Work on data here */
13. });
Note: Many times, failure handler might not be defined if catch() is
written already.
EDIT: reject() result in invoking catch() only if the error
handler in then() is not defined. Notice #3 in the picture to
the catch(). It is invoked when handler in line# 8 and 9 are not
defined.
It makes sense because promise returned by then() does not have an error if a callback is taking care of it.
Instead of words, good example. Following code (if first promise resolved):
Promise.resolve()
.then
(
() => { throw new Error('Error occurs'); },
err => console.log('This error is caught:', err)
);
is identical to:
Promise.resolve()
.catch
(
err => console.log('This error is caught:', err)
)
.then
(
() => { throw new Error('Error occurs'); }
)
But with rejected first promise, this is not identical:
Promise.reject()
.then
(
() => { throw new Error('Error occurs'); },
err => console.log('This error is caught:', err)
);
Promise.reject()
.catch
(
err => console.log('This error is caught:', err)
)
.then
(
() => { throw new Error('Error occurs'); }
)
I've defined some API calls in Futures that make API calls to Mashery and Stripe
val stripeFuture = Future { // api call }
val masheryFuture = Future { //api call }
For the stripeFuture -The main logic is to set the stripeCustomerId on a Client object within the onSuccess block
stripeFuture onSuccess {
//client.stripeCustomerId
}
I've wrapped up the API calls in a for-comprehension similar to the example in Futures and Promises
val apiCalls = for {
masheryInfo <- masheryFuture
stripeCustomer <- stripeFuture
}
There is a rollback if one of the API calls fail
apiCalls onFailure {
case pse: MasheryException => {
// delete stripe customer id
}
case e: StripeException => {
//delete mashery api key
}
The problem is when the call to Mashery fails 'masheryFuture', I want to rollback 'get the stripe id' from the Client object but there is a around a 1 second delay til that call finishes and it doesn't set the stripeCustomerId until it hits the onSuccess block so within the ase pse: MasheryException => { } block, client.getstripeCustomerId returns null.
Is there a way of getting around this race condition for both of the API calls
Use Future.andThen.
The doc:
Applies the side-effecting function to the result of this future, and
returns a new future with the result of this future.
This method allows one to enforce that the callbacks are executed in a
specified order.
for (f <- Future(x).andThen { y }) etc.
Update:
for (f <- Future(x) andThen {
case Success(x) => use(x)
case _ => // ignore
}) yield result