I build the reactive pipeline like this on netty server with spring web-flux:
someService.getSettings(key) //Mono<Settings>
.filter(Settings::isEnabled)
.switchIfEmpty(Mono.error(
new Exception("Setting is disabled.")
)).then(/*Mono representing API fetch*/ someApi.fetch(param));
As per my understanding, whenever the Settings::isEnabled would return false, the pipeline would be empty and I will get a Mono representing error. However, this behavior does not happens. My someApi.fetch(param) call always fires without the filter Mono being completed. So even if Settings::isEnabled is false or true, someApi.fetch(param) is always fired without completion of Mono.filter.
How do I make sure that error is thrown if filter returns empty Mono and the fetch call happens only when I have checked that the setting is not disabled?
this is still java: this call to someApi.fetch(param) is not done inside a lambda, so it can't be lazy. the moment the execution reaches the then line, your fetch method is called. the Flux.defer operator is tailor-made to wrap that call and make it lazy.
Related
I have a simple Kafka streams scenario where I am doing a groupyByKey then reduce and then an action. There could be duplicate events in the source topic hence the groupyByKey and reduce
The action could error and in that case, I need the streams app to reprocess that event. In the example below I'm always throwing an error to demonstrate the point.
It is very important that the action only ever happens once and at least once.
The problem I'm finding is that when the streams app reprocesses the event, the reduce function is being called and as it returns null the action doesn't get recalled.
As only one event is produced to the source topic TOPIC_NAME I would expect the reduce to not have any values and skip down to the mapValues.
val topologyBuilder = StreamsBuilder()
topologyBuilder.stream(
TOPIC_NAME,
Consumed.with(Serdes.String(), EventSerde())
)
.groupByKey(Grouped.with(Serdes.String(), EventSerde()))
.reduce { current, _ ->
println("reduce hit")
null
}
.mapValues { _, v ->
println(Id: "${v.correlationId}")
throw Exception("simulate error")
}
To cause the issue I run the streams app twice. This is the output:
First run
Id: 90e6aefb-8763-4861-8d82-1304a6b5654e
11:10:52.320 [test-app-dcea4eb1-a58f-4a30-905f-46dad446b31e-StreamThread-1] ERROR org.apache.kafka.streams.KafkaStreams - stream-client [test-app-dcea4eb1-a58f-4a30-905f-46dad446b31e] All stream threads have died. The instance will be in error state and should be closed.
Second run
reduce hit
As you can see the .mapValues doesn't get called on the second run even though it errored on the first run causing the streams app to reprocess the same event again.
Is it possible to be able to have a streams app re-process an event with a reduced step where it's treating the event like it's never seen before? - Or is there a better approach to how I'm doing this?
I was missing a property setting for the streams app.
props["processing.guarantee"]= "exactly_once"
By setting this, it will guarantee that any state created from the point of picking up the event will rollback in case of a exception being thrown and the streams app crashing.
The problem was that the streams app would pick up the event again to re-process but the reducer step had state which has persisted. By enabling the exactly_once setting it ensures that the reducer state is also rolled back.
It now successfully re-processes the event as if it had never seen it before
I have a common rest controller:
private final KafkaReceiver<String, Domain> receiver;
#GetMapping(produces = MediaType.APPLICATION_STREAM_JSON_VALUE)
public Flux<Domain> produceFluxMessages() {
return receiver.receive().map(ConsumerRecord::value)
.timeout(Duration.ofSeconds(2));
}
What I am trying to achieve is to collect messages from Kafka topic for a certain period of time, and then just stop consuming and consider this flux completed. If I remove timeout and open this in a browser, I am getting messages forever, downloading never stops. And with this timeout consuming stops after 2 seconds, but I'm getting an exception:
java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 2000ms in 'map' (and no fallback has been configured)
Is there a way to successfully complete Flux after timeout?
There's multiple overloads of the timeout() method - you're using the standard one that throws an exception on timeout.
Instead, just use the overloaded timeout method to provide an empty default publisher to fallback to:
timeout(Duration.ofSeconds(2), Mono.empty())
(Note in a general case you could explicitly capture the TimeoutException and fallback to an empty publisher using onErrorResume(TimeoutException.class, e -> Mono.empty()), but that's much less preferable to using the above option where possible.)
It may happen that data that enters Flink job triggers exception either due to bug in code or lack of validation.
My goal is to provide consistent way of exception handling that our team could use within Flink jobs that won't cause any downtime in production.
Restart strategies do not seem to be applicable here as:
simple restart won't fix issue and we fall into restart loop
we cannot simply skip event
they can be good for OOME or some transient issues
we cannot add custom one
try/catch block in "keyBy" function does not fully help as:
there's no way to skip event in "keyBy" after exception is handled
Sample code:
env.addSource(kafkaConsumer)
.keyBy(keySelector) // must return one result for one entry
.flatMap(mapFunction) // we can skip some entries here in case of errors
.addSink(new PrintSinkFunction<>());
env.execute("Flink Application");
I'd like to have ability to skip processing of event that caused issue in "keyBy" and similar methods that are supposed to return exactly one result.
Beside the suggestion of #phanhuy152 (which seems totally legit to me) why not filter before keyBy?
env.addSource(kafkaConsumer)
.filter(invalidKeys)
.keyBy(keySelector) // must return one result for one entry
.flatMap(mapFunction) // we can skip some entries here in case of errors
.addSink(new PrintSinkFunction<>());
env.execute("Flink Application");
Can you reserve a special value like "NULL" for the keyBy to return in such case? Then your flatMap function can skip when encounter such value?
So I have a workflow which is supposed to throw an error after a certain condition is satisfied. (False condition) As you can see in the log directly below, it works: I do a loop exit first for the group 'coms' and an error is thrown. However, Flowgear seems to only read the last executed node and then determine the workflows status from that. Since the loop finishes last and is successful, if you look in the second log, you can see that the workflow has been evaluated as 'successful' although an error was thrown inside.
Any ideas how to make the loop break? Also why does flowgear only consider the last node? There should be an option in the error node to stop all execution.
Iterator nodes (Splitter and Loop) will consume the errors. The only way at this stage to get the workflow to return an error is to cause an error in the AnyError or UnhandledError part of the workflow. I've created a workflow to demonstrate this here: http://flowgear.me/s/UdpGBbd
Hope this helps.
I am facing the following problem:
I have multiple HTTP Requests in my testplan.
I want every request to be repeated 4 times if they fail.
I realized that with a BeanShell Assertion, and its already working fine.
My problem is, that I don't want requests to be executed if a previous Request failed 5 times,
BUT I also dont want the thread to end.
I just want the current thread iteration to end,
so that the next iteration of the thread can start again with the 1st request (if the thread is meant to be repeated).
How do I realize that within the BeanShell Assertion?
Here is just a short extract of my code where i want the solution to have
badResponseCounter is being increased for every failed try of the request, this seems to work so far. Afterwards, the variable gets resetted.
if (badResponseCounter = 5) {
badResponseCounter = 0;
// Stop current iteration
}
I already checked the API, methods like setStopTest() or setStopThread() are given, but nothing for quitting the current iteration. I also need the preference "continue" in the thread group, as otherwise the entire test will stop after 1 single request failed.
Any ideas of how to do this?
In my opinion the easiest way is using following combination:
If Controller to check ${JMeterThread.last_sample_ok} and badResponseCounter variables
Test Action Sampler as a child of If Controller configured to "Go to next loop iteration"
Try this.
ctx.setRestartNextLoop(true);
if the thread number is 2, i tried to skip. I get the below result as i expected (it does not call b-2). It does not kill the thread either.