Controller calls the service method which in turn has blocking call to DB. But we are using subscribeOn to schedule the subscription on another thread. The question i have is, does this approach work without blocking other event loop threads?
//Controller
#GetMapping("configuration")
#ResponseStatus(HttpStatus.OK)
public Mono<Command> getConfiguration(Principal agent) {
Mono<String> mono = service.getConfiguration(agent.getName());
return mono.flatMap(cmd -> {
return Mono.just(Command.toCmd(cmd));
}).onErrorMap(throwable -> new ResponseStatusException(HttpStatus.INTERNAL_SERVER_ERROR,
"Error getting config", throwable));
}
//Service method
public Mono<String> getConfiguration(String name) {
return Mono.fromCallable(() -> {
return dbService.getConfigCommandFromServer(name);
}).subscribeOn(Schedulers.boundedElastic());
}
~~~
Related
I have a web app (Spring Cloud Gateway with Project Reactor) where I have to logout (send another http request) when something goes wrong and set 401 to the main response. The problem is when I execute another request in onErrorResume block, the root response seems to ignore finishWithStatus() logic entirely and returns 200.
#Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
return someFunctionWhichReturnsMono()
.flatMap(chain::filter)
.onErrorResume(e -> {
log.error("Unexpected Filter Error, logging out user", e);
// A. this doesn't set 401, seems like ignoring finishWithStatus(..)
// called inside this method in onErrorResume block
return logout(exchange);
// B. this works fine and I get 401 as a response
// return finishWithStatus(exchange, HttpStatus.UNAUTHORIZED);
});
}
protected Mono<Void> finishWithStatus(ServerWebExchange exchange, HttpStatus status) {
exchange.getResponse().setStatusCode(status);
return exchange.getResponse().setComplete();
}
protected void logout(ServerWebExchange exchange) {
webClient
.post()
.uri(....)
.retrieve()
.bodyToMono(Void.class)
.doOnSuccess(any -> {
log.info("Successfully logged out user");
})
.then(finishWithStatus(exchange, HttpStatus.UNAUTHORIZED))
.onErrorResume(e -> {
log.error("Failed to logout user", e);
//the following line has no effect when error happens
return finishWithStatus(exchange, HttpStatus.UNAUTHORIZED);
});
}
Could somebody explain why is that despite I return Mono in both cases. However, in case A I have nested onErrorResume (at onErrorResume of the "root" Mono I create another Mono with its own onErrorResume).
I feel I miss something fundamental like I need to "join" two Monos or say bubble up some Mono.error from the deepest onErrorResume to the top one?
What would be a generic approach to handle nested errors (like the case above when on an error you have to send another request which in turn might end up with an error).
I would greatly appreciate any advice or sample on this matter.
Fundamental rule in reactive programming: Nothing happens until you subscribe
finishWithStatus() works because you are passing the publisher value to the next filter by returning Mono<Void> type.
You are making chain of reactive operators in logout() method but never subscribing or passing the publisher to the next filter. Therefore nothing happens even if you call this method
Another point is you can't return logout() method in filter() method since the return type for filter() is Mono<Void> not void.
You should change the return type for logout() method:
protected Mono<Void> logout(ServerWebExchange exchange) {
return webClient
.post()
.uri(...)
.retrieve()
.bodyToMono(Void.class)
.doOnSuccess(any -> {
log.info("Successfully logged out user");
})
.then(finishWithStatus(exchange, HttpStatus.UNAUTHORIZED))
.onErrorResume(e -> {
log.error("Failed to logout user", e);
return finishWithStatus(exchange, HttpStatus.UNAUTHORIZED);
});
}
I have this flow that I am trying to test but nothing works as expected. The flow itself works well but testing seems a bit tricky.
This is my flow:
#Configuration
#RequiredArgsConstructor
public class FileInboundFlow {
private final ThreadPoolTaskExecutor threadPoolTaskExecutor;
private String filePath;
#Bean
public IntegrationFlow fileReaderFlow() {
return IntegrationFlows.from(Files.inboundAdapter(new File(this.filePath))
.filterFunction(...)
.preventDuplicates(false),
endpointConfigurer -> endpointConfigurer.poller(
Pollers.fixedDelay(500)
.taskExecutor(this.threadPoolTaskExecutor)
.maxMessagesPerPoll(15)))
.transform(new UnZipTransformer())
.enrichHeaders(this::headersEnricher)
.transform(Message.class, this::modifyMessagePayload)
.route(Map.class, this::channelsRouter)
.get();
}
private String channelsRouter(Map<String, File> payload) {
boolean isZip = payload.values()
.stream()
.anyMatch(file -> isZipFile(file));
return isZip ? ZIP_CHANNEL : XML_CHANNEL; // ZIP_CHANNEL and XML_CHANNEL are PublishSubscribeChannel
}
#Bean
public SubscribableChannel xmlChannel() {
var channel = new PublishSubscribeChannel(this.threadPoolTaskExecutor);
channel.setBeanName(XML_CHANNEL);
return channel;
}
#Bean
public SubscribableChannel zipChannel() {
var channel = new PublishSubscribeChannel(this.threadPoolTaskExecutor);
channel.setBeanName(ZIP_CHANNEL);
return channel;
}
//There is a #ServiceActivator on each channel
#ServiceActivator(inputChannel = XML_CHANNEL)
public void handleXml(Message<Map<String, File>> message) {
...
}
#ServiceActivator(inputChannel = ZIP_CHANNEL)
public void handleZip(Message<Map<String, File>> message) {
...
}
//Plus an #Transformer on the XML_CHANNEL
#Transformer(inputChannel = XML_CHANNEL, outputChannel = BUS_CHANNEL)
private List<BusData> xmlFileToIngestionMessagePayload(Map<String, File> xmlFilesByName) {
return xmlFilesByName.values()
.stream()
.map(...)
.collect(Collectors.toList());
}
}
I would like to test multiple cases, the first one is checking the message payload published on each channel after the end of fileReaderFlow.
So I defined this test classe:
#SpringBootTest
#SpringIntegrationTest
#ExtendWith(SpringExtension.class)
class FileInboundFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#TempDir
static Path localWorkDir;
#BeforeEach
void setUp() {
copyFileToTheFlowDir(); // here I copy a file to trigger the flow
}
#Test
void checkXmlChannelPayloadTest() throws InterruptedException {
Thread.sleep(1000); //waiting for the flow execution
PublishSubscribeChannel xmlChannel = this.getBean(XML_CHANNEL, PublishSubscribeChannel.class); // I extract the channel to listen to the message sent to it.
xmlChannel.subscribe(message -> {
assertThat(message.getPayload()).isInstanceOf(Map.class); // This is never executed
});
}
}
As expected that test does not work because the assertThat(message.getPayload()).isInstanceOf(Map.class); is never executed.
After reading the documentation I didn't find any hint to help me solved that issue. Any help would be appreciated! Thanks a lot
First of all that channel.setBeanName(XML_CHANNEL); does not effect the target bean. You do this on the bean creation phase and dependency injection container knows nothing about this setting: it just does not consult with it. If you really would like to dictate an XML_CHANNEL for bean name, you'd better look into the #Bean(name) attribute.
The problem in the test that you are missing the fact of async logic of the flow. That Files.inboundAdapter() works if fully different thread and emits messages outside of your test method. So, even if you could subscribe to the channel in time, before any message is emitted to it, that doesn't mean your test will work correctly: the assertThat() will be performed on a different thread. Therefore no real JUnit report for your test method context.
So, what I'd suggest to do is:
Have Files.inboundAdapter() stopped in the beginning of the test before any setup you'd like to do in the test. Or at least don't place files into that filePath, so the channel adapter doesn't emit messages.
Take the channel from the application context and if you wish subscribe or use a ChannelInterceptor.
Have an async barrier, e.g. CountDownLatch to pass to that subscriber.
Start the channel adapter or put file into the dir for scanning.
Wait for the async barrier before verifying some value or state.
i need help to understand why the error message is not logged when subscribing to the stream with .subscribe(). However, the error is logged when subscribing with .block(). I understand that it might be to the block() call subscribing on the main thread. I was wondering which changes should i do in order to see the error logged when subscribing with .subscribe(). Also not sure whether it is the best way to deal listen for sqs messages on webflux.
Thanks,
#SqsListener(value = "${sqs.queues.email-notifications}", deletionPolicy = NEVER)
public void listenNotifications(MessageDto message, Acknowledgment acknowledgment) {
Mono.just(message)
.log("NotificationsSQSConsumer.listenNotifications")
.map(this::toSendEmailCommand)
.flatMap(sendEmailUseCase::handle)
.then(sendAck(acknowledgment))
.log("NotificationsSQSConsumer.sendAck")
//.block(); //.subscribe() nop**
}
private Mono<?> sendAck(Acknowledgment acknowledgment) {
return Mono.fromCallable(() -> acknowledgment.acknowledge().get())
.doOnError(throwable -> log.error("ERROR!:" + throwable.getMessage()))
.subscribeOn(Schedulers.boundedElastic());
}
class SendEmailUseCase {
public Mono<Void> handle(SendEMailCommand sendEMailCommand) {
return Mono.just(sendEMailCommand.getTemplateId())
.flatMap(templateLoader::getTemplate) //do a Mono.fromCallable on a bounded elastic
.map(s -> Tuples.of(new StringReader(s), sendEMailCommand.getTemplateId()))
.map(this::newMustache)
.map(c -> Tuples.of(c, sendEMailCommand.getTemplateData()))
.map(this::build)
.map(mailText -> Tuples.of(sendEMailCommand.getReceiver(), sendEMailCommand.getSubject(), mailText))
.map(this::newEmailMessage)
.flatMap(emailSender::sendMail); //do a Mono.fromCallable on bounded elastic
}
}
//Json
{
"dd.span_id":"0",
"dd.trace_id":"0",
"timestamp":"2021-03-04T15:53:00.588 03:00",
"event_type":"logging",
"level":"ERROR",
"thread_name":"boundedElastic-1",
"logger_name":"c.n.c.m.i.m.NotificationsSQSConsumer",
"message":"ERROR!:com.amazonaws.services.sqs.model.AmazonSQSException: Value AQEB/60/Nzg2oC9tdLlZT4BiztR8UAgd503EZ/bHQ7bE6WuKRfF59Y5l/2+gmqMZtSJs8sug6qUNuUt7pmXVM8G2S3TVt9yVc05t
}
I realized that i was missing the error callback on .subscribe()
#Async
#SqsListener(value = "${sqs.queues.email-notifications}", deletionPolicy = NEVER)
public void handle(MessageDto message, Acknowledgment acknowledgment) {
Mono.just(message)
.log("NotificationsSQSConsumer.listenNotifications")
.map(this::toSendEmailCommand)
.flatMap(sendEmailUseCase::handle)
.then(sendAck(acknowledgment))
.log("NotificationsSQSConsumer.sendAck")
.subscribe(o -> {
}, t -> log.error("ERROR!:" + t.getMessage()));
}
private Mono<?> sendAck(Acknowledgment acknowledgment) {
return Mono.fromCallable(() -> acknowledgment.acknowledge().get())
.subscribeOn(Schedulers.boundedElastic());
}
When I lock on a thread on the ThreadPool like this the thread is blocked:
private static object _testServerLock = new object();
private static TestServer _testServer = null;
public TestServer GetServer()
{
lock (_testServerLock)
{
if (_testServer == null)
{
_testServer = new TestServer(); // does some async stuff internally
}
}
return _testServer;
}
If I have too more concurrent threads calling this than I have threads in the ThreadPool all of them will end up waiting for the lock, while async code happening elsewhere can't continue since it is waiting for a free thread in the ThreadPool.
So I don't want to block the thread, I need to return it to the ThreadPool while I am waiting.
Is there some other way to lock which returns the waiting thread to the ThreadPool?
Whatever has to be done inside a lock should be moved into a Task, which is started before the tests and finishes, when it has created its resource.
Whenever a test wants to get the resource created by the task, it can block with an await on the creator-task before accessing the resource. So all accesses to the resource are in tasks and can't block all threads of the pool.
Something like:
private static object _testServerLock = new object();
private static TestServer _testServer = null;
private static Task _testTask = null;
private async Task<TestServer> CreateTestServerAsync()
{
...
}
// Constructor of the fixture
public TestFixture()
{
// The lock here may be ok, because it's before all the async stuff
// and it doesn't wait for something inside
lock (_testServerLock)
{
if (_testTask == null)
{
_testTask = Task.Run(async () => {
// it's better to expose the async nature of the call
_testServer = await CreateTestServerAsync();
});
// or just, whatever works
//_testTask = Task.Run(() => {
// _testServer = new TestServer();
//});
}
}
}
public async Task<TestServer> GetServerAsync()
{
await _testTask;
return _testServer;
}
Update:
You can remove the lock using the initialization of the static member.
private static TestServer _testServer = null;
private static Task _testTask = Task.Run(async () => {
_testServer = await CreateTestServerAsync();
});
private static async Task<TestServer> CreateTestServerAsync()
{
...
}
public TestFixture()
{
}
public async Task<TestServer> GetServerAsync()
{
await _testTask;
return _testServer;
}
With xUnit ~1.7+, the main thing you can do is make your Test Method return Task<T> and then use async/await which will limit your hard-blocking/occupation of threads
xUnit 2.0 + has parallel execution and a mechanism for controlling access to state to be shared among tests. Note however that this fundamentally operates by running one tests in the Test Class at a time and giving the Class Fixture to one at a time (which is equivalent to what normally happens - only one Test Method per class runs at a time). (If you use a Collection Fixture, effectively all the Test Classes in the collection become a single Test Class).
Finally, xUnit 2 offers switches for controlling whether or not:
Assemblies run in parallel with other [Assemblies]
Test Collections/Test Classes run in parallel with others
Both of the prev
You should be able to manage your issue by not hiding the asyncness as you've done and instead either exposing it to the Test Method or by doing build up/teardown via IAsyncLifetime
If you call a web service from Silverlight like this:
MyServiceClient serviceClient = new MyServiceClient();
void MyMethod()
{
serviceClient.GetDataCompleted += new EventHandler<GetDataCompletedEventArgs>(serviceClient_GetDataCompleted);
serviceClient.GetDataAsync();
// HOW DO I WAIT/JOIN HERE ON THE ASYNC CALL, RATHER THAN BEING FORCE TO LEAVE THIS METHOD?
}
I would rather wait/join with the asych service thread inside "MyMethod" rather than leaving "MyMethod" after calling "GetDataAsync", what is the best way to do this?
Thanks,
Jeff
No you cannot do this way. You will end up in a deadlock. GetDataCompleted is called by the mainthreed. The same threed thait is waiting in WaitOne.
I have to ask; why? The point is to provide your user with a fluid experience and waiting on a web service call will not necessarily do that. I suppose you want the full block of content to load before the Silverlight control loads. In that case, I would turn to caching the content rather than forcing the client to wait indefinitely.
To do this you would use a ManualResetEvent in your class (class level variable) and then wait on it.
void MyMethod()
{
wait = new ManualResetEvent(false);
// call your service
wait.WaitOne();
// finish working
}
and in your event handler code
void serviceClient_GetDataCompleted(...)
{
// Set values you need from service
wait.Set();
}
You could also use a lambda and closure to get similar behavior:
serviceClient.GetDataCompleted += (s,e) =>
{
// Your code here
};
serviceClient.GetDataAsync();
If you had a base class provide the mechanics of building a WCF channel, it could then be used to build the BeginX / EndX methods for a async call.
public class ServiceFooCoordinator : CoordinatorBase<IServiceFoo>
{
public IAsyncResult BeginMethodFoo ()
{
IAsyncResult ar = null;
IServiceFoo channel = null;
channel = _factory.GetChannel();
Begin( channel, () => ar = channel.BeginMethodFoo( null, channel ) );
return ar;
}
public Bar[] EndMethodFoo ( IAsyncResult ar )
{
IServiceFoo channel = null;
channel = _factory.GetChannel();
return channel.EndMethodFoo( ar );
}
}
Which can then be used in a method:
ServiceFooCoordinator _coordinator;
var asyncResult = _coordinator.BeginMethodFoo();
try
{
var result = _coordinator.EndMethodFoo( asyncResult );
}
catch ( Exception )
{ }
Which gets you your asynchronous call in a sychronous manner.