Is Spring webclient non-blocking client? - spring-webflux

I don't understand reactive webclient works. It says that spring webclient is non-blocking client, but this webclient seems waiting signal onComplete() from remote api, then it can process each item that emitted from the remote api.
I'm expecting that webclient can process each item when onNext() is fired from the target api
I'm new in the spring webflux worlds. I read about it and it says it uses netty as default server. And this netty using eventloop. So to understand how it works I try to create 2 small apps, client and server.
Server app only return simple flux with delay 1 second each item.
Client app using webclient to call remote api.
Server:
#GetMapping(ITEM_END_POINT_V1)
public Flux<Item> getAllItems(){
return Flux.just(new Item(null, "Samsung TV", 399.99),
new Item(null, "LG TV", 329.99),
new Item(null, "Apple Watch", 349.99),
new Item("ABC", "Beats HeadPhones",
149.99)).delayElements(Duration.ofSeconds(1)).log("Item : ");
}
Client:
WebClient webClient = WebClient.create("http://localhost:8080");
#GetMapping("/client/retrieve")
public Flux<Item> getAllItemsUsingRetrieve() {
return webClient.get().uri("/v1/items")
.retrieve()
.bodyToFlux(Item.class).log();
}
Log from server:
2019-05-01 22:44:20.121 INFO 19644 --- [ctor-http-nio-2] Item : : onSubscribe(FluxConcatMap.ConcatMapImmediate)
2019-05-01 22:44:20.122 INFO 19644 --- [ctor-http-nio-2] Item : : request(unbounded)
2019-05-01 22:44:21.126 INFO 19644 --- [ parallel-1] Item : : onNext(Item(id=null, description=Samsung TV, price=399.99))
2019-05-01 22:44:22.129 INFO 19644 --- [ parallel-2] Item : : onNext(Item(id=null, description=LG TV, price=329.99))
2019-05-01 22:44:23.130 INFO 19644 --- [ parallel-3] Item : : onNext(Item(id=null, description=Apple Watch, price=349.99))
2019-05-01 22:44:24.131 INFO 19644 --- [ parallel-4] Item : : onNext(Item(id=ABC, description=Beats HeadPhones, price=149.99))
2019-05-01 22:44:24.132 INFO 19644 --- [ parallel-4] Item : : onComplete()
Log from client:
2019-05-01 22:44:19.934 INFO 24164 --- [ctor-http-nio-2] reactor.Flux.MonoFlatMapMany.1 : onSubscribe(MonoFlatMapMany.FlatMapManyMain)
2019-05-01 22:44:19.936 INFO 24164 --- [ctor-http-nio-2] reactor.Flux.MonoFlatMapMany.1 : request(unbounded)
2019-05-01 22:44:19.940 TRACE 24164 --- [ctor-http-nio-2] o.s.w.r.f.client.ExchangeFunctions : [7e73de5c] HTTP GET http://localhost:8080/v1/items, headers={}
2019-05-01 22:44:24.159 TRACE 24164 --- [ctor-http-nio-6] o.s.w.r.f.client.ExchangeFunctions : [7e73de5c] Response 200 OK, headers={masked}
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=null, description=Samsung TV, price=399.99))
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=null, description=LG TV, price=329.99))
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=null, description=Apple Watch, price=349.99))
2019-05-01 22:44:24.204 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onNext(Item(id=ABC, description=Beats HeadPhones, price=149.99))
2019-05-01 22:44:24.205 INFO 24164 --- [ctor-http-nio-6] reactor.Flux.MonoFlatMapMany.1 : onComplete()
I'm expecting that client won't wait for 4 seconds then get the actual result.
As you can see that server start emit onNext() on 22:44:21.126, and client get result on 22:44:24.159.
So I don't understand why webclient is called non-blocking client if it has this behaviour.

The WebClient is non-blocking in a sense that the thread sending HTTP requests through the WebClient is not blocked by the IO operation.
When the response is available, netty will notify one of the worker threads and it will process the response according to the reactive stream operations that you defined.
In your example the server will wait until all the elements in a Flux are available (4 seconds), serialize them to the JSON array, and send it back in a single HTTP response.
The client waits for this single response, but non of its threads are blocked during this period.
If you want to achieve the streaming effect, you need to leverage different content-type or the underlying protocol like WebSockets.
Check-out the following SO thread about the application/stream+json content-type:
Spring WebFlux Flux behavior with non streaming application/json

Related

Sleuth 3.0.1 + Spring Cloud Gateway = traceids do not correlate on request/response

Well I have a new setup with Sleuth 3.0.1 where by default spring.sleuth.reactor.instrumentation-type is set to manual
Therefore I wrapped all my places where I need to trace with WebFluxSleuthOperators.withSpanInScope(tracer, currentTraceContext, exchange, () -> {}) logic and it generally puts span in scope but the traceid is always different before and after the response is received:
INFO [gateway,2b601780f2242aa9,2b601780f2242aa9] 9 --- [or-http-epoll-1] ...LogFilter : Request - Username: ..., method: GET, uri: ...
INFO [gateway,,] 9 --- [or-http-epoll-1] ...ValidationFilter : ... Validation passed
INFO [gateway,6a396715ff0979a6,6a396715ff0979a6] 9 --- [or-http-epoll-4] ...LogFilter : Request - Username: ..., method: GET, uri: ...
INFO [gateway,,] 9 --- [or-http-epoll-4] ...ValidationFilter : ... Validation passed
INFO [gateway,020c11ab0c37f8ae,020c11ab0c37f8ae] 9 --- [or-http-epoll-2] ...LogFilter : Request - Username: ..., method: GET, uri: ...
INFO [gateway,,] 9 --- [or-http-epoll-2] ...ValidationFilter : Validation passed
INFO [gateway,189a49553566f771,f51aaaf68e5738d0] 9 --- [or-http-epoll-1] ...LogFilter : Response - Status: 200 OK
INFO [gateway,92f25dcff9833079,87db5f7b1cbefedb] 9 --- [or-http-epoll-2] ...LogFilter : Response - Status: 200 OK
INFO [gateway,d8b133f2200e7808,1743df4b5ad37d07] 9 --- [or-http-epoll-1] ...LogFilter : Response - Status: 400 BAD_REQUEST
Digging deeper it seems the span returned on exchange.getAttribute(Span.class.getName()); has different trace ID in post routing phase.
Of course changing instrumentation-type to decorate_on_last fixes the issue but would like to avoid unnecessary performance degradations.

Spring webflux testing

I have a Spring webflux application with the service below.
#Configuration
public class RouterConfiguration {
#Autowired
private RegistrationHandler regHandler;
#Bean
public RouterFunction<ServerResponse> regRoute() {
return RouterFunctions.route(POST(REGISTER_PATH).and(accept(APPLICATION_JSON)), regHandler::register);
}
}
public Mono<ServerResponse> register(ServerRequest request) {
return request.bodyToMono(Register.class).flatMap(reg -> {
return buildOrg(reg).flatMap(orgRepository::save).flatMap(org ->
ServerResponse.created(URI.create(USER_RESOURCE_URI + p.getUserId())).build());
});
}
I would like to test the above API and my test example,
#ExtendWith({ SpringExtension.class, RestDocumentationExtension.class })
#WebFluxTest({ RegistrationHandler.class })
#AutoConfigureWebTestClient(timeout = "100000")
class RegistrationHandlerTest {
#Autowired
ApplicationContext context;
#MockBean
PasswordEncoder passwordEncoder;
#MockBean
private OrgRepository orgRepository;
#MockBean
private UserRepository usrRepository;
#Captor
private ArgumentCaptor<Organization> orgInputCaptor;
#Captor
private ArgumentCaptor<Mono<Organization>> orgMonoInputCaptor;
#Captor
private ArgumentCaptor<User> usrInputCaptor;
private WebTestClient webTestClient;
#BeforeEach
void setUp(RestDocumentationContextProvider restDocumentation) {
webTestClient = WebTestClient.bindToApplicationContext(context).configureClient()
.filter(documentationConfiguration(restDocumentation)).responseTimeout(Duration.ofMillis(100000))
.build();
}
#Test
#WithMockUser(username = "admin", roles = { "USER", "ADMIN" })
public void testRegister() {
final Register register = new Register();
register.setOrgName("testOrg");
register.setUsername("testUser");
register.setPassword("testPwd");
register.setCountry(1);
final Organization org = new Organization();
final User usr = new User();
given(orgRepository.save(orgInputCaptor.capture())).willReturn(Mono.just(org));
given(usrRepository.save(usrInputCaptor.capture())).willReturn(Mono.just(usr));
webTestClient.mutateWith(csrf()).post().uri(REGISTER_PATH).contentType(APPLICATION_JSON).bodyValue(register)
.exchange().expectStatus().is2xxSuccessful().expectBody();
StepVerifier.create(orgMonoInputCaptor.getValue()).expectNext(org).expectComplete();
then(usrRepository).should().save(usrInputCaptor.capture());
}
}
The test case is failing for 404 NOT_FOUND error.
2021-02-02 08:06:37.488 INFO [AdminService,,] 28571 --- [ Test worker] s.a.s.h.RegistrationRequestHandlerTest : Starting RegistrationHandlerTest using Java 15.0.2 on
2021-02-02 08:06:37.489 INFO [AdminService,,] 28571 --- [ Test worker] s.a.s.h.RegistrationHandlerTest : No active profile set, falling back to default profiles: default
2021-02-02 08:06:38.218 INFO [AdminService,,] 28571 --- [ Test worker] c.test.service.admin.AdminApplication : Starting up Admin Application
2021-02-02 08:06:38.458 INFO [AdminService,,] 28571 --- [ Test worker] ctiveUserDetailsServiceAutoConfiguration :
Using generated security password: fcd6963d-cf13-4eb8-a705-d46755493538
2021-02-02 08:06:38.861 INFO [AdminService,,] 28571 --- [ Test worker] s.a.s.h.RegistrationRequestHandlerTest : Started RegistrationHandlerTest in 1.951 seconds (JVM running for 3.765)
RegistrationHandlerTest > testRegister() STANDARD_OUT
2021-02-02 08:06:39.188 ERROR [AdminService,,] 28571 --- [ Test worker] o.s.t.w.reactive.server.ExchangeResult : Request details for assertion failure:
> POST http://localhost:8080/register
> WebTestClient-Request-Id: [1]
> Content-Type: [application/json]
> Content-Length: [89]
{"orgName":"testOrg","username":"testUser","password":"testPwd","country":1,"state":null}
< 404 NOT_FOUND Not Found
< Content-Type: [application/json]
< Content-Length: [135]
< Cache-Control: [no-cache, no-store, max-age=0, must-revalidate]
< Pragma: [no-cache]
< Expires: [0]
< X-Content-Type-Options: [nosniff]
< X-Frame-Options: [DENY]
< X-XSS-Protection: [1 ; mode=block]
< Referrer-Policy: [no-referrer]
{"timestamp":"2021-02-02T16:06:39.150+00:00","path":"/register","status":404,"error":"Not Found","message":null,"requestId":"21d38b1c"}
Gradle Test Executor 13 finished executing tests.
> Task :test-admin:test FAILED
RegistrationRequestHandlerTest > testRegister() FAILED
java.lang.AssertionError: Range for response status value 404 NOT_FOUND expected:<SUCCESSFUL> but was:<CLIENT_ERROR>
at org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:59)
at org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:122)
at org.springframework.test.web.reactive.server.StatusAssertions.lambda$assertSeriesAndReturn$5(StatusAssertions.java:235)
at org.springframework.test.web.reactive.server.ExchangeResult.assertWithDiagnostics(ExchangeResult.java:227)
at org.springframework.test.web.reactive.server.StatusAssertions.assertSeriesAndReturn(StatusAssertions.java:233)
at org.springframework.test.web.reactive.server.StatusAssertions.is2xxSuccessful(StatusAssertions.java:177)
at com.test.service.admin.spring.handler.RegistrationHandlerTest.testRegister(RegistrationHandlerTest.java:90)
1 test completed, 1 failed
Am I missing anything?
I missed the RouterConfiguration class in the WebFluxTest annotation and it's working fine after adding the same.
#WebFluxTest({ RegistrationHandler.class, RouterConfiguration.class })
#WebFluxTest is intended for testing the slice of an application that's using WebFlux's annotation-based programming model. It includes #Controller beans (among others) but not RouterFunction beans. The value attribute of #WebFluxTest should be used to identify a specific #Controller that you want to test rather than testing all #Controller beans. #WebFluxTest({ RegistrationHandler.class }) doesn't look right as I don't think RegistrationHandler is a controller.
You should use #SpringBootTest or #Import your RouterFunction implementation instead, as noted in the Spring Boot reference documentation:
#WebFluxTest cannot detect routes registered via the functional web framework. For testing RouterFunction beans in the context, consider importing your RouterFunction yourself via #Import or using #SpringBootTest.

Spring Cloud Reactor Sleuth - Not setting MDC with new span id when request is sent with WebClient

WebClient Initialisation
Code to call backend and initialisation as shown below. The sample code is using the spring-cloud-version Hoxton.SR4 , 'org.springframework.boot' version '2.3.1.RELEASE'. Full sample at https://github.com/malika15/sleuth-resttemplate-webclient-comparison
#EnableAutoConfiguration
#RestController
public class FrontendUsingWebClient {
#Autowired
WebClient webClient;
private Logger log = LoggerFactory.getLogger("FrontendUsingWebClient");
public static void main(String[] args) {
SpringApplication.run(FrontendUsingWebClient.class,
"--spring.application.name=FrontendUsingWebClient",
"--server.port=8081");
}
#Bean
WebClient webClient() {
return WebClient.builder().exchangeStrategies(ExchangeStrategies.builder().codecs(c ->
c.defaultCodecs().enableLoggingRequestDetails(true)).build()
).baseUrl("http://localhost:9000").build();
}
#RequestMapping("/")
public Mono<String> callBackend() {
log.info("Frontend WebClient::Begin");
Mono<String> response = webClient.get().uri("/api").retrieve().bodyToMono(String.class)
.doFirst(() -> log.info("Frontend WebClient ::doFirst"))
.doOnSubscribe(subscription -> log.info("Frontend WebClient::doOnSubscribe"))
.doOnRequest(value -> log.info("Frontend WebClient::doOnRequest"))
.doOnSuccess(s -> log.info("Frontend WebClient::doOnSuccess"))
.doOnEach(stringSignal -> log.info("Frontend WebClient::doOnEach"))
.doOnNext(s -> log.info("Frontend WebClient::doOnNext"))
.doAfterTerminate(() -> log.info("Frontend::doAfterTerminate"))
.doFinally(signalType -> log.info("Frontend WebClient::doFinally"))
.doOnCancel(() -> log.info("Frontend WebClient::doOnCancel"));
log.info("Frontend WebClient::End");
return response;
}
}
Webclient - New span id in HTTP headers but not in MDC
Notice in log output below, HTTP header has X-B3-SpanId:"e8c842b87aa2c176" where as MDC has d6737613f8c125f7,d6737613f8c125f7. Sleuth starts a new span for the webclient call and sets it on request X-B3 Headers but fails to set on the TraceContext(MDC)
2020-06-19 13:55:29.516 TRACE [FrontendUsingWebClient,d6737613f8c125f7,d6737613f8c125f7,false] 14008 --- [ctor-http-nio-3] o.s.w.r.f.client.ExchangeFunctions : [418bc045] HTTP GET http://localhost:9000/api, headers=[X-B3-TraceId:"d6737613f8c125f7", X-B3-SpanId:"e8c842b87aa2c176", X-B3-ParentSpanId:"d6737613f8c125f7", X-B3-Sampled:"0"]
RestTemplate - New span id in HTTP headers and in MDC
Notice in log output below new span being created Starting scope for span: 2d795955f8643a11/79ec6c794e86cd58 and set on MDC 2d795955f8643a11,79ec6c794e86cd58. Sleuth starts new span for RestTemplate call and closes after response is received. Setting it on MDC appropriately
2020-06-19 13:52:15.520 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.web.client.RestTemplate : HTTP GET http://localhost:9000/api
2020-06-19 13:52:15.524 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.web.client.RestTemplate : Accept=[text/plain, application/json, application/*+json, */*]
2020-06-19 13:52:15.525 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean 'tracingClientHttpRequestInterceptor'
2020-06-19 13:52:15.526 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] o.s.c.sleuth.log.Slf4jScopeDecorator : Starting scope for span: 2d795955f8643a11/79ec6c794e86cd58
2020-06-19 13:52:15.526 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] o.s.c.sleuth.log.Slf4jScopeDecorator : With parent: 3276748429663156753
2020-06-19 13:52:15.526 INFO [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] FrontendUsingRestTemplate : Frontend RestTemplate::Sending request to Backend http://localhost:9000/api
2020-06-19 13:52:15.533 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection : ProxySelector Request for http://localhost:9000/api
2020-06-19 13:52:15.535 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection : Proxy used: DIRECT
2020-06-19 13:52:15.536 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection : sun.net.www.MessageHeader#161711d99 pairs: {GET /api HTTP/1.1: null}{Accept: text/plain, application/json, application/*+json, */*}{X-B3-TraceId: 2d795955f8643a11}{X-B3-SpanId: 79ec6c794e86cd58}{X-B3-ParentSpanId: 2d795955f8643a11}{X-B3-Sampled: 0}{User-Agent: Java/11.0.2}{Host: localhost:9000}{Connection: keep-alive}
2020-06-19 13:52:15.766 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection : KeepAlive stream used: http://localhost:9000/api
2020-06-19 13:52:15.768 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] s.n.www.protocol.http.HttpURLConnection : sun.net.www.MessageHeader#1c1638c03 pairs: {null: HTTP/1.1 200 OK}{Content-Type: text/plain;charset=UTF-8}{Content-Length: 39}
2020-06-19 13:52:15.769 TRACE [FrontendUsingRestTemplate,2d795955f8643a11,79ec6c794e86cd58,false] 12564 --- [ctor-http-nio-3] o.s.c.sleuth.log.Slf4jScopeDecorator : Closing scope for span: 2d795955f8643a11/79ec6c794e86cd58
2020-06-19 13:52:15.771 DEBUG [FrontendUsingRestTemplate,2d795955f8643a11,2d795955f8643a11,false] 12564 --- [ctor-http-nio-3] o.s.web.client.RestTemplate : Response 200 OK
Looking for ideas on how to get span generated for Webclient calls into MDC for it to show up in the logs for request and response

When using the node driver, notarisation in flows hangs with a handshake failure

Whenever I try and test using the node driver, I find at the point of notarisation, my flows will hang.
After examining the node logs, it shows that the notary's message broker was unreachable:
[INFO ] 09:33:26,653 [nioEventLoopGroup-3-3] (AMQPClient.kt:91)
netty.AMQPClient.run - Retry connect {}
[INFO ] 09:33:26,657 [nioEventLoopGroup-3-4] (AMQPClient.kt:76)
netty.AMQPClient.operationComplete - Connected to localhost:10001 {}
[INFO ] 09:33:26,658 [nioEventLoopGroup-3-4]
(AMQPChannelHandler.kt:49) O=Notary Service, L=Zurich,
C=CH.channelActive - New client connection db926eb8 from
localhost/127.0.0.1:10001 to /127.0.0.1:63781 {}
[INFO ] 09:33:26,658
[nioEventLoopGroup-3-4] (AMQPClient.kt:86)
netty.AMQPClient.operationComplete - Disconnected from localhost:10001
{}
[ERROR] 09:33:26,658 [nioEventLoopGroup-3-4]
(AMQPChannelHandler.kt:98) O=Notary Service, L=Zurich,
C=CH.userEventTriggered - Handshake failure
SslHandshakeCompletionEvent(java.nio.channels.ClosedChannelException)
{}
[INFO ] 09:33:26,659 [nioEventLoopGroup-3-4]
(AMQPChannelHandler.kt:74) O=Notary Service, L=Zurich,
C=CH.channelInactive - Closed client connection db926eb8 from
localhost/127.0.0.1:10001 to /127.0.0.1:63781 {}
[INFO ] 09:33:26,659
[nioEventLoopGroup-3-4] (AMQPBridgeManager.kt:115)
peers.DLF1ZmHt1DXc9HbxzDNm6VHduUABBbNsp7Mh4DhoBs6ifd ->
localhost:10001:O=Notary Service, L=Zurich, C=CH.onSocketConnected -
Bridge Disconnected {}
While the notary logs display the following:
[INFO ] 13:24:21,735 [main] (ActiveMQServerImpl.java:540)
core.server.internalStart - AMQ221001: Apache ActiveMQ Artemis Message
Broker version 2.2.0 [localhost,
nodeID=7b3df3b8-98aa-11e8-83bd-ead493c8221e] {}
[DEBUG] 13:24:21,735 [main] (ArtemisRpcBroker.kt:51)
rpc.ArtemisRpcBroker.start - Artemis RPC broker is started. {}
[INFO ] 13:24:21,737 [main] (ArtemisMessagingClient.kt:28)
internal.ArtemisMessagingClient.start - Connecting to message broker:
localhost:10001 {}
[ERROR] 13:24:22,298 [main] (NettyConnector.java:713)
core.client.createConnection - AMQ214016: Failed to create netty
connection {} java.nio.channels.ClosedChannelException: null
at io.netty.handler.ssl.SslHandler.channelInactive(...)(Unknown Source) ~[netty-all-4.1.9.Final.jar:4.1.9.Final]
[DEBUG] 13:24:22,362 [main] (PersistentIdentityService.kt:137)
identity.PersistentIdentityService.verifyAndRegisterIdentity -
Registering identity O=Notary Service, L=Zurich, C=CH {}
[WARN ] 13:24:22,363 [main] (AppendOnlyPersistentMap.kt:79)
utilities.AppendOnlyPersistentMapBase.set - Double insert in
net.corda.node.utilities.AppendOnlyPersistentMap for entity class
class
net.corda.node.services.identity.PersistentIdentityService$PersistentIdentity
key 69ACAA32A0C7934D9454CB53EEA6CA6CCD8E4090B30C560A5A36EA10F3DC13E8,
not inserting the second time {}
[ERROR] 13:24:22,368 [main] (NodeStartup.kt:125) internal.Node.run -
Exception during node startup {}
org.apache.activemq.artemis.api.core.ActiveMQNotConnectedException:
AMQ119007: Cannot connect to server(s). Tried with all available
servers.
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:787)
~[artemis-core-client-2.2.0.jar:2.2.0]
at net.corda.nodeapi.internal.ArtemisMessagingClient.start(ArtemisMessagingClient.kt:39)
~[corda-node-api-3.2-corda.jar:?]
at net.corda.nodeapi.internal.bridging.AMQPBridgeManager.start(AMQPBridgeManager.kt:195)
~[corda-node-api-3.2-corda.jar:?]
at net.corda.nodeapi.internal.bridging.BridgeControlListener.start(BridgeControlListener.kt:35)
~[corda-node-api-3.2-corda.jar:?]
at net.corda.node.internal.Node.startMessagingService(Node.kt:301) ~[corda-node-3.2-corda.jar:?]
How do I fix this?
IntelliJ Ultimate ships with the Yourkit profiler, which by default starts when IntelliJ starts and listens on port 100001 - the default port for the Notary in Driver.
You can locate the config for this using here and alter it to use a different port as per this
Your new config line will look something like this:
-agentlib:yjpagent=delay=10000,probe_disable=*,port=30000

Unable to join Akka.NET cluster

I am having a problem joining and debugging joining to Akka.NET cluster. I am using version 1.3.8. My setup is following:
Lighthouse
Almost default code from github. Runs in console akka.hocon is following:
lighthouse {
actorsystem: "sng"
}
petabridge.cmd{
host = "0.0.0.0"
port = 9110
}
akka {
loglevel = DEBUG
loggers = ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
debug {
receive = on
autoreceive = on
lifecycle = on
event-stream = on
unhandled = on
}
}
remote {
log-sent-messages = on
log-received-messages = on
log-remote-lifecycle-events = on
enabled-transports = ["akka.remote.dot-netty.tcp"]
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
hostname = "0.0.0.0"
port = 4053
}
log-remote-lifecycle-events = DEBUG
}
cluster {
auto-down-unreachable-after = 5s
seed-nodes = []
roles = [lighthouse]
}
}
Working node
Also console (net461) application with as simple as possible startup and joining. It works as excpected. akka.hocon:
akka {
loglevel = DEBUG
loggers = ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
}
remote {
log-sent-messages = on
log-received-messages = on
log-remote-lifecycle-events = on
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
hostname = "0.0.0.0"
port = 0
}
}
cluster {
auto-down-unreachable-after = 5s
seed-nodes = ["akka.tcp://sng#127.0.0.1:4053"]
roles = [monitor]
}
}
Not working node
An .NET 4.6.1 library, registerd as COM and started in other (Media Monkey) application with VBA code:
Sub OnStartup
Set o = CreateObject("MediaMonkey.Akka.Agent.MediaMonkeyAkkaProxy")
o.Init(SDB)
End Sub
Akka system is, as in console aplikation, created with standard ActorSystem.Create("sng", config);
akka.hocon:
akka {
loglevel = DEBUG
loggers = ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
}
remote {
log-sent-messages = on
log-received-messages = on
log-remote-lifecycle-events = on
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
hostname = "0.0.0.0"
port = 0
}
}
cluster {
auto-down-unreachable-after = 5s
seed-nodes = ["akka.tcp://sng#127.0.0.1:4053"]
roles = [mediamonkey]
}
}
Debugging workflow
Startup Lighthouse application:
Configuration Result:
[Success] Name sng.Lighthouse
[Success] ServiceName sng.Lighthouse
Topshelf v4.0.0.0, .NET Framework v4.0.30319.42000
[Lighthouse] ActorSystem: sng; IP: 127.0.0.1; PORT: 4053
[Lighthouse] Performing pre-boot sanity check. Should be able to parse address [akka.tcp://sng#127.0.0.1:4053]
[Lighthouse] Parse successful.
[21:01:35 INF] Starting remoting
[21:01:35 INF] Remoting started; listening on addresses : [akka.tcp://sng#127.0.0.1:4053]
[21:01:35 INF] Remoting now listens on addresses: [akka.tcp://sng#127.0.0.1:4053]
[21:01:35 INF] Cluster Node [akka.tcp://sng#127.0.0.1:4053] - Starting up...
[21:01:35 INF] Cluster Node [akka.tcp://sng#127.0.0.1:4053] - Started up successfully
The sng.Lighthouse service is now running, press Control+C to exit.
[21:01:35 INF] petabridge.cmd host bound to [0.0.0.0:9110]
[21:01:35 INF] Node [akka.tcp://sng#127.0.0.1:4053] is JOINING, roles [lighthouse]
[21:01:35 INF] Leader is moving node [akka.tcp://sng#127.0.0.1:4053] to [Up]
Started and stopped working console node
Lighthouse logs:
[21:05:40 INF] Node [akka.tcp://sng#0.0.0.0:37516] is JOINING, roles [monitor]
[21:05:40 INF] Leader is moving node [akka.tcp://sng#0.0.0.0:37516] to [Up]
[21:05:54 INF] Connection was reset by the remote peer. Channel [[::ffff:127.0.0.1]:4053->[::ffff:127.0.0.1]:37517](Id=1293c63a)
[21:05:54 INF] Message AckIdleCheckTimer from akka://sng/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsng%400.0.0.0%3A37516-1/endpointWriter to akka://sng/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsng%400.0.0.0%3A37516-1/endpointWriter was not delivered. 1 dead letters encountered.
[21:05:55 INF] Message GossipStatus from akka://sng/system/cluster/core/daemon to akka://sng/deadLetters was not delivered. 2 dead letters encountered.
[21:05:55 INF] Message Heartbeat from akka://sng/system/cluster/core/daemon/heartbeatSender to akka://sng/deadLetters was not delivered. 3 dead letters encountered.
[21:05:56 INF] Message GossipStatus from akka://sng/system/cluster/core/daemon to akka://sng/deadLetters was not delivered. 4 dead letters encountered.
[21:05:56 INF] Message Heartbeat from akka://sng/system/cluster/core/daemon/heartbeatSender to akka://sng/deadLetters was not delivered. 5 dead letters encountered.
[21:05:57 INF] Message GossipStatus from akka://sng/system/cluster/core/daemon to akka://sng/deadLetters was not delivered. 6 dead letters encountered.
[21:05:57 INF] Message Heartbeat from akka://sng/system/cluster/core/daemon/heartbeatSender to akka://sng/deadLetters was not delivered. 7 dead letters encountered.
[21:05:58 INF] Message GossipStatus from akka://sng/system/cluster/core/daemon to akka://sng/deadLetters was not delivered. 8 dead letters encountered.
[21:05:58 INF] Message Heartbeat from akka://sng/system/cluster/core/daemon/heartbeatSender to akka://sng/deadLetters was not delivered. 9 dead letters encountered.
[21:05:59 WRN] Cluster Node [akka.tcp://sng#127.0.0.1:4053] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://sng#0.0.0.0:37516, Uid=1060233119 status = Up, role=[monitor], upNumber=2)]. Node roles [lighthouse]
[21:06:01 WRN] AssociationError [akka.tcp://sng#127.0.0.1:4053] -> akka.tcp://sng#0.0.0.0:37516: Error [Association failed with akka.tcp://sng#0.0.0.0:37516] []
[21:06:01 WRN] Tried to associate with unreachable remote address [akka.tcp://sng#0.0.0.0:37516]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Association failed with akka.tcp://sng#0.0.0.0:37516] Caused by: [System.AggregateException: One or more errors occurred. ---> Akka.Remote.Transport.InvalidAssociationException: No connection could be made because the target machine actively refused it tcp://sng#0.0.0.0:37516
at Akka.Remote.Transport.DotNetty.TcpTransport.<AssociateInternal>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Remote.Transport.DotNetty.DotNettyTransport.<Associate>d__22.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at Akka.Remote.Transport.ProtocolStateActor.<>c.<InitializeFSM>b__11_54(Task`1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) Akka.Remote.Transport.InvalidAssociationException: No connection could be made because the target machine actively refused it tcp://sng#0.0.0.0:37516
at Akka.Remote.Transport.DotNetty.TcpTransport.<AssociateInternal>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Remote.Transport.DotNetty.DotNettyTransport.<Associate>d__22.MoveNext()<---
]
[21:06:04 INF] Cluster Node [akka.tcp://sng#127.0.0.1:4053] - Leader is auto-downing unreachable node [akka.tcp://sng#127.0.0.1:4053]
[21:06:04 INF] Marking unreachable node [akka.tcp://sng#0.0.0.0:37516] as [Down]
[21:06:05 INF] Leader is removing unreachable node [akka.tcp://sng#0.0.0.0:37516]
[21:06:05 WRN] Association to [akka.tcp://sng#0.0.0.0:37516] having UID [1060233119] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.
Working node logs:
[21:05:38 INF] Starting remoting
[21:05:38 INF] Remoting started; listening on addresses : [akka.tcp://sng#0.0.0.0:37516]
[21:05:38 INF] Remoting now listens on addresses: [akka.tcp://sng#0.0.0.0:37516]
[21:05:38 INF] Cluster Node [akka.tcp://sng#0.0.0.0:37516] - Starting up...
[21:05:38 INF] Cluster Node [akka.tcp://sng#0.0.0.0:37516] - Started up successfully
[21:05:40 INF] Welcome from [akka.tcp://sng#127.0.0.1:4053]
[21:05:40 INF] Member is Up: Member(address = akka.tcp://sng#127.0.0.1:4053, Uid=439782041 status = Up, role=[lighthouse], upNumber=1)
[21:05:40 INF] Member is Up: Member(address = akka.tcp://sng#0.0.0.0:37516, Uid=1060233119 status = Up, role=[monitor], upNumber=2)
//shutdown logs are missing
Started and stopped COM node
Lighthouse logs:
[21:12:02 INF] Connection was reset by the remote peer. Channel [::ffff:127.0.0.1]:4053->[::ffff:127.0.0.1]:37546](Id=4ca91e15)
COM node logs:
[WARNING][18. 07. 2018 19:11:15][Thread 0001][ActorSystem(sng)] The type name for serializer 'hyperion' did not resolve to an actual Type: 'Akka.Serialization.HyperionSerializer, Akka.Serialization.Hyperion'
[WARNING][18. 07. 2018 19:11:15][Thread 0001][ActorSystem(sng)] Serialization binding to non existing serializer: 'hyperion'
[21:11:15 DBG] Logger log1-SerilogLogger [SerilogLogger] started
[21:11:15 DBG] StandardOutLogger being removed
[21:11:15 DBG] Default Loggers started
[21:11:15 INF] Starting remoting
[21:11:15 DBG] Starting prune timer for endpoint manager...
[21:11:15 INF] Remoting started; listening on addresses : [akka.tcp://sng#0.0.0.0:37543]
[21:11:15 INF] Remoting now listens on addresses: [akka.tcp://sng#0.0.0.0:37543]
[21:11:15 INF] Cluster Node [akka.tcp://sng#0.0.0.0:37543] - Starting up...
[21:11:15 INF] Cluster Node [akka.tcp://sng#0.0.0.0:37543] - Started up successfully
[21:11:15 DBG] [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
[21:11:15 DBG] [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
[21:11:16 DBG] [Uninitialized] Received Akka.Cluster.InternalClusterAction+JoinSeedNodes
[21:11:16 DBG] [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
[21:11:26 WRN] Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://sng#127.0.0.1:4053]
[21:11:31 WRN] Couldn't join seed nodes after [3] attempts, will try again. seed-nodes=[akka.tcp://sng#127.0.0.1:4053]
[21:11:36 WRN] Couldn't join seed nodes after [4] attempts, will try again. seed-nodes=[akka.tcp://sng#127.0.0.1:4053]
[21:11:40 ERR] No response from remote. Handshake timed out or transport failure detector triggered.
[21:11:40 WRN] AssociationError [akka.tcp://sng#0.0.0.0:37543] -> akka.tcp://sng#127.0.0.1:4053: Error [Association failed with akka.tcp://sng#127.0.0.1:4053] []
[21:11:40 WRN] Tried to associate with unreachable remote address [akka.tcp://sng#127.0.0.1:4053]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [Association failed with akka.tcp://sng#127.0.0.1:4053] Caused by: [Akka.Remote.Transport.AkkaProtocolException: No response from remote. Handshake timed out or transport failure detector triggered.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Remote.Transport.AkkaProtocolTransport.<Associate>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Akka.Remote.EndpointWriter.<AssociateAsync>d__23.MoveNext()]
[21:11:40 DBG] Disassociated [akka.tcp://sng#0.0.0.0:37543] -> akka.tcp://sng#127.0.0.1:4053
[21:11:40 INF] Message InitJoin from akka://sng/system/cluster/core/daemon/joinSeedNodeProcess-1 to akka://sng/deadLetters was not delivered. 1 dead letters encountered.
[21:11:40 INF] Message InitJoin from akka://sng/system/cluster/core/daemon/joinSeedNodeProcess-1 to akka://sng/deadLetters was not delivered. 2 dead letters encountered.
[21:11:40 INF] Message InitJoin from akka://sng/system/cluster/core/daemon/joinSeedNodeProcess-1 to akka://sng/deadLetters was not delivered. 3 dead letters encountered.
[21:11:40 INF] Message InitJoin from akka://sng/system/cluster/core/daemon/joinSeedNodeProcess-1 to akka://sng/deadLetters was not delivered. 4 dead letters encountered.
[21:11:40 INF] Message InitJoin from akka://sng/system/cluster/core/daemon/joinSeedNodeProcess-1 to akka://sng/deadLetters was not delivered. 5 dead letters encountered.
[21:11:40 INF] Message AckIdleCheckTimer from akka://sng/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsng%40127.0.0.1%3A4053-1/endpointWriter to akka://sng/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsng%40127.0.0.1%3A4053-1/endpointWriter was not delivered. 6 dead letters encountered.
[21:11:41 WRN] Couldn't join seed nodes after [5] attempts, will try again. seed-nodes=[akka.tcp://sng#127.0.0.1:4053]
[21:11:41 INF] Message InitJoin from akka://sng/system/cluster/core/daemon/joinSeedNodeProcess-1 to akka://sng/deadLetters was not delivered. 7 dead letters encountered.
[21:11:46 WRN] Couldn't join seed nodes after [6] attempts, will try again. seed-nodes=[akka.tcp://sng#127.0.0.1:4053]
[21:11:51 WRN] Couldn't join seed nodes after [7] attempts, will try again. seed-nodes=[akka.tcp://sng#127.0.0.1:4053]
Do you have any idea how to debug and/or resolve this?
As I can see that the first thing I notice in the non-working node the hocon configuration contains different "seed-nodes" address from the working node.
IMHO the "seed-nodes" in all the applications [nodes as called in cluster] withinvthe cluster needs to be same. So in the non-working node instead of
seed-nodes = ["akka.tcp://songoulash#127.0.0.1:4053"]
replace with the below which is in the working node
seed-nodes = ["akka.tcp://sng#127.0.0.1:4053"]
Also, please check the github link for sample https://github.com/AJEETX/Akka.Cluster
and another link https://github.com/AJEETX/AkkaNet.Cluster.RoundRobinGroup
#Rok, Kindly let me know if this was helpful or I can further try to investigate.