akka http error while calling lagom streaming service - lagom

I am following this.
Code snippet is as:
public ServerServiceCall<String, Source<String, ?>> tick(int intervalMs) {
return tickMessage -> {
Duration interval = Duration.ofMillis(intervalMs);
return completedFuture(Source.tick(interval, interval, tickMessage));
};
}
But when I try to hit service: http://localhost:9000/tick/2000, I get HTTP error 500.
Background traces are available here

Related

Dotnet site behind kestrel stops working (Requests return 503 from Apache server)

I run .NET core 3.1 app behind in Linux machine with Kestrel behind reverse proxy (Apache).
I constantly push some data to certain endpoint and then relay that data through signalR hub to users:
[HttpPut("{tagName}/live")]
[Authorize(Roles = "Datasource")]
[ProducesResponseType(StatusCodes.Status200OK)]
[ProducesResponseType(StatusCodes.Status400BadRequest)]
[ProducesResponseType(StatusCodes.Status401Unauthorized)]
[ProducesResponseType(StatusCodes.Status403Forbidden)]
[TypeFilter(typeof(TagControllerFilterAttribute))]
public async Task<IActionResult> PutLiveTag(Tag tag, string tagName)
{
bool status = await tagRepository.UpdateLiveData(tagName, tag);
if (status)
{
await tagHubContext.Clients.Group(tagName).SendAsync(tagName, tag);
return Ok();
}
return BadRequest();
}
Users joins and leaves that signalr hub by simply removing them from group:
public class TagDataHub : Hub
{
[HubMethodName("subscribe")]
public async Task JoinTagGroup(string tagName)
{
await Groups.AddToGroupAsync(Context.ConnectionId, tagName);
}
[HubMethodName("unsubscribe")]
public async Task LeaveTagGroup(string tagName)
{
await Groups.RemoveFromGroupAsync(Context.ConnectionId, tagName);
}
}
My app runs in deployment for few hours and then when I try use that app's API endpoints or anything related to that app I get 503 response from originated from Apache. I really don't know why Kestrel stops working...
When I run service dotnet-site status I get that app is still running.
I checked processes in server, I see that dotnet uses lots of resources, but apart from code provided above there is nothing else running in that app:
Please send help I really don't know how to debug/figure out this on linux server, I would value any suggestion.
I also looked this up and did exactly as this person, but it didn't help me: apache mpm worker

TcpClient Only Returns First Result in Spring WebFlux Controller

I have an HTTP service exposing a GET endpoint that connects to a simple echo server via TCP. The HTTP service is running on Netty.
#RestController
public class OurTcpClient {
private Connection connection;
#GetMapping("echo1")
public Mono<String> echo(#RequestParam("value") final String value) {
this.connection.outbound()
.sendString(Mono.just(String.format("%04d", value.length()) + value)) // prepend length
.then()
.subscribe();
return this.connection.inbound()
.receive()
.asString()
.next();
}
#PostConstruct
public void init() {
this.connection = TcpClient.create()
.host("localhost")
.port(10002)
.wiretap(true)
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10000)
.option(ChannelOption.SO_KEEPALIVE, true)
.connectNow();
}
}
My expectation is that I can query the service at, for example, http://localhost:8081/echo1?value=hi, as many times as I like, and receive "hi" back in each response. This works for the first request. The second request hangs indefinitely. If I then cancel the second request and attempt another, I get the following error:
{
"timestamp": "2020-04-13T18:56:40.221+0000",
"path": "/echo1",
"status": 500,
"error": "Internal Server Error",
"message": "Only one connection receive subscriber allowed."
}
Any help would be greatly appreciated.
In the example you use next()
return this.connection.inbound()
.receive()
.asString()
.next();
According to the Flux#next javadoc Emit only the first item emitted by this Flux, into a new Mono., then the subscription will be canceled.
In the context of Reactor Netty when you use next, timeout, take etc. operators that cancel the subscription this means that the connection will be closed.

How to send pings on Ktor websockets

I tried to search in api docs as well as examples, but there weren't any example demonstrating how to send pings and receive pings. The only example was of how to connect to websocket and to send a text here.
I also saw chat sample here of server side and i carefully followed that as well (i.e. set ping interval in server side configuration of WebSocket installation).
I start listening both side for pongs but none of the side was receiving any ping messages.
And there is no option to configure the client side for pings as you can see here.
I'm so much confused about how to send pings.
This is my server side:
embeddedServer(
CIO,
80
) {
install(io.ktor.websocket.WebSockets) {
pingPeriod = Duration.ofSeconds(20)
}
routing {
webSocket("/ws") {
for (frame in incoming) {
when (frame) {
is Frame.Pong -> {
println("ping's response recieved")
}
is Frame.Ping -> {
// just temporary block
println("ping recieved")
}
is Frame.Text -> {
println(frame.readText())
}
}
}
}
}
}.apply { start() }
This is my client side:
val client = HttpClient(CIO) {
install(WebSockets)
}
client.ws(
method = HttpMethod.Get,
host = "127.0.0.1",
port = 80,
path = "/ws"
) {
send(Frame.Text("Hello World!"))
for (frame in incoming) {
when (frame) {
is Frame.Pong -> {
println("ping's response received")
}
is Frame.Ping -> {
// just temporary block
println("ping recieved from server")
}
is Frame.Text -> {
println(frame.readText())
}
}
}
}
Result:
Hello World!
i.e websocket is connected, text are able to transferred, but unfortunately can't use ping/pong feature.
I also found some functions for this here pinger and ponger but now it says its part of api and gets automatically start with initiation of WebsocketSession and i also tried to put pinger in client side but that didn't send ping to server whatsoever.
Result of above code is simply Hello world gets printed in server console as sent from client side but no ping received messages.
I was having trouble getting OkHttp to use ping/pong so I filed this issue https://github.com/ktorio/ktor/issues/1803 and one of the developers replied "well the only thing I can recommend you is to try CIO out. Is does support manual Ping/Pong processing using RawWebSockets."
Haven't tried it myself but you should check out https://github.com/ktorio/ktor/blob/master/ktor-features/ktor-websockets/jvm/test/io/ktor/tests/websocket/RawWebSocketTest.kt

Not getting response with Http Async Client

I am stuck with this weird situation where sometimes my HTTP requests don't go out or I don't get a HTTP response to my request sporadically. My application makes several (100s) http requests to other 3rd party service periodically most of which work absolutely fine.
I use the CloseableHttpAsyncClient (Version 4.0) with a custom HttpRequestIntercerptor and HttpResponseInterceptor. These were mainly added for debugging purpose with the RequestInterceptor is the last interceptor in the chain and the ResponseInterceptor is the first one. The idea was to log each http request at the last stage before it sends the actual request and to log each http response when it is first received.
I have the following pattern to setup the async client:
HttpAsyncClientBuilder asyncClientBuilder = HttpAsyncClientBuilder.create();
asyncClientBuilder.addInterceptorLast(new MyHttpRequestInterceptor());
asyncClientBuilder.addInterceptorFirst(new MyHttpResponseInterceptor());
IOReactorConfig reactorConfig = IOReactorConfig.DEFAULT;
reactorConfig.setConnectTimeout(5 * 60 * 1000); // 5 mins
reactorConfig.setSoTimeout(5 * 60 * 1000); // 5 mins
asyncClientBuilder.setDefaultIOReactorConfig(reactorConfig);
System.setProperty("http.maxConnections", "100");
this.asyncHttpClient = asyncClientBuilder.useSystemProperties().build();
this.asyncHttpClient.start();
To make the request I do:
HttpGet httpGet = new HttpGet("some url");
asyncHttpClient.execute(httpGet, new AsyncHTTPResponseHandler(requestMetadata));
Here is my AsyncHTTPResponseHandler class:
class AsyncHTTPResponseHandler implements FutureCallback<HttpResponse> {
// local copy of the request for reference while processing the response.
private RequestMetadata requestMetadata;
public AsyncHTTPResponseHandler(final RequestMetadata requestMetadata) {
this.setRequestMetadata(requestMetadata);
Thread.currentThread().setUncaughtExceptionHandler(new HttpUncaughtExceptionHandler(requestMetadata));
}
#Override
public void cancelled() {
logger.error("AsyncHTTPResponseHandler#Http request id: {} cancelled",
requestMetadata.getRequestId()));
}
#Override
public void completed(HttpResponse response) {
logger.debug("Received HTTP Response for request id: {}",
requestMetadata.getRequestId());
//handleHttpResponse(requestMetadata, response);
}
#Override
public void failed(Exception e) {
logger.error("AsyncHTTPResponseHandler#Error in Http request id: " + requestMetadata.getRequestId(), e);
}
}
Based on this setup, I see the following cases based on my interceptors logs:
1. My application http request triggers an asyncclient HttpRequest and I get the HttpResponse -- Success.
2. My application http request triggers an asyncclient HttpRequest (the interceptor logs it) and I don't get the HttpResponse for this request --- Don't know why?
3. My application http request does not trigger an asyncclient HttpRequest (the interceptor does not log it) and I don't get the HttpResponse for this request --- Don't know why?
Any tips or suggestions on what I can do fix this or debug this problem further?
Thanks!!
So, thought I will share my findings and solution here.
We were experiencing symptoms similar to this bug: https://issues.apache.org/jira/browse/HTTPASYNC-79
If you enable DEBUG logging for "org.apache.http.impl.nio" package, then you can see the exchanges. Note: The logs will be very verbose.
The issue was resolved by upgrading the HttpAsyncClient library from 4.0 to 4.0.2. I have also enabled socket and Connection timeouts. You should see timeout exceptions in the log files with this.
Here is how my HttpAsyncClient instance looks now:
HttpAsyncClientBuilder asyncClientBuilder = HttpAsyncClientBuilder.create();
asyncClientBuilder.addInterceptorLast(new MyHttpRequestInterceptor());
asyncClientBuilder.addInterceptorFirst(new MyHttpResponseInterceptor());
// reactor config
IOReactorConfig reactorConfig = IOReactorConfig.custom()
.setConnectTimeout(TIMEOUT_5_MINS_IN_MILLIS)
.setSoTimeout(TIMEOUT_5_MINS_IN_MILLIS).build();
asyncClientBuilder.setDefaultIOReactorConfig(reactorConfig);
// request config
RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(TIMEOUT_5_MINS_IN_MILLIS)
.setConnectionRequestTimeout(TIMEOUT_5_MINS_IN_MILLIS)
.setSocketTimeout(TIMEOUT_5_MINS_IN_MILLIS).build();
asyncClientBuilder.setDefaultRequestConfig(requestConfig);
// connection config
ConnectionConfig connectionConfig = ConnectionConfig.custom()
.setMalformedInputAction(CodingErrorAction.IGNORE)
.setUnmappableInputAction(CodingErrorAction.IGNORE)
.build();
asyncClientBuilder.setDefaultConnectionConfig(connectionConfig);
System.setProperty("http.maxConnections", "100");
System.setProperty("http.conn-manager.timeout", "300000"); // 5 mins
this.asyncHttpClient = asyncClientBuilder.useSystemProperties().build();

What WCF Exceptions should I retry on failure for? (such as the bogus 'xxx host did not receive a reply within 00:01:00')

I have a WCF client that has thrown this common error, just to be resolved with retrying the HTTP call to the server. For what it's worth this exception was not generated within 1 minute. It was generated in 3 seconds.
The request operation sent to xxxxxx
did not receive a reply within the
configured timeout (00:01:00). The
time allotted to this operation may
have been a portion of a longer
timeout. This may be because the
service is still processing the
operation or because the service was
unable to send a reply message. Please
consider increasing the operation
timeout (by casting the channel/proxy
to IContextChannel and setting the
OperationTimeout property) and ensure
that the service is able to connect to
the client
How are professionals handling these common WCF errors? What other bogus errors should I handle.
For example, I'm considering timing the WCF call and if that above (bogus) error is thrown in under 55 seconds, I retry the entire operation (using a while() loop). I believe I have to reset the entire channel, but I'm hoping you guys will tell me what's right to do.
What other
I make all of my WCF calls from a custom "using" statement which handles exceptions and potential retires. My code optionally allows me to pass a policy object to the statement so I can easily change the behavior, like if I don't want to retry on error.
The gist of the code is as follows:
[MethodImpl(MethodImplOptions.NoInlining)]
public static void ProxyUsing<T>(ClientBase<T> proxy, Action action)
where T : class
{
try
{
proxy.Open();
using(OperationContextScope context = new OperationContextScope(proxy.InnerChannel))
{
//Add some headers here, or whatever you want
action();
}
}
catch(FaultException fe)
{
//Handle stuff here
}
finally
{
try
{
if(proxy != null
&& proxy.State != CommunicationState.Faulted)
{
proxy.Close();
}
else
{
proxy.Abort();
}
}
catch
{
if(proxy != null)
{
proxy.Abort();
}
}
}
}
You can then use the call like follows:
ProxyUsing<IMyService>(myService = GetServiceInstance(), () =>
{
myService.SomeMethod(...);
});
The NoInlining call probably isn't important for you. I need it because I have some custom logging code that logs the call stack after an exception, so it's important to preserve that method hierarchy in that case.