I use netty to transfer files between the server and the client, but because I use FileRegion to send and receive, I don't know the progress of the file transfer. This is the server code:
RandomAccessFile(file, "r").run file#{
ctx.channel().run {
writeAndFlush(DefaultFileRegion(this#file.channel, 0, this#file.length())).run {
addListener {
this#file.close()
}
}
}
}
And client:
RandomAccessFile("Repository\\Music\\${this}\\${this}.mp3", "rw").run file#{
readBytes(channel, 0, length)
close()
}
I tried to monitor bytebuf's reading progress on my main decoder (MessageToMessageCodec), but I failed. Netty seems to be calling my decoder after reading all the data. How should I monitor the transmission progress? Be deeply grateful
Related
My application set up is mentioned as part of issue# Correct way of using spring webclient in spring amqp
where I am trying to use Spring webclient to make API calls in Spring AMQP rabbit MQ consumer threads.
Issue seems to be that parallel flux blocking call just stalls or takes a very long time after first few requests are fired.
To simulate this, I did below minimalistic set up -
Dependencies used
Spring boot 2.2.6.RELEASE
spring-boot-starter-web
spring-boot-starter-webflux
reactor-netty 0.9.14.RELEASE
As mentioned in the other linked issue, below is configuration for webclient -
#Bean
public WebClient webClient() {
ConnectionProvider connectionProvider = ConnectionProvider
.builder("fixed")
.lifo()
.pendingAcquireTimeout(Duration.ofMillis(200000))
.maxConnections(100)
.pendingAcquireMaxCount(3000)
.maxIdleTime(Duration.ofMillis(290000))
.build();
HttpClient client = HttpClient.create(connectionProvider);
client.tcpConfiguration(<<connection timeout, read timeout, write timeout is set here....>>);
Webclient.Builder builder =
Webclient.builder().baseUrl(<<base URL>>).clientConnector(new ReactorClientHttpConnector(client));
return builder.build();
}
Below is #Service class with parallel flux webclient calls -
#Service
public class FluxtestService {
public Flux<Response> getFlux(List<Request> reqList) {
return Flux
.fromIterable(reqList)
.parallel()
.runOn(Schedulers.elastic())
.flatMap(s -> {
return webClient
.method(POST)
.uri(<<downstream url>>)
.body(BodyInserters.fromValue(s))
.exchange()
.flatMap(response -> {
if(response.statusCode().isError()){
return Mono.just(new Response());
}
return response.bodyToMono(Response.class);
})
}).sequential();
}
}
}
To simulate Spring AMQP rabbit mq consumer/listener, I created below #RestController -
#RestController
public class FluxTestController
#Autowired
private FluxtestService service;
#PostMapping("/fluxtest")
public List<Response> getFlux (List<Request> reqlist) {
return service.getFlux(reqlist).collectList().block();
}
I tried firing requests from jmeter with around 15 threads. First few set of requests are processed very quickly. While requests are being served, I can see below set of logs in log file -
Channel cleaned, now 32 active connections and 68 inactive connections
Once I submit more set of requests, the active connections keeps increasing till it reaches max configured 100. I don't see it decreasing at all. Till this point, response time is ok.
But any subsequent requests start taking very long time. Also I don't see the active connections reducing much at all even though there are no requests being fired.
Also after some time, I see below exceptions -
reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 200000 ms
This probably shows that the downstream connection is not being released. Please help advise on this issue and possible fixes.
Seems issue was because the underlying connection was not being properly released in case webclient downstream call responded with error status. While using "exchange" with "webclient", it seems we need to ensure that the response is properly released; else it can lead to connections leak. Below are the changes that seemed to fix this issue -
Replace
.flatMap(response -> {
if(response.statusCode().isError()) {
return Mono.just(new Response());
}
return response.bodyToMono(Response.class);
})
with
.flatMap(response -> {
if(response.statusCode().isError()) {
response.releaseBody().thenReturn(Mono.just(new Response()));
}
return response.bodyToMono(Response.class);
})
I have read the spring cloud stream binder reference document which mentioned DLQ processing using #RabbitListener. https://docs.spring.io/spring-cloud-stream-binder-rabbit/docs/3.0.10.RELEASE/reference/html/spring-cloud-stream-binder-rabbit.html#rabbit-dlq-processing
Can we achieve the same via Spring cloud function like we can do the same for consumers?
Like
#Bean
public Consumer<Message> dlqprocess(DLQProcess dlqprocess) {
return t -> dlqprocess.do(t);
}
I am not sure whether we can do this or not. If this allows what are the other configuration we have to do?
If you aim is to requeue failed messages, the function can just throw exceptions as described in docs.
Furthermore, if you need more fine-grained control about send and requeued messages you can use StreamBrdidge. Here you need to explicitly define DLQ binding in the configuration file:
spring.cloud.stream.bindings.myDlq-out-0.destination=DLX
spring.cloud.stream.rabbit.bindings.myDlq-out-0.producer.exchangeType=direct
spring.cloud.stream.rabbit.bindings.myDlq-out-0.producer.routingKeyExpression='myDestination.myGroup'
spring.cloud.stream.source=myDlq
Finally, the function controls whether to send and requeue the message:
#Bean
public Consumer<Message> process(StreamBridge streamBridge) {
return t -> {
// ....
if(republish) streamBridge.send("myDlq-out-0", t);
if(sendToDestination) streamBridge.send("myDestination-out-0", t);
// ....
};
}
I tried to search in api docs as well as examples, but there weren't any example demonstrating how to send pings and receive pings. The only example was of how to connect to websocket and to send a text here.
I also saw chat sample here of server side and i carefully followed that as well (i.e. set ping interval in server side configuration of WebSocket installation).
I start listening both side for pongs but none of the side was receiving any ping messages.
And there is no option to configure the client side for pings as you can see here.
I'm so much confused about how to send pings.
This is my server side:
embeddedServer(
CIO,
80
) {
install(io.ktor.websocket.WebSockets) {
pingPeriod = Duration.ofSeconds(20)
}
routing {
webSocket("/ws") {
for (frame in incoming) {
when (frame) {
is Frame.Pong -> {
println("ping's response recieved")
}
is Frame.Ping -> {
// just temporary block
println("ping recieved")
}
is Frame.Text -> {
println(frame.readText())
}
}
}
}
}
}.apply { start() }
This is my client side:
val client = HttpClient(CIO) {
install(WebSockets)
}
client.ws(
method = HttpMethod.Get,
host = "127.0.0.1",
port = 80,
path = "/ws"
) {
send(Frame.Text("Hello World!"))
for (frame in incoming) {
when (frame) {
is Frame.Pong -> {
println("ping's response received")
}
is Frame.Ping -> {
// just temporary block
println("ping recieved from server")
}
is Frame.Text -> {
println(frame.readText())
}
}
}
}
Result:
Hello World!
i.e websocket is connected, text are able to transferred, but unfortunately can't use ping/pong feature.
I also found some functions for this here pinger and ponger but now it says its part of api and gets automatically start with initiation of WebsocketSession and i also tried to put pinger in client side but that didn't send ping to server whatsoever.
Result of above code is simply Hello world gets printed in server console as sent from client side but no ping received messages.
I was having trouble getting OkHttp to use ping/pong so I filed this issue https://github.com/ktorio/ktor/issues/1803 and one of the developers replied "well the only thing I can recommend you is to try CIO out. Is does support manual Ping/Pong processing using RawWebSockets."
Haven't tried it myself but you should check out https://github.com/ktorio/ktor/blob/master/ktor-features/ktor-websockets/jvm/test/io/ktor/tests/websocket/RawWebSocketTest.kt
Here is my paho client code
// Create a client instance
client = new Paho.MQTT.Client('127.0.0.1', 1883, "clientId");
// set callback handlers
client.onConnectionLost = onConnectionLost;
client.onMessageArrived = onMessageArrived;
// connect the client
client.connect({onSuccess:onConnect});
// called when the client connects
function onConnect() {
// Once a connection has been made, make a subscription and send a message.
console.log("onConnect");
client.subscribe("/World");
message = new Paho.MQTT.Message("Hello");
message.destinationName = "/World";
client.send(message);
}
// called when the client loses its connection
function onConnectionLost(responseObject) {
if (responseObject.errorCode !== 0) {
console.log("onConnectionLost:"+responseObject.errorMessage);
}
}
// called when a message arrives
function onMessageArrived(message) {
console.log("onMessageArrived:"+message.payloadString);
}
On Rabbitmq server everything is default seetings. When i run this code i get WebSocket connection to 'ws://127.0.0.1:1883/mqtt' failed: Connection closed before receiving a handshake response
What i am missing ?
From my personal experience with Paho MQTT JavaScript library and RabbitMQ broker on windows, here is a list of things that you need to do to be able to use MQTT from JS from within a browser:
Install rabbitmq_web_mqtt plugin (you may find latest binary here, copy it to "c:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.2\plugins\", and enable from command line using "rabbitmq-plugins enable rabbitmq_web_mqtt".
Of course, MQTT plugin also needs to be enabled on broker
For me, client was not working with version 3.6.1 of RabbitMQ, while it works fine with version 3.6.2 (Windows)
Port to be used for connections is 15675, NOT 1883!
Make sure to specify all 4 parameters when making instance of Paho.MQTT.Client. In case when you omit one, you get websocket connection error which may be quite misleading.
Finally, here is a code snippet which I tested and works perfectly (just makes connection):
client = new Paho.MQTT.Client("localhost", 15675, "/ws", "client-1");
//set callback handlers
client.onConnectionLost = onConnectionLost;
client.onMessageArrived = onMessageArrived;
//connect the client
client.connect({
onSuccess : onConnect
});
//called when the client connects
function onConnect() {
console.log("Connected");
}
//called when the client loses its connection
function onConnectionLost(responseObject) {
if (responseObject.errorCode !== 0) {
console.log("onConnectionLost:" + responseObject.errorMessage);
}
}
//called when a message arrives
function onMessageArrived(message) {
console.log("onMessageArrived:" + message.payloadString);
}
It's not clear in the question but I assume you are running the code above in a web browser.
This will be making a MQTT connection over Websockets (as shown in the error). This is different from a native MQTT over TCP connection.
The default pure MQTT port if 1883, Websocket support is likely to be on a different port.
You will need to configure RabbitMQ to accept MQTT over Websockets as well as pure MQTT, this pull request for RabbitMQ seams to talk about adding this capability. It mentions that this capability was only added in version 3.6.x and that the documentaion is still outstanding (as of 9th Feb 2016)
I am trying to implement WebRTC. I am trying to create P2P connection using
var peerC = new webkitDeprecatedRTCPeerConnection("STUN stun.l.google.com:19302", onSignal);
Here, onSignal is:
// when PeerConn is created, send setup data to peer via WebSocket
function onSignal(message) {
socket.send(message);
}
// when remote adds a stream, hand it on to the local video element
function onRemoteStreamAdded(event) {
remotevid.src = window.webkitURL.createObjectURL(event.stream);
document.getElementById('anim').style.visibility='visible';
}
// when remote removes a stream, remove it from the local video element
function onRemoteStreamRemoved(event) {
remotevid.src = "";
}
But this do not work for me. Even if I use any of webkitDeprecatedRTCPeerConnection, DeprecatedRTCPeerConnection, RTCPeerConnection, nothing proceeds.
Some help would be really thankfull.
Use webkitRTCPeerConnection or even better, use https://github.com/webrtc/adapter which shims RTCPeerConnection.
See also https://webrtc.github.io/samples/ for a large amount of maintained samples how to use the WebRTC API in the browser.