I have a Ktor application that return multiple files as a stream but when the client has a slow internet connection, apparently, the stream gets full and blows launching an io.netty.handler.timeout.WriteTimeoutException.
kotlinx.coroutines.JobCancellationException: Parent job is Cancelling
Caused by: io.netty.handler.timeout.WriteTimeoutException: null
Is there a way to prevent that ?
Here's my method:
suspend fun ApplicationCall.returnMultiFiles(files: List<StreamFile>) {
log.debug("Returning Multiple Files: ${files.size}")
val call = this // Just to make it more readable
val bufferSize = 16 * 1024
call.response.headers.append(
HttpHeaders.ContentDisposition,
ContentDisposition.Attachment.withParameter(ContentDisposition.Parameters.FileName, "${UUID.randomUUID()}.zip").toString())
call.respondOutputStream(ContentType.parse("application/octet-stream")) {
val returnOutputStream = this.buffered(bufferSize) // Just to make it more readable
ZipOutputStream(returnOutputStream).use { zipout ->
files.forEach{ record ->
try {
zipout.putNextEntry(ZipEntry(record.getZipEntry()))
log.debug("Adding: ${record.getZipEntry()}")
record.getInputStream().use { fileInput ->
fileInput.copyTo(zipout, bufferSize)
}
} catch (e: Exception) {
log.error("Failed to add ${record.getZipEntry()}", e)
}
}
}
}
}
Related
When saving the text file it is being saved for example in the Documents/ folder. I'm aware of that I need to use append to add new line of text to that existing file but I am not entirely sure how and where to add this line of code and check that file exists.
This is the current code that is handling saving the file.
...
save.setOnClickListener {
saveFile()
}
}
private fun saveFile() {
val fileIntent = Intent(Intent.ACTION_CREATE_DOCUMENT).apply {
addCategory(Intent.CATEGORY_OPENABLE)
type = "text/plain"
putExtra(Intent.EXTRA_TITLE, "measurements.txt")
// Optionally, specify a URI for the directory that should be opened in
// the system file picker before your app creates the document.
//putExtra(DocumentsContract.EXTRA_INITIAL_URI, pickerInitialUri)
}
saveLauncher.launch(fileIntent)
}
private var saveLauncher = registerForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
if (result.resultCode == Activity.RESULT_OK) {
val uri = result.data?.data
try {
val outputStream = uri?.let { contentResolver.openOutputStream(it) }
outputStream?.write(getMeasurement?.toByteArray())
outputStream?.close()
} catch (e: Exception) {
Toast.makeText(this, "Error is ${e.localizedMessage}", Toast.LENGTH_SHORT).show()
}
}
}
You can use a Writer to append text. Also, you should not put your close call inside the try block, or it could get skipped and you won't release the file. close should go in a finally block, but it's easier to use use { } instead.
When you open an output stream from the content resolver, you can specify write and append mode by passing "wa" as the second argument. See here and here in the documentation.
openOutputStream throws an exception if the file doesn't already exist. If you want to create a file if it doesn't exist yet, you'll need to add that logic.
I like to exit early from a function rather than nest everything in an if-statement, so I rearranged it that way in my example, but you don't have to do that.
private var saveLauncher = registerForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
val uri = result.data?.data
val content = getMeasurement
if (result.resultCode != Activity.RESULT_OK || uri == null || content == null) {
return#registerForActivityResult
}
try {
contentResolver.openOutputStream(uri, "wa")
.writer().use { it.write(content) }
} catch (e: Exception) {
Toast.makeText(this, "Error is ${e.localizedMessage}", Toast.LENGTH_SHORT).show()
}
}
Ktor (1.4.2) seems to be suppresing exceptions. According to documentation I need to re-throw exception like this
install(StatusPages) {
exception<Throwable> { ex ->
val callstack: String = ....
call.respond(HttpStatusCode.InternalServerError,
ApiError(ex.javaClass.simpleName, ex.message ?: "Empty Message", callstack)
)
throw ex
}
}
I wrote wrapper around Logger interface to make sure it's calling Logger.error() (it doesn't) and configured it like this
install(CallLogging) {
level = org.slf4j.event.Level.INFO
logger = Log4j2Wrapper(logger2)
}
I can see INFO-level logs. What am I missing ?
Turned out the problem is in rather akward design of ktor api. Summary of my findings:
Contrary to common expectation to see log events of specified log level and above it doesn't that way with CallLogging. It expects exact match of the logging level which minimizes usefulness of this feature.
You need to overwrite logger specified in ApplicationEnvironment.log. But if you follow official ktor examples and do it like this
val server = embeddedServer(Netty, port) {
applicationEngineEnvironment {
log = YOUR_LOGGER
}
}
it won't work because value call.application.environment inside function io.ktor.server.engine.defaultEnginePipeline(env) is still original one, created from logger name in config file or NOP logger.
Hence you'll need to initialize your server like this
val env = applicationEngineEnvironment {
log = YOUR_LOGGER
module {
install(CallLogging) {
level = org.slf4j.event.Level.INFO
}
install(StatusPages) {
exception<Throwable> { ex ->
val callstack: String = ...
call.respond(HttpStatusCode.InternalServerError,
ApiError(ex.javaClass.simpleName, ex.message ?: "Empty Message", callstack)
)
throw ex
}
}
routing {
}
}
connector {
host = "0.0.0.0"
port = ...
}
}
val server = embeddedServer(Netty, env)
I want to implement circuit breaker using vert.x for rabbit mq. if rabbit mq is down then circuit is open based on the configuration.
I did separate POC for vert.x circuit breaker and also able to connect with Rabbit MQ client . using vert.x .
Now, I want to build circuit breaker if circuit is open then client will not to push messages in queue , and keep in data base, once circuit is closed then it will start pushing db messages in mq. Please help.
I've use below links for create working poc.
https://vertx.io/docs/vertx-circuit-breaker/java/
https://vertx.io/docs/vertx-rabbitmq-client/java/
Snippet Used for Circuit Breaker
CircuitBreakerOptions options = new CircuitBreakerOptions()
.setMaxFailures(0)
.setTimeout(5000)
.setMaxRetries(3)
.setFallbackOnFailure(true);
CircuitBreaker breaker =
CircuitBreaker.create("my-circuit-breaker", vertx, options)
.openHandler(v -> {
System.out.println("Circuit opened");
}).closeHandler(v -> {
System.out.println("Circuit closed");
}).retryPolicy(retryCount -> retryCount * 100L);
breaker.executeWithFallback(promise -> {
vertx.createHttpClient().getNow(8080, "localhost", "/", response -> {
if (response.statusCode() != 200) {
promise.fail("HTTP error");
} else {
response.exceptionHandler(promise::fail).bodyHandler(buffer -> {
promise.complete(buffer.toString());
});
}
});
}, v -> {
// Executed when the circuit is opened
return "Hello (fallback)";
}, ar -> {
// Do something with the result
System.out.println("Result: " + ar.result());
});
}
Snippet Used for RAbbit MQ Client
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
RabbitMQOptions config = new RabbitMQOptions();
// Each parameter is optional
// The default parameter with be used if the parameter is not set
config.setUser("guest");
config.setPassword("guest");
config.setHost("localhost");
config.setPort(5672);
// config.setVirtualHost("vhost1");
// config.setConnectionTimeout(6000); // in milliseconds
config.setRequestedHeartbeat(60); // in seconds
// config.setHandshakeTimeout(6000); // in milliseconds
// config.setRequestedChannelMax(5);
// config.setNetworkRecoveryInterval(500); // in milliseconds
// config.setAutomaticRecoveryEnabled(true);
// vertx.
RabbitMQClient client = RabbitMQClient.create(vertx, config);
CircuitBreakerOptions options = new CircuitBreakerOptions().setMaxFailures(0).setTimeout(5000).setMaxRetries(0)
.setFallbackOnFailure(true);
CircuitBreaker breaker = CircuitBreaker.create("my-circuit-breaker", vertx, options).openHandler(v -> {
System.out.println("Circuit opened");
}).closeHandler(v -> {
System.out.println("Circuit closed");
}).retryPolicy(retryCount -> retryCount * 100L);
breaker.executeWithFallback(promise -> {
vertx.createHttpClient().getNow(8080, "localhost", "/", response -> {
if (response.statusCode() != 200) {
promise.fail("HTTP error");
} else {
response.exceptionHandler(promise::fail).bodyHandler(buffer -> {
promise.complete(buffer.toString());
});
}
});
}, v -> {
// Executed when the circuit is opened
return "Hello (fallback)";
}, ar -> {
// Do something with the result
System.out.println("Result: " + ar.result());
});
client.start(rh -> {
if (rh.failed()) {
System.out.println("failed");
} else {
for (int i = 0; i > 5; i++) {
System.out.println(rh.succeeded());
System.out.println("client : " + client.isConnected());
breaker.executeWithFallback(promise -> {
if(!client.isConnected()) {
promise.fail("MQ client is not connected");
}
},v -> {
// Executed when the circuit is opened
return "Hello (fallback)";
}, ar -> {
// Do something with the result
System.out.println("Result: " + ar.result());
});
/* RabbitMQClient client2 = RabbitMQClient.create(vertx); */
JsonObject message = new JsonObject().put("body", "Hello RabbitMQ, from Vert.x !");
client.basicPublish("eventExchange", "order.created", message, pubResult -> {
if (pubResult.succeeded()) {
System.out.println("Message published !");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} else {
pubResult.cause().printStackTrace();
}
});
}
}
});
}
Although its working but, circuit open in case rabbit mq down . But its not looks a standard way of ding it with vert.x .
CAn you please suggest of give input how i can implement it.
I'm trying to write an infinite length response body and detect when a client disconnects so I can stop writing. I'm used to getting socket exceptions or similar when a client closes the connection but that doesn't seem to be happening when writing directly to Response.Body. I can close the client applications and the server side just keeps on writing. I've included the relevant code below. It's entirely possible there is a better way to do it but this came to mind. Basically I have a live video feed which should go on forever. I'm writing to ResponseBody as chunked content (No content length, flushing after each video frame). The video frames are received via an event callback from elsewhere in the program so I'm subscribing to the events in the controller method and then forcing it to stay open with the await Task.Delay loop so the Response stream isn't closed. The callback for H264PacketReceived is formatting the data as a streaming mp4 file and writing it to the Response Stream. This all seems to work fine, I can play the live stream with ffmpeg or chrome, but when I close the client application I don't get an exception or anything. It just keeps writing to the stream without any errors.
public class LiveController : ControllerBase
{
[HttpGet]
[Route("/live/{cameraId}/{stream}.mp4")]
public async Task GetLiveMP4(Guid cameraId, int stream)
{
try
{
Response.StatusCode = 200;
Response.ContentType = "video/mp4";
Response.Headers.Add("Cache-Control", "no-store");
Response.Headers.Add("Connection", "close");
ms = Response.Body;
lock (TCPVideoReceiver.CameraStreams)
{
TCPVideoReceiver.CameraStreams.TryGetValue(cameraId, out cameraStream);
}
if (this.PacketStream == null)
{
throw new KeyNotFoundException($"Stream {cameraId}_{stream} not found");
}
else
{
connected = true;
this.PacketStream.H264PacketReceived += DefaultStream_H264PacketReceived;
this.PacketStream.StreamClosed += PacketStream_StreamClosed;
}
while(connected)
{
await Task.Delay(1000);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
finally
{
connected = false;
this.PacketStream.H264PacketReceived -= DefaultStream_H264PacketReceived;
this.PacketStream.StreamClosed -= PacketStream_StreamClosed;
}
}
private bool connected = false;
private PacketStream PacketStream;
private Mp4File mp4File;
private Stream ms;
private async void PacketStream_StreamClosed(PacketStream source)
{
await Task.Run(() =>
{
try
{
Console.WriteLine($"Closing live stream");
connected = false;
ms.Close();
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
});
}
private async void DefaultStream_H264PacketReceived(PacketStream source, H264Packet packet)
{
try
{
if (mp4File == null && packet.IsIFrame)
{
mp4File = new Mp4File(null, packet.sps, packet.pps);
var _p = mp4File.WriteHeader(0);
await ms.WriteAsync(mp4File.buffer, 0, _p);
}
if (mp4File != null)
{
var _p = mp4File.WriteFrame(packet, 0);
var start = mp4File._moofScratchIndex - _p;
if (_p > 0)
{
await ms.WriteAsync(mp4File._moofScratch, start, _p);
await ms.FlushAsync();
}
}
return;
}
catch (Exception ex)
{
connected = false;
Console.WriteLine(ex.ToString());
}
}
Answering my own question.
When the client disconnects mvc core sets the cancellation token HttpContext.RequestAborted
By monitoring and/or using that cancellation token you can detect a disconnect and clean everything up.
That said, the entire design can be improved by creating a custom stream which encapsulates the event handling (producer/consumer). Then the controller action can be reduced to.
return File(new MyCustomStream(cameraId, stream), "video/mp4");
The File Method already monitors the cancellation token and everything works as you'd expect.
I'm subscribing to a MQTT Topic(in my case it is app unique user id).I'm using AWS IOT core services for subscription.Whenever home screen opens up and I got connected callback from awsConnectClient,I make the call for subscription. Now what is happening if app gets open up three times It subscribed to the same topic 3 time.Now whenever any message publish to that topic.It received by app 3 times.
Now what I want to do that I want to know that if this userId is already subscribed from this device I would not make a call for subscription again from same device.
One approach could be If I save in my app that I had already subscribed to this topic and do not make the call for subscription again. but I doubt if this approach could be correct for all scenarios.Could it be possible that we could drive this logic from the server end only, if any aws iot api could give me that this is already subscribed.
fun connectClick() {
Log.d(TAG, "clientId = $clientId")
try {
mqttManager.connect(clientKeyStore) { status, throwable ->
Log.d(TAG, "Status = " + status.toString())
var formattedStatus = String.format(getString(R.string.status_msg),status.toString())
if (status == AWSIotMqttClientStatusCallback.AWSIotMqttClientStatus.Connected) {
Log.i(TAG, " subscribed to - " + VoiceXPreference(this).rosterName)
unsubscribe()
subscribeClick(VoiceXPreference(this).rosterName)
}
runOnUiThread {
tv_iot_status.text = formattedStatus
if (throwable != null) {
Log.e(TAG, "Connection error.", throwable)
}
}
}
} catch (e: Exception) {
Log.e(TAG, "Connection error.", e)
}
}
Above is my subscription code.Although I'm always unsubscribing before subscribing but this is not working for me.
Following is my initClient call which makes the connection request. I have added the if check if mqttManager is already initialised first disconnect and then make connect request. Although I have put initRequest inside onCreate() call back of app screen which calls only once when the app opens up. I have checked the logs it is being called only once.
AWSMobileClient.getInstance().initialize(this, object : Callback<UserStateDetails> {
override fun onResult(result: UserStateDetails) {
Log.i(TAG,"connect request called");
if(mqttManager != null){
mqttManager?.disconnect()
}
initIoTClient()
}
override fun onError(e: Exception) {
Log.e(TAG, "onError: ", e)
}
})
Following is my subscribe code snippet which is subscribing to unique userId
fun subscribeClick(topic: String) {
Log.d(TAG, "topic = $topic")
try {
mqttManager?.subscribeToTopic(topic, AWSIotMqttQos.QOS0,
{ topic, data ->
runOnUiThread {
try {
val message = String(data, Charsets.UTF_8)
Log.d(TAG, "Message arrived:")
Log.d(TAG, " Topic: $topic")
Log.d(TAG, " Message: $message")
val gson = Gson()
val notificationModel = gson.fromJson(message, NotificationModel::class.java)
var orderServiceMapperResponseModel = OrderServiceMapperResponseModel()
orderServiceMapperResponseModel.seatId = notificationModel.seatId
orderServiceMapperResponseModel.serviceName = notificationModel.service
orderServiceMapperResponseModel.id = notificationModel.id
orderServiceMapperResponseModel.createdDate = notificationModel.createdDate
serviceList.add(orderServiceMapperResponseModel)
if (isPictureInPictureMode) {
if (isShownNotification) {
updateNotificationCount()
} else {
updatePIPWindowContent()
}
} else {
updateAdapterDataSource()
}
} catch (e: UnsupportedEncodingException) {
Log.e(TAG, "Message encoding error.", e)
}
}
})
} catch (e: Exception) {
Log.e(TAG, "Subscription error.", e)
}
}
I'm also always making disconnect() request inside onDestroy() of my app screen
mqttManager?.disconnect()
But Still I'm getting 3 subscription messages instead of 1.
You receive 3 duplicated messages not because you subscribe 3 times but because you create 3 individual connections.
The MQTT specification clearly states that
If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription.
meaning duplicated subscriptions per connection never happen, unless the server has a broken implementation.
Your code looks like that it never send disconnect requests while a new connection is created whenever the code block is invoked.
You should keep a single MQTT session, or make sure you close the connection when the app is closed.