how to add feedback between client and server in Kotlin? - kotlin

I am writing a simple client - server in Kotlin. There was a problem sending data from the server to the client.
Server: without the "writer" everything works fine and the message comes from the client
Client: without "reader" everything works fine and the server accepts the message.
As soon as I try to add "feedback" to the server and client, everything immediately stops working.
I understand that streams need to be closed, but I cannot understand at what point exactly. I tried many options - it does not work
Server:
import java.net.ServerSocket
import java.util.*
fun main() {
val server = ServerSocket(9000)
val client = server.accept()
val reader = Scanner(client.getInputStream())
val writer = client.getOutputStream()
print(reader.nextLine())
writer.write("Yes".toByteArray())
writer.flush()
}
Client
import java.net.Socket
import java.util.*
fun main() {
val client = Socket("192.168.1.8", 9000)
var writer = client.getOutputStream()
var reader = Scanner(client.getInputStream())
writer.write("Hi".toByteArray())
writer.flush()
print(reader.nextLine())
}

It looks like you're missing newlines in both messages, so the calls to reader.nextLine() never return, because the 'next line' hasn't been completely received. When using nextLine() you need to think of the communication protocol as "newline delimited messages".
Note that this also means you can't have newlines in your messages. If you need more complex messages you may want to consider base 64 encoding the message content.
Working with low level socket protocols can be very challenging, using a higher level protocol like HTTP adds complexity but can also be simpler in some ways.
If you need to use sockets you're likely better off reading a byte at a time and implementing a state machine to process bytes to output, though this will be much more complex than calling nextLine().
The following code works for me:
Server:
import java.net.ServerSocket
import java.util.*
fun main() {
val server = ServerSocket(9000)
val client = server.accept()
val reader = Scanner(client.getInputStream())
val writer = client.getOutputStream()
print(reader.nextLine())
writer.write("Yes\n".toByteArray()) // <<<<<< Added \n
writer.flush()
}
Client:
import java.net.Socket
import java.util.*
fun main() {
val client = Socket("127.0.0.1", 9000) // << Also changed to localhost IP.
var writer = client.getOutputStream()
var reader = Scanner(client.getInputStream())
writer.write("Hi\n".toByteArray()) // <<<<< Added \n
writer.flush()
print(reader.nextLine())
}

Related

TCP/IP Client in Kotlin that does not wait forever for server message

I have simple TCP/IP client code in Kotlin below.
This code works.
The client opens the socket and loops forever, first sending a message to the server, and then waiting forever for a response form the server.
I know this code isn’t great, looping forever, keeping the socket connection open etc., but it is just for test purposes right now.
fun tcp_client() {
thread {
val client1 = Socket(SERVER_IP_ADDRESS, SERVER_IP_PORT)
val output1 = PrintWriter(client1.getOutputStream(), true)
val input1 = BufferedReader(InputStreamReader(client1.inputStream))
while (true) {
output1.println(str_user_text)
str_rcvd_data = input1.readLine()
}
}
client1.close()
}
The line:
str_rcvd_data = input1.readLine()
waits forever for a server response.
My question: Is it possible to modify this code so that the client does NOT wait forvever for a server response? Something like this:
If (server data received) {
// process the data
} else {
// do something else for now and check again in a short while
}
Thanks in advance for any suggestions
Garrett
I eventually worked this out - I am not sure how 'correct' this solution is, but it works for me:
Connecting to the server....
My old code would hang if it couldn't connect, because the call to Socket() with the IP address and Port is a Blocking call - i.e.e wait forever:
val client1 = Socket(SERVER_IP_ADDRESS, SERVER_IP_PORT)
So I replaced the code with this:
try {
client1 = Socket()
client1.connect(InetSocketAddress(SERVER_IP_ADDRESS, SERVER_IP_PORT), 3000)
output1 = DataOutputStream (client1.getOutputStream())
input1 = DataInputStream (client1.getInputStream())
} catch (ex : Exception) {
// do something
} finally {
// do something
}
This isn't perfect, but it works.
For reading the data, my old code called readline() which is blocking:
str_rcvd_data = input1.readLine()
Now, my code first checks if there is any data and then grabs each byte
iRxDataAvailable = input1.available()
while (iRxDataAvailable > 0)
{
iRxDataAvailable--
// Take a copy of the received byte
byRcvdByte = input1.readByte()
// Do something with this data byte...
}
Finally, to send data to the server, the data is placed in a byte array, and then:
output1.write(byArray)

How to make the application self-recover from Reactor Kafka's RetriableCommitFailedException caused by request timeout?

I have a Kafka processor that is defined like this.
import org.apache.kafka.clients.consumer.ConsumerConfig
import org.apache.kafka.common.serialization.StringDeserializer
import org.slf4j.LoggerFactory
import org.springframework.context.annotation.Bean
import org.springframework.stereotype.Component
import reactor.core.publisher.Mono
import reactor.core.scheduler.Schedulers
import reactor.kafka.receiver.KafkaReceiver
import reactor.kafka.receiver.ReceiverOptions
import reactor.kafka.receiver.ReceiverRecord
import reactor.kotlin.core.publisher.toMono
import reactor.util.retry.Retry
import java.time.Duration
import java.util.*
#Component
class KafkaProcessor {
private val logger = LoggerFactory.getLogger(javaClass)
private val consumerProps = hashMapOf(
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG to StringDeserializer::javaClass,
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG to StringDeserializer::javaClass,
ConsumerConfig.GROUP_ID_CONFIG to "groupId",
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to "earliest",
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG to "localhost:9092"
)
private val receiverOptions = ReceiverOptions.create<String, String>(consumerProps)
.subscription(Collections.singleton("some-topic"))
.commitInterval(Duration.ofSeconds(1))
.commitBatchSize(1000)
.maxCommitAttempts(1)
private val kafkaReceiver: KafkaReceiver<String, String> = KafkaReceiver.create(receiverOptions)
#Bean
fun processKafkaMessages(): Unit {
kafkaReceiver.receive()
.groupBy { m -> m.receiverOffset().topicPartition() }
.flatMap { partitionFlux ->
partitionFlux.publishOn(Schedulers.elastic())
.concatMap { receiverRecord ->
processRecord(receiverRecord)
.map { it.receiverOffset().acknowledge() }
}
}
.retryWhen(
Retry.backoff(3, Duration.ofSeconds(1))
.maxBackoff(Duration.ofSeconds(3))
.doBeforeRetry { rs ->
logger.warn("Retrying: ${rs.totalRetries() + 1}/3 due to ${rs.failure()}")
}
.onRetryExhaustedThrow { _, u ->
logger.error("All ${u.totalRetries() + 1} attempts failed with the last exception: ${u.failure()}")
u.failure()
}
)
.subscribe()
}
private fun processRecord(record: ReceiverRecord<String, String>): Mono<ReceiverRecord<String, String>> {
return record.toMono()
}
}
Sometimes, I got this error.
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.
Caused by: org.apache.kafka.common.errors.TimeoutException: The request timed out.
The first retry looks like this.
Retrying: 1/3 due to org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets
The second and third look like this.
Retrying: 2/3 due to reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
Retrying: 3/3 due to reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
And once all the 3 retries are exhausted, the message looks like this.
All 4 attempts failed with the last exception: reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
When I do get that error, I need to restart the application in order to reconnect to the Kafka broker and commit the record.
I am aware that by setting maxCommitAttempts to 1 means that once it hits a RetriableCommitFailedException, it won't retry again. I thought that the retryWhen clause I put in the end of the processKafkaMessages() function would do the trick so that the pipeline can recover by itself.
The reason I set the maxCommitAttempts is because it does not have the retry with backoff as discussed here and the default 100 max commit attempts is done within 10ms. So, I thought I should write my own retry logic with a backoff.
The question is, how should I do the retry with backoff for the auto commit correctly? And is it possible to write a unit test for that using EmbeddedKafka?
Language: Kotlin
Reactor Kafka library: io.projectreactor.kafka:reactor-kafka:1.2.2.RELEASE
retryWhen() merely attempts to re-subscribe. Since the Kafka consumer is in error state, it will reject the re-subscription. You need to defer the kafkaReceiver.receive() call, thus:
Flux.defer(() -> kafkaReceiver.receive())
.groupBy { m -> m.receiverOffset().topicPartition() }
// etc
Thus re-subscription will call kafkaReceiver.receive() again and create a new consumer.

How to replace blocking code for reading bytes in Kotlin

I have ktor application which expects file from multipart in code like this:
multipart.forEachPart { part ->
when (part) {
is PartData.FileItem -> {
image = part.streamProvider().readAllBytes()
}
else -> // irrelevant
}
}
The Intellij IDEA marks readAllBytes() as inappropriate blocking call since ktor operates on top of coroutines. How to replace this blocking call to the appropriate one?
Given the reputation of Ktor as a non-blocking, suspending IO framework, I was surprised that apparently for FileItem there is nothing else but the blocking InputStream API to retrieve it. Given that, your only option seems to be delegating to the IO dispatcher:
image = withContext(Dispatchers.IO) { part.streamProvider().readBytes() }

Spring WebFlux Webclient receiving an application/octet-stream file as a Mono

I'm prototyping a small Spring WebFlux application in Kotlin. This application needs to GET a tar archive from a remote REST endpoint and store it locally on disk. Sounds simple.
I first created an integration test that starts the spring server and one other WebFlux server with a mock REST endpoint that serves the tar archive.
The test should go like:
1) app: GET mock-server/archive
2) mock-server: response with status 200 and tar archive in body as type attachment
3) app: block until all bytes received, then untar and use files
The problem I'm having is that when I try and collect the bytes into a ByteArray on the app, it blocks forever.
My mock-server/archive routes to the following function:
fun serveArchive(request: ServerRequest): Mono<ServerResponse> {
val tarFile = FileSystemResource(ARCHIVE_PATH)
assert(tarFile.exists() && tarFile.isFile && tarFile.contentLength() != 0L)
return ServerResponse
.ok()
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.contentLength(tarFile.contentLength())
.header("Content-Disposition", "attachment; filename=\"$ARCHIVE_FNAME\"")
.body(fromResource(tarFile))
}
Then my app calls that with the following:
private fun retrieveArchive {
client.get().uri(ARCHIVE_URL).accept(MediaType.APPLICATION_OCTET_STREAM)
.exchange()
.flatMap { response ->
storeArchive(response.bodyToMono())
}.subscribe()
}
private fun storeArchive(archive: Mono<ByteArrayResource>): Mono<Void> {
val archiveContentBytes = archive.block() // <- this blocks forever
val archiveContents = TarArchiveInputStream(archiveContentBytes.inputStream)
// read archive
}
I've see How to best get a byte array from a ClientResponse from Spring WebClient? and that's why I'm trying to use the ByteArrayResource.
When I step through everything, I see that serveArchive seems to be working (the assert statement says the file I'm passing exists and there are some bytes in it). In retrieveArchive I get a 200 and can see all the appropriate information in the .headers (content-type, content-length all look good). When I get down to storeArchive and try to retrieve the bytes from the Mono using block, it simply blocks forever.
I'm at a complete loss of how to debug something like this.
You just have to return the converted body from the flatMap so it transforms from Mono<T> to T:
client.get().uri(ARCHIVE_URL).accept(MediaType.APPLICATION_OCTET_STREAM)
.exchange()
.flatMap { response ->
response.bodyToMono(ByteArrayResource::class.java)
}
.map { archiveContentBytes ->
archiveContentBytes.inputStream
}
.doOnSuccess { inputStream ->
//here is you code to do anything with the inputStream
val archiveContents = TarArchiveInputStream(inputStream)
}
.subscribe()

Why the producer with unlimited buffered channel doesn't return or return invalid data from channel?

Look at this adopted example from official Kotlin documentation:
package com.example
import kotlinx.coroutines.experimental.channels.produce
import kotlinx.coroutines.experimental.runBlocking
fun main(args: Array<String>) = runBlocking {
val producer = produce {
for (x in 1..5) send(x)
}
for (i in producer) {
println(i)
}
println("Done!")
}
If I run it will print:
1
2
3
4
5
Done!
As you can see here is using unbuffered channel by default. Let's change it to buffered channel:
package com.example
import kotlinx.coroutines.experimental.channels.Channel
import kotlinx.coroutines.experimental.channels.produce
import kotlinx.coroutines.experimental.runBlocking
fun main(args: Array<String>) = runBlocking {
val producer = produce(capacity = Channel.UNLIMITED) {
for (x in 1..5) send(x)
}
for (i in producer) {
println(i)
}
println("Done!")
}
If I run it several times it will print:
1
Done!
or
2
Done!
or just
Done!
I suppose that the producer will put data to the buffered channel and the for loop will read data from it until it can do it (i.e. it will read if data is exists in channel OR channel is not closed). Thus I think the for loop should read all data from buffered channel even it was closed. Am I right?
Can anybody explain me why buffered channel cause such strange behavior of producer? Is it bug?
P.S.
kotlin-stdlib v.1.2.21 (it is last version today)
kotlinx-coroutines-core v.0.22.2 (it is last version today)
The Kotlin team has confirmed that this is a bug and opened an issue for it in response to this question.
The description of the issue mentions that it only appears with the produce builder and provides a workaround, basically the inlined code of that convenience construct:
val producer = Channel<Int>(1)
launch(coroutineContext) {
for(i in 1..2) {
producer.send(i)
println("Sent: $i")
}
producer.close()
}