Kotlin Kafka consumer - kotlin

I am new to kotlin and kafka. I am trying to read message from a kafka consumer and then process it later in kotlin.
I am able to print the message consumed from kafka topic.
However, I am not able to add this message into a list and process it later.
How to add the message from kafka topic to a list to be processed later?
Below is my kafka consumer's consume method.
KafkaConsumer.kt
fun consume(handler: (value: String) -> Unit) = thread(start = true) {
keepGoing = true
consumer.use { consumer ->
while (keepGoing) {
consumer.poll(Duration.ofMillis(500))?.forEach {
println(it?.value() ?: "no value")
handler(it?.value() ?: "empty_message")
}
}
}
}
I am invoking the consume() as shown below
fun main(){
<read config>
val kafkaConsumer = KafkaConsumer(config.get(<topicname>))
kafkaConsumer.consume {
println(it)
}
}
PS: I tried to append the to a mutable list in the main() which did not work.

I'm not sure if it's a formatting issue or your actual code, but Kotlin lambdas are defined by { } and you call consume() with 2 pairs of braces:
kafkaConsumer.consume { it -> {
println(it)
} }
This is effectively passing a function that takes one argument and returns a function. That returned function takes itself an argument and prints it.
This should not even compile given the signature of consume().
What you probably want is this instead:
kafkaConsumer.consume { it ->
println(it)
}
Note that it is also the implicit argument of lambda expressions, so you don't even need to specify it at all:
kafkaConsumer.consume {
println(it)
}

Related

How to concat multiple Kotlin Flows sequencially?

I'm looking for a way to subscribe to multiple Kotlin Flows sequencially, something similar to rxjs's concat operator. Next Flow should only be subscribed once previous one is completed.
Example:
val flow1 = flowOf(0,1,2).onEach { delay(10) }
val flow2 = flowOf(3,4,5).onEach { delay(10) }
runBlocking{
listOf(flow1,flow2)
.merge()
.onEach { println(it) }
.collect()
}
-> prints 0,3,1,4,2,5 // Flow order is not preserved
I imagine a solution could be to replace merge() with concat(), but this operator sadly doesn't exist in Kotlin Flows
val flow1 = flowOf(0,1,2).onEach { delay(10) }
val flow2 = flowOf(3,4,5).onEach { delay(10) }
runBlocking{
listOf(flow1,flow2)
.concat()
.onEach { println(it) }
.collect()
}
-> prints 0,1,2,3,4,5 // Flow order is now preserved
You're looking for flattenConcat.
It has a Flow<Flow<T>> receiver, but you could easily make a Flow out of your list of flows, for instance using list.asFlow(). Or you could create a Flow directly instead of a list by using flowOf instead of listOf.

Kotlin add custom method to stream chaining

I need to add a custom method (which is a Consumer) to the dot chaining in stream api, i not sure how to do it, following is my code.
If that is not possible, is there anyway to do it with other operation? Maybe like with .map or something else?
fun main(args: Array<String>) {
var countries: List<String> = listOf("India", "Germany", "Japan")
var firstCountry = countries.stream()
.filter{it == "Germany"}
.performOperation{} //not sure what to do here
.findFirst()
println(firstCountry)
}
fun performOperation(country: String) {
if(country.length > 3) {
throw InvalidLengthException("Error")
}
//do some operation, won't return any value
doCustomOperation(country)
}
You may already be aware that when it comes to steams there are two types of operations, one is your map, filter etc. known as intermediate opeartion and others are terminal operations such as forEach. You said your custom operation wont return any value, hence making it a terminal operation. moreover it seems to me that you want to perform same operation for all the elements, basically a forEach. for this you can define an extension function on Stream as
fun <T> Stream<T>.someOperation(operation: (T) -> Unit){
this.forEach { operation(it) }
}
There is two ways to do what you want.
var firstCountry = countries.stream()
.filter{it == "Germany"}
.also(::performOperation)
.findFirst()
The :: is a function reference and is basically the same as .also { performOperation(it)}
The second one would be to make your own extension method on list. I wouldn't recommend it until you understand kotlin lambdas and extension methods
fun Stream<String>.performOperation(): Stream<String> {
for(country in this) {
if(country.length > 3) {
throw InvalidLengthException("Error")
}
doCustomOperation(country)
}
return this
}
You would just call that one like .performOperation() where you have the .performOperation{}

How to execute a program with Kotlin and Arrow

I'm trying to learn a bit of Functional Programming using Kotlin and Arrow and in this way I've already read some blogposts like the following one: https://jorgecastillo.dev/kotlin-fp-1-monad-stack, which is good, I've understand the main idea, but when creating a program, I can't figure out how to run it.
Let me be more explicit:
I have the following piece of code:
typealias EitherIO<A, B> = EitherT<ForIO, A, B>
sealed class UserError(
val message: String,
val status: Int
) {
object AuthenticationError : UserError(HttpStatus.UNAUTHORIZED.reasonPhrase, HttpStatus.UNAUTHORIZED.value())
object UserNotFound : UserError(HttpStatus.NOT_FOUND.reasonPhrase, HttpStatus.NOT_FOUND.value())
object InternalServerError : UserError(HttpStatus.INTERNAL_SERVER_ERROR.reasonPhrase, HttpStatus.INTERNAL_SERVER_ERROR.value())
}
#Component
class UserAdapter(
private val myAccountClient: MyAccountClient
) {
#Lazy
#Inject
lateinit var subscriberRepository: SubscriberRepository
fun getDomainUser(ssoId: Long): EitherIO<UserError, User?> {
val io = IO.fx {
val userResource = getUserResourcesBySsoId(ssoId, myAccountClient).bind()
userResource.fold(
{ error -> Either.Left(error) },
{ success ->
Either.right(composeDomainUserWithSubscribers(success, getSubscribersForUserResource(success, subscriberRepository).bind()))
})
}
return EitherIO(io)
}
fun composeDomainUserWithSubscribers(userResource: UserResource, subscribers: Option<Subscribers>): User? {
return subscribers.map { userResource.toDomainUser(it) }.orNull()
}
}
private fun getSubscribersForUserResource(userResource: UserResource, subscriberRepository: SubscriberRepository): IO<Option<Subscribers>> {
return IO {
val msisdnList = userResource.getMsisdnList()
Option.invoke(subscriberRepository.findAllByMsisdnInAndDeletedIsFalse(msisdnList).associateBy(Subscriber::msisdn))
}
}
private fun getUserResourcesBySsoId(ssoId: Long, myAccountClient: MyAccountClient): IO<Either<UserError, UserResource>> {
return IO {
val response = myAccountClient.getUserBySsoId(ssoId)
if (response.isSuccessful) {
val userResource = JacksonUtils.fromJsonToObject(response.body()?.string()!!, UserResource::class.java)
Either.Right(userResource)
} else {
when (response.code()) {
401 -> Either.Left(UserError.AuthenticationError)
404 -> Either.Left(UserError.UserNotFound)
else -> Either.Left(UserError.InternalServerError)
}
}
}.handleError { Either.Left(UserError.InternalServerError) }
}
which, as you can see is accumulating some results into an IO monad. I should run this program using unsafeRunSync() from arrow, but on javadoc it's stated the following: **NOTE** this function is intended for testing, it should never appear in your mainline production code!.
I should mention that I know about unsafeRunAsync, but in my case I want to be synchronous.
Thanks!
Instead of running unsafeRunSync, you should favor unsafeRunAsync.
If you have myFun(): IO<A> and want to run this, then you call myFun().unsafeRunAsync(cb) where cb: (Either<Throwable, A>) -> Unit.
For instance, if your function returns IO<List<Int>> then you can call
myFun().unsafeRunAsync { /* it (Either<Throwable, List<Int>>) -> */
it.fold(
{ Log.e("Foo", "Error! $it") },
{ println(it) })
}
This will run the program contained in the IO asynchronously and pass the result safely to the callback, which will log an error if the IO threw, and otherwise it will print the list of integers.
You should avoid unsafeRunSync for a number of reasons, discussed here. It's blocking, it can cause crashes, it can cause deadlocks, and it can halt your application.
If you really want to run your IO as a blocking computation, then you can precede this with attempt() to have your IO<A> become an IO<Either<Throwable, A>> similar to the unsafeRunAsync callback parameter. At least then you won't crash.
But unsafeRunAsync is preferred. Also, make sure your callback passed to unsafeRunAsync won't throw any errors, at it's assumed it won't. Docs.

Observe many times from same Observable (RxAndroidBle)

I'm using the RxAndroidBle library with RxJava2 to read from a BLE Characteristic. I think this question is just an RxJava question, but including the detail that I'm using RxAndroidBle in case that is useful.
I get connection, and then use it to call readCharacteristic(), which itself returns a Single<ByteArray>. At this point, I don't just want to just get the one ByteArray though. I need to read from this characteristic several times, because the BLE device is set up to let me get a small file back, and characteristics can only send 20 bytes back at a time, hence my need to read repeatedly.
Is it possible to modify this code so that the switchMap() below returns an Observable that will emit many ByteArrays, instead of just the single one?
I'm new to RxJava.
val connection: Observable<RxBleConnection> = selectedDevice.record.bleDevice.establishConnection(false, Timeout(30, TimeUnit.SECONDS))
return connection
.subscribeOn(Schedulers.io())
.switchMap {
// I want to get an Observable that can read multiple times here.
it.readCharacteristic(serverCertCharacteristicUUID).toObservable()
}
.doOnNext {
Timber.e("Got Certificate bytes")
}
.map {
String(it as ByteArray)
}
.doOnNext {
Timber.e("Got certificate: $it")
}
.singleOrError()
To repeat a read multiple times until a specific value is emitted one needs to change this part:
// I want to get an Observable that can read multiple times here.
it.readCharacteristic(serverCertCharacteristicUUID).toObservable()
to something like what was suggested by the RxJava author in the first answer that google gives for phrase rxjava single repeat:
// this will repeat until a `checkRepeatIf` returns false
Observable.defer {
val successValue = AtomicReference<ByteArray>()
connection.readCharacteristic(serverCertCharacteristicUUID)
.doOnSuccess { successValue.lazySet(it) }
.repeatWhen { completes -> completes.takeWhile { checkRepeatIf(successValue.get()) } }
}
I was able to get this working by sending a signal to stop both the connectionObservable, and the read on the Bluetooth characteristic. Of note is that you need to call toObservable() AFTER repeat() or this doesn't work, although I don't know why exactly.
override fun readMultipartCharacteristic(macAddress: String): Single<String> {
val CERTIFICATE_TERMINATOR = 0x30.toByte()
val device = bluetoothService.getBleDevice(macAddress)
if (connectionObservable == null || !device.connectionState.equals(RxBleConnection.RxBleConnectionState.CONNECTED)) {
connectionObservable = device.establishConnection(false, Timeout(30, TimeUnit.SECONDS))
}
val stop: PublishSubject<Unit> = PublishSubject.create()
return connectionObservable!!
.subscribeOn(Schedulers.io())
.takeUntil(stop)
.switchMap {
it.readCharacteristic(UUID("my-uuid"))
.repeat()
.toObservable()
.takeUntil(stop)
}
.collectInto(ByteArrayOutputStream(), { buffer, byteArray ->
// Watch for the signal of the end of the stream
if (byteArray.size == 1 && byteArray.get(0).equals(CERTIFICATE_TERMINATOR)) {
stop.onComplete()
} else {
buffer.write(byteArray)
}
})
.map {
String(it.toByteArray())
}
}
You can use the notification to buffer your data.
device.establishConnection(false)
.flatMap(rxBleConnection -> rxBleConnection.setupNotification(characteristicUuid))
.flatMap(notificationObservable -> notificationObservable) // <-- Notification has been set up, now observe value changes.
.subscribe(
bytes -> {
// Given characteristic has been changes, here is the value.
},
throwable -> {
// Handle an error here.
}
);

Vert.x Reactive Kafka client: chaining not working when writing?

I am using io.vertx.reactivex.kafka.client.producer.KafkaProducer client. The client has a
rxWrite function which returns Single<RecordMetadata>. However I need to log error if any, during write operation. It apparently is not getting executed.
I have written following working example.
test(): Function to test the chaining and logging
fun test(): Single<Int> {
val data = Single.just(ArrayList<String>().apply {
add("Hello")
add("World")
})
data.flattenAsObservable<String> { list -> list }
.flatMap { advertiser ->
//does not work with writeKafka
writeError(advertiser).toObservable().doOnError({ println("Error $data") })
}
.subscribe({ record -> println(record) }, { e -> println("Error2 $e") })
return data.map { it.size }
}
writeKafka: Writes the given given string into Kafka and returns Single
fun writeKafka(param: String): Single<RecordMetadata> {
//null topic to produce IllegalArgumentException()
val record = KafkaProducerRecord.create(null, UUID.randomUUID().toString(), param)
return kafkaProducer.rxWrite(record)
}
writeError: Always return a single with error of same type
fun writeError(param: String): Single<RecordMetadata> {
return Single.error<RecordMetadata>(IllegalArgumentException())
}
So when I call writeKafka It only prints Error2 but if I use writeError it prints both Error and Error2. Looks like the single returned by writeKafka is still waiting for result, but then why even Error2 is printed?
I am pretty newbie in RxJava2, could somebody point out any error in that?
It is important to read and post the stacktrace of errors so that the problem can be isolated.
In this case, looks like you get the IllegalArgumentException from create and you don't get any Single because the relevant Kafka class throws it. return kafkaProducer.rxWrite(record) never executes at all and you practically crash the flatMap. doOnError never gets into play hence only the "Error2" is printed.