Make na abstraction to kafka consumer using ktor and kotlin - kotlin

Question

I'm creating a abstraction for consumer and producer Kafka, to avoid duplicate code all the time. So i created a lib using kotlin and gradle, named "kafka-commons", and put the following code:
For Kafka producer:
fun producer(
bootstrapServers: String,
idempotence: Boolean,
acks: Acks,
retries: Int,
requestPerConnection: Int,
compression: Compression,
linger: Int,
batchSize: BatchSize
): KafkaProducer<String, Any> {
val prop: HashMap<String, Any> = HashMap()
prop[BOOTSTRAP_SERVERS_CONFIG] = bootstrapServers
prop[KEY_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java.name
prop[VALUE_SERIALIZER_CLASS_CONFIG] = StringSerializer::class.java.name
prop[ENABLE_IDEMPOTENCE_CONFIG] = idempotence
prop[ACKS_CONFIG] = acks.value
prop[RETRIES_CONFIG] = retries
prop[MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION] = requestPerConnection
prop[COMPRESSION_TYPE_CONFIG] = compression.value
prop[LINGER_MS_CONFIG] = linger
prop[BATCH_SIZE_CONFIG] = batchSize.value
return KafkaProducer(prop)
}
suspend inline fun <reified K : Any, reified V : Any> KafkaProducer<K, V>.dispatch(record: ProducerRecord<K, V>) =
suspendCoroutine<RecordMetadata> { continuation ->
val callback = Callback { metadata, exception ->
if (metadata == null) {
continuation.resumeWithException(exception!!)
} else {
continuation.resume(metadata)
}
}
this.send(record, callback)
}
So, i created a default Command object with the following structure
data class Command(
val id: UUID,
val status: CommandStatus,
val message: Any
)
id: create a unique ID
status: crete a message status (can be: Open /Processing / Closed / Error)
message: an object from http reques (for example: If have an Post
For example: if have a "insert user: POST" with body :
{ "id": 1, "name" : "John", "lastName" : "Wick" }
so message will be this object, and so on.
And for create this command, i made this function:
suspend fun creatCommand(
topicName: String,
id: UUID,
commandStatus: CommandStatus,
request: Any,
bootstrapServers: String,
idempotence: Boolean,
acks: Acks,
retries: Int,
requestPerConnection: Int,
compression: Compression,
linger: Int,
batchSize: BatchSize
): Unit {
val producer = producer(
bootstrapServers,
idempotence,
acks,
retries,
requestPerConnection,
compression,
linger,
batchSize)
val command = toCommand(processStarted(id, commandStatus, request))
val record = ProducerRecord<String, Any>(topicName, id.toString(), command)
coroutineScope { launch { producer.dispatch(record) } }
}
So, I created other APIs and just call this function to create a producer that sends a command to kafka. Like:
fun Route.user(service: Service) =
route("/api/access") {
post("/test") {
call.respond(service.command(call.receive()))
}
}
>>>>>> other class <<<<<<<<
classService () {
fun command( all parameters) { creatCommand(all parameters)}
}
Sor far so good. All works great.
Now my problemas begin. I'm trying to create a consumer.
First i made this:
fun consumer(
bootstrapServers: String,
group: String,
autoCommit: Boolean,
offsetBehaviour: OffsetBehaviour,
pollMax: Int
): KafkaConsumer<String, Any> {
val prop: HashMap<String, Any> = HashMap()
prop[BOOTSTRAP_SERVERS_CONFIG] = bootstrapServers
prop[KEY_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java.name
prop[VALUE_DESERIALIZER_CLASS_CONFIG] = StringDeserializer::class.java.name
prop[GROUP_ID_CONFIG] = group
prop[AUTO_OFFSET_RESET_CONFIG] = offsetBehaviour
prop[ENABLE_AUTO_COMMIT_CONFIG] = autoCommit
prop[MAX_POLL_RECORDS_CONFIG] = pollMax
return KafkaConsumer(prop)
}
And after:
fun<T> recordingCommand(
command: Class<T>,
topic: String,
bootstrapServers: String,
group: String,
autoCommit: Boolean,
offsetBehaviour: OffsetBehaviour,
pollMax: Int
) {
val consumer = consumer(bootstrapServers, group, autoCommit, offsetBehaviour, pollMax)
consumer.subscribe(mutableListOf(topic))
while (true) {
val records = consumer.poll(Duration.ofMillis(100))
for (record in records) {
val om = ObjectMaper
om.readvalue(record.value(), command::class.java)
>>>> I GOT LOST HERE <<<<<<
}
}
}
What i need: creating a abstract consumer that record all data inside a Command.message() (only the message) in a database.
For example, i need to record the user above (id 1, john wick ) into a postgresql database.
So if i had a service with insert method, i can call it, passing insertmethod like:
service.insert(recordingCommand(all parameters)).
Anyone can help me with that?

Answer 1

If I understood correctly, you are having issues with JSON mapping into objects
val mapper = ObjectMaper
while (true) {
val records = consumer.poll(Duration.ofMillis(100))
for (record in records) {
val cmd = mapper.readvalue(record.value(), command::class.java)
// do things with cmd
}
}
Note: Kafka has its own JSON to POJO deserializer, and if you want to send data to a database, Kafka Connect is generally more fault-tolerant than your simple consume loop.