I've created a consumer to receive messages from a Kafka topic but I'm parsing the message manually, is there a way to parse automatically with a custom deserializer for this single topic consumer?
// I've tried to replace the second "String" with "MyMessage" but it didn't work
// e.g: onMessage(message: ConsumerRecord<String, MyMessage>)
fun onMessage(message: ConsumerRecord<String, String>) {
val myMessage = try {
jacksonObjectMapper.readValue<MyMessage>(message.value())
} catch (ex: Exception) {
logger.error(ex.message)
null
}
myMessage?.let{ // do stuff }
}
The expected result is:
fun onMessage(message: ConsumerRecord<String, MyMessage>) {
// do something with message.value() parameter that will be for sure a MyMessage instance.
}
In case of a parse error can I set a callback to handle this error?
Yes; just implement your own and add it to the consumer configuration value.deserializer.
https://kafka.apache.org/documentation/#consumerconfigs_value.deserializer
If you are using Spring Boot:
https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.integration.spring.kafka.consumer.value-deserializer
If configured this way, it is not managed by Spring; if you want Spring to manage it, add it to the DefaultKafkaProducerFactory via a constructor or setter.
Spring for Apache Kafka provides a JSON deserializer:
https://docs.spring.io/spring-kafka/docs/current/reference/html/#json-serde
Related
I have read the set-based consistency validation blog and I want to validate through a dispatch interceptor. I follow the example, but I use reactive repository and it doesn't really work for me. I have tried both block and not block. with block it throws error, but without block it doesn't execute anything. here is my code.
class SubnetCommandInterceptor : MessageDispatchInterceptor<CommandMessage<*>> {
#Autowired
private lateinit var privateNetworkRepository: PrivateNetworkRepository
override fun handle(messages: List<CommandMessage<*>?>): BiFunction<Int, CommandMessage<*>, CommandMessage<*>> {
return BiFunction<Int, CommandMessage<*>, CommandMessage<*>> { index: Int?, command: CommandMessage<*> ->
if (CreateSubnetCommand::class.simpleName == (command.payloadType.simpleName)){
val interceptCommand = command.payload as CreateSubnetCommand
privateNetworkRepository
.findById(interceptCommand.privateNetworkId)
// ..some validation logic here ex.
// .filter { network -> network.isSubnetOverlap() }
.switchIfEmpty(Mono.error(IllegalArgumentException("Requested subnet is overlap with the previous subnet.")))
// .block() also doesn't work here it throws error
// block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-
}
command
}
}
}
Subscribing to a reactive repository inside a message dispatcher is not really recommended and might lead to weird behavior as underling ThreadLocal (used by Axox) is not adapted to be used in reactive programing
Instead, check out Axon's Reactive Extension and reactive interceptors section.
For example what you might do:
reactiveCommandGateway.registerDispatchInterceptor(
cmdMono -> cmdMono.flatMap(cmd->privateNetworkRepository
.findById(cmd.privateNetworkId))
.switchIfEmpty(
Mono.error(IllegalArgumentException("Requested subnet is overlap with the previous subnet."))
.then(cmdMono)));
I am currently working on a personal project - in which I need my Spring application to take queries from an EMQX (MQTT Server) and query its data for corresponding results, and then push the results to a topic with the query UUID.
This is working - after many hours understanding how the Spring Integration framework works. But I think the way in which the handler is using "block" is incorrect - and not in keeping with the manner in which the Integration Flow should operate. Whilst this works I do want to make sure it is being done properly - out of respect for the work - and to avoid future issues.
The code snippet below should be enough to understand what it is that I'm trying to achieve - and where the potential issue lies.
#Bean
fun mqttInFlow() : Publisher<Message<String>> {
return IntegrationFlows.from(inbound())
.handle<String> { payload, headers ->
val emotionalOutput: EmotionalOutput = gson.fromJson(payload, EmotionalOutput::class.java)
emotionalPrintService.populateEmotionalOutput(emotionalOutput).map {
MessageBuilder.withPayload(gson.toJson(it))
.copyHeaders(headers)
.setHeader(MqttHeaders.TOPIC, "query/" + it.query_uuid).build()
}.block()
}
.channel(outgoingChannel())
.toReactivePublisher()
}
EDIT - Thanks for the advice - here is what I understood to be the potential edit for the Kotlin DSL solution - this is now producing an error - complaining that an output-channel or replyChannel was not available - nothing outside of the this function has been changed.
#Bean
fun newMqttInFlow() =
integrationFlow (inbound()) {
wireTap {
handle<String> { payload, headers ->
gson.fromJson<EmotionalOutput>(payload, EmotionalOutput::class.java).let { emotionalOutput ->
emotionalPrintService.populateEmotionalOutput(emotionalOutput).map { populatedEmotionalOutput ->
MessageBuilder.withPayload(gson.toJson(populatedEmotionalOutput))
.copyHeaders(headers)
.setHeader(MqttHeaders.TOPIC, populatedEmotionalOutput.query_uuid)
}
}
}
}
channel("outgoingChannel")
}
Exception is :
exception is org.springframework.messaging.core.DestinationResolutionException: no output-channel or replyChannel header available
Although I have many years experience with Java - this approach is new - so thank you very much for your assistance. It's appreciated. If the whole class would be useful - I can post that.
EDIT
Here is the Configuration file - which might give a better insight into what might be causing this secondary error -
021-03-28 21:59:48.008 ERROR 84492 --- [T Call: divnrin] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageHandlingException: error occurred in message handler [bean 'mqttOutbound'; defined in: 'class path resource [io/divnr/appserver/configuration/MQTTConfiguration.class]'; from source: 'org.springframework.core.type.classreading.SimpleMethodMetadata#4a9419d7']; nested exception is java.lang.IllegalArgumentException: This default converter can only handle 'byte[]' or 'String' payloads; consider adding a transformer to your flow definition, or provide a BytesMessageMapper, or subclass this converter for reactor.core.publisher.MonoMapFuseable payloads, failedMessage=GenericMessage [payload=MonoMapFuseable, headers={mqtt_receivedRetained=false, mqtt_id=0, mqtt_duplicate=false, id=c5a75283-c0fe-ebac-4168-dabddd989da9, mqtt_receivedTopic=source/d9e50e8f-67e0-4505-7ca2-4d05b1242207, mqtt_receivedQos=0, timestamp=1616961588004}]
at org.springframework.integration.support.utils.IntegrationUtils.wrapInHandlingExceptionIfNecessary(IntegrationUtils.java:192)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:65)
at
The full class is provided here.
#Configuration
#EnableIntegration
#IntegrationComponentScan
class MQTTConfiguration(val emotionalPrintService: EmotionalPrintService,
val gson: Gson,
val applicationConfiguration: ApplicationConfiguration) {
#Bean
fun mqttServiceFactory() : MqttPahoClientFactory {
return DefaultMqttPahoClientFactory().apply {
connectionOptions = MqttConnectOptions().apply {
serverURIs = arrayOf<String>(applicationConfiguration.mqttServerAddress)
}
}
}
#Bean
fun newMqttInFlow() =
integrationFlow (inbound()) {
handle<String> { payload, headers ->
gson.fromJson<EmotionalOutput>(payload, EmotionalOutput::class.java).let { emotionalOutput ->
emotionalPrintService.populateEmotionalOutput(emotionalOutput).map { populatedEmotionalOutput ->
MessageBuilder.withPayload(gson.toJson(populatedEmotionalOutput))
.copyHeaders(headers)
.setHeader(MqttHeaders.TOPIC, populatedEmotionalOutput.query_uuid).build()
}
}
}
channel(outgoingChannel())
}
#Bean
#ServiceActivator(requiresReply = "false", inputChannel = "outgoingChannel")
fun mqttOutbound(): MessageHandler {
val messageHandler = MqttPahoMessageHandler("divnrout", mqttServiceFactory())
messageHandler.setAsync(true)
return messageHandler
}
#Bean
fun outgoingChannel() : FluxMessageChannel {
return FluxMessageChannel()
}
#Bean
fun inbound(): MessageProducerSupport {
return MqttPahoMessageDrivenChannelAdapter("divnrin", mqttServiceFactory(),
"source/" + applicationConfiguration.sourceUuid).apply {
setConverter(DefaultPahoMessageConverter())
setQos(1)
}
}
}
You indeed don't need that block() in the end of your handle(). You just can return the Mono from that emotionalPrintService.populateEmotionalOutput() and the framework will take for you about the proper subscription and back-pressure handling.
What you would need yet is to make that outgoingChannel() as a FluxMessageChannel.
See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/reactive-streams.html#reactive-streams
Plus consider to move your IntegrationFlow solution to the proper Kotlin DSL: https://docs.spring.io/spring-integration/docs/current/reference/html/kotlin-dsl.html#kotlin-dsl
Also: when it is a FluxMessageChannel in the end of flow, there is no reason to worry about a toReactivePublisher() - the FluxMessageChannel is a Publisher<Message<?>> by itself.
UPDATE
The problem is here:
handle<String>( { payload, headers ->
gson.fromJson<EmotionalOutput>(payload, EmotionalOutput::class.java).let { emotionalOutput ->
emotionalPrintService.populateEmotionalOutput(emotionalOutput).map { populatedEmotionalOutput ->
MessageBuilder.withPayload(gson.toJson(populatedEmotionalOutput))
.copyHeaders(headers)
.setHeader(MqttHeaders.TOPIC, populatedEmotionalOutput.query_uuid).build()
}
}
}) { async(true) }
See that async(true) option. Unfortunately in the current version we don't let it to process reactive reply in the reactive manner by default. You have to say that you'd like to be an async at this end-point. So, your Publisher reply and and FluxMessageChannel as an output is going to do the proper trick.
I am testing an event driven architecture in KTOR. My Core logic is held in a class that reacts to different Event types being emitted by a StateFlow. EventGenerators push Events into the StateFlow which are picked up by the Core.
However, when the Core attempts to respond to an ApplicationCall embedded in one of my Events I receive an ResponseAlreadySentException and I'm not sure why this would be the case. This does not happen if I bypass the StateFlow and call the Core class directly from the EventGenerator. I am not responding to ApplicationCalls anywhere else in my code, and have checked with breakpoints that the only .respond line is not being hit multiple times.
MyStateFlow class:
class MyStateFlow {
val state: StateFlow<CoreEvent>
get() = _state
private val _state = MutableStateFlow<CoreEvent>(CoreEvent.NothingEvent)
suspend fun update(event: CoreEvent) {
_state.value = event
}
}
My Core class:
class Core(
myStateFlow: MyStateFlow,
coroutineContext: CoroutineContext = SupervisorJob() + Dispatchers.IO
) {
init {
CoroutineScope(coroutineContext).launch {
myStateFlow.state.collect {
onEvent(it)
}
}
}
suspend fun onEvent(event: CoreEvent) {
when(event) {
is FooEvent {
event.call.respond(HttpStatusCode.OK, "bar")
}
...
}
}
}
One of my EventGenerators is a Route in my KTOR Application class:
get("/foo") {
myStateFlow.update(CoreEvent.FooEvent(call))
}
However, hitting /f00 in my browser returns either an ResponseAlreadySentException or an java.lang.UnsupportedOperationException with message: "Headers can no longer be set because response was already completed". The error response can flip between the two while I'm tinkering with different attempted solutions, but they seem to be saying the same thing: The call has already been responded to before I attempt to call call.respond(...).
If I change my Route instead to call the Core.onEvent() directly, hitting /foo returns "bar" in my browser as is the intended behaviour:
get("/foo") {
core.onEvent(CoreEvent.FooEvent(call))
}
For completeness, my dependency versions are:
implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.4.10"
implementation "io.ktor:ktor-server-netty:1.4.1"
Thank you in advanced for any insight you can offer.
I am trying create Observable, which will retryWhen when the network connection will be established.
I've created subject:
private val retrySubject = PublishSubject.create<Unit>()()
And I am using it like this:
private fun publishNetworkReconnection() {
compositeDisposable?.add(
connectionHelper.observeConnection()
.subscribe {connected: Boolean
if(connected){
retrySubject.onNext(null)
}
}
)
}
Then I am trying to use it in my retryWhen operator:
val disposable =
Flowable.interval(0, UPDATE_INTERVAL, TimeUnit.SECONDS, Schedulers.io())
.onBackpressureDrop()
.flatMapCompletable {
revocationRepository.sync(event.id)
}
.retryWhen { retryHandler -> retryHandler.flatMap({ nothing -> retrySubject.asObservable() }) }
.observeOn(AndroidSchedulers.mainThread())
.subscribe({ }, { Timber.e(it, "Unable to sync blacklist") })
compositeDisposable?.add(disposable)
}
How to use it properly in this case?
I am getting this error in Android Studio:
Type mismatch. Required: Publisher<< out (???..???) >>! Found: Observable<
Void
!>!
The problem is, you are mixing RxJava 1 - Subject.asObservable and RxJava 2 - Flowable.
The signature of retryWhen from RxJava 2 is:
Flowable<T> retryWhen(Function<? super Flowable<Throwable>,? extends Publisher<?>> handler)
so lambda inside retryWhen should return something that extends Publisher. Instead, you are returning Subject<Unit> which is converted to Observable<Void>, because apparently your Subject is from RxJava 1. And - because of RxJava version difference - obviously it doesn't extend Publisher.
So error message is correct, retryWhen expects Publisher from RxJava 2, but you give Obervable from RxJava 1. You didn't notice different packages, because they are not included in message.
The main issue is mixing code from RxJava 1 and RxJava 2 which is never good.
I am building an API using Ratpack and Groovy. The POST API is always giving:
405-Method not Found Error
This is a snippet from POST Endpoint Handler. In this code, promiseSingle, then, observe, map, doOnNext, doOnError, etc.
RxJAVA functions are not working. Is there any reason why RxJava methods are not working?
saveJsonAsData(context, id)
.promiseSingle()
.then { Data updateddata ->
context.response.headers
.add(HttpHeaderNames.LOCATION, "/api/save/${updateddata.id}/${updateddata.value}")
context.response.status(HttpResponseStatus.CREATED.code())
.send()
}
}
protected Observable<Data> saveJsonAsData(GroovyContext context, String id) {
context.request.body.observe()
.map { TypedData typedData -> extractData(context, typedData) }
.doOnNext { Data data ->
data.id = id
validatorWrapper.validate(data)
}
.flatMap(data.&save as Func1)
.doOnError { Throwable throwable -> log.error("Error saving data", throwable) }
}
The issue is not so much with Rx as it is with the usage of the Context.
You should try to keep the response handling logic within your Handler, that is don't pass the Context around, rather get the objects you need and pass them to your services.
As an example
path('myendpoint') { MyRxService service ->
byMethod {
get {
// do something when request is GET
}
post {
request.body.map { typedData ->
extractItem(typeData) // extract your item from the request first
}.flatMap { item ->
service.saveJsonAsItemLocation(item).promiseSingle() // then once it's extracted pass the item to your "saveJsonAsItemLocation" method
}.then { ItemLocationStore updatedItem ->
response.headers.add(HttpHeaderNames.LOCATION, "/itemloc/v1/save/${updatedItem.tcin}/${updatedItem.store}")
context.response.status(HttpResponseStatus.CREATED.code()).send()
}
}
}
}
My guess is that you have something like this:
get {
// get stuff
}
post {
// post stuff
}
The reason this doesn't work is that Ratpack doesn't use Routing Table for handling incoming requests, instead it uses chain delegation. The get {} binds to root path and GET http method and post {} binds to root path and POST http method. Because get {} matches the path, Ratpack considers the handler matched and since the handler is for GET it considers it a 405.
There are chain methods available that binds regardless of HTTP Method such as all {} and path {}. Chain#all will handle all paths and methods where as Chain#path(String) matches against specific path.
Hope this helps.