I have two separate message listener classes written in Kotlin to listen to two separate AWS SQS queues (one is main SQS queue and another is Dead Letter Queue (DLQ)).
The problem is that the message listener for DLQ is unable to consume the messages from its DLQ. This is a tricky problem because my listener code of DLQ is absolutely correct and can consume the messages from other SQS queues but not for this DLQ particularly. I tested its working functionality by replacing DLQ name by the main queue name. There are messages on DLQ as well. By the way both queues are standard queues on AWS (not FIFO).
Here is the code snippet where I'm initializing the SQS connection:-
#PostConstruct
fun init() {
try {
if (enableCalmQueue) {
initializeListeners("$podName-${Util.MAIN_QUEUE}", mainQueueListener)
initializeListeners("$podName-${Util.DLQ}", dlqListener)
LOG.info("Initialized SQS Queue Response Handler")
}
} catch (e: Exception) {
LOG.warn("Failed to Initialize response handler due to ${e.message}")
}
}
private fun initializeListeners(queueName: String, messageListener: MessageListener) {
val connection = getSqsConnectionObject()
val session = connection.createSession(false, SQSSession.UNORDERED_ACKNOWLEDGE)
val consumer = session.createConsumer(session.createQueue(queueName))
consumer.messageListener = messageListener
connection.start()
LOG.info("Initialized $messageListener for $queueName")
}
private fun getSqsConnectionObject(): SQSConnection {
val connectionFactory = SQSConnectionFactory(
ProviderConfiguration(),
AmazonSQSClientBuilder.defaultClient()
)
return connectionFactory.createConnection()
}
DLQ Listener:-
#Named("dlqListener")
class DeadLetterQueueListener : MessageListener {
#Value("\${pod_name}")
private val podName: String = "dev"
#Inject
private lateinit var tradeStatusService: TradeStatusService
companion object {
private val LOG = LoggerFactory.getLogger(DeadLetterQueueListener::class.java)
}
override fun onMessage(message: Message?) {
try {
if (message is TextMessage) {
LOG.info("Received message ${message.text} in Dead Letter Queue Listener " +
"with jmsMessageId: ${message.jmsMessageID}")
if(message.getStringProperty(Util.QUEUE_NAME) == "$podName-${Util.MAIN_QUEUE}") {
message.acknowledge()
val jsonMessage = message.text
val objectMapper = ObjectMapper()
val externalTradeInputs = objectMapper.readValue(jsonMessage, ExternalTradeInputs::class.java)
val tradeStatusData = TradeStatusData().apply {
this.tradeBookingId = externalTradeInputs.tradeBookingId
this.statusDateTime = Date()
}
}
}
} catch (e: Exception) {
LOG.error("Error processing in Dead Letter Queue Listener with : ${e.message}")
}
}
}
Like I mentioned, this dlqListener works fine if I pass the main queue name but not DLQ queue name.
I tried to debug a lot but could not find anything concrete. Both main and DLQ exists in AWS. I suspect something some wrong in the way DLQ has been configured but to my best I have verified but nothing found any issues over there.
I would really appreciate if someone can give me some reference or hints about its root cause.
Related
emphasized textI am trying to use Kotlin Flow to process some data asynchronously and in parallel, and stream the responses to the client as they occur, as opposed to waiting until all the jobs are complete.
After unsuccessfully trying to just send the flow itself to the response, like this: call.respond(HttpStatusCode.OK, flow.toList())
... I tinkered for hours trying to figure it out, and came up with the following. Is this correct? It seems there should be a more idiomatic way of sending a Flow<MyData> as a response, like one can with a Flux<MyData> in Spring Boot.
Also, it seems that using the below method does not cancel the Flow when the HTTP request is cancelled, so how would one cancel it in Ktor?
data class MyData(val number: Int)
class MyService {
fun updateAllJobs(): Flow<MyData> =
flow {
buildList { repeat(10) { add(MyData(Random.nextInt())) } }
// Docs recommend using `onEach` to "delay" elements.
// However, if I delay here instead of in `map`, all elements are held
// and emitted at once at the very end of the cumulative delay.
// .onEach { delay(500) }
.map {
// I want to emit elements in a "stream" as each is computed.
delay(500)
emit(it)
}
}
}
fun Route.jobRouter() {
val service: MyService by inject() // injected with Koin
put("/jobs") {
val flow = service.updateAllJobs()
// Just using the default Jackson mapper for this example.
val mapper = jsonMapper { }
// `respondOutputStream` seems to be the only way to send a Flow as a stream.
call.respondOutputStream(ContentType.Application.Json, HttpStatusCode.OK) {
flow.collect {
println(it)
// The data does not stream without the newline and `flush()` call.
write((mapper.writeValueAsString(it) + "\n").toByteArray())
flush()
}
}
}
}
The best solution I was able to find (although I don't like it) is to use respondBytesWriter to write data to a response body channel. In the handler, a new job to collect the flow is launched to be able to cancel it if the channel is closed for writing (HTTP request is canceled):
fun Route.jobRouter(service: MyService) {
put("/jobs") {
val flow = service.updateAllJobs()
val mapper = jsonMapper {}
call.respondBytesWriter(contentType = ContentType.Application.Json) {
val job = launch {
flow.collect {
println(it)
try {
writeStringUtf8(mapper.writeValueAsString(it))
flush()
} catch (_: ChannelWriteException) {
cancel()
}
}
}
job.join()
}
}
}
I am building a testing library that abstracts away some 3rd party resources such as Google PubSub.
The issue is when I create a topic and a subscription, messages arrive constantly. Example output:
ack_id: "projects/test-project-123/subscriptions/my-subscription:2"
message { data: "My message!-1" message_id: "2" publish_time {
seconds: 1614597484 } }
ack_id: "projects/test-project-123/subscriptions/my-subscription:4"
message { data: "My message!-1" message_id: "2" publish_time {
seconds: 1614597484 } }
Curiously enough, publish time is the same.
The code that pulls the messages:
fun poll(size: Int, subscriptionId: String): List<String> {
val subscriberStubSettings: SubscriberStubSettings = SubscriberStubSettings.newBuilder()
.setTransportChannelProvider(channelProvider)
.setCredentialsProvider(credentialsProvider)
.build()
GrpcSubscriberStub.create(subscriberStubSettings).use { subscriber ->
val pullRequest: PullRequest = PullRequest.newBuilder()
.setMaxMessages(size)
.setSubscription(ProjectSubscriptionName.format(projectId, subscriptionId))
.build()
val pullResponse: PullResponse = subscriber.pullCallable().call(pullRequest)
val acknowledgeRequest = AcknowledgeRequest.newBuilder()
.setSubscription(ProjectSubscriptionName.format(projectId, subscriptionId))
.addAllAckIds(
pullResponse.receivedMessagesList
.stream()
.map { it.ackId }.toList()
).build()
return pullResponse.receivedMessagesList
.map { it.message.data.toStringUtf8() }
.toList()
}
}
I am trying to pull messages one by one from each subscription:
fun purge() {
for (subscription in listSubscriptionIds()) {
var messages = poll(MESSAGE_BATCH_SIZE, subscription)
while (messages.isNotEmpty()) {
messages = poll(MESSAGE_BATCH_SIZE, subscription)
}
}
}
Extra functions:
private val channelProvider: TransportChannelProvider
get() {
return FixedTransportChannelProvider
.create(
GrpcTransportChannel.create(channel())
)
}
private fun channel(): ManagedChannel {
return if (channels.isEmpty()) {
val endpoint = emulator.emulatorEndpoint
val channel = ManagedChannelBuilder
.forTarget(endpoint)
.usePlaintext()
.build()
channels.add(channel)
channel
} else {
channels.first()
}
}
var emulator: PubSubEmulatorContainer = PubSubEmulatorContainer(
DockerImageName.parse("gcr.io/google.com/cloudsdktool/cloud-sdk:latest")
)
How can I overcome it? Is it a bug, or am I doing anything wrong?
Your code should acknowledge that you have received (and, presumably, handled) each message:
Once a message is sent to a subscriber, the subscriber should acknowledge the message. A message is considered outstanding once it has been sent out for delivery and before a subscriber acknowledges it. Pub/Sub will repeatedly attempt to deliver any message that has not been acknowledged. (ref)
Those repeated attempts are what you are seeing here, so it looks like that acknowledgement isn't happening. Try the Acknowledge RPC call.
I have a server that relays between two (different) clients. When the User (first client, through websockets) sends a message, the server needs to repeat this message every X milliseconds to the Device (second client) until a new message is received, or the websocket is closed.
I consume the websocket as a flow, and I've created the following operator:
fun <T> flowEvery(value: T, everMilliSeconds: Long): Flow<T> =
flow {
while (true) {
emit(value)
delay(everMilliSeconds)
}
}.cancellable()
#ExperimentalCoroutinesApi
fun <T> Flow<T>.repeatEvery(mSec: Long): Flow<T> =
this.flatMapLatest {
flowEvery(it, mSec)
}
Problem is, once the socket is closed the last message is kept on being sent for ever.
My call site is:
try {
oscConnections.sendTo(
deviceIdentifier,
incoming.consumeAsFlow().repeatEvery(50).mapNotNull { frame ->
when (frame) {
is Frame.Text -> listOf(frame.readText().toFloat())
else -> null
}
})
} finally {
close(CloseReason(CloseReason.Codes.NORMAL, "Ended"))
}
the incoming channel is closed (onCompletion is called) but the stream sent to sendTo is not. sendTo it self consumes the input stream and send a UDP message for every element it consumes.
How can I force the flow to stop?
By using flatMapLatest or transformLatest you replace the last value of the upstream Flow with a never-ending Flow.
You must stop that Flow somehow and CancellationExceptions are used everywhere in coroutines to signal the cancellation of coroutines. You can wrap your never-ending Flow logic in a coroutineScope to precisely cancel only that scope once the upstream flow has completed.
fun <T> Flow<T>.repeatEvery(delay: Long): Flow<T> =
flow<T> {
try {
coroutineScope {
onCompletion { this#coroutineScope.cancel() }
.transformLatest { value ->
while (true) {
emit(value)
delay(delay)
}
}
.collect(::emit)
}
}
catch (e: CancellationException) {
// done
}
}
PS: .cancellable() doesn't do much in your example. As per documentation Flows built using flow builders like flow { … } are automatically cancellable.
I my application I have two verticles (Standard one and NOT workers), one of which, say VerticleA, produces some messages and the other one, say VerticleB, consumes them. In VerticleB I create several consumers to consume these messages.
class VerticleA : AbstractVerticle() {
val publisher: MessageProducer<String>
fun start(promise: Promise<Void>) {
publisher = vertx.eventBus().sender<String>("address")
.setWriteQueueMaxSize(queueSize)
vertx.setPeriodic(timeout) {
if(publisher.writeQueueFull())
return
getAsyncMessages() { messages ->
messages.forEach { publisher.wirte(it) }
}
}
}
}
class VerticleB : AbstractVerticle() {
val consumers: List<MyConsumers>
override fun start(promise: Promise<Void>) {
// Some initialization
consumers = (1..count).map { createConsumer() }
}
fun createConsumer(): MessageConsumer<String> {
val consumer = vertx.eventBus().consumer<String>("address")
consumer.handler { message ->
consumer.pause()
asyncProcess(message) { consumer.resume() }
}
return consumer
}
}
When I deploy the application with only single instance of VerticleB i.e. with DeploymentOptions.setInstances(1) then everything works fine. But when I set the number instances to anything more than one then the consumers stop processing messages after processing few initial messages.
From logs I can see that each of the consumer consumes couple of messages and then stops consuming. The pause and resume logs are also in pairs i.e. for each pause() call there is a resume() call so the asyncProcess() does calls the callback even in case of error. Also the consumers have not ended.
I read the core manual here https://vertx.io/docs/vertx-core/java/ but I do not see anything that can point to solution of this.
I want to use Channel as a Queue, but I need to clear it periodically. I didn't found clear method for the Channel and I make workaround with Channel.cancel and create new Channel, but it looks bad.
The question is:
How can I implement using Kotlin's channel as a queue with cleaning? Recreating a channel looks not so good...
Simplified context.
I have methods called by an external system: enqueue(id: Int) and cancel() and I don't have access to system that invokes these methods (React native methods in my case).
enqueue(id: Int) - enqueue id into the queue for processing (only one item at time can be processed) and starts processing the queue if is not started before.
cancel() - cancel pending processing but allow finishing current processing for current processing item.
My Processor is a singleton and enqueue(id: Int) can be called multiple times before canceling (to add items into queue) and after (for new processing).
My solution is to use channel as a queue and consume its items as a flow. cancel() will cancel the channel that allow current item processint to finish.
The problem is that after channel.cancel() channel is closed and I need to create new channel that is not so beautiful.
fun main() = runBlocking<Unit> {
val processor = Processor()
repeat(3) { processor.enqueue(it) }
delay(150)
processor.cancelPending()
delay(500)
println("Run processing one more time.")
repeat(3) { processor.enqueue(it) }
delay(500)
}
class Processor : CoroutineScope by CoroutineScope(Dispatchers.Default) {
private var channel = Channel<Int>(50)
private var processJob: Job? = null
fun enqueue(id: Int) {
channel.offer(id)
if (processJob?.isActive == true) return
processJob = launch {
channel.consumeAsFlow().collect { process(it) }
}
}
private suspend fun process(id: Int) {
delay(100)
println("[$id] processed.")
}
fun cancelPending() {
println("Cancel.")
channel.cancel()
channel = Channel(50)
}
}