Exponential backoff in Kotlin with RxJava2 - kotlin

I am trying to do an exponential retry for a request, so that if the request fails (i.e. your internet is down) the app retries endlessly until it works (for the amount of time the app is in the foreground)
I tried with this solution
public class RetryWithDelay implements Function<Observable<? extends Throwable>, Observable<?>> {
private final int maxRetries;
private final int retryDelayMillis;
private int retryCount;
public RetryWithDelay(final int maxRetries, final int retryDelayMillis) {
this.maxRetries = maxRetries;
this.retryDelayMillis = retryDelayMillis;
this.retryCount = 0;
}
#Override
public Observable<?> apply(final Observable<? extends Throwable> attempts) {
return attempts
.flatMap(new Function<Throwable, Observable<?>>() {
#Override
public Observable<?> apply(final Throwable throwable) {
if (++retryCount < maxRetries) {
// When this Observable calls onNext, the original
// Observable will be retried (i.e. re-subscribed).
return Observable.timer(retryDelayMillis,
TimeUnit.MILLISECONDS);
}
// Max retries hit. Just pass the error along.
return Observable.error(throwable);
}
});
}
}
But when I try to convert this to kotlin it says Function only takes one generic parameter.

I copy-pasted the Java code into IntelliJ and it did half the work for me:
import java.util.concurrent.TimeUnit
import io.reactivex.functions.Function
import io.reactivex.*
class RetryWithDelay(private val maxRetries: Int, private val retryDelayMillis: Long) : Function<Observable<Throwable>, Observable<Long>> {
override fun apply(attempts: Observable<Throwable>): Observable<Long> {
return attempts
.flatMap(object : Function<Throwable, Observable<Long>> {
private var retryCount: Int = 0
override fun apply(throwable: Throwable): Observable<Long> {
return if (++retryCount < maxRetries) {
// When this Observable calls onNext, the original
// Observable will be retried (i.e. re-subscribed).
Observable.timer(retryDelayMillis,
TimeUnit.MILLISECONDS)
} else Observable.error<Long>(throwable)
// Max retries hit. Just pass the error along.
}
})
}
}
Note that the retryCount has been moved into the inner flatMap so that it is not shared between multiple Observers.

Related

How to test ApplicationEvent in Spring Integration Flow

In my Spring project(WebFlux/Kotlin Coroutines/Java 17), I defined a bean like this.
#Bean
fun sftpInboundFlow(): IntegrationFlow {
return IntegrationFlows
.from(
Sftp.inboundAdapter(sftpSessionFactory())
.preserveTimestamp(true)
.deleteRemoteFiles(true) // delete files after transfer is done successfully
.remoteDirectory(sftpProperties.remoteDirectory)
.regexFilter(".*\\.csv$")
// local settings
.localFilenameExpression("#this.toUpperCase() + '.csv'")
.autoCreateLocalDirectory(true)
.localDirectory(File("./sftp-inbound"))
) { e: SourcePollingChannelAdapterSpec ->
e.id("sftpInboundAdapter")
.autoStartup(true)
.poller(Pollers.fixedDelay(5000))
}
/* .handle { m: Message<*> ->
run {
val file = m.payload as File
log.debug("payload: ${file}")
applicationEventPublisher.publishEvent(ReceivedEvent(file))
}
}*/
.transform<File, DownloadedEvent> { DownloadedEvent(it) }
.handle(downloadedEventMessageHandler())
.get()
}
#Bean
fun downloadedEventMessageHandler(): ApplicationEventPublishingMessageHandler {
val handler = ApplicationEventPublishingMessageHandler()
handler.setPublishPayload(true)
return handler
}
And write a test for asserting the application event.
#OptIn(ExperimentalCoroutinesApi::class)
#SpringBootTest(
classes = [SftpIntegrationFlowsTestWithEmbeddedSftpServer.TestConfig::class]
)
#TestPropertySource(
properties = [
"sftp.hostname=localhost",
"sftp.port=2222",
"sftp.user=user",
"sftp.privateKey=classpath:META-INF/keys/sftp_rsa",
"sftp.privateKeyPassphrase=password",
"sftp.remoteDirectory=${SftpTestUtils.sftpTestDataDir}",
"logging.level.org.springframework.integration.sftp=TRACE",
"logging.level.org.springframework.integration.file=TRACE",
"logging.level.com.jcraft.jsch=TRACE"
]
)
#RecordApplicationEvents
#TestInstance(TestInstance.Lifecycle.PER_CLASS)
class SftpIntegrationFlowsTestWithEmbeddedSftpServer {
companion object {
private val log = LoggerFactory.getLogger(SftpIntegrationFlowsTestWithEmbeddedSftpServer::class.java)
}
#Configuration
#Import(
value = [
SftpIntegrationFlows::class,
IntegrationConfig::class
]
)
#ImportAutoConfiguration(
value = [
IntegrationAutoConfiguration::class
]
)
#EnableConfigurationProperties(value = [SftpProperties::class])
class TestConfig {
#Bean
fun embeddedSftpServer(sftpProperties: SftpProperties): EmbeddedSftpServer {
val sftpServer = EmbeddedSftpServer()
sftpServer.setPort(sftpProperties.port ?: 22)
//sftpServer.setHomeFolder()
return sftpServer
}
#Bean
fun remoteFileTemplate(sessionFactory: SessionFactory<LsEntry>) = RemoteFileTemplate(sessionFactory)
}
#Autowired
lateinit var uploadGateway: UploadGateway
#Autowired
lateinit var embeddedSftpServer: EmbeddedSftpServer
#Autowired
lateinit var template: RemoteFileTemplate<LsEntry>
#Autowired
lateinit var applicationEvents: ApplicationEvents
#BeforeAll
fun setup() {
embeddedSftpServer.start()
}
#AfterAll
fun teardown() {
embeddedSftpServer.stop()
}
#Test
//#Disabled("application events can not be tracked in this integration tests")
fun `download the processed ach batch files to local directory`() = runTest {
val testFilename = "foo.csv"
SftpTestUtils.createTestFiles(template, testFilename)
eventually(10.seconds) {
// applicationEvents.stream().forEach{ log.debug("published event:$it")}
applicationEvents.stream(DownloadedEvent::class.java).count() shouldBe 1
SftpTestUtils.fileExists(template, testFilename) shouldBe false
SftpTestUtils.cleanUp(template)
}
}
}
It can not catch the application events by ApplicationEvents.
I tried to replace the ApplicationEventPublishingMessageHandler with a constructor autowired ApplicationEventPublisher, it also does not work as expected.
Check the complete test source codes: SftpIntegrationFlowsTestWithEmbeddedSftpServer
Update: The applicationEvents does not work in an async thread, either applying a #Async on the listener method or invoking applicationEvents in a async thread, the application event records did not work as expected.
I'm not familiar with that #RecordApplicationEvents, so I would register an #EventListener(File payload) in the support #Configuration with some async barrier to wait form an event from that scheduled task.
You can turn on a DEBUG logging for org.springframework.integration and Message History to see in logs how your message travels. If there is one at all according to your SFTP state.

Eventbus subscribers not registering fully

My debug output is showing this;
D/EventBus: No subscribers registered for event class com.me.MyEvent
My concern is the index is not working fully and as a result EventBus is using reflection to find the subscriber/s.
The event is published from button click inside a RecyclerView's adapter, and is received by the Fragment containing the RecyclerView. Send and receive work fine. As I said, my concern is reflection is being used.
The index seems to be built correctly. My build.gradle
def eventbus_version = '3.3.1'
implementation "org.greenrobot:eventbus:$eventbus_version"
kapt "org.greenrobot:eventbus-annotation-processor:$eventbus_version"
and
kapt {
arguments {
arg('eventBusIndex', 'com.me.MyEventBusIndex')
}
Here's part of the generated index;
public class MyEventBusIndex implements SubscriberInfoIndex {
private static final Map<Class<?>, SubscriberInfo> SUBSCRIBER_INDEX;
static {
SUBSCRIBER_INDEX = new HashMap<Class<?>, SubscriberInfo>();
putIndex(new SimpleSubscriberInfo(MyFragment.class, true, new SubscriberMethodInfo[] {
new SubscriberMethodInfo("handleEvent", com.me.MyEvent.class,
ThreadMode.POSTING, 0, true),
}));
}
and some fragment code;
override fun onStart() {
super.onStart()
EventBus.builder().addIndex(MyEventBusIndex()).build().register(this)
}
override fun onStop() {
EventBus.getDefault().unregister(this)
super.onStop()
}
and finally a fragment method;
#Subscribe(sticky = true)
fun handleEvent(event: MyEvent) {
//dostuff
EventBus.getDefault().removeStickyEvent(event)
}

Kotlin flows as a message queue between coroutines

I'm attempting to use Kotlin's Flow class as a message queue to transfer data from a producer (a camera) to a set of workers (image analyzers) running on separate coroutines.
The producer in my case is a camera, and will run substantially faster than the workers. Back pressure should be handled by dropping data so that the image analyzers are always operating on the latest images from the camera.
When using channels, this solution works, but seems messy and does not provide an easy way for me to translate the data between the camera and the analyzers (like flow.map).
class ImageAnalyzer<Result> {
fun analyze(image: Bitmap): Result {
// perform some work on the image and return a Result. This can take a long time.
}
}
class CameraAdapter {
private val imageChannel = Channel<Bitmap>(capacity = Channel.RENDEZVOUS)
private val imageReceiveMutex = Mutex()
// additional code to make this camera work and listen to lifecycle events of the enclosing activity.
protected fun sendImageToStream(image: CameraOutput) {
// use channel.offer to ensure the latest images are processed
runBlocking { imageChannel.offer(image) }
}
#OnLifecycleEvent(Lifecycle.Event.ON_DESTROY)
fun onDestroy() {
runBlocking { imageChannel.close() }
}
/**
* Get the stream of images from the camera.
*/
fun getImageStream(): ReceiveChannel<Bitmap> = imageChannel
}
class ImageProcessor<Result>(workers: List<ImageAnalyzer<Result>>) {
private val analysisResults = Channel<Result>(capacity = Channel.RENDEZVOUS)
private val cancelMutex = Mutex()
var finished = false // this can be set elsewhere when enough images have been analyzed
fun subscribeTo(channel: ReceiveChannel<Bitmap>, processingCoroutineScope: CoroutineScope) {
// omit some checks to make sure this is not already subscribed
processingCoroutineScope.launch {
val workerScope = this
workers.forEachIndexed { index, worker ->
launch(Dispatchers.Default) {
startWorker(channel, workerScope, index, worker)
}
}
}
}
private suspend fun startWorker(
channel: ReceiveChannel<Bitmap>,
workerScope: CoroutineScope,
workerId: Int,
worker: ImageAnalyzer
) {
for (bitmap in channel) {
analysisResults.send(worker.analyze(bitmap))
cancelMutex.withLock {
if (finished && workerScope.isActive) {
workerScope.cancel()
}
}
}
}
}
class ExampleApplication : CoroutineScope {
private val cameraAdapter: CameraAdapter = ...
private val imageProcessor: ImageProcessor<Result> = ...
fun analyzeCameraStream() {
imageProcessor.subscribeTo(cameraAdapter.getImageStream())
}
}
What's the proper way to do this? I would like to use a ChannelFlow instead of a Channel to pass data between the camera and the ImageProcessor. This would allow me to call flow.map to add metadata to the images before they're sent to the analyzers. However, when doing so, each ImageAnalyzer gets a copy of the same image instead of processing different images in parallel. Is it possible to use a Flow as a message queue rather than a broadcaster?
I got this working with flows! It was important to keep the flows backed by a channel throughout this sequence so that each worker would pick up unique images to operate on. I've confirmed this functionality through unit tests.
Here's my updated code for posterity:
class ImageAnalyzer<Result> {
fun analyze(image: Bitmap): Result {
// perform some work on the image and return a Result. This can take a long time.
}
}
class CameraAdapter {
private val imageStream = Channel<Bitmap>(capacity = Channel.RENDEZVOUS)
private val imageReceiveMutex = Mutex()
// additional code to make this camera work and listen to lifecycle events of the enclosing activity.
protected fun sendImageToStream(image: CameraOutput) {
// use channel.offer to enforce the drop back pressure strategy
runBlocking { imageChannel.offer(image) }
}
#OnLifecycleEvent(Lifecycle.Event.ON_DESTROY)
fun onDestroy() {
runBlocking { imageChannel.close() }
}
/**
* Get the stream of images from the camera.
*/
fun getImageStream(): Flow<Bitmap> = imageChannel.receiveAsFlow()
}
class ImageProcessor<Result>(workers: List<ImageAnalyzer<Result>>) {
private val analysisResults = Channel<Result>(capacity = Channel.RENDEZVOUS)
private val cancelMutex = Mutex()
var finished = false // this can be set elsewhere when enough images have been analyzed
fun subscribeTo(flow: Flow<Bitmap>, processingCoroutineScope: CoroutineScope): Job {
// omit some checks to make sure this is not already subscribed
return processingCoroutineScope.launch {
val workerScope = this
workers.forEachIndexed { index, worker ->
launch(Dispatchers.Default) {
startWorker(flow, workerScope, index, worker)
}
}
}
}
private suspend fun startWorker(
flow: Flow<Bitmap>,
workerScope: CoroutineScope,
workerId: Int,
worker: ImageAnalyzer
) {
while (workerScope.isActive) {
flow.collect { bitmap ->
analysisResults.send(worker.analyze(bitmap))
cancelMutex.withLock {
if (finished && workerScope.isActive) {
workerScope.cancel()
}
}
}
}
}
fun getAnalysisResults(): Flow<Result> = analysisResults.receiveAsFlow()
}
class ExampleApplication : CoroutineScope {
private val cameraAdapter: CameraAdapter = ...
private val imageProcessor: ImageProcessor<Result> = ...
fun analyzeCameraStream() {
imageProcessor.subscribeTo(cameraAdapter.getImageStream())
}
}
It appears that, so long as the flow is backed by a channel, the subscribers will each get a unique image.

Hiding base class constructor parameters in Kotlin

I am trying to understand how to hide a base constructor parameter in a subclass in kotlin. How do you put a facade over a base constructor? This doesn't work:
import com.android.volley.Request
import com.android.volley.Response
class MyCustomRequest(url: String)
: Request<String>(Request.Method.POST, url, hiddenListener) {
private fun hiddenListener() = Response.ErrorListener {
/* super secret listener */
}
...
}
I think I understand the problem:
During construction of a new instance of a derived class, the base
class initialization is done as the first step (preceded only by
evaluation of the arguments for the base class constructor) and thus
happens before the initialization logic of the derived class is run.
I'm trying to solve this problem for Volley, where I need my custom request to be be a Request so that it can be passed into a RequestQueue. It would be easier of RequestQueue took in some kind of interface but since it doesn't I have to subclass. There are other ways I can hide these complexities from the caller, but this limitation has come up for me other times in Kotlin and I'm not sure how to solve it.
I am not familiar with volley but I tried to come up with an example that should give you some insight how to solve your problem. What you can do is use a companion object:
interface MyListener {
fun handleEvent()
}
open class Base<T>(anything: Any, val listener: MyListener) { // this would be your Request class
fun onSomeEvent() {
listener.handleEvent()
}
}
class Derived(anything: Any) : Base<Any>(anything, hiddenListener) { // this would be your MyCustomRequest class
private companion object {
private val hiddenListener = object : MyListener {
override fun handleEvent() {
// do secret stuff here
}
}
}
}
So if you apply this to your problem, the result should look something like this:
class MyCustomRequest(url: String)
: Request<String>(Request.Method.POST, url, hiddenListener) {
private companion object {
private val hiddenListener = Response.ErrorListener {
/* super secret listener */
}
}
...
}
A different way would be to use a decorator, create your Request withing that decorator and just delegate the calls to it:
class Decorator(anything: Any) {
private var inner: Base<Any>
private val hiddenListener: MyListener = object : MyListener {
override fun handleEvent() { }
}
init {
inner = Base(anything, hiddenListener)
}
}
And once again for your example that would look like this:
class MyCustomRequest(url: String) {
private var inner: Request<String>
private val hiddenListener = Response.ErrorListener {
/* super secret listener */
}
init {
inner = Request<String>(Request.Method.POST, url, hiddenListener)
}
...
}

What is the reason for twitter4j.StreamListner IllegalAccessError in Kotlin?

When implementing a twitter4j.StatusListner in Kotlin, I get the following IllegalAccessError and associated stack trace:
Exception in thread "main" java.lang.IllegalAccessError: tried to access class twitter4j.StreamListener from class rxkotlin.rxextensions.TwitterExampleKt$observe$1
at rxkotlin.rxextensions.TwitterExampleKt$observe$1.subscribe(TwitterExample.kt:50)
at io.reactivex.internal.operators.observable.ObservableCreate.subscribeActual(ObservableCreate.java:40)
at io.reactivex.Observable.subscribe(Observable.java:10700)
at io.reactivex.Observable.subscribe(Observable.java:10686)
at io.reactivex.Observable.subscribe(Observable.java:10615)
at rxkotlin.rxextensions.TwitterExampleKt.main(TwitterExample.kt:8)
Produced by the following code:
val twitterStream = TwitterStreamFactory().instance
// See https://stackoverflow.com/questions/37672023/how-to-create-an-instance-of-anonymous-interface-in-kotlin/37672334
twitterStream.addListener(object : StatusListener {
override fun onStatus(status: Status?) {
if (emitter.isDisposed) {
twitterStream.shutdown()
} else {
emitter.onNext(status)
}
}
override fun onException(e: Exception?) {
if (emitter.isDisposed) {
twitterStream.shutdown()
} else {
emitter.onError(e)
}
}
// Other overrides.
})
emitter.setCancellable { twitterStream::shutdown }
If I don't use Rx, it makes the exception a bit simpler:
twitterStream.addListener(object: twitter4j.StatusListener {
override fun onStatus(status: Status) { println("Status: {$status}") }
override fun onException(ex: Exception) { println("Error callback: $ex") }
// Other overrides.
})
Exception in thread "main" java.lang.IllegalAccessError: tried to access class twitter4j.StreamListener from class rxkotlin.rxextensions.TwitterExampleKt
at rxkotlin.rxextensions.TwitterExampleKt.main(TwitterExample.kt:14)
However, if I implement a Java wrapper function, no error is thrown and the behaviour is as expected:
Wrapper -
public class Twitter4JHelper {
public static void addStatusListner(TwitterStream stream, StatusListener listner) {
stream.addListener(listner);
}
}
Revised implementation -
val twitterStream = TwitterStreamFactory().instance
val listner = object: StatusListener {
override fun onStatus(status: Status?) {
if (emitter.isDisposed) {
twitterStream.shutdown()
} else {
emitter.onNext(status)
}
}
override fun onException(e: Exception?) {
if (emitter.isDisposed) {
twitterStream.shutdown()
} else {
emitter.onError(e)
}
}
// Other overrides.
}
Twitter4JHelper.addStatusListner(twitterStream, listner)
emitter.setCancellable { twitterStream::shutdown }
This revised solution comes from a blog post, which I think tries to explain the cause but Google translate is not being my friend. What is causing the IllegalAccessError? Is there a purely Kotlin based solution, or will I have to live with this workaround?
Yep that's not going to work.
addListener method takes a StreamListener param and StreamListener is non-public (package private). I would definitely raise a bug against Kotlin compiler for this.
The code Kotlin compiler generates is:
TwitterStream twitterStream = (new TwitterStreamFactory()).getInstance();
twitterStream.addListener((StreamListener)(new StatusListener() {
// ..overrides ...
}));
StatusListener already implements StreamListener so I don't see why the cast is required.
I worked around this by using a java utility class:
public class T4JCompat {
public static void addStatusListener(TwitterStream stream, StatusListener listener) {
stream.addListener(listener);
}
public static void removeStatusListener(TwitterStream stream, StatusListener listener) {
stream.removeListener(listener);
}
}
You can call these methods from Kotlin and things work as expected.