I need to load some data from server page by page until all the data is loaded. The data is considered to be fully loaded if at some point I received fewer items than I've requested. This is the working solution that I have right now:
return Observable.fromCallable { 0 }
.delay(500, TimeUnit.MILLISECONDS)
.repeat()
.scan { previousPage, _ -> previousPage + 1}
.concatMap { doLongFetch(it) }
.takeUntil { it.size < 100 }
fun doLongFetch(page: Int): Observable<List<ListItem>>() {
//Here I do the loading
}
However, there's a problem with the source observable. As you can see, it emits new values every 500 milliseconds to provide some input for the scan function. The delay is required since otherwise, it would emit thousands of values in a very short period of time, which is not required at all. Ideally, I want to remove that delay completely and make sure that the source observable emits another value only after the downstream has handled the previous one (meaning that the data has been requested and processed).
Any ideas on how I can do that?
I would like to consume some stream-data using Kotlin actors
I was thinking to put my consumer inside an actor, while it polls in an infinite loop while(true). Then, when I decide, I send a message to stop the consumer.
Currently I have this:
while(true) {
for (message in channel){ <--- blocked in here, waiting
when(message) {
is MessageStop -> consumer.close()
else -> {}
}
}
consumer.poll()
}
The problem
The problem with this is that it only runs when I send a message to the actor, so my consumer is not polling the rest of the time because channel is blocking waiting to receive the next message
Is there any alternative?, someone with the same issue?, or something similar to actors but not blocked by channel in Kotlin?
Since the channel is just a Channel (https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines.channels/-channel/index.html) you can first check if the channel is empty and if so start your polling. Otherwise handle the messages.
E.g.
while(true) {
while (channel.isNotEmpty()) {
val message = channel.receive()
when(message) {
is MessageStop -> consumer.close()
else -> {}
}
}
consumer.poll()
}
In the end I used AKKA with Kotlin, I'm finding much easier this way
You should use postDelayed(), for example:
final Runnable r = new Runnable() {
public void run() {
// your code here
handler.postDelayed(this, 1000)
}
}
You can change 1000 with the the millisecond delay you want. Also I highly recommend to put your code inside a thread (if you are not already have) to prevent ANR (App Not Responding)
I am sending a value via MyRepository.myConflatedChannel.offer(myvalue).
I then expect to receive it in collect { } or onEach { } blocks in my ViewModel. However, neither function is invoked. It is as if nothing is passed down the ConflatedBroadcastChannel.
Has anybody seen a similar problem?
Make sure you properly work with receiving values.
If you use the ConflatedBroadcastChannel, you can use either OpenSubscription to get a ReceiveChannel or you can represent it as flow (with asFlow).
Note that consume and consumeEach are terminal, they perform an action and then cancel the channel after the execution of the block. See this.
First case:
val receivingChannel = MyRepository.myConflatedChannel.openSubscription()
// then you can consume values using for example a for loop, e.g.:
launch {
for (value in receivingChannel) {
// do something
}
}
Second case:
val receivingFlow = MyRepository.myConflatedChannel.asFlow()
launch {
receivingFlow.collect {
// do something
}
}
I am receiving messages on UDP in multiple threads. After each reception I raise MessageReceived.OnNext(message).
Because I am using multiple threads the messages raised unordered which is a problem.
How can I order the raise of the messages by the message counter?
(lets say there is a message.counter property)
Must take in mind a message can get lost in the communication (lets say if we have a counter hole after X messages that the hole is not filled I raise the next message)
Messages must be raised ASAP (if the next counter received)
In stating the requirement for detecting lost messages, you haven't considered the possibility of the last message not arriving; I've added a timeoutDuration which flushes the buffered messages if nothing arrives in the given time - you may want to consider this an error instead, see the comments for how to do this.
I will solve this by defining an extension method with the following signature:
public static IObservable<TSource> Sort<TSource>(
this IObservable<TSource> source,
Func<TSource, int> keySelector,
TimeSpan timeoutDuration = new TimeSpan(),
int gapTolerance = 0)
source is the stream of unsorted messages
keySelector is a function that extracts an int key from a message. I assume the first key sought is 0; amend if necessary.
timeoutDuration is discussed above, if omitted, there is no timeout
tolerance is the maximum number of messages held back while waiting for an out of order message. Pass 0 to hold any number of messages
scheduler is the scheduler to use for the timeout and is supplied for test purposes, a default is used if not given.
Walkthrough
I'll present a line-by-line walkthrough here. The full implementation is repeated below.
Assign Default Scheduler
First of all we must assign a default scheduler if none was supplied:
scheduler = scheduler ?? Scheduler.Default;
Arrange Timeout
Now if a time out was requested, we will replace the source with a copy that will simply terminate and send OnCompleted if a message doesn't arrive in timeoutDuration.
if(timeoutDuration != TimeSpan.Zero)
source = source.Timeout(
timeoutDuration,
Observable.Empty<TSource>(),
scheduler);
If you wish to send a TimeoutException instead, just delete the second parameter to Timeout - the empty stream, to select an overload that does this. Note we can safely share this with all subscribers, so it is positioned outside the call to Observable.Create.
Create Subscribe handler
We use Observable.Create to build our stream. The lambda function that is the argument to Create is invoked whenever a subscription occurs and we are passed the calling observer (o). Create returns our IObservable<T> so we return it here.
return Observable.Create<TSource>(o => { ...
Initialize some variables
We will track the next expected key value in nextKey, and create a SortedDictionary to hold the out of order messages until they can be sent.
int nextKey = 0;
var buffer = new SortedDictionary<int, TSource>();
Subscribe to the source, and handle messages
Now we can subscribe to the message stream (possibly with the timeout applied). First we introduce the OnNext handler. The next message is assigned to x:
return source.Subscribe(x => { ...
We invoke the keySelector function to extract the key from the message:
var key = keySelector(x);
If the message has an old key (because it exceeded our tolerance for out of order messages) we are just going to drop it and be done with this message (you may want to act differently):
// drop stale keys
if(key < nextKey) return;
Otherwise, we might have the expected key, in which case we can increment nextKey send the message:
if(key == nextKey)
{
nextKey++;
o.OnNext(x);
}
Or, we might have an out of order future message, in which case we must add it to our buffer. If we do this, we must also ensure our buffer hasn't exceeded our tolerance for storing out of order messages - in this case, we will also bump nextKey to the first key in the buffer which because it is a SortedDictionary is conveniently the next lowest key:
else if(key > nextKey)
{
buffer.Add(key, x);
if(gapTolerance != 0 && buffer.Count > gapTolerance)
nextKey = buffer.First().Key;
}
Now regardless of the outcome above, we need to empty the buffer of any keys that are now ready to go. We use a helper method for this. Note that it adjusts nextKey so we must be careful to pass it by reference. We simply loop over the buffer reading, removing and sending messages as long as the keys follow on from each other, incrementing nextKey each time:
private static void SendNextConsecutiveKeys<TSource>(
ref int nextKey,
IObserver<TSource> observer,
SortedDictionary<int, TSource> buffer)
{
TSource x;
while(buffer.TryGetValue(nextKey, out x))
{
buffer.Remove(nextKey);
nextKey++;
observer.OnNext(x);
}
}
Dealing with errors
Next we supply an OnError handler - this will just pass through any error, including the Timeout exception if you chose to go that way.
Flushing the buffer
Finally, we must handle OnCompleted. Here I have opted to empty the buffer - this would be necessary if an out of order message held up messages and never arrived. This is why we need a timeout:
() => {
// empty buffer on completion
foreach(var item in buffer)
o.OnNext(item.Value);
o.OnCompleted();
});
Full Implementation
Here is the full implementation.
public static IObservable<TSource> Sort<TSource>(
this IObservable<TSource> source,
Func<TSource, int> keySelector,
int gapTolerance = 0,
TimeSpan timeoutDuration = new TimeSpan(),
IScheduler scheduler = null)
{
scheduler = scheduler ?? Scheduler.Default;
if(timeoutDuration != TimeSpan.Zero)
source = source.Timeout(
timeoutDuration,
Observable.Empty<TSource>(),
scheduler);
return Observable.Create<TSource>(o => {
int nextKey = 0;
var buffer = new SortedDictionary<int, TSource>();
return source.Subscribe(x => {
var key = keySelector(x);
// drop stale keys
if(key < nextKey) return;
if(key == nextKey)
{
nextKey++;
o.OnNext(x);
}
else if(key > nextKey)
{
buffer.Add(key, x);
if(gapTolerance != 0 && buffer.Count > gapTolerance)
nextKey = buffer.First().Key;
}
SendNextConsecutiveKeys(ref nextKey, o, buffer);
},
o.OnError,
() => {
// empty buffer on completion
foreach(var item in buffer)
o.OnNext(item.Value);
o.OnCompleted();
});
});
}
private static void SendNextConsecutiveKeys<TSource>(
ref int nextKey,
IObserver<TSource> observer,
SortedDictionary<int, TSource> buffer)
{
TSource x;
while(buffer.TryGetValue(nextKey, out x))
{
buffer.Remove(nextKey);
nextKey++;
observer.OnNext(x);
}
}
Test Harness
If you include nuget rx-testing in a console app, the following will run given you a test harness to play with:
public static void Main()
{
var tests = new Tests();
tests.Test();
}
public class Tests : ReactiveTest
{
public void Test()
{
var scheduler = new TestScheduler();
var xs = scheduler.CreateColdObservable(
OnNext(100, 0),
OnNext(200, 2),
OnNext(300, 1),
OnNext(400, 4),
OnNext(500, 5),
OnNext(600, 3),
OnNext(700, 7),
OnNext(800, 8),
OnNext(900, 9),
OnNext(1000, 6),
OnNext(1100, 12),
OnCompleted(1200, 0));
//var results = scheduler.CreateObserver<int>();
xs.Sort(
keySelector: x => x,
gapTolerance: 2,
timeoutDuration: TimeSpan.FromTicks(200),
scheduler: scheduler).Subscribe(Console.WriteLine);
scheduler.Start();
}
}
Closing comments
There's all sorts of interesting alternative approaches here. I went for this largely imperative approach because I think it's easiest to follow - but there's probably some fancy grouping shenanigans you can employ to do this to. One thing I know to be consistently true about Rx - there's always many ways to skin a cat!
I'm also not entirely comfortable with the timeout idea here - in a production system, I would want to implement some means of checking connectivity, such as a heartbeat or similar. I didn't get into this because obviously it will be application specific. Also, heartbeats have been discussed on these boards and elsewhere before (such as on my blog for example).
Strongly consider using TCP instead if you want reliable ordering - that's what it's for; otherwise, you'll be forced to play a guessing game with UDP and sometimes you'll be wrong.
For example, imagine that you receive the following datagrams in this order: [A, B, D]
When you receive D, how long should you wait for C to arrive before pushing D?
Whatever duration you choose you may be wrong:
What if C was lost during transmission and so it will never arrive?
What if the duration you chose is too short and you end up pushing D but then receive C?
Perhaps you could choose a duration that heuristically works best, but why not just use TCP instead?
Side Note:
MessageReceived.OnNext implies that you're using a Subject<T>, which is probably unnecessary. Consider converting the async UdpClient methods into observables directly instead, or convert them by writing an async iterator via Observable.Create<T>(async (observer, cancel) => { ... }).
i am processing a ragged semicolon delimited file using script component as transformation.
The component is able to process the data and load to oledb destination. But when error is found it should stop processing further. As i am using try catch block the component doesn't fail and continue to process till the end.
Is there any way i could stop the processing further without failing the component/package?
Let me know if any other information/details required?
sample code:
str.split(";");
if(column[0] == "H")
{
col1=Column[3];
}
if(column[0] != "T")
{
try
{
Row.col1=Column[0];
Row.col2=Column[1];
.....
}
catch
{
update the variable to check if we have error in file.
}
}
Thank you for your time.
The general idea will be that you want to use try/catch blocks to ensure the data processing itself doesn't abort. Once you know your script isn't reporting a failure back to the engine, it's a simple process to not call the AddRow()
Pseudocode
foreach(line in fileReader)
{
try
{
// perform dangerous operations here
// Only add row if you have been able to parse current line
Output0Buffer.AddRow();
Output0Buffer.Col1 = parsedContent;
}
catch
{
// Signal that we should break out of the loop
// do not propagate the error
// You might want to do something though so you know you
// have an incomplete load
break;
}
}
If you are looking to just skip the current bad line, you can substitute continue for the break above.
C# loop - break vs. continue
I didn't get any help from anywhere, But as a work around i have placed a return statement in the code. It checks the error variable if it's true then i will return without processing further. But the thing still is it processes the whole file :(. But it works!!!