val seq1 = sequenceOf(1, 2, 3)
val seq2 = sequenceOf(5, 6, 7)
sequenceOf(seq1, seq2).flatten().forEach { ... }
That's how I'm doing sequence concatenation but I'm worrying that it's actually copying elements, whereas all I need is an iterator that uses elements from the iterables (seq1, seq2) I gave it.
Is there such a function?
Your code doesn't copy the sequence elements, and sequenceOf(seq1, seq2).flatten() actually does what you want: it generates a sequence that takes items first from seq1 and then, when seq1 finishes, from seq2.
Also, operator + is implemented in exactly this way, so you can just use it:
(seq1 + seq2).forEach { ... }
The source of the operator is as expected:
public operator fun <T> Sequence<T>.plus(elements: Sequence<T>): Sequence<T> {
return sequenceOf(this, elements).flatten()
}
You can take a look at the implementation of .flatten() in stdlib that uses FlatteningSequence, which actually switches over the original sequences' iterators. The implementation can change over time, but Sequence is intended to be as lazy as possible, so you can expect it to behave in a similar way.
Example:
val a = generateSequence(0) { it + 1 }
val b = sequenceOf(1, 2, 3)
(a + b).take(3).forEach { println(it) }
Here, copying the first sequence can never succeed since it's infinite, and iterating over (a + b) takes items one by one from a.
Note, however, that .flatten() is implemented in a different way for Iterable, and it does copy the elements. Find more about the differences between Iterable and Sequence here.
Something else you might need to do is create a sequence of sequences:
val xs = sequence {
yield(1)
yield(2)
}
val twoXs = sequence {
yieldAll(xs)
// ... interesting things here ...
yieldAll(xs)
}
This doesn't do anything that xs + xs doesn't do, but it gives you a place to do more complex things.
Related
How to "take(N)" iteratively - get a Sequence<Sequence>, each inner sequences having next N elements?
I am writing a high-load application in Kotlin.
I have tens of thousands of entries to insert to a database.
I want to batch them by, say, 1000.
So I created a loop:
val itemsSeq = itemsList.iterator().asSequence()
while (true) {
log.debug("Taking $BATCH_SIZE from $itemsSeq")
val batchSeq = itemsSeq.take(BATCH_SIZE)
val squareBatch = applySomething(batchSeq, something)
?: break
}
fun applySomething(batch: Sequence<Item>, something: Something) {
/* Fully consumes batch. Bulk-loads from DB by IDs, applies, bulk-saves. */
}
I thought that take() would advance the itemsSeq and the next call to take() would give me a sequence "view" of itemsSeq starting at the 10th item.
But with this code, I am getting:
DEBUG Taking 10 from kotlin.sequences.ConstrainedOnceSequence#53fe15ff
Exception in thread "main" java.lang.IllegalStateException: This sequence can be consumed only once.
at kotlin.sequences.ConstrainedOnceSequence.iterator(SequencesJVM.kt:23)
at kotlin.sequences.TakeSequence$iterator$1.<init>(Sequences.kt:411)
at kotlin.sequences.TakeSequence.iterator(Sequences.kt:409)
So it seems that the take() "opens" the itemsSeq again, while that can be consumed only once.
As a workaround, I can use chunked():
public fun <T> Sequence<T>.chunked(size: Int): Sequence<List<T>> {
But I would prefer not to create Lists, rather Sequences.
What I am looking for is something between take() and chunked().
Is there anything such in Kotlin SDK?
I can possibly create my own sequence { ... } but for readability, I would prefer something built-in.
There is a way to construct a Sequence by handing it over an Iterator, see Sequence.
Given an iterator function constructs a Sequence that returns values
through the Iterator provided by that function. The values are
evaluated lazily, and the sequence is potentially infinite.
Wrapped in an extension function it could look like this:
fun <T> Iterable<T>.toValuesThroughIteratorSequence(): Sequence<T> {
val iterator = this.iterator()
return Sequence { iterator }
}
Quick test:
data class Test(val id: Int)
val itemsList = List(45) { Test(it) }
val batchSize = 10
val repetitions = itemsList.size.div(batchSize) + 1
val itemsSeq = itemsList.toValuesThroughIteratorSequence()
(0 until repetitions).forEach { index ->
val batchSeq = itemsSeq.take(batchSize)
println("Batch no. $index: " + batchSeq.map { it.id.toString().padStart(2, ' ') }.joinToString(" "))
}
Output:
Batch no. 0: 0 1 2 3 4 5 6 7 8 9
Batch no. 1: 10 11 12 13 14 15 16 17 18 19
Batch no. 2: 20 21 22 23 24 25 26 27 28 29
Batch no. 3: 30 31 32 33 34 35 36 37 38 39
Batch no. 4: 40 41 42 43 44
Background
First of all, we need to be aware there is a big difference between an object that we can iterate over and object that represents a "live" or already running iteration process. First group means Iterable (so List, Set and all other collections), Array, Flow, etc. Second group is mostly Iterator or old Java Enumeration. The difference could be also compared to file vs file pointer when reading or database table vs database cursor.
Sequence belongs to the first group. Sequence object does not represent a live, already started iteration, but just a set of elements. These elements can be produced lazily, sequence could have unbounded size and usually internally it works by using iterators, but conceptually sequence is not an iterator itself.
If we look into the documentation about sequences it clearly compares them to Iterable, not to Iterator. All standard ways to construct sequences like: sequenceOf(), sequence {}, Iterable.asSequence() produce sequences that return the same list of items every time we iterate over them. Iterator.asSequence() also follows this pattern, but because it can't re-produce same items twice, it is intentionally protected against iterating multiple times:
public fun <T> Iterator<T>.asSequence(): Sequence<T> = Sequence { this }.constrainOnce()
Problem
Your initial attempt with using take() didn't work, because this is a misuse of sequences. We expect that subsequent take() calls on the same sequence object will produce exactly the same items (usually), not next items. Similarly as we expect multiple take() calls on a list always produce same items, each time starting from the beginning.
Being more specific, your error was caused by above constrainOnce(). When we invoke take() multiple times on a sequence, it has to restart from the beginning, but it can't do this if it was created from an iterator, so Iterator.asSequence() explicitly disallows this.
Simple solution
To fix the problem, you can just skip constrainOnce() part, as suggested by #lukas.j. This solution is nice, because stdlib already provides tools like Sequence.take(), so if used carefully, this is the easiest to implement and it just works.
However, I personally consider this a kind of workaround, because the resulting sequence doesn't behave as sequences do. It is more like an iterator on steroids than a real sequence. You need to be careful when using this sequence with existing operators or 3rd party code, because such sequence may work differently than they expect and as a result, you may get incorrect results.
Advanced solution
We can follow your initial attempt of using subsequent take() calls. In this case our object is used for live iteration, so it is no longer a proper sequence, but rather an iterator. The only thing we miss in stdlib is a way to create a sub-iterator with a single chunk. We can implement it by ourselves:
fun main() {
val list = (0 .. 25).toList()
val iter = list.iterator()
while (iter.hasNext()) {
val chunk = iter.limited(10)
println(chunk.asSequence().toList())
}
}
fun <T> Iterator<T>.limited(n: Int): Iterator<T> = object : Iterator<T> {
var left = n
val iterator = this#limited
override fun next(): T {
if (left == 0)
throw NoSuchElementException()
left--
return iterator.next()
}
override fun hasNext(): Boolean {
return left > 0 && iterator.hasNext()
}
}
I named it limited(), because take() suggests we read items from the iterator. Instead, we only create another iterator on top of the provided iterator.
Of course, sequences are easier to use than iterators and typical solution to this problem is by using chunked(). With above limited() it is pretty straightforward to implement chunkedAsSequences():
fun main() {
val list = (0 .. 25).toList()
list.asSequence()
.chunkedAsSequences(10)
.forEach { println(it.toList()) }
}
fun <T> Sequence<T>.chunkedAsSequences(size: Int): Sequence<Sequence<T>> = sequence {
val iter = iterator()
while (iter.hasNext()) {
val chunk = iter.limited(size)
yield(chunk.asSequence())
chunk.forEach {} // required if chunk was not fully consumed
}
}
Please also note there is a tricky case of chunk being not fully consumed. chunkedAsSequences() is protected against this scenario. Previous simpler solutions aren't.
I would like to generate random numbers from a union of ranges in Kotlin. I know I can do something like
((1..10) + (50..100)).random()
but unfortunately this creates an intermediate list, which can be rather expensive when the ranges are large.
I know I could write a custom function to randomly select a range with a weight based on its width, followed by randomly choosing an element from that range, but I am wondering if there is a cleaner way to achieve this with Kotlin built-ins.
Suppose your ranges are nonoverlapped and sorted, if not, you could have some preprocessing to merge and sort.
This comes to an algorithm choosing:
O(1) time complexity and O(N) space complexity, where N is the total number, by expanding the range object to a set of numbers, and randomly pick one. To be compact, an array or list could be utilized as the container.
O(M) time complexity and O(1) space complexity, where M is the number of ranges, by calculating the position in a linear reduction.
O(M+log(M)) time complexity and O(M) space complexity, where M is the number of ranges, by calculating the position using a binary search. You could separate the preparation(O(M)) and generation(O(log(M))), if there are multiple generations on the same set of ranges.
For the last algorithm, imaging there's a sorted list of all available numbers, then this list can be partitioned into your ranges. So there's no need to really create this list, you just calculate the positions of your range s relative to this list. When you have a position within this list, and want to know which range it is in, do a binary search.
fun random(ranges: Array<IntRange>): Int {
// preparation
val positions = ranges.map {
it.last - it.first + 1
}.runningFold(0) { sum, item -> sum + item }
// generation
val randomPos = Random.nextInt(positions[ranges.size])
val found = positions.binarySearch(randomPos)
// binarySearch may return an "insertion point" in negative
val range = if (found < 0) -(found + 1) - 1 else found
return ranges[range].first + randomPos - positions[range]
}
Short solution
We can do it like this:
fun main() {
println(random(1..10, 50..100))
}
fun random(vararg ranges: IntRange): Int {
var index = Random.nextInt(ranges.sumOf { it.last - it.first } + ranges.size)
ranges.forEach {
val size = it.last - it.first + 1
if (index < size) {
return it.first + index
}
index -= size
}
throw IllegalStateException()
}
It uses the same approach you described, but it calls for random integer only once, not twice.
Long solution
As I said in the comment, I often miss utils in Java/Kotlin stdlib for creating collection views. If IntRange would have something like asList() and we would have a way to concatenate lists by creating a view, this would be really trivial, utilizing existing logic blocks. Views would do the trick for us, they would automatically calculate the size and translate the random number to the proper value.
I implemented a POC, maybe you will find it useful:
fun main() {
val list = listOf(1..10, 50..100).mergeAsView()
println(list.size) // 61
println(list[20]) // 60
println(list.random())
}
#JvmName("mergeIntRangesAsView")
fun Iterable<IntRange>.mergeAsView(): List<Int> = map { it.asList() }.mergeAsView()
#JvmName("mergeListsAsView")
fun <T> Iterable<List<T>>.mergeAsView(): List<T> = object : AbstractList<T>() {
override val size = this#mergeAsView.sumOf { it.size }
override fun get(index: Int): T {
if (index < 0 || index >= size) {
throw IndexOutOfBoundsException(index)
}
var remaining = index
this#mergeAsView.forEach { curr ->
if (remaining < curr.size) {
return curr[remaining]
}
remaining -= curr.size
}
throw IllegalStateException()
}
}
fun IntRange.asList(): List<Int> = object : AbstractList<Int>() {
override val size = endInclusive - start + 1
override fun get(index: Int): Int {
if (index < 0 || index >= size) {
throw IndexOutOfBoundsException(index)
}
return start + index
}
}
This code does almost exactly the same thing as short solution above. It only does this indirectly.
Once again: this is just a POC. This implementation of asList() and mergeAsView() is not at all production-ready. We should implement more methods, like for example iterator(), contains() and indexOf(), because right now they are much slower than they could be. But it should work efficiently already for your specific case. You should probably test it at least a little. Also, mergeAsView() assumes provided lists are immutable (they have fixed size) which may not be true.
It would be probably good to implement asList() for IntProgression and for other primitive types as well. Also you may prefer varargs version of mergeAsView() than extension function.
As a final note: I guess there are libraries that does this already - probably some related to immutable collections. But if you look for a relatively lightweight solution, it should work for you.
I have the following data for my task:
Input: nums = [1,2,3,4]
Output: [1,3,6,10]
Explanation: Running sum is obtained as follows: [1, 1+2, 1+2+3, 1+2+3+4].
As u see, I need to return IntArray, the first thing I used was runningReduce() , but this function is used in the version of Kotlin 1.4.30.
fun runningSum(nums: IntArray): IntArray {
return nums.runningReduce { sum, element -> sum + element }.toIntArray()
}
Yes, this solution works, but how can I solve the same problem using reduce() or fold()?
Try the following:
nums.fold(listOf(0)) { acc, i -> acc + (acc.last() + i) }.drop(1).toIntArray()
The solution is sub-optimal though: it copies the list in each iteration. But looks fancy.
To avoid copying, you could write it as follows:
nums.fold(mutableListOf(0)) { acc, i -> acc += (acc.last() + i); acc }.drop(1).toIntArray()
I think I prefer the first version, the second one is not pure from functional perspective.
Mafor is right, fold() is not a good choice when producing a collection because it copies the collection every time so you need to workaround with mutable collections, which defeats the point of the functional style.
If you really want to work with arrays, which are mutable, doing it the old fashioned procedural way may be best:
val array = intArrayOf(1, 2, 3, 4)
for (i in array.indices.drop(1)) {
array[i] += array[i - 1]
}
println(array.joinToString(", "))
Here's a slightly modified version of Mafor's answer that gives you an IntArray and avoids the use of multiple statements - it still uses the mutable list though:
val input = intArrayOf(1, 2, 3, 4)
val output = input.fold(mutableListOf(0)) { acc, cur ->
acc.apply { add(last() + cur) }
}.drop(1).toIntArray()
println(output.joinToString(", "))
Both of these print 1, 3, 6, 10.
I am pretty confused with both functions fold() and reduce() in Kotlin, can anyone give me a concrete example that distinguishes both of them?
fold takes an initial value, and the first invocation of the lambda you pass to it will receive that initial value and the first element of the collection as parameters.
For example, take the following code that calculates the sum of a list of integers:
listOf(1, 2, 3).fold(0) { sum, element -> sum + element }
The first call to the lambda will be with parameters 0 and 1.
Having the ability to pass in an initial value is useful if you have to provide some sort of default value or parameter for your operation. For example, if you were looking for the maximum value inside a list, but for some reason want to return at least 10, you could do the following:
listOf(1, 6, 4).fold(10) { max, element ->
if (element > max) element else max
}
reduce doesn't take an initial value, but instead starts with the first element of the collection as the accumulator (called sum in the following example).
For example, let's do a sum of integers again:
listOf(1, 2, 3).reduce { sum, element -> sum + element }
The first call to the lambda here will be with parameters 1 and 2.
You can use reduce when your operation does not depend on any values other than those in the collection you're applying it to.
The major functional difference I would call out (which is mentioned in the comments on the other answer, but may be hard to understand) is that reduce will throw an exception if performed on an empty collection.
listOf<Int>().reduce { x, y -> x + y }
// java.lang.UnsupportedOperationException: Empty collection can't be reduced.
This is because .reduce doesn't know what value to return in the event of "no data".
Contrast this with .fold, which requires you to provide a "starting value", which will be the default value in the event of an empty collection:
val result = listOf<Int>().fold(0) { x, y -> x + y }
assertEquals(0, result)
So, even if you don't want to aggregate your collection down to a single element of a different (non-related) type (which only .fold will let you do), if your starting collection may be empty then you must either check your collection size first and then .reduce, or just use .fold
val collection: List<Int> = // collection of unknown size
val result1 = if (collection.isEmpty()) 0
else collection.reduce { x, y -> x + y }
val result2 = collection.fold(0) { x, y -> x + y }
assertEquals(result1, result2)
Another difference that none of the other answers mentioned is the following:
The result of a reduce operation will always be of the same type (or a super type) as the data that is being reduced.
We can see that from the definition of the reduce method:
public inline fun <S, T : S> Iterable<T>.reduce(operation: (acc: S, T) -> S): S {
val iterator = this.iterator()
if (!iterator.hasNext()) throw UnsupportedOperationException("Empty collection can't be reduced.")
var accumulator: S = iterator.next()
while (iterator.hasNext()) {
accumulator = operation(accumulator, iterator.next())
}
return accumulator
}
On the other hand, the result of a fold operation can be anything, because there are no restrictions when it comes to setting up the initial value.
So, for example, let us say that we have a string that contains letters and digits. We want to calculate the sum of all the digits.
We can easily do that with fold:
val string = "1a2b3"
val result: Int = string.fold(0, { currentSum: Int, char: Char ->
if (char.isDigit())
currentSum + Character.getNumericValue(char)
else currentSum
})
//result is equal to 6
reduce - The reduce() method transforms a given collection into a single result.
val numbers: List<Int> = listOf(1, 2, 3)
val sum: Int = numbers.reduce { acc, next -> acc + next }
//sum is 6 now.
fold - What would happen in the previous case of an empty list? Actually, there’s no right value to return, so reduce() throws a RuntimeException
In this case, fold is a handy tool. You can put an initial value by it -
val sum: Int = numbers.fold(0, { acc, next -> acc + next })
Here, we’ve provided initial value. In contrast, to reduce(), if the collection is empty, the initial value will be returned which will prevent you from the RuntimeException.
Simple Answer
Result of both reduce and fold is "a list of items will be transformed into a single item".
In case of fold,we provide 1 extra parameter apart from list but in case of reduce,only items in list will be considered.
Fold
listOf("AC","Fridge").fold("stabilizer") { freeGift, itemBought -> freeGift + itemBought }
//output: stabilizerACFridge
In above case,think as AC,fridge bought from store & they give stabilizer as gift(this will be the parameter passed in the fold).so,you get all 3 items together.Please note that freeGift will be available only once i.e for the first iteration.
Reduce
In case of reduce,we get items in list as parameters and can perform required transformations on it.
listOf("AC","Fridge").reduce { itemBought1, itemBought2 -> itemBought1 + itemBought2 }
//output: ACFridge
The difference between the two functions is that fold() takes an initial value and uses it as the accumulated value on the first step, whereas the first step of reduce() uses the first and the second elements as operation arguments on the first step.
I need to iterate part of an array backwards. I'd like to do that "functionally" as it's more comprehensible, like that
for (b in buf.sliceArray(0 until bufLimit).reversedArray()) {}
But both sliceArray and reversedArray are not lazy. Is there a lazy version or should I probably fall back to
for (bIdx in bufLimit - 1 downTo 0) {
val b = buf[bIdx]
}
which is more confusing and verbose?
If you use a list instead of an array, then you can reverse it and then convert to a Sequence:
val buf: List = listOf(1, 2, 3, 4, 5)
val bufLimit = 3
for (b in buf.asReversed().asSequence().drop(buf.size - bufLimit)) {
println(b)
}
Functions with the as prefix only wrap objects without copying, so the code above does not copy the buf content.
Note that you shouldn't loose any performance compared to Array if you use an ArrayList.
However this solution does involve several iterators, so it is somewhat less efficient than the index code you have suggested in the question:
for (bIdx in bufLimit - 1 downTo 0) {
val b = buf[bIdx]
}
I suggest creating an extension function to handle your specific use case. e.g.:
/**
* Performs the given [action] on each element at the specified [indices].
*/
inline fun ByteArray.forEachAt(indices: Iterable<Int>, action: (Byte) -> Unit): Unit {
indices.forEach { index -> action(this[index]) }
}
Usage:
buf.forEachAt((0 until bufLimit).reversed)) {}
// or
buf.forEachAt(bufLimit - 1 downTo 0) {}