Related
This code below I understand and was helpful.
But I would like to make this a generic approach, but cannot actually get started, and think that it is not possible actually with the case statement. I am looking at another approach, but am interested if a generic approach is also possible here.
import spark.implicits._
import org.apache.spark.sql.Encoders
// Creating case classes with the schema of your json objects. We're making
// these to make use of strongly typed Datasets. Notice that the MyChgClass has
// each field as an Option: this will enable us to choose between "chg" and
// "before"
case class MyChgClass(b: Option[String], c: Option[String], d: Option[String])
case class MyFullClass(k: Int, b: String, c: String, d: String)
case class MyEndClass(id: Int, after: MyFullClass)
// Creating schemas for the from_json function
val chgSchema = Encoders.product[MyChgClass].schema
val beforeSchema = Encoders.product[MyFullClass].schema
// Your dataframe from the example
val df = Seq(
(1, """{"b": "new", "c": "new"}""", """{"k": 1, "b": "old", "c": "old", "d": "old"}""" ),
(2, """{"b": "new", "d": "new"}""", """{"k": 2, "b": "old", "c": "old", "d": "old"}""" )
).toDF("id", "chg", "before")
// Parsing the json string into our case classes and finishing by creating a
// strongly typed dataset with the .as[] method
val parsedDf = df
.withColumn("parsedChg",from_json(col("chg"), chgSchema))
.withColumn("parsedBefore",from_json(col("before"), beforeSchema))
.drop("chg")
.drop("before")
.as[(Int, MyChgClass, MyFullClass)]
// Mapping over our dataset with a lot of control of exactly what we want. Since
// the "chg" fields are options, we can use the getOrElse method to choose
// between either the "chg" field or the "before" field
val output = parsedDf.map{
case (id, chg, before) => {
MyEndClass(id, MyFullClass(
before.k,
chg.b.getOrElse(before.b),
chg.c.getOrElse(before.c),
chg.d.getOrElse(before.d)
))
}
}
output.show(false)
parsedDf.printSchema()
We have many such situations, but with differing payload. I can get the fields of the case class, but cannot see the forest for the trees how to make this generic. E,g, [T] type approach for the below. I am wondering if this can be done in fact?
I can get a List of attributes, and am wondering if something like attrList.map(x => ...) with substitution can be used for the chg.b etc?
val output = parsedDf.map{
case (id, chg, before) => {
MyEndClass(id, MyFullClass(
before.k,
chg.b.getOrElse(before.b),
chg.c.getOrElse(before.c),
chg.d.getOrElse(before.d)
))
}
}
Does the following macro work for your use case?
// libraryDependencies += scalaOrganization.value % "scala-reflect" % scalaVersion.value
import scala.language.experimental.macros
import scala.reflect.macros.blackbox
def mkInstance[A, B](before: A, chg: B): A = macro mkInstanceImpl[A]
def mkInstanceImpl[A: c.WeakTypeTag](c: blackbox.Context)(before: c.Tree, chg: c.Tree): c.Tree = {
import c.universe._
val A = weakTypeOf[A]
val classAccessors = A.decls.collect {
case m: MethodSymbol if m.isCaseAccessor => m
}
val arg = q"$before.${classAccessors.head}"
val args = classAccessors.tail.map(m => q"$chg.${m.name}.getOrElse($before.$m)")
q"new $A($arg, ..$args)"
}
// in a different subproject
val output = parsedDf.map{
case (id, chg, before) => {
MyEndClass(id,
mkInstance(before, chg)
)
}
}
// scalacOptions += "-Ymacro-debug-lite"
// scalac: new MyFullClass(before.k, chg.b.getOrElse(before.b), chg.c.getOrElse(before.c), chg.d.getOrElse(before.d))
https://scastie.scala-lang.org/bXq5FHb3QuC5PqlhZOfiqA
Alternatively you can use Shapeless
// libraryDependencies += "com.chuusai" %% "shapeless" % "2.3.10"
import shapeless.{Generic, HList, LabelledGeneric, Poly2}
import shapeless.ops.hlist.{IsHCons, Mapped, ZipWith}
import shapeless.ops.record.Keys
def mkInstance[A, B, L <: HList, H, T <: HList, OptT <: HList, L1 <: HList, T1 <: HList, T2 <: HList, K <: HList](
before: A, chg: B
)(implicit
// checking that field names in tail of A are equal to field names in B
aLabelledGeneric: LabelledGeneric.Aux[A, L1],
bLabelledGeneric: LabelledGeneric.Aux[B, T2],
isHCons1: IsHCons.Aux[L1, _, T1],
keys: Keys.Aux[T1, K],
keys1: Keys.Aux[T2, K],
// checking that field types in B are Options of field types in tail of A
aGeneric: Generic.Aux[A, L],
isHCons: IsHCons.Aux[L, H, T],
mapped: Mapped.Aux[T, Option, OptT],
bGeneric: Generic.Aux[B, OptT],
zipWith: ZipWith.Aux[OptT, T, getOrElsePoly.type, T],
): A = {
val aHList = aGeneric.to(before)
aGeneric.from(isHCons.cons(isHCons.head(aHList), zipWith(bGeneric.to(chg), isHCons.tail(aHList))))
}
object getOrElsePoly extends Poly2 {
implicit def cse[A]: Case.Aux[Option[A], A, A] = at(_ getOrElse _)
}
Since all the classes are now known at compile-time it's better to use compile-time reflection (macros themselves or macros hidden inside type classes as in Shapeless) but in principle runtime reflection also can be used
import scala.reflect.ClassTag
import scala.reflect.runtime.{currentMirror => rm}
import scala.reflect.runtime.universe._
def mkInstance[A: TypeTag : ClassTag, B: TypeTag : ClassTag](before: A, chg: B): A = {
val A = typeOf[A]
val B = typeOf[B]
val classAccessors = A.decls.collect {
case m: MethodSymbol if m.isCaseAccessor => m
}.toList
val arg = rm.reflect(before).reflectMethod(classAccessors.head)()
val args = classAccessors.tail.map(m =>
rm.reflect(chg).reflectMethod(B.decl(m.name).asMethod)()
.asInstanceOf[Option[_]].getOrElse(
rm.reflect(before).reflectMethod(m)()
)
)
rm.reflectClass(A.typeSymbol.asClass)
.reflectConstructor(A.decl(termNames.CONSTRUCTOR).asMethod)(arg :: args : _*)
.asInstanceOf[A]
}
I'm solving exercises for a programming book in Kotlin. The task is to implement function using "zip()" and return a "List" of Pairs, where the first item in a "Pair" is the element, and the second item is the index of that element.
I solved the exercise, the solution works but I cannot understand the book solution.
Here is mine solution:
fun zipWithIndex(listToTake: List<Any>): List<Pair<Any, Any>> {
val finalList = mutableListOf<Any>()
var num = 0
for(element in listToTake) {
finalList += num
num ++
}
return (listToTake zip finalList)
}
fun main() {
val listToCall = listOf<String>("a", "b", "c")
println(zipWithIndex(listToCall))
}
And here is the book solution:
fun <T> List<T>.zipWithIndex(): List<Pair<T, Int>> =
zip(indices)
fun main() {
val list = listOf('a', 'b', 'c')
list.zipWithIndex() eq
"[(a, 0), (b, 1), (c, 2)]"
}
Can somebody please explain how does the book solution get the indexes of the elements in the list or tell me the topic that I need to read about to figure out how the code from the book works.
Thanks in advance for any help.
indices is a property of every kotlin List: https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/-list/#extension-properties
It's an IntRange of all valid indices, so essentially the range (https://kotlinlang.org/docs/ranges.html) equivalent of [0, 1, 2]. An IntRange is an Iterable, so it can be zipped with (the third zip overload in the api docs of list).
So it is equivalent to the zip you did, except you constructed [0, 1, 2] yourself while they used the pre-existing property of the List.
They also defined an extension function on List (https://kotlinlang.org/docs/extensions.html#extension-functions) instead of passing the list as a parameter.
I am playing with kotlin language and I tried the following code:
data class D( val n: Int, val s: String )
val lst = listOf( D(1,"one"), D(2, "two" ) )
val res = lst.reduce { acc:String, d:D -> acc + ", " + d.toString() }
The last statement causes the following errors:
Expected parameter of type String
Expected parameter of type String
Type mismatch: inferred type is D but String was expected
while this version of the last statement works:
val res = lst.map { e -> e.toString() }.reduce { acc, el -> acc + ", " + el }
I do not understand why the first version does not work. The formal definition of the reduce function, found here, is the following:
inline fun <S, T : S> Iterable<T>.reduce(
operation: (acc: S, T) -> S
): S
But this seems in contrast with the following sentence, on the same page:
Accumulates value starting with the first element and applying
operation from left to right to current accumulator value and each
element.
That is, as explained here:
The difference between the two functions is that fold() takes an
initial value and uses it as the accumulated value on the first step,
whereas the first step of reduce() uses the first and the second
elements as operation arguments on the first step.
But, to be able to apply the operation on first and second element, and so on, it seems to me tha the operation shall have both arguments of the base type of the Iterable.
So, what am I missing ?
Reduce is not the right tool here. The best function in this case is joinToString:
listOf(D(1, "one"), D(2, "two"))
.joinToString(", ")
.let { println(it) }
This prints:
D(n=1, s=one), D(n=2, s=two)
reduce is not designed for converting types, it's designed for reducing a collection of elements to a single element of the same type. You don't want to reduce to a single D, you want a string. You could try implementing it with fold, which is like reduce but takes an initial element you want to fold into:
listOf(D(1, "one"), D(2, "two"))
.fold("") { acc, d -> "$acc, $d" }
.let { println(it) }
However, this will add an extra comma:
, D(n=1, s=one), D(n=2, s=two)
Which is exactly why joinToString exists.
You can see the definition to understand why its not working
To make it work, you can simply create an extension function:
fun List<D>.reduce(operation: (acc: String, D) -> String): String {
if (isEmpty())
throw UnsupportedOperationException("Empty list can't be reduced.")
var accumulator = this[0].toString()
for (index in 1..lastIndex) {
accumulator = operation(accumulator, this[index])
}
return accumulator
}
you can use it as:
val res = lst.reduce { acc:String, d:D -> acc + ", " + d.toString() }
or simply:
val res = lst.reduce { acc, d -> "$acc, $d" }
You can modify the function to be more generic if you want to.
TL;DR
Your code acc:String is already a false statement inside this line:
val res = lst.reduce { acc:String, d:D -> acc + ", " + d.toString() }
Because acc can only be D, never a String! Reduce returns the same type as the Iterable it is performed on and lst is Iterable<D>.
Explanation
You already looked up the definition of reduce
inline fun <S, T : S> Iterable<T>.reduce(
operation: (acc: S, T) -> S
): S
so lets try to put your code inside:
lst is of type List<D>
since List extends Iterable, we can write lst : Iterable<D>
reduce will look like this now:
inline fun <D, T : D> Iterable<T>.reduce(
operation: (acc: D, T) -> D //String is literally impossible here, because D is not a String
): S
and written out:
lst<D>.reduce { acc:D, d:D -> }
This question already has answers here:
Kotlin Error: Unresolved reference: merge (for list.merge())
(2 answers)
Closed 3 years ago.
i'm reading anotonio leava book about kotlin
there is a function named "merge" it can merge two list with each other
this is a code that author write in his book:
val list = listOf(1,2,3,4,5,6)
val anotherlist = listOf(7,8,9,10,11,12)
assertEquals(listOf(8,10,12,14,16,18),list.merge(anotherlist){it1,it2 - > it1+it2})
it means merge is plusing two list
but when i write code myself
my idea intelij doesnt know the func name
and it doesnt work
can you help me
sorry because of my english
You can do it with the use of map
val x: List<Int> = listOf(1,2,3,4,5,6)
val y: List<Int> = listOf(2,3,4,5,6,7)
val sums = (x.indices).map { x[it] + y[it] }
output
[3, 5, 7, 9, 11, 13]
instead of map you can use zip function
fun main(args: Array<String>) {
val list1 = listOf(4, 5, 6)
val list2 = listOf(1, 2, 3)
val result = list1.zip(list2) { e1, e2 -> e1 + e2 }
println(result) // prints [5, 7, 9]
}
merge is neither a member function of List, nor does the kotlin standard library provide an extension function. But you can define one by your own (and I suspect, that's what the author did)
fun <E> List<E>.merge(anotherlist: List<E>, operation: (E, E) -> E): List<E> {
// you probably want to check cardinalities of the two lists
return indices.map { operation(this[it], anotherlist[it]) }
}
Kotlin often uses very pragmatic approaches. I wonder whether there is some I don't know of to simplify a filter predicate which just asks for some known values.
E.g. consider the following list:
val list = listOf("one", "two", "2", "three")
To filter out "two" and "2" filtering can be accomplished in several ways, e.g.:
list.filter {
it in listOf("two", "2") // but that creates a new list every time... (didn't check though)
}
// extracting the list first, uses more code... and may hide the list somewhere sooner or later
val toCheck = listOf("two", "2")
list.filter { it in toCheck }
// similar, but probably less readable due to naming ;-)
list.filter(toCheck::contains)
// alternative using when, but that's not easier for this specific case and definitely longer:
list.filter {
when (it) {
"two", "2" -> true
else -> false
}
}
// probably one of the simplest... but not so nice, if we need to check more then 2 values
list.filter { it == "two" || it == "2" }
I wonder... is there something like list.filter { it in ("two", "2") } or any other simple way to create/use a short predicate for known values/constants? In the end that's all I wanted to check.
EDIT: I just realised that the sample doesn't make much sense as listOf("anything", "some", "other").filter { it in listOf("anything") } will always be just: listOf("anything"). However, the list intersection makes sense in constellations where dealing with, e.g. a Map. In places where the filter actually doesn't return only the filtered value (e.g. .filterKeys). The subtraction (i.e. list.filterNot { it in listOf("two", "2") }) however also makes sense in lists as well.
Kotlin provides some set operations on collections which are
intersect (what both collections have in common)
union (combine both collections)
subtract (collections without elements of the other)
In your case, instead of filter, you may use the set operation subtract
val filteredList = list.subtract(setOf("two","2"))
and there you go.
EDIT:
and the fun (pun intended) doesn't end there: you could extend the collections with your own functions such as a missing outerJoin or for filtering something like without or operators i.e. / for intersect
For example, by adding these
infix fun <T> Iterable<T>.without(other Iterable<T>) = this.subtract(other)
infix fun <T> Iterable<T>.excluding(other Iterable<T>) = this.subtract(other)
operator fun <T> Iterable<T>.div(other: Iterable<T>) = this.intersect(other)
Your code - when applied to your example using the intersect - would become
val filtered = list / filter //instead of intersect filter
or - instead of substract:
val filtered = list without setOf("two", "2")
or
val filtered = list excluding setOf("two", "2")
Pragmatic enough?
I ended up with the following now:
fun <E> containedIn(vararg elements: E) = { e:E -> e in elements }
fun <E> notContainedIn(vararg elements: E) = { e:E -> e !in elements }
which can be used for maps & lists using filter, e.g.:
list.filter(containedIn("two", "2"))
list.filter(notContainedIn("two", "2"))
map.filterKeys(containedIn("two", "2"))
map.filterValues(notContainedIn("whatever"))
In fact it can be used for anything (if you like):
if (containedIn(1, 2, 3)(string.toInt())) {
My first approach inspired by Gerald Mückes answer, but with minus instead of subtract (so it only covers the subtraction-part):
(list - setOf("two", "2"))
.forEach ...
Or with own extension functions and using vararg:
fun <T> Iterable<T>.without(vararg other: T) = this - other
with the following usage:
list.without("two", "2")
.forEach... // or whatever...
With the above variant however no infix is possible then. For only one exclusion an infix can be supplied as well... otherwise the Iterable-overload must be implemented:
infix fun <T> Iterable<T>.without(other : T) = this - other
infix fun <T> Iterable<T>.without(other : Iterable<T>) = this - other
Usages:
list without "two"
list without listOf("two", "2")
I don't think there is anything simpler than to create the filtering list/set and then apply it:
val toCheck = listOf("two", "2")
val filtered = list.filter { it in toCheck }
or
val toCheck = setOf("two", "2")
val filtered = list.filter { it in toCheck }
but if you prefer you can create a Predicate:
val predicate: (String) -> Boolean = { it in listOf("2", "two") }
val filtered = list.filter { predicate(it) }
Edit: as for the approach with minus, which is not the case here but has been mentioned, it does not provide simplicity or efficiency since itself is using filter:
/**
* Returns a list containing all elements of the original collection except the elements contained in the given [elements] collection.
*/
public operator fun <T> Iterable<T>.minus(elements: Iterable<T>): List<T> {
val other = elements.convertToSetForSetOperationWith(this)
if (other.isEmpty())
return this.toList()
return this.filterNot { it in other }
}
(from Collections.kt)