Apache beam seems to be refusing to recognise Kotlin's Iterable. Here is a sample code:
#ProcessElement
fun processElement(
#Element input: KV<String, Iterable<String>>, receiver: OutputReceiver<String>
) {
val output = input.key + "|" + input.value.toString()
println("output: $output")
receiver.output(output)
}
I get the following weird error:
java.lang.IllegalArgumentException:
...PrintString, #ProcessElement processElement(KV, OutputReceiver), #ProcessElement processElement(KV, OutputReceiver):
#Element argument must have type org.apache.beam.sdk.values.KV<java.lang.String, java.lang.Iterable<? extends java.lang.String>>
Sure enough, if I replace Iterable with java.lang.Iterable, the same code works just fine. What am I doing wrong?
Version of depedencies:
kotlin-jvm: 1.3.21
org.apache.beam: 2.11.0
Here is a gist with full codes and stack trace:
https://gist.github.com/marcoslin/e1e19afdbacac9757f6974592cfd8d7f#file-apache-beam-iterable-notworking-kt
Update:
After a bit of trial and error, I found out that while List<String> throws similar exception but MutableList<String> actually works:
class PrintString: DoFn<KV<String, MutableList<String>>, String>() {
#ProcessElement
fun processElement(
#Element input: KV<String, MutableList<String>>, receiver: OutputReceiver<String>
) {
val output = input.key + "|" + input.value.toString()
println("output: $output")
receiver.output(output)
}
}
So, this reminded me that Kotlin's Immutable collection are actually only interface and that underlying collection is still mutable. However, attempt to replace Iterable with MutableIterable continue to raise the error.
Update 2:
I deployed my Kotlin Dataflow job using the MutableList per above and job failed with:
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.ClassCastException:
org.apache.beam.runners.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn$WindowReiterable cannot be cast to java.util.List
at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:184)
at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
I had to switch back to use java.lang.Iterable.
I ran into this problem as well, when using a ParDo following a GroupByKey. It turns out that a #JvmWildcard annotation is needed in the Iterable generic type when writing a transformation that accepts the result of a GroupByKey.
See the contrived example below that reads a file and groups by the first character of each line.
class BeamPipe {
class ConcatLines : DoFn<KV<String, Iterable<#JvmWildcard String>>, KV<String, String>>() {
#ProcessElement
fun processElement(#Element input: KV<String, Iterable<#JvmWildcard String>>, receiver: OutputReceiver<KV<String, String>>) {
receiver.output(KV.of(input.key, input.value.joinToString("\n")))
}
}
fun pipe(options: PipelineOptions) {
val file =
"testFile.txt"
val p = Pipeline.create(options)
p.apply(TextIO.read().from(file))
.apply("Key lines by first character",
WithKeys.of { line: String -> line[0].toString() }
.withKeyType(TypeDescriptors.strings()))
.apply("Group lines by first character", GroupByKey.create<String, String>())
.apply("Concatenate lines", ParDo.of(ConcatLines()))
.apply("Write to files", FileIO.writeDynamic<String, KV<String, String>>()
.by { it.key }
.withDestinationCoder(StringUtf8Coder.of())
.via(Contextful.fn(ProcessFunction { it.value }), TextIO.sink())
.to("whatever")
.withNaming { key -> FileIO.Write.defaultNaming(key, ".txt") }
)
p.run()
}
}
This looks like a bug in the Beam Kotlin SDK. The Reflection analysis for your #ProcessElement method does not work correctly. You can probably work around this by using ProcessContext ctx instead of using the #Element parameter.
I am not very familiar with kotlin but it seems that you need to import import java.lang.Iterable before using it in your code.
May I know how to fix the issue when we get the iterable from groupbykey.create(). i could not groupbykey as you did javalang iterable
For those are experiencing this issue and found their way here, my current workaround to keep writing the pipeline in kotlin is to create a Java static class with function(s) that creates, contains, and processes your Iterable(s). The result (in non-iterable format) can then be passed back to the kotlin.
Related
I am trying to create a dynamic service based on the data class model defined by user and they registerDataModels() method appDataModule() it should automatically create all based method in the router service. When I try to achive using generics in those method I am getting a compiler error. Is there any other better way to dynamically create route methods like by defining the datamodel by developer and then service should be automatically created?
org.jetbrains.kotlin.backend.common.BackendException: Backend Internal error: Exception during IR lowering
File being compiled: */api/AppConfigService.kt
The root cause java.lang.RuntimeException was thrown at: org.jetbrains.kotlin.backend.jvm.codegen.FunctionCodegen.generate(FunctionCodegen.kt:47)
File is unknown
The root cause java.lang.AssertionError was thrown at: org.jetbrains.kotlin.codegen.coroutines.CoroutineTransformerMethodVisitor.spillVariables(CoroutineTransformerMethodVisitor.kt:636)
fun Application.registerDataModels() {
appDataModule<M1>()
appDataModule<M2>()
appDataModule<M3>()
}
inline fun <reified T: DBModel> Application.appDataModule() {
routing {
createAppData<T>()
updateAppData<T>()
deleteAppData<T>()
}
}
inline fun <reified T: DBModel> Route.createAppData() {
put("/api/data/${getName<T>()}/create") {
authenticated {
create<T>{}
}
}
}
inline fun <reified T: DBModel> Route.updateAppData() {
put("/api/data/${getName<T>()}/update") {
authenticated {
update<T>{}
}
}
}
inline fun <reified T: DBModel> Route.deleteAppData() {
put("/api/data/${getName<T>()}/delete") {
authenticated {
delete<T>{}
}
}
}
Note: This answer assumed that code would be loaded at runtime, which seems not to be the case, and is therefore not completely matching OP's question.
You are using inline functions with reified.
To make a long story short, inline functions are compiled and 'copied' to the location where they are being used, already with a fixed (thats what reified does) class. So when you use an inline function
inline fun <reified T> foo(t: T): T { ... }
and you call it like this:
val myVal = foo("test").uppercase()
then at compile time of that calling line of code, the type of T is known to be String and the target line is compiled accordingly, so you know at runtime which type T is within your function.
It is (for this one calling line) as if that function was like this to begin with:
fun foo(t: String): String { ... }
Because you want to compile these classes dynamically, however, this process fails, because the class obviously does not exist yet. This is simply due to the nature of reified. If you can somehow remove it, it might work.
I agree that the error message of the compiler could be more telling here. Maybe you can raise a task on kotlin's issue tracking platform?: https://youtrack.jetbrains.com/issues/kt?_gl=1*5r6x4d*_ga*MTQyMDYxMjc2MS4xNjMzMzQwMzk5*_ga_9J976DJZ68*MTY2OTM1NjM1MS4yMS4xLjE2NjkzNTYzNTcuMC4wLjA.&_ga=2.265829455.1332696793.1669356352-1420612761.1633340399
I have Kotlin some code that works as a class but when I try and run it as a Kotlin script I am getting " error: 'getNameAndVersion' is a member and an extension at the same time. References to such elements are not allowed"
enum class Env { Test, Sit }
data class ImageVersions(val apiName: String, val versions: Map<Env, String?>)
fun String.getNameAndVersion() = substringBefore(':') to substringAfter(':')
val testList = listOf("api-car-v1:0.0.118", "api-dog-v1:0.0.11", "api-plane-v1:0.0.36")
val sitList = listOf("api-car-v1:0.0.119", "api-plane-v1:0.0.37", "api-dog-v1:0.0.12")
getVersions(
mapOf(
Env.Test to testList,
Env.Sit to sitList
)
).forEach(::println)
fun getVersions(envMap: Map<Env, List<String>>): List<ImageVersions> {
val envApiNameMap = envMap.mapValues { it.value.associate(String::getNameAndVersion) }
val allApiNames = envApiNameMap.flatMap { it.value.keys }.distinct()
return allApiNames.map { apiName ->
ImageVersions(apiName, envApiNameMap.mapValues { it.value[apiName] })
}
}
I don't think I'm doing anything wrong with the way I'm using the method reference but according to my compiler I'm wrong. Appreciate some help with this. thanks
kotlinc -script .\myscript.kts
error: 'getNameAndVersion' is a member and an extension at the same time. References to such elements are not allowed
I don't have any experience with scripts but this error occurs when you try to reference a function inside a class that is also an extension function. (Here it is pointing to String::getNameAndVersion). Maybe when you run a script, the entire code is wrapped inside a class and then executed.
To fix this you can do one of the following:
Convert the function to a normal function which accepts a String parameter.
fun getNameAndVersion(s: String) = s.substringBefore(':') to s.substringAfter(':')
And replace String::getNameAndVersion with just ::getNameAndVersion in associate function
Other option is to directly call the function in the associate's lambda instead of passing a reference of this function.
.associate { it.getNameAndVersion() }
Taking my first steps in Kotlin, I'm struggling to find the correct signature for a function that receives an instance of a known class along with the desired output class and then looks in a map of converter lambdas whether the conversion can be done.
Here's an example for Long:
private fun <T> castLong(value: Long, clazz: Class<out T>): T {
// map lookup removed for simplicity
return when (clazz) {
String::class.java -> { value.toString() }
else -> { throw IllegalArgumentException("Unsupported Cast") }
}
}
Where T is the class of the desired return value - let's say String. One should be able to call castLong(aLongValue, String::class.java) and receive an instance of String.
But the compiler says:
Type mismatch: inferred type is String but T was expected
This seems like it should be possible as it is quite straightforward so far but even playing around with reified and other constructs didn't yield any better results.
It happens because it can't smart cast String to T, you have to manually cast it.
Furthermore, since you said you are taking your first steps in Kotlin, I leave here two other "advices" not strictly related to your question:
you can get the class of T making it reified
the brackets of a case using when aren't necessary if the case is one line
private inline fun <reified T> castLong(value: Long): T {
// map lookup removed for simplicity
return when (T::class.java) {
String::class.java -> value.toString()
else -> throw IllegalArgumentException("Unsupported Cast")
} as T
}
I have this code snippet in Java (this is an MCVE; actual code is more complex but has exact same issue):
enum StatusEnum { A, B, C; }
[...]
final static Set<String> names = Arrays.asList(StatusEnum.values())
.stream().map(StatusEnum::name).collect(Collectors.toSet());
IntelliJ gave me the following automated conversion to Kotlin:
internal val names = Arrays.asList(*StatusEnum.values())
.stream().map<String>(Function<StatusEnum, String> { it.name })
.collect<Set<String>, Any>(Collectors.toSet())
This unfortunately has compile errors:
Interface Function does not have constructors
Type inference failed. Expected type mismatch: inferred type is Collector!>! but Collector!>! was expected
Unresolved reference: it
This is my very first attempt at converting some code to Kotlin. I have reviewed the Functions and Lambdas section of the documentation. Still not clear what's going on here or how to fix it.
Use Kotlin methods instead of Java streams:
val names = StatusEnum.values()
.map { it.name }
.toSet()
Is the code (show below) correct? It was take from page 63 of the Kotlin-docs.pdf, which is also the last code snippet of https://kotlinlang.org/docs/reference/generics.html
fun <T> cloneWhenGreater(list: List<T>, threshold: T): List<T>
where T : Comparable, T : Cloneable {
return list.filter { it > threshold }.map { it.clone() }
}
Taken by as is, the compiler fails with:
1. One type argument expected for interface Comparable defined in kotlin
2. Type inference failed. Expected type mismatch: inferred type is List but List was expected
3. Cannot access 'clone': it is protected in 'Cloneable'
The first two errors are easily resolved by changing the code to the following:
fun <T> cloneWhenGreater(list: List<T>, threshold: T): List<Any>
where T : Comparable<in T>, T : Cloneable {
return list.filter { it > threshold }.map { it.clone() }
}
I still get the following error:
Cannot access 'clone': it is protected in 'Cloneable'
I'm using Kotlin 1.1.2-release-IJ2017.1-1
Am I missing something? or is there an error in the documentation?
Thanks.
it.clone() returns Any and you get error casting List to List.
So, you have changed it to List.
Your next error is Cannot access clone: it is protected in Cloneable.
This problem can be solved by creating our own Cloneable interface with public method.