I see that Scio fallsback to Kryo coder rather than Java Serializer which is default coder used for Dataflow when coder cannot be inferred/found via CoderRegistry. I don't see any reference to setFallbackCoderProvider anywhere, how does Scio registers the fallback to KryoAtomicCoder?
Up to Scio v0.6.x it happens here:
https://github.com/spotify/scio/blob/v0.6.1/scio-core/src/main/scala/com/spotify/scio/Implicits.scala#L55
Coder lookup and derivation has changed significantly since v0.7.x. It's still under heavy development but for more details see:
https://github.com/spotify/scio/wiki/Coders
Related
Simply, we are very sensitive to ABI changes, and we want to prevent people on our team using experimental features because of issues this can cause at runtime. Is there a way either via gradle configuration and/or a custom gradle plugin to detect when such features are used and elicit a compiler warning, preferably with a custom error message?
The Kotlin compiler (and transitively Gradle) will warn you by default (or even show an error) when experimental features are used.
To change this behaviour, the developer has to opt in via an #OptIn annotation or -opt-in compiler argument. If you're trying to avoid accidental usages of experimental features, this should therefore be sufficient (if we consider that opting in to an experimental feature is not an accident).
If you want to prevent usages of #OptIn itself, this is a different story. Technically #OptIn itself is still experimental at the moment, so there will be a warning for usages of it anyway unless someone adds the compiler argument to opt in to the #OptIn experimental annotation :) So currently just having a culture of not modifying compiler arguments should be enough, but that of course will change once #OptIn gets stable.
I am not aware of plugins that already do this, but you should be able to write a plugin with Kotlin Symbol Processing that checks for those annotations (never tried it myself, though).
As a side note about ABI compatibility, there is also the binary-compatibility-validator, but that's for your own code, so I don't think it's related to the question.
I use Python, but I don't know how it works in Kotlin. This is an example
example => exec("""print("hello")""") output => hello
exec("""print("hello")""") output => hello
Kotlin supports JSR-223. You can use the jvm scripting engine to eval kts files.
val engine = ScriptEngineManager().getEngineByExtension("kts")
engine.eval("""print("hello")""")
You need JSR-223 library dependency. Refer to example
implementation("org.jetbrains.kotlin:kotlin-scripting-jsr223:$kotlinVersion")
Short answer: this isn't practical in Kotlin.
Technically, there may be ways, but they're likely to be far more trouble than they're worth; you're far better looking for a different approach to your problem.
Unlike a dynamic (‘scripting’) language like Python, Kotlin is statically-compiled. In the case of Kotlin/JVM, you run the Kotlin compiler to generate .class files with Java bytecode, which is then run by a JVM.
So if you really need to convert a string into code and run it, you'd have to find a way to ensure that a Kotlin compiler is available on the platform where your code is running (which it often won't be; compiled bytecode can run on any platform with a JVM, and most of those won't have Kotlin installed too). You'd then have to find a way to run the compiler; this will probably mean writing your source code out to a file, starting up the compiler program as a separate process (as I don't think there's an API for calling it directly), and checking the results. Then you'd have find the resulting bytecode and load into the JVM, which will probably mean setting up a separate classloader instance.
All of which is likely to be slow, fragile, and very awkward.
(See these previous questions which cover some of the same ground.)
(The details will be different for Kotlin/JS and Kotlin/Native, but I think the principles are roughly the same.)
In general, each computer language has its own approach, its own mind-set and ways of doing things, and it's best to try to understand that and accept that patterns and techniques from one language don't always translate well into another. (In the Olden Days™, it used to be said that a determined programmer could write FORTRAN programs in any language — but only in satire.)
Perhaps if you could explain why you want to do this, and what sort of problem you're trying to solve (probably as a separate question), we might be able to suggest more natural solutions in Kotlin.
I'm trying to create a Kotlin Vert.x language support module and I need a way to compile Kotlin files and load the results with a ClassLoader. I've tried using kotlin-compiler library and found K2JVMCompiler class, but it seems to support only command-line-style arguments with its exec method. Is there a way to compile a Kotlin file in runtime (possibly without having to save and read .class files) and immediately load the generated classes? (Kind of like Groovy does.) If not, do you have any useful compiler arguments suggestions or pretty much any advices?
This feels like an XY Problem. You want to know how to compile Kotlin on the fly so that you can more easily use Vert.x by running from Kotlin source files instead of compiled code. But really the recommended path for Vert.x usage is to create a simple bit of code that deploys your verticle within compiled code.
In the question, your link for language support says Vert.x 2 in the path "vertx.io/vertx2/language_support.html"; which is different than how it is now done in Vert.x 3. I think you are merging two thoughts into one. First that Vert.x 3 wants you to run Java/Kotlin files from source (it doesn't really; that was a Vert.x 2 thing they moved away from for compiled languages), and second that you need custom language support (you don't).
You should try to use Vert.x 3 by running compiled code. To do so, build your classes and run your own main() that deploys a verticle programatically. Your code would be simple as:
import io.vertx.core.Vertx
fun main(args: Array<String>) {
val vertx = Vertx.vertx()
vertx.deployVerticle(SomeVerticleOfMine())
}
Alternatively, the docs for running and deploying from the command-line say:
Vert.x will compile the Java source file on the fly before running it. This is really useful for quickly prototyping verticles and great for demos. No need to set-up a Maven or Gradle build first to get going!
And really it is indeed just for prototyping and quick testing, and it isn't any faster than letting your IDE do the same and running from the compiled classes. You also then have debugging features of the IDE which are infinitely valuable.
For a few helper libraries for using Kotlin with Vert.x, view these options:
Vert.x 3 module for Klutter - I am the author, one of my libraries
Vert.x 3 helpers for Kotlin - by Cy6erGn0m
Kovert, a REST framework for Vert.x 3 - I am the author, one of my libraries
Vert.x nubes - not Kotlin specific, but makes Vert.x-Web friendlier for JVM languages.
There is a full sample project of running Vert.x + Kovert (specifically start with the App class). You can look at the code of Kovert to do your own similar work of starting and running Vert.x nicely, with Promises or however you wish. The docs for Kovert have links to code for starting Vertx and also starting a Verticle to use Vert.x-Web, so more sample code you can read. But it helps to understand Injekt (light-weight dependency registry), Kovenant (promises library), and Klutter configuration injection to understand the complete sample.
Other quick note, Vert.x has codegen support for other languages, but since you can call all of the Java version directly, it does not need to support Kotlin either.
Kotlin 1.1 comes with javax.script (JSR-223) support, which means you can use it as scripting engine similarly to JavaScript with Nashorn.
I am writing Python code to develop some Spark applications. I am really curious how Python interact with running JVM and started reading the source code of Spark.
I can see that in the end, all the Spark transformations/actions ended up be calling certain jvm methods in the following way.
self._jvm.java.util.ArrayList(),
self._jvm.PythonAccumulatorParam(host, port))
self._jvm.org.apache.spark.util.Utils.getLocalDir(self._jsc.sc().conf())
self._jvm.org.apache.spark.util.Utils.createTempDir(local_dir, "pyspark") \
.getAbsolutePath()
...
As a Python programmer, I am really curious what is going on with this _jvm object. However, I have briefly read all the source code under pyspark and only found _jvm to be an attribute of Context class, beyond that, I know nothing about neither _jvm's attributes nor methods.
Can anyone help me understand how pyspark translate into JVM operations? should I read some scala code and see if _jvm is defined there?
It uses py4j. There is a special protocol to translate python calls into JVM calls. All of this you can find in Pyspark code, see java_gateway.py
I have a script that uses quicklisp to load zs3 for accessing Amazon's S3.
When I run the script with clisp, when (zs3:bucket-exists-p "Test") is run, USOCKET:TIMEOUT-ERROR occurs.
However, when I run it with sbcl, it runs properly.
Do they access usocket differently?
What are the pros and cons of each?
usocket is a compatibility layer which hides the underlying socket API of each Lisp implementation. There is bound to be an impedance mismatch in some cases, but for the most part it should just work.
I suspect zs3 is not often used with CLISP (or perhaps not at all!), and you're seeing the result of that. On the other hand one can generally expect libraries to be well-tested under SBCL since that is the most popular implementation.
Note also that threads are still experimental in CLISP; they are not enabled by default. The fact that sockets are often mixed with threads only decreases the relative use of CLISP + usocket.