Databricks Connect: Unable to run scala program in databricks cluster from IntelliJ - intellij-idea

Followed the steps mentioned in this docs. databricks-connect test command works fine. However, when I launch the test scala program from Intellij, I'm seeing following error:
Exception in thread "main" java.lang.NoSuchMethodError: com.databricks.spark.util.MetricDefinitions$.EVENT_INITIAL_CONFIG_LOG()Lcom/databricks/spark/util/MetricDefinition;
at com.databricks.spark.util.InitialConfigLogging$.recordInitialConfigs(InitialConfigLogging.scala:47)
at org.apache.spark.sql.internal.SQLConf.recordSessionInitialConfs(SQLConf.scala:4166)
at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:150)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1033)
at com.srikanth.Demo$.main(Demo.scala:13)
at com.srikanth.Demo.main(Demo.scala)
Environment details:
python - 3.8
java - 1.8
databricks-connect - 9.1

I had the same error message, turns out I did not add the databricks-connect library properly. You should add the library like described here: https://docs.databricks.com/dev-tools/databricks-connect.html#intellij-scala-or-java

Related

Problems in IntelliJ since version 2022.1 / java.lang.NoClassDefFoundError

As of today, when I try to run some of my programs in IntelliJ today, this message was shown:
Executing pre-compile tasks...
Loading Ant configuration...
Running Ant tasks...
Running 'before' tasks
Checking sources
Finished, saving caches…
Module 'Listen' production: java.lang.NoClassDefFoundError: com/intellij/DynamicBundle
Executing post-compile tasks...
Loading Ant configuration...
Running Ant tasks...
Synchronizing output directories...
13.04.22, 14:54 - Build completed with 1 error and 0 warnings in 4 sec, 39 ms
This line is printed in red: Module 'Listen' production: java.lang.NoClassDefFoundError: com/intellij/DynamicBundle
I don't really have too much of an idea where I should look for the problem.
It seems to not have anything to do with the project itself. I already created a new Project and created a Hello World-program, which resulted in the same error.
Remarkably, yesterday my IntelliJ IDEA version was updated to version 2022.1.
Runtime version: 11.0.14.1+1-b2043.25 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
I have no idea either whether it has anything to do with the new version either.
I maintain another Project, which uses Scala language. That one is messed up even more:
Cannot process toolwindow sbt
java.lang.AbstractMethodError: Missing implementation of resolved method 'abstract com.intellij.openapi.externalSystem.settings.AbstractExternalSystemSettings getSettings(com.intellij.openapi.project.Project)' of abstract class com.intellij.openapi.externalSystem.service.task.ui.AbstractExternalSystemToolWindowFactory.
at com.intellij.openapi.externalSystem.service.task.ui.AbstractExternalSystemToolWindowFactory.shouldBeAvailable(AbstractExternalSystemToolWindowFactory.java:35)
at com.intellij.toolWindow.ToolWindowSetInitializerKt.beanToTask(ToolWindowSetInitializer.kt:262)
at com.intellij.toolWindow.ToolWindowSetInitializerKt.beanToTask(ToolWindowSetInitializer.kt:248)
at com.intellij.toolWindow.ToolWindowSetInitializerKt.access$beanToTask(ToolWindowSetInitializer.kt:1)
at com.intellij.toolWindow.ToolWindowSetInitializerKt$computeToolWindowBeans$1.accept(ToolWindowSetInitializer.kt:278)
at com.intellij.toolWindow.ToolWindowSetInitializerKt$computeToolWindowBeans$1.accept(ToolWindowSetInitializer.kt)
at com.intellij.openapi.extensions.impl.ExtensionPointImpl.processWithPluginDescriptor(ExtensionPointImpl.java:301)
at com.intellij.toolWindow.ToolWindowSetInitializerKt.computeToolWindowBeans(ToolWindowSetInitializer.kt:274)
at com.intellij.toolWindow.ToolWindowSetInitializer$1.get(ToolWindowSetInitializer.kt:101)
at com.intellij.toolWindow.ToolWindowSetInitializer$1.get(ToolWindowSetInitializer.kt:83)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
at java.base/java.lang.Thread.run(Thread.java:829)
Any advice?
Best
Stapelüberläufer
I had the same problem after upgrading to Intellij 2022.1
For me, the fix came from Gen's answer. I also had an old Scala plugin, and just removed it as I'm not using it anymore.
If you are looking to upgrade it, go to File -> Settings -> Plugins. From there just hit Upgrade for the plugins you want. After that restart the IDE.
I found that Scala plugin was not updated and updating it to at least version 2022.1.13 fixed it for me.
I upgraded to Intellij 2022.1 and encountered this issue when trying to sync the old gradles projects that used to work for me :
Caused by: java.lang.NoClassDefFoundError: org/gradle/internal/impldep/com/google/common/collect/Lists
Cleaning the caches folder's content (Gradle_HOME/caches) resolved this issue for me
Make sure to also check that the SDK is correctly configured for your project (Java 17 by default) and the Gradle home is well specified in the Settings>Gradle section.
For info, I'm using Java 8 and Gradle 5.5.1 (yes I know :))
Just upgrade the Scala Plugin and its done!
visiting
scala plugin's page
and install it from there worked for me on ultimate version 2022.2.3

Karate 0.9.1 - Exception in thread "main" java.lang.StackOverflowError

When I'm trying to run *.feature file or a single scenario with "right-click" (IntelliJ Idea), I've always received an exception:
Exception in thread "main" java.lang.StackOverflowError
at java.util.HashMap.<init>(HashMap.java:457)
at java.util.LinkedHashMap.<init>(LinkedHashMap.java:347)
at java.util.HashSet.<init>(HashSet.java:162)
at java.util.LinkedHashSet.<init>(LinkedHashSet.java:154)
at jdk.nashorn.internal.runtime.ScriptObject$KeyIterator.init(ScriptObject.java:2467)
at jdk.nashorn.internal.runtime.ScriptObject$ScriptObjectIterator.hasNext(ScriptObject.java:2441)
at jdk.nashorn.api.scripting.ScriptObjectMirror$13.call(ScriptObjectMirror.java:368)
at jdk.nashorn.api.scripting.ScriptObjectMirror$13.call(ScriptObjectMirror.java:363)
at jdk.nashorn.api.scripting.ScriptObjectMirror.inGlobal(ScriptObjectMirror.java:858)
at jdk.nashorn.api.scripting.ScriptObjectMirror.entrySet(ScriptObjectMirror.java:363)
at net.minidev.json.reader.JsonWriter$7.writeJSONString(JsonWriter.java:135)
at net.minidev.json.reader.JsonWriter$7.writeJSONString(JsonWriter.java:128)
at com.intuit.karate.JsonUtils$NashornObjectJsonWriter.writeJSONString(JsonUtils.java:77)
at com.intuit.karate.JsonUtils$NashornObjectJsonWriter.writeJSONString(JsonUtils.java:67)
...
Same scenario works fine if I run with TestRunner.java. Looks like that problem is in IJ cucumber plugin.
Maybe someone has a solution or workaround for this issue.
I'm using karate 0.9.1, cucumber for Java plugin: v183.4284.148, Idea 2018.3.3
No one has reported this - and from the stack-trace it looks like some JSON that you are using is being printed to the console - has some circular references, maybe you are using a Map of object references. But yes I can't explain why this works fine in the runner.
Can you follow the instructions here and submit a sample project, we can try open it in IntelliJ: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue
EDIT: this was a cyclic reference, but we have fixed it to be safe in future.

NoClassDefFoundError in Liquibase

Getting an Exception when initializing Liquibase. We are using version 3.3.2 (because offline is broken in 3.3.3)
java.lang.NoClassDefFoundError: Could not initialize class liquibase.servicelocator.ServiceLocator
at liquibase.logging.LogFactory.getLog(LogFactory.java:44)
at liquibase.logging.LogFactory.getLogger(LogFactory.java:37)
at liquibase.integration.spring.SpringLiquibase.<init>(SpringLiquibase.java:142)
We are running java 8, and it seems like this error is only visible when using sun-java-1.8.0_latest-x86_64-1.8.0-45.x86_64, not sun-java-1.8.0_latest-x86_64-1.8.0-12.x86_64.
I have created https://liquibase.jira.com/browse/CORE-2372, but maybe some of you guys have some tips

PlayORM Example compilation error

Samples run fine on my Windows 7.
However I get the following error on my Ubuntu machine:
Exception in thread "main" java.lang.NoSuchMethodError: org.joda.time.LocalDateTime.toDate()Ljava/util/Date;
at com.alvazan.orm.api.base.spi.UniqueKeyGenerator.generateKey(UniqueKeyGenerator.java:47)
at com.alvazan.orm.api.base.spi.UniqueKeyGenerator.generateNewKey(UniqueKeyGenerator.java:42)
at com.alvazan.orm.api.base.spi.UniqueKeyGenerator.generateNewKey(UniqueKeyGenerator.java:21)
at com.alvazan.orm.impl.meta.data.MetaIdField.fillInAndFetchId(MetaIdField.java:106)
at com.alvazan.orm.impl.meta.data.MetaIdField.translateToColumn(MetaIdField.java:56)
at com.alvazan.orm.impl.meta.data.MetaClassSingle.translateToRow(MetaClassSingle.java:82)
at com.alvazan.orm.layer0.base.BaseEntityManagerImpl.putImpl(BaseEntityManagerImpl.java:117)
at com.alvazan.orm.layer0.base.BaseEntityManagerImpl.put(BaseEntityManagerImpl.java:100)
at com.example.PlayORMExample.createTestData(PlayORMExample.java:89)
at com.example.PlayORMExample.main(PlayORMExample.java:31)
I looked at similar questions on the "stack", but they do not seem to apply to my case.
you either
have the wrong version of the joda-time jar OR
have two joda time jars one of which is the wrong version
later,
Dean (I am a committer on PlayOrm).

NullPointerException org.gradle.wrapper.BootstrapMainStarter.findLauncherJar(BootstrapMainStarter.java:37)

Got the following stacktrace when launching gradle 1.1, anyone know how to resolve them:
Exception in thread "main" java.lang.NullPointerException
at org.gradle.wrapper.BootstrapMainStarter.findLauncherJar(BootstrapMainStarter.java:37)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:28)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:130)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:47)
I think the automatic unzip of the dists/gradle-1.1-bin/13d7lnhcrghv2i5e54el41jpgr/gradle-1.1-bin.zip might be failing. I checked permissions and that I have access to that directory.
If I unzip manually, then I get the following error:
Exception in thread "main" java.lang.RuntimeException: Gradle distribution 'http://services.gradle.org/distributions/gradle-1.1-bin.zip' contains too many directories. Expected to find exactly 1 directory.
at org.gradle.wrapper.Install.createDist(Install.java:73)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:47)
I did a google search for gradle nullpointerexception and it mentioned the JAVA_HOME needs to be set for compiling, but I've already checked it is set correctly and been able to compile stuff with ant in that environment.
I was getting exactly same error and I changed the version of gradle that I was using. Inside my gradle-wrapper.properties, changed version 2.4 to 2.2.1 and error is gone.