error while building apache spark - maven-2

i get the following error while building apache spark on open suse. any pointers?
test:/siva/spark-1.1.0 # sbt/sbt assembly
[info] Loading project definition from /siva/spark-1.1.0/project/project
[info] Loading project definition from /root/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Loading project definition from /siva/spark-1.1.0/project
org.apache.maven.model.building.ModelBuildingException: 1 problem was encountered while building the effective model for org.apache.spark:spark-parent:1.1.0
[FATAL] Non-resolvable parent POM: Could not transfer artifact org.apache:apache:pom:14 from/to central ( http://repo.maven.apache.org/maven2): Error transferring file: repo.maven.apache.org from http://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom and 'parent.relativePath' points at wrong local POM # line 21, column 11
at org.apache.maven.model.building.DefaultModelProblemCollector.newModelBuildingException(DefaultModelProblemCollector.java:195)
at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:841)
....
at xsbt.boot.Boot.main(Boot.scala)
[error] org.apache.maven.model.building.ModelBuildingException: 1 problem was encountered while building the effective model for org.apache.spark:spark-parent:1.1.0
[error] [FATAL] Non-resolvable parent POM: Could not transfer artifact org.apache:apache:pom:14 from/to central ( http://repo.maven.apache.org/maven2): Error transferring file: repo.maven.apache.org from http://repo.maven.apache.org/maven2/org/apache/apache/14/apache-14.pom and 'parent.relativePath' points at wrong local POM # line 21, column 11
[error] Use 'last' for the full log.
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: backward-delete-word
[INFO] Unable to bind key for unsupported operation: down-history
[INFO] Unable to bind key for unsupported operation: up-history
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
building the spark did not throw such error in ubuntu and redhat.

Related

How to install Apache Atlas 1.1.0 on MacOS Mojave?

I am trying to setup Apache Atlas on my system. I am encountering the following error
I read the article as suggested (http://cwiki.apache.org/confluence/display/MAVEN/PluginConfigurationException) but I was unable to understand what it was trying to convey.
[INFO] >>> findbugs-maven-plugin:3.0.1:check (findbugs-check) > :findbugs # apache-atlas >>>
[INFO]
[INFO] --- findbugs-maven-plugin:3.0.1:findbugs (findbugs) # apache-atlas ---
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass$3$1 (file:/Users/r0r017e/.m2/repository/org/codehaus/groovy/groovy-all/2.4.0/groovy-all-2.4.0.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass$3$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
...
[INFO] Apache Atlas Server Build Tools 1.0 ................ SUCCESS [ 0.632 s]
[INFO] apache-atlas 1.1.0 ................................. FAILURE [ 3.341 s]
...
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:3.0.1:findbugs (findbugs) on project apache-atlas: Unable to parse configuration of mojo org.codehaus.mojo:findbugs-maven-plugin:3.0.1:findbugs for parameter pluginArtifacts: Cannot assign configuration entry 'pluginArtifacts' with value '${plugin.artifacts}' of type java.util.Collections.UnmodifiableRandomAccessList to property of type java.util.ArrayList -> [Help 1]
I need to setup Apache Atlas via Cassandra as a backend store. I am new to data governance and open source solutions, any help would be appreciated!
If you are just testing Atlas I suggest using a docker image. The following worked for me. https://github.com/michalmiklas/atlas-docker

Error Compiling Scala Spark using SBT Console in IntelliJ IDEA

I tried to compile Scala Spark project that I got using get-idea following this tutorial https://anitatailor.wordpress.com/2013/10/09/setting-up-spark0-8-0-with-intellijidea/
But when I tried to compile Scala Spark using SBT Console in IntelliJ IDEA, I got some errors.
You may see the error logs below
java -Dsbt.log.noformat=true -Djline.terminal=jline.UnsupportedTerminal -Xmx512M -XX:MaxPermSize=256M -jar C:\Users\Priska\.IdeaIC14\system\sbt\sbt-launch.jar
[info] Loading project definition from D:\spark-incubating\project\project
[info] Compiling 1 Scala source to D:\spark-incubating\project\project\target\scala-2.9.2\sbt-0.12\classes...
[error] error while loading CharSequence, class file 'D:\IDEA\jre\jre\lib\rt.jar(java/lang/CharSequence.class)' is broken
[error] (bad constant pool tag 15 at byte 1501)
[error] error while loading Comparator, class file 'D:\IDEA\jre\jre\lib\rt.jar(java/util/Comparator.class)' is broken
[error] (bad constant pool tag 15 at byte 5003)
[error] two errors found
[error] (compile:compile) Compilation failed
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
Can you help me to solve the problem?
Any help would be appreciated.
Thank you

Why does building Spark sources give "object sbt is not a member of package com.typesafe"?

I tried to compile the https://github.com/apache/spark project using IntelliJ IDEA with the sbt plugin on Windows.
I'm facing an error about sbt. Since I'm not familiar with sbt, I don't know how to fix it.
The error messages are as follows:
[info] Loading project definition from F:\codeReading\sbtt\spark-master\project
[info] Compiling 3 Scala sources to F:\codeReading\sbtt\spark-master\project\target\scala-2.10\sbt-0.13\classes...
[error] F:\codeReading\sbtt\spark-master\project\SparkBuild.scala:26: object sbt is not a member of package com.typesafe
[error] import com.typesafe.sbt.pom.{PomBuild, SbtPomKeys}
[error] ^
[error] F:\codeReading\sbtt\spark-master\project\SparkBuild.scala:51: not found: type PomBuild
[error] object SparkBuild extends PomBuild {
[error] ^
[error] F:\codeReading\sbtt\spark-master\project\SparkBuild.scala:118: not found: value SbtPomKeys
[error] otherResolvers <<= SbtPomKeys.mvnLocalRepository(dotM2 => Seq(Resolver.file("dotM2", dotM2))),
[error] ^
[error] F:\codeReading\sbtt\spark-master\project\SparkBuild.scala:178: value projectDefinitions is not a member of AnyRef
[error] super.projectDefinitions(baseDirectory).map { x =>
[error] ^
[error] four errors found
[error] (plugins/compile:compile) Compilation failed
Spark is built with Maven. The SBT build is only a convenience. You will have far better results importing it as a Maven project.
It looks like IDEA doesn't like project references to git projects that the Spark build definition uses for sbt-pom-reader.
It's showed up when I ran sbt within the cloned project:
➜ spark git:(master) ✗ xsbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/oss/spark/project/project
[info] Loading project definition from /Users/jacek/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Loading project definition from /Users/jacek/oss/spark/project
[info] Set current project to spark-parent (in build file:/Users/jacek/oss/spark/)
>
You can see the project reference to the git project of sbt-pom-reader when accessing the plugins project for which the reference is defined:
> reload plugins
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/oss/spark/project/project
[info] Updating {file:/Users/jacek/oss/spark/project/project/}project-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Loading project definition from /Users/jacek/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Updating {file:/Users/jacek/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project/}sbt-pom-reader-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Loading project definition from /Users/jacek/oss/spark/project
> projects
[info] In file:/Users/jacek/oss/spark/project/
[info] * plugins
[info] spark-style
[info] In https://github.com/ScrapCodes/sbt-pom-reader.git
[info] sbt-pom-reader
A solution could be executing sbt gen-idea to generate the project files for IDEA. It's just a guess, though.

Binary incompatibility in plugins detected in Play 2.1.3 project after upgrading to IDEA 13.1?

After updating IntelliJ IDEA to 13.1 last night I can't use Run/Debug config anymore. It happens in a project I've been working on in IDEA 13.0.x for the past couple of months -- now it shows me the message and I have no idea how I can solve this.
[info] Loading project definition from /www/play20apps/cnproject/project
[error] sbt.IncompatiblePluginsException: Binary incompatibility in plugins detected.
[error] Note that conflicts were resolved for some dependencies:
[error] asm:asm
[error] asm:asm-tree
[error] asm:asm-util
[error] org.avaje.ebeanorm:avaje-ebeanorm-agent
[error] jline:jline
[error] junit:junit
[error] com.jcraft:jsch
[error] commons-logging:commons-logging
[error] commons-codec:commons-codec
[error] Use 'last' for the full log.
[error] java.lang.ExceptionInInitializerError
[error] Use 'last' for the full log.
[error] Not a valid command: run
[error] run
[error] ^
Process finished with exit code 1
I didn't change Play version since yesterday of course and running application with common command line works as expected.
Also tried to create new run config after plugins' update with no luck.
Please advise.
Adding -Djline.terminal= to JVM Options should fix it.
last log reports this.
The relevant part:
Caused by: java.lang.IllegalArgumentException: java.lang.ClassNotFoundException: none
at jline.Terminal.setupTerminal(Terminal.java:65)
at jline.Terminal.getTerminal(Terminal.java:26)
at sbt.JLine$.sbt$JLine$$terminal(LineReader.scala:65)
where none is probably passed via -Djline.terminal=none.

maven in 5 min not working

I have downloaded latest
maven 3.0.3, java version "1.6.0_18".
mvn –version
Apache Maven 3.0.3 (r1075438; 2011-02-28 18:31:09+0100)
Maven home: C:\software\apache-maven-3.0.3
Java version: 1.5.0_05, vendor: Sun Microsystems Inc.
Java home: C:\software\jdk1.5.0.15\jre
Default locale: en_GB, platform encoding: Cp1252
OS name: "windows xp", version: "5.1", arch: "x86", family: "windows"
My application located at: C:\Temp\ecm_esv\app\ENT_APP\ESV>. I am getting following error message:
C:\Temp\ecm_esv\app\ENT_APP\ESV>mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
[INFO] Scanning for projects...
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-clean-plugin:2.4.1: Plugin org.apache.maven.plugins:maven-clean-plugin:2.4.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-clean-plugin:jar:2.4.1
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3.1/maven-install-plugin-2.3.1.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-install-plugin:2.3.1: Plugin org.apache.maven.plugins:maven-install-plugin:2.3.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-install-plugin:jar:2.3.1
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-deploy-plugin/2.5/maven-deploy-plugin-2.5.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-deploy-plugin:2.5: Plugin org.apache.maven.plugins:maven-deploy-plugin:2.5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-deploy-plugin:jar:2.5
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-site-plugin/2.0.1/maven-site-plugin-2.0.1.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-site-plugin:2.0.1: Plugin org.apache.maven.plugins:maven-site-plugin:2.0.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-site-plugin:jar:2.0.1
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-antrun-plugin/1.3/maven-antrun-plugin-1.3.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-antrun-plugin:1.3: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.3 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-antrun-plugin:jar:1.3
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/2.2-beta-5/maven-assembly-plugin-2.2-beta-5.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5: Plugin org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-assembly-plugin:jar:2.2-beta-5
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.1/maven-dependency-plugin-2.1.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-dependency-plugin:2.1: Plugin org.apache.maven.plugins:maven-dependency-plugin:2.1 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-dependency-plugin:jar:2.1
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.0/maven-release-plugin-2.0.pom
[WARNING] Failed to retrieve plugin descriptor for org.apache.maven.plugins:maven-release-plugin:2.0: Plugin org.apache.maven.plugins:maven-release-plugin:2.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-release-plugin:jar:2.0
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading: http://repo1.maven.org/maven2/org/codehaus/mojo/maven-metadata.xml
[WARNING] Could not transfer metadata org.apache.maven.plugins/maven-metadata.xml from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org
[WARNING] Could not transfer metadata org.codehaus.mojo/maven-metadata.xml from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org
Downloading: http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloading: http://repo1.maven.org/maven2/org/codehaus/mojo/maven-metadata.xml
[WARNING] Could not transfer metadata org.apache.maven.plugins/maven-metadata.xml from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org
[WARNING] Could not transfer metadata org.codehaus.mojo/maven-metadata.xml from/to central (http://repo1.maven.org/maven2): Error transferring file: repo1.maven.org
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.406s
[INFO] Finished at: Mon Apr 18 15:51:05 CEST 2011
[INFO] Final Memory: 1M/3M
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'archetype' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (C:\Temp\ecm_esv\app\ENT_APP\ESV), central (http://repo1.maven.org/maven2)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException
C:\Temp\ecm_esv\app\ENT_APP\ESV>
C:\software\apache-maven-3.0.3\conf\settings.xml
I have enabled proxies here as:
….
<proxies>
<!-- proxy
| Specification for one proxy, to be used in connecting to the network.
|-->
<proxy>
<id>optional</id>
<active>true</active>
<protocol>http</protocol>
<host>www.BT.co.uk</host>
<port>80</port>
<nonProxyHosts>localhost|google.com</nonProxyHosts>
</proxy>
</proxies>
What is the problem I am facing here? Why it is not downloading?
Found solution:
Solution #1:
issue was with proxy in settings.xml file:
<host>webproxy</host>
to get host goto IE->tools->connection->LAN settings->advanced->http.
=====
Solution #2:
if auto configured proxy is given: then
1> open IE(or any browser)
2> get the url address from your browser through IE->Tools->internet option->connections->LAN Settings-> get address and give in url eg: as http://autocache.abc.com/ and enter, a file will be downloaded with .pac format, save to desktop
3> open .pac file in textpad, identify PROXY:
In your editor, it will come something like:
return "PROXY web-proxy.ind.abc.com:8080; PROXY proxy.sgp.abc.com:8080";
4> go to Maven settings.xml and enter as:
<proxy>
<id>optional</id>
<active>true</active>
<protocol>http</protocol>
<host>web-proxy.ind.abc.com</host>
<port>8080</port>
</proxy>
5> run mvn:install through command prompt or run through eclipse.
=====
Solution #3:
For any other issues delete maven local repository and run mvn:install again.
Clearly this is an issue of proxy. Either use VPN or edit procy setting in setting.xml.
I searched lot about this error. Finally, I used VPN(Virtual private network ) to disable the proxy and was able to use cemtral repository of Maven.
Window-Preferences-Maven-User Settings. Check, does settings.xml exists? if eclipse says no, get settings.xml from you friend or set the path to it.
Try checking proxy settings in settings.xml file located under following directory:
C:\Users\UserName\.m2
I just copied setting.xml which comes under "\apache-maven-3.1.1\conf" directory, to the directory "${user.home}/.m2/)" of windows machine and change the required proxy setting according to my network as below and it worked for me.
<proxy>
<id>optional</id>
<active>true</active>
<protocol>http</protocol>
<host>www-proxy.xxxx</host>
<port>8080</port>
</proxy>
Thanks --
Kamran Shahzad
We have observed that the maven project creation fails when it tries to download the archtype from this URL: Maven Repository which falls under "https" protocol.
So solution #2 mentioned above worked only after a little change to the settings.xml. Instead of using "http" protocol I have to use "https"
<proxy>
<id>optional</id>
<active>true</active>
<protocol>https</protocol>
<host>web-proxy.ind.abc.com</host>
<port>8080</port>
Hope it helps someone.
Note: Make sure that you check the following internet settings.
Internet Explorer > Internet Options > Connections [TAB] > LAN Settings > Under Proxy server check Use proxy server for LAN > Advanced > Check "same proxy server for all protocols" and Click OK.
Adding below under proxy tag in settings.xml worked for me
<proxy>
<id>optional</id>
<active>true</active>
<protocol>https</protocol>
<host>web-proxy.ind.abc.com</host>
<port>8080</port>
</proxy>