I am trying to use Holden Karau's spark-testing-base using sbt and get 4 errors. It looks like sbt is generating invalid references to 4 jars.
The errors are:
[error] 4 not found
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.test-jar
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-hdfs/2.6.0/hadoop-hdfs-2.6.0.test-jar
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.6.0/hadoop-mapreduce-client-jobclient-2.6.0.test-jar
[error] https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-server-tests/2.6.0/hadoop-yarn-server-tests-2.6.0.test-jar
My build.sbt contains:
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "2.2.1",
"org.scalacheck" %% "scalacheck" % "1.12.4",
"com.holdenkarau" %% "spark-testing-base" % "1.6.1_0.3.3"
)
parallelExecution in Test := false
lazy val root = (project in file(".")).
settings(
name := "core",
version := "1.0",
scalaVersion := "2.11.8"
)
And my test class is from Holden's wiki example:
import org.scalatest._
import com.holdenkarou.SharedSparkContext
class SampleTest extends FunSuite with SharedSparkContext {
test("test initializing spark context") {
val list = List(1, 2, 3, 4)
val rdd = sc.parallelize(list)
assert(rdd.count === list.length)
}
}
When I execute sbt test I get a bunch of maven updates followed by the errors listed above.
It appears that sbt is generating invalid jar names - for example https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.test-jar should be hadoop-common-2.6.0-test.jar (i.e. the separator after the 2.6.0 should be -, not .).
Can someone guide me as to why this is occurring, and how to fix it?
Related
I am testing the idea of making my dsl Jvm compatible and I wanted to test the possibility of extending Xbase and using the interpreter. I have tried to make a minimal test project to use with the interpreter but I am getting a runtime error. I think I understand the general concepts of adapting Xbase, but am unsure about how the setup/entrypoints for the interpreter and could not find any information regarding the error I am getting or how to resolve. Here are the relevant files for my situation:
Text.xtext:
import "http://www.eclipse.org/xtext/xbase/Xbase" as xbase
import "http://www.eclipse.org/xtext/common/JavaVMTypes" as types
Program returns Program:
{Program}
'program' name=ID '{'
variables=Var_Section?
run=XExpression?
'}'
;
Var_Section returns VarSection:
{VarSection}
'variables' '{'
decls+=XVariableDeclaration+
'}'
;
#Override // Change syntax
XVariableDeclaration returns xbase::XVariableDeclaration:
type=JvmTypeReference name=ID '=' right=XLiteral ';'
;
#Override // Do not allow declarations outside of variable region
XExpressionOrVarDeclaration returns xbase::XExpression:
XExpression;
TestJvmModelInferrer:
def dispatch void infer(Program element, IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
acceptor.accept(element.toClass(element.fullyQualifiedName)) [
documentation = element.documentation
if (element.variables !== null) {
for (decl : element.variables.decls) {
members += decl.toField(decl.name, decl.type) [
static = true
initializer = decl.right
visibility = JvmVisibility.PUBLIC
]
}
}
if (element.run !== null) {
members += element.run.toMethod('main', typeRef(Void::TYPE)) [
parameters += element.run.toParameter("args", typeRef(String).addArrayTypeDimension)
visibility = JvmVisibility.PUBLIC
static = true
body = element.run
]
}
]
}
Test case:
#Inject ParseHelper<Program> parseHelper
#Inject extension ValidationTestHelper
#Inject XbaseInterpreter interpreter
#Test
def void basicInterpret() {
val result = parseHelper.parse('''
program program1 {
variables {
int var1 = 0;
double var2 = 3.4;
}
var1 = 13
}
''')
result.assertNoErrors
var interpretResult = interpreter.evaluate(result.run)
println(interpretResult.result)
Partial stack trace:
java.lang.IllegalStateException: Could not access field: program1.var1 on instance: null
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter._assignValueTo(XbaseInterpreter.java:1262)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.assignValueTo(XbaseInterpreter.java:1221)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter._doEvaluate(XbaseInterpreter.java:1213)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.doEvaluate(XbaseInterpreter.java:216)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.internalEvaluate(XbaseInterpreter.java:204)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.evaluate(XbaseInterpreter.java:190)
at org.eclipse.xtext.xbase.interpreter.impl.XbaseInterpreter.evaluate(XbaseInterpreter.java:180)
The interpreter does only support expressions, but does not work with types that are created by a JvmModelInferrer. Your code tries to work with fields of such an inferred type.
Rather than using the interpreter, I'd recommend to use an InMemoryCompiler in your test. The domainmodel example may serve as an inspiration: https://github.com/eclipse/xtext-eclipse/blob/c2b15c3ec118c4c200e2b28ea72d8c9116fb6800/org.eclipse.xtext.xtext.ui.examples/projects/domainmodel/org.eclipse.xtext.example.domainmodel.tests/xtend-gen/org/eclipse/xtext/example/domainmodel/tests/XbaseIntegrationTest.java
You may find this project interesting, which (among other stuff) implements an interpreter for Xtend based on the Xbase interpreter. It might be a bit outdated, though, and also will not fully support all Xtend concepts. But it could be a starting point, and your contrbutions are welcome :-)
https://github.com/kbirken/xtendency
I am struggling to get the sbt-proguard plugin to work. I have a class library that I want to obfuscate but I cannot seem to get the plugin to output without the above error. I have specified the keep option, or at lease I think that I have, but I have had no luck. I copied the keep options from the Proguard website which said it was meant for class libraries. In addition, I do not think the plug-in is responding the the options that I have configured.
For example, I wanted to have more verbose output to see if the output could give me a clue as to what I am doing wrong. However, whenever I look at the log files, it always specifies the default options. Below is my configuration. Can someone help me out with this one? I am completely lost. Thanks
import sbt.Keys._
import com.typesafe.sbt.SbtProguard._
import ProguardKeys._
lazy val commonDependencies = Seq(
Dependencies.Libraries.junit,
Dependencies.Libraries.springBootLogging,
Dependencies.Libraries.scalaMock,
Dependencies.Libraries.joda,
Dependencies.Libraries.scalaTestPlus,
Dependencies.Libraries.scalaXml,
Dependencies.Libraries.commonsCodec,
Dependencies.Libraries.typeSafeConfig
)
val keepClasses =
"""
|-injars in.jar
|-outjars out.jar
|-libraryjars <java.home>/lib/rt.jar
|-printmapping out.map
|
|-keepparameternames
|-renamesourcefileattribute SourceFile
|-keepattributes Exceptions,InnerClasses,Signature,Deprecated,
| SourceFile,LineNumberTable,*Annotation*,EnclosingMethod
|
|-keep public class * {
| public protected *;
|}
|
|-keepclassmembernames class * {
| java.lang.Class class$(java.lang.String);
| java.lang.Class class$(java.lang.String, boolean);
|}
|
|-keepclasseswithmembernames,includedescriptorclasses class * {
| native <methods>;
|}
|
|-keepclassmembers,allowoptimization enum * {
| public static **[] values();
| public static ** valueOf(java.lang.String);
|}
|
|-keepclassmembers class * implements java.io.Serializable {
| static final long serialVersionUID;
| private static final java.io.ObjectStreamField[] serialPersistentFields;
| private void writeObject(java.io.ObjectOutputStream);
| private void readObject(java.io.ObjectInputStream);
| java.lang.Object writeReplace();
| java.lang.Object readResolve();
|}
""".stripMargin
proguardSettings
lazy val skedaddleCore = (project in file(".")).
settings(BuildSettings.buildSettings: _*).
settings(
name := "core",
resolvers := Resolvers.all,
libraryDependencies ++= commonDependencies,
merge in Proguard := true,
proguardVersion in Proguard := "5.2.1",
options in Proguard --= Seq("-dontnote", "-dontwarn", "-ignorewarnings"),
options in Proguard ++= Seq("-verbose", "-dontshrink"),
options in Proguard += keepClasses
)
I finally got the plugin to do what I wanted with the following configuration.
import sbt.Keys._
import com.typesafe.sbt.SbtProguard._
lazy val commonDependencies = Seq(
Dependencies.Libraries.junit,
Dependencies.Libraries.springBootLogging,
Dependencies.Libraries.scalaMock,
Dependencies.Libraries.joda,
Dependencies.Libraries.scalaTestPlus,
Dependencies.Libraries.scalaXml,
Dependencies.Libraries.commonsCodec,
Dependencies.Libraries.typeSafeConfig
)
proguardSettings
ProguardKeys.proguardVersion in Proguard := "5.2.1"
ProguardKeys.options in Proguard ++= Seq("-dontnote", "-dontwarn", "-ignorewarnings")
ProguardKeys.inputs in Proguard <<= (dependencyClasspath in Compile) map { _.files }
ProguardKeys.filteredInputs in Proguard <++= (packageBin in Compile) map ProguardOptions.noFilter
val keepClasses =
"""
|-keepparameternames
|-keepattributes Exceptions,InnerClasses,Signature,Deprecated,
| SourceFile,LineNumberTable,*Annotation*,EnclosingMethod
|
|-keep,includedescriptorclasses interface com.** {
| <methods>;
|}
""".stripMargin
ProguardKeys.options in Proguard += keepClasses
lazy val skedaddleCore = (project in file(".")).
settings(BuildSettings.buildSettings: _*).
settings(
name := "core",
resolvers := Resolvers.all,
libraryDependencies ++= commonDependencies
)
I'm dealing with a problem working on a Play project and sbt.
The problem is:
I have tests, some of them are using PowerMockito, some of them aren't
For the tests using PowerMockito, I've added #RunWith(PowerMockRunner.class) at the beginning of the class.
For the tests not using, I've added nothing.
Here is an example for a test that uses PowerMockito
#RunWith(PowerMockRunner.class)
#PrepareForTest({SomeMockedClass.class})
public class SomeClassTest {
#Test
public void aTest() {
PowerMockito
.stub(PowerMockito.method(SomeMockedClass.class, "aMethod"))
.toReturn(something);
assertThat(someCall).isEqualTo(something);
}
}
Here is an example for a test that doesn't use PowerMockito
public class AnotherClassTest {
#Test
public void anotherTest(){
assertThat(something).isEqualTo(something);
}
}
And here is my build.sbt dep:
libraryDependencies ++= Seq(
cache,
...
// Testing
"org.easytesting" % "fest-assert" % "1.4" % "test",
"junit" % "junit" % "4.12" % "test",
"org.powermock" % "powermock-mockito-release-full" % "1.6.2" % "test",
"org.powermock" % "powermock-module-junit4-rule-agent" % "1.6.2" % "test",
"org.easymock" % "easymock" % "3.3.1" % "test",
"com.novocode" % "junit-interface" % "0.11" % "test"
)
Here is the problem:
When I run my tests from IntelliJ, it founds all tests and everything is ok
When I run my tests from activator (activator test), it does NOT found the tests with the #RunWith.
I've read some post about it, like the sbt version or something
(even https://github.com/sbt/jacoco4sbt/issues/15).
But the sbt version is : 0.13.8-M5 so it seems ok.
Play version is : 2.3.8
If you guys have any clue that'd be great.
Thanks
I have problem setting up querydsl framework in play 2.2.6 with scala 2.10.3 and java 1.7
I have done installation exactly like it was in documentation. But it doesn't work.
I am gettign an error:
dany#dany1L:~/git/app$ playFramework-2.2.6
[info] Loading project definition from /home/dany/git/app/project
/home/dany/git/app/build.sbt:11: error: not found: value QueryDSLPlugin
val current = project.in(file(".")).configs(QueryDSLPlugin.QueryDSL)
^
[error] sbt.compiler.EvalException: Type error in expression
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
Here is my project/plugins.sbt:
// Comment to get more information during initialization
logLevel := Level.Warn
// The Typesafe repository
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
// Use the Play sbt plugin for Play projects
// changed to support play 2.2.4 addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.2.1")
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.2.6")
addSbtPlugin("com.code-troopers.play" % "play-querydsl" % "0.1.2")
And my build.sbt:
import com.typesafe.config._
import play.Project._
import sbt._
import Keys._
//javacOptions ++= Seq("-Xlint:unchecked")
playJavaSettings
playJavaSettings ++ QueryDSLPlugin.queryDSLSettings
val current = project.in(file(".")).configs(QueryDSLPlugin.QueryDSL)
val conf = ConfigFactory.parseFile(new File("conf/application.conf")).resolve()
name := conf.getString("app.name")
version := conf.getString("app.version")+"_("+conf.getString("app.releaseDate")+")"
libraryDependencies ++= Seq(
javaJdbc,
javaJpa,
"org.hibernate" % "hibernate-entitymanager" % "3.6.9.Final",
"mysql" % "mysql-connector-java" % "5.1.27",
"org.mindrot" % "jbcrypt" % "0.3m",
"org.jasypt" % "jasypt" % "1.9.2",
"org.apache.poi" % "poi" % "3.10.1",
"com.googlecode.genericdao" % "dao" % "1.2.0",
"com.googlecode.genericdao" % "search-jpa-hibernate" % "1.2.0",
"com.google.code.gson" % "gson" % "2.3.1",
"com.googlecode.json-simple" % "json-simple" % "1.1.1",
"javax.mail" % "javax.mail-api" % "1.5.3",
"javax.activation" % "activation" % "1.1.1",
"com.sun.mail" % "javax.mail" % "1.5.3",
"com.querydsl" % "querydsl-jpa" % "4.0.2",
"com.querydsl" % "querydsl-apt" % "4.0.2",
cache
)
Please give me some help.
After adding:
import codetroopers._
on top of build.sbt I am getting an error:
[info] Loading project definition from /home/dany/git/app/project
error: bad symbolic reference. A signature in QueryDSLPlugin.class refers to type AutoPlugin
in package sbt which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling QueryDSLPlugin.class.
[error] sbt.compiler.EvalException: Type error in expression
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
Thanks to #Nathan
and his answer here
Finally after few days of struggle I've make it working.
Here are my configuration files:
plugins.sbt
// Comment to get more information during initialization
logLevel := Level.Warn
// The Typesafe repository
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
// Use the Play sbt plugin for Play projects
// changed to support play 2.2.4 addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.2.1")
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.2.6")
addSbtPlugin("com.code-troopers.play" % "play-querydsl" % "0.1.1")
and build.sbt
import com.typesafe.config._
import play.Project._
import sbt._
import Keys._
//javacOptions ++= Seq("-Xlint:unchecked")
playJavaSettings
val conf = ConfigFactory.parseFile(new File("conf/application.conf")).resolve()
name := conf.getString("app.name")
version := conf.getString("app.version")+"_("+conf.getString("app.releaseDate")+")"
libraryDependencies ++= Seq(
javaJdbc,
javaJpa,
"org.hibernate" % "hibernate-entitymanager" % "3.6.9.Final",
"mysql" % "mysql-connector-java" % "5.1.27",
"org.mindrot" % "jbcrypt" % "0.3m",
"org.jasypt" % "jasypt" % "1.9.2",
"org.apache.poi" % "poi" % "3.10.1",
"com.googlecode.genericdao" % "dao" % "1.2.0",
"com.googlecode.genericdao" % "search-jpa-hibernate" % "1.2.0",
"com.google.code.gson" % "gson" % "2.3.1",
"com.googlecode.json-simple" % "json-simple" % "1.1.1",
"javax.mail" % "javax.mail-api" % "1.5.3",
"javax.activation" % "activation" % "1.1.1",
"com.sun.mail" % "javax.mail" % "1.5.3",
"com.querydsl" % "querydsl-jpa" % "4.0.2",
"com.querydsl" % "querydsl-apt" % "4.0.2",
cache
)
playJavaSettings ++ QueryDSLPlugin.queryDSLSettings
val current = project.in(file(".")).configs(QueryDSLPlugin.QueryDSL)
QueryDSLPlugin.queryDSLPackage := "models"
and build.properties
sbt.version=0.13.0
I am having difficulties getting sbt (version 0.12.1) to recognize any tests in src/test/scala.
I have tried both JUnit style tests and scalatest style tests but to no avial
To make things simple
I have moved my tests to the root package (src/test/scala)
I have included both org.scalatest and junit-interface in my build.sbt
libraryDependencies ++= List(
"org.scalatest" %% "scalatest" % "1.8" % "test",
"com.novocode" % "junit-interface" % "0.8" % "test->default"
)
I have made the tests as simple as possible:
scalatest example
import org.scalatest.FunSuite
import scala.collection.mutable.Stack
class ExampleSuite extends FunSuite {
test("math still works") {
assert(1+1 == 2)
}
}
junit test example:
import org.junit.Assert._
import org.junit.Test
class SimpleTest {
#Test
def testPass() {
assertEquals(1+1, 2)
}
}
my test structure is:
src/test/scala
├── FunSuiteExample.scala
└── SimpleTest.scala
What am I missing?
based on instructions at:
https://github.com/szeiger/junit-interface
modfied build.sbt
removed "junit" % "junit" % "4.10" % "test" from build.sbt
added "com.novocode" % "junit-interface" % "0.11" % "test"
put test in src/test/scala
import org.junit._
import org.junit.Assert._
class SimpleTeset {
#Test
def testTrue() {
assertEquals(1+1, 2)
}
}