Nutch deployment on hadoop will not index to solr - apache

I have oozie workflow that does a nutch crawl I designed using hue.
All steps in the process work, except for indexing to solr.
The oozie action that defines the solrindex is as follows
`
<start to="solr-test"/>
<action name="solr-test">
<java>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<main-class>org.apache.nutch.indexer.IndexingJob</main-class>
<java-opts>solr.server.url=http://ip-redacted:8983/solr/raw</java-opts>
<arg>hdfs://ip-redacted:8020/user/admin/c</arg>
<arg>-dir</arg>
<arg>hdfs://ip-redacted:8020/user/admin/s000</arg>
</java>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
`
When I run the action I get the following error message
Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [-1]
The locations hdfs://ip-redacted:8020/user/admin/c and
hdfs://ip-redacted:8020/user/admin/s000 are locations that contain the crawldb and the segments respectively.
The stderr of the job says ::
`Log Length: 122
Intercepting System.exit(-1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], exit code [-1]`
The syslog says::
`ERROR [main] org.apache.nutch.indexer.IndexingJob: Indexer: java.lang.RuntimeException: org.apache.nutch.indexer.IndexWriter not found.
at org.apache.nutch.indexer.IndexWriters.<init>(IndexWriters.java:51)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:100)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:185)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:195)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:55)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)`
have verified that the class exists in the apache-nutch-1.7.jar file.
And if I request hadoop to run as a map-reduce job in the command shell as follows::
`hadoop jar apache-nutch-1.7.jar org.apache.nutch.indexer.IndexingJob -D solr.server.url=http://ip-redacted:8983/solr/raw hdfs://ip-redacted:8020/user/admin/c -dir hdfs://ip-redacted:8020/user/admin/s000`
It works!! But, when I do it as a oozie job, created through Hue, it fails...
Also, other actions, like inject, generate, fetch, parse work fine in Hue. It's only solrindex step that fails and I don't know what to do to fix it. Any input on this will be great!

Did you put the Nutch jar (and dependencies if needed) in a 'lib' directory in the HDFS workspace of the workflow?

Ah, I'm beginning to loathe the packaging of Nutch!
Try extracting the classes/plugins folder from the job archive, copy it to HDFS (something like hdfs dfs -put -r plugins lib) and then add the HDFS path of the plugins folder to the "files" list of the indexing step.
Best,
Edoardo

Related

Exception in thread "main" java.lang.ClassNotFoundException error

I run hadoop jar /home/apache-nutch-2.3.1/runtime/deploy/apache-nutch-2.3.1.job org.apache.nutch.crawl.Crawl urls -dir crawl -depth 3 -topN 5
But I get the following error:
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.nutch.crawl.Crawl
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:316)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
I created a urls/seed.text file in /home/apache-nutch-2.3.1/build/ that contains the following URLs:
http://nutch.apache.org
http://apache.org
and I edited conf/regex-urlfilter.txt as follow:
+^http://([a-z0-9]*\.)*apache.org/
The class org.apache.nutch.crawl.Crawl has been removed since version 1.8. It's recommended to run the shell script bin/crawl instead. It will launch Hadoop jobs for every step of a crawl: inject, generate, fetch, parse, etc.

java.lang.NoClassDefFoundError spark-submit in yarn cluster mode, cluster being setup using Ambari

I'm using the spark-submit command as below:
spark-submit --class com.example.hdfs.spark.RawDataAdapter --master yarn --deploy-mode cluster --jars /home/hadoop/emr/deployment/server/emr-core-1.0-SNAPSHOT.jar home/hadoop/emr-spark-1.0-SNAPSHOT.jar hdfs://111.11.11.111:8020/user/hdfsinputfile.zip 8000
However, it gives me the error java.lang.NoClassDefFoundError: com/example/emr/parser/IParser3. Though the IParser3.class is present in emr-core-1.0-SNAPSHOT.jar. I don't understand why it throws that error. I tried several ways but couldn't succeed. How can I resolve this?
I am able to run the same command in client mode and also as a standalone spark application. Getting this error only when in yarn cluster mode.
Exception from container-launch. Container id: container_e37_1526066605784_0014_02_000001 Exit code: 15 Container exited with a non-zero exit code 15. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : g.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) at com.example.hdfs.spark.utils.SimpleClassLoader.loadJarFile(SimpleClassLoader.java:126) at com.example.hdfs.spark.utils.SimpleClassLoader.(SimpleClassLoader.java:38) at com.example.hdfs.spark.input RawInputFormat.loadPlugins(RawInputFormat.java:71) at com.example.hdfs.spark.RawDataAdapter.run(RawDataAdapter.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at com.example.hdfs.spark.RawDataAdapter.main(RawDataAdapter.java:33) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$anon$3.run(ApplicationMaster.scala:646) 18/05/14 14:00:13 ERROR ApplicationMaster: Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:423) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:282) at org.apache.spark.deploy.yarn.ApplicationMaster$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:768) at org.apache.spark.deploy.SparkHadoopUtil$anon$2.run(SparkHadoopUtil.scala:67) at org.apache.spark.deploy.SparkHadoopUtil$anon$2.run(SparkHadoopUtil.scala:66) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:766) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: java.util.concurrent.ExecutionException: Boxed Error at scala.concurrent.impl.Promise$.resolver(Promise.scala:55) at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$resolveTry(Promise.scala:47) at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244) at scala.concurrent.Promise$class.tryFailure(Promise.scala:112) at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153) at org.apache.spark.deploy.yarn.ApplicationMaster$anon$3.run(ApplicationMaster.scala:664) Caused by: java.lang.NoClassDefFoundError: com/example/emr/parser/IParser3 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) at com.example.hdfs.spark.utils.SimpleClassLoader.findClass(SimpleClassLoader.java:152) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) at com.example.hdfs.spark.utils.SimpleClassLoader.loadJarFile(SimpleClassLoader.java:126) at com.example.hdfs.spark.utils.SimpleClassLoader.(SimpleClassLoader.java:38) at com.example.hdfs.spark.input.RawInputFormat.loadPlugins(RawInputFormat.java:71) at com.example.hdfs.spark.RawDataAdapter.run(RawDataAdapter.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at com.example.hdfs.spark.RawDataAdapter.main(RawDataAdapter.java:33) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$anon$3.run(ApplicationMaster.scala:646) Failing this attempt. Failing the application.
Quoting from Spark Documentation :-
http://spark.apache.org/docs/latest/running-on-yarn.html
In client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
In cluster mode, the Spark driver runs inside an application master process which is managed by YARN on the cluster, and the client can go away after initiating the application
So in cluster mode, the jar is executed on any available node so , so you can try these 2 ways :-
1) Copy the dependency jar to each node .
2) You can try to copy the jar to Distributed (HDFS system) and then use it .
For more details you can have a look into :
https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management

PredictionIO - Error when trainning kmean clustering

I followed the guidance below to train and deploy KMean clustering.
But I got error with pio train:
[WARN] [Template$] template.json does not exist. Template metadata will not be available. (This is safe to ignore if you are not working on a template.)
[INFO] [Runner$] Submission command: /home/lavalamp/PredictionIO/vendors/spark-1.4.1/bin/spark-submit --class io.prediction.workflow.CreateWorkflow --jars file:/home/lavalamp/PredictionIO/MyKmeans/target/scala-2.10/template-scala-parallel-vanilla_2.10-0.1-SNAPSHOT.jar,file:/home/lavalamp/PredictionIO/MyKmeans/target/scala-2.10/template-scala-parallel-vanilla-assembly-0.1-SNAPSHOT-deps.jar --files file:/home/lavalamp/PredictionIO/conf/log4j.properties --driver-class-path /home/lavalamp/PredictionIO/conf file:/home/lavalamp/PredictionIO/lib/pio-assembly-0.9.4.jar --engine-id gYCE4NX4ODPQkryp9Jq9by3OEXxa4fxQ --engine-version b972fa8f340c142fb6dffbebc6d276b3bb32eeda --engine-variant file:/home/lavalamp/PredictionIO/MyKmeans/engine.json --verbosity 0 --json-extractor Both
--env PIO_ENV_LOADED=1,PIO_STORAGE_SOURCES_MYSQL_PASSWORD=123456,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/lavalamp/.pio_store,PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://192.168.1.73/pio,PIO_HOME=/home/lavalamp/PredictionIO,
PIO_FS_ENGINESDIR=/home/lavalamp/.pio_store/engines,PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=MYSQL,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=MYSQL,
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_MYSQL_USERNAME=root,PIO_FS_TMPDIR=/home/lavalamp/.pio_store/tmp,
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=MYSQL,
PIO_CONF_DIR=/home/lavalamp/PredictionIO/conf
Exception in thread "main" java.lang.ClassCastException: com.biglabs.VanillaEngine$ cannot be cast to io.prediction.controller.EngineFactory
at io.prediction.workflow.WorkflowUtils$.getEngine(WorkflowUtils.scala:69)
at io.prediction.workflow.CreateWorkflow$.liftedTree1$1(CreateWorkflow.scala:193)
at io.prediction.workflow.CreateWorkflow$.main(CreateWorkflow.scala:192)
at io.prediction.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Can anyone help me with this issue?
try this solution, https://github.com/singsanj/KMeans-parallel-template
hope this solve your issues.
just dont forget to update the scripts/loadData.py with you newly created app access key and engine.json with your appId.
if you still have issues.. happy to solve.

Oozie hive script scheduling in tez mode

I tried to schedule the hive workflow xml file to run the hive script in tez mode by passing the hadoop properties for referring the tez jar files in workflow xml file as shown below.
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>hive-site.xml</job-xml>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>tez.lib.uris</name>
<value>${nameNode}/apps/Tez/,${nameNode}/apps/Tez/lib/</value>
</property>
</configuration>
Also I had changed the hive-site xml file property hive.execution.engine as tez mode.
<property>
<name>hive.execution.engine</name>
<value>tez</value>
</property>
When I scheduled the workflow using oozie. I got the error as follows
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, org/apache/tez/dag/api/SessionNotRunning
java.lang.NoClassDefFoundError: org/apache/tez/dag/api/SessionNotRunning
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:479)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:306)
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:290)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:295)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 24 more
Can any one please say how to rectify this issue and to schedule my workflow xml file and run the hive script in tez mode.
I observed the above error before and I was able to resolve it and was able to run Hive(tez engine) on Oozie.
Here's the steps I followed.
Class not found error:
As the error says, Oozie Launcher container cannot find the SessionNotRunning class.
This class is part of tez-api-0.x.x.jar. you can confirm that using :
jar tvf /usr/lib/tez/tez-api-0.7.0.jar | grep SessionNotRunning
You need to make sure your Oozie launcher container(which is YARN) localize this and other TEZ JAR's for it to pass it to HiveClient.
The expectation is that if we include the following config property in the workflow.xml , oozie should pick up all those JAR's.
<property>
<name>tez.lib.uris</name>
<value>hdfs:///apps/tez/,hdfs:///apps/tez/lib/</value>
</property>
However, it may not do that.(Not sure why)
So, I copied all TEZ JAR's to Hive-action's share library in HDFS (Ex: to /user/oozie/share/lib/lib_20160405125827/hive/). The oozie hive-action on your workflow should use JAR's present in that path and localize those JAR's.
While doing that, make sure the new JAR's have same permission as previous JAR's present in that HDFS directory. Oozie also need a refresh of share library.
Example commands can be:
hadoop fs -copyFromLocal /usr/lib/tez/*.jar /user/oozie/share/lib/lib_20160405125827/hive/
hadoop fs -copyFromLocal /usr/lib/tez/lib/*.jar /user/oozie/share/lib/lib_20160405125827/hive/
hadoop fs -chown oozie:oozie /user/oozie/share/lib/lib_20160405125827/hive/*.jar
oozie admin -sharelibupdate
Now, if you list your hive share library, oozie admin -shareliblist hive , you should be able to see all TEZ libraries.
With those steps, you should no longer see NoClassDefFoundError's or ClassNotFoundException's from TEZ jars.
Missing Hadoop Dependencies:
At This time, the TEZ job should be submitted, but there's another error that you may encounter on the OOZIE launcher .
14972 [uber-SubtaskRunner] ERROR org.apache.hadoop.hive.ql.exec.Task - Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1459860815404_0033 failed 2 times due to AM Container for appattempt_1459860815404_0033_000002 exited with exitCode: 1
looking at container logs, I see
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/service/AbstractService
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.AbstractService
This is because the my TEZ installation is minimal and doesn't come with hadoop dependencies.
https://github.com/apache/tez/blob/release-0.7.0/docs/src/site/markdown/install.md#hadoop-installation-dependent-installdeploy-instructions
So, you need to tell TEZ to use your cluster's hadoop libraries using the following property in your workflow.xml.
<property>
<name>tez.use.cluster.hadoop-libs</name>
<value>true</value>
</property>
So, with the above steps , I was able to run a hive script successfully on TEZ engine via Oozie.

jetty 9 and intellij won't start

I'm playing around with intellij for a new project and am having a bit of a problem. When I try to run jetty 9 from intellij 12, I get this error
"C:\Program Files\Java\jdk1.7.0_13\bin\java" -DSTOP.PORT=0 -Dcom.sun.management.jmxremote= -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -DOPTIONS=jmx -Didea.launcher.port=7553 "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA 12.0.4\bin" -Dfile.encoding=windows-1252 -classpath "start.jar;C:\Program Files (x86)\JetBrains\IntelliJ IDEA 12.0.4\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain org.eclipse.jetty.start.Main etc/jetty-jmx.xml C:\Users\willie\AppData\Local\Temp\context9038140457899104277config\jetty-contexts.xml
[2013-02-17 01:49:18,784] Artifact Armor:war exploded: Server is not connected. Press 'Deploy' to start deployment.
2013-02-17 01:49:20.055:WARN:oejx.XmlParser:main: FATAL#file:/C:/Code/jetty-9.0.0.RC0/start.d/ line:1 col:1 : org.xml.sax.SAXParseException; systemId: file:/C:/Code/jetty-9.0.0.RC0/start.d/; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:453)
at org.eclipse.jetty.start.Main.start(Main.java:595)
at org.eclipse.jetty.start.Main.main(Main.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.xml.sax.SAXParseException; systemId: file:/C:/Code/jetty-9.0.0.RC0/start.d/; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:198)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:441)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:368)
at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1388)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:998)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:116)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302)
at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:204)
at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:220)
at org.eclipse.jetty.xml.XmlConfiguration.<init>(XmlConfiguration.java:138)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1209)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1160)
... 12 more
Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information
Process finished with exit code -2
Disconnected from server
This is with the base system. I have deleted the profile, recreated it, I've downloaded another copy of jetty, I've done everything I can think of. The curious thing is the only thing in the start.d folder is the default test ini file that came with the distribution. I opened the xml file that ini file references and there is nothing wrong with it. I'm completely stumped. Does anyone know what could be going on here?
Update,
jetty runs just fine from the command line, but not from Intellij
Update 2, It seems that the way IntelliJ does its deployments is thru the plugin, it creates a new jetty-contexts.xml file. That file seems to be what is not liked by jetty. here is the file
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">
<Configure class="org.eclipse.jetty.server.Server" id="Server">
<Ref id="DeploymentManager">
<Call name="addAppProvider">
<Arg>
<New class="org.eclipse.jetty.deploy.providers.ContextProvider">
<Set name="monitoredDir">C:\Users\willie\AppData\Local\Temp\context7950837742823871110deploy</Set>
<Set name="scanInterval">1</Set>
</New>
</Arg>
</Call>
</Ref>
</Configure>
and a new dump, now that i've deleted the test wars and removed the start.d from the config.
"C:\Program Files\Java\jdk1.7.0_13\bin\java" -DSTOP.PORT=0 -Dcom.sun.management.jmxremote= -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -DOPTIONS=jmx -Didea.launcher.port=7538 "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA 12.0.4\bin" -Dfile.encoding=windows-1252 -classpath "start.jar;C:\Program Files (x86)\JetBrains\IntelliJ IDEA 12.0.4\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain org.eclipse.jetty.start.Main etc/jetty-jmx.xml C:\Users\willie\AppData\Local\Temp\context826007528789372946config\jetty-contexts.xml
[2013-02-17 05:38:31,987] Artifact Armor:war exploded: Server is not connected. Press 'Deploy' to start deployment.
2013-02-17 05:38:33.509:WARN:oejx.XmlConfiguration:main: Config error at <Call name="addAppProvider"><Arg>|????<New class="org.eclipse.jetty.deploy.providers.ContextProvider"><Set name="monitoredDir">C:\Users\willie\AppData\Local\Temp\context7950837742823871110deploy</Set><Set name="scanInterval">1</Set></New>|???</Arg></Call> java.lang.ClassNotFoundException: org.eclipse.jetty.deploy.providers.ContextProvider in file:/C:/Users/willie/AppData/Local/Temp/context826007528789372946config/jetty-contexts.xml
2013-02-17 05:38:33.510:WARN:oejx.XmlConfiguration:main: Config error at <Ref id="DeploymentManager"><Call name="addAppProvider"><Arg>|????<New class="org.eclipse.jetty.deploy.providers.ContextProvider"><Set name="monitoredDir">C:\Users\willie\AppData\Local\Temp\context7950837742823871110deploy</Set><Set name="scanInterval">1</Set></New>|???</Arg></Call></Ref> java.lang.ClassNotFoundException: org.eclipse.jetty.deploy.providers.ContextProvider in file:/C:/Users/willie/AppData/Local/Temp/context826007528789372946config/jetty-contexts.xml
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:453)
at org.eclipse.jetty.start.Main.start(Main.java:595)
at org.eclipse.jetty.start.Main.main(Main.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.deploy.providers.ContextProvider
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at org.eclipse.jetty.util.Loader.loadClass(Loader.java:100)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.nodeClass(XmlConfiguration.java:354)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.newObj(XmlConfiguration.java:743)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.itemValue(XmlConfiguration.java:1111)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.value(XmlConfiguration.java:1016)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.call(XmlConfiguration.java:710)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:407)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.refObj(XmlConfiguration.java:819)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:419)
at org.eclipse.jetty.xml.XmlConfiguration$JettyXmlConfiguration.configure(XmlConfiguration.java:344)
at org.eclipse.jetty.xml.XmlConfiguration.configure(XmlConfiguration.java:262)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1221)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1160)
... 12 more
Usage: java -jar start.jar [options] [properties] [configs]
java -jar start.jar --help # for more information
The use of org.eclipse.jetty.start.Main is meant for use from a full and complete jetty distribution.
It is just a bootstrap to establish the correct classpath and configuration xmls that will eventually start Jetty.
Your command line could (I have not tested this) look like this instead ...
"C:\Program Files\Java\jdk1.7.0_13\bin\java"
-Djetty.home=C:\path\to\jetty\distribution
-DSTOP.PORT=0
-Didea.launcher.port=7553
"-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA 12.0.4\bin"
-Dfile.encoding=windows-1252
-jar start.jar
"-Dpath=C:\Program Files (x86)\JetBrains\IntelliJ IDEA 12.0.4\lib\idea_rt.jar"
org.eclipse.jetty.start.Main
Note that you will likely need the following ...
To define jetty.home property so that jetty knows where to look for its files
To have the extra IntelliJ jars defined via the -Dpath=${classpath} configurable that jetty will pass through into the running server
To have the IntelliJ properties defined in the ${jetty.home}/start.ini
To have the OPTIONS command line defined in the ${jetty.home/start.ini
To have the various jmx properties defined in the ${jetty.home}/start.ini
Put your custom jetty-context.xml into the ${jetty.home}/webapps/ directory, or change the deployment directory (in the etc/jetty-deploy.xml) to where your jetty-context.xml file is located. Be sure that the reference to the war file make sense.
Finally, know that you can ask jetty what the server classpath will look like easily enough with the command line option --version which will list the classpath the server will use (in the order that the server will use it) based on what you have configured for the start bootstrap process.