Nutch 2.3 + Elasticsearch 2.1.1. Cannot load Elasticsearch dependencies - apache

Question

I'm trying to integrate Nutch 2.3 in order to push data to the latest Elasticsearch 2.1.1.
I started updating versions and dependencies in the following files:
indexer-elastic/plugin.xml
<plugin id="indexer-elastic" name="ElasticIndexWriter" version="1.0.0"
provider-name="nutch.apache.org">
<runtime>
<library name="indexer-elastic.jar">
<export name="*" />
</library>
<library name="elasticsearch-2.1.1.jar"/>
<library name="hppc-0.7.1.jar"/>
<library name="jackson-core-2.6.2.jar"/>
<library name="jackson-dataformat-cbor-2.6.2.jar"/>
<library name="jackson-dataformat-smile-2.6.2.jar"/>
<library name="jackson-dataformat-yaml-2.6.2.jar"/>
<library name="guava-18.0.jar"/>
<library name="compress-lzf-1.0.2.jar"/>
<library name="t-digest-3.0.jar"/>
<library name="jsr166e-1.1.0.jar"/>
<library name="commons-cli-1.3.1.jar"/>
<library name="netty-3.10.5.Final.jar"/>
<library name="joda-time-2.8.2.jar"/>
<library name="lucene-analyzers-common-5.3.1.jar"/>
<library name="lucene-backward-codecs-5.3.1.jar"/>
<library name="lucene-core-5.3.1.jar"/>
<library name="lucene-highlighter-5.3.1.jar"/>
<library name="lucene-join-5.3.1.jar"/>
<library name="lucene-memory-5.3.1.jar"/>
<library name="lucene-queries-5.3.1.jar"/>
<library name="lucene-queryparser-5.3.1.jar"/>
<library name="lucene-spatial-5.3.1.jar"/>
<library name="lucene-suggest-5.3.1.jar"/>
<library name="HdrHistogram-2.1.6.jar"/>
<library name="joda-convert-1.2.jar"/>
</runtime>
<requires>
<import plugin="nutch-extensionpoints" />
</requires>
<extension id="org.apache.nutch.indexer.elastic"
name="Elasticsearch Index Writer"
point="org.apache.nutch.indexer.IndexWriter">
<implementation id="ElasticIndexWriter"
class="org.apache.nutch.indexwriter.elastic.ElasticIndexWriter" />
</extension>
</plugin>
indexer-elastic/ivy.xml
<ivy-module version="1.0">
<info organisation="org.apache.nutch" module="${ant.project.name}">
<license name="Apache 2.0" />
<ivyauthor name="Apache Nutch Team" url="http://nutch.apache.org" />
<description>Apache Nutch</description>
</info>
<configurations>
<include file="../../..//ivy/ivy-configurations.xml" />
</configurations>
<publications>
<!--get the artifact from our module name -->
<artifact conf="master" />
</publications>
<dependencies>
<dependency org="org.elasticsearch" name="elasticsearch"
rev="2.1.1" conf="*->default" />
<dependency org="com.google.guava" name="guava" rev="18.0" />
</dependencies>
</ivy-module>
I also reworked org.apache.nutch.indexwriter.elastic.ElasticIndexWriter to work against the new interface of the elasticsearch 2.1.1 client.
So what is the problem?
It seems that the dependencies listed in indexer-elastic/plugin.xml are not loaded automatically at runtime. Therefore elasticsearch client cannot benefit from them and throws exceptions..
So i tried a different approach adding the dependencies one by one according to the exception it gives me in $NUTCH_ROOT/ivy/ivy.xml where the main dependencies of Apache Nutch are listed. That's not the right approach but it's kind of working.
How to deal with plugin dependencies?
What is the strategy for using newer version of a library in the plugin. For example Nutch uses Guava v11.0.2 but Elasticsearch 2.1.1 requires Guava v18.0. Although i'm specifying it explicitly in indexer-elastic/ivy.xml it seems to load the old version at runtime.

Answer 1

Plugin dependencies should be declared both in the plugin's ivy.xml and in the plugin.xml files. I haven't tested the files you included but cannot see anything immediately wrong with them. As you pointed out, declaring the deps in the main ivy file is not great.
See this note on how to upgrade the Tika plugin, the same logic applies to all the plugins.
As for resolving conflicts between the main dependencies and the ones from the plugins, unfortunately you'll have to deal with it yourself e.g. force the version you need in the main ivy.xml as Nutch does not handle the plugins as dependencies (in the Maven sense) of the main code.