SpanNotQuery giving unexpected results (exclude is ignored) - lucene

Question

We're having some problems with a SpanNotQuery in elasticsearch. It looks like the exclude part of the query is ignored.
To reproduce the problem I created a set of documents:
fiets kopen
fiets lopen
harrie kopen
harrie lopen
harrie fiets
kopen lopen
A SpanTermQuery for harrie will result in (3, 4, 5)
A SpanTermQuery for kopen will result in (1, 3, 6)
Now I want to combine this in a SpanNotQuery where the include is 'harrie' and exclude 'kopen'
I would expect the result to be (4, 5), but it is (3, 4, 5).
We have to use SpanQueries, this is just a small subset of the trouble we're running in to.
I created a unit test with only Lucene to show our problem
public class LuceneTest {
#Test
public void test() throws Exception {
RAMDirectory ram = new RAMDirectory();
createAndFillIndex(ram);
DirectoryReader directoryReader = DirectoryReader.open(ram);
IndexSearcher searcher = new IndexSearcher(directoryReader);
SpanQuery include = new SpanTermQuery(new Term("dummy", "harrie"));
SpanQuery exclude = new SpanTermQuery(new Term("dummy", "kopen"));
Query spanNot = new SpanNotQuery(include, exclude);
TopDocs search = searcher.search(spanNot, 100);
for (ScoreDoc scoreDoc : search.scoreDocs) {
Document result = searcher.doc(scoreDoc.doc);
String dummy = result.get("dummy");
System.out.println(scoreDoc.doc + ": " + dummy);
}
}
private void createAndFillIndex(RAMDirectory ram) throws IOException {
IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_47, new SimpleAnalyzer(Version.LUCENE_47));
IndexWriter writer = new IndexWriter(ram, conf);
add(writer, "nul"); //0
add(writer, "fiets kopen"); //1
add(writer, "fiets lopen"); //2
add(writer, "harrie kopen"); //3
add(writer, "harrie lopen"); //4
add(writer, "harrie fiets"); //5
add(writer, "kopen lopen"); //6
writer.close();
}
private void add(IndexWriter writer, String value) throws IOException {
Document doc = new Document();
IndexableField f = new TextField("dummy", value, Field.Store.YES);
doc.add(f);
writer.addDocument(doc);
}
}
Does anyone know what we're doing wrong?
Thanks!

Answer 1

The documentation gives a hint here. It matches:
spans from include which have no overlap with spans from exclude
We're dealing with spans, not whole documents. The matching span for a simple term query though, is just the single term. In each of the three matched documents in your example, the matched span is harrie, which does not have any overlap with the term kopen in any of them.
It's probably more helpful to look at an example that shows how it's intended to work. You should be able to copy-paste the following fragments into your example (and by the way, thanks for the MCVE!). Let's try this query:
SpanQuery include = new SpanTermQuery(new Term("dummy", "harrie"));
SpanQuery exclude = new SpanTermQuery(new Term("dummy", "kopen"));
SpanQuery matchterm = new SpanTermQuery(new Term("dummy", "match"));
SpanQuery[] clauses = {include, matchterm};
SpanQuery nearQuery = new SpanNearQuery(clauses, 2, true);
Query spanNot = new SpanNotQuery(nearQuery, exclude);
against these documents:
add(writer, "harrie kopen match"); //1
add(writer, "harrie match kopen"); //2
add(writer, "harrie other stuff match kopen"); //3
You should see 2 hits.
Document 1: matches nearQuery with the span: "harrie kopen match". This contains "kopen" (that is, overlaps with the span matching exclude), and so it is eliminated by the SpanNotQuery
Document 2: matches nearQuery with the span: "harrie match". The document contains "kopen", but not within the matched span, so the document remains matched.
Document 3: matches nearQuery with the span: "marrie other stuff match". Again, the document contains "kopen", but not within the matched span, so it get through.
If you want the negation to be over the entire document, rather than just the matched span, use a BooleanQuery instead.
SpanQuery include = new SpanTermQuery(new Term("dummy", "harrie"));
SpanQuery exclude = new SpanTermQuery(new Term("dummy", "kopen"));
Query query = new BooleanQuery();
query.add(new BooleanClause(include, BooleanClause.Occur.MUST))
query.add(new BooleanClause(exclude, BooleanClause.Occur.MUST_NOT))