NOT operator doesn't work in query lucene - lucene

I use lucene version 3.0.3.0, but some expression that i search, doesn't work properly. for example if i search "!Fiesta OR Astra" on field "Model", "vauxhallAstra" is returned only and "fordFocus" is not returned. my code is below:
var fordFiesta = new Document();
fordFiesta.Add(new Field("Id", "1", Field.Store.YES, Field.Index.NOT_ANALYZED));
fordFiesta.Add(new Field("Make", "Ford", Field.Store.YES, Field.Index.ANALYZED));
fordFiesta.Add(new Field("Model", "Fiesta", Field.Store.YES, Field.Index.ANALYZED));
var fordFocus = new Document();
fordFocus.Add(new Field("Id", "2", Field.Store.YES, Field.Index.NOT_ANALYZED));
fordFocus.Add(new Field("Make", "Ford", Field.Store.YES, Field.Index.ANALYZED));
fordFocus.Add(new Field("Model", "Focus", Field.Store.YES, Field.Index.ANALYZED));
var vauxhallAstra = new Document();
vauxhallAstra.Add(new Field("Id", "3", Field.Store.YES, Field.Index.NOT_ANALYZED));
vauxhallAstra.Add(new Field("Make", "Vauxhall", Field.Store.YES, Field.Index.ANALYZED));
vauxhallAstra.Add(new Field("Model", "Astra", Field.Store.YES, Field.Index.ANALYZED));
Directory directory = FSDirectory.Open(new DirectoryInfo(Environment.CurrentDirectory + "\\LuceneIndex"));
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_30);
var writer = new IndexWriter(directory, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
writer.AddDocument(fordFiesta);
writer.AddDocument(fordFocus);
writer.AddDocument(vauxhallAstra);
writer.Optimize();
writer.Close();
IndexReader indexReader = IndexReader.Open(directory, true);
Searcher indexSearch = new IndexSearcher(indexReader);
var queryParser = new QueryParser(Version.LUCENE_30, "Model", analyzer);
var query = queryParser.Parse("!Fiesta OR Astra");
Console.WriteLine("Searching for: " + query.ToString());
TopDocs resultDocs = indexSearch.Search(query, 200);
Console.WriteLine("Results Found: " + resultDocs.MaxScore);
var hits = resultDocs.ScoreDocs;
foreach (var hit in hits)
{
var documentFromSearcher = indexSearch.Doc(hit.Doc);
Console.WriteLine(documentFromSearcher.Get("Make") + " " + documentFromSearcher.Get("Model"));
}
indexSearch.Close();
directory.Close();
Console.ReadKey();

!Fiesta OR Astra doesn't mean what you think it means. The !Fiesta portion does NOT mean, "get everything except Fiesta", as you might expect, but rather more like "forbid Fiesta". A NOT term in a Lucene query only filters out results, it does not find anything.
The only query you have defined that will actually fetch results is Astra. So everything containing Astra will be found, then anything with Fiesta will be filtered out.
In order to perform the query I believe you are expecting, you would need something like:
Astra OR (*:* !Fiesta)
*:* as a MatchAllDocsQuery. Since you do need to match all the documents to perform this sort of query, it can be expected to perform poorly.
Confusing interpretation of "boolean" logic like this are why I really don't like AND/OR/NOT syntax for Lucene. +/- is much clearer, more powerful, and doesn't introduce the oddball gotchas like this.
This excellent article on the topic clarifies somewhat why you should be thinking in terms of MUST/MUST_NOT/SHOULD, rather than traditional boolean logic.

Related

Lucene problems searchinh hyphenated field

I'm having some problems with Lucene that are driving me nuts. I have the following field:
doc.Add(new Field("cataloguenumber", i.CatalogueNumber.ToLower(), Field.Store.YES, Field.Index.ANALYZED));
Which will contain a catalogue number that looks something like this:
DF-GH5
DF-FJ4
DF-DOG
AC-DP
AC-123
AC-DOCO
i.e. two characters followed by a hyphen followed by 2-5 alphanumeric characters.
I'm trying to run a boolean query to allow users to search over the data:
// specify the search fields, lucene search in multiple fields
string[] searchfields = new string[] { "cataloguenumber", "title", "author", "categories", "year", "length", "keyword", "description" };
// Making a boolean query for searching and get the searched hits
BooleanQuery mainQuery = new BooleanQuery();
QueryParser parser;
//Add filter for main keyword
parser = new MultiFieldQueryParser(Lucene.Net.Util.Version.LUCENE_30, searchfields, new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30));
parser.AllowLeadingWildcard = true;
mainQuery.Add(parser.Parse(GetMainSearchQueryString(SearchPhrase)), Occur.MUST);
The system is working fine for all fields EXCEPT cataloguenumber which for whatever reason is not working at all.
Ideally we would like to be able to search by full or partial cataloguenumber so for example "DF-" should return all items prefixed DF
Does anyone know how I can make this work?
Thanks very much in advance
Olly
A common source of problems is to use different analyzers on index-time and query-time. You should be able to get good results by using a StandardAnalyzer - it treats the text DF-GH5 as a single token so you will be able to search using fx df-gh5 or df-* but make sure to use it for the IndexWriter and the QueryParser.
Here is a simple example which builds an in-memory index with a single document, and tries to query the index by cataloguenumber.
public static void Test()
{
// Use an in-memory index.
RAMDirectory indexDirectory = new RAMDirectory();
// Make sure to use the same analyzer for indexing
Analyzer analyzer = new StandardAnalyzer(Lucene.Net.Util.Version.LUCENE_30);
// Add single document to the index.
using (IndexWriter writer = new IndexWriter(indexDirectory, analyzer, IndexWriter.MaxFieldLength.UNLIMITED))
{
Document document = new Document();
document.Add(new Field("content", "This is just some text", Field.Store.YES, Field.Index.ANALYZED));
document.Add(new Field("cataloguenumber", "DF-GH5", Field.Store.YES, Field.Index.ANALYZED));
writer.AddDocument(document);
}
var parser = new MultiFieldQueryParser(
Lucene.Net.Util.Version.LUCENE_30,
new[] { "cataloguenumber", "content" },
analyzer);
var searcher = new IndexSearcher(indexDirectory);
DoSearch("df-gh5", parser, searcher);
DoSearch("df-*", parser, searcher);
}
private static void DoSearch(string queryString, MultiFieldQueryParser parser, IndexSearcher searcher)
{
var query = parser.Parse(queryString);
TopDocs docs = searcher.Search(query, 10);
foreach (ScoreDoc scoreDoc in docs.ScoreDocs)
{
Document searchHit = searcher.Doc(scoreDoc.Doc);
string cataloguenumber = searchHit.GetValues("cataloguenumber").FirstOrDefault();
string content = searchHit.GetValues("content").FirstOrDefault();
Console.WriteLine($"Found object: {cataloguenumber} {content}");
}
}

Lucene white space analyzer ignoring phrases?

I'm updating a document in Lucene, but when I search for the full value in one of the fields no results come back. If I search for just one word, then I get a result back.
This example comes from chapter 2 of the Lucene in Action 2nd Edition book and I'm using the Lucene 3 Java Library.
Here's the main logic
"Document fields show new value when updated, and not old value" in {
getHitCount("city", "Amsterdam") must equal(1)
val update = new Document
update add new Field("id", "1", Field.Store.YES, Field.Index.NOT_ANALYZED)
update add new Field("country", "Netherlands", Field.Store.YES, Field.Index.NO)
update add new Field("contents", "Den Haag has a lot of museums", Field.Store.NO, Field.Index.ANALYZED)
update add new Field("city", "Den Haag", Field.Store.YES, Field.Index.ANALYZED)
wr updateDocument(new Term("id", "1"), update)
wr close
getHitCount("city", "Amsterdam") must equal(0)
getHitCount("city", "Den Haag") must equal(1)
}
It's the last line in the above that fails - the hit count is 0. If I change the query to either "Den" or "Haag" then I get 1 hit.
Here is all the setup and dependencies. Note how the writer uses a white space query analyzer as the book suggests. Is this the problem?
override def beforeEach{
dir = new RAMDirectory
val wri = writer
for (i <- 0 to ids.length - 1) {
val doc = new Document
doc add new Field("id", ids(i), Field.Store.YES, Field.Index.NOT_ANALYZED)
doc add new Field("country", unindexed(i), Field.Store.YES, Field.Index.NO)
doc add new Field("contents", unstored(i), Field.Store.NO, Field.Index.ANALYZED)
doc add new Field("city", text(i), Field.Store.YES, Field.Index.ANALYZED)
wri addDocument doc
}
wri close
wr = writer
}
var dir: RAMDirectory = _
def writer = new IndexWriter(dir, new WhitespaceAnalyzer, IndexWriter.MaxFieldLength.UNLIMITED)
var wr: IndexWriter = _
def getHitCount(field: String, q: String): Int = {
val searcher = new IndexSearcher(dir)
val query = new TermQuery(new Term(field, q))
val hitCount = searcher.search(query, 1).totalHits
searcher.close()
hitCount
}
You may want to look at PhraseQuery instead of TermQuery.

How Lucene search works?

I'm testing Lucene indexing/searchin and I have a doubt. To test I create some simple files.
Example:
mark_test_mark.txt
mark test mark
a.txt
mark
test
mark
mark
test
mark
mark
test
mark
mark
test
mark
I extrac the files' content and I'm indexing this too.
I'm creating the document to indexing this way:
doc.add(new Field(FILE_NAME, index.getFileName().trim(), Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES));
doc.add(new Field(FILE_NAME_LOWER, index.getFileName().toLowerCase().trim(), Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES));
doc.add(new Field(CONTENT, index.getFileContent(), Field.Store.YES, Field.Index.ANALYZED, Field.TermVector.YES));
My question is when I do a seach for a keyword like 'mark'.
Lucene returns to me the following result:
mark_test_mark.txt -> 0.36452034
a.txt -> 0.36452034
Where, the 1st part represent the file name, and the second, the search score.
In my opinion, these 2 files don't have the same score and the first file should be a.txt.
Am I wrong?
EDIT:
I forget to say that I'm searching by name and content, so I do a multi-field search.
I'm using this code to do this:
IndexReader reader = IndexReader.open(Indexer.getFSDirectory(searchDirectory));
IndexSearcher searcher = new IndexSearcher(reader);
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_36);
MultiFieldQueryParser queryParser = new MultiFieldQueryParser(Version.LUCENE_36, new String[] {Indexer.FILE_NAME_LOWER, Indexer.CONTENT}, analyzer);
TopDocs topDocs = null;
try {
topDocs = searcher.search(queryParser.parse(searchQuery.getQuery()), getHitsPerPage());
} catch (ParseException e) {
e.printStackTrace();
}
ScoreDoc[] hits = topDocs.scoreDocs;

Lucene.net return all fields in a document

I am storing data in lucene.Net
I add a document with multiple fields :
var doc = new Document();
doc.Add(new Field("CreationDate", dt, Field.Store.YES, Field.Index.ANALYZED));
doc.Add(new Field("FileName", path, Field.Store.YES, Field.Index.ANALYZED));
doc.Add(new Field("Directory", dtpath, Field.Store.YES, Field.Index.NOT_ANALYZED));
doc.Add(new Field("Text", text.ToString(), Field.Store.YES, Field.Index.ANALYZED));
...
writer.AddDocument(doc);
I want to move through all items and return fields "CreationDate" and "Directory" for each document.
But a Term can only except 1 field :
var termEnum = reader.Terms(new Term("CreationDate"));
How do i make it return the 2 fields ???
Thanks
Martin
When you iterate over search results, read the document and load the values from it:
int docId = hits[i].doc;
Document doc = searcher.Doc(docId);
String creationDate = doc.Get("CreationDate");
String directory = doc.get("Directory");
// ...and so on
You can ontain a enumeration of all documents containing a given term with:
var termDocEnum = reader.TermDocs(new Term("CreationDate"));
You can use that enumeration to fetch documents using the docId:
Document doc = searcher.doc(termDocEnum.doc);
Which should make it easy enough to use the Document API to get the information you are looking for.
Note, as hinted at earlier, this will only get documents for which the term given has a value specified! If that is a problem, you can either call TermDocs once for each pertinent argument and merge the sets as needed (done easily enough with a hash table indexed on docId, or some such), or call TermDocs with no argument, and use seek to find the appropriate Terms (merging will again need to be done manually if necessary).
A TermEnum, as passed from the 'terms' method does not give you a docId (you would have to use the termDocs method to get them with each term in the enum, I believe.)

Deleting document by Term from lucene

The following code does not delete the document by Term as expected:
RAMDirectory idx = new RAMDirectory();
IndexWriter writer = new IndexWriter(idx,
new SnowballAnalyzer(Version.LUCENE_30, "English"),
IndexWriter.MaxFieldLength.LIMITED);
Document doc = new Document();
doc.add(new Field("title", "mydoc", Field.Store.YES, Field.Index.NO));
doc.add(new Field("content", "some content, deleteme", Field.Store.YES, Field.Inde
x.ANALYZED));
writer.addDocument(doc);
Document doc2 = new Document();
doc2.add(new Field("title", "mydoc2", Field.Store.YES, Field.Index.NO));
doc2.add(new Field("content", "other content, don't deleteme", Field.Store.YES, Field.I
ndex.ANALYZED));
writer.addDocument(doc2);
writer.optimize();
writer.close();
/*
IndexReader reader = IndexReader.open(idx, false);
int docs_up_for_deletion = reader.docFreq(new Term("title"));
int before = reader.numDocs();
int docs_deleted = reader.deleteDocuments(new Term("title", "mydoc"));
reader.close();
*/
IndexWriter writer2 = new IndexWriter(idx,
new SnowballAnalyzer(Version.LUCENE_30, "English"),
IndexWriter.MaxFieldLength.LIMITED);
int before = writer2.numDocs();
writer2.deleteDocuments(new Term("title", "mydoc"));
writer2.commit();
writer2.optimize();
int after = writer2.numDocs();
writer2.close();
int docs_deleted = before - after;
I've tried deleting with the IndexReader and IndexWriter and neither works.
I've also tried adding another IndexReader search after the above code just in case the number only gets updated after closing writer2 (mentioned in this FAQ), but that doesn't help. Doing a writer.deleteAll() works, just not the delete by Term.
I found an old reference to the fact that only fields of type Field.Keyword can be deleted, but this is no longer a valid field type in Lucene 3.x
Your title field is not indexed. Change
new Field("title", "mydoc", Field.Store.YES, Field.Index.NO)
to
new Field("title", "mydoc", Field.Store.YES, Field.Index.ANALYZED)
or
new Field("title", "mydoc", Field.Store.YES, Field.Index.NOT_ANALYZED)
depending on whether or not you want your field analyzed.