I'm new to apache lucene. I started using lucene. I had faced an issue. I started indexing all the files in the directory. I didn't close the indexwriter and tried to open in luke. It prompted with an error "Index not closed". The problem over here is the code execution has been completed. How to unlock the index? If I instantiate a new IndexWriter with the same directory, will it overwrite the existing index?
I am not an expert either..
If I were you, I'd do the following..
1) Add the following snippet to code at the end, which is a must at any cost.
myIndexWriter.close();
2) Delete the existing directory manually, and rerun the whole code.
If you instantiate the new IndexWriter without deleting the directory, it will add docs to the existing index. Yes, it will result in duplicate index entries.
However, in the Lucene's perspective, all those entries are still unique, i.e. every addDocument() creates a new entry in the Index with a new unique lucene-internal-doc-id.
Related
I have a few basic questions regarding the usage of SearcherManager with IndexWriter.
I need to periodically re-build the Lucene index in the application and currently it happens on a different thread other than the one that serves the search requests.
Can I use the same IndexWriter instance through the lifetime of the application to rebuild the index periodically? Currently, I create / open it once during the startup and just call IndexWriter#commit whenever a new index is built.
I'm using SearcherManagerto acquire and release IndexSearcher instances for each search request. After the index is periodically built, I'm planning to use SearcherManager#maybeRefresh method to get refreshed IndexSearcher instances.SearcherManager instance is also created once during the startup and I intend to maintain it through out.
I do not close the IndexWriter or SearcherManager throughout the app's lifetime.
Now for the questions,
If I create a new IndexWriter every time I need to rebuild the index, will SearcherManager#maybeRefresh be able to detect that it's a new IndexWriter Instance? Or do I need to create a new SearcherManager using the newly created IndexWriter ?
What's the difference between creating a SearcherManager instance using an IndexWriter, creating it using a DirectoryReader or creating it using a Directory ?
The answers depend on how you construct your SearcherManager:
If you construct it with a DirectoryReader, all future IndexSearchers acquired from the SearcherManager will be based on that reader, i.e. all searches will provide results from the point in time you instantiated the SearcherManager. If you write data to the index/directory and run SearcherManager.maybeRefresh() afterwards, the reader will not be updated and your search results will be outdated.
If you construct the SearcherManager with an IndexWriter, SearcherManager.maybeRefresh() will update the SearcherManager's reader if data has been written and commited by the writer. All newly acquired IndexSearchers will then reflect the new state of the underlying index.
Despite having limited experience, I recommend using the latter approach. It provides a very simple way to implement near-real-time searching: At application start you create an IndexWriter and construct a SearcherManager with it. Afterwards you start a background thread that periodically commits all changes in the IndexWriter and refreshes the SearcherManager. For the lifetime of your application you can keep using the initial IndexWriter and SearcherManager without having to close/reopen them.
PS: I have only started working with Lucene a few days ago, so don't take everything I wrote here as 100% certain.
We use Lucene as a search engine. Our Lucene index is created by a master server, which is then deployed to slave instances.
This deployment is currently done by a script that deletes the files, and copy the new ones.
We needed to know if there was any good practice to do a "hot deployment" of a Lucene index. Do we need to stop or suspend Lucene? Do we need to inform Lucene the index has changed?
Thanks
The first step is to open the index in append mode for writing. You can achieve this by calling IndexWriter with the open mode named IndexWriterConfig.OpenMode.CREATE_OR_APPEND.
Once this is done, you are ready to both update existing documents and add new documents. For updating documents, you need to provide some kind of a unique identifier for a document (could be the URL or something else that is guaranteed to be unique). Now if you want to update a document with id say "Doc001" simply call the updateDocument function of Lucene passing "Doc001" as the Term (the very first) argument.
By this you can update an existing index without deleting it.
I'm making a web app that uses Lucene as search engine. First, the user has to select a file/directory to index and after that he is capable to search it (duh!). My problem happens when the user is trying to index a huge amount of data: for example, if it's taking too long and the user refreshs the page and try to index another directory, an exception is thrown because the first indexing is still running (write.lock shows up). Known that, how is it possible to stop the first indexing? I tried closing the IndexWriter with no success.
Thanks in advance.
Why do you want to interrupt the first indexing operation and restart it again?
In my opinion you should display a simple image which shows that the system is working (as Nielsen says: "The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.") and when the user press refresh you should intercept the event and prevent the execution of another indexing process.
You are probably trying to open an indexwriter instance on the index directory which has already indexwriter open on it. If you have opened indexwriter on two different index directory then the exception with write.lock won't happen. Could you please check that the new indexwriter instance is not writing to previously opened index directory which has already indexwriter opened on it.
I am creating a tagging system for my site
I got the basics of adding a document to lucene but i can seem to figure out how to delete a document or update one when the user changes the tags of something. I found pages that say use the document index and i need to optimize before effect but how do i get the document index? Also i seen another that said to use IndexWriter to delete but i couldnt figure out how to do it with that either.
I am using C# asp.net and i dont have java install on that machine
What version of Lucene are you using? The IndexWriter class has an update method that lets you update (BTW an update under the hood is really a delete followed by an add). You will need to have some identifier (such as as document id) which lets you update. When you index the document, add a unique document identifier such as a URL, a counter etc. Then the "Term" will be the ID of the document you wish to update. For example using URL you can update thus:
IndexWriter writer = ...
writer.update(new Term("id","http://somedomain.org/somedoc.htm"), doc);
You need an IndexReader to delete a document, I'm not sure about the .net version but the Java and C++ versions of the Lucene API have an IndexModifier class that hides the differences between IndexReader and IndexWriter classes and just uses the appropriate one as you call addDocument() and removeDocument().
Also, there is no concept of updating a document in Lucene you have to remove it an them re-add it again. In order to do this you will need to make sure that every document has a unique stored id in the index.
I am working on a job portal site and have been using Lucene for job search functionality.
Users will be posting a number jobs on our site on a daily basis.We need to make sure that new job posted is searchable on the site as soon as possible.
In this context, how do I update Lucene index when a new job is posted or when an existing job is edited?
Can lucene index updating and search work in parallel?
Also,can I know any tips/best practices with respect to Lucene indexing,optimizing,performance etc?
Appreciate ur help!
Thanks!
Yes, Lucene can search from and write to an index at the same time as long as no more than 1 IndexWriter writes to it. If you want the new records visible ASAP, have the IndexWriter call the commit() function often (see IndexWriter's JavaDoc for details).
These Wiki pages might also help:
ImproveIndexingSpeed
ImproveSearchingSpeed
I have used Lucene.Net on a web site similar to what you are doing. Yes, you can do live indexes, updating to keep everything up to date? What platform are you using Lucene on, .NET, Java?
Make sure you create a new IndexSearcher as any additions after an IndexSearcher has been created are not visible to that instance.
A better approach may be to ReOpen the IndexReader if you want to resuse the same index searcher.