I am working on a job portal site and have been using Lucene for job search functionality.
Users will be posting a number jobs on our site on a daily basis.We need to make sure that new job posted is searchable on the site as soon as possible.
In this context, how do I update Lucene index when a new job is posted or when an existing job is edited?
Can lucene index updating and search work in parallel?
Also,can I know any tips/best practices with respect to Lucene indexing,optimizing,performance etc?
Appreciate ur help!
Thanks!
Yes, Lucene can search from and write to an index at the same time as long as no more than 1 IndexWriter writes to it. If you want the new records visible ASAP, have the IndexWriter call the commit() function often (see IndexWriter's JavaDoc for details).
These Wiki pages might also help:
ImproveIndexingSpeed
ImproveSearchingSpeed
I have used Lucene.Net on a web site similar to what you are doing. Yes, you can do live indexes, updating to keep everything up to date? What platform are you using Lucene on, .NET, Java?
Make sure you create a new IndexSearcher as any additions after an IndexSearcher has been created are not visible to that instance.
A better approach may be to ReOpen the IndexReader if you want to resuse the same index searcher.
Related
What is different between IndexWriter.Close() andIndexWriter.Commit() when I hava just single instance of indexWriter?
Note:The Data that I going to make index is very big then I can't close IndexWriter runtime.
Note:I want to search in documents when data are indexing at sametime.
Commit() commits pending, buffered changes to the index (which can then be found with IndexReader() ). The IndexWriter can then continue to be used for more changes. Close() also performs a Commit(), but additionally closes the IndexWriter. Note that IndexWriter implements IDisposable(), and I recommend using it.
By your first note, if you mean there are lots of documents to index, that's fine. You can use the same IndexWriter for many documents without closing it. Just loop through however many documents you want to index within the same IndexWriter using() statement.
With regards to your second note, you must perform a commit() ( or close()) before your IndexWriter() changes will be seen by an IndexReader(). You can always search with IndexReader(), but it will only see the index as it was since the last IndexWriter.Commit().
I recommend Lucene In Action for these important details. It helped me a great deal.
We use Lucene as a search engine. Our Lucene index is created by a master server, which is then deployed to slave instances.
This deployment is currently done by a script that deletes the files, and copy the new ones.
We needed to know if there was any good practice to do a "hot deployment" of a Lucene index. Do we need to stop or suspend Lucene? Do we need to inform Lucene the index has changed?
Thanks
The first step is to open the index in append mode for writing. You can achieve this by calling IndexWriter with the open mode named IndexWriterConfig.OpenMode.CREATE_OR_APPEND.
Once this is done, you are ready to both update existing documents and add new documents. For updating documents, you need to provide some kind of a unique identifier for a document (could be the URL or something else that is guaranteed to be unique). Now if you want to update a document with id say "Doc001" simply call the updateDocument function of Lucene passing "Doc001" as the Term (the very first) argument.
By this you can update an existing index without deleting it.
I am creating a tagging system for my site
I got the basics of adding a document to lucene but i can seem to figure out how to delete a document or update one when the user changes the tags of something. I found pages that say use the document index and i need to optimize before effect but how do i get the document index? Also i seen another that said to use IndexWriter to delete but i couldnt figure out how to do it with that either.
I am using C# asp.net and i dont have java install on that machine
What version of Lucene are you using? The IndexWriter class has an update method that lets you update (BTW an update under the hood is really a delete followed by an add). You will need to have some identifier (such as as document id) which lets you update. When you index the document, add a unique document identifier such as a URL, a counter etc. Then the "Term" will be the ID of the document you wish to update. For example using URL you can update thus:
IndexWriter writer = ...
writer.update(new Term("id","http://somedomain.org/somedoc.htm"), doc);
You need an IndexReader to delete a document, I'm not sure about the .net version but the Java and C++ versions of the Lucene API have an IndexModifier class that hides the differences between IndexReader and IndexWriter classes and just uses the appropriate one as you call addDocument() and removeDocument().
Also, there is no concept of updating a document in Lucene you have to remove it an them re-add it again. In order to do this you will need to make sure that every document has a unique stored id in the index.
I can't find a straightforward yes or no answer to this!
I know I can send multiple reads in parallel but can I query an index while a seperate process/thread is updating it?
It's been a while since I used Lucene. However, assuming you are talking about the Java version, the FAQ has this to say:
Does Lucene allow searching and indexing simultaneously?
Yes. However, an IndexReader only searches the index as of the "point in time" that it was opened. Any updates to the index, either added or deleted documents, will not be visible until the IndexReader is re-opened. So your application must periodically re-open its IndexReaders to see the latest updates. The IndexReader.isCurrent() method allows you to test whether any updates have occurred to the index since your IndexReader was opened.
See also Lucene's near-real-time feature, which gives fast turnaround upon changes (adds, deletes, updates) to the index to being able to search those changes. For example, using near-real-time you could make changes to the index and then reopen the reader every few seconds.
I've a .net web application which uses Lucene.net for company search functionality.
When registered users add a new company,it is saved to database and also gets indexed in Lucene based company search index in real time.
When adding company in Lucene index, how do I handle use case of two or more logged-in users posting a new company at the same time?Also, will both these companies get indexed without any file lock, lock time out, etc. related issues?
Would appreciate if i could help with code as well.
Thanks.
By default Lucene.Net has inbuilt index locking using a text file. However if the default locking mode isn't good enough then there are others that you can use instead (which are included in the Lucene.Net source code).