There are lots of articles on how search engines perform indexing, but couldn't find any information on how they store these indexed records in a way that enables fast querying with scalability. Could someone explain the index storing mechanisms used in search engines or point to any article ?
Solr is able to achieve fast search responses because, instead of searching the text directly, it searches an index instead. This is like retrieving pages in a book related to a keyword by scanning the index at the back of a book, as opposed to searching every word of every page of the book.
This type of index is called an inverted index, because it inverts a page-centric data structure (page->words) to a keyword-centric data structure (word->pages).
Inverted index is a major term in the domain of Information Retrieval and Natural Language Processing. Take a document, note down all the unique words appearing in that document as well as frequency of the words. Here you are ready with your own inverted index. Solr creates similar inverted index of the documents posted to its core using a defined schema. Schema is a blue print which helps Solr in creating invered index of the documents by giving a set of predefined fields in the schema.xml file.
Related
as you know, there are different technique to index documents for search engines.
such as inverted index, Distributed Dynamic Indexing, Semantic Indexing, NGram Indexing, Context Indexing, Big Data, Multilingual Indexing and so on.
I am working with Solr now. I wonder which techniques does Solr use to index documents and how does Solr (or Lucene) use these techniques?
First - this is a very wide area and most of the terms you're listing isn't index types. They describe product features (or buzzwords) that could be supported regardless of how the index is built behind the scene.
Solr uses Lucene - which at the core is an inverted index.
The index stores statistics about terms in order to make term-based search more efficient. Lucene's index falls into the family of indexes known as an inverted index. This is because it can list, for a term, the documents that contain it. This is the inverse of the natural relationship, in which documents list terms.
There is also many support structures in place to make Lucene even more efficient for certain queries and features. On such feature is the DocValues support - which can be described as a column oriented store with document -> term mappings to speed up things like faceting.
You can see most of these support features in the Codecs API Doc for Lucene 6.3.0. As it's quite a large list, I'll leave it out from the comment itself.
To answer which techniques - Under the hood , Solr uses Lucene APIs and Lucene indexing technique is - Inverted Indexing. Solr is simply a complete application with infrastructure wrapper but underlying document indexing technique is the one provided by Lucene APIs.
How does Solr (or Lucene) use these techniques?
Here is a nice overview of Lucene indexing for beginners. Its just a very simplistic overview but explains the basics.
Since Solr is a product, most of its available documentations are functional ones ( not explaining actual indexing techniques etc) and since raw usage of Lucene is minimal, Lucene documentation is not up to the mark so most of the time, one needs to dig Lucene code or API documentation to understand working of Lucene.
Hope it helps !!
In some blogs and lucene website,I know lucene use data structure "skip list" in inverted index. But I have some puzzle about it.
1:In general,skip list maybe used in memory ,but inverted index is stored in disk. So how lucene use it when search on the index? just scanning it on disk or load it to memory?
2:skip list's insert operator often use random(0,1) to decide whether insert to next level,but in luncene introdution,it seems a fixed interval in every terms,so how lucene create the "skip list" different or not?
Please correct me if I am wrong.
Lucene uses memory in a couple different ways, even though the index is persisted on a disk when the IndexReader is created for searching and for operations like sorting (field cache):
http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
Basically those binary files get copied into RAM for much faster scanning and reducing I/O. You get a hint in the above link how searching with some parameters can force Lucene to "skip terms in searching" Hence, where that data structure can be used.
Lucene is open source, so you can see the code for yourself what is being used in Java or Lucene.NET for the C# implementation.
see To accelerate posting list skips, Lucene uses skip lists
Let's say I have a large database with product information. I want to create a search engine for that database, preferably with indexing and autocorrect features. How do I go about doing this? Are there any good libraries I could use, so that I don't have to start from scratch with basic SQL? Just some basic recommendations, links, would be much appreciated.
I am familiar with PHP, C#, VB, and Java, but I know very little about databases.
If your product database creates web pages, you would be best served using lucene or htdig. Those will do really good text searching based on your content.
Otherwise you will want to search the large fields of your database using the full text search capabilities in mysql.
To do the autocomplete you will need to have an offline indexing process that works similarly to google. Create another table called wordIndex. It contains words and the number of occurrences in your product db.
When a user starts to type, you do an ajax lookup on this table and autocomplete based on that.
If mySQL FULLTEXT searching doesn't do all you need it to (databases have indexes of their own you can set up), two good choices are Solr (based on Lucene) and Sphinx. Both are often used to provide a full featured search index on top of a mySQL database. Here's a comparison of the two.
In addition to text content my documents have tags which can be searched too. The problem now is that the tags change quite often and every time a tag gets added or removed I have to call UpdateDocument which is quite slow when done for hundreds of documents.
Are there any well performing strategies for storing tags that change often and need to be searched with Lucene? I have been thinking about keeping the tags in separate documents to keep them smaller but I can't figure out how to quickly search for tags AND content.
Store [tag, UID] pairs in a relational database. Every time a tag is added or updated, it is added and updated in this table in the database.
When performing a Lucene search that includes both tag data (stored in a database) and content (indexed in Lucene) you will need to merge the results together. One way you can do this is to:
Make a database query to pull up all the UID's for the tag in question
Translate all the UID's to Lucene doc ID's and set a bit in a BitSet for every matching Lucene doc ID
Create a Filter that wraps the BitSet, and pass that filter in to your search.
We implemented this approach in our system, and it works well. You might need to put a cache in front of the database for performance reasons, though. The particulars of step (3) will vary depending on which version of Lucene you're using.
I'm looking for some documentation on how Information Retrieval systems (e.g., Lucene) store their indexes for speedy "relevancy" lookups. My Google-fu is failing me: I've found a page which describes Lucene's file format, but it's more focused on how many bits each number is than on how the database is used in producing speedy queries.
Surely someone has some useful bookmarks lying around that they can refer me to.
Thanks!
The Lucene index is an inverted index, so any search on this topic should be relevant, like:
http://en.wikipedia.org/wiki/Inverted_index
http://www.ibm.com/developerworks/library/wa-lucene/