In my project, we use solr to index a lot of different kind of documents, by example Books and Persons, with some common fields (like the name) and some type-specific fields (like the category, or the group people belong to).
We would like to do queries that can find both books and persons, with for each document type some filters applied. Something like:
find all Books and Persons with "Jean" in the name and/or content
but only Books from category "fiction" and "fantasy"
and only Persons from the group "pangolin"
everything sorted by score
A very simple way to do that would be:
q = name:jean content:jean
&
fq=
(type:book AND category:(fiction fantasy))
OR
(type:person AND group:pangolin)
But alas, as fq are cached, I'd prefer something allowing me simpler and so more reusable fq like :
fq=type:book,
fq=type:person,
fq=category(fiction fantasy),
fq=group:pangolin.
Is there a way to tell solr to merge or combine many queries? Something like 'grouping' fq together.
I read a bit about nested queries with _query_, but the very few documentation about it makes me think it's not the solution I'm looking for.
As Geert-Jan mentioned it in his answer, the possibility to do OR between fq is a solr asking feature, but with very little support by now: https://issues.apache.org/jira/browse/SOLR-1223
So I managed to simulate what I want to in a simple way:
for each field a document type can have, we have to define everytime a value (so if in my own example Books can have no category, at index time we still have to define something like category=noCategoryCode
when using a filter on one of this fields in a query on multiple types, we add a non-present condition in the filter, so fq=category:fiction becomes fq=category:fiction (*:* AND -category:*)
By this way, all other types (like Person) will pass through this filter, and the filter stands quite atomic and often used - so caching is still useful.
So, my full example becomes:
q = name:jean content:jean
&
fq= type:(book person)
&
fq= category:(fiction fantasy) (*:* AND -category:*)
&
fq= group:(pangolin) (*:* AND -group:*)
Still, can't wait SOLR-1223 to be patched :)
You can apply multiple filter queries at the same time
q=name:jean content:jean&fq=type:book&fq=type:person&fq=category(fiction fantasy)&fq=group:pangolin
Perhaps I am not understanding your issue, but the only difference between a query and a filter is that the filter is cached. If you don't care about the caching, just modify their query:
real query +((type:book category:fiction) (type:person group:pangolin))
Related
I'm using Lucene 4.10.3 with Java 1.7
I'm wondering whether it's possible to order query results the matching term?
Simply put, if my documents conatin a text field;
The query is
text:a*
I want documents with ab, then ac, then ad etc.
The real case is more complex however, what I'm actually trying to accomplish is to "stuff" a relational DB into my lucene Index (probably not the best idea?).
An appropriate example would be :
I have documents representing books in a library. every book has a title and also a list of people who has borrowed this book and the date of borrowing.
when a user searches for a book with title containing "JAVA", I want to give priority to books that were borrowed by this user. This could be accomplished by adding a TextField "borrowers", adding a SHOULD clause on it and ordering by score)
also, if there are several books with "JAVA" that this user has borrowed before, I want to show the most recent borrowed ones first. so I thought to create a TextField "borrowers" that will look like
borrowers : "user1__20150505 user2__20150506" etc.
I will add a BooleanClause borrowers: user1* and order by matching term.
any other solution ideas will be welcome
I understand your real problem is more complex, but maybe this is helpful anyway.
You could first search for Tokens in the index that match your query, then for each matching token executing a query using this token specifically.
See https://lucene.apache.org/core/6_0_1/core/org/apache/lucene/index/TermsEnum.html for that. Just seek to the prefix and iterate until the prefix stops matching.
In general it is sometimes easy to just issue two queries. For example one within the corpus of books the user as borrowed before and another witin the whole corpus.
These approaches may not work, but in that case you could implement a custom Scorer somehow mapping the ordering to a number.
See http://opensourceconnections.com/blog/2014/03/12/using-customscorequery-for-custom-solrlucene-scoring/
I have two requirements for my SOLR implementation:
I need to be able to search on multiple fields at the same time (preferably with field boosting). This is possible using dismax parser.
I also have a specific set of indexed fields (example gender field). I need to be able to apply such specific filters (example: select?q=david&gender:male&status:married). As per my understanding of dismax, this is not possible.
Please suggest if the second requirement can be handled using dismax (or edismax)? For now i am forced to use standard query parser, even though i really liked dismax.
There is nothing stopping you from using dismax or edismax. Use qf to tell it which fields to search by default, and use fq to apply queries that act as filters.
/select?q=david&fq=gender:male&fq=status:married&qf=name^10 address^3
Filter Queries doesn't affect score, and will be cached separately. If you always filter on both gender and status, you could combine them to get a single query cache instead (fq=gender:male AND status:married).
I am not sure this is a duplicated question or not (I don't think so) but its very interesting question for me:
In SQL we can create custom field and put it in the result:
SELECT *.p, totalOrder=(SELECT sum(price) from orders where id=p.id)
FROM products p;
so the result is a list of products with totalSales value.
What is best approach in NoSQL(MongoDB),
I am sure we should have two types of socuments(products and orders) so I know we don't have Join but the question is do we have custom field assignment in finding queries?
When you use aggregation, you have the $project operation which is exactly that. It is used to rename fields or derive field values through some simple operators. But as usual with MongoDB, you can not get any data from another collection.
When you need to do something which is too complex to express with aggregation, you can use MapReduce and build your output-documents with Javascript. But again, no breaking out of the collection.
I have indexed documents in Lucene based on three fields: title, address, city. Now I want to build my query say, C A B so that I can retrieve the documents as follows:
C must be present in the title field of the documents and either A or B must be present in either of address and city fields of the matched documents. The documents that have A present in either of those fields should get higher score or higher boost. Here A, B, C may be single terms or phrases.
I am new to Lucene. I do not have any experience of framing such complex queries. In this context I have read the post Boost factor in MultiFieldQueryParser
But this post does not answer my question. So if anyone please help me to solve this I will be really grateful.
title:C AND (address:A^2 OR city:A^2 OR address:B OR city:B)
Don't get caught up on reading about MultiFieldQueryParser, that isn't really what you need for this. Standard QueryParser syntax will serve your purposes you fine.
See the Lucene QueryParser syntax documentation
A query like:
+title:C +((address:A city:A)^2 address:B city:B)
Should do nicely.
To explain a bit:
+title:C - require a match on title:C. No results will be returned that don't match this condition.
+(....) - require a match on the subquery contained inside. As long as a match is found on any one of the optional queries contained within the parentheses is matched, this will be satisfied.
(address:A city:A)^2 - You prefer a match on A, these two queries are boosted more heavily with ^2.
I'm looking for a pattern for performing a dynamic search on multiple tables.
I have no control over the legacy (and poorly designed) database table structure.
Consider a scenario similar to a resume search where a user may want to perform a search against any of the data in the resume and get back a list of resumes that match their search criteria. Any field can be searched at anytime and in combination with one or more other fields.
The actual sql query gets created dynamically depending on which fields are searched. Most solutions I've found involve complicated if blocks, but I can't help but think there must be a more elegant solution since this must be a solved problem by now.
Yeah, so I've started down the path of dynamically building the sql in code. Seems godawful. If I really try to support the requested ability to query any combination of any field in any table this is going to be one MASSIVE set of if statements. shiver
I believe I read that COALESCE only works if your data does not contain NULLs. Is that correct? If so, no go, since I have NULL values all over the place.
As far as I understand (and I'm also someone who has written against a horrible legacy database), there is no such thing as dynamic WHERE clauses. It has NOT been solved.
Personally, I prefer to generate my dynamic searches in code. Makes testing convenient. Note, when you create your sql queries in code, don't concatenate in user input. Use your #variables!
The only alternative is to use the COALESCE operator. Let's say you have the following table:
Users
-----------
Name nvarchar(20)
Nickname nvarchar(10)
and you want to search optionally for name or nickname. The following query will do this:
SELECT Name, Nickname
FROM Users
WHERE
Name = COALESCE(#name, Name) AND
Nickname = COALESCE(#nick, Nickname)
If you don't want to search for something, just pass in a null. For example, passing in "brian" for #name and null for #nick results in the following query being evaluated:
SELECT Name, Nickname
FROM Users
WHERE
Name = 'brian' AND
Nickname = Nickname
The coalesce operator turns the null into an identity evaluation, which is always true and doesn't affect the where clause.
Search and normalization can be at odds with each other. So probably first thing would be to get some kind of "view" that shows all the fields that can be searched as a single row with a single key getting you the resume. then you can throw something like Lucene in front of that to give you a full text index of those rows, the way that works is, you ask it for "x" in this view and it returns to you the key. Its a great solution and come recommended by joel himself on the podcast within the first 2 months IIRC.
What you need is something like SphinxSearch (for MySQL) or Apache Lucene.
As you said in your example lets imagine a Resume that will composed of several fields:
List item
Name,
Adreess,
Education (this could be a table on its own) or
Work experience (this could grow to its own table where each row represents a previous job)
So searching for a word in all those fields with WHERE rapidly becomes a very long query with several JOINS.
Instead you could change your framework of reference and think of the Whole resume as what it is a Single Document and you just want to search said document.
This is where tools like Sphinx Search do. They create a FULL TEXT index of your 'document' and then you can query sphinx and it will give you back where in the Database that record was found.
Really good search results.
Don't worry about this tools not being part of your RDBMS it will save you a lot of headaches to use the appropriate model "Documents" vs the incorrect one "TABLES" for this application.