I've seen two solutions mentioned, and was wondering what most people do.
Use logstash
Code your application to make writes to Elasticsearch alongside SQL. For example,
public saveRecord() {
saveToElasticsearch();
saveToSQL();
}
Another question is how to handle actually searching the entity? Do you ONLY use Elasticsearch?
If not, I would assume you fetch from Elasticsearch based on keywords and use the IDs returned to filter your SQL query. My question then, is how do you handle pagination? For example let's say you only want results 50 to 100. First you query Elasticsearch which returns 50-100. Then the SQL query reduces that to 20 results - the other 30 results are in what would've been the next Elasticsearch query (100 - 150 for example). Do you keep going back and forth?
As for your first question check here
As for the second question, if you plan to use elasticsearch as your search layer then better do it for all the searchable/filterable fields. As you've described, the alternative will get very messy very soon. Use elasticsearch for all your searches/filters and even aggregations if it suits your needs. Use the sql database as your point of truth and just get the full payload from there.
In general, if you will need to paginate then your search should better be in one place otherwise it will get ugly.
I'm trying to get a list of Change Requests that match certain conditions, some of these conditions are met by using functions like has_attr().
I would like to ask is it at all possible, I need for instance to use such function has_associated_task(cvtype="task") is it possible to do that?
For queries I'm using the following pattern:
http://ip[:port]/change/oslc/db/dbURI/role/User/cr?oslc_cm.query=change:cvtype="problem" and request_type="Change_Request" and has_associated_task(cvtype="task")&oslc_cm.properties=problem_synopsis
this does work without the function term but I would like to extend the search criteria further, is there any other way besides doing a predefined query in change? Is there somewhere a list of terms? like change:cvtype (I've tried to see this [http://www.ibm.com/xmlns/prod/rational/change/1.0/][1] but I got a "whoops" from the web server)
There are some ways you could solve this:
OSLC Resource Shapes - some OSLC providers associate shapes (like schemas) that describe what you can expect from an OSLC Query Capability.
There isn't a way in the simple query syntax to test for null (or not null), assuming you want to have some condition such as (cvtype="task" and linkedTask != NULL). To get around this you can simply query based on cvtype="task" and locally filter the results using tools such as XPath or Jena. Alternatively you can do is look for extensions to the tool you are working with to see if they provide any extensions to the query syntax to support your use case, I don't have this information off hand.
Does this gremlin script (executed via REST API of Neo4j) executes the sorting on the lucene index? Or are the nodes sorted in Neo4j?
g.idx('myIndex').get('name', 'aaa').sort{it.name}
Two additional questions:
1. How to set ordering? ASC/DESC
2. How to perform a fulltext search (LIKE). I already tried *, %, nothing worked
sort is a Groovy method. To reverse the order, use reverse:
g.idx('myIndex').get('name', 'aaa').sort{it.name}.reverse()
See:
http://groovy.codehaus.org/groovy-jdk/java/util/Collection.html
http://groovy.codehaus.org/groovy-jdk/java/util/List.html
Besides doing what espeed suggested, which is using Gremlin's facilities to sort etc, you may also be interested in passing additional instructions down to Lucene itself. This can be done by prefixing the second argument into get with a magic string %query%. Like so:
... .get(null, "%query% _start_node_id_:15815486")
The key argument can be null if you don't need to use it.
Is there a way to issue join queries (http://www.searchworkings.org/blog/-/blogs/query-time-joining-in-lucene) in lucene without directly using Query API? Is it possible to issue query in text form for this requirement? For example:
title:derivatives join(comments:great)
Apache Solr (4.0, not released yet) has a query parser which can handle join queries.
If I understand your question, I think you want a query like 'title:derivatives AND comments:great'. Or you can use code like 'queryParser.setDefaultOperator(QueryParser.Operator.AND)' to change the default conjunction operator to AND instead of OR ('OR' is used by default unless you tell Lucene otherwise).
I'm using Hibernate for ORM of my Java app to an Oracle database (not that the database vendor matters, we may switch to another database one day), and I want to retrieve objects from the database according to user-provided strings. For example, when searching for people, if the user is looking for people who live in 'fran', I want to be able to give her people in San Francisco.
SQL is not my strong suit, and I prefer Hibernate's Criteria building code to hard-coded strings as it is. Can anyone point me in the right direction about how to do this in code, and if impossible, how the hard-coded SQL should look like?
Thanks,
Yuval =8-)
For the simple case you describe, look at Restrictions.ilike(), which does a case-insensitive search.
Criteria crit = session.createCriteria(Person.class);
crit.add(Restrictions.ilike('town', '%fran%');
List results = crit.list();
Criteria crit = session.createCriteria(Person.class);
crit.add(Restrictions.ilike('town', 'fran', MatchMode.ANYWHERE);
List results = crit.list();
If you use Spring's HibernateTemplate to interact with Hibernate, here is how you would do a case insensitive search on a user's email address:
getHibernateTemplate().find("from User where upper(email)=?", emailAddr.toUpperCase());
You also do not have to put in the '%' wildcards. You can pass MatchMode (docs for previous releases here) in to tell the search how to behave. START, ANYWHERE, EXACT, and END matches are the options.
The usual approach to ignoring case is to convert both the database values and the input value to upper or lower case - the resultant sql would have something like
select f.name from f where TO_UPPER(f.name) like '%FRAN%'
In hibernate criteria restrictions.like(...).ignoreCase()
I'm more familiar with Nhibernate so the syntax might not be 100% accurate
for some more info see pro hibernate 3 extract and hibernate docs 15.2. Narrowing the result set
This can also be done using the criterion Example, in the org.hibernate.criterion package.
public List findLike(Object entity, MatchMode matchMode) {
Example example = Example.create(entity);
example.enableLike(matchMode);
example.ignoreCase();
return getSession().createCriteria(entity.getClass()).add(
example).list();
}
Just another way that I find useful to accomplish the above.
Since Hibernate 5.2 session.createCriteria is deprecated. Below is solution using JPA 2 CriteriaBuilder. It uses like and upper:
CriteriaBuilder builder = session.getCriteriaBuilder();
CriteriaQuery<Person> criteria = builder.createQuery(Person.class);
Root<Person> root = criteria.from(Person.class);
Expression<String> upper = builder.upper(root.get("town"));
criteria.where(builder.like(upper, "%FRAN%"));
session.createQuery(criteria.select(root)).getResultList();
Most default database collations are not case-sensitive, but in the SQL Server world it can be set at the instance, the database, and the column level.
You could look at using Compass a wrapper above lucene.
http://www.compass-project.org/
By adding a few annotations to your domain objects you get achieve this kind of thing.
Compass provides a simple API for working with Lucene. If you know how to use an ORM, then you will feel right at home with Compass with simple operations for save, and delete & query.
From the site itself.
"Building on top of Lucene, Compass simplifies common usage patterns of Lucene such as google-style search, index updates as well as more advanced concepts such as caching and index sharding (sub indexes). Compass also uses built in optimizations for concurrent commits and merges."
I have used this in the past and I find it great.