There are three search strategies for own entities: trait, free-text and keywords, as explained in the documentation.
What I can't understand, is the allowed combinations of these options. I am able to choose either:
trait
keywords
free-text and keywords
Why can't free-text be chosen on it's own, only in combination with keywords?
Edit:
Definition of free-text, from the documentation:
When you need to extract a substring of the message, and this
substring does not belong to a predefined list of possible values.
Definition of keywords:
When the entity value belongs to a predefined list, and you just need
substring matching to look it up in the sentence.
From the definition, free-text and keywords look mutually exclusive to me. Therefore I can't understand why I can't choose free-text on its own, and why it is possible to choose both simultaneously.
I think solution in the word 'extract' in free-text and 'matching' in keywords, cause in case of free-text it extract substring from message but this substring has to have some words matches with original message.
in this example there is 'i will be' mutual with all messages, so we need both free-text to extract and keyword to match.
Related
Lets say that I'm indexing a string value "useridA;useridB,userdidC,useridA,useridA"
The field is set to ANALYZED and uses a custom CharTokenizer which looks for a boundary comma char.
What is the expected behavior in the index, as the token "useridA" occurs multiples times within the same field?
Will it just re-index the same value an preserve the same space as if it would have been just one occurrence?
At the basic level lucene is an "inverted term index" it stores term->docID. So if a term occurs many times it'll only be recorded once.
Obviously this is a huge simplification. Positional information will also be stored depending on the TermVector value used when adding the field (you will need this to use phrase and slop queries).
Depending only your use-case I'd suggest you de-dupe the list either when indexing or just use a HashSet< string> for that property of whatever your class is.
So I've implemented the sunspot_rails gem into my application to utilize the powerful Solr search engine. I recently checked out Ryan's railscast on full-text searching and I noticed he was using additional parameters in his search queries such as "-" to denote something that should NOT be including in the full-text search.
I never heard about this until now, I was wondering if there was a user-friendly usage guide somewhere both me and my users can reference to take my search functionality to it's maximum capability.
I think ideally I would like to make an abridged version similar to Github's markdown cheat-sheet for my search forms that users can quickly reference.
Sunspot uses Solr's DisMax Query Parser, which has a very simple query syntax. For the most part, it is intended to flexibly parse user-created queries.
DisMax recognizes three special characters: +, -, and ". From the documentation:
[DisMax] is designed to be support raw input strings provided by users with no special escaping. '+' and '-' characters are treated as "mandatory" and "prohibited" modifiers for the subsequent terms. Text wrapped in balanced quote characters '"' are treated as phrases, any query containing an odd number of quote characters is evaluated as if there were no quote characters at all.
There are a few other "behind the scenes" options to tune the relevancy of matched documents. For example, "minimum match" specifies the number or proportion of "optional" fields (i.e., not prefixed with - or +) which must be present. As well as options to boost term matches in specific fields, or term matches within close proximity to each other, and so on.
In Sunspot, these are all exposed in the options parameter to the fulltext method, or as methods within a block supplied to that method.
I am trying to index in Lucene a field that could have RDF literal in different languages.
Most of the approaches I have seen so far are:
Use a single index, where each document has a field per each language it uses, or
Use M indexes, M being the number of languages in the corpus.
Lucene 2.9+ has a feature called Payload that allows to attach attributes to term. Is anyone use this mechanism to store language (or other attributes such as datatypes) information ? How is performance compared to the two other approaches ? Any pointer on source code showing how it is done would help. Thanks.
It depends.
Do you want to allow something like: "Search all english text for 'foo'"? If so, then you will need one field per language.
Or do you want "Search all text for 'foo' and present the user with which language the match was found in?" If this is what you want, then either payloads or separate fields will work.
An alternative way to do it is to index all your text in one field, then have another field saying the language of the document. (Assuming each document is in a single language.) Then your search would be something like +text:foo +language:english.
In terms of efficiency: you probably want to avoid payloads, since you would have to repeat the name of the language for every term, and you can't search based on payloads (at least not easily).
so basically lucene is a ranking algorithm, it just looks at strings and compares them to other string. they can be encoded in different character encodings but their similarity is the same non the less. Just make sure you load the SnowBallAnalyzer with the supported langugage stemmer and you should get results. Like say Spanish or Chinese
I'm looking for an Oracle Regular Expression that will match tokens in any order.
For example, say I'm looking for "one two".
I would want it to match both,
"one token two"
"two other one"
The number of tokens might grow larger than two, so generating the permutations for the regex would be a hassel.
Is there an easier way to do this, than this
'(ONE.*TWO)|(TWO.*ONE)'
i.e
select *
from some_table t
where regexp_like(t.NAME_KEY, '(ONE.*TWO)|(TWO.*ONE)')
Here's an alternative query that uses Full Text Search (FTS) functionality:
WHERE CONTAINS(t.name_key, 'ONE & TWO') > 0
See the Precedence Examples for criteria evaluation explanation.
Related:
Introduction to Oracle Text
You can use several different regular expressions:
SELECT *
FROM some_table t
WHERE regexp_like(t.NAME_KEY, 'ONE')
AND regexp_like(t.NAME_KEY, 'TWO')
One issue is that this will also match 'TWONE' which the original regular expression would not match. This can be fixed if you also check for some separating tokens or word boundary.
Also a regular expression is not necessary to match a constant string. You could just use LIKE instead.
I have below values in my database.
been Lorem Ipsum and scrambled ever
scrambledtexttextofandtooktooktypetexthastheunknownspecimenstandardsincetypesett
Here is my query:
SELECT
nBusinessAdID,
MATCH (`sHeadline`) AGAINST ("text" IN BOOLEAN MODE) AS score
FROM wiki_businessads
WHERE MATCH (`sHeadline`) AGAINST ("text" IN BOOLEAN MODE)
AND bDeleted ="0" AND nAdStatus ="1"
ORDER BY score DESC, bPrimeListing DESC, dDateCreated DESC
It's not fetching first result, why? It should fetch first result because its contain text word in it. I have disabled the stopword filtering.
This one is also not working
SELECT
nBusinessAdID,
MATCH (`sHeadline`) AGAINST ('"text"' IN BOOLEAN MODE) AS score
FROM wiki_businessads
WHERE MATCH (`sHeadline`) AGAINST ('"text"' IN BOOLEAN MODE)
AND bDeleted ="0" AND nAdStatus ="1"
ORDER BY score DESC, bPrimeListing DESC, dDateCreated DESC
The full text search only matches words and word prefixes. Because your data in the database does not contain word boundaries (spaces) the words are not indexed, so they are not found.
Some possible choices you could make are:
Fix your data so that it contains spaces between words.
Use LIKE '%text%' instead of a full text search.
Use an external full-text search engine.
I will expand on each of these in turn.
Fix your data so that it contains spaces between words.
Your data seems to have been corrupted somehow. It looks like words or sentences but with all the spaces removed. Do you know how that happened? Was it intentional? Perhaps there is a bug elsewhere in the system. Try to fix that. Find out where the data came from and see if it can be reimported correctly.
If the original source doesn't contain spaces, perhaps you could use some natural language toolkit to guess where the spaces should be and insert them. There most likely already exist libraries that can do this, although I don't happen to know any. A Google search might find something.
Use LIKE '%text%' instead of a full text search.
A workaround is to use LIKE '%text%' instead but note that this will be much slower as it will not be able to use the index. However it will give the correct result.
Use an external full-text search engine.
You could also look at Lucene or Sphinx. For example I know that Sphinx supports finding text using *text*. Here is an extract from the documentation which explains how to enable infix searching, which is what you need.
9.2.16. min_infix_len
Minimum infix prefix length to index. Optional, default is 0 (do not index infixes).
Infix indexing allows to implement wildcard searching by 'start*', '*end', and 'middle' wildcards (refer to enable_star option for details on wildcard syntax). When mininum infix length is set to a positive number, indexer will index all the possible keyword infixes (ie. substrings) in addition to the keywords themselves. Too short infixes (below the minimum allowed length) will not be indexed.
For instance, indexing a keyword "test" with min_infix_len=2 will result in indexing "te", "es", "st", "tes", "est" infixes along with the word itself. Searches against such index for "es" will match documents that contain "test" word, even if they do not contain "es" on itself. However, indexing infixes will make the index grow significantly (because of many more indexed keywords), and will degrade both indexing and searching times.