I'm using SQLite's FTS module to try to implement a full text search with windowing (or paging as some might call it) of results as well as highlighting of the search hits within each window. As far as I can tell it's straightforward to have either windowing or highlighting of search hits, but not both.
To implement windowing, I run the FTS and retrieve and store the docid of every hit in a temporary table, along the lines of:
CREATE VIRTUAL TABLE FullTextIndex USING FTS4 ( fulltext TEXT );
...
CREATE TEMP TABLE SearchResults ( id INTEGER );
INSERT INTO SearchResults
SELECT docid FROM FullTextIndex WHERE fullText MATCH 'mySearchExpression';
When I'm ready to retrieve the actual full text based on the windowing criteria, I do something like this:
SELECT FullTextIndex.fullText FROM FullTextIndex
INNER JOIN SearchResults
ON FullTextIndex.docid = SearchResults.id
WHERE *windowing criteria applied to SearchResults*;
That part works great and is very efficient because it's a rowid match.
However, at this point I cannot make use of SQLite's snippet function (or any other FTS auxiliary function) in order to highlight my search hits because I'm not using the text index in the query. Therefore I tried this:
SELECT snippet(FullTextIndex) FROM FullTextIndex
WHERE FullTextIndex.docID IN
(SELECT id FROM SearchResults WHERE *windowing criteria*)
AND FullTextIndex.fullText MATCH 'mySearchExpression';
It works, but it takes almost exactly as long as the original FTS because it is apparently running the FTS first, without applying the narrowing criteria based on SearchResults (and I suppose fundamentally it can't because of the way FTS indexing works).
My best solution so far is to make a second temporary table - a temporary full text index containing only the search hits for each window. In essence this:
CREATE VIRTUAL TABLE temp.TextResultsWindow USING FTS4 (id INTEGER, fullText text);
INSERT INTO temp.TextResultsWindow
SELECT FullTextIndex.docid, FullTextIndex.fullText
FROM FullTextIndex
WHERE FullTextIndex.docid IN
(SELECT id from SearchResults WHERE *windowing criteria*);
SELECT snippet(temp.TextResultsWindow.TextResultsWindow)
FROM temp.TextResultsWindow
WHERE temp.TextResultsWindow.fullText
MATCH 'mySearchExpression';
This does work, and it seems to be relatively fast for modest window sizes, but it seems terribly inefficient to be re-indexing a window of search hits just to have highlighting. If the volume of text in a particular window happened to be very large I could also easily see the window retrieval taking longer than the original FTS, just because it has to spend so much time re-indexing for each window just to use the snippet function. So while it works academically I don't think it'll be very good in practice.
Can anyone suggest a better approach? Plenty of applications have full text search with both windowing and highlighting. Do I have no choice but to just apply a manual scan and highlighting function to each full text result that I retrieve?
Thanks a lot.
UPDATE:
For what it's worth, there's a section in the SQLite documentation that discusses FTE3/4 and snippet, and paging/windowing. They're suggesting using LIMIT and OFFSET to do the windowing, so the scenario they're describing would indeed result in the full text search being re-run on the whole index for every window. I guess if the SQLite developers don't have a more efficient way of doing this it probably doesn't exist, but still would appreciate any other thoughts folks might have.
Context: this is a section I have cut out of a much larger stored Proc and found this is the part that is causing my error.
I have a table of data, put very simply it contains 2 records:
[Title]
Mitchell
Andrew
and I have a super simple query:
SELECT TOP 1
[Title]
FROM [Post] p
where CONTAINS (p.Title, '"Andrew"' )
This return 1 row successfully (this shows whatever indexing on this column needs to be there does work), however:
SELECT TOP 1
[Title]
FROM [Post] p
where CONTAINS (p.Title, '"Mitchell"' )
does not return any rows.
I know it is selectable because this does return the row:
SELECT TOP 1
[Title]
FROM [Post] p
where title like 'Mitchell'
I think the data is in error, because when I run this on a local backup of the database it returns the row fine.
So my question is what else can I do to investigate the fault?
Currently tried rebuilding indexes (although this column isn't explicitly indexed)
You appear to be using CONTAINS incorrectly. According to the documentation, the searched-for text is a word, meaning a string of characters without spaces or punctuation. Thus '"Mitchell"' is invalid as a search term. The surprise is not that it returns no rows, but that '"Andrew"' does.
Do you need CONTAINS, with all its complexity? The data you provide (which I understand is simplified) doesn't even require pattern-matching, let alone matching inflected forms and synonyms. (Andrew and Mitchell don't have inflected forms or synonyms.) A simple = would suffice.
So, the answer is, you're doing it wrong. If CONTAINS isn't needed, simplify your query and your life by using something more direct. And, whether it is or isn't, provide a search term consistent with the data and semantics of the function you're using.
Thanks for the input guys,
I had ran the indexes on the individual table but not rebuild the full text index catalogs using the following command:
ALTER FULLTEXT CATALOG [PostCatalog] REBUILD
After this completed, I can find both Andrew and Mitchell.
I think the reason I couldn't replicate locally is that when I restored the database, it rebuild it automatically.
Cheers guys
In Sql Server, I have a table containing 46 million rows.
In "Title" column of table, I want make search. The word may be at any index of field value.
For example:
Value in table: BROTHERS COMPANY
Search string: ROTHER
I want this search to match the given record. This is exactly what LIKE '%ROTHER%' do. However, LIKE '%%' usage should not be used on large tables because of performance issues. How can I achieve it?
Though I don't know your requirements, your best approach may be to challenge them. Middle-of-the-string searches are usually not very practical. If you can get your users to perform prefix searches (broth%) then you can easily use Full Text's wildcard search (CONTAINS(*, '"broth*"')). Full Text can also handle suffix searches (%rothers) with a little extra work.
But when it comes to middle-of-the-string searches with SQL Server, you're stuck using LIKE. However you may be able to improve performance of LIKE by using a binary collation as explained in this article. (I hate to post a link without including its content but it is way too long of an article to post here and I don't understand the approach enough to sum it up.)
If that doesn't help and if middle-of-the-string searches are that important of a requirement then you should consider using a different search solution like Lucene.
Add Full-Text index if you want.
You can search the table using CONTAINS:
SELECT *
FROM YourTable
WHERE CONTAINS(TableColumnName, 'SearchItem')
I have a query which slows down immensely when i add an addition where part
which essentially is just a like lookup on a varchar(500) field
where...
and (xxxxx.yyyy like '% blahblah %')
I've been racking my head but pretty much the query slows down terribly when I add this in.
I'm wondering if anyone has suggestions in terms of changing field type, index setup, or index hints or something that might assist.
any help appreciated.
sql 2000 enterprise.
HERE IS SOME ADDITIONAL INFO:
oops. as some background unfortunately I do need (in the case of the like statement) to have the % at the front.
There is business logic behind that which I can't avoid.
I have since created a full text catalogue on the field which is causing me problems
and converted the search to use the contains syntax.
Unfortunately although this has increased performance on occasion it appears to be slow (slower) for new word searchs.
So if i have apple.. apple appears to be faster the subsequent times but not for new searches of orange (for example).
So i don't think i can go with that (unless you can suggest some tinkering to make that more consistent).
Additional info:
the table contains only around 60k records
the field i'm trying to filter is a varchar(500)
sql 2000 on windows server 2003
The query i'm using is definitely convoluted
Sorry i've had to replace proprietary stuff.. but should give you and indication of the query:
SELECT TOP 99 AAAAAAAA.Item_ID, AAAAAAAA.CatID, AAAAAAAA.PID, AAAAAAAA.Description,
AAAAAAAA.Retail, AAAAAAAA.Pack, AAAAAAAA.CatID, AAAAAAAA.Code, BBBBBBBB.blahblah_PictureFile AS PictureFile,
AAAAAAAA.CL1, AAAAAAAA.CL1, AAAAAAAA.CL2, AAAAAAAA.CL3
FROM CCCCCCC INNER JOIN DDDDDDDD ON CCCCCCC.CID = DDDDDDDD.CID
INNER JOIN AAAAAAAA ON DDDDDDDD.CID = AAAAAAAA.CatID LEFT OUTER JOIN BBBBBBBB
ON AAAAAAAA.PID = BBBBBBBB.Product_ID INNER JOIN EEEEEEE ON AAAAAAAA.BID = EEEEEEE.ID
WHERE
(CCCCCCC.TID = 654321) AND (DDDDDDDD.In_Use = 1) AND (AAAAAAAA.Unused = 0)
AND (DDDDDDDD.Expiry > '10-11-2010 09:23:38') AND
(
(AAAAAAAA.Code = 'red pen') OR
(
(my_search_description LIKE '% red %') AND (my_search_description LIKE '% nose %')
AND (DDDDDDDD.CID IN (63,153,165,305,32,33))
)
)
AND (DDDDDDDD.CID IN (20,32,33,63,64,65,153,165,232,277,294,297,300,304,305,313,348,443,445,446,447,454,472,479,481,486,489,498))
ORDER BY AAAAAAAA.f_search_priority DESC, DDDDDDDD.Priority DESC, AAAAAAAA.Description ASC
You can see throwing in the my_search_description filter also includes a dddd.cid filter (business logic).
This is the part which is slowing things down (from a 1.5-2 second load of my pages down to a 6-8 second load (ow ow ow))
It might be my lack of understanding of how to have the full text search catelogue working.
Am very impressed by the answers so if anyone has any tips I'd be most greatful.
If you haven't already, enable full text indexing.
Unfortunately, using the LIKE clause on a query really does slow things down. Full Text Indexing is really the only way that I know of to speed things up (at the cost of storage space, of course).
Here's a link to an overview of Full-Text Search in SQL Server which will show you how to configure things and change your queries to take advantage of the full-text indexes.
More details would certainly help, but...
Full-text indexing can certainly be useful (depending on the more details about the table and your query). Full Text indexing requires a good bit of extra work both in setup and querying, but it's the only way to try to do the sort of search you seek efficiently.
The problem with LIKE that starts with a Wildcard is that SQL server has to do a complete table scan to find matching records - not only does it have to scan every row, but it has to read the contents of the char-based field you are querying.
With or without a full-text index, one thing can possibly help: Can you narrow the range of rows being searched, so at least SQL doesn't need to scan the whole table, but just some subset of it?
The '% blahblah %' is a problem for improving performance. Putting the wildcard at the beginning tells SQL Server that the string can begin with any legal character, so it must scan the entire index. Your best bet if you must have this filter is to focus on your other filters for improvement.
Using LIKE with a wildcard at the beginning of the search pattern forces the server to scan every row. It's unable to use any indexes. Indexes work from left to right, and since there is no constant on the left, no index is used.
From your WHERE clause, it looks like you're trying to find rows where a specific word exists in an entry. If you're searching for a whole word, then full text indexing may be a solution for you.
Full text indexing creates an index entry for each word that's contained in the specified column. You can then quickly find rows that contain a specific word.
As other posters have correctly pointed out, the use of the wildcard character % within the LIKE expression is resulting in a query plan being produced that uses a SCAN operation. A scan operation touches every row in the table or index, dependant on the type of scan operation being performed.
So the question really then becomes, do you actually need to search for the given text string anywhere within the column in question?
If not, great, problem solved but if it is essential to your business logic then you have two routes of optimization.
Really go to town on increasing the overall selectivity of your query by focusing your optimization efforts on the remaining search arguments.
Implement a Full Text Indexing Solution.
I don't think this is a valid answer, but I'd like to throw it out there for some more experienced posters comments...are these equivlent?
where (xxxxx.yyyy like '% blahblah %')
vs
where patindex(%blahbalh%, xxxx.yyyy) > 0
As far as I know, that's equivlent from a database logic standpoint as it's forcing the same scan. Guess it couldn't hurt to try?
So apparently if a Mysql table's fulltext index contains a keyword that appears in 50% of the data rows, that keyword will be ignored by the match query
So if I have a table with the fulltext index 'content' which contains 50 entries
and 27 of the entries contains the word 'computer' in the content field, and I run the query:
SELECT *
FROM `table`
WHERE MATCH(`content`) AGAINST ('computer');
...the computer query will return zero results since computer appears in more than 50% of the entries and hence the keyword is ignored...
is there a way to disable this functionality especially since this is problematic in the beginning phase of the database's lifespan
Use BOOLEAN full-text searches to bypass 50% feature.
http://dev.mysql.com/doc/refman/5.0/en/fulltext-boolean.html
Yes, use the Boolean Full-Text Searched option
See here Mysql Manual
The manual says that "They do not use the 50% threshold."
The search in MySQL is not intuitive and the manual is slightly confusing so you need to read it carefully as well as preparing some test cases to make sure it is working and that you have implemented it correctly (from bitter experience).
There are a number of other search plugin's of varying complexity that you might want to look at that perform better but it is worth getting to grips with the native version as it is quick and (easy?) dirty.