I'm very noob in relation to Full Text search and I was told to do a full text search over 2 tables and sort results by relevance.
I will look on table "Posts" and table "PostComments". I must look for the search term (let's say "AnyWord") on Posts.Description, Posts.Title and PostComments.Comments.
I have to return Posts order by relevance but since I'm looking on Posts AND PostComments I don't know if this make sense. I'd say I need all the information on the same table in order to sort by relevance.
Could you help me to figure out if this make sense and if it does how to achieve it?
EDIT
I'll try to explain a little better what I need.
A Post is relevant for the search if the searched term is present on the title, on the description or on any of the related PostComments.
But on the front end I will show a list of post. The title of the post on this list is a link to the post itself. The post comments are visible there but not on the search result list, although they are involved on the search process.
So you could have posts on the search result that matched JUST because the search term is present on one or more comments
Only ContainsTable returns an evaluation of relevance. You did not mention what needed to be returned so I simply returned the name of the table from where the value is stored along with the given table's primary key (you would replace "PrimaryKey" with your actual primary key column name e.g. PostId or PostCommentsId), the value and its rank.
Select Z.TableName, Z.PK, Z.Value, Z.Rank
From (
Select 'Posts' As TableName, Posts.PrimaryKey As PK, Posts.Description As Value, CT.Rank
From Posts
Join ContainsTable( Posts, Description, 'Anyword' ) As CT
On CT.Key = Posts.PrimaryKey
Union All
Select 'PostComments', PostComments.PrimaryKey, PostComments.Comments, CT.Rank
From PostComments
Join ContainsTable( PostComments, Comments, 'Anyword' ) As CT
On CT.Key = PostComments.PrimaryKey
) As Z
Order By Z.Rank Desc
EDIT Given the additional information, it is much clearer. First, it would appear that the ranking of the search has no bearing on the results. So, all that is necessary is to use an OR between the search on post information and the search on PostComments:
Select ...
From Posts
Where Contains( Posts.Description, Posts.Title, 'searchterm' )
Or Exists (
Select 1
From PostComments
Where PostComments.PostId = Posts.Id
And Contains( PostComments.Comments, 'searchterm' )
)
Related
I've searched the way of improving this dangerous combination of functions in one SQL sentence...
To put you in a context, i have a table with several information about articles (article_id, author, ...) and another one containing the article_id with one tag_id. As an article is able to have several tags, that second table could have 2 rows with the same article_id and different tag_id.
In order to get a list of the 8 articles that have more tags in common with the one that i want (in this case the 1354) I have written the following query:
SELECT articles.article_id, articles.author, count(articles_tags.article_id) as times
FROM articles
INNER JOIN articles_tags ON (articles.article_id=articles_tags.article_id)
WHERE id_tag IN
(SELECT article_id FROM articles_tags WHERE article_id=1354)
AND article_id <> 1354
GROUP BY article_id
ORDER BY times DESC
LIMIT 8
It is EXTREMELY slow... like 90 seconds for half million articles.
By deleting the "order by times" sentence, it works almost instantly, but if i do so, i won't get the most similar articles.
What can i do?
Thanks!!
a query on a sub-select is ALWAYS a time-killer... Also, as the query didn't really appear to be accurate, or missing, I am making an assumption that your articles_tags table has two columns... one for the actual article ID, and another for the tag_ID associated with it.
That said, I would pre-query just the TAG IDs for article 1354 (the on you are interested in). Use that as a Cartesian join to the article tags again on the tag IDs being the same. From that, you are grabbing the SECOND version of article tags alias and getting ITs article ID, and then the count that MATCH (via Join and not a left-join). Apply the group by on the article ID as you had, And for grins, join to the articles table to get the author.
Now, note. Some SQL engines require you to group by all non-aggregate fields, so you MAY have to either add the author to the group by (which will always be the same per article ID anyway), or change it to MAX( A.author ) as Author which would give the same results.
I would have an index on the (tag_id, article_id) so the tags are found from the "common" tags you are looking to find in common. You could have one article with 10 tags, and another article with 10 completely different tags resulting in 0 in common. This will prevent the other article from even appearing in the result set.
You STILL have the time associated with blowing through half-million articles as you described, which could be millions of actual tag entries.
select
AT2.article_id,
A.Author,
count(*) as Times
from
( select ATG.id_tag
from articles_tags ATG
where ATG.Article_ID = 1354
order by ATG.id_tag ) CommonTags
JOIN articles_tags AT2
on CommonTags.ID_Tag = AT2.ID_Tag
AND AT2.Article_ID <> 1354
JOIN articles A
on AT2.Article_ID = A.Article_ID
group by
AT2.article_id
order by
Times DESC
limit 8
It seems that it should be possible to do this without any subqueries, and then a quicker query may result.
Here the article of interest is joined to its tags, and then further to other articles having these tags. Then the number of tags for each article is counted and ordered:
SELECT a2.article_id, a2.author, COUNT(t2.tag_id) AS times
FROM articles a1
INNER JOIN articles_tags t1
ON t1.article_id = a1.article_id -- find tags for staring article
INNER JOIN tags t2
ON t2.tag_id = t1.tag_id -- find other instances of those tags
AND t2.articles_id <> t1.articles_id
INNER JOIN articles a2
ON a2.articles_id = t2.articles_id -- and the articles where they are used
WHERE a1.article_id = 1354
GROUP BY a2.article_id, a2.author -- count common tags by articles
ORDER BY times DESC
LIMIT 8
If you know a lower bound on the number of tags in common (e.g. 3), inserting HAVING times > 2 before ORDER BY times DESC could give a further speed improvement.
Background: I'm working on a homebrew project for managing a collection of my own images, and have been trying to implement a tag-based search so I can easily sift through them.
Right now, I'm working with RedBean's tagging API for applying tags to each image's database entry, however I'm stuck on a specific detail of my implementation; currently, to allow search of tags where multiple tags will refine the search (when searching for "ABC XYZ", tagged image must have tags "ABC" and "XYZ"),
I'm having to handle some of the processing in the server-side language and not SQL, and then run an (optional) second query to verify that any returned images don't have a tag that has been explicitly excluded from results. (when searching for "ABC -XYZ", tagged image must have tag "ABC" and not "XYZ").
The problem here is that my current method requires that I run all results by my server-side code, and makes any attempts at sensible pagination/result offsets inaccurate.
My goal is to just grab the rows of the post table that contain the requested tags (and not contain any excluded tags) with one query, and still be able to use LIMIT/OFFSET arguments for my query to obtain reasonably paginated results.
Table schemas is as follows:
Table "post"
Columns:
id (PRIMARY KEY for post table)
(image metadata, not relevant to tag search)
Table "tag"
Columns:
id (PRIMARY KEY for tag table)
title (string of varying length - assume varchar(255))
Table "post_tag"
Columns:
id (PRIMARY KEY for post_tag table)
post_id (associated with column "post.id")
tag_id (associated with column "tag.id")
If possible, I'd like to also be able to have WHERE conditions specific to post table columns as well.
What should I be using for query structure? I've been playing with left joins but haven't been able to get the exact structure I need to solve this.
Here is the basic idea:
The LEFT OUTER JOIN is the set of posts that match tags you want to exclude. The final WHERE clause in the query makes sure that none of these posts match the entries in the first post table.
The INNER JOIN is the set of posts that match all of the tags. Note that the number two must match the number of unique tag names that you provide in the IN clause.
select p.*
from post p
left outer join (
select pt.post_id
from post_tag pt
inner join tag t on pt.tag_id = t.id
where t.title in ('UVW', 'XYZ')
) notag on p.id = notag.post_id
inner join (
select pt.post_id
from post_tag pt
inner join tag t on pt.tag_id = t.id
where t.title in ('ABC', 'DEF')
group by pt.post_id
having count(distinct t.title) = 2
) yestag on p.id = yestag.post_id
where notag.post_id is null
--add additional WHERE filters here as needed
I'm trying to select a maximum of 10 related articles, where a related article is an article that has 3 or more of the same keywords as the other article.
My table structure is as follows:
articles[id, title, content, time]
tags[id, tag]
articles_tags[article_id, tag_id]
Can I select the related articles id and title all in one query?
Any help is greatly appreciated.
Assuming that title is also unique
SELECT fA.ID, fA.Title
from
Articles bA,
articles_tags bAT,
articles_tags fAT,
Articles fA
where
bA.title = 'some name' AND
bA.id = bAT.Article_Id AND
bAT.Tag_ID = fAT.Tag_ID AND
fAT.Article_ID = fA.ID AND
fA.title != 'some name'
GROUP BY
fA.ID, fA.Title
HAVING
count(*) >= 3
Where to exclude the 'seed' article
Because I don't care exactly WHICH tags I match on, just THAT I match on 3 tags, I only need tag_id and avoid the join to the tags table completely. So now I join the many-to-many table to itself to find the articles which have an overlap.
The problem is that the article will match itself 100% so we need to eliminate that from the results.
You can exclude that record in 3 ways. You can filter it from the table to before joining, you can have it fall out of the join, or you can filter it when you're finished.
If you eliminate it before you begin the join, you're not gaining much of an advantage. You've got thousands or millions of articles and you're only eliminating 1. I also believe this will not be useful based on the best index for the article_tag mapping table.
If you do it as part of the join, the inequality will prevent that clause from being part of the index scan and be applied as a filter after the index scan.
Consider the index on article_tags as (Tag_ID, Article_ID). If I join the index to itself on tag_id = tag_id then I'll immediately define the slice of the index to process by walking the index to each tag_id my 'seed' article has. If I add the clause article_id != article_id, that can't use the index to define the slice to be processed. That means it will be applied as a filter. e.g. Say my first tag is "BLUE". I walk the index to get all the articles which have "BLUE". (by ID of course). Say there are 50 rows. We know that 1 is my seed article and 49 are matches. If I don't include the inequality, I include all 50 records and move on. If I do include the inequality, I then have to check each of the 50 records to see which is my seed and which isn't. The next tag is "Jupiter" and it matches 20,000 articles. Again I have to check each row in that slice of the index to exclude my seed article. After I go through this 2,5,20 times (depends on tags for that seed article), I now have a completely clean set of articles to do the COUNT(*) and HAVING on. If I don't include the inequality as part of my join but instead just filter the SEED ID out after the group by and having then I only do that filer once on a very short list.
#updated to exclude the searched article itself!
Something along these lines
select *
from articles
inner join (
select at2.article_id, COUNT(*) cnt
from articles a
inner join articles_tags at on at.article_id = a.id
# find all matching tags to get the article ids
inner join articles_tags at2 on at2.tag_id = at.tag_id
and at2.article_id != at.article_id
where a.id = 1234 # the base article to find matches for
group by at2.article_id
having count(*) >= 3 # at least 3 matching keywords
) matches on matches.article_id = articles.id
order by matches.cnt desc
limit 10; # up to 10 matches required
If you can write a query to get ids of records that have matches, then you can certainly have that same query return you the titles. If your real question is 'how do I write the query to return the matches?', then please say so and I'll edit this answer with more details along those lines.
Let's assume I have a database with two tables: categories and articles. Every article belongs to a category.
Now, let's assume I want to fetch the latest article of each category that fits a specific criteria (read: the article does). If it weren't for that extra criteria, I could just add a column called last_article_id or something similar to the categories table - even though that wouldn't be properly normalized.
How can I do this though? I assume there's something using GROUP BY and HAVING?
Try with:
SELECT *
FROM categories AS c
LEFT JOIN (SELECT * FROM articles ORDER BY id DESC) AS a
ON c.id = a.id_category
AND /criterias about joining/
WHERE /more criterias/
GROUP BY c.id
If you provide us with the Tables schemas, we could be a little more specific, but you could try something like (12.2.9.6. EXISTS and NOT EXISTS, SELECT Syntax for LIMIT)
SELECT *
FROM articles a
WHERE EXISTS (
SELECT 1
FROM articles
where category_id = a.category_id
AND <YourCriteria Here>
ORDER BY <Order Required : ID DESC, LastDate DESC or something?
LIMIT 1
)
Assuming the id's in the articles table represent always increasing numbers, this should work. Using the id is not semantically correct IMHO, you should actually use a time/date tamp field if one is available.
SELECT * FROM articles WHERE article_id IN
(
SELECT
MAX(article_id)
FROM
articles
WHERE [your filters here]
GROUP BY
category_id
)
I am trying to write a sql query which fetches all the tags related to every topic being displayed on the page.
like this
TITLE: feedback1
POSTED BY: User1
CATEGORY: category1
TAGS: tag1, tag2, tag3
TITLE: feedback2
POSTED BY: User2
CATEGORY: category2
TAGS: tag2, tag5, tag7,tag8
TITLE: feedback3
POSTED BY: User3
CATEGORY: category3
TAGS: tag1, tag5, tag6, tag3
The relationship of tags to topics is many to many.
Right now I am first fetching all the topics from the "topics" table and to fetch the related tags of every topic I loop over the returned topics array for fetching tags.
But this method is very expensive in terms of speed and not efficient too.
Please help me write this sql query.
Query for fetching all the topics and its information is as follows:
SELECT
tbl_feedbacks.pk_feedbackid as feedbackId,
tbl_feedbacks.type as feedbackType,
DATE_FORMAT(tbl_feedbacks.createdon,'%M %D, %Y') as postedOn,
tbl_feedbacks.description as description,
tbl_feedbacks.upvotecount as upvotecount,
tbl_feedbacks.downvotecount as downvotecount,
(tbl_feedbacks.upvotecount)-(tbl_feedbacks.downvotecount) as totalvotecount,
tbl_feedbacks.viewcount as viewcount,
tbl_feedbacks.title as feedbackTitle,
tbl_users.email as userEmail,
tbl_users.name as postedBy,
tbl_categories.pk_categoryid as categoryId,
tbl_clients.pk_clientid as clientId
FROM
tbl_feedbacks
LEFT JOIN tbl_users
ON ( tbl_users.pk_userid = tbl_feedbacks.fk_tbl_users_userid )
LEFT JOIN tbl_categories
ON ( tbl_categories.pk_categoryid = tbl_feedbacks.fk_tbl_categories_categoryid )
LEFT JOIN tbl_clients
ON ( tbl_clients.pk_clientid = tbl_feedbacks.fk_tbl_clients_clientid )
WHERE
tbl_clients.pk_clientid = '1'
What is the best practice that should be followed in such cases when you need to display all the tags related to every topic being displayed on a single page.
How do I alter the above sql query, so that all the tags plus related information of topics is fetched using a single query.
For a demo of what I am trying to achieve is similar to the'questions' page of stackoverflow.
All the information (tags + information of every topic being displayed) is properly displayed.
Thanks
To do this, I would have three tables:
Topics
topic_id
[whatever else you need to know for a topic]
Tags
tag_id
[etc]
Map
topic_id
tag_id
select t.[whatever], tag.[whatever]
from topics t
join map m on t.topic_id = m.topic_id
join tags tag on tag.tag_id = m.tag_id
where [conditionals]
Set up partitions and/or indexes on the map table to maximize the speed of your query. For example, if you have many more topics than tags, partition the table on topics. Then, each time you grab all the tags for a topic, it will be 1 read from 1 area, no seeking needed. Make sure to have both topics and tags indexed on their _id.
Use your 'explain plan' tool. (I am not familiar with mysql, but I assume there is some tool that can tell you how a query will be run, so you can optimize it)
EDIT:
So you have the following tables:
tbl_feedbacks
tbl_users
tbl_categories
tbl_clients
tbl_tags
tbl_topics
tbl_topics_tags
The query you provide as a starting point shows how feedback, users, categories and clients relate to each other.
I assume that tbl_topics_tags contains FKs to tags and topics, showing which topic has which tag. Is this correct?
What of (feedbacks, users, categories, and clients) has a FK to topics or tags? Or, do either topics or tags have a FK to any of the initial 4?
Once I know this, I'll be able to show how to modify the query.
EDIT #2
There are two different ways to go about this:
The easy way is the just join on your FK. This will give you one row for each tag. It is much easier and more flexible to put together the SQL to do it this way. If you are using some other language to take the results of the query and translate them to present them to the user, this method is better. If nothing else, it will be far more obvious what is going on, and will be easier to debug and maintain.
However, you may want each row of the query results to contain one feedback (and the tags that go with it).
SQL joining question <- this is a question I posted on how to do this. The answer I accepted is an oracle-only answer AFAIK, but there are other non-oracle answers.
Adapting Kevin's answer (which is supposed to work in SQL92 compliant systems):
select
[other stuff: same as in your post],
(select tag
from tbl_tag tt
join tbl_feedbacks_tags tft on tft.tag_id = tt.tag_id
where tft.fk_feedbackid = tbl_feedbacks.pk_feedbackid
order by tag_id
limit 1
offset 0 ) as tag1,
(select tag
from tbl_tag tt
join tbl_feedbacks_tags tft on tft.tag_id = tt.tag_id
where tft.fk_feedbackid = tbl_feedbacks.pk_feedbackid
order by tag_id
limit 1
offset 1 ) as tag2,
(select tag
from tbl_tag tt
join tbl_feedbacks_tags tft on tft.tag_id = tt.tag_id
where tft.fk_feedbackid = tbl_feedbacks.pk_feedbackid
order by tag_id
limit 1
offset 2 ) as tag3
from [same as in the OP]
This should do the trick.
Notes:
This will pull the first three tags. AFAIK, there isn't a way to have an arbitrary number of tags. You can expand the number of tags shown by copying and pasting more of those parts of the query. Make sure to increase the offset setting.
If this does not work, you'll probably have to write up another question, focusing on how to do the pivot in mysql. I've never used mysql, so I'm only guessing that this will work based on what others have told me.
One tip: you'll usually get more attention to your question if you strip away all the extra details. In the question I linked to above, I was really joining between 4 or 5 different tables, with many different fields. But I stripped it down to just the part I didn't know (how to get oracle to aggregate my results into one row). I know some stuff, but you can usually do far better than just one person if you trim your question down to the essentials.