Search entities by weighted keywords and spelling correction - sql

Question

For starters, a little diagram relations-entities
Diagram relations-entities http://img11.hostingpics.net/pics/32979039DB.png
And now, a dataset
Archive
create :
CREATE TABLE archive (
id integer NOT NULL,
parent_id integer,
code character varying(15) NOT NULL,
label text NOT NULL
);
ALTER TABLE ONLY archive ADD CONSTRAINT archive_pkey PRIMARY KEY (id);
CREATE INDEX idx_142 ON archive USING btree (parent_id);
CREATE UNIQUE INDEX uniq_14242 ON archive USING btree (code);
ALTER TABLE ONLY archive ADD CONSTRAINT fk_14242 FOREIGN KEY (parent_id) REFERENCES archive(id);
insert :
INSERT INTO archive VALUES (1, NULL, 'B28', 'Confidential');
INSERT INTO archive VALUES (2, 1, 'B28.0', 'Nuclear zone');
Keyword
create :
CREATE TABLE keyword (
id integer NOT NULL,
label text NOT NULL,
label_double_metaphone text NOT NULL
);
ALTER TABLE ONLY keyword ADD CONSTRAINT eyword_pkey PRIMARY KEY (id);
CREATE UNIQUE INDEX uniq_242 ON keyword USING btree (label);
insert :
INSERT INTO keyword VALUES (1, 'SECURITY', 'SKRT');
INSERT INTO keyword VALUES (2, 'AREA', 'AR');
INSERT INTO keyword VALUES (3, 'NUCLEAR', 'NKLR');
Assoc_kw_archive
create :
CREATE TABLE assoc_kw_archive (
id integer NOT NULL,
keyword_id integer,
archive_id integer,
weight integer NOT NULL
);
ALTER TABLE ONLY assoc_kw_archive ADD CONSTRAINT assoc_kw_archive_pkey PRIMARY KEY (id);
CREATE INDEX idx_3421 ON assoc_kw_archive USING btree (archive_id);
CREATE INDEX idx_3422 ON assoc_kw_archive USING btree (keyword_id);
ALTER TABLE ONLY assoc_kw_archive ADD CONSTRAINT fk_3421 FOREIGN KEY (archive_id) REFERENCES archive(id);
ALTER TABLE ONLY assoc_kw_archive ADD CONSTRAINT fk_3422 FOREIGN KEY (keyword_id) REFERENCES keyword(id);
insert :
INSERT INTO assoc_kw_archive VALUES (1, 1, 1, 10);
INSERT INTO assoc_kw_archive VALUES (2, 1, 2, 20);
INSERT INTO assoc_kw_archive VALUES (3, 2, 2, 30);
INSERT INTO assoc_kw_archive VALUES (4, 3, 2, 30);
The target
The goal here is to search in the database. The research is based on a string typed by a user. Output a list of archives sorted by relevance. Relevant archive depends on three factors:
The people can make a mistake in the spelling of a word, etc...
The weight of a word to give it importance
Give a gain to the archives include the x keywords typed by the user
I worked on different versions of sql query, but, now I can't to step back and look at the overall problem.
The archive table is composed of 100,000 tuples, 80 000 for the table of keywords and 1,000,000 associations between these two entities.
This is my last version, she is functional, but is very slowly :
select f.id, f.code, f.label, min(f.dist) as distF, max(f.poid) as poidF
from
(
select
a.id,
a.code,
a.label,
( ( levenshtein(lower('Security'), lower(k1.label)) + 1 ) + ( levenshtein(lower('Nuclear'), lower(k2.label)) + 1 ) ) as dist,
( ka1.weight + ka2.weight ) as poid
from archive a
inner join assoc_kw_archive ka1
on ka1.archive_id = a.id
inner join keyword k1
on k1.id = ka1.keyword_id
inner join assoc_kw_archive ka2
on ka2.archive_id = a.id
inner join keyword k2
on k2.id = ka2.keyword_id
where levenshtein(dmetaphone('Security'), k1.label_double_metaphone) < 2
and levenshtein(dmetaphone('Nuclear'), k2.label_double_metaphone) < 2
) as f
group by f.id, f.code, f.label
order by distF asc, poidF desc
limit 10;
I made one join by keyword, it's this that makes it slow! But I can't find another solution.

Answer 1

I think the problem is doing the full join with the distance calculation. here is an alternative approach. Filter the keywords first. Keep the information in the where clause by using a subquery. Then use conditional aggregation to get the information you want.
The query ends up looking something like:
select a.id, a.code, a.label,
min( (levenshtein(lower('Security'), lower(case when securityl < 2 then k.label end)) + 1 ) +
(levenshtein(lower('Nuclear'), lower(case when nuclearl < 2 then k.label end)) + 1 )
) as mindist,
sum(weight) as poid
from archive a inner join
assoc_kw_archive ka
on ka.archive_id = a.id inner join
(select k.*, levenshtein(dmetaphone('Security'), k.label_double_metaphone) as securityl,
levenshtein(dmetaphone('Nuclear'), k.label_double_metaphone) as nuclearl
from keyword k
having securityl < 2 or
nuclearl < 2
) k
on k.id = ka.keyword_id
group by a.id, a.code, a.label