Find similar entries in SQL column and rank by frequency - sql

Question

I have a column of 10k URIs in my SQLite database. I would like to identify which of these URIs are subdomains of the same website.
For instance, for the given set...
1. daiquiri.rum.cu
2. mojito.rum.cu
3. cubalibre.rum.cu
4. americano.campari.it
5. negroni.campari.it
6. hemingway.com
... I would like to run a query that returns:
Website | Occurrences
----------------------------
rum.cu | 3
campari.it | 2
hemingway.com | 1
That is, the domain names / patterns that were matched, ranked by the number of times they were found in the database.
The heuristic I would use is: for every URI with 3+ domains, replace first domain with '%'and execute the pseudoquery: COUNT(uris from website where uris LIKE '%.remainderofmyuri').
Note that I don't care much about execution speed (in fact, not at all). The number of entries is within the range of 10k-100k.

Answer 1

The only problem is to find the domain. In order to find an algorithm imagine your urls with an additional dot in front (like '.negroni.campari.it' and '.hemingway.com'). You see then it's always the string that comes after the second dot from right. All we have to do is look for that occurrence and strip part of the string. Unfortunately, however, SQLite's string functions are rather poor. There is no function that gives you the second occurence of a dot, not even when counting from left. So the agorithm is great for most dbms, but it isn't for SQLite. We need another approach. (I am writing this anyhow, to show how to usually approach the problem.)
Here is the SQLite solution: The difference between a domain and the subdomains is that in the domain there is exactly one dot, whereas a subdomain has at least two. So when there is more than one dot, we must remove the first part including the first dot in order to get to the domain. Moreover we want this to work even with sub domains like abc.def.geh.ijk.com, so we must do this recursively.
with recursive cte(uri) as
(
select uri from uris
union all
select substr(uri, instr(uri, '.') + 1) as uri from cte where instr(uri, '.') > 0
)
select uri, count(*)
from cte
where length(uri) = length(replace(uri,'.','')) + 1 -- domains only
group by uri
order by count(*) desc;
Here we generate 'daiquiri.rum.cu' and 'rum.cu' and 'cu' from 'daiquiri.rum.cu' etc. So for every uri we get the domain (here 'rum.cu') and some other strings. At last we filter with LENGTH to get those strings that have exactly one dot - the domains. The rest is group by and count.
Here is the SQL fiddle: http://sqlfiddle.com/#!5/c1f35/37.

Answer 2

select x.site, count(*)
from mytable a
inner join
(
select 'rum.cu' as site
union all select 'campari.it'
union all select 'hemingway.com'
) x on a.url like '%' + x.site + '%'
group by x.site -- EDIT I missed out the GROUP BY on the first go - sorry!
(This is how I'd do it in SQL-Server; not sure how SQLite differs in syntax.)
'mytable' is your table whuch has a column called url containing 'mojito.rum.cu' etc. I haven't put the '%.' in the like because that would miss out hemmingway.com. However you could get around that by using this line instead:
) x on a.url like '%.' + x.site + '%' or a.url = x.site
You may not need the fimal + '%' - I put it in to catch urls like 'hemingway.com/some-page.html. If you don't have urls like that you can skip that.
EDIT for dynamic names
select x.site, count(*)
from mytable a
inner join
(
select distinct ltrim(url, instr(url, '.')) as site
from mytable
where url like '%.%.%'
union
select distinct url
from mytable
where url like '%.%' and url not like '%.%.%'
) x on a.url like '%' + x.site + '%'
group by x.site
Something like that should do it. I haven't tested that the INSTR() function is correct. You may need to add or subtract 1 from the offset it generates when you test it. It may not be the fastest query but it should work.