Parsing simplified SQL queries with Perl into SQLite - sql

Question

I am trying to turn a "simplified" sql query into a working SQLite one to use against XnViews databases, meaning the database layout is at best suboptimal for what I'm trying to do AND I can't change anything about that.
Example would be "(cat_10 and cat_12) and (cat_5 or cat_7)".
This should be used against the table "t3", which has the fields "if" (fileID) and "ic" (categoryID).
The entries look like this:
if, ic
7, 10
7, 12
7, 4
9, 10
9, 12
9, 5
10, 10
10, 12
10, 7
The simplified query above should only select the files 9 and 10 as 7 does have the wanted categories 10 and 12 but has neither 5 nor 7.
The actual problem now is building that hell of a query statement because it took me already some hours to simply get an AND between two categories working.
SELECT if FROM t3 WHERE ic IN (10, 12) GROUP BY if HAVING count(if) = 2
This gives me all fileIDs that contain category 10 and 12, but I have no idea how I should combine that with the remaining " and (cat_5 or cat_7)".
When I planned these simplified sql statements (made by a click-it-together-builder made in html and js) I was planning to simply replace "cat_5" with "t3.ic = 5" and leave the rest as it is.
Of course I didn't forsee that it wouldn't work as where checks the entry as a whole and there can't be ic = 5 AND ic = 7. That pretty much broke everything.
So I'm wondering if anyone would have an idea how I could translate these simple queries in actual working ones, keeping in mind that it might not be limited to ( x and y ) pairs.
Edit: I worked out how to do the example I've given, I think atleast:
SELECT if FROM t3 WHERE ic IN (10, 12) GROUP BY if HAVING count(if) = 2
INTERSECT
SELECT if FROM t3 WHERE ic IN (5, 7) GROUP BY if
But the main problem now is resolving the ( ) in the right order.
Edit 2: I think I'm giving grouping the categories into one field with group_concat() a try, then I should be able to simply to cats LIKE " " AND which would be small blocks I could easy throw together, then just the brackets and it should work. Highlighting the 'should'.

Answer 1

Your original query doesn't do what is intended. WHERE ic IN (10, 12) GROUP BY if HAVING count(if) = 2 would yield the right result even when you have ics in 10 and 10 again but not 12 at all. This is against your textual description of what you want. This is where an inner query to fetch results for 12 and 10 is needed. You can test your query to fail in the fiddle link I have posted below.
Bit tricky, but this is how I would interpret it straightforward.
SELECT DISTINCT ifc
FROM t3
WHERE ifc IN (
SELECT ifc
FROM t3
WHERE ic = 10
GROUP BY ifc
HAVING COUNT(*) > 0
INTERSECT
SELECT ifc
FROM t3
WHERE ic = 12
GROUP BY ifc
HAVING COUNT(*) > 0
)
AND ic IN (5, 7)
Try fiddle
I did not bring in any optimization, you may try yours. The fiddle link is of Postgres but this should work ( did not get SQLite to work in my browser :( )
Edit: CL. points out an interesting thing about not having to include HAVING clauses in the inner query which is true. I was interpreting OP's requirement in SQL terms with an intent to make things clear without thinking of any optimizations.
Here is a better looking query:
SELECT DISTINCT ifc
FROM t3
WHERE ifc IN (
SELECT ifc
FROM t3
WHERE ic = 10
INTERSECT
SELECT ifc
FROM t3
WHERE ic = 12
)
AND ic IN (5, 7)

Answer 2

Ok I got it working as I originally planned surprisingly.
SELECT Folders.Pathname || Images.Filename AS File FROM Images
JOIN Folders ON Images.FolderID = Folders.FolderID
LEFT JOIN (
SELECT f, Cats, t4.if AS Tagged FROM t2
JOIN (
SELECT if, ' ' || group_concat(ic,' ') || ' ' AS Cats FROM t3 GROUP BY if
) st3 ON t2.i = st3.if
LEFT JOIN t4 ON t2.i = t4.if
) st2 ON File = st2.f
$selectWhereImage $sqlqry
ORDER BY ModifiedDate $order LIMIT $offset, $limit
I know this is one hell of a query but it combines all things I'd be looking for (category ids, tagged or not, rating, color) sortable by date with the full filepath as result.
It's probably a horrible way to do it but if anyone finds a better working way where I can simply replace placeholders like "cat_5" while keeping the rest like it is, needed for brackets and operators, then I'm all ears :D
Oh and $selectWhereImage contains just a longer WHERE that limits File to be ending with an imageformat, $sqlqry is the refittet thing from above, cat_5 would just turn into cats LIKE '% 5 %', due to the additional spaces left and right of cats I can match any number without finding "1" in "10" since " 1 " isn't in " 10 " :D

Answer 3

A hackish approach which would be simpler and I believe faster is
SELECT DISTINCT ifc
FROM t3
WHERE ifc IN (
SELECT ifc
FROM t3
WHERE ic = 10
)
AND ifc IN (
SELECT ifc
FROM t3
WHERE ic = 12
)
AND ic IN (5, 7)

Answer 4

If you have to use an intersect as you have done you should change your upper query which is wrong. Since you have to ensure every if has a 10 and 12 as ic then you can't get away without two separate queries for that. Something like:
SELECT ifc
FROM t3
WHERE ifc IN (
SELECT ifc
FROM t3
WHERE ic = 10
)
AND ifc IN (
SELECT ifc
FROM t3
WHERE ic = 12
)
INTERSECT
SELECT ifc FROM t3 WHERE ic IN (5, 7)
The INTERSECT will handle the group by here so you dont have to explicitly add but this will not be as efficient as my other queries. If you have to get away with subqueries, you can use JOIN:
SELECT DISTINCT t.ifc
FROM t3 AS t
JOIN t3 AS v ON v.ifc = t.ifc
JOIN t3 AS p ON p.ifc = t.ifc
WHERE v.ic = 10 AND p.ic = 12 AND t.ic IN (5, 7)
The second one has the advantage that it works on databases that doesn't know INTERSECT like MySQL.