apache derby hangs up executing query - sql

Question

i'd really appreciate some help here.
I've got this derby database from my customer, and their application based on Hibernate. I can't make it work because Hibernate generates query that hangs up the whole database:
select houses0_.STREET_ID as STREET5_1_, houses0_.ID as ID1_, houses0_.ID as ID42_0_, houses0_.NAME as NAME42_0_, houses0_.SHORT_NAME as SHORT3_42_0_, houses0_.INDEX as INDEX42_0_ from SRG.HOUSE_REFERENCE houses0_ where houses0_.STREET_ID in (select streets0_.ID from SRG.STREET_REFERENCE streets0_ where streets0_.TOWNSHIP_ID in (select townships0_.ID from SRG.TOWNSHIP_REFERENCE townships0_ where townships0_.CITY_ID='555'))
I dont see anything suspicious in it, but when i launch derby with it - derby just does nothing. Here are my logs:
2011-10-31 21:58:10.328 GMT Thread[DRDAConnThread_3,5,main] (XID = 1152939), (SESSIONID = 1), (DATABASE = dev), (DRDAID = NF000001.FC97-434877518919679629{1}), Begin compiling prepared statement: select houses0_.STREET_ID as STREET5_1_, houses0_.ID as ID1_, houses0_.ID as ID42_0_, houses0_.NAME as NAME42_0_, houses0_.SHORT_NAME as SHORT3_42_0_, houses0_.INDEX as INDEX42_0_ from HOUSE_REFERENCE houses0_ where houses0_.STREET_ID in (select streets0_.ID from STREET_REFERENCE streets0_ where streets0_.TOWNSHIP_ID in (select townships0_.ID from TOWNSHIP_REFERENCE townships0_ where townships0_.CITY_ID='555')) :End prepared statement
2011-10-31 21:58:10.682 GMT Thread[DRDAConnThread_3,5,main] (XID = 1152939), (SESSIONID = 1), (DATABASE = dev), (DRDAID = NF000001.FC97-434877518919679629{1}), End compiling prepared statement: select houses0_.STREET_ID as STREET5_1_, houses0_.ID as ID1_, houses0_.ID as ID42_0_, houses0_.NAME as NAME42_0_, houses0_.SHORT_NAME as SHORT3_42_0_, houses0_.INDEX as INDEX42_0_ from SRG.HOUSE_REFERENCE houses0_ where houses0_.STREET_ID in (select streets0_.ID from SRG.STREET_REFERENCE streets0_ where streets0_.TOWNSHIP_ID in (select townships0_.ID from SRG.TOWNSHIP_REFERENCE townships0_ where townships0_.CITY_ID='555')) :End prepared statement
2011-10-31 21:58:10.758 GMT Thread[DRDAConnThread_3,5,main] (XID = 1152939), (SESSIONID = 1), (DATABASE = dev), (DRDAID = NF000001.FC97-434877518919679629{1}), Executing prepared statement: select houses0_.STREET_ID as STREET5_1_, houses0_.ID as ID1_, houses0_.ID as ID42_0_, houses0_.NAME as NAME42_0_, houses0_.SHORT_NAME as SHORT3_42_0_, houses0_.INDEX as INDEX42_0_ from SRG.HOUSE_REFERENCE houses0_ where houses0_.STREET_ID in (select streets0_.ID from SRG.STREET_REFERENCE streets0_ where streets0_.TOWNSHIP_ID in (select townships0_.ID from SRG.TOWNSHIP_REFERENCE townships0_ where townships0_.CITY_ID='555')) :End prepared statement
And NOTHING after that.
I've set some properties:
derby.drda.portNumber=1529
derby.locks.deadlockTrace=true
derby.locks.monitor=true
derby.locks.deadlockTimeout=1
derby.locks.waitTimeout=1
derby.language.logStatementText=false
derby.infolog.append=true
derby.stream.error.logSeverityLevel=0
derby.drda.logConnections=true
derby.language.logStatementText=true
derby.language.logQueryPlan=true
to see whats going on, but there's nothing in logs, this query just takes eternity to execute, either its done from hibernate or squirrelsql or derby's ij tool.
As you can see timeouts are pretty small but derby doesnt stop executing query, it leaves it to do whatever the query wants.
Query is not that complex - i rewrote it using joins and it took 1 sec to execute. But not in that form - i dont get it, why? Can you give me any hints?
UPD: house_reference contains 7508 rows, selecting * from house_reference takes less than a second, street_reference contains ~44k rows, and subquery
select streets0_.ID from STREET_REFERENCE streets0_ where streets0_.TOWNSHIP_ID in (select townships0_.ID from TOWNSHIP_REFERENCE townships0_ where townships0_.CITY_ID='52000001000')
returns 111 rows in less than a second. so selecting * from house_reference with in clause containing 111 rows takes 22 minutes to execute, and it returns 0 results. That doesnt look ok to me, but im not that experienced in database administration to judge, thats why im asking the community :)

Answer 1

Queries specifying "in (subquery)" cannot always be replaced with a corresponding join. This process is called "flattening", and there are rules that the database has to follow in order to give the right answer. See this documentation: http://db.apache.org/derby/docs/10.8/tuning/ctuntransform36368.html
Unfortunately it's not uncommon to have significant performance problems with subqueries in joins. I'm glad you found a workaround (assuming that your re-written query is actually an acceptable workaround for you).
Object-relational mapping libraries can generate some complex queries. If you can get the query to eventually run, you may be able to learn something from the query plan.
Another possibility is that the database statistics have become inaccurate, causing Derby to choose a poor query plan. Recent versions of Derby have improved support for re-computing the statistics, which can help the optimizer choose a better query plan.
There is a wealth of information in the Derby Tuning guide at http://db.apache.org/derby/docs/10.8/tuning/index.html