I have Events, which can have many EventTypes. I'm attempting to query the database by using the NHibernate mappings I have defined. The issue is that when defining criteria on child objects, NHibernate makes repeated calls to the database, rather than making one call for all the data it needs. This leads to an extremely slow search.
The database tables are as follows:
Event
-EventID
EventType
-EventTypeID
EventEventTypeID
-EventEventTypeID
-EventID
-EventTypeID
The Event class looks like this:
Id as Integer
Types as IList(Of EventType)
The EventType class looks like this:
Id as Integer
FullTitle as String
There is no EventEventType class.
The NHibernate mappings are as follows:
EventMapping
Table("event.Event")
Id(Function(x) x.Id).Column("EventID").GeneratedBy().Identity()
HasManyToMany(Of EventType)(Function(x) .Types).Table("event.EventEventType").ParentKeyColumn("EventID").ChildKeyColumn("EventTypeID").Cascade.All()
EventTypeMapping
Table("event.EventType")
Id(Function(x) x.Id).Column("EventTypeID").GeneratedBy().Identity()
Map(Function(x) x.FullTitle).Column("EventTypeFullTitle")
On the open of my form, the following function is called, which sets the FetchMode for the Types property of Event.
Public Function CreateListViewQuery(currentNHibernateSession As ISession) As NHibernate.ICriteria Implements IListViewQueryable.CreateListViewQuery
Return currentNHibernateSession.CreateCriteria(Of [Event])().SetFetchMode("Types", FetchMode.Join)
End Function
This is used to populate the listview, which happens extremely quickly, with just one call to the database for all the data:
SELECT TOP (50)
this_.EventID as EventID
, t3_.EventTypeID as EventTyp1_15_0_
, t3_.EventTypeFullTitle as EventTyp2_15_0_
FROM event.Event this_
inner join event.EventEventType types5_ on this_.EventID=types5_.EventID
inner join event.EventType t3_ on types5_.EventTypeID=t3_.EventTypeID
However, when I add a Criterion.Expression, like this (where QuickSearch is my user input):
.CreateAlias("Types", "t", NHibernate.SqlCommand.JoinType.InnerJoin)
.Add(NHibernate.Criterion.Expression.Like("t.FullTitle", QuickSearch))
I get 1 call which looks like this:
SELECT TOP 50
this_.EventID as EventID12_6_
, types8_.EventID as EventID
, t3_.EventTypeID as EventTyp2_
, t3_.EventTypeFullTitle as EventTyp2_15_0_
FROM
event.Event this_
inner join event.EventEventType types8_ on this_.EventID=types8_.EventID
inner join event.EventType t3_ on types8_.EventTypeID=t3_.EventTypeID
WHERE t3_.EventTypeFullTitle like #p1
And 50 more calls which look like this (50 as I have selected TOP 50):
SELECT
types0_.*
FROM
event.EventEventType types0_
left outer join event.EventType eventtype1_ on types0_.EventTypeID=eventtype1_.EventTypeID
WHERE types0_.EventID=#p0
(where #p0 is each of the TOP 50 Events which are returned by the search)
I feel like only the first call should be necessary for this.
Is it the nature of a many-to-many relationship which means NHibernate needs these additional calls? Is there something in my mapping which I've missed?
Perhaps importantly, I have used exactly the same technique for String properties of Event, and one-to-many relationships from Event and only one call is required for the search. The problem seems to only exist with many-to-many relationships.
Apologies for the long question, and thank you for getting this far. I've read many questions on a similar topic but couldn't find any addressing the issues of repeated database calls for many-to-many relationships.
What you are describing is an nHibernate n+1 issue (basically the number of queries executed is in direct proportion to the result set size), which can be difficult to resolve depending on how complex your query is.
Although not an obvious solution what has worked for me in the past is to change the join type to left outer join as below:
.CreateAlias("Types", "t", NHibernate.SqlCommand.JoinType.LeftOuterJoin)
Related
Imagine we have the following models:
type Company struct {
gorm.Model
Name string
Addresses []Address
}
type Address struct {
gorm.Model
CompanyID uint64
Street string
City string
Country string
}
I want to take all the companies(and their addresses) which have address in a specific location. Something like this:
SELECT company.*, address.* FROM company
INNER JOIN address ON address.company_id = company.id AND address.country = 'Bulgaria'
So if a company does not have address at the specific location, I will not get it as a result at all. I was trying something like that:
db.Joins("Addresses", "addresses.country = ?", "Bulgaria").Find(&companies)
However, it doesn't work, because GORM doesn't take the second argument of Joins(when preloading join used), so I should check the generated query and make something like that:
db.Where(`"Address".country = ?`, "Bulgaria").Joins("Addresses").Find(&companies)
Is there a better way/not hacky way? Have in mind all of the above code is mock of the real problem, I didn't want to expose the original models/queries.
You can use Preload to load Addresses into the Company object.
Based on your described conditions, where you don't want to load companies that don't match your filter, you should use an INNER JOIN
Two options:
First, if your table is named company, then your query should look like this:
db.Table("company").
Preload("Addresses").
Joins("INNER JOIN addresses a ON a.company_id = company.id").
Where("a.country = ?", "Bulgaria").
Find(&companies)
Second, if your table is named companies, then you should try this:
db.Preload("Addresses").
Joins("INNER JOIN addresses a ON a.company_id = companies.id").
Where("a.country = ?", "Bulgaria").
Find(&companies)
If you are using Gorm v2 you perform CRUD operations on has-one and one-to-many via associations:
var companies []Company
var addresses []Address
countries := []string{"Bulgaria"}
db.Model(&Address).Where("country IN (?)", countries).Find(&addresses)
// The key is using the previously fetched variable as the model for the association query
db.Model(&addresses).Association("CompanyID").Find(&companies)
This also works for many-to-many relations.
When comparing Gorm VS raw SQL queries (db.Raw(SQLquery)) you will typically see a performance hit. On the upside, you will not have to deal with the added complexity of raw sql string building in Go, which is pretty nice. I personally use a combination of both raw and gorm-built queries in my projects :)
The title is self-explanatory. Using 2.0.0-M3, I'd like to avoid unnecessary verbosity is the form of explicitly naming the columns to be joined on, since they are appropriately named, and since NATURAL JOIN is part of the SQL standard. Not to mention, Wikipedia itself even says that "The natural join is arguably one of the most important operators since it is the relational counterpart of logical AND."
I think the foregoing ought to be clear enough, but just if not, read on. Suppose I want to know the supplier-name and part-number of each part. Assuming appropriate case classes not shown:
class Suppliers(tag: Tag) extends Table[Supplier](tag, "suppliers") {
def snum = column[String]("snum")
def sname = column[String]("sname")
def * = (snum, sname) <> (Supplier.tupled, Supplier.unapply _)
}
class Shipments(tag: Tag) extends Table[Shipment](tag, "shipments") {
def snum = column[String]("snum")
def pnum = column[String]("pnum")
def * = (snum, pnum) <> (Shipment.tupled, Shipment.unapply _)
}
val suppliers = TableQuery[Suppliers]
val shipments = TableQuery[Shipments]
Given that both tables have the snum column I want to join on, seems as if
( suppliers join shipments ).run
ought to return a Vector with my desired data, but I get a failed attempt at an INNER JOIN, failing (at run-time) since it's missing any join condition.
I know I can do
suppliers.flatMap( s => shipments filter (sp => sp.snum === s.snum) map (sp => (s.sname, sp.pnum)) )
but, even without the names of all the columns I omitted for clarity of this question, it's still quite a lot more typing (and proofreading) than simply
suppliers join shipments
or, for that matter
SELECT * FROM suppliers NATURAL JOIN shipments;
If the Scala code is messier than the SQL code, then I really start questioning things. Is there no way simply to do a natural join in Slick?
Currently not supported by Slick. Please submit a ticket or pull request.
To improve readability of query code, you can put your join conditions into re-usable values. Or you can put the whole join in a function or method extension of Query[Suppliers,Supplier].
Alternatively you could look at the AutoJoin pattern (which basically makes your join conditions implicit) described here http://slick.typesafe.com/docs/#20130612_slick_vs_orm_scaladays_2013 and implemented here https://github.com/cvogt/play-slick/blob/scaladays2013/samples/computer-database/app/util/autojoin.scala
First off, I'm a Ruby/Rails newbie, so I apologize if this question is basic.
I've got a DB that (among other things) looks like this:
organizations { id, name, current_survey_id }
surveys { id, organization_id }
responses { id, survey_id, question_response_integer }
I'm trying to create a scope method that adds the average of the current survey answers to a passed-in Organization relation. In other words, the scope that's getting passed into the method would generate SQL that looks like more-or-less like this:
select * from organizations
And I'd like the scope, after it gets processed by my lambda, to generate SQL that looks like this:
select o.id, o.name, cs.average_responses
from organizations o join
(select r.id, avg(r.question_response_integer) as average_responses
from responses r
group by r.id) cs on cs.id = o.current_survey_id
The best I've got is something like this:
current_survey_average: lambda do |scope, sort_direction|
average_answers = Responses.
select("survey_id, avg(question_response_integer) as average_responses").
group("survey_id")
scope.joins(average_answers).order("average_responses #{sort_direction}")
end
That's mostly just a stab in the dark - among other things, it doesn't specify how the scope could be expected to join to average_answers - but I haven't been able to find any documentation about how to do that sort of join, and I'm running out of things to try.
Any suggestions?
EDIT: Thanks to Sean Hill for the answer. Just to have it on record, here's the code I ended up going with:
current_survey_average: lambda do |scope, sort_direction|
scope_table = scope.arel.froms.first.name
query = <<-QUERY
inner join (
select r.survey_id, avg(r.question_response_integer) as average_responses
from responses r
group by r.survey_id
) cs
on cs.survey_id = #{scope_table}.current_survey_id
QUERY
scope.
joins(query).
order("cs.average_responses #{sort_direction}")
end
That said, I can see the benefit of putting the averaged_answers scope directly onto the Responses class - so I may end up doing that.
I have not been able to test this, but I think the following would work, either as-is or with a little tweaking.
class Response < ActiveRecord::Base
scope :averaged, -> { select('r.id, avg(r.question_response_integer) as average_responses').group('r.id') }
scope :current_survey_average, ->(incoming_scope, sort_direction) do
scope_table = incoming_scope.arel.froms.first.name
query = <<-QUERY
INNER JOIN ( #{Arel.sql(averaged.to_sql)} ) cs
ON cs.id = #{scope_table}.current_survey_id
QUERY
incoming_scope.joins(query).order("average_responses #{sort_direction}")
end
end
So what I've done here is that I have split out the inner query into another scope called averaged. Since you do not know which table the incoming scope in current_survey_average is coming from, I got the scope table name via scope.arel.froms.first.name. Then I created a query string that uses the averaged scope and joined it using the scope_table variable. The rest is pretty self-explanatory.
If you do know that the incoming scope will always be from the organizations table, then you don't need the extra scope_table variable. You can just hardcode it into the join query string.
I would make one suggestion. If you do not have control over sort_direction, then I would not directly input that into the order string.
I have so category and this categories have unlimited sub category.
In Database Table, Fields are ID, UpperID and Title.
If I call a category and its subcategory in DataTable with recursive method in program(ASP.NET project)
performance is very bad.
And many user will use this application so everything goes bad.
Maybe All categories Fill to A Cache object and then we musnt go to Database.
But category count is 15000 or 20000.
So I think isn't a good method.
What can I do for fast performance?
Are you give me any suggestion?
caching or other in-memory-persistance is by far better than doing this on a relational system :) ... hey... it's oop!
just my 2 cents!
eg.
var categories = /* method for domain-objects*/.ToDictionary(category => category.ID);
foreach (var category in categories.Values)
{
if (!category.ParentCategoryID.HasValue)
{
continue;
}
Category parentCategory;
if (categories.TryGetValue(category.ParentCategoryID.Value, out parentCategory))
{
parentCategory.AddSubCategory(category);
}
}
et voila ... your tree is ready to go!
edit:
do you exactly know where your performance bottle-neck is?...
to give you some ideas, eg:
loading from database
building up the structure
querying the structure
loading from database:
then you should load it once and be sure to have some changetracking/notifying to get changes (if made) or optimize your query!
building up the structure:
the way i create the tree (traversal part) is the wastest you can do with a Dictionary<TKey, TValue>
querying the structure:
the structure i've used in my example is faster than List<T>. Dictionary<TKey, TValue> uses an index over the keys - so you may use int for the keys (IDs)
edit:
So you use DataTable to fix the
problem. Now you've got 2 problems: me
and DataTable
what do you have right now? where are you starting from? can you determine where your mudhole is? give us code!
Thanks All,
I find my solution with Common Table Expressions(CTE) fifty- fifty.
Its allow fast recursive queries.
WITH CatCTE(OID, Name, ParentID)
AS
(
SELECT OID, Name, ParentID FROM Work.dbo.eaCategory
WHERE OID = 0
UNION ALL
SELECT C.OID, C.Name, C.ParentID FROM Work.dbo.eaCategory C JOIN CatCTE as CTE ON C.ParentID= CTE.OID
)
SELECT * FROM CatCTE
I have a simple test object model in which there are schools, and a school has a collection of students.
I would like to retrieve a school and all its students who are above a certain age.
I carry out the following query, which obtains a given school and the children which are above a certain age:
public School GetSchoolAndStudentsWithDOBAbove(int schoolid, DateTime dob)
{
var school = this.Session.CreateCriteria(typeof(School))
.CreateAlias("Students", "students")
.Add(Expression.And(Expression.Eq("SchoolId", schoolid), Expression.Gt("students.DOB", dob)))
.UniqueResult<School>();
return school;
}
This all works fine and I can see the query going to the database and returning the expected number of rows.
However, when I carry out either of the following, it gives me the total number of students in the given school (regardless of the preceding request) by running another query:
foreach (Student st in s.Students)
{
Console.WriteLine(st.FirstName);
}
Assert.AreEqual(s.Students.Count, 3);
Can anyone explain why?
You made your query on the School class and you restricted your results on it, not on the mapped related objects.
Now there are many ways to do this.
You can make a static filter as IanL said, however its not really flexible.
You can just iterate the collection like mxmissile but that is ugly and slow (especially considering lazy loading considerations)
I would provide 2 different solutions:
In the first you maintain the query you have and you fire a dynamic filter on the collection (maintaining a lazy-loaded collection) and doing a round-trip to the database:
var school = GetSchoolAndStudentsWithDOBAbove(5, dob);
IQuery qDob = nhSession.CreateFilter(school.Students, "where DOB > :dob").SetDateTime("dob", dob);
IList<Student> dobedSchoolStudents = qDob.List<Student>();
In the second solution just fetch both the school and the students in one shot:
object result = nhSession.CreateQuery(
"select ss, st from School ss, Student st
where ss.Id = st.School.Id and ss.Id = :schId and st.DOB > :dob")
.SetInt32("schId", 5).SetDateTime("dob", dob).List();
ss is a School object and st is a Student collection.
And this can definitely be done using the criteria query you use now (using Projections)
Unfortunately s.Students will not contain your "queried" results. You will have to create a separate query for Students to reach your goal.
foreach(var st in s.Students.Where(x => x.DOB > dob))
Console.WriteLine(st.FirstName);
Warning: That will still make second trip to the db depending on your mapping, and it will still retrieve all students.
I'm not sure but you could possibly use Projections to do all this in one query, but I am by no means an expert on that.
You do have the option of filtering data. If it there is a single instance of the query mxmissle option would be the better choice.
Nhibernate Filter Documentation
Filters do have there uses, but depending on the version you are using there can be issues where filtered collections are not cached correctly.