Performing PostgreSQL LEFT OUTER JOINS and CASEs in Django - sql

So I have a fairly involved sql query here.
SELECT links_link.id, links_link.created, links_link.url, links_link.title, links_category.title, SUM(links_vote.karma_delta) AS karma, SUM(CASE WHEN links_vote.user_id = 1 THEN links_vote.karma_delta ELSE 0 END) AS user_vote
FROM links_link
LEFT OUTER JOIN auth_user ON (links_link.user_id = auth_user.id)
LEFT OUTER JOIN links_category ON (links_link.category_id = links_category.id)
LEFT OUTER JOIN links_vote ON (links_vote.link_id = links_link.id)
WHERE (links_link.id = links_vote.link_id)
GROUP BY links_link.id, links_link.created, links_link.url, links_link.title, links_category.title
ORDER BY links_link.created DESC
LIMIT 20
All my relations are good (I think) and this query works perfectly when I run it in my navicat for postgresql but turning it into something Django can use has been quite the challenge. I am using the pre-alpha 1.2 development verison (from the subversion repositories) so I have full range of tools from the docs.
Here are my models for grins:
class Category (models.Model):
created = models.DateTimeField(auto_now_add = True)
modified = models.DateTimeField(auto_now = True)
title = models.CharField(max_length = 128)
def __unicode__(self):
return self.title
class Link (models.Model):
category = models.ForeignKey(Category)
user = models.ForeignKey(User)
created = models.DateTimeField(auto_now_add = True)
modified = models.DateTimeField(auto_now = True)
fame = models.PositiveIntegerField(default = 1)
url = models.URLField(max_length = 2048)
title = models.CharField(max_length = 256)
active = models.BooleanField(default = True)
def __unicode__(self):
return self.title
class Vote (models.Model):
link = models.ForeignKey(Link)
user = models.ForeignKey(User)
created = models.DateTimeField(auto_now_add = True)
modified = models.DateTimeField(auto_now = True)
karma_delta = models.SmallIntegerField(default = 1)
def __unicode__(self):
return str(self.karma_delta)
How I am able to turn
def latest(request):
links = Link.objects.all().order_by('-created')[:20]
return render_to_response('links/list.html', {'links': links})
Into the above query?
I've only been able to make some progress using things like Aggregation but how to tackle my use of CASE is beyond me. Any help would be much appreciated. I always prefer to work in a framework's built in ORM but if raw SQL is necessary...

I don't have time at the moment to attempt a full translation of that query, but if the CASE is your main stumbling block, I can tell you right now it isn't supported natively, you'll need to use a call to .extra() with some raw SQL for that. Something like:
.extra(select={'user_vote': 'SUM(CASE WHEN links_vote.user_id = 1 THEN links_vote.karma_delta ELSE 0 END')})
But if this query works well as-is, why bother translating it into the ORM? Just grab a cursor and run it as a SQL query. Django's ORM is intentionally not a 100% solution, there's a reason it exposes the raw cursor API.
Update: And since Django 1.2, there's also Manager.raw() to let you make raw SQL queries and get model objects back (thanks Van Gale).

Related

API return breaks when using inner join?

I've been building an API which was working absolutely fine until I tried to add an inner join
The SQL I'm passing returns as I'd expect when I run it in Beaver (am using Mac)
However, when I try and access it via my API end point, instead of the combined results, I get only results from the table I added in the join
I presume am doing something really stupid ...
From my controller:
result = dbtest.FromDatabase(
"SELECT A.FAMILY_ID
,A.START_TIME
,A.END_TIME
,A.WORKER_ID
,A.WEEK_NO
,A.ID
,A.SHIFT_NO
,A.DAY_OF_WEEK
,A.HOLIDAY_OR_TERM
,B.WORKER_NAME
FROM SHIFT_REQ_TBL A
INNER JOIN WORKER_TBL B ON A.WORKER_ID = B.WORKER_ID
WHERE A.FAMILY_ID = '" + FAMILY.FAMILY_ID + "'");
From my model
if (query.Contains("SHIFT_REQ_TBL"))
{
var tbl_type = new TimekeeperTables.SHIFT_REQ_TBL();
tbl_type.FAMILY_ID = Convert.ToInt32(reader["FAMILY_ID"]);
tbl_type.ID = Convert.ToInt32(reader["ID"]);
tbl_type.WEEK_NO = Convert.ToInt32(reader["WEEK_NO"]);
tbl_type.WORKER_ID = Convert.ToInt32(reader["WORKER_ID"]);
tbl_type.SHIFT_NO = reader["SHIFT_NO"].ToString();
tbl_type.START_TIME = reader["START_TIME"].ToString();
tbl_type.END_TIME = reader["END_TIME"].ToString();
tbl_type.DAY_OF_WEEK = reader["DAY_OF_WEEK"].ToString();
tbl_type.HOLIDAY_OR_TERM = reader["HOLIDAY_OR_TERM"].ToString();
tbl_type.WORKER_NAME = reader["WORKER_NAME"].ToString();
db_results.Add(tbl_type);
jsonDoc = JsonConvert.SerializeObject(db_results);
}
Results (from postman)
"[{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"},{\"WORKER_ID\":1,\"WORKER_NAME\":\"UNASSIGNED\"}]"
I'm sorry, I was being stupid and figured it out
I was using a really dumb way of identifying the calling URL which already was checking for SHIFT_TBL in an earlier case

The entity or complex type 'AdministratorModel' cannot be constructed in a LINQ to Entities query

I am attempting to join two tables in a linq statement, and construct an 'Administrator Model' with a "Administrator Contract Model" attached to the parent model.
Dim response = (From a In db.Administrators
Join ac In db.AdministratorContracts
On a.NDE_Staff_ID Equals ac.NDE_Staff_ID
Order By a.LastName, a.FirstName
Select New AdministratorModel With {
.NDE_Staff_ID = a.NDE_Staff_ID,
.FirstName = a.FirstName,
.LastName = a.LastName,
.Contract = New AdministratorContractModel With {
.AdministratorContractsID = ac.AdministratorContractsID,
.NDE_Staff_ID = ac.NDE_Staff_ID,
.NDE_Number = ac.NDE_Number,
.ForYear = ac.ForYear,
.Contract = ac.Contract,
.ContractDays = ac.ContractDays,
.Salary = ac.Salary,
.Salary_AddComp = ac.Salary_AddComp,
.Salary_Benefits = ac.Salary_Benefits,
.Experience_System = ac.Experience_System,
.Experience_Total = ac.Experience_Total,
.EducationAttained = ac.EducationAttained,
.BenefitsExplanation = ac.BenefitsExplanation,
.AddCompExplanation = ac.AddCompExplanation}})
The problem is that LinqToEntities is trying to convert your query to SQL, but doesn't know anything about your model types (assuming that these are not the same as the database types).
If you enumerate the query before trying to create the types, it should work. Try calling ToList() after the OrderBy() and before the Select(). That will create an in-memory dataset of the database types, which is then detached from the database, allowing the rest of your code to work.
Note that if you want to do any filtering, you should do this before calling ToList(), as this will reduce the size of the in-memory dataset.

How to search across fields for a range of optional values using ActiveRecord in Ruby?

I'm coding up a search query where the user can search for Items by creator, title, or description, or any combination of the above.
So in my search controller logic I grab the params thusly:
creator = params['creator']
title = params['title']
description = params['description']
# todo: do some input validation here
results = nil
cr = User.roughly_named(creator).first
What I am doing now is:
q = []
q << "creator_id IS #{cr.id}" if cr
q << "title LIKE '%#{title}%'" if title != ''
q << "description LIKE '%#{description}%'" if title != ''
results = Item.where(q.join(' AND ')
but surely there is a better way. I am open to suggestions.
How about using scopes :
res = Item.scoped
res = res.where(["creator_id is ?", cr.id]) if cr
res = res.where(["title like ?", "%#{title}%"]) unless title.empty?
res = res.where(["description like ?", "%#{description}%"]) unless description.empty?
When doing Item.scoped, you basically do lazy loading. Iterating over res will actually execute the query. This is handy when chaining optional where clauses.
PS: prefer the ? syntax to prevent SQL injections.
You could simply achieve this with regular where clause
Or try dynamic finders? If that does not help either, you could use method_missing to create dynamic method call. It's explained elsewhere

multiple string queries on a single table

making a site for game trailers and on the front page I organize the games in terms of their category, so I end up doing this (rails):
def index
#newGames = Game.order("created_at DESC").limit(3)
#casualGames = Game.where("category = 'casual'").limit(9)
#actionGames = Game.where("category = 'action'").limit(8)
#strategyGames = Game.where("category = 'strategy'").limit(9)
#adventureGames = Game.where("category = 'adventure'").limit(8)
#puzzleGames = Game.where("category = 'puzzle'").limit(9)
end
Is there a way to accomplish the same thing but without making 6 separate queries on the sable table?
Thanks
As your search parameters are different querying DB multiple times is unavoidable. However you can make your controller skinny. Create a class method in Game class and collect and return everything you need in a hash.
Game.rb
def self.for_index_page
games = {}
games.merge!(new: order("created_at DESC").limit(3))
games.merge!(casual: category_with_limit('casual', 9)
games.merge!(action: category_with_limit('action', 8)
...
end
def self.category_with_limit(category, limit)
where(category: category).limit(limit)
end
GamesController.rb
def index
#games = Game.for_index_page
end
index.erb
<%=#games[:new] %>
<%=#games[:casual] %>
...

Chaining multiple joins with wheres in rails3

Trying to construct a query such that I have multiple statement specifying joins, each with a where message chained onto them. When the query is run, I get all the joins, but only the where from my first call. Here's the method body that's doing the query:
observations_joins = Observation.joins(:obs_session => :project).where(:obs_sessions=>{:project_id=>self.project.id})
descriptor_hash = descriptor_where_hash if tag_descriptors && tag_descriptors.size > 0
puts "The descriptor_hash: #{descriptor_hash}"
observations = observations_joins.joins(:obs_descriptors).where("#{descriptor_hash['query_string']}", descriptor_hash['match_values']) if tag_descriptors && tag_descriptors.size > 0
arel = observations.arel
puts "The arel sql should be: #{arel.to_sql}"
observations
I have another method that gets called from inside the second joins statement, that iterates over the potential match values and generates the string and the values used; body here:
match_values = []
query_string = "obs_descriptors.tag_name = ?"
tag_descriptors.each_index do |index|
query_string = query_string + " #{tag_descriptors.fetch(index).qualifier_key} obs_descriptors.tag_name = ?" if index != 0
match_values << tag_descriptors.fetch(index).tag_name
end
{:match_values=>match_values, :query_string=>query_string}
So the sql getting generated looks like:
SELECT `observations`.* FROM `observations` INNER JOIN `obs_sessions` ON `obs_sessions`.`id` = `observations`.`obs_session_id` INNER JOIN `projects` ON `projects`.`id` = `obs_sessions`.`project_id` INNER JOIN `obs_descriptors` ON `obs_descriptors`.`observation_id` = `observations`.`id` WHERE (`obs_sessions`.`project_id` = 1)
and doesn't include the second set of where conditions. I also print the hash, just to make sure I'm not losing my mind and there are values in there, and there indeed are.
So, what am I missing to make this go as I'd expect it to?
Answering my own question here. The most elegant, concise way I found to get this working was to drop down to arel directly. Also, there were some issues with the original code posted, but even still, I needed to use arel to get properly grouped conditions. For context, I've got an object that, based on it's related data, needs to dynamically construct a semi advanced query, so I wanted to do things like checking for the existence of certain related data, and if present, then tack on the additional joins and wheres. Here's the final versions of the relevant methods:
def find_observations
observations = Observation.select('distinct observations.*').includes(:obs_session).includes(:judgements).includes(:concepts).includes(:obs_descriptors)
observations = observations.joins(:obs_session => :project).where(:obs_sessions=>{:project_id=>self.project.id})
if tag_descriptors && tag_descriptors.size > 0
observations = observations.where(descriptor_predicate)
end
if session_descriptors && session_descriptors.size > 0
observations = observations.where(session_predicate)
end
if user_descriptors && user_descriptors.size > 0
observations = observations.where(user_predicate)
end
#puts "observations sql is: #{observations.to_sql}"
observations.all
end
The above method optionally calls the remaining methods, which return the arel used in the where calls when chaining the AR object while building up the eventual query. Notice the disctinct; I'd had a version of this using arel entirely, that appeared to be working, but was in fact returning duplicates. I found references to using group(some_attribute) to fake things, but that turned out to cause problems down the chain, so to speak. So I fell back to using ActiveRelation to specify the distinct, joins and includes, and arel for the rest.
The next one was the part that was originally giving me lots of trouble; there are a variable number of possibilities, and each one could be either an AND or OR condition, and needed to be grouped separately so as not to mess up the rest of the generated where clause.
def descriptor_predicate
od = Arel::Table.new :obs_descriptors
predicate = nil
self.tag_descriptors.each_index do |index|
descriptor = self.tag_descriptors.fetch(index)
qual_key = descriptor.qualifier_key
tag_name = descriptor.tag_name
if index == 0
predicate = od[:descriptor].eq(tag_name)
else
if qual_key == "OR"
predicate = predicate.or(od[:descriptor].eq(tag_name))
else
predicate = predicate.and(od[:descriptor].eq(tag_name))
end
end
end
predicate
end
And finally the other predicate methods for the potential joined entity values:
def session_predicate
o = Arel::Table.new :observations
predicate = nil
self.session_descriptors.each_index do |index|
obs = self.session_descriptors.fetch(index)
if index == 0
predicate = o[:obs_session_id].eq(obs.entity_id)
else
predicate = predicate.or(o[:obs_session_id].eq(obs.entity_id))
end
end
predicate
end
def user_predicate
o = Arel::Table.new :observations
predicate = nil
self.user_descriptors.each_index do |index|
obs = self.user_descriptors.fetch(index)
if index == 0
predicate = o[:contributor_id].eq(obs.entity_id)
else
predicate = predicate.or(o[:contributor_id].eq(obs.entity_id))
end
end
predicate
end
def descriptor_where_string(included_where_statements)
tag_descriptors.each_index do |index|
qual_key = tag_descriptors.fetch(index).qualifier_key
tag_name = tag_descriptors.fetch(index).tag_name
if index == 0
query_string = "obs_descriptors.descriptor = #{tag_name}"
else
if qual_key == "OR"
query_string = query_string + " #{qual_key} obs_descriptors.descriptor = #{tag_name} AND #{included_where_statements} "
else
query_string = query_string + " #{qual_key} obs_descriptors.descriptor = ?"
end
end
end
query_string
end
Ultimately, I found the best solution involved leveraging both ActiveRelation chaining for providing the distinct and includes, and using arel directly for the conditions on the related values. Hope this helps somebody at some point.