Trying to construct a query such that I have multiple statement specifying joins, each with a where message chained onto them. When the query is run, I get all the joins, but only the where from my first call. Here's the method body that's doing the query:
observations_joins = Observation.joins(:obs_session => :project).where(:obs_sessions=>{:project_id=>self.project.id})
descriptor_hash = descriptor_where_hash if tag_descriptors && tag_descriptors.size > 0
puts "The descriptor_hash: #{descriptor_hash}"
observations = observations_joins.joins(:obs_descriptors).where("#{descriptor_hash['query_string']}", descriptor_hash['match_values']) if tag_descriptors && tag_descriptors.size > 0
arel = observations.arel
puts "The arel sql should be: #{arel.to_sql}"
observations
I have another method that gets called from inside the second joins statement, that iterates over the potential match values and generates the string and the values used; body here:
match_values = []
query_string = "obs_descriptors.tag_name = ?"
tag_descriptors.each_index do |index|
query_string = query_string + " #{tag_descriptors.fetch(index).qualifier_key} obs_descriptors.tag_name = ?" if index != 0
match_values << tag_descriptors.fetch(index).tag_name
end
{:match_values=>match_values, :query_string=>query_string}
So the sql getting generated looks like:
SELECT `observations`.* FROM `observations` INNER JOIN `obs_sessions` ON `obs_sessions`.`id` = `observations`.`obs_session_id` INNER JOIN `projects` ON `projects`.`id` = `obs_sessions`.`project_id` INNER JOIN `obs_descriptors` ON `obs_descriptors`.`observation_id` = `observations`.`id` WHERE (`obs_sessions`.`project_id` = 1)
and doesn't include the second set of where conditions. I also print the hash, just to make sure I'm not losing my mind and there are values in there, and there indeed are.
So, what am I missing to make this go as I'd expect it to?
Answering my own question here. The most elegant, concise way I found to get this working was to drop down to arel directly. Also, there were some issues with the original code posted, but even still, I needed to use arel to get properly grouped conditions. For context, I've got an object that, based on it's related data, needs to dynamically construct a semi advanced query, so I wanted to do things like checking for the existence of certain related data, and if present, then tack on the additional joins and wheres. Here's the final versions of the relevant methods:
def find_observations
observations = Observation.select('distinct observations.*').includes(:obs_session).includes(:judgements).includes(:concepts).includes(:obs_descriptors)
observations = observations.joins(:obs_session => :project).where(:obs_sessions=>{:project_id=>self.project.id})
if tag_descriptors && tag_descriptors.size > 0
observations = observations.where(descriptor_predicate)
end
if session_descriptors && session_descriptors.size > 0
observations = observations.where(session_predicate)
end
if user_descriptors && user_descriptors.size > 0
observations = observations.where(user_predicate)
end
#puts "observations sql is: #{observations.to_sql}"
observations.all
end
The above method optionally calls the remaining methods, which return the arel used in the where calls when chaining the AR object while building up the eventual query. Notice the disctinct; I'd had a version of this using arel entirely, that appeared to be working, but was in fact returning duplicates. I found references to using group(some_attribute) to fake things, but that turned out to cause problems down the chain, so to speak. So I fell back to using ActiveRelation to specify the distinct, joins and includes, and arel for the rest.
The next one was the part that was originally giving me lots of trouble; there are a variable number of possibilities, and each one could be either an AND or OR condition, and needed to be grouped separately so as not to mess up the rest of the generated where clause.
def descriptor_predicate
od = Arel::Table.new :obs_descriptors
predicate = nil
self.tag_descriptors.each_index do |index|
descriptor = self.tag_descriptors.fetch(index)
qual_key = descriptor.qualifier_key
tag_name = descriptor.tag_name
if index == 0
predicate = od[:descriptor].eq(tag_name)
else
if qual_key == "OR"
predicate = predicate.or(od[:descriptor].eq(tag_name))
else
predicate = predicate.and(od[:descriptor].eq(tag_name))
end
end
end
predicate
end
And finally the other predicate methods for the potential joined entity values:
def session_predicate
o = Arel::Table.new :observations
predicate = nil
self.session_descriptors.each_index do |index|
obs = self.session_descriptors.fetch(index)
if index == 0
predicate = o[:obs_session_id].eq(obs.entity_id)
else
predicate = predicate.or(o[:obs_session_id].eq(obs.entity_id))
end
end
predicate
end
def user_predicate
o = Arel::Table.new :observations
predicate = nil
self.user_descriptors.each_index do |index|
obs = self.user_descriptors.fetch(index)
if index == 0
predicate = o[:contributor_id].eq(obs.entity_id)
else
predicate = predicate.or(o[:contributor_id].eq(obs.entity_id))
end
end
predicate
end
def descriptor_where_string(included_where_statements)
tag_descriptors.each_index do |index|
qual_key = tag_descriptors.fetch(index).qualifier_key
tag_name = tag_descriptors.fetch(index).tag_name
if index == 0
query_string = "obs_descriptors.descriptor = #{tag_name}"
else
if qual_key == "OR"
query_string = query_string + " #{qual_key} obs_descriptors.descriptor = #{tag_name} AND #{included_where_statements} "
else
query_string = query_string + " #{qual_key} obs_descriptors.descriptor = ?"
end
end
end
query_string
end
Ultimately, I found the best solution involved leveraging both ActiveRelation chaining for providing the distinct and includes, and using arel directly for the conditions on the related values. Hope this helps somebody at some point.
Related
I'm coding up a search query where the user can search for Items by creator, title, or description, or any combination of the above.
So in my search controller logic I grab the params thusly:
creator = params['creator']
title = params['title']
description = params['description']
# todo: do some input validation here
results = nil
cr = User.roughly_named(creator).first
What I am doing now is:
q = []
q << "creator_id IS #{cr.id}" if cr
q << "title LIKE '%#{title}%'" if title != ''
q << "description LIKE '%#{description}%'" if title != ''
results = Item.where(q.join(' AND ')
but surely there is a better way. I am open to suggestions.
How about using scopes :
res = Item.scoped
res = res.where(["creator_id is ?", cr.id]) if cr
res = res.where(["title like ?", "%#{title}%"]) unless title.empty?
res = res.where(["description like ?", "%#{description}%"]) unless description.empty?
When doing Item.scoped, you basically do lazy loading. Iterating over res will actually execute the query. This is handy when chaining optional where clauses.
PS: prefer the ? syntax to prevent SQL injections.
You could simply achieve this with regular where clause
Or try dynamic finders? If that does not help either, you could use method_missing to create dynamic method call. It's explained elsewhere
In the below code retrive data from database arrange those & delete from database again insert
page_nos = [10,11,12,13,14,20,21,22,23,24,30,31,32,33,34,40,41,42,43,44]
page_nos.each do |page|
#page_data = HomePageSetting.where("page_no = '#{page}'").order("score desc")
p(#page_data)
HomePageSetting.where("page_no = #{page}").delete_all
#page_data.each do |data|
#home_page = HomePageSetting.new
#home_page.subject_id = data.subject_id
#home_page.subject_type = data.subject_type
#home_page.score = data.score
#home_page.flag = data.flag
#home_page.flag_detail = data.flag_detail
#home_page.page_no = data.page_no
#home_page.release_date = data.release_date
#home_page.item_created_at = data.item_created_at
#home_page.save
end
end
that is not working if i remove p(#page_data) line
I don't quite understand what you're trying to do, but I suspect I do know the reason that outputting the #page_data variable makes a difference.
#page_data is an ActiveRecord::Relation object - that is, a stored SQL query. Because executing a query and loading the resulting objects into memory takes time, Rails doesn't actually execute the query until it has to. When you output #page_data, its has to instantiate those objects to print the output. They then exist in memory, and can be referenced even after you delete their rows from the database in the next line.
If you don't print out that data, it stays uninstantiated on the next line, where you delete all those rows. Then, you call .each on it, and it gets instantiated at that point, and there are no objects to fetch, because you just deleted them all.
You can force Rails to instantiate the relation when you first define #page_data by calling .all, on it, thus:
#page_data = HomePageSetting.where("page_no = '#{page}'").order("score desc").all
It looks as if one problem at least is that you're calling .save inside the .new block -- you should save the record after.
Here's the changed code:
page_nos = [10,11,12,13,14,20,21,22,23,24,30,31,32,33,34,40,41,42,43,44]
page_nos.each do |page|
#page_data = HomePageSetting.where("page_no = '#{page}'").order("score desc")
p(#page_data)
HomePageSetting.where("page_no = #{page}").delete_all
#page_data.each do |data|
#home_page = HomePageSetting.new
#home_page.subject_id = data.subject_id
#home_page.subject_type = data.subject_type
#home_page.score = data.score
#home_page.flag = data.flag
#home_page.flag_detail = data.flag_detail
#home_page.page_no = data.page_no
#home_page.release_date = data.release_date
#home_page.item_created_at = data.item_created_at
end
#home_page.save
end
When I'm having problems like this I also sometimes will use .save! -- it will halt processing and throw an exception if the data doesn't save correctly. .save can fail silently and simply return false. To try that use #home_page.save! instead.
Good luck!
I need to build an array of objects of class ID using arrayfun:
% ID.m
classdef ID < handle
properties
id
end
methods
function obj = ID(id)
obj.id = id;
end
end
end
But get an error:
>> ids = 1:5;
>> s = arrayfun(#(id) ID(id), ids)
??? Error using ==> arrayfun
ID output type is not currently implemented.
I can build it alternatively in a loop:
s = [];
for k = 1 : length(ids)
s = cat(1, s, ID(ids(k)));
end
but what is wrong with this usage of arrayfun?
Edit (clarification of the question): The question is not how to workaround the problem (there are several solutions), but why the simple syntax s = arrayfun(#(id) ID(id), ids); doesn't work. Thanks.
Perhaps the easiest is to use cellfun, or force arrayfun to return a cell array by setting the 'UniformOutput' option. Then you can convert this cell array to an array of obects (same as using cat above).
s = arrayfun(#(x) ID(x), ids, 'UniformOutput', false);
s = [s{:}];
You are asking arrayfun to do something it isn't built to do.
The output from arrayfun must be:
scalar values (numeric, logical, character, or structure) or cell
arrays.
Objects don't count as any of the scalar types, which is why the "workarounds" all involve using a cell array as the output. One thing to try is using cell2mat to convert the output to your desired form; it can be done in one line. (I haven't tested it though.)
s = cell2mat(arrayfun(#(id) ID(id), ids,'UniformOutput',false));
This is how I would create an array of objects:
s = ID.empty(0,5);
for i=5:-1:1
s(i) = ID(i);
end
It is always a good idea to provide a "default constructor" with no arguments, or at least use default values:
classdef ID < handle
properties
id
end
methods
function obj = ID(id)
if nargin<1, id = 0; end
obj.id = id;
end
end
end
Here's another one:
ValidFirings = ((DwellTimes > 30/(24*60*60)) | (GroupCount > 1));
for i = length(ValidFirings):-1:2
if(~ValidFirings(i))
DwellTimes(i-1) = DwellTimes(i)+DwellTimes(i-1);
GroupCount(i-1) = GroupCount(i)+GroupCount(i-1);
DwellTimes(i) = [];
GroupCount(i) = [];
ReducedWallTime(i) = [];
ReducedWallId(i) = [];
end
end
It appears that the intent is to sum up 'dwelltimes' based on whether or not the sensor firing is considered valid. So I have a vector of sensor firings that Im walking through backwards and summing into the previous row if the current row is not marked as valid.
I can visualize this in C/C++ but I don't know how to translate it into better Matlab vector notation. As it stands now, this loop is v slow.
EDIT:
Could I use some form of DwellTimes = DwellTimes( cumsum( ValidFirings ))?
As with your previous question, replacing the for loop should improve the performance.
%# Find the indices for invalid firings
idx = find(~(DwellTimes > 30/(24*60*60)) | (GroupCount > 1));
%# Index the appropriate elements and add them (start the addition
%# from the second element)
%# This eliminates the for loop
DwellTimes(idx(2:end)-1) = DwellTimes(idx(2:end)-1)+DwellTimes(idx(2:end));
GroupCount(idx(2:end)-1) = GroupCount(idx(2:end)-1)+GroupCount(idx(2:end));
%# Now remove all the unwanted elements (this removes the
%# first element if it was a bad firing. Modify as necessary)
GroupCount(idx)=[];
DwellTimes(idx)=[];
I would consolidate first as shown, then eliminate the invalid data. This avoids the constant resizing of the data. Note that you can't reverse the order of the FOR loop due to the way that the values propagate.
ValidFirings = ((DwellTimes > 30/(24*60*60)) | (GroupCount > 1));
for i = length(ValidFirings):-1:2
if (~ValidFirings(i))
DwellTimes(i-1) = DwellTimes(i) + DwellTimes(i-1);
GroupCount(i-1) = GroupCount(i) + GroupCount(i-1);
end
end
DwellTimes = DwellTimes(ValidFirings);
GroupCount = GroupCount(ValidFirings);
ReducedWallTime = ReducedWallTime(ValidFirings);
ReducedWallId = ReducedWallId(ValidFirings);
So I have a fairly involved sql query here.
SELECT links_link.id, links_link.created, links_link.url, links_link.title, links_category.title, SUM(links_vote.karma_delta) AS karma, SUM(CASE WHEN links_vote.user_id = 1 THEN links_vote.karma_delta ELSE 0 END) AS user_vote
FROM links_link
LEFT OUTER JOIN auth_user ON (links_link.user_id = auth_user.id)
LEFT OUTER JOIN links_category ON (links_link.category_id = links_category.id)
LEFT OUTER JOIN links_vote ON (links_vote.link_id = links_link.id)
WHERE (links_link.id = links_vote.link_id)
GROUP BY links_link.id, links_link.created, links_link.url, links_link.title, links_category.title
ORDER BY links_link.created DESC
LIMIT 20
All my relations are good (I think) and this query works perfectly when I run it in my navicat for postgresql but turning it into something Django can use has been quite the challenge. I am using the pre-alpha 1.2 development verison (from the subversion repositories) so I have full range of tools from the docs.
Here are my models for grins:
class Category (models.Model):
created = models.DateTimeField(auto_now_add = True)
modified = models.DateTimeField(auto_now = True)
title = models.CharField(max_length = 128)
def __unicode__(self):
return self.title
class Link (models.Model):
category = models.ForeignKey(Category)
user = models.ForeignKey(User)
created = models.DateTimeField(auto_now_add = True)
modified = models.DateTimeField(auto_now = True)
fame = models.PositiveIntegerField(default = 1)
url = models.URLField(max_length = 2048)
title = models.CharField(max_length = 256)
active = models.BooleanField(default = True)
def __unicode__(self):
return self.title
class Vote (models.Model):
link = models.ForeignKey(Link)
user = models.ForeignKey(User)
created = models.DateTimeField(auto_now_add = True)
modified = models.DateTimeField(auto_now = True)
karma_delta = models.SmallIntegerField(default = 1)
def __unicode__(self):
return str(self.karma_delta)
How I am able to turn
def latest(request):
links = Link.objects.all().order_by('-created')[:20]
return render_to_response('links/list.html', {'links': links})
Into the above query?
I've only been able to make some progress using things like Aggregation but how to tackle my use of CASE is beyond me. Any help would be much appreciated. I always prefer to work in a framework's built in ORM but if raw SQL is necessary...
I don't have time at the moment to attempt a full translation of that query, but if the CASE is your main stumbling block, I can tell you right now it isn't supported natively, you'll need to use a call to .extra() with some raw SQL for that. Something like:
.extra(select={'user_vote': 'SUM(CASE WHEN links_vote.user_id = 1 THEN links_vote.karma_delta ELSE 0 END')})
But if this query works well as-is, why bother translating it into the ORM? Just grab a cursor and run it as a SQL query. Django's ORM is intentionally not a 100% solution, there's a reason it exposes the raw cursor API.
Update: And since Django 1.2, there's also Manager.raw() to let you make raw SQL queries and get model objects back (thanks Van Gale).