Group By Year, Month & then Count in ActiveRecord 3 - sql

I'm trying to pull a count of all users created in a year, and month but the following doesn't seem to work as expected.
User.group("YEAR(created_AT), MONTH(created_at)").
count("DISTINCT(id), YEAR(created_at), MONTH(created_at)")
i'm looking for something like
{2011 => {1 => 222, 2 => 333, 4 => 444, 5 => 667 ... }}
but i'm getting
{1 => 222, 2 => 333, 4 => 444, 5 => 667 ... }
Am i missing something, or can ActiveRecord not give me this result in one query?

The count method doesn't work like you think it does. You end up doing this:
select count(distinct(id), year(created_at), month(created_at))
from users
group by year(created_at), month(created_at)
That SELECT clause is pretty dodgy but MySQL will, in its usual sloppy manner, let it through. I think you want this query:
select count(distinct(id)), year(created_at), month(created_at)
from users
group by year(created_at), month(created_at)
I'd probably go straight to select_all like this:
a = User.connection.select_all(%q{
select count(distinct(id)) as c, year(created_at) as y, month(created_at) as m
from users
group by y, m
})
Or you could do it like this:
a = User.connection.select_all(
User.select('count(distinct(id)) as c, year(created_at) as y, month(created_at) as m').
group('y, m')
)
Those will give you an array, a, of Hashes with c, y, and m keys like this:
a = [
{ 'c' => '23', 'y' => '2010', 'm' => '11' },
{ 'c' => '1', 'y' => '2011', 'm' => '1' },
{ 'c' => '5', 'y' => '2011', 'm' => '3' },
{ 'c' => '2', 'y' => '2011', 'm' => '4' },
{ 'c' => '11', 'y' => '2011', 'm' => '8' }
]
Then a bit of data wrangling is all you need to finish the job:
h = a.group_by { |x| x['y'] }.each_with_object({}) do |(y,v), h|
h[y.to_i] = Hash[v.map { |e| [e['m'].to_i, e['c'].to_i] }]
end
# {2010 => {11 => 23}, 2011 => {1 => 1, 3 => 5, 4 => 2, 8 => 11}}

Related

Athena query error : extraneous input 'select' expecting

Got an error while running query :
extraneous input 'select' expecting {'(', 'add', 'all', 'some', 'any', 'at', 'no', 'substring', 'position', 'tinyint', 'smallint', 'integer', 'date', 'time', 'timestamp', 'interval', 'year', 'month', 'day', 'hour', 'minute', 'second', 'zone', 'filter', 'over', 'partition', 'range', 'rows', 'preceding', 'following', 'current', 'row', 'schema', 'comment', 'view', 'replace', 'grant', 'revoke', 'privileges', 'public', 'option', 'explain', 'analyze', 'format', 'type', 'text', 'graphviz', 'logical', 'distributed', 'validate', 'show', 'tables', 'views', 'schemas', 'catalogs', 'columns', 'column', 'use', 'partitions', 'functions', 'to', 'system', 'bernoulli', 'poissonized', 'tablesample', 'unnest', 'array', 'map', 'set', 'reset', 'session', 'data', 'start', 'transaction', 'commit', 'rollback', 'work', 'isolation', 'level', 'serializable', 'repeatable', 'committed', 'uncommitted', 'read', 'write', 'only', 'call', 'input', 'output', 'cascade', 'restrict', 'including', 'excluding', 'properties', 'function', 'lambda_invoke', 'returns', 'sagemaker_invoke_endpoint', 'nfd', 'nfc', 'nfkd', 'nfkc', 'if', 'nullif', 'coalesce', identifier, digit_identifier, quoted_identifier, backquoted_identifier}
query is right join table ss with dd ,like:
select * from
(
select platform, id, nextMonth
FROM "logs"."user_records" as ss
right join
select id as idRight, platform, month
FROM "logs"."user_records" as dd
on ss.platform = dd.platform and ss.userid = dd.useridRight and ss.nextMonth=dd.month )
You probably need to surround the subquery after your right join in parentheses. Untested, but I'd guess:
select * from
(
select platform, id, nextMonth
FROM "logs"."user_records" as ss
right join
( select id as idRight, platform, month
FROM "logs"."user_records" ) as dd
on ss.platform = dd.platform and ss.userid = dd.useridRight and ss.nextMonth=dd.month )

Postgresql convert 1200 to 1.2K

Is there a way to convert 1200 to 1.2K with Postgres like the Rails helper (number_to_human) method does?
number_to_human(1200, format: '%n%u', units: { thousand: 'K', million: 'M', billion: 'B' })
# "1.2K"
number_to_human(1200000, format: '%n%u', units: { thousand: 'K', million: 'M', billion: 'B' })
# "1.2M"

How to subtract one row to other rows in a grouped by dataframe?

I've got this data frame with some 'init' values ('value', 'value2') that I want to subtract to the mid term value 'mid' and final value 'final' once I've grouped by ID.
import pandas as pd
df = pd.DataFrame({
'value': [100, 120, 130, 200, 190,210],
'value2': [2100, 2120, 2130, 2200, 2190,2210],
'ID': [1, 1, 1, 2, 2, 2],
'state': ['init','mid', 'final', 'init', 'mid', 'final'],
})
My attempt was tho extract the index where I found 'init', 'mid' and 'final' and subtract from 'mid' and 'final' the value of 'init' once I've grouped the value by 'ID':
group = df.groupby('ID')
group['diff_1_f'] = group['value'].iloc[group.index[group['state'] == 'final'] - group['value'].iloc[group.index[dfs['state'] == 'init']]]]
group['diff_2_f'] = group['value2'].iloc[group.index[group['state'] == 'final'] - group['value'].iloc[group.index[dfs['state'] == 'init']]]
group['diff_1_m'] = group['value'].iloc[group.index[group['state'] == 'mid'] - group['value'].iloc[group.index[dfs['state'] == 'init']]]
group['diff_2_m'] = group['value2'].iloc[group.index[group['state'] == 'mid'] - group['value'].iloc[group.index[dfs['state'] == 'init']]]
But of course it doesn't work. How can I obtain the following result:
df = pd.DataFrame({
'diff_value': [20, 30, -10,10],
'diff_value2': [20, 30, -10,10],
'ID': [ 1, 1, 2, 2],
'state': ['mid', 'final', 'mid', 'final'],
})
Also in it's grouped form.
Use:
#columns names in list for subtract
cols = ['value', 'value2']
#new columns names created by join
new = [c + '_diff' for c in cols]
#filter rows with init
m = df['state'].ne('init')
#add init rows to new columns by join and filter no init rows
df1 = df.join(df[~m].set_index('ID')[cols], lsuffix='_diff', on='ID')[m]
#subtract with numpy array by .values for prevent index alignment
df1[new] = df1[new].sub(df1[cols].values)
#remove helper columns
df1 = df1.drop(cols, axis=1)
print (df1)
value_diff value2_diff ID state
1 20 20 1 mid
2 30 30 1 final
4 -10 -10 2 mid
5 10 10 2 final

Inheritance of rows in SQL

I have a table of rows, here is a sample (as hashes):
{:id => 8, :n => 774, :inherits_from => [], :properties => ["a", "b", "c"]}
{:id => 9, :n => 915, :inherits_from => [8], :properties => ["d"]}
{:id => 10, :n => 754, :inherits_from => [1, 2], :properties => []}
{:id => 11, :n => 774, :inherits_from => [10], :properties => ["a", "b"]}
The idea is that properties can be inherited from other rows:
table[:id => 9] #=> {:id => 9, :n => 915, :inherits_from => [8], :properties => ["a", "b", "c", "d"]}
Rows can have one or several inheritances, or can have none. There can be many levels of inheritance, i.e., 5 inherits from 2 and 3, 7 inherits from 5 (which means that 7 inherits from 5, but also from 2 and 3, because 5 inherits from them) etc.
I know I could do it with brute force (i.e., for every queried row check if it inherits from somewhere). Also, I could use Class.new to generate a class for each row and set inheritances (I use ruby), but there will be thousands of rows.
Any advise on how could I do it in the most efficient way?

thinking sphinx search has_many not working no error

model
class Person < ActiveRecord::Base
attr_accessible :alignment, :description, :first_name, :last_name
has_many :roles #table roles with active as one of the field with value equal t or f (boolean)
end
class Role < ActiveRecord::Base
attr_accessible :active, :organization_id, :person_id, :position_id
belongs_to :person
belongs_to :organization
belongs_to :position
end
person_index.rb
ThinkingSphinx::Index.define :person, :with => :active_record do
#Fields
indexes last_name, :sortable => true
indexes first_name, :sortable => true
indexes alignment
#indexes role(:active), :as => :active
indexes role.active, :as => :active
#Attributes
has created_at, updated_at
has professions(:id), :as => :profession_ids
has positions(:id), :as => :position_id
has organizations(:id), :as => :organization_id
end
people_controller
filters = {}
filters[:profession_ids] = params[:profession_ids] if params[:profession_ids].present?
filters[:organization_id] = params[:organization_id] if params[:organization_id].present?
filters[:position_id] = params[:position_id] if params[:position_id].present?
filters[:active_ids] = role if params[:active].present? #added
#people = Person.search " #{params[:lastname]} #{params[:firstname]} #{params[:alignmemt]}",
:with => filters,
:star => true,
:condition => {:alignment => params[:alignment], :active => params[:active]},
:order => 'last_name ASC, first_name ASC',
:page => params[:page],
:per_page => 20
When i search active and/or alignment it is not filtering result and doesn't give me error. these are both string field, alignment is in the people table and active is in another table (roles)
Why? What am i missing?
update
Tried the recommended solution for active on this question and same result...
person_index.rb
ThinkingSphinx::Index.define :person, :with => :active_record do
#Fields
indexes last_name, :sortable => true
indexes first_name, :sortable => true
indexes alignment
#Attributes
has created_at, updated_at
has professions(:id), :as => :profession_ids
has positions(:id), :as => :position_id
has organizations(:id), :as => :organization_id
has roles.active, :as => :active_ids
end
people_controller
def index
#role = Role.find_by_active(params[:active]) #ADDED
filters = {}
filters[:profession_ids] = params[:profession_ids] if params[:profession_ids].present?
filters[:organization_id] = params[:organization_id] if params[:organization_id].present?
filters[:position_id] = params[:position_id] if params[:position_id].present?
#people = Person.search " #{params[:lastname]} #{params[:firstname]} #{params[:alignmemt]}",
:with => filters,
:star => true,
:condition => {:alignment => params[:alignment], :active_ids => #role}, #CHANGED
:order => 'last_name ASC, first_name ASC',
:page => params[:page],
:per_page => 20
but still have same result... WHY?
controller updated after Pat answer
def index
if params[:active].present?
role = Array.new
rolepid = Array.new
role = Role.find_all_by_active(params[:active])
role.each do |num|
puts num.person_id
rolepid << num.person_id #get all the person id whith the params[:active]
end
end
filters = {}
filters[:profession_ids] = params[:profession_ids] if params[:profession_ids].present?
filters[:organization_id] = params[:organization_id] if params[:organization_id].present?
filters[:position_id] = params[:position_id] if params[:position_id].present?
filters[:active_ids] = rolepid if params[:active].present?
#people = Person.search " #{params[:lastname]} #{params[:firstname]} #{params[:alignent]}",
#:classes => [Person, Role],
:with => filters,
:star => true,
:condition => {:alignment => params[:alignment]},
:order => 'last_name ASC, first_name ASC',
:page => params[:page],
:per_page => 20
But now it is looking for active in people table when it should look in roles table. So i added #:classes => [Person, Role], but no luck....
Role Load (0.7ms) SELECT "roles".* FROM "roles" WHERE "roles"."active" = 'f'
Sphinx Query (0.7ms) SELECT * FROM `person_core` WHERE `active_ids` IN (304, 34, 306, 308, 334, 295, 344, 348, 352, 354, 365, 367, 308, 429, 468, 9, 544, 590, 609, 110, 1643, 1652, 1653, 1655, 1669, 628, 1687, 1691, 1709) AND `sphinx_deleted` = 0 ORDER BY `last_name` ASC, first_name ASC LIMIT 0, 20
Sphinx Found 0 results
So i change in the controller
filters[:active_ids] = rolepid if params[:active].present?
to
filters[:id] = rolepid if params[:active].present?
Since rolepid is an array of integer with the person ids.
But Sphinx is just looking for 4 ids that are not in rolepid... I am confused :|
Parameters: {"utf8"=>"✓", "firstname"=>"", "lastname"=>"", "alignment"=>"", "organization_id"=>"", "position_id"=>"", "active"=>"f", "commit"=>"Search"}
Role Load (0.8ms) SELECT "roles".* FROM "roles" WHERE "roles"."active" = 'f'
Sphinx Query (0.6ms) SELECT * FROM `person_core` WHERE `id` IN (304, 34, 306, 308, 334, 295, 344, 348, 352, 354, 365, 367, 308, 429, 468, 9, 544, 590, 609, 110, 1643, 1652, 1653, 1655, 1669, 628, 1687, 1691, 1709) AND `sphinx_deleted` = 0 ORDER BY `last_name` ASC, first_name ASC LIMIT 0, 20
Sphinx Found 4 results
Person Load (0.4ms) SELECT "people".* FROM "people" WHERE "people"."id" IN (84, 1, 61, 50)
Why is it not returning the 29 records from rolepid array ?
filtering for alignment IS working. thanks for catching the misspelled word.
If you're using active_ids as an attribute (which, if it's integers, is certainly appropriate), then it should be a filter in the :with option, not in the :conditions option.
I'm not sure if this is related, but it's worth noting you've misspelled alignment in the query string (you've got alignmemt instead).