I tried this query on MySQL server (5.1.41)...
SELECT max(volume), dateofclose, symbol, volume, close, market FROM daily group by market
I got this result:
max(volume) dateofclose symbol volume close market
287031500 2010-07-20 AA.P 500 66.41 AMEX
242233000 2010-07-20 AACC 16200 3.98 NASDAQ
1073538000 2010-07-20 A 4361000 27.52 NYSE
2147483647 2010-07-20 AAAE.OB 400 0.01 OTCBB
437462400 2010-07-20 AAB.TO 31400 0.37 TSX
61106320 2010-07-20 AA.V 0 0.24 TSXV
As you can see, the maximum volume is VERY different from the 'real' value of the volume column?!?
The volume column is define as int(11) and I got 2 million rows in this table but it's very far from the max of MyISAM storage so I cannot believed this is the problem!? What is also strange is data get show from the same date (dateofclose). If I force a specific date with a WHERE clause, the same symbol came out with different max(volume) result. This is pretty weird...
Need some help here!
UPDATE :
Here's my edited "working" request:
SELECT a.* FROM daily a
INNER JOIN (
SELECT market, MAX(volume) AS max_volume
FROM daily
WHERE dateofclose = '20101108'
GROUP BY market
) b ON
a.market = b.market AND
a.volume = b.max_volume
So this give me, by market, the highest volume's stock (for nov 8, 2010).
As you can see, the maximum volume is VERY different from the 'real' value of the volume column?!?
This is because MySQL rather bizarrely doesn't GROUP things in a sensical way.
Selecting MAX(column) will get you the maximum value for that column, but selecting other columns (or column itself) will not necessarily select the entire row that the found MAX() value is in. You essentially get an arbitrary (and usually useless) row back.
Here's a thread with some workarounds using subqueries:
How can I SELECT rows with MAX(Column value), DISTINCT by another column in SQL?
This is a subset of the "greatest n per group" problem. (There is a tag with that name but I am a new user so I can't retag).
This is usually best handled with an analytic function, but can also be written with a join to a sub-query using the same table. In the sub-query you identify the max value, then join to the original table on the keys to find the row that matches the max.
Assuming that {dateofclose, symbol, market} is the grain at which you want the maximum volume, try:
select
a.*, b.max_volume
from daily a
join
(
select
dateofclose, symbol, market, max(volume) as max_volume
from daily
group by
dateofclose, symbol, market
) b
on
a.dateofclose = b.dateofclose
and a.symbol = b.symbol
and a.market = b.market
Also see this post for reference.
Did you try adjusting your query to include Symbol in the grouping?
SELECT max(volume), dateofclose, symbol,
volume, close, market FROM daily group by market, symbol
Related
I downloaded the entire FDIC bank call reports dataset, and uploaded it to BigQuery.
The table I currently have looks like this:
What I am trying to accomplish is adding a column showing the deposit growth rate since the last quarter for each bank:
Note:The first reporting date for each bank (e.g. 19921231) will not have a "Quarterly Deposit Growth". Hence the two empty cells for the two banks.
I would like to know if a bank is increasing or decreasing its deposits each quarter/call report (viewed as a percentage).
e.g. "On their last call report (19921231)First National Bank had deposits of 456789 (in 1000's). In their next call report (19930331)First National bank had deposits of 567890 (in 1000's). What is the percentage increase (or decrease) in deposits"?
This "_%_Change_in_Deposits" column would be displayed as a new column.
This is the code I have written so far:
select
SFRNLL.repdte, SFRNLL.cert, SFRNLL.name, SFRNLL.city, SFRNLL.county, SFRNLL.stalp, SFRNLL.specgrp AS `Loan_Specialization`, SFRNLL.lnreres as `_1_to_4_Residential_Loans`, AL.dep as `Deposits`, AL.lnlsnet as `loans_and_leases`,
IEEE_DIVIDE(SFRNLL.lnreres, AL.lnlsnet) as SFR2TotalLoanRatio
FROM usa_fdic_call_reports_1992.All_Reports_19921231_1_4_Family_Residential_Net_Loans_and_Leases as SFRNLL
JOIN usa_fdic_call_reports_1992.All_Reports_19921231_Assets_and_Liabilities as AL
ON SFRNLL.cert = AL.cert
where SFRNLL.specgrp = 4 and IEEE_DIVIDE(SFRNLL.lnreres, AL.lnlsnet) <= 0.10
UNION ALL
select
SFRNLL.repdte, SFRNLL.cert, SFRNLL.name, SFRNLL.city, SFRNLL.county, SFRNLL.stalp, SFRNLL.specgrp AS `Loan_Specialization`, SFRNLL.lnreres as `_1_to_4_Residential_Loans`, AL.dep as `Deposits`, AL.lnlsnet as `loans_and_leases`,
IEEE_DIVIDE(SFRNLL.lnreres, AL.lnlsnet) as SFR2TotalLoanRatio
FROM usa_fdic_call_reports_1993.All_Reports_19930331_1_4_Family_Residential_Net_Loans_and_Leases as SFRNLL
JOIN usa_fdic_call_reports_1993.All_Reports_19930331_Assets_and_Liabilities as AL
ON SFRNLL.cert = AL.cert
where SFRNLL.specgrp = 4 and IEEE_DIVIDE(SFRNLL.lnreres, AL.lnlsnet) <= 0.10
The table looks like this:
Additional notes:
I would also like to view the last column (SFR2TotalLoansRatio) as a percentage.
This code runs correctly, however, previously I was getting a "division by zero" error when attempting to run 50,000 rows (1992 to the present).
Addressing each of your question individually.
First) Retrieving SFR2TotalLoanRatio as percentage, I assume you want to see 9.988% instead of 0.0988 in your results. Currently, in BigQuery you can achieve this by casting the field into a STRING then, concatenating the % sign. Below there is an example with sample data:
WITH data as (
SELECT 0.0123 as percentage UNION ALL
SELECT 0.0999 as percentage UNION ALL
SELECT 0.3456 as percentage
)
SELECT CONCAT(CAST(percentage*100 as String),"%") as formatted_percentage FROM data
And the output,
Row formatted_percentage
1 1.23%
2 9.99%
3 34.56%
Second) Regarding your question about the division by zero error. I am assuming IEEE_DIVIDE(arg1,arg2) is a function to perform the division, in which arg1 is the divisor and arg2 is the dividend. Therefore, I would adivse your to explore your data in order to figured out which records have divisor equals to zero. After gathering these results, you can determine what to do with them. In case you decide to discard them you can simply add within your WHERE statement in each of your JOINs: AL.lnlsnet = 0. On the other hand, you can also modify the records where lnlsnet = 0 using a CASE WHEN or IF statements.
UPDATE:
In order to add this piece of code your query, you u have to wrap your code within a temporary table. Then, I will make two adjustments, first a temporary function in order to calculate the percentage and format it with the % sign. Second, retrieving the previous number of deposits to calculate the desired percentage. I am also assuming that cert is the individual id for each of the bank's clients. The modifications will be as follows:
#the following function MUST be the first thing within your query
CREATE TEMP FUNCTION percent(dep INT64, prev_dep INT64) AS (
Concat(Cast((dep-prev_dep)/prev_dep*100 AS STRING), "%")
);
#followed by the query you have created so far as a temporary table, notice the the comma I added after the last parentheses
WITH data AS(
#your query
),
#within this second part you need to select all the columns from data, and LAG function will be used to retrieve the previous number of deposits for each client
data_2 as (
SELECT repdte, cert, name, city, county, stalp, Loan_Specialization, _1_to_4_Residential_Loans,Deposits, loans_and_leases, SFR2TotalLoanRatio,
CASE WHEN cert = lag(cert) OVER (PARTITION BY id ORDER BY d) THEN lag(Deposits) OVER (PARTITION BY id ORDER BY id) ELSE NULL END AS prev_dep FROM data
)
SELECT repdte, cert, name, city, county, stalp, Loan_Specialization, _1_to_4_Residential_Loans,Deposits, loans_and_leases, SFR2TotalLoanRatio, percent(Deposits,prev_dep) as dept_growth_rate FROM data_2
Note that the built-in function LAG is used together with CASE WHEN in order to retrieve the previous amount of deposits per client.
So I have two tables of sales, budget and actual.
"budget" has two columns: location and sales. For example,
location sales
24 $20000
36 $100300
40 $24700
Total $145000
"actual" has three columns: invoice_number, location, and sales. For example,
invoice location sales
10000 36 $5000
10001 40 $6000
10002 99 $7000
and so forth
Total $110000
In summary, "actual" records transactions at the invoice level, whereas "budget" is done at the location level only (no individual invoices).
I'm trying to create a summary table that lists actual and budget sales side by side, grouped by location. The total of the actual column should be $110000, and $145000 for budget. This is my attempt at it (on pgAdmin/ postgresql):
SELECT actual.location, SUM(actual.sales) AS actual_sales, SUM(budget.sales) AS budget_sales
FROM actual LEFT JOIN budget
ON actual.location = budget.location
GROUP BY actual.location;
I used LEFT JOIN because "actual" has locations that "budget" doesn't have (e.g. location 99).
I ended up with some gigantic numbers ($millions) on both the actual_sales and budget_sales columns, far exceeding the total actual ($110000) or budget sales ($145,000).
Is this because the way I wrote my query is basically asking SQL to join each invoice in "actual" to each line in "budget," therefore duplicating things many times over? If so how should I have written this?
Thanks in advance!
Based on your description, you seem to have duplicates in both tables. There are various ways to solve this problem. Here is one using union all and group by:
select Location,
sum(actual_sales) as actual_sales,
sum(budget_sales) as budget_sales
from ((select a.location, a.sales as actual_sales, null as budget_sales
from actual a
) union all
(select b.location, null, b.sales
from budget b
)
) ab
group by location;
This structure guarantees that each value is counted only once, regardless of the table.
The query looks fine to me. However, it is difficult to find out why the figures are wrong. My suggestion is that you do the sum by location separately for budget and actual into 2 temporary tables, and later put them together using LEFT JOIN.
Yes, you're joining the budget in once for each actual sales row. However, your Actual Sales sum shouldn't have been larger unless there were multiple budget rows for the same location. You should check for that, because it doesn't sound like there should be.
What you need to do in a case like this is sum the actual sales first in a CTE or subquery, then later join the result to the budget. That way you only have one row for each location. This does it for the actual sales. If you really do have more than one row for a location for budget as well, you might need to subquery the budget as well the same way.
Select Act.Location, Act.actual_sales, budget.sales as budget_sales
From
(
SELECT actual.location, SUM(actual.sales) AS actual_sales
FROM actual
GROUP BY actual.location
) Act
left join budget on Act.location = budget.location
Gordon's suggestion is good, an alternative using WITH statements is:
WITH aloc AS (
SELECT location, SUM(sales) FROM actual GROUP BY 1
), bloc AS (
SELECT location, SUM(sales) FROM budget GROUP BY 1
)
SELECT location, a.sum AS actual_sales, b.sum AS budget_sales
FROM aloc a LEFT JOIN bloc b USING (location)
This is equivalent to:
SELECT location, a.sum AS actual_sales, b.sum AS budget_sales
FROM (SELECT location, SUM(sales) FROM actual GROUP BY 1) a LEFT JOIN
(SELECT location, SUM(sales) FROM budget GROUP BY 1) b USING (location)
but I find WITH statements more readable.
The purpose of the subqueries is to get tables into a state where a row means something relevant, i.e. aloc contains a row per location, and hence cause the join to evaluate to what you want.
I'm relatively new to SQL but have learned some cool stuff. I'm getting results that don't make sense. I've got a query with several subqueries and what-not but I have a windowed function that isn't working like I'm expecting.
The part that isn't working is this (simplified from the 300 line query):
SELECT AVG(table.sales_amount)
OVER (PARTITION BY table.month, table.sales_rep, table.department)
FROM table
The problem is that when I pull the data non aggregated I get a value different (107) than the above returns (95).
I've used windowed functions for COUNT and SUM and they work fine, but AVG is acting strangely. Am I missing something about how this works with AVG?
The subquery that table is a standin for looks like:
sales_rep, month, department, sales_amount
1, 2017-1, abc, 125.20
1, 2017-2, abc, 120.00
2, 2017-1, def, 100.00
...etc
Working out of Sql Server Management studio
SOLVED: I did finally figure it out, the results i was joining this subquery to had the sales rep multiple times in a month selling objects A&B which caused whoever sold both to be counted twice. whoops, my bad.
The results that you get should be the same values as in:
SELECT AVG(table.sales_amount)
FROM table
GROUP BY table.month, table.sales_rep, table.department;
Of course, the rows will be different. You need to match up the three key columns.
Based on your sample data, it looks like the partitioning keys uniquely define each row. Perhaps you really intend:
SELECT AVG(table.sales_amount) OVER () as overall_average
FROM table;
EDIT:
For the departmental average:
SELECT AVG(table.sales_amount) OVER (partition by table.department) as department_average
FROM table;
After some bruteforcing of potential errors I finally figured out the issue. I was joining that subquery to the another which had multiple instances of a sales_rep in a given month (selling objects a & b) which caused the average of those with sales of both objects to be counted twice instead of once.
so sales rep 1 sold objects a & b which made his avg count as 66% of the dept avg instead of 50%, and sales rep 2 count only 33%.
I am trying to get a total hours from a dataset and because you can have the same asset with the same company (company_B) twice at two different times I have this join issue. I know I want the min for company_B gone and the Max for company_B gone because they represent wrong dates being matched. The negative is easy but what about the Max?
I have:
AssetID------StartDate-------FinishDate-------CompanyName----HoursOnSite
22222-------2016-02-12-------2016-02-20-------Company_A--------192
22222-------2016-02-01-------2016-02-09-------Company_B--------208 (keep)
22222-------2016-02-12-------2016-02-09-------Company_B-------(-56) (remove)
22222-------2016-02-01-------2016-02-21-------Company_B--------480 (remove)
22222-------2016-02-12-------2016-02-21-------Company_B--------216 (keep)
55555-------2016-02-18-------2016-02-22-------Company_C--------96
99584-------2016-02-22-------2016-02-25-------Company_D--------63
I think you can do the query for the records with max and min HoursOnSite for company B, and use (not in) or not equal to exclude those records.
If you still have concern, please paste your query.
I'm assuming that there has to be atleast 3 instances of unique assetid - companyname combination for the Max, Min filters to work. You can change it in the final where statement tO suit your requirement
WITH CTE
AS (
SELECT *
,count(CompanyName) OVER (PARTITION BY AssetID,CompanyName) AS a
FROM <TABLE_NAME>
)
SELECT *
FROM CTE
WHERE HoursOnSite NOT IN (
SELECT MAX(HoursOnSite)
FROM <TABLE_NAME>
)
AND gdp NOT IN (
SELECT min(HoursOnSite)
FROM <TABLE_NAME>
)
AND a > 2 --MODIFY AS PER YOUR REQUIREMENT
I have seen other questions like this one but feel mine is a bit different, or didn't quite understand the SQL in the other questions...so my apologies if this one is redundant or very easy..
Anyway, I have an accounting transaction DB that stores every transaction posting within our financial system on one line. What I am trying to do is net the sum of the debits and the credits for each GL account.
Here are the two basic queries I am executing to get the results that I would like to net.
Query 1 gives me the sum of all debit transactions posting to each gl account:
Select gl_debit, sum (amt) from FISC_YEAR2014 where fund = 'XXX'
group by gl_debit
Query 2 gives me the sum of all credit transactions posting to each gl account:
select gl_credit, sum (amt) from FISC_YEAR2014 where fund = 'XXX'
group by gl_credt
Now I would to subtract the credit amounts from the debit amounts to get net totals for each gl account. Make sense?
Thanks.
There are two ways to do this depending our your table definition. I think your situation is the first.
This is the normal way assuming credits and debits are in separate columns:
SELECT sum(gl_debit)-sum(gl_credit) as net_debit
FROM FISC_YEAR2014
WHERE fund = 'XXX'
This is the other way assuming direction is indicated by a separate column:
SELECT SUM(IF(is_debit=1,amount,-1*amount)) as net_debit
FROM FISC_YEAR2014
WHERE fund = 'XXX'
See also:
MySQL 'IF' in 'SELECT' statement
Can't calculate totals in general ledger report
What's a good way to store a financial ledger?
I believe this is what you need:
select
gl_account,
sum(amt)
from
(
select gl_debit gl_account,
sum(-amt) amt
from fisc_year2014
where fund = 'XXX'
group by gl_debit
union all
select gl_credit,
sum(amt)
from fisc_year2014
where fund = 'XXX'
group by gl_credit
)
group by
gl_account
There are two SELECTs: one to get the (negative) debits and another to get the credits. They are UNIONed to create a two-column result. The outer SELECT then aggregates the total sum by the gl_account code. If there is a mismatch (a gl_debit without a gl_credit, or vice-versa), then its amount would still be displayed.
SQLFiddle here (I added another row to show the effect of mismatched IDs)
To do this you should SUM the debits and credits separately in subqueries, then join those subqueries on gl_credit = gl_debit.
SELECT COALESCE(gl_credit, gl_debit) AS Id
,COALESCE(d.amt,0)-COALESCE(c.amt,0) AS Net
FROM (
SELECT gl_debit, SUM(amt) AS amt
FROM FISC_YEAR2014
GROUP BY gl_debit
) d
FULL OUTER JOIN (
SELECT gl_credit, SUM(amt) AS amt
FROM FISC_YEAR2014
GROUP BY gl_credit
) c ON d.gl_debit = c.gl_credit
ORDER BY COALESCE(gl_credit, gl_debit)
SQLFiddle
Outputs:
ID Net
-----------
101 -475
201 225
301 500
501 -250
If I were you rather than using a FULL OUTER JOIN, I'd select the ids from the accounts table or wherever you store them, then LEFT JOIN both of the subqueries to it, you haven't shown any other tables though so I can only speculate.