Total disclosure: I'm a SQL beginner.
I have a data set of certain accounting and governance metrics for US companies. It has about 15 columns and roughly 18 million rows. Each row is a unique combination of company, date and metric being measured. The columns include certain identifiers like isin number, ticker symbol, etc, the date the metric was released, the metric description, and the metric itself.
What I'm trying to do is write a query that will yield the NEWEST values for a certain metric for all companies. In my hopeless search over the past few days I've come to think that the GROUP BY clause may be what I'm looking for. However, it doesn't seem to do exactly what I need. I've got it working with just 2 columns: isin number (company identifier), and date. In other words, I can spit out a list that shows the most recent date for each company, but I'm not sure how to add more columns to this, how to specify what metric to look at.
Any guidance would be appreciated, even if it's just pointing me in the right direction towards what kind of commands I should be looking into.
Thanks!
EDIT: Wow. Thanks for the quick and thorough replies. And point taken on the clarity and example data sets/starting query. Update: I think I have it working. Here's what I used:
SELECT a1.["id_isin_number"], a1.["metric_description"], a1.["date_period_ends"], a1.["company_metric_value"], a2.maxdate
FROM [AGR Metrics].[dbo].[Audit_Integrity_Metric_Data_File_NA Original_0] a1
INNER JOIN (
SELECT a2.["id_isin_number"], MAX(a2.["date_period_ends"]) AS maxdate
FROM [AGR Metrics].[dbo].[Audit_Integrity_Metric_Data_File_NA Original_0] a2
GROUP BY a2.["id_isin_number"]
) a2
ON a1.["date_period_ends"] = a2.maxdate
AND a1.["id_isin_number"] = a2.["id_isin_number"]
WHERE a1.["metric_description"] = '"Litigation: Class Action"'
I'm looking over the responses now to make sure I'm doing this as efficiently as possible.
You can use the ROW_NUMBER() function for this (if using SQL Server 2005 or newer):
SELECT *
FROM (SELECT *,ROW_NUMBER() OVER(PARTITION BY isin ORDER BY [date] DESC) AS RowRank
FROM YourTable
)sub
WHERE RowRank = 1
Just list out the fields you want in place of * if you don't want them all returned.
The ROW_NUMBER() function adds a number to each row, PARTITION BY is optional and is used to define a group for which numbering will start over at 1, in this case, you want the most recent for each value of isin so we PARTITION BY that. ORDER BY is required and defines the order of the numbering, in this case by date.
Your current query can also be used, but the ROW_NUMBER() method is simpler and more efficient:
SELECT a.*
FROM YourTable a
JOIN (SELECT isin, MAX([date])
FROM YourTable
GROUP BY isin
)b
ON a.isin = b.isin
AND a.[date] = b.[date]
Well as you quote the date the metric was released , So you can use it to sort your table using Order By .
This is a very basic example which can be used to simply sort data and selecting top 1 value.
Please refer This
CREATE TABLE trialOne (
Id INT NULL,
NAME VARCHAR(50) NULL,
[Date] DATETIME NULL
)
SELECT * FROM dbo.ETProgram
INSERT INTO trialone VALUES(1,'john','2009-01-06 11:39:51.827')
INSERT INTO trialone VALUES(2,'joseph','2010-01-06' )
INSERT INTO trialone VALUES(3,'Ajay','2009-05-06' )
INSERT INTO trialone VALUES(4,'Dave','2009-11-06' )
INSERT INTO trialone VALUES(5,'jonny','2004-01-06')
INSERT INTO trialone VALUES(6,'sunny','2005-01-06')
INSERT INTO trialone VALUES(7,'elle','2013-01-06' )
INSERT INTO trialone VALUES(8,'mac','2012-01-06' )
INSERT INTO trialone VALUES(8,'Sam','2008-01-06' )
INSERT INTO trialone VALUES(10,'xxxxx','2013-08-06')
SELECT TOP(1)name FROM trialone ORDER BY Date DESC
Related
I haven't been able to find any similar question but I am looking for a way to delete all but 1 of similar rows that have 2 specific columns that contain identical data. For example:
price
symbol
date
13
RT
2020-10-1
80.9
DX
2020-10-2
81
DX
2020-10-2
90
AP
2020-10-3
89.9
AP
2020-10-3
90
AP
2020-10-3
85
DX
2020-10-4
In this example, I'd like to be able to run a query in the BQ console to find any of the rows with that have both the date AND the symbol as identical and delete one of them (which one gets deleted doesn't matter much.) The query would delete 1 of the DX rows on 2020-10-2 and 2 of the AP rows on 2020-10-3.
I appreciate the help!!
As you are using the big-query, I would suggest you to use CREATE OR REPLACE TABLE as follows:
CREATE OR REPLACE TABLE your_table
AS SELECT DISTINCT price, symbol, date
FROM your_table;
You can use this example code.
DELETE FROM [SampleDB].[dbo].[Employee]
WHERE ID NOT IN
(
SELECT MAX(ID) AS MaxRecordID
FROM [SampleDB].[dbo].[Employee]
GROUP BY [FirstName],
[LastName],
[Country]
);
Check this link for more info: https://www.sqlshack.com/different-ways-to-sql-delete-duplicate-rows-from-a-sql-table/
You specifically say that you want to delete based on two columns, not all three. In your example data, the price is the same on all rows, but that might not be the case in the real data.
You can use create or replace table, but I would recommend:
CREATE OR REPLACE TABLE t AS
SELECT ARRAY_AGG(t LIMIT 1)[ORDINAL(1)].*
FROM `t` t
GROUP BY symbol, date;
You can also express this using window functions:
CREATE OR REPLACE TABLE t AS
SELECT t.* EXCEPT (seqnum)
FROM (SELECT t.*,
ROW_NUMBER() OVER (PARTITION BY symbol, date ORDER BY price) as seqnum
FROM `t` t
) t
WHERE seqnum = 1;
Below is for BigQuery Standard SQL
create or replace table your_table as
with temp as (
select as value array_agg(t order by price limit 1) [offset(0)]
from your_table t
group by symbol, date
)
select * from temp;
Note: you can remove order by price part if you don't care about which exactly row to survive out of those with duplicate symbol and date
if applied to sample data from your question - resulted table is
i´m working with a table that looks like this:
Start
https://i.stack.imgur.com/uibc3.png
My desired result would look like this:
Result
https://i.stack.imgur.com/v0sic.png
So i´m triyng to select the max value from two "combined" colums. If the values are the same amount (Part C), the outcome doesn't matter.
I tried to order the table by max value and then using distinct but the result didn't turn out as expected
Could you please offer a solution or some insight to this? Thanks in advance!
Use row_number():
select *
from (
select t.*, row_number() over(partition by part order by amount desc, zone) rn
from mytable t
) t
where rn = 1
For each part, this gives you the row with the highest amount; if there are top ties, column zone is used to break them.
If you want to allow ties, then use rank() instead, like:
rank() over(partition by part order by amount desc) rn
You can achieve this by using SUB Query
DECLARE #T TABLE(
PART VARCHAR(50),
ZONE VARCHAR(10),
Amt INT)
Insert Into #T Values ('PartA','71H',1),('PartA','75H',2),('PartB','98D',1),('PartB','98A',3),('PartC','75H',1),('PartC','52H',1)
SELECT M.PART,MIN(M.Zone) AS ZONE,S.AMOUNT
FROM #T M
INNER JOIN (
SELECT Part,MAX(Amt) as AMOUNT From #T
GROUP BY PART) S ON S.AMOUNT=M.Amt AND S.PART=M.PART
GROUP BY M.PART,S.AMOUNT
ORDER BY M.PART
I am having a little bit of trouble figuring out a way to select just the third or fourth row in a query I am writing, any help would be greatly appreciated.
This is an example of the code I came up with, this however only selects the first row.
Left Outer Join (select ap_attachments.ap_table_key, ap_description, ap_creation_date, ap_creation_time, ap_file_name, ap_attach_id
from ap_attachments
inner join (select Min(ap_attachment_id) ap_attach_id, ap_table_key
from ap_attachments
where ap_file_name like '%jpg%'
group by ap_table_key) C
On ap_attachments.ap_attachment_id = C.ap_attach_id) apImgThree_attach
On apImgTwo_attach.ap_table_key = order_link.to_order_id
You can do this with the ROW_NUMBER() function:
select ap_attachment_id, ap_table_key,ROW_NUMBER() OVER(PARTITION BY ap_table_key ORDER BY ap_attachment_id) AS RN
from ap_attachments
where ap_file_name like '%jpg%'
Then you can specify which row you'd like to return using the RN value. This may require some adapting depending on your source data, the DENSE_RANK() function may be more appropriate.
The ROW_NUMBER() function assigns a number to each row. PARTITION BY is optional, but used to start the numbering over for each value in that group, ie: if you PARTITION BY Some_Date then for each unique date value the numbering would start over at 1. ORDER BY of course is used to define how the counting should go, and is required in the ROW_NUMBER() function.
Look up the docs on lead and lag. You can also use the PARTITION clause to create the window within a specific date, for example;
declare #table table(
[flower] [sysname]);
insert into #table
([flower])
values (N'rose'),
(N'tulip'),
(N'chamomile'),
(N'lily');
select [flower] from #table order by [flower];
select [flower]
, lag ([flower]
, 1
, 0)
over (
order by [flower] desc) as [previous_flower]
, lead ([flower]
, 1
, 0)
over (
order by [flower] desc) as [next_flower]
from #table;
I have a query that partitions and ranks "Note" records, grouping them by ID_Task (users add notes for each task). I want to rank the notes by date, but I also want to restrict it so they're ranked between two dates.
I'm using SQL Server 2008. So far my SELECT looks like this:
SELECT Note.ID,
Note.ID_Task,
Note.[Days],
Note.[Date],
ROW_NUMBER() OVER (PARTITION BY ID_Task ORDER BY CAST([Date] AS DATE), Edited ASC) AS Rank
FROM
Note
WHERE
Note.Locked = 1 AND Note.Deleted = 0
Now, I assume that if I put the WHERE clause at the bottom, although they'll still have ranks, I might or might not get item with rank 1, as it might get filtered out. So is there a way I can only partition records WHERE , ignoring all of the others? I could partition a sub-query I guess.
The intention is to use the rank number to find the most recent note for each task, in another query. So in that query I'll join with this result WHERE rank = 1.
row_number() operates after where. You'll always get a row 1.
For example:
declare #t table (id int)
insert #t values (3), (1), (4)
select row_number() over (order by id)
from #t
where id > 1
This prints:
1
2
Each of my rows have a date. I want the database to keep the good date. But I am in a situation where I want only the first date. But I still want all the other rows. So I would like to fill the date column with all the same date in my result.
For an example (Because I don't think I expressed myself well)
I have this:
name value date
a 10 5/13
b 14 2/13
c 20 1/13
a 11 7/13
a 5 8/13
b 8 9/13
I want it to become like this in the result:
name value date
a 26 5/13
b 22 5/13
c 20 5/13
I searched for this information but I only find the way to select the first row.
for now I'm doing
SELECT name, SUM(value), date FROM table
ORDER BY name
And I'm kind of clueless for what to do next.
Thanks :)
Databases don't have a concept of "first". Here is an attempt, but no guarantees unless you have a way of ordering to determine first:
select name, sum(value), const.date
from table cross join
(select top 1 date from table) const
group by name, const.date
If you only want to do this for a query, to provide this aggregated data for some specific client requirement, then #freshPrince's answer is appropriate. But if want to actually modify the data in the table itself, and prevent the issue from arising again, then you need to change the schema.
Create Table newTable(
name varChar(30) not null,
date datetime not null,
value decimal(10,2) not null default(0),
primary key (name, date) )
Insert newTable (name, date, value)
Select name, SUM(value), Min(date)
FROM currentTable
Group By Name
and delete the old table... then rename the new table to whatever...
You will also have to modify the process used to insert new rows so that instread of always inserting a new row, it updates the existing row for a specified name and date if it already exists...
Your question is slightly confusing since your desired result is showing a date that does not exists with either b or c but if that is the result that you want want you could use something similar to the following:
select name, sum(value) value, d.date
from yt
cross join
(
select min(date) date
from yt
where name = (select min(name)
from yt)
) d
group by name, d.date;
See SQL Fiddle with Demo
But it seems like you actually would want the min(date) for each name:
select name, sum(value) value, min(date)
from yt
group by name;
See SQL Fiddle with Demo.
If the order of the date should be the determined by the name then you could use:
select t.name, sum(value) value, d.date
from yt t
cross join
(
select top 1 name, date
from yt
order by name, date
) d
group by t.name, d.date;
See Demo