I have a university graduate database and would like to extract a random sample of data of around 1000 records.
I want to ensure the sample is representative of the population so would like to include the same proportions of courses eg
I could do this using the following:
select top 500 id from degree where coursecode = 1 order by newid()
union
select top 300 id from degree where coursecode = 2 order by newid()
union
select top 200 id from degree where coursecode = 3 order by newid()
but we have hundreds of courses codes so this would be time consuming and I would like to be able to reuse this code for different sample sizes and don't particularly want to go through the query and hard code the sample sizes.
Any help would be greatly appreciated
You want a stratified sample. I would recommend doing this by sorting the data by course code and doing an nth sample. Here is one method that works best if you have a large population size:
select d.*
from (select d.*,
row_number() over (order by coursecode, newid) as seqnum,
count(*) over () as cnt
from degree d
) d
where seqnum % (cnt / 500) = 1;
EDIT:
You can also calculate the population size for each group "on the fly":
select d.*
from (select d.*,
row_number() over (partition by coursecode order by newid) as seqnum,
count(*) over () as cnt,
count(*) over (partition by coursecode) as cc_cnt
from degree d
) d
where seqnum < 500 * (cc_cnt * 1.0 / cnt)
Add a table for storing population.
I think it should be like this:
SELECT *
FROM (
SELECT id, coursecode, ROW_NUMBER() OVER (PARTITION BY coursecode ORDER BY NEWID()) AS rn
FROM degree) t
LEFT OUTER JOIN
population p ON t.coursecode = p.coursecode
WHERE
rn <= p.SampleSize
It is not necessary to partition the population at all.
If you are taking a sample of 1000 from a population among hundreds of course codes, it stands to reason that many of those course codes will not be selected at all in any one sampling.
If the population is uniform (say, a continuous sequence of student IDs), a uniformly-distributed sample will automatically be representative of population weighting by course code. Since newid() is a uniform random sampler, you're good to go out of the box.
The only wrinkle that you might encounter is if a student ID is a associated with multiple course codes. In this case make a unique list (temporary table or subquery) containing a sequential id, student id and course code, sample the sequential id from it, grouping by student id to remove duplicates.
I've done similar queries (but not on MS SQL) using a ROW_NUMBER approach:
select ...
from
( select ...
,row_number() over (partition by coursecode order by newid()) as rn
from degree
) as d
join sample size as s
on d.coursecode = s.coursecode
and d.rn <= s.samplesize
Related
I need to perform TREAMMEAN in Access, which does not have this function.
In a table I have many Employees, each has many records.
I need to TRIMMEAN Values for each Employee separately.
Following queries perform TOP 10 percent for all records:
qry_data_TOP10_ASC
qry_data_TOP10_DESC
unionqry_TOP10_ASCandDESC
qry_data_ALL_minus_union_qry
After that, I can use Avg (Average).
But I don't know how to do it for each employee.
Visualization:
Note:
This question is edited to simplify problem.
You don't really give information in your pseudo code about your data fields but using your example that DOES have basic field information I can suggest the following should work as you described
It assumes field1 is your unique record ID - but you make no mention of which fields are keys
SELECT AVG(qry_data.field2) FROM qry_data WHERE qry_data.field1 NOT IN
(SELECT * FROM
(SELECT TOP 10 PERCENT qry_data.field1, qry_data.field2
FROM qry_data
ORDER BY qry_data.field2 ASC)
UNION
(SELECT TOP 10 PERCENT qry_data.field1, qry_data.field2
FROM qry_data
ORDER BY qry_data.field2 DESC)
)
This should give you what you want, the two sub-queries should correlate the TOP 10s (ascending and descending) for every employee. The two NOT INs should then remove those from the Table1 records and then you group the Employees and Average the Scores.
SELECT Table1.Employee, AVG(Table1.Score) AS AvgScore
FROM Table1
WHERE ID NOT IN
(
SELECT TOP 10 ID
FROM Table1 a
WHERE a.Employee = Table1.Employee
ORDER BY Score ASC, Employee, ID
)
AND ID NOT IN
(
SELECT TOP 10 ID
FROM Table1 b
WHERE b.Employee = Table1.Employee
ORDER BY Score DESC, Employee, ID
)
GROUP BY Table1.Employee;
What I am trying to do: I have a table, "band_style" with schema (band_id, style).
One band_id may occur multiple times, listed with different styles.
I want ALL rows of band_id, NUM (where NUM is the number of different styles a band has) for the band ids with the SECOND MOST number of styles.
I have spent hours on this query- almost nothing seems to be working.
This is how far I got. The table (data) successfully computes all bands with styles less than the maximum value of band styles. Now, I need ALL rows that have the Max NUM for the resulting table. This will give me bands with the second most number of styles.
However, this final result seems to be ignoring the MAX function and just returning the table (data) as is. Can someone please provide some insight/working method? I have over 20 attempts of this query with this being the closest.
Using SQL*PLUS on Oracle
WITH data AS (
SELECT band_id, COUNT(*) AS NUM FROM band_style GROUP BY band_id HAVING COUNT(*) <
(SELECT MAX(c) FROM
(SELECT COUNT(band_id) AS c
FROM band_style
GROUP BY band_id)))
SELECT data.band_id, data.NUM FROM data
INNER JOIN ( SELECT band_id m, MAX(NUM) n
FROM data GROUP BY band_id
) t
ON t.m = data.band_id
AND t.n = data.NUM;
Something like this... based on a Comment under your post, you are looking for DENSE_RANK()
select band_id
from ( select band_id, dense_rank() over (order by count(style) desc) as drk
from band_style
group by band_id
)
where drk = 2;
I would use a windowing function (RANK() in this case) - which is great for find the 'n' ranked thing in a set.
SELECT DISTINCT bs.band_id
FROM band_style bs
WHERE EXISTS (
SELECT NULL
FROM (
SELECT
bs2.band_id,
bs2.num,
RANK() OVER (ORDER BY bs2.num) AS numrank
FROM (
SELECT bs1.band_id, COUNT(*) as num
FROM band_style bs1
GROUP BY bs1.band_id ) bs2 ) bs3
WHERE bs.band_id = bs3.band_id
AND bs3.numrank = 2 )
I'm still learning the tricks and trade of PostgreSQL. I need a way of taking data and giving me the first 100 rows of each dataset.
My problem:
I have a table on the server that has over 60 columns. One column has the country. I need a breakdown where I get the first 100 rows by each country alphabetically.
There are 70 countries in this table. So the total results should be 7,000. How do I break this down?
Use ROW_NUMBER analytic function:
CREATE VIEW my_View AS
SELECT col1, col2, col3, ...... col60
FROM (
SELECT *,
row_number() over (Partition by country ) as Rn
FROM table
) x
WHERE rn <= 100
ORDER BY country
Thie above will give 100 random records for each country. If you do not want such a randomness, then please use ORDER BY some_columnclause in that way:
row_number() over (Partition by country ORDER BY country) as Rn
It is known that GROUP BY produces one row per group. I want to produce multiple rows per group. The particular use case is, for example, selecting two cheapest offerings for each item.
It is trivial for two or three elements in the group:
select type, variety, price
from fruits
where price = (select min(price) from fruits as f where f.type = fruits.type)
or price = (select min(price) from fruits as f where f.type = fruits.type
and price > (select min(price) from fruits as f2 where f2.type = fruits.type));
(Select n rows per group in mysql)
But I am looking for a query that can show n rows per group, where n is arbitrarily large. In other words, a query that displays 5 rows per group should be convertible to a query that displays 7 rows per group by just replacing some constants in it.
I am not constrained to any DBMS, so I am interested in any solution that runs on any DBMS. It is fine if it uses some non-standard syntax.
For any database that supports analytic functions\ window functions, this is relatively easy
select *
from (select type,
variety,
price,
rank() over ([partition by something]
order by price) rnk
from fruits) rank_subquery
where rnk <= 3
If you omit the [partition by something], you'll get the top three overall rows. If you want the top three for each type, you'd partition by type in your rank() function.
Depending on how you want to handle ties, you may want to use dense_rank() or row_number() rather than rank(). If two rows tie for first, using rank, the next row would have a rnk of 3 while it would have a rnk of 2 with dense_rank. In both cases, both tied rows would have a rnk of 1. row_number would arbitrarily give one of the two tied rows a rnk of 1 and the other a rnk of 2.
To save anyone looking some time, at the time of this writing, apparently this won't work because https://dev.mysql.com/doc/refman/5.7/en/subquery-restrictions.html.
I've never been a fan of correlated subqueries as most uses I saw for them could usually be written more simply, but I think this has changed by mind... a little. (This is for MySQL.)
SELECT `type`, `variety`, `price`
FROM `fruits` AS f2
WHERE `price` IN (
SELECT DISTINCT `price`
FROM `fruits` AS f1
WHERE f1.type = f2.type
ORDER BY `price` ASC
LIMIT X
)
;
Where X is the "arbitrary" value you wanted.
If you know how you want to limit further in cases of duplicate prices, and the data permits such limiting ...
SELECT `type`, `variety`, `price`
FROM `fruits` AS f2
WHERE (`price`, `other_identifying_criteria`) IN (
SELECT DISTINCT `price`, `other_identifying_criteria`
FROM `fruits` AS f1
WHERE f1.type = f2.type
ORDER BY `price` ASC, `other_identifying_criteria` [ASC|DESC]
LIMIT X
)
;
"greatest N per group problems" can easily be solved using window functions:
select type, variety, price
from (
select type, variety, price,
dense_rank() over (partition by type) order by price as rnk
from fruits
) t
where rnk <= 5;
Windows functions only work on SQL Server 2012 and above. Try this out:
SQL Server 2005 and Above Solution
DECLARE #yourTable TABLE(Category VARCHAR(50), SubCategory VARCHAR(50), price INT)
INSERT INTO #yourTable
VALUES ('Meat','Steak',1),
('Meat','Chicken Wings',3),
('Meat','Lamb Chops',5);
DECLARE #n INT = 2;
SELECT DISTINCT Category,CA.SubCategory,CA.price
FROM #yourTable A
CROSS APPLY
(
SELECT TOP (#n) SubCategory,price
FROM #yourTable B
WHERE A.Category = B.Category
ORDER BY price DESC
) CA
Results in two highest priced subCategories per Category:
Category SubCategory price
------------------------- ------------------------- -----------
Meat Chicken Wings 3
Meat Lamb Chops 5
I am looking for an optimized query
let me show you a small example.
Lets suppose I have a table having three field studentId, teacherId and subject as
Now I want those data in which a physics teacher is teaching to only one student, i.e
teacher 300 is only teaching student 3 and so on.
What I have tried till now
select sid,tid from tabletesting with(nolock)
where tid in (select tid from tabletesting with(nolock)
where subject='physics' group by tid having count(tid) = 1)
and subject='physics'
The above query is working fine. But I want different solution in which I don't have to scan the same table twice.
I also tried using Rank() and Row_Number() but no result.
FYI :
I have showed you an example, this is not the actual table i am playing with, my table contain huge number of rows and columns and where clause is also very complex(i.e date comparison etc.), so I don't want to give the same where clause in subquery and outquery.
You can do this with window functions. Assuming that there are no duplicate students for a given teacher (as in your sample data):
select tt.sid, tt.tid
from (select tt.*, count(*) over (partition by teacher) as scnt
from TableTesting tt
) tt
where scnt = 1;
Another way to approach this, which might be more efficient, is to use an exists clause:
select tt.sid, tt.tid
from TableTesting tt
where not exists (select 1 from TableTesting tt1 where tt1.tid = tt.tid and tt1.sid <> tt.sid)
Another option is to use an analytic function:
select sid, tid, subject from
(
select sid, tid, subject, count(sid) over (partition by subject, tid) cnt
from tabletesting
) X
where cnt = 1