So this should be fairly simple, and I'm sure there's an embarrassingly easily solution I'm missing, but here goes:
I want to create a grid of numbers based on two numeric variables.
More specifically, I want to select the 5th and 95th percentile of each variable, then cut up the difference between those two values into 100 parts, and then group by those.
So basically what I need is in pseudocode
(5th percentile)+(95th percentile-5th percentile)/100*[all numbers from 0 to 100]
I can pick out the 5th and 95th percentile with the following query:
SELECT Min(subq.lat) as latitude, percentile FROM
(SELECT round(latitude,2) as lat, ntile(100) OVER (order by latitude desc) as
'percentile' FROM table ORDER BY latitude DESC) AS subq
where percentile in (5,95)
group by 2
And I can can create a list of numbers from 0 to 100 as well.
But how to combine those two is something that's a little beyond me.
Help would be much appreciated.
I'm not entirely sure I follow what you're after, but it could be as simple as looping through 1-100, performing your calculation for each set and inserting them into a results table:
CREATE TABLE #Results (Counter_ INT, Calc_Value FLOAT)
GO
DECLARE #intFlag INT
SET #intFlag = 1
WHILE (#intFlag <=100)
BEGIN
--Do Stuff
INSERT INTO #Results
SELECT Counter_ = #intFlag
,Calc_Value = (calculation logic)/#intFlag
SET #intFlag = #intFlag + 1
END
GO
The 'Do Stuff' portion gets executed for each value 1 - 100, the (Calculation logic) would obviously need to be replaced with whatever logic you use. If those values are constant for 1-100, you could set them as variables so they don't have to run 100 times. Roughly:
CREATE TABLE #Results (Counter_ INT, Calc_Value FLOAT)
GO
DECLARE #intFlag INT, #Percentile_Value FLOAT = (Calculation Logic)
SET #intFlag = 1
WHILE (#intFlag <=100)
BEGIN
--Do Stuff
INSERT INTO #Results
SELECT Counter_ = #intFlag
,Calc_Value = #Percentile_Value/#intFlag
SET #intFlag = #intFlag + 1
END
GO
You can do what you want to do with window functions. Basically, do the percentile calculation with row_number() and the total count.
Here is an example:
SELECT lat,
((seqnum - 0.05 * cnt) / (0.95 * cnt - 0.05 * cnt)) * 100 as NewPercentile
FROM (SELECT round(latitude,2) as lat,
row_number() over (order by latitude) as seqnum,
count(*) over () as cnt
FROM table
ORDER BY latitude DESC
) AS subq
where seqnum between 0.05 * cnt and 0.95 * cnt
Related
I have a SQL Server database. I need to loop through a table to get the count of each value in the column 'RevID'. Each value should only be in the table a certain number of times - for example 125 times. If the count of the value is greater than 125 or less than 125, I need to update the column to ensure all values in the RevID (are over 25 different values) is within the same range of 125 (ok to be a few numbers off)
For example, the count of RevID = "A2" is = 45 and the count of RevID = 'B2' is = 165 then I need to update RevID so the 45 count increases and the 165 decreases until they are within the 125 range.
This is what I have so far:
DECLARE #i INT = 1,
#RevCnt INT = SELECT RevId, COUNT(RevId) FROM MyTable group by RevId
WHILE(#RevCnt >= 50)
BEGIN
UPDATE MyTable
SET RevID= (SELECT COUNT(RevID) FROM MyTable)
WHERE RevID < 50)
#i = #i + 1
END
I have also played around with a cursor and instead of trigger. Any idea on how to achieve this? Thanks for any input.
Okay I cam back to this because I found it interesting even though clearly there are some business rules/discussion that you and I and others are not seeing. anyway, if you want to evenly and distribute arbitrarily there are a few ways you could do it by building recursive Common Table Expressions [CTE] or by building temp tables and more. Anyway here is a way that I decided to give it a try, I did utilize 1 temp table because sql was throwing in a little inconsistency with the main logic table as a cte about every 10th time but the temp table seems to have cleared that up. Anyway, this will evenly spread RevId arbitrarily and randomly assigning any remainder (# of Records / # of RevIds) to one of the RevIds. This script also doesn't rely on having a UniqueID or anything it works dynamically over row numbers it creates..... here you go just subtract out test data etc and you have what you more than likely want. Though rebuilding the table/values would probably be easier.
--Build Some Test Data
DECLARE #Table AS TABLE (RevId VARCHAR(10))
DECLARE #C AS INT = 1
WHILE #C <= 400
BEGIN
IF #C <= 200
BEGIN
INSERT INTO #Table (RevId) VALUES ('A1')
END
IF #c <= 170
BEGIN
INSERT INTO #Table (RevId) VALUES ('B2')
END
IF #c <= 100
BEGIN
INSERT INTO #Table (RevId) VALUES ('C3')
END
IF #c <= 400
BEGIN
INSERT INTO #Table (RevId) VALUES ('D4')
END
IF #c <= 1
BEGIN
INSERT INTO #Table (RevId) VALUES ('E5')
END
SET #C = #C+ 1
END
--save starting counts of test data to temp table to compare with later
IF OBJECT_ID('tempdb..#StartingCounts') IS NOT NULL
BEGIN
DROP TABLE #StartingCounts
END
SELECT
RevId
,COUNT(*) as Occurences
INTO #StartingCounts
FROM
#Table
GROUP BY
RevId
ORDER BY
RevId
/************************ This is the main method **********************************/
--clear temp table that is the main processing logic
IF OBJECT_ID('tempdb..#RowNumsToChange') IS NOT NULL
BEGIN
DROP TABLE #RowNumsToChange
END
--figure out how many records there are and how many there should be for each RevId
;WITH cteTargetNumbers AS (
SELECT
RevId
--,COUNT(*) as RevIdCount
--,SUM(COUNT(*)) OVER (PARTITION BY 1) / COUNT(*) OVER (PARTITION BY 1) +
--CASE
--WHEN ROW_NUMBER() OVER (PARTITION BY 1 ORDER BY NEWID()) <=
--SUM(COUNT(*)) OVER (PARTITION BY 1) % COUNT(*) OVER (PARTITION BY 1)
--THEN 1
--ELSE 0
--END as TargetNumOfRecords
,SUM(COUNT(*)) OVER (PARTITION BY 1) / COUNT(*) OVER (PARTITION BY 1) +
CASE
WHEN ROW_NUMBER() OVER (PARTITION BY 1 ORDER BY NEWID()) <=
SUM(COUNT(*)) OVER (PARTITION BY 1) % COUNT(*) OVER (PARTITION BY 1)
THEN 1
ELSE 0
END - COUNT(*) AS NumRecordsToUpdate
FROM
#Table
GROUP BY
RevId
)
, cteEndRowNumsToChange AS (
SELECT *
,SUM(CASE WHEN NumRecordsToUpdate > 1 THEN NumRecordsToUpdate ELSE 0 END)
OVER (PARTITION BY 1 ORDER BY RevId) AS ChangeEndRowNum
FROM
cteTargetNumbers
)
SELECT
*
,LAG(ChangeEndRowNum,1,0) OVER (PARTITION BY 1 ORDER BY RevId) as ChangeStartRowNum
INTO #RowNumsToChange
FROM
cteEndRowNumsToChange
;WITH cteOriginalTableRowNum AS (
SELECT
RevId
,ROW_NUMBER() OVER (PARTITION BY RevId ORDER BY (SELECT 0)) as RowNumByRevId
FROM
#Table t
)
, cteRecordsAllowedToChange AS (
SELECT
o.RevId
,o.RowNumByRevId
,ROW_NUMBER() OVER (PARTITION BY 1 ORDER BY (SELECT 0)) as ChangeRowNum
FROM
cteOriginalTableRowNum o
INNER JOIN #RowNumsToChange t
ON o.RevId = t.RevId
AND t.NumRecordsToUpdate < 0
AND o.RowNumByRevId <= ABS(t.NumRecordsToUpdate)
)
UPDATE o
SET RevId = u.RevId
FROM
cteOriginalTableRowNum o
INNER JOIN cteRecordsAllowedToChange c
ON o.RevId = c.RevId
AND o.RowNumByRevId = c.RowNumByRevId
INNER JOIN #RowNumsToChange u
ON c.ChangeRowNum > u.ChangeStartRowNum
AND c.ChangeRowNum <= u.ChangeEndRowNum
AND u.NumRecordsToUpdate > 0
IF OBJECT_ID('tempdb..#RowNumsToChange') IS NOT NULL
BEGIN
DROP TABLE #RowNumsToChange
END
/***************************** End of Main Method *******************************/
-- Compare the results and clean up
;WITH ctePostUpdateResults AS (
SELECT
RevId
,COUNT(*) as AfterChangeOccurences
FROM
#Table
GROUP BY
RevId
)
SELECT *
FROM
#StartingCounts s
INNER JOIN ctePostUpdateResults r
ON s.RevId = r.RevId
ORDER BY
s.RevId
IF OBJECT_ID('tempdb..#StartingCounts') IS NOT NULL
BEGIN
DROP TABLE #StartingCounts
END
Since you've given no rules for how you'd like the balance to operate we're left to speculate. Here's an approach that would find the most overrepresented value and then find an underrepresented value that can take on the entire overage.
I have no idea how optimal this is and it will probably run in an infinite loop without more logic.
declare #balance int = 125;
declare #cnt_over int;
declare #cnt_under int;
declare #revID_overrepresented varchar(32);
declare #revID_underrepresented varchar(32);
declare #rowcount int = 1;
while #rowcount > 0
begin
select top 1 #revID_overrepresented = RevID, #cnt_over = count(*)
from T
group by RevID
having count(*) > #balance
order by count(*) desc
select top 1 #revID_underrepresented = RevID, #cnt_under = count(*)
from T
group by RevID
having count(*) < #balance - #cnt_over
order by count(*) desc
update top #cnt_over - #balance T
set RevId = #revID_underrepresented
where RevId = #revID_overrepresented;
set #rowcount = ##rowcount;
end
The problem is I don't even know what you mean by balance...You say it needs to be evenly represented but it seems like you want it to be 125. 125 is not "even", it is just 125.
I can't tell what you are trying to do, but I'm guessing this is not really an SQL problem. But you can use SQL to help. Here is some helpful SQL for you. You can use this in your language of choice to solve the problem.
Find the rev values and their counts:
SELECT RevID, COUNT(*)
FROM MyTable
GROUP BY MyTable
Update #X rows (with RevID of value #RevID) to a new value #NewValue
UPDATE TOP #X FROM MyTable
SET RevID = #NewValue
WHERE RevID = #RevID
Using these two queries you should be able to apply your business rules (which you never specified) in a loop or whatever to change the data.
I wonder if someone knows how can I generate in Sql Server random values within a range uniformly distributed. This is what I did:
SELECT ID, AlgorithmType, AlgorithmID
FROM TEvaluateAlgorithm
I want AlgorithmID takes values from 0 to 15, has to be uniformly distributed
UPDATE TEA SET TEA.AlgorithmID = FLOOR(RAND(CONVERT(VARBINARY, NEWID()))*(16))
-- FROM TEvaluateAlgorithm TEA
I do not know what happen with the random, but is not distributing uniform random values between 0 and 15, not with the same amount.
For example from 0 to 9 is greater than from 10 to 15.
Thanks in advance!
EDITED:
Here is my data you can see the difference...
AlgorithmID COUNT(*)
0 22254
1 22651
2 22806
3 22736
4 22670
5 22368
6 22690
7 22736
8 22646
9 22536
10 14479
11 14787
12 14553
13 14546
14 14574
15 14722
rand() doesn't do a good job with this. Because you want integers, I would suggest the following:
select abs(checksum(newid()) % 16
I just checked this using:
select val, count(*)
from (select abs(checksum(newid()) % 16
from master..spt_values
) t
group by val
order by val;
and the distribution looks reasonable.
Here's a quick proof of concept.
Set #Loops to something big enough to make the statistics meaningful. 50k seems like a decent starting point.
Set #MinValue to the lowest integer in your set and set #TotalValues to how many integers you want in your set. 0 and 16 get you the 16 values [0-15], as noted in the question.
We're going to use a random function to cram 50k outputs into a temp table, then run some stats on it...
DECLARE #MinValue int
DECLARE #TotalValues int
SET #MinValue = 0
SET #TotalValues = 16
DECLARE #LoopCounter bigint
SET #LoopCounter = 0
DECLARE #Loops bigint
SET #Loops = 50000
CREATE TABLE #RandomValues
(
RandValue int
)
WHILE #LoopCounter < #Loops
BEGIN
INSERT INTO #RandomValues (RandValue) VALUES (FLOOR(RAND()*(#TotalValues-#MinValue)+#MinValue))
--you can plug into the right side of the above equation any other randomize formula you want to test
SET #LoopCounter = #LoopCounter + 1
END
--raw data query
SELECT
RandValue AS [Value],
COUNT(RandValue) AS [Occurrences],
((CONVERT(real, COUNT(RandValue))) / CONVERT(real, #Loops)) * 100.0 AS [Percentage]
FROM
#RandomValues
GROUP BY
RandValue
ORDER BY
RandValue ASC
--stats on your random query
SELECT
MIN([Percentage]) AS [Min %],
MAX([Percentage]) AS [Max %],
STDEV([Percentage]) AS [Standard Deviation]
FROM
(
SELECT
RandValue AS [Value],
COUNT(RandValue) AS [Occurrences],
((CONVERT(real, COUNT(RandValue))) / CONVERT(real, #Loops)) * 100.0 AS [Percentage]
FROM
#RandomValues
GROUP BY
RandValue
--ORDER BY
-- RandValue ASC
) DerivedRawData
DROP TABLE #RandomValues
Note that you can plug in any other randomizing formula into the right side of the INSERT statement within the WHILE loop then re-run to see if you like the results better. "Evenly distributed" is kinda subjective, but the standard deviation result is quantifiable and you can determine if it is acceptable or not.
I'm in SQL 2005 and I'm trying to convert this Cursor into something that isn't a Cursor to determine if this is the most efficient way to do this.
--Create cursor to determint total cost
DECLARE CostCursor CURSOR FAST_FORWARD
FOR SELECT ReceiptQty
,Price
FROM #temp_calculate
ORDER BY UpdateDate DESC
OPEN CostCursor
FETCH Next FROM CostCursor INTO #ReceiptQty,#Price
WHILE ##FETCH_STATUS = 0
BEGIN
IF #OnHandQty >= #ReceiptQty
BEGIN
--SELECT #ReceiptQty,#Price, 1,#OnHandQty
SET #Cost = #ReceiptQty * #Price
SET #OnHandQty = #OnHandQty - #ReceiptQty
SET #TotalCost = #TotalCost + #Cost
END
ELSE
BEGIN
IF #OnHandQty < #ReceiptQty
BEGIN
--SELECT #ReceiptQty,#Price, 2,#OnHandQty
SET #Cost = #OnHandQty * #Price
SET #OnHandQty = 0
SET #TotalCost = #TotalCost + #Cost
BREAK;
END
END
FETCH Next FROM CostCursor INTO #ReceiptQty,#Price
END
CLOSE CostCursor
DEALLOCATE CostCursor
The system needs to go through and use the newest recieved inventory and price to determine what the paid for the on-hand is.
Ex. 1st Iteration: #OnHandQty = 8 RecievedQty = 5 Price = 1 UpdateDate = 1/20 Results: #HandQty = 3 #TotalCost = $5
2nd Iteration: #OnHandQty = 3 RecievedQty = 6 Price = 2 UpdateDate = 1/10 Results: #HandQty = 0 #TotalCost = $11
The Final Results tell me that the inventory I have on hand I paid $11 for. If I was doing this in C# or any other Object Oriented langauge this screams Recursion to me. I thought about a Recursive CTE could be more efficient. I've only successfully done any Recursive CTE's for Heirarchy following types of Queries and I haven't been able to successfully wrap my head around a query that would achieve this another way.
Any help or a simple thats how it has to be would be appreciated.
Here's a recursive CTE solution. A row number column has to be present to make it work. So I derived a new temp table (#temp_calculate2) containing a row number column. Ideally, the row number column would be present in #temp_calculate, but I don't know enough about your situation as to whether or not you can modify the structure of #temp_calculate.
It turns out there are four basic ways to calculate a running total in SQL Server 2005 and later: via a join, a subquery, a recursive CTE, and a cursor. I ran across a blog entry by Jerry Nixon that demonstrates the first three. The results are quite stunning. A recursive CTE is almost unbelievably fast compared to the join and subquery solutions.
Unfortunately, he didn't include a cursor solution. I created one and ran it on my computer using his example data. The cursor solution is only a little slower than the recursive CTE - 413ms vs. 273ms.
I don't know how much memory a cursor solution uses compared to a recursive CTE. I'm not good enough with SQL Profiler to get that data, but I'd be curious to see how the two approaches compare regarding memory usage.
SET NOCOUNT OFF;
DECLARE #temp_calculate TABLE
(
ReceiptQty INT,
Price FLOAT,
UpdateDate DATETIME
);
INSERT INTO #temp_calculate (ReceiptQty, Price, UpdateDate) VALUES (5, 1.0, '2012-1-20');
INSERT INTO #temp_calculate (ReceiptQty, Price, UpdateDate) VALUES (6, 2.0, '2012-1-10');
INSERT INTO #temp_calculate (ReceiptQty, Price, UpdateDate) VALUES (4, 3.0, '2012-1-08');
DECLARE #temp_calculate2 TABLE
(
RowNumber INT PRIMARY KEY,
ReceiptQty INT,
Price FLOAT
);
INSERT INTO #temp_calculate2
SELECT
RowNumber = ROW_NUMBER() OVER(ORDER BY UpdateDate DESC),
ReceiptQty,
Price
FROM
#temp_calculate;
;WITH LineItemCosts (RowNumber, ReceiptQty, Price, RemainingQty, LineItemCost)
AS
(
SELECT
RowNumber,
ReceiptQty,
Price,
8, -- OnHandQty
ReceiptQty * Price
FROM
#temp_calculate2
WHERE
RowNumber = 1
UNION ALL
SELECT
T2.RowNumber,
T2.ReceiptQty,
T2.Price,
LIC.RemainingQty - LIC.ReceiptQty,
(LIC.RemainingQty - LIC.ReceiptQty) * T2.Price
FROM
LineItemCosts AS LIC
INNER JOIN #temp_calculate2 AS T2 ON LIC.RowNumber + 1 = T2.RowNumber
)
/* Swap these SELECT statements to get a view of
all of the data generated by the CTE. */
--SELECT * FROM LineItemCosts;
SELECT
TotalCost = SUM(LineItemCost)
FROM
LineItemCosts
WHERE
LineItemCost > 0
OPTION
(MAXRECURSION 10000);
Here is one thing you can try. Admittedly, this isn't the type of thing I have to deal with real world, but I stay away from cursors. I took your temp table #temp_calculate and added an ID ordered by UPDATEDATE. You could also add the fields you are wanting in your output to your temp table - #HandQty and #TotalCost as well as a new one called IndividulaCost - and run this one query and use it to UPDATE #HandQty and IndividulaCost . Run one more UPDATE after, taking the same concept used here to get and update the total cost. (In fact you may be able to use some of this on your insert to your temp table and eliminate a step.)
I don't think it is great, but I do believe it is better than a cursor. Play with it and see what you think.
DECLARE #OnHandQty int
set #OnHandQty = 8
SELECT a.ID,
RECEIPTQty + TOTALOFFSET AS CURRENTOFFSET,
TOTALOFFSET,
CASE WHEN #OnHandQty - (RECEIPTQty + TOTALOFFSET) > 0 THEN RECEIPTQTY * PRICE
ELSE (#OnHandQty - TOTALOFFSET) * Price END AS CALCPRICE,
CASE WHEN #OnHandQty - RECEIPTQTY - TOTALOFFSET > 0 THEN #OnHandQty - RECEIPTQTY - TOTALOFFSET
ELSE 0 END AS HandQuantity
FROM SO_temp_calculate a
CROSS APPLY ( SELECT ISNULL(SUM(ReceiptQty), 0) AS TOTALOFFSET
FROM SO_temp_calculate B where a.id > b.id
) X
RETURNS:
ID CURRENTOFFSET TOTALOFFSET CALCPRICE HandQuantity
----------------------------------------------------------------
1 5 0 5 3
2 11 5 6 0
If you were using SQL SERVER 2012 you could use RANK functions with OVER clause and ROWS UNBOUNDED PRECEDING. Until you go there, this is one way to deal with Sliding Aggregations.
CREATE CLUSTERED INDEX IDX_C_RawData_ProductID_UpdateDate ON #RawData (ProductID ASC , UpdateDate DESC , RowNumber ASC)
DECLARE #TotalCost Decimal(30,5)
DECLARE #OnHandQty Decimal(18,5)
DECLARE #PreviousProductID Int
UPDATE #RawData
SET #TotalCost = TotalCost = CASE
WHEN RowNumber > 1
AND #OnHandQty >= ReceiptQuantity THEN #TotalCost + (ReceiptQuantity * Price)
WHEN RowNumber > 1
AND #OnHandQty < ReceiptQuantity THEN #TotalCost + (#OnHandQty * Price)
WHEN RowNumber = 1
AND OnHand >= ReceiptQuantity THEN (ReceiptQuantity * Price)
WHEN RowNumber = 1
AND OnHand < ReceiptQuantity THEN (OnHand * Price)
END
,#OnHandQty = OnHandQty = CASE
WHEN RowNumber > 1
AND #OnHandQty >= ReceiptQuantity THEN #OnHandQty - ReceiptQuantity
WHEN RowNumber > 1
AND #OnHandQty < ReceiptQuantity THEN 0
WHEN RowNumber = 1
AND OnHand >= ReceiptQuantity THEN (OnHand - ReceiptQuantity)
WHEN RowNumber = 1
AND OnHand < ReceiptQuantity THEN 0
END/*,
#PreviousProductID = ProductID*/
FROM #RawData WITH (TABLOCKX)
OPTION (MAXDOP 1)
Welp, this was the solution I ended up coming up with. I like to think the fine folks watching the #sqlhelp hashtag for pointing me to this article by Jeff Moden:
http://www.sqlservercentral.com/articles/T-SQL/68467/
I did end up having to use a Rownumber on the table because it wasn't getting the first set of cases correctly. Using this construct I brought retrieiving the dataset down from 17 minutes, best I been able to do, to 12 Seconds on my vastly slower dev box. I'm confident production will lower that even more.
I've tested the output and I get the exact same results as the old way except for when 2 items for the same product have different price and the update time is the exact same. One way may pick a different order then the other. It of 15,624 items that only happened once where the varience was >= a penny.
Thanks everyone who answered here. I ultimately went a different way but I wouldn't have found it without you.
We have had a request to provide some data to an external company.
They require only a sample of data, simple right? wrong.
Here is their sampling criteria:
Total Number of records divided by 720 (required sample size) - this gives sampling interval (if result is a fraction, round down to next whole number).
Halve the sampling interval to get the starting point.
Return each record by adding on the sampling interval.
EXAMPLE:
10,000 Records - Sampling interval = 13 (10,000/720)
Starting Point = 6 (13/2 Rounded)
Return records 6, 19 (6+13), 32 (19+13), 45 (32+13) etc.....
Please can someone tell me how (if) something like this is possible in SQL.
If you have use of ROW_NUMBER(), then you can do this relatively easily.
SELECT
*
FROM
(
SELECT
ROW_NUMBER() OVER (ORDER BY a, b, c, d) AS record_id,
*
FROM
yourTable
)
AS data
WHERE
(record_id + 360) % 720 = 0
ROW_NUMBER() gives all your data a sequential identifier (this is important as the id field must both be unique and NOT have ANY gaps). It also defines the order you want the data in (ORDER BY a, b, c, d).
With that id, if you use Modulo (Often the % operator), you can test if the record is the 720th record, 1440th record, etc (because 720 % 720 = 0).
Then, if you offset your id value by 360, you can change the starting point of your result set.
EDIT
After re-reading the question, I see you don't want every 720th record, but uniformly selected 720 records.
As such, replace 720 with (SELECT COUNT(*) / 720 FROM yourTable)
And replace 360 with (SELECT (COUNT(*) / 720) / 2 FROM yourTable)
EDIT
Ignoring the rounding conditions will allow a result of exactly 720 records. This requires using non-integer values, and the result of the modulo being less than 1.
WHERE
(record_id + (SELECT COUNT(*) FROM yourTable) / 1440.0)
%
((SELECT COUNT(*) FROM yourTable) / 720.0)
<
1.0
declare #sample_size int, #starting_point int
select #sample_size = 200
select top (#sample_size) col1, col2, col3, col4
from (
select *, row_number() over (order by col1, col2) as row
from your_table
) t
where (row % ((select count(*) from your_table) / #sample_size)) - (select count(*) from your_table) / #sample_size / 2) = 0
It's going to work in SQL Server 2005+.
TOP (#variable) is used to limit rows (where condition because of integers rounding might not be enough, may return more rows then needed) and ROW_NUMBER() to number and order rows.
Working example: https://data.stackexchange.com/stackoverflow/query/62315/sql-data-sampling below code:
declare #tab table (id int identity(1,1), col1 varchar(3), col2 varchar(3))
declare #i int
set #i = 0
while #i <= 1000
begin
insert into #tab
select 'aaa', 'bbb'
set #i = #i+1
end
declare #sample_size int
select #sample_size = 123
select ((select count(*) from #tab) / #sample_size) as sample_interval
select top (#sample_size) *
from (
select *, row_number() over (order by col1, col2, id desc) as row
from #tab
) t
where (row % ((select count(*) from #tab) / #sample_size)) - ((select count(*) from #tab) / #sample_size / 2) = 0
SQL server has in-built function for it.
SELECT FirstName, LastName
FROM Person.Person
TABLESAMPLE (10 PERCENT) ;
You can use rank to get a row-number. The following code will create 10000 records in a table, then select the 6th, 19th, 32nd, etc, for a total of 769 rows.
CREATE TABLE Tbl (
Data varchar (255)
)
GO
DECLARE #i int
SET #i = 0
WHILE (#i < 10000)
BEGIN
INSERT INTO Tbl (Data) VALUES (CONVERT(varchar(255), NEWID()))
SET #i = #i + 1
END
GO
DECLARE #interval int
DECLARE #start int
DECLARE #total int
SELECT #total = COUNT(*),
#start = FLOOR(COUNT(*) / 720) / 2,
#interval = FLOOR(COUNT(*) / 720)
FROM Tbl
PRINT 'Start record: ' + CAST(#start as varchar(10))
PRINT 'Interval: ' + CAST(#interval as varchar(10))
SELECT rank, Data
FROM (
SELECT rank()
OVER (ORDER BY t.Data) as rank, t.Data AS Data
FROM Tbl t) q
WHERE ((rank + 1) + #start) % #interval = 0
I have a query that returns a large number of heavy rows.
When I transform this rows in a list of CustomObject I have a big memory peak, and this transformation is made by a custom dotnet framework that I can't modify.
I need to retrieve a less number of rows to do "the transform" in two passes and then avoid the memory peak.
How can I split the result of a query by half? I need to do it in DB layer. I thing to do a "Top count(*)/2" but how to get the other half?
Thank you!
If you have identity field in the table, select first even ids, then odd ones.
select * from Table where Id % 2 = 0
select * from Table where Id % 2 = 1
You should have roughly 50% rows in each set.
Here is another way to do it from(http://www.tek-tips.com/viewthread.cfm?qid=1280248&page=5). I think it's more efficient:
Declare #Rows Int
Declare #TopRows Int
Declare #BottomRows Int
Select #Rows = Count(*) From TableName
If #Rows % 2 = 1
Begin
Set #TopRows = #Rows / 2
Set #BottomRows = #TopRows + 1
End
Else
Begin
Set #TopRows = #Rows / 2
Set #BottomRows = #TopRows
End
Set RowCount #TopRows
Select * From TableName Order By DisplayOrder
Set RowCount #BottomRows
Select * From TableNameOrder By DisplayOrderDESC
--- old answer below ---
Is this a stored procedure call or dynamic sql? Can you use temp tables?
if so, something like this would work
select row_number() OVER(order by yourorderfield) as rowNumber, *
INTO #tmp
FROM dbo.yourtable
declare #rowCount int
SELECT #rowCount = count(1) from #tmp
SELECT * from #tmp where rowNumber <= #rowCount / 2
SELECT * from #tmp where rowNumber > #rowCount / 2
DROP TABLE #tmp
SELECT TOP 50 PERCENT WITH TIES ... ORDER BY SomeThing
then
SELECT TOP 50 PERCENT ... ORDER BY SomeThing DESC
However, unless you snapshot the data first, a row in the middle may slip through or be processed twice
I don't think you should do that in SQL, unless you will always have a possibility to have the same record 2 times.
I would do it in an "software" programming language, not SQL. Java, .NET, C++, etc...