I have a table with huge records in sql.(I have a procedure which displays top 100) How do i iterate through the whole records and display first 100 and then the next 100 and so on in a loop. ie
Process in batches of 100. An external query or process can execute this procedure in a loop
In C#, you could use the SqlDataReaderclass. With this, you can read through all records, row by row.
You can pass two parameters offset and limit to SP. Then keep limit to 100 and add the 100 in offset every time you execute the SP.
Starting value of offset is zero.
i want to get records from 0 - 100 first time, next time 101 - 200 . Im thinking of cursors to open whole records and then select top 100 where id deos not exist in the temp table and insert the unique id into temp table.
Related
I have a huge table in my database and a stored procedure accessing it, which needs pagination.
To achieve this I want total records of the table, and for that, I am facing performance issue because for doing that I need to run this query twice:
First time to get count for all records
Secondly when I need to select records in that page range
Is there any way I can avoid the first query for getting the total count instead of I can use row count or something else?
One way to do it would be something like this:
SELECT
(your list of columns),
COUNT(*) OVER ()
FROM
dbo.YourTable
ORDER BY
(whatever column you want to order by)
OFFSET x ROWS FETCH NEXT y ROWS ONLY;
With the OFFSET / FETCH, you retrieve only a page of data - and the COUNT(*) OVER() will give you the total count of the rows in the table - all in a single query
I am using sql server 2012. I have one table, contains 1.7 million records. I am just selecting all record using (select * from table_name). But it takes 1 Hour 2 minutes to fetch.
What should i do to fetch records quickly?
All you can do is limit your Result by Using
Top(100) Result or Where Clause Your Desired Data instead of
SELECT * FROM table
This will help you get only concerning and limited data in a less amount of time.
Another thing you can do is to Get Concerning columns only which gives you desired results.
This will significantly enhance the fetching time.
I need to get 10 random rows from table at each time, but rows shall never repeat when I repeat the query.
But if I get all rows it will repeat again from one, like table has 20 rows, at first time I get 10 random rows, 2nd time I will need to get remaining 10 rows and at my 3rd query I need to get 10 rows randomly.
Currently my query for getting 10 rows randomly:
SELECT TOP 10 *
FROM tablename
ORDER BY NEWID()
But MSDN suggest this query
SELECT TOp 10 * FROM Table1
WHERE (ABS(CAST(
(BINARY_CHECKSUM(*) *
RAND()) as int)) % 100) < 10
For good performance. But this query not return constant rows. Could you please suggest something on this
Since required outcome of your second query depends on the (random) outcome of the first query, the querying cannot be stateless. You'll need to store the state (info about the previous query/queries) somewhere, somehow.
The simplest solution would probably be storing the already-retrieved rows or their IDs in a temporary table and then querying ... where id not in (select id from temp_table) in the second query.
As Jiri Tousek said, each query that you run has to know what previous queries returned.
Instead of inserting the IDs of previously returned rows in a table and then checking that new result is not in that table yet, I'd simply add a column to the table with the random number that would define a new random order of rows.
You populate this column with random numbers once.
This will remember the random order of rows and make it stable, so all you need to remember between your queries is how many random rows you have requested so far. Then just fetch as many rows as needed starting from where you stopped in the previous query.
Add a column RandomNumber binary(8) to the table. You can choose a different size. 8 bytes should be enough.
Populate it with random numbers. Once.
UPDATE tablename
SET RandomNumber = CRYPT_GEN_RANDOM(8)
Create an index on RandomNumber column. Unique index. If it turns out that there are repeated random numbers (which is unlikely for 20,000 rows and random numbers 8 bytes long), then re-generate random numbers (run the UPDATE statement once again) until all of them are unique.
Request first 10 random rows:
SELECT TOP(10) *
FROM tablename
ORDER BY RandomNumber
As you process/use these 10 random rows remember the last used random number. The best way to do it depends on how you process these 10 random rows.
DECLARE #VarLastRandomNumber binary(8);
SET #VarLastRandomNumber = ...
-- the random number from the last row returned by the previous query
Request next 10 random rows:
SELECT TOP(10) *
FROM tablename
WHERE RandomNumber > #VarLastRandomNumber
ORDER BY RandomNumber
Process them and remember the last used random number.
Repeat. As a bonus you can request different number of random rows on each iteration (it doesn't have to be 10 each time).
what I would do is have two new fields, SELECTED (int) and TimesSelected (integer) then
UPDATE tablename SET SELECTED = 0;
WITH CTE AS (SELECT TOP 10 *
FROM tablename
ORDER BY TimesSelected ASC, NEWID())
UPDATE CTE SET SELECTED = 1, TimesSelected = TimesSelected + 1;
SELECT * from tablename WHERE SELECTED = 1;
so if you use that each time, once selected a record goes to the top of the pile, and records below it are selected randomly.
you might want to put an index on SELECTED and do
UPDATE tablename SET SELECTED = 0 WHERE SELECTED = 1; -- for performance
The most elegant solution, provided you do the consecutive queries within a certain amount of time, would be to use a cursor:
DECLARE rnd_cursor CURSOR FOR
SELECT col1, col2, ...
FROM tablename
ORDER BY NEWID();
OPEN rnd_cursor;
FETCH NEXT FROM rnd_cursor; -- Repeat ten times
Keep the cursor open and just keep fetching rows as you need them. Close the cursor when you're done:
CLOSE rnd_cursor;
DEALLOCATE rnd_cursor;
As for the second part of your question, once you fetched the last row, open a new cursor:
IF ##FETCH_STATUS <> 0
BEGIN
CLOSE rnd_cursor;
OPEN rnd_cursor;
END;
I have a sqlite3 database worth 4GB and 400k rows. The id column is a consecutive number starting at 1. If I query the following
select * from game where id = 1
After printing the first match the query continues until it reach the 400k row, thus taking a few seconds to finish the query.
How do I make the query stop at the first match?
Or how do I go directly to specific row since id and rowcount are the same?
Just add a LIMIT 1 to your query:
SELECT * FROM game WHERE id = 1 LIMIT 1;
I am planning to create a mysql 5 (myISAM) table that will contain x thousand rows of data. Each row will have a count field, & the rows with the 20 highest count values will be retrieved quite a lot, probably on a one-for-one ratio to every row update. The x thousand rows not in this 20 will not typically be retrieved.
What are the choices wrt choosing an efficient method of querying for the 20 top count values. Data does not have to be realtime live but it helps!
Add an index to the count field, then just a normal query:
SELECT * FROM myTable ORDER BY countField DESC LIMIT 20;
Create an index on the count field. Done.
Every five minutes, empty out and then copy the top 20 to a separate table. To get the top 20, just consult that table. Done!
(Or, you could spend the next few hours tweaking your query and parameters and indices and making it all "just right".)