Thanks in advance for putting up with me.
Pulling a 33,000-record recordset from the database took LESS execution time than using Count() in the SQL and just grabbing 20 rows.
How is that possible?
A bit more detail:
Before, we were grabbing the entire recordset yet only displaying 20 rows of it on a page at a time for pagination. That was cringeworthy and wasteful, so I redesigned the page to only grab 20 rows at a time and to simply use an index variable to grab the next page, and so on.
All well and good, but that lacked a record count, which our people needed.
So after the record query, I added (what I thought would be) a quick query just on the index of the table using the Count(index) function in Structured Query Language.
A side by side comparison of the original page and my new page indicates my new page takes roughly 10% longer to execute than the original! I was flabbergasted. I thought for sure it would be lightning fast, way faster than the original.
Any thoughts on why and what I might do to remedy that?
Is it because the script has to run two queries, regardless of the data retrieved?
Update:
Here is the SQL.
(Table names and field names are fictionalized in this post for security, but the structure is the same as the real page).
The main recordset select query contains:
SELECT
top 21 roster_id, roster_pplid, roster_pplemailid, roster_emailid, roster_firstname,
roster_lastname, roster_since, roster_pplsubscrid, roster_firstppldone, roster_pmtcurrent,
roster_emailverified, roster_active, roster_selfcanceled, roster_deactreason
FROM roster
WHERE
roster_siteid = 22
AND roster_isdeleted = false
order by roster_id desc
The record count query contains:
SELECT
COUNT(roster_id)
FROM
roster
WHERE
roster_siteid = 22
AND roster_isdeleted = false
The first query runs, then the second. The second always dynamically has the same matching WHERE filter.
I think I know why it is slower, I'm using GetRows to grab the recordset in the new page, was not using that in the old page. That seems to be the slowdown. But I have to use it, cannot step beyond the 21st record otherwise.
Nick.McDermaid : The SQL shown is selecting the TOP 21 rows, that is how it is grabbing just 20 rows (number 21 is just to populate the index for the "Next" page link).
Related
I have a web application in which I get data from my database and show in a datatable. I am facing an issue doing this as the data that I am fetching has too many rows(200 000). So when I query something like select * from table_name;
my application gets stuck.
Is there a way to handle this problem with JavaScript?
I tried pagination but I cannot figure how would i do that as datatable creates pagination for already rendered data?
Is there a way through which I can run my query through pagination at
the backend?
I have come across the same problem when working with mongodb and angularjs. I used server side paging. Since you have huge number of records, You can try using the same approach.
Assuming a case that you are displaying 25 records in one page.
Backend:
Get the total count of the records using COUNT query.
select * from table_name LIMIT 25 OFFSET
${req.query.pageNumber*25} to query limited records based on the page number;
Frontend:
Instead of using datatable, display the data in HTML table it self.
Define buttons for next page and previous page.
Define global variable in the controller/js file for pageNumber.
Increment pageNumber by 1 when next page button is clicked and
decrement that by 1 when prev button is pressed.
use result from COUNT query to put upper limit to pageNumber
variable.(if 200 records are there limit will be 200/25=8).
So basically select * from table_name LIMIT 25 OFFSET
${req.query.pageNumber*25} will limit the number of records to 25. when req.query.pageNumber=1, it will offset first 25records and sends next 25 records. similarly if req.query.pageNumber=2, it will offset first 2*25 records and sends 51-75 records.
There are two ways to handle.
First way - Handling paging in client side
Get all data from database and apply custom paging.
Second way - Handling paging in server side
Every time you want to call in database and get records according to pagesize.
You can use LIMIT and OFFSET constraints for pagination in MySQL. I understand that at a time 2 lacs data makes performance slower. But as you mention that you have to use JS for that. So make it clear that if you wants js as frontend then it is not going to help you. But as you mention that you have a web application, If that application is on Node(as server) then I can suggest you the way, which can help you a lot.
use 2 variables, named var_pageNo and var_limit. Now use the row query of mysql as
select * form <tbl_name> LIMIT var_limit OFFSET (var_pageNo * var_limit);
Do code according to this query. Replace the variable with your desire values. This will make your performance faster, and will fetch the data as per your specified limit.
hope this will helpful.
I successfully deal with selection from SAP Tables by using RFC function module. Issue I'm facing is to understand best practice of selecting data from arbitrary row.
Example: First RFC call will fetch 1000 records from KNA1 (I will log in custom transparent table how many records in total where considered so far).
New RFC Call should take next 1000 rows but starting from row 1000 till 2000. Is there some elegant way of dealing with this situation?
Using of Cursor is not possible since there are 2 consecutive calls of same RFC cursor value will be reset.
Otherwise I should always selecting everything and distinguish requested data by looping the total data which will consume a lot of time.
Thanks for any suggestions!
Use OFFSET
In the SELECT with OFFSET:
SELECT * FROM kna1
UP TO 1000 ROWS
OFFSET (lv_offset)
WHERE ...
ORDER BY ...
If lv_offset contains 2000 for example, it will return the rows 2001-3000 by the ordering.
According to the online help, you have to use ORDER BY in the SELECT.
This one has got me stumped but good. Using VBA in MS Access, I sometimes get different results when running the same code against the same table. I can run the code 2, 5, 6, 10 times and get the same results, then run it again and get a different result. I can run the code twice and get the same results and then I can run the code twice and get different results - all with the same code against the same table.
The code is used to group trips so they can be billed correctly. I do this by taking the raw SQL data and putting it into an Access table, then via several sorts and some cross-checking, I label each trip in the access table with a GR or an ML in the last field of the table. The result set is all trips for the specified time frame which are now labeled: ML (multi-loaded), GR (Grouped) or blank (demand).
I have even tried putting in MoveLast/MoveFirst to make sure the table is fully loaded each time (per suggestion from others).
Here is a link to the code and data after 2 runs of the same code on the same data:
Code&DataI removed the trip ID and client ID data for privacy concerns. The trip ID is unique but the client id will be used many times depending on how many trips the client took during the time period.Any and all help you can give to make this code produce the same results each time it is run is GREATLY appreciated. I don't want to have to go back to doing this report labeling by hand. This is the smallest of 4 that must be done twice a month.Thanks!David R. Mohr.................................................end of line........................................................................
When opening t_BillableTrips, I do not think it is safe to assume that the data will be sorted the way you want. That could potentially change from run to run. I would suggest to use a query with an explicit sort order instead of opening the table directly. My second suggestion is to use the Recordset Clone method to get Intable2 and Intable3. The recordsets will be sharing the same underlying in memory data but will be able to be positioned at different records.
First of all, Thanks to everyone that gave an answer. Your prompts guided me to find the solution.
1st problem is that I was directly editing the TABLE while being under the impression that my MAKE TABLE ORDER BY command was actually creating a table in the order I specified - it only worked most of the time and we can't have that.
So, after digging deeper I found more and more evidence that trying to sort the actual table - especially with a MAKE TABLE command is not good practice and can give unpredictable results as well as generates a lot more overhead. I am now basing my positioning and updating on a QUERY of the table and not the actual table. I.E. changed this:
Set InTable = dbsBilling.OpenRecordset("t_BillableTrips", dbOpenTable)
Set InTable2 = dbsBilling.OpenRecordset("t_BillableTrips", dbOpenTable)
Set InTable3 = dbsBilling.OpenRecordset("t_BillableTrips", dbOpenTable)
Set InTable4 = dbsBilling.OpenRecordset("t_BillableTrips", dbOpenTable)
to this:
Set InTable = dbsBilling.OpenRecordset("q_BillableTripsSort1B", dbOpenDynaset)
Set InTable2 = dbsBilling.OpenRecordset("q_BillableTripsSort1B", dbOpenDynaset)
Set InTable3 = dbsBilling.OpenRecordset("q_BillableTripsSort1B", dbOpenDynaset)
Set InTable4 = dbsBilling.OpenRecordset("q_BillableTripsSort1B", dbOpenDynaset)
So far, this seems to have fixed the problem and, of course, the proc runs much faster since it does not have to create the table twice to run/update for two different sorts.
I have a table called Photos. It has a little over 3000 rows and includes an image type field called Photo.
This query runs instantaneously:
Select PhotoFileName, DATALENGTH(Photo)
From Photos
Order by DATALENGTH(Photo)
This query hangs intermittently (sometimes takes several minutes to complete, then after completing once, runs instantaneously).
Select PhotoFileName, DATALENGTH(Photo)
From Photos
Where DATALENGTH(Photo)>0
Same with this query:
Select PhotoFileName, DATALENGTH(Photo)
From Photos
Where Photo is not NULL
What could possibly be going on?
I'm not sure why you're seeing this problem, but I think you could resolve it by adding a computed column to the table to hold the Length of the photo, perhaps tied with an index on that new column.
I am having a problem trying to achieve the following:
I'd like to have a page with 'infinite' scrolling functionality and all the results fetched to be sorted by certain attributes. The way the code currently works is, it places the query, sorts the results, and displays them. The problem is, that once the user reaches the bottom of the page and new query is placed, the results from this query are sorted, but in its own context. That is, if you have a total of 100 results, and the first query display only 50, then they are sorted. But the next query (for the next 50) sorts the results only based on these 50 results, not based on the 100 (total results).
So, do I have to fetch all the results at once, sort them, and then apply some pagination logic to them or there's a way for MongoDB to actually have infinite scrolling (AJAX requests) with sorting applying to the results?
There's a few ways to do this with MongoDB. You can use the .skip() and .limit() commands (documented here: http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-CursorMethods) to apply pagination to the query.
Alternatively, you could add a clause to your query like: {sorted_field : {$gt : <value from last record>}}. In other words, filter out matches of the query whose sorted value is less than that of the last resulting item from the current page of results. For example, if page 1 of results returns documents A through D, then to retrieve the next page 2 you repeat the same query with the additional filter x > D.
Let me preface this by saying that I have no experience with MongoDB (though I am aware that it is a NoSQL database).
This question, however, is somewhat of a general database one (you'd probably get more responses tagging it as such). I've implemented such a feature using Cassandra (another, albiet quite different NoSQL database), however the same principles apply.
Use the sorted-by attribute of the last retrieved record, and conduct a range search based on it in the database. So, assuming your database consists of the following set of letters:
A
B
C
D
E
F
G
..and you were retrieving 2 letters at a time, you'd retrieve A, B first. When more records are needed, you'd use B to conduct a range search on the set of letters in the database. In plain English this would be something like:
Get the letters that appear after B, limit the results to 2
From a brief look at the MongoDB tutorial, it looks like you have conditional operators to help you implement this.