I am using Datatables JQuery plugin 1.9 with CakePHP 2.4.
My application is an online Database interface. The database is not relatively huge. We have only 26,000 records. However, I found out that even with recursive = -1, and the use of "Containable" behavior, CakePHP is still limited to find 5000 rows only, before I get a memory exhausted error, and it even takes 5 minutes to load, in the view!
Of course, using the LIMIT option was just experimental as I need to list/paginate/search through the entire database records.
My question is, have anyone built a CakePHP application that dealt with similar number of rows or larger?? How did you do it? Any detailed documentation reference about your approach would be greatly appreciated.
I've been looking for a week for setting up the server-side processing of Datatables in CakePHP and none of the solutions/plugins suggested out there worked (e.g. cnizzdotcom's). Increasing the memory limit (up to 1 GB!) didn't help much too.
Unfortunately, if this limitation continues this will be our last time using CakePHP. But for now, it's very critical to find a solution of the problem.
you really need to display your 26k items at once on your datatables??
I'd change it and use an ajax pagination. You could still keep your datatables pagination and make an async request to cakephp (cake should return a json), in this request you'll need to pass the limit and the page (you could also pass the sort and direction) as parameters
Search the datatable's doc for ajax sources
create a view for this ajax
in this view create paginate your items using the "page", "limit", "sort" and "direction" parameters passed to the view (via GET probablly)
this view should return a json in a format that datatables could understand
But if you really need to display the 26k at once.. then ignore this answer.
Hope this helps
Related
Is it possible to implement the view(the table) that socrata gives to users to implement in a mvc application? Especially the scroll effect where it takes only ms to load data.
If not can anyone suggest any lightweight grid for same (scroll with faster loading)
The scroll effect is making use of paging by loading data dynamically, rather than trying to load the entire dataset at once.
If you are using SODA.NET, the Resource object has a GetRows(limit, offset) method that you can use to retrieve paged sets of rows in the dataset.
There are any number of ways you could implement the scroll-to-load, including client-side AJAX requests. See this answer to a related question for a starting point.
I am working on enhancing the a search functionality of a website.
The current search is working as
1.reading all the rows from the database
2.find keywords from each rows and return the result.
The problem is it is too slow and it has to prepare all the data in the backend which mean read all the data from different database and put them to html.
The solution comes to my mind is:
show partial search results (like 10) which means as long as it find enough result in the databse it will stop reading and searching rows.
once user scroll down the page, using ajax to trigger another process of searching
My questions is:
Is it a good way(possible way) to do that?
Any tutorial source I should look up.
i know it is kinda abstract question, but I need advice for this.
Thanks in advance.
Update my research:
https://github.com/webcreate/infinite-ajax-scroll
this jquery lib can do the front end job
I have developed a web page for a database using catalyst and template toolkit. I have a table in my webpage. I want to have a search option in the same web page, which fetches data dynamically from the back end(database) and allows the user to search in the database.
I have jQuery datatables in my page, but it doesnot fetch the data from the database instead it will search in the table and give the result. As I am a new user to perl and catalyst, I request to tell me how can I have a search option in my webpage.
If my question is not clear,I would explain in detail.
What you're asking is not trivial to achieve, but can be done.
You need to first have a good read of the documentation on DataTables regarding server-side processing, and then rewrite the Catalyst Controller in your app that generates your data so that it mirrors the behaviour of Allan's PHP code: applying filters, sorts and limits, then returning JSON results. If you're using DBIx::Class and/or SQL::Abstract modules, it should be possible to create the equivalent functionality in far less code.
It's not for the faint-hearted, though. I've written and manage an application that uses Catalyst, TT and DT to render CRUD screens and I'm still not rushing to do a conversion to server-side processing until it proves absolutely necessary. (YAGNI principle + time-poor life.)
Is there a way to search and retrieve the results from On-Line Encyclopedia of Integer Sequences (http://oeis.org) programmatically?
I have searched their site and the results are always returned in html. They do not seem to provide an API but in the policy statement they say its acceptable to access the database programmatically. But how to do it without screen scraping?
Thanks a lot for your help.
The OEIS now provides several points of access, not just ones using their internal format. These seem largely undocumented, so here are all of the endpoints that I have found:
https://oeis.org/search?fmt=json&q=<sequenceTerm>&start=<itemToStartAt>
Returns a JSON formatted response of the results found from the sequenceTerm given. If too many results were returned, count will be > 0 whilst results will be null. If no results were returned, count will be 0. itemToStartAt is used for pagination of results, as only a maximum of 10 are ever returned. This starts at 0. If you wanted to return a second page of results, this would equal 10. Information about what each of the entries means can be found here.
https://oeis.org/search?fmt=text&q=<sequenceTerm>&start=<itemToStartAt>
Exactly the same arguments as before, however this returns it in the OEIS internal format. Which is largely written about here. Unless your project requires it, I'd highly recommend using the JSON format over this.
https://oeis.org/search?fmt=<json|text>&q=id:A<sequenceNumber>
Will return a single result if the sequenceNumber is found. This is the suggested method for obtaining single sequences, as it appears to be far more optimised than some of the alternative methods that can be used as queries. Requests often take under a second. Alternative search query methods can be found on this page.
https://oeis.org/A<sequenceNumber>/graph?png=1
This endpoint can be used to grab the images used to graph the data points. Alternatively, setting png to equal to zero returns the HTML page containing a graph of it.
https://oeis.org/recent.txt
This returns a list of recently updated entries in the OEIS internal format. There are no parameters available, or JSON format, as this seems like a static text file that is simply being served to the client. Due to the length of replies from the OEIS database (for some sequences replies can take above five seconds), I'd highly recommend heavily caching requests and using the above endpoint to update them when they change.
A URL of the form http://oeis.org/search?fmt=text&q=2,5,14,50,233 gives a nicely formatted text output.
But it seems there is no way to get a single sequence in text form.
If you happen to use Mathematica, it sounds like the following notebook might help. It allows you to specify a sequence and automatically import a detailed list of matching entries from the OEIS:
http://www.brotherstechnology.com/math/oeis_mathematica.html
It looks like direct use of their CGI program is the only API they provide.
URL for Searching the Database
https://oeis.org/search?q=id:A000032&fmt=text
gives the plain text form of an entry in their internal format
https://oeis.org/eishelp1.html
I have a Rails 3 application that uses jQuery DataTable to display a list of products that a user has drilled down to. However, in the case of a generic type of product, there may be over 3,000 different products to show. It takes about 30 seconds to render the page for such result sets, even though there are only 25 visible in the datatable (I understand that they ALL render, then datatable pages through the results).
My question is, I would like to only render the first page worth of results, and only call the others if the user clicks the next or previous buttons. Is that even possible?
All the processing i.e. pagination, sorting, etc. can be done serverside with this plugin. This example from the plugin docs shows how to handle the server side using php (sorry, I know you are using Rails) but it might give you an idea. They key is to set
"bServerSide": true
in the initialization of your datatable. Once you do that, you should be able to do whatever you like on the server side, instead of returning the entire dataset in one shot.
Note: Correct me if I am wrong but I assume you are using DataTables.
Much better than doing this server side is to set bDeferRender to true. This does just what it sounds like and only renders the rows it is displaying. We found that every browser later than IE6 has a good enough JS engine to handle many thousands of rows with that option enabled.