How to traverse a tree of MP_Node (django-treebeard) categories and display effectively with lease amount of queries? I tried looking the docs but I see the queries number increasing with more categories.
Is there a method to limit the number of queries to display a menu like amazon.com and get all the categories in an optimized manner?
I see that dump_bulk() api in treebeard gets all the categories in a single query. Is it advisable to use it? If not why? Where is its practical usage?
A sample code using twitter-bootstrap nav menu would be appreciated.
I'm looking to reduce the number of queries. Answer with explanation with least number of queries will be accepted.
I chose django-mptt in lieu of Treebeard. It's much more simple, and uses only one tree management methodology (MPTT, hence the name). See this function (disclaimer, I submitted a patch which rewrote this function to be more optimized) - it caches an entire tree below a given node, allowing you to go up and down the tree as much as you want (e.g., node.get_children(), node.parent, etc.), without running any more queries. In other words, it's ideal for doing exactly what you're wanting to do.
Related
Hello World,
I'm in research mode for one of feature to be built in our software and there one new thing that we have never faced.
The thing is, on one form we have a drop down with list of items. User can select default which means all items needs to be considered or else he can selectively opt for certain list items.
Actually the form is related to filter functionality depending upon users input the data is going to get filtered and displayed on UI.
The main problem we are trying to solve is suppose user selects default, which means all list items ID's are gonna be considered in POST call of API. The list can be huge, say 1 to 1K and above too.
So under such circumstances we can build the query string but, it seems its gonna be so huge. I have also studied that certain browsers support limited query string as per their standard limits.
So currently I have following doubts in mind.
Will shortening of query string work here ?
By which technique it can be handled efficiently ?
What performance considerations I need to take care during during so ?
Any suggestions or thoughts are welcome. That would boast my software design thinking.
based on what I understand from your question, here is my opinion:[if I understood wrong, please correct me, so I can help you]
You need to send query in URL and not in body or using JSON!is that correct?
I think you don't need to send every one of the selected items one by one!
If there are selected in serial, you can perform a range in your query!
Like http://abcd/test?id=1-43,6-765(take ID as string and then export the useful data in back-end) with this approach, you can shorten your query!
And also think about the database too (if there is any).querying this much data is use a lot of IO and make query low performance.
Suppose I am to search against two types [cars] and [buildings], and I would want the results to be separated. Is there a way one can group results by types?
I understand one simple way will be to query each types separately, but for other use cases one may actually need to query tens or hundreds of types together. Is there a native way or hacky way(like using sort) to achieve this?
This type of grouping behavior is (currently) not available in elasticsearch. It has been a long standing request:
https://github.com/elasticsearch/elasticsearch/issues/256
There are two approaches that can help, both of which are far from perfect, but may be good enough for some use cases.
Client side aggregation. Request a lot more results than you plan on displaying and the then bucket those.
Using multi-query. This allows you to easily pass down some number of queries in a single batch, but will have potential scaling problems if the number of queries gets to large.
This is one feature that Solr has that elasticsearch doesn't, but I have never tried it. I used a similar feature with Autonomy IDOL years back, but the performance was abysmal.
If you want the results separated in groups of documents, you're going to have to restructure your documents, since, elasticsearch is focused on finding matching documents. You might get around this by designing a document that has child documents then you can query for matches on the parent document that represents your type.
I guess there might be some common field (let's say it's [price]) if you want to search against different types. Then it would be reasonable to add some different type like [price_aggregator] and put into it fields [type] and [price]. And then you could easily build your query against just one type. This requires some additional work while indexing and more memory to store index but it's much performant when you search.
Maybe this is a shot in the dark here but I'm trying to find out if anyone has thoughts on this problem we have been presented with.
The situation is that we have a database that contains all kinds of data about a large list of projects. There are dozens of tables that all provide supporting info about a project, both in 1 to 1 manner, where some specific type of info about projects (say ProjectInfoTypeA) might be stored in a table called ProjectInfoTypeA, and we'd do a inner join between that and the projects table, as well as 1 to many, like maybe ProjectScopeKeywords, where a project can be assigned N attributes or in this case "keywords" for a number of different attribute/lookup tables.
In the end we need to have the user in our web app build up things like:
Show me all projects completed in the last 5 years that took at least 4 years to do, cost at least $1MM, and have all 3 of these keywords ( x,y,z ) associated with it.
We also want users to be able to save their queries so they, and other users, can select them from a list of saved queries.
Once we get the list of projects from their filter, we need to then work with it in all different ways: but not as a report. If this were a report I'd just give them some report builder of some kind, but we need to work with their filtered list in the web app.
Currently we are thinking of 2 different ideas:
1) being that we just try to write our own UI for building up the query, and then create some giant SQL statement.
2) we store the data about each of their filters in the database, and then when they slick "Search" we would essentially prune down the list of projects by iteratively stripping off the projects that didn't match each query, based on the data they stored in the database.
I'm guessing no one out there has had to deal with something like this, but if any of you had, I'd be interested to hear any suggestions/patterns that would be worth looking into.
I would recommend choosing option 1. I have used a query-builder approach on a number of projects, with varying degrees of sophistication depending on the complexity of the requirements.
If you are in a position to use a ready-made solution, you can find several on the web: http://www.google.com/search?q=sql+query+builder
For a custom built solution, at a minimum, you would probably want to provide flattened views for the user to query from; this will simplify the designer complexity, reduce the learning curve for the user, and provide some abstraction against future schema changes.
After defining your base data sources, you need to provide means by which the user can select specific columns, define filter criteria, specify value aggregation, and define sub-queries (based on your example query requirement). The column selection and filter definition should not be too difficult, but the value aggregation and sub-query creation would be non-trivial to define. You should be able to use the ready-made solutions as examples of how to present this functionality to the user.
My members will have the ability to customise their profile page with X amount of widgets, each widget displays different data such as a list of music, list of people they are following etc.
Several widgets include:
- List of media they have uploaded
- List of people they are following
- List of people following them
- Html/Text widget
- Media Statistics (num downloads etc)
- Comments widget for other members to leave comments
Some widgets will have to page the data returned because there could be hundreds of results.
I haven't done any optimisation at the moment so it is doing lots of DB work to return all the data...what would be the most efficient way to retrieve the data...would 1 DB call per widget be acceptable? There could be around 5-20 widgets per page.
If you need more information about my situation please feel free to ask.
Paul
Short answer: It depends.
Start off from the unoptimised state, then use SQL profiler or a C# profiler like dotTrace to work out the best places to make improvements. Set a realistic goal to work towards (e.g. 'less than 800 milliseconds to load the page').
Generally I find performance starts to suffer after about 20-30 database calls in a request, but this is going to depend on your server, the location of the database etc.
There are many things that you can try: pre-caching, eager fetch using joins rather than selects etc. Nothing is going to guarantee better performance though unless it is applied intelligently.
For a page with lots of widgets, a common design pattern is to load each widget asynchronously using AJAX, rather than loading the entire page in one go.
since you've cut out out your work to widgets the proper thing to do would be for each widget to do a single query for all its required functionality. This would also be the case even if you retrieved widgets via AJAX (which as cbp noted is not a bad idea).
Secondly, i would set up some kind of mechanism for each widget to register its existence and then after all widgets have registered then i would fire a single query that would include all widget queries. (technically its again multiple queries but in a single round-trip, see MulriCriteria and MultiQueries in NH reference).
Also do not forget that lazy loads are hidden db retrievals and you could have a huge performance impact by using lazy load in a situation where an eager load is proper (for example Foo.Bar.Name where you always show the Bar.Name value when you present the Foo entity)
Performance degradation can occur even with less that 20-30 database call per request, but it depends on the size and complexity of your entities, queries, filters as well as the size of the data sets retrieved.
I have a couple specific needs for my search and I'm interested to get people's opinions on what search approach makes the most sense. Based on my explanation below, would you recommend that I use basic sql queries? Or step up to a more advanced search solution, like Sphinx?
I have two models that I want to search in: products and varieties.
product has_many :varieties
variety belongs_to :product
I need my search to recognize the relationship between products and varieties. However, varieties do not have their own existence on the site. So, when a user searches for a variety that's in the system, I need the search to return the corresponding product page on which the variety resides.
For example, let's say that the product is ball and the variety is bouncy. If a user searches for 'bouncy', I want the search to return the ball/show view.
The other tweak involves the results. If there's only one result for a given search, I want to render the product/show page. However, if there are multiple results, I want to render the product/index page, displaying the multiple results. My dataset is a pretty limited universe, so I think it's going to be fairly common that we have only one result.
Those are my requirements. Can I satisfy these requirements with standard sql queries and conditions? Or would you recommend a more advanced search approach?
Thanks!
Either solution will satisfy your requirement, but you can satisfy it with standard SQL queries only if your dataset is small. In that case, a DB index on the searched queries is important. You could take a look at scoped_search which I've used for small projects and gets the job done.
If you have a big dataset and plain SQL queries slow you down, sphinx (and thinking-sphinx) is the way to go. The only disadvantage of this approach is having to monitor and maintain another daemon, although it is very stable and lightweight. This solution is also very easy to implement, and there's a good community around thinking-sphinx.
Lastly, you may consider your database's full text search capabilities. If you are using PostgreSQL, tsearch is a great solution because it is very fast and built into your database process. There are a couple of Rails plugins for interacting with it: acts-as-tsearch and tsearchable. Try them out and see which one feels better to you.