I am making an application in vb.net that involves listing a large number of files in a listview. However, after 30-40 000 items this gets rather slow and resource demanding. I need a better way to present a large number of items to users. I have though of using pages much like google search results showing 1000 items per page... I have also though of switching between sets of items every time the user hits min and max scrolling position.
What would be the best slution to handeling and presenting large numbers of items in listview? thanks.
Showing large amount of items in a listview is always not a good solution as it cause performance and memory issues. Also the user could not see all the items in a single scroll. So always choose to show items with less amount of data. But some requirements may force to show large number of items. VirtualizationPanel and VirtualizationStackPanel is a solution to improve the UI responsiveness and performance by recycling and loading items based on the scrolling like Windows Explorer.
Please see the below links will helps you to solve your problems
http://msdn.microsoft.com/en-us/library/system.windows.controls.virtualizingstackpanel.aspx
http://msdn.microsoft.com/en-IN/library/system.windows.controls.virtualizingpanel(v=vs.90).aspx
Related
I'm creating a client dashboard website that displays many different graphs and charts of different views of data in our database.
The data is of records of medical patients and companies that they work for for insurance purposes. The data is displayed as aggregate charts but there is a filter feature on the page that the user can use to filter individual patient records. The fields that they can filter by are
Date range of the medical claim
Relationship to the insurance holder
Sex
Employer groups (user selects a number of different groups they work with, and can turn them on and off in the filter)
User Lists (the user of the site can create arbitrary lists of patients and save their IDs and edit them later). Either none, one, or multiple lists can be selected. There is also an any/all selector if multiple are chosen.
A set of filters that the user can define (with preset defaults) from other, more internally structured pieces of data. The user can customize up to three of them and can select any one, or none of them, and they return a list of patient IDs that is stored in memory until they're changed.
The problem is that loading the data can take a long time, some pages taking from 30 seconds to a minute to load (the page is loaded first and the data is then download as JSON via an ajax function while a loading spinner is displayed). Some of the stored procedures we use are very complex, requiring multiple levels of nested queries. I've tried using the Query Analyzer to simplify them, but we've made all the recommended changes and it still takes a long time. Our database people have looked and don't see any other way to make the queries simpler while still getting the data that we need.
The way it's set up now, only changes to the date range and the employer groups cause the database to be hit again. The database never filters on any of the other fields. Any other changes to the filter selection are made on the front end. I tried changing the way it worked and sending all the fields to the back end for the database to filter on, and it ended up taking even longer, not to mention having to wait on every change instead of just a couple.
We're using MS SQL 2014 (SP1). My question is, what are our options for speeding things up? Even if it means completely changing the way our data is stored?
You don't provide any specifics - so this is pretty generic.
Speed up your queries - this is the best, easiest, least error-prone option. Modern hardware can cope with huge datasets and still provide sub-second responses. Post your queries, DDL, sample data and EXPLAINs to Stack Overflow - it's very likely you can get significant improvements.
Buy better hardware - if you really can't speed up the queries, figure out what the bottleneck is, and buy better hardware. It's so cheap these days that maxing out on SSDs, RAM and CPU will probably cost less than the time it takes to figure out how to deal with the less optimal routes below.
Caching - rather than going back to the database for everything, use a cache. Figure out how "up to date" your dashboards need to be, and how unique the data is, and cache query results if at all possible. Many development frameworks have first-class support for caching. The problem with caching is that it makes debugging hard - if a user reports a bug, are they looking at cached data? If so, is that cache stale - is it a bug in the data, or in the caching?
Pre-compute if caching is not feasible, you can pre-compute data. For instance, when you create a new patient record, you could update the reports for "patient by sex", "patient by date", "patience by insurance co" etc. This creates a lot of work - and even more opportunity for bugs.
De-normalize - this is the nuclear option. Denormalization typically improves reporting speed at the expense of write speed, and at the expense of introducing lots of opportunities for bugs.
I'm working on a very high throughput site with many items, am looking into implementing "trending now" type functionality, that would allow users to quickly get a prioritized list of the top N items that have been viewed recently by many people, that gradually fade away as they get fewer views.
One idea about how to do this is to give more weight to recent views of an item, something like a weight of 16 for every view of an item the past 15 minutes, a weight of 8 for every view of an item in the past 1 hour, a weight of 4 for things in the past 4 hours, etc but I do not know if this is the right way to approach it.
I'd like to do this in Redis, we've had good success with Redis in the past for other projects.
What is the best way to do this, both technologically and the determination of what is trending?
The first answer hints at a solution but I'm looking for more detail -- starting a bounty.
These are both decent ideas, but not quite detailed enough. One got half the bounty but leaving the question open.
So, I would start with a basic time ordering (zset of item_id scored by timestamp, for example), and then float things up based on interactions. So you might decided that a single interaction is worth 10 minutes of 'freshness', so each interaction adds that much time to the score of the relevant item. If all interactions are valued equally, you can do this with one zset and just increment the scores as interactions occur.
If you want to have some kind of back-off, say, scoring by the square root of the interaction count instead of the interaction count directly, you could build a second zset with your score for interactions, and use zunionstore to combine this with your timestamp index. For this, you'll probably want to pull out the existing score, do some math on it and put a new score over it (zadd will let you overwrite a score)
The zunionstore is potentially expensive, and for sufficiently large sets even the zadd/zincrby gets expensive. To this end, you might want to keep only the N highest scoring items, for N=10,000 say, depending on your application needs.
These two links are very helpful:
http://stdout.heyzap.com/2013/04/08/surfacing-interesting-content/
http://word.bitly.com/post/41284219720/forget-table
The Reddit Ranking algorithm does a pretty good job of what you describe. A good write up here that talks through how it works.
https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9
consider an ordered set with the number of views as the scores. whenever an item is accessed, increment its score (http://redis.io/commands/zincrby). this way you can get top items out of the set ordered by scores.
you will need to "fade" the items too, maybe with an external process that would decrement the scores.
I am working on creating an Iphone application. It will contain over 100k+ records and that number might grow with time to even 500k records. Each record will not be big. It will contain few fields with text in them. At some point the application will need to show a large subset in an uitableview. Might even be 1k records in the view. I understand that people will not want to scroll through 1k records so i am not sure how i will design it so it will make sense. Might implement some kind of search which will reload view with every letter a person pressed.
My question is which is better to use for my purpose. Core data or sqlite. Which will give me the ability to scroll through the data with out loading 1k dataset into memory and for user be able to quickly scroll through the records.
Thank You in advance!
Yan
Use CoreData and NSFetchedResultsController. It handles large resources efficient.
I am writing an application that can potentially have hundreds of custom user controls on a canvas. All of them can be dragged around the screen, and there are lines connecting all of then to at least one other user control. Once you reach 250 custom user controls or so performance starts to markedly decline, with frame rates going to 10 or below.
Are there any best practices or ideas on how to optimize performance in this sort of situation? The problem seems to be in the sheer number of controls on the canvas at any one time.
Thanks!
~Steve
Unfortunately no...it becomes a system resourcing issue.
One way to alleaviate this issue is using virtualization in some form or another as was done with the VirtualizingStackPanel. This boils down to removing items from the visual tree and holding a reference to the needed item once it needs to become visible. Not forcing the item to render will significantly reduce the needed system resources. If all the items need to be rendered at a given time it becomes a system resourcing issue at that point. Keep the control as small as possible and keep things such as anotmation to a minimum as they are taxing on the system.
I'm wondering how big of a data can UITableViewController handle... I understand that most of the time we use arrays to represent data in each of the table view's cell.
Suppose that the array contains a fair amount of strings and some kind of integer (as an id for that row, to be passed when you select that row). Then each element of the array is something like 4 bytes for the string and 4 bytes for the integer. I'm not sure though how to calculate the maximum elements we can have in a single tableview.
However this leads to another question: even if the iOS device can handle very big data (say 2000-3000 rows), is it a good design decision to present user with such large amount of data? In Apple's Contacts app, they present data of so many contacts all at once while giving the A-Z quick jump on the right hand side. In the Twitter app, users are presented with hundreds of tweets (although cut off in the middle if it's older than a day).
So which way is best when presenting user with such large amount of data?
You don't need to keep all the data in memory at once. Sure, an array is the easiest way of doing it but if you're going to have many rows (which I guess you are because you asked this question!) then you really need to be caching the recently used values.
But that's not the only thing you need to consider.
When the table is first displayed it first asks the delegate how many sections and then it asks every section how many rows it has (even if they're not going to be displayed initially). This means that the more sections you have, the longer the table will take to display.
One of the big changes I made between version 1.x and 2.x of my app was to reduce the number of sections displayed on the main screen. I also stored meta-data -- such as the number of rows in each section -- so that I wouldn't need to calculate them each time the view became visible. This meant it could deal with far more rows and was much quicker even when the volume increased
UITableView can handle a huge number items because it only requires the data that is being displayed. However, the limitation of the device means that you may not be able to store all data in memory. If this is the case you can use Core Data to keep the amount of data stored in memory to a minimum. NSFetchedResultsControlleris your friend.
As far as the UI is concerned I think the index bar displayed on the right of the table can make large data sets possible. However, lets take an example of 3000 items. If we have 25 index points (eg A, B, C, D etc), then this would be 120 items between index points. I would advise against using an strict alphabetical index in this case because the distribution would be very uneven. I would parse the data and have the first 2 or 3 letters of each 120th item. With this in mind I would guess that 3000 wouldn't be a great UX.
I think that for large amout of data, all data are not in an array in RAM, but in a base on flash drive (core data or sqlite).
Just data for visible rows are in memory.