A large amount of data (+-15 000 records) is loaded via AJAX which is then used to populate two arrays, one sorted by ID the other by a name. Both the arrays contains the same objects just ordered differently.
The data is displayed in a tree view but the hierarchy can be changed by the user. The tree is virtualised on the first level so only the records displayed + 50% is 'materialized' as vue components.
Supplying both of the arrays as data to a vue instance is very slow. I suspect vue.js is adding observers to the objects twice, or change notifications is sent multiple times, I don't really know.
So only one of the arrays is added to vue the other is used out of band.
Vue slows down the addition of elements to an array a lot. If the array is populated before it is bound to the vue instance it takes +-20s before the tree view is displayed. If I bind it before populating the arrays it takes about +-50s before the tree view becomes usable (the elements are displayed almost instantly). This could be because of notifications going for all these elements added.
Is there a way to add a second array with duplicate data so vue.js watches it for changes, but it doesn't slow vue down as much?
Is there a way to switch watching/notifications of temporarily so elements could be added to an array without the penalty, yet be 'watched' when notifications is switched back on?
I'm not sure that my reasoning behind the slowdowns is correct, so maybe my questions are misguided.
O another thing I need the arrays to be watched and only one of the properties of the elements.
var recordsById = [];
var recordsByName = [];
// addRecord gets called for every record AJAX returns, so +-15 000
// calling addRecord 15 000 times before 'binding' takes 20 sec (20 sec with no display)
// calling addRecord after 'binding' takes > 50 sec (instant display but CPU usage makes treeview unausable)
function addRecord(record) {
var pos = binarySearch(recordsById, record);
recordsById.splice(0, pos, record);
pos = binarySearch(recordsByName, record);
recordsByName.splice(0, pos, record);
}
var treeView = new Vue({
el: '#treeView',
data: {
// If I uncomment following line, vue becomes very slow, not just for initial loading, but as a whole
//recordsById: recordsById,
recordsByName: recordsByName
},
computed: {
virtualizedList: function() {.....}
}
})
There are a couple techniques which might improve your performance.
Using the keys
For rendering large lists, the first thing you want to do is use a key. A great candidate for this is the id you speak about.
Use pagination
An obvious solution to "displaying a lot of data is slow" is "display less data". Let's face it, 15 000 is a lot. Even if Vue itself could insert and update so many rows quickly, think of the user's browser: will it be able to keep up? Even if it was the simplest possible list of text nodes, it would still be a lot of nodes.
Use virtual scrolling technique
If you don't like the look of pagination, a more advanced approach would be virtual scrolling. As user browses this large list, dynamically add the elements below, and remove the ones that the user has already seen. You keep the total number of elements from the list at the DOM at once to minimum.
Related
The requirements:
100k lines
One of the columns is not text - its custom painted with wxDC*.
The items addition is coming from another thread using wxThreadEvent.
Up until now I used wxDataViewListCtrl, but it takes too long to AppendItem 100 thousand time.
wxListCtrl (in virtual mode) does not have the ability to use wxDC* - please correct me if I am wrong.
The only thing I can think of is using wxDataViewCtrl + wxDataViewModel. But I can't understand how to add items.
I looked at the samples (https://github.com/wxWidgets/wxWidgets/tree/WX_3_0_BRANCH/samples/dataview), too complex for me.
I cant understand them.
I looked at the wiki (https://wiki.wxwidgets.org/WxDataViewCtrl), also too complex for me.
Can somebody please provide a very simple example of a wxDataViewCtrl + wxDataViewModel with one string column and one wxDC* column.
Thanks in advance.
P.S.
Per #HajoKirchhoff's request in the comments, I am posting some code:
// This is called from Rust 100k times.
extern "C" void Add_line_to_data_view_list_control(unsigned int index,
const char* date,
const char* sha1) {
wxThreadEvent evt(wxEVT_THREAD, 44);
evt.SetPayload(ViewListLine{index, std::string(date), std::string(sha1)});
wxQueueEvent(g_this, evt.Clone());
}
void TreeWidget::Add_line_to_data_view_list_control(wxThreadEvent& event) {
ViewListLine view_list_line = event.GetPayload<ViewListLine>();
wxVector<wxVariant> item;
item.push_back(wxVariant(static_cast<int>(view_list_line.index)));
item.push_back(wxVariant(view_list_line.date));
item.push_back(wxVariant(view_list_line.sha1));
AppendItem(item);
}
Appending 100k items to a control will always be slow. That's because it requires moving 100k items from your storage to the controls storage. A much better way for this amount of data is to have a "virtual" list control or wxGrid. In both cases the data is not actually transferred to the control. Instead when painting occurs, a callback function will transfer only the data required to paint. So for a 100k list you will only have "activity" for the 20-30 lines that are visible.
With wxListCtrl see https://docs.wxwidgets.org/3.0/classwx_list_ctrl.html, specify the wxLC_VIRTUAL flag, call SetItemCount and then provide/override
OnGetItemText
OnGetItemImage
OnGetItemColumnImage
Downside: You can only draw items contained in a wxImageList, since the OnGetItemImage return indizes into the list. So you cannot draw arbitrary items using a wxDC. Since the human eye will be overwhelmed with 100k different images anyway, this is usually acceptable. You may have to provide 20/30 different images beforehand, but you'll have a fast, flexible list.
That said, it is possible to override the OnPaint function and use that wxDC to draw anything in the list. But that'll get difficult pretty soon.
So an alternative would be to use wxGrid, create a wxGridTableBase derived class that acts as a bridge between the grid and your actual 100k data and create wxGridCellRenderer derived classes to render the actual data onscreen. The wxGridCellRenderer class will get a wxDC. This will give you more flexibility but is also much more complex than using a virtual wxListCtrl.
The full example of doing what you want will inevitably be relatively complex. But if you decompose in simple parts, it's really not that difficult: you do need to define a custom model, but if your list is flat, this basically just means returning the value of the item at the N-th position, as you can trivially implement all model methods related to the tree structure. An example of such a model, although with multiple columns can be found in the sample, so you just need to simplify it to a one (or two) column version.
Next, you are going to need a custom renderer too, but this is not difficult neither and, again, there is an example of this in the sample too.
If you have any concrete questions, you should ask them, but it's going to be difficult to do much better than what the sample shows and it does already show exactly what you want to do.
Thank you every one who replied!
#Vz.'s words "If you have any concrete questions, you should ask them" got me thinking and I took another look at the samples of wxWidgets. The full code can be found here. Look at the following classes:
TreeDataViewModel
TreeWidget
TreeCustomRenderer
I have {data.word} component.
And, I have data about 100,000 words loading from JSON file.
When searching with this code below. If the result returns 10,000 words. It would take 4-5 seconds. Then, the just updated respectively.
for(let i = 0, l = data.length; i < l; ++i) {
if(data[i].word.toLowerCase().indexOf(inputText) === 0){
filteredWords.push(glossaries[i]);
}
}
My question is how can I load searching data and start rendering flatlist simultaneously or is there any better code to perform faster searching. (My current app, when I input searching word on searchbox, it takes 4-5 seconds to display the updated flatlist which looks awkward.
Thank you
100000 entries is quite a lot to search through, especially when you are doing it on a JS runtime on a mobile device that's also responsible for the rest of your app. A few optimisations I'd suggest:
If you just want to match on the start of the word (eg, "mat" will match "mattress" but "at" won't), look into a data structure like a Suffix Tree. Implementing this should dramatically speed up your search times.
I see you're converting all the words to lower case while you're looping through them, try to supply the word data with all the words already in lower case so you don't need to run this operation 100000 times with each keystroke.
Make sure you're only rendering once and aren't rerendering the list every time a match is found. Only update your component state when you have finished finding strings.
These tweaks should make your app a lot snappier.
I am implementing a SectionList showing a list of weeks which needs to scroll to the current week, thus I am using initialScrollIndex.
The problem I face is that I don't really understand what the index given to my _getItemLayout is?
Sometimes data[index] === undefined, which doesn't make sense to me.
I need to know which section it is (data[?]) as each section contains another SectionList for the events that week, thus height isn't a constant.
_getItemLayout = (data, index) => {
const rows = (data[index] === undefined) ? 1 : data[index].data.reduce((sum, value) => value.data.length + sum, 0);
const height = (rows * 94);
return {
length: height,
offset: height * index,
index
};
}
Its been long time, but for future readers thought to share some info.
There is a good article that explains the getItemLayout, please find it here
I also faced data[index] as undefined. The reason is that index is calculated considering section.data.length + 2 (1 for section header and 1 for section footer), you can find the code here (RN-52).
With SectionList we have to be very careful while processing index.
Similar to uxxi we ended up writing up our own implementation based off of react-native-get-item-layout but there is an important distinction to make.
Wiring either method to the SectionList getItemLayout parameter re-executes the same iterations over the data for every item being rendered. The overhead added by this is substantial and after correcting for the same there is a significant performance improvement.
Essentially the key is to calculate your offset data whenever the data provided to the component changes it's shape and then call against that calculated data to obtain the offset for getItemLayout. This will result in a single iteration per data change vs endless iterations per interaction with the list.
For some reason the react-native-get-item-layout package keeps crashing with "height: <<NaN>>" so I had to write my own RN SectionList getItemLayout . It uses the same interface as the former.
Like the package it's also an O(n).
I have an application where markers/features are loading into layers/layerGroups (what's the right term?) from multiple sources, and they're loading dynamically (based on some attribute in feature.properties and other conditions). I want to be able to tell on the sidepanel the number of markers currently loaded into the layer on display. Given just the layer's variable/identifier, how can one find the number of markers/features loaded into it?
var layer1= L.layerGroup();
layerControl.addOverlay(layer1, 'Layer 1');
... // loading stuff into this layer from different sources
console.log(layer1.length); // doesn't work, gives "undefined"
console.log(JSON.stringify(layer1)); // doesn't work, "TypeError: cyclic object value"
..so I guess layers can't be treated like JSON objects.
I found a related question, but the answer there only addresses markers loaded from one geoJson source and advises a simple counter++ in the onEachFeature. I'm working with a lot of layers in my application and would appreciate not having to put a separate counter variable for each and every one, rather want to just use the layer's variable/identifier to count. If we can add a layer to a map or clustergroup so simply then we ought to be able to count what's in it, right?
The getLayers() function returns an array containing all the features in your object. Then you can get the length of that array.
layer_variable.getLayers().length;
I'm running an update() method n times per second to "update" the keyboard input from the user so I can read it later in the logic part of the program. So I find two ways of implementing this in the SDL Docs and I'm not sure which one should I use.
1; Loop for all events using SDL_PollEvent searching for key down/up events and saving the key states in a map so I can check for each key state in the logic of the program.
Note: Alternatively, I can also use SDL_PeepEvents instead of SDL_PollEvent to take only the event types that matter; so, it would not "thrown away" the events on the queue.
std::map<int, bool> keyboard; // Saves the state(true=pressed; false=released) of each SDL_Key.
void update()
{
SDL_Event event;
while(SDL_PollEvent(&event))
{
switch(event.type)
{
case SDL_KEYDOWN:
keyboard[event.key.keysym.sym] = false;
break;
case SDL_KEYUP:
keyboard[event.key.keysym.sym] = true;
break;
}
}
}
2; Taking a snapshot from the keyboard each frame so I can read it easily.
Uint8* keyboard;
void update()
{
SDL_PumpEvents();
keyboard = SDL_GetKeyState(NULL);
}
With any of above implementations I can read keyboard just like this:
if (key_map[SDLK_Return]) printf("Return has been pressed.");
Also, is there another way to do so?
I prefer to do a variation of 1, where I fill three arrays, indicating not only the current state, but also which keys just went down and which keys just went up. This allows me to easily check for those events in code (without comparing to the previous snapshot), but, most importantly, it won't miss events that last less than a frame. For example, if your game is running at 10 fps due to a slow machine, the user might press and release an important key between two calls of your update routine, and then your system will never register it. This is extremely frustrating.
SDL also sends key events when the key is held down, which allow you to have multiple key down events for each key up. I find this particularly useful when implementing keyboard scrolling through a list of items, e.g. a keyboard-controlled menu.
You should use solution 2.
Why? As SDL_GetKeyState() docs point out, before using it you are expected to call SDL_PumpEvents() to update the state array.
When you are calling SDL_PollEvent(), it implicitly calls SDL_PumpEvents(). So, it basically updates the array for SDL_GetKeyState() anyway. By parsing these events manually, you just create a second array (well, actually a much slower map) holding the same information which SDL already collected for you.
So, I would dare say that first solution means doing the same thing twice. And if you ever decide to support things such as repeated keystrokes (SDL_EnableKeyRepeat()), you'll be reimplementing even a larger part of SDL.
I realize this question is quite old, but my answer could benefit someone. Personally, I use two arrays with SDL_GetKeyState. I store one array holding the current frame's keyboard state, and one array holding that last frame's keyboard state. (With some memcpy commands, it's really easy to update them.) Along with those two arrays, I have a map that converts strings like "A" to the SDL scancode values, but that is optional.
Then, when you need to check if something is released or pressed, you can combine the two arrays to check. (I made this a function.) For example, if you know that the key is pressed this frame, but wasn't pressed last frame, it was clearly just pressed this frame. if (currentFrame["A"] == true && lastFrame["A"] == false) {/*just pressed*/}
You would then do the opposite for the released. I find that method super easy to implement and use.