I have 50K+ records in my db,I want to add search filter without affecting performance of the application .Please suggest me what approach I should adopt to search the large data.
I am calling api to fetch data from server. And used react native search functionality. But due to large data i have implemented pagination at server side so each time new api being called and new data fetched from server. Now issue is it will search only from fetched page records and i want to search from all 50K+ records. And i want to search for each character typed. So I think it is not fissible to call api at each character typed.
So what is the best approch?
I have a quote finder app which holds 400k quotes on mongodb and I am using Node JS as backend. In my point of view if you are going to search more than 100 items in your front end lets say you are going to use flatlist you can create your searching algoritms in front-end side, than you can dynamically render your list according to search results. 100 items are not a specific limit it is just my idea. Because in lists more data than that would look ugly.
For 50k search you definitely have to come up with search algorithms on server-side. After you get your search data you can use
https://github.com/UnPourTous/react-native-search-list
And if your specific aim is to search on server-side I would recommend elastic-search.
But for 50k data it is better you implement your own algoritms. When you send fetch request let your server run the search and in response get the data you want.
You can use redux for this situation. When you start your application get all records(50+K) from server it will take time(You can get records in splashscreen) based on your server and store all records in redux store. Now you can search data from your redux store so don't need to call API on every search.
Make sure your server send only those data in response which you required to show in mobile application. So for getting records it will reduce response time.
You can use redux-search for this.
Related
I'm designing a website with a REST API using Django Rest Framework and Vue for the front end and I'm trying to work out what the proper way to do filtering is.
As far as I can see, I can either:-
a) Allow filtering via the API by using URL parameters like /?foo=bar
or
b) Do all the filtering on the Vue side by only displaying the items that are returned that have foo=bar
Are there any strong reasons for doing one over the other?
The real answer to this question is "it depends".
Here are a few questions to ask yourself to help determine what the best approach is:
How much data will be returned if I don't filter at the API level?
If you're returning just a few records, there won't be a noticeable performance hit when the query runs. If you're returning thousands, you'll likely want to consider server side querying/paging.
If you're building an application where the amount of data will grow over time, it's best to build the server side querying from the get-go.
What do I want the front-end experience to be like?
For API calls that return small amounts of data, the user experience will be much more responsive if you return all records up front and do client-side filtering. That way if users change filters or click through paged data, the UI can update almost instantaneously.
Will any other applications be consuming my API?
If you plan to build other apps that consume the API, you may want to build the filtering at the API level so you don't need to recreate front-end filtering logic in every consuming application.
Hopefully these questions can help guide you to the best answer for your use case.
Whenever I come across this issue I ask myself just one question: How many items are you working with? If you're only returning a few items from the API you can easily do the filtering on the front-end side and save yourself a bunch of requests whenever the results are filtered. Also, if the result set is quite small, it's a lot faster to do it this way rather than sending off a request every time the filters change.
However, if you're working with a very large number of items, it's probably best to just filter them out in the API, or even via your database query if that's what you're working with. This will save you from returning a large number of results to the front-end. Also, filtering large numbers of items on the front-end can significantly impact performance since it usually involves looping over a collection.
I have an app in which I want to show a list of around 19000 products. You can search this list and you can filter it by categories. I added paging for my API where the data comes from and I plan on using LoadMore for React FlatList. Now, when I type a search-query inside the search-bar, I want to search within all the data, not the filtered data. Where should I perform the search then? Should i either load all 19000 Items from the API, store them locally and search it? Then the paging would be kind of unnecessary.. Or should I send the search-term to the API and handle the search there? The latter also seems kind of inperformant, because It would send an API-Request every moment I type a letter to the search-bar. Or is there a third approach? What would be the proper way for this?
Thanks in Advance.
Within my app there are multiple pages that display a drop down of "clients". The select options are loading via an GET call made in Axios. Every time a page is displayed it makes that get call.
I'm curious if it's better to store those clients in Vuex, and then just load them that way so I don't make a call every time? The only thing I am concerned about is when a new "client" is added the best way to tell the app it needs to make a new get call to update the data in Vuex.
There are many possible solutions to this.
You could use a cache in back-end suchlike Redis, or as you said, cache it in the front-end.
You can abstract this caching with a get function which will check a maximum threshold of cache age.
For example, you can set it to last for 15 minutes. If another request is made before it you could answer with the last obtained data, else it will request the data to the server again.
I have a search functionality that gets data from HERE API's Search endpoint. I maintain records of each search's results so I can add metadata that I need for my own purposes and also so I can provide results without always going back to HERE API. The problem I have is with paginating, specifically with providing a starting index when fetching results from HERE. Similar to how Algolia does it, I want to be able to search for a term and begin with the results at a certain index, the offset. HERE API apparently doesn't allow this at all. The closest it comes to such a feature is that it provides the URL for the next search, as described here. This is limited because it doesn't allow me to start the search results at a particular index that I specify. So essentially I want to know if there's a "standard" way of getting such functionality even when it's not provided by the API.
My own solution
The HERE API provides a size parameter that allows specifying the total number of results that I want, so I can specify a larger size than I need, and basically use code to start the results from my desired index. But this feels a bit hacky, and I wonder if there's a better/more established way of doing this.
Happy to listen to any ideas! Thanks. :)
Such a kind of an 'offset' for starting the paging after a specific number of results is indeed not supported by the Places API itself.
You have to set up a workaround within your application.
Is it possible to create a Kapow Robot that can search Google for the Operating hours of the Businesses from our list/database and update the timings if changes are made?
Please share if there are any other more efficient ways than the KAPOW robot that can be implemented with minimal effort and cost-effectiveness.
That's what the Google Places API is there for. While you could in theory just open Google Maps in a Load Page action, enter the query string and then parse the results, I would advise against it. Here's why:
The API will be faster, returning results in a structured manner (JSON)
Kapow has actions for calling RESTful services and parsing/modifying JSON
Google does not like robots parsing their pages, and most likely will lock you out (i.e. present you with Captchas sooner or later)
If you decide to go for the API, here's what you should do:
Get your API key first, see this page for details: https://developers.google.com/places/web-service/get-api-key. Note that the free plan allows for 1,000 requests within a 24-hours limit (https://developers.google.com/places/web-service/usage)
Maintain the place ids for all the businesses you'd like to query regularly, and update your list.
For each place, retrieve the details as described in the API documentation. The opening hours will be within the JSON response: https://developers.google.com/places/web-service/details
Update your list. I'd recommend using a definite type in Kapow for that, and using the actions Store in Database and Query Database. In case you need the data elsewhere, you may create additional robots (e.g. for Excel files, sending data per email, et cetera).