What would be best practice for the following situation. I have an ecommerce store that pulls down inventory levels from a distributor. Should the site, for everytime a user loads a product detail page use the third party API for the most up to date data? Or, should the site using third party APIs and then store that data for a certain amount of time in it's own system and update it periodically?
To me it seems obvious that it should be updated everytime the product detail page is loaded but what about high traffic ecommerce stores? Are completely different solutions used for that case?
In this case I would definitely cache the results from the distributor's site for some period of time, rather than hitting them every time you get a request. However, I would not simply use a blanket 5 minute or 30 minute timeout for all cache entries. Instead, I would use some heuristics. If possible, for instance if your application is written in a language like Python, you could attach a simple script to every product which implements the timeout.
This way, if it is an item that is requested infrequently, or one that has a large amount in stock, you could cache for a longer time.
if product.popularityrating > 8 or product.lastqtyinstock < 20:
cache.expire(productnum)
distributor.checkstock(productnum)
This gives you flexibility that you can call on if you need it. Initially, you can set all the rules to something like:
cache.expireover("3m",productnum)
distributor.checkstock(productnum)
In actual fact, the script would probably not include the checkstock function call because that would be in the main app, but it is included here for context. If python seems too heavyweiaght to include just for this small amount of flexibilty, then have a look at TCL which was specifically designed for this type of job. Both can be embedded easily in C, C++, C# and Java applications.
Actually, there is another solution. Your distributor keeps the product catalog on their servers and gives you access to it via Open Catalog Interface. When a user wants to make an order he gets redirected in-place to the distributor's catalog, chooses items then transfers selection back to your shop.
It is widely used in SRM (Supplier Relationship Management) branch.
It depends on many factors: the traffic to your site, how often the inventory levels change, the business impact of displaing outdated data, how often the supplers allow you to call their API, their API's SLA in terms of availability and performance, and so on.
Once you have these answers, there are of course many possibilities here. For example, for a low-traffic site where getting the inventory right is important, you may want to call the 3rd-party API on every call, but revert to some alternative behavior (such as using cached data) if the API does not respond within a certain timeout.
Sometimes, well-designed APIs will include hints as to the validity period of the data. For example, some REST-over-HTTP APIs support various HTTP Cache control headers that can be used to specify a validity period, or to only retrieve data if it has changed since last request.
Related
I don't have problem with getting inventory items. But I can't calculate items prices efficiently.
Valve doesn't have an api for prices.
What i have tried (using "steamcommunity.com") (javascript for example):
itemHashNameArray.forEach((hashname) => {
let url = `https://steamcommunity.com/market/priceoverview/&appid=730&market_hash_name=${hashname}`;
let itemDetails = steamApi(url);
//"steamApi" function just sends get request to site and returns response as json.
let itemPrice = parseFloat(itemDetails.lowest_price.split(" ")[0].replace(",", "."));
})
"steamcommunity.com/market" allows you to get 1 item price per request.
It is very slow / not efficient. Also steam blocks you after so many request.
Third partie apis that I found, allows 1 item per request too.
And they don't even support other currencies except dollar.
I need to calculate for other currencies too.
Is there faster and better way/api?
Just to clarify what's going on:
Most webpages and APIs have limits as far as requests, this helps prevent malicious DoS attacks or simply too much of a resource drain (often costly too). I've ran into this trouble with trying to pull Wikipedia/DBPedia information.
While this is annoying, often times a company or a third-party will recognize the need for doing bulk transactions against their data, and open up an API (usually for a fee): https://partner.steamgames.com/doc/gettingstarted
Some third parties include SteamApis and Steamlytics.
There is a whole subreddit dedicated to this too, here's a relevant post for the API call to get all of a user's inventory items: https://www.reddit.com/r/SteamBot/comments/jey4sg/help_i_am_trying_to_make_a_script_that_counts_the/
Potentially another option is to roll-your-own API service, which could somehow pull inventory data at a rate that doesn't hit limits. Not sure of the legality of this or what kind of hoops third parties have had to jump through, and it would be the most costly probably unless you start charging for its usage. I think loading the full html and then writing a parser to dissect some of that information with straight Javascript or JQuery would alleviate some of the calls looking like spam, but that probably has a limit too and I'm unsure that would include all of the relevant information easily available.
I am using the Bigcommerce API to develop a small standalone application for a client. I store product information in a local database anytime I fetch products from Bigcommerce, to reduce latency and network load. However, products can change on Bigcommerce, and while it is acceptable for my application to show mildly outdated information, I will need to update my local cache at some point. My current plan is to do this by storing the original date I requested the product, after which I will need to perform another request to refresh the cache.
My question is, given a list of products (including their Bigcommerce IDs), is there a way to request updates to all of them through a single call to the Products Resource? I can make a request for each individual product by calling:
GET {api}/v2/products/{id}
I can also request all products within an unbroken ID range by calling:
GET {api}/v2/products?min_id={value}&max_id={value}
I am able to successfully call both of the above methods, and I can chain them together in loops to fetch all products. What I want to do is request multiple products with unrelated IDs in a single call. So, something like this:
//THIS IS NOT A REAL METHOD!
GET {api}/v2/products?id[]={value1}&id[]={value2}
Is there any way I can do this? Or is there another approach to solving this that I haven't considered? My main requirements are:
Minimal API requests. My application is small but my client's bigcommerce store is not, and I will be processing tens of thousands of products. I have limited CPU and network resources available, and I simply cannot process that many requests.
Scalable. As I said, my client's store is large, and growing. I need a solution whose overhead scales at a manageable rate with number of products.
Note: my application is a small web application written in PHP running on a Linux shared hosting environment. It is a back of house system which will likely only be used by single user at a time, during standard business hours. I haven't tagged the question with PHP because my question is about the API, which is language agnostic.
One approch can be.
First get all products from BigCommerce using simple products call.
Set some interval time to get updated product list.
You can use min_date_modified and max_date_modified OR min_date_created and max_date_created in products API call to get updated products details.
There appear to be two lines of APIs for adding, authenticating and aggregating sites. Depending upon which version of the Documentation/SDK set your rep started you off on, or where in the SDK Guide you started implementing from determines where you start.
Path #1 starts at
ContentServiceTraversal which allows for the retrieval of all ContentServiceInfo (by container type (such as BANK)
ItemManagementService is used to add these items
Refresh is done through RefreshService (most API not containing the word Site)
Path #2 starts at
SiteTranversalService which allows for the retrieval of all SiteInfo (no apparent support for Container Type filter)
SiteAccountManagementService is used to add these items
Refresh is done through Refreshservice (all API containing the word Site).
From the best that I can tell the aforementioned API have a lot of functionality duplication. I have noticed certain API that exist on one branch and not the other but usually they are minor changes (e.g. things you are able to filter by).
I started off with ContentServiceInfo because the documentation and samples that our rep initially gave us started there. Additionally this API started off by providing greater granularity (e.g. simply being able to filter by Container type since we were pretty much only interested in Banks and Processor sites (which I do not believe you guys support)).
My questions are:
Do the two branches of API do the exact same thing?
Do they mostly behave the same way?
Do they back-end to the exact same
System
Data store
Scraper?
Is one line of API supposed to be deprecated sooner in the future than another?
Does one line of API have more future in terms of actually adding new or augmenting existing functionality?
Site-level addition has been introduced through Yodlee APIs to overcome the fact that though a user had bank,creditcard,loan,rewards account at the same end site, user had to provide credentials for each of these containers. Site level addition APIs try to add all these containers with only 1 set of credentials. That's the only difference between container based addition and site based addition.
As to answer your questions:
Do the two branches of API do the exact same thing?
Do they mostly behave the same way?
If you mean the aggregation functionality, Yes.Except for the fact that Site level adds/refreshes all the container(bank,creditcard,loan,rewards) and Container level can add/refresh only one container per API call, all the other behavior will remain the same.
Do they back-end to the exact same
System
Data store
Scraper?
If you are referring to the Yodlee data gathering components, Yes.
Is one line of API supposed to be deprecated sooner in the future than another?
No.Both these sets of APIs cater to different needs. If you are a company who solely rely on Creditcard data, using site level addition will be overkill as it will take longer time for the aggregation and it makes more sense to use container based addition. There is also the factor of backward compatibility, which rules out deprecation of APIs.
Let's assume I have a POST /orders operation that takes as input a collection of order items. An order can't contain more than 50 items, but where do I perform this validation?
Validating the order size in both the client and the server would be redundant, and increase the maintenance cost if I decide to change the order size limit,.
Validating it only in the server would prevent clients from "failing fast" (i.e., you add a thousand items to the order and is informed of the limit only when completing it).
I'm assuming client-only validation is not an option, as the API may have other clients.
The problem gets more complicated if I have dynamic validation rules. Suppose retail customers can have orders 50 items, but wholesale customers can have 500 items in their orders. Should the API expose an operation so clients can fetch the current validation rules?
You have to do both, but differently.
To guarantee valid operations, all critical validation must happen on the server/web service side. The client side UI is just that - a user interface to make interacting with the web service convenient for a person. Once the web service is stable and secure, create a default method to pass web service errors through the client to the user. After that, features in the UI layer are usability issues and should be based on testing (even if that is informal testing by watching over a user's shoulder or listening to feedback.)
I agree with what was said before.
Although, I think if you can predict almost every situation a user may come into, you could also create client-side validation.
As per your example about wholesale/retail, you could first create a drop-down that asks the client to choose whether they're wholesale or retail and then apply the 500/50 rule to the input box based on the first option.
The obvious problem comes in the fact that if your API is released to other developers, they may not be aware of the 50/500 rule and that is where I agree with the previous answer about critical validation happening on the server. If you're building the API for your own use then you could go either way because you're aware of the input restrictions. It will also save quite a bit on server-costs if the app is very big (validation on the server will be taxing).
The project that I'm currently on is consuming a large volume of 3rd-party information exposed via APIs. These datasets are constantly changing and in the order of millions of entries for each.
Users are to denote their favorites and recall that data when they need it. An example may be that the user wants to "bookmark" an inventory level to their "analyze later" list.
My current thinking is that during actions like searching users are presented with "live" data from the 3rd parties. If they flag something they're interested in I copy that data to a database I control. Subsequent views of that info are served from my database, not the 3rd party, since the 3rd party entry may change (or cease to exist entirely).
Is this good API practice? What object keys are sent to the client-facing application on search? The 3rd party keys? Or do I preprocess the results of a search and determine which items I have locally, thus returning local keys in those instances? Or do I completely abstract the 3rd party sources and generate unique local keys for every returned item, which is then subsequently used if someone saves [that seems REALLY heavy, tho]? Or do I put that processing off and do the lookup as to whether something exists locally to after someone bookmarks something?