I apologize for being rather vague here, but I'm working on a project involving stock data and stock purchases. I'm sure I'm going to end up having to get a broker involved, but I was wondering if anyone knows of any documentation on the underlying technology involved with existing trading sites, as well as the channels through which systems like google finance get their information.
Note that I already know of the APIs from yahoo and tdameritrade that send out the data, I'm interested in the channels through which that data travels to them in the first place.
They're most likely getting the data feeds from one or more of the usual suspects (ie, Reuters, Bloomberg and the like). You've probably noticed that the feeds on the publically accessible websites are delayed by 15-20 minutes compared to the real time feeds. Keep that in mind in your application, if you need proper up to date/real time market data it'll cost you a fair penny.
Those firms trading directly on the exchanges obviously have access to the data from the actual exchange - that's what you (have to) use in real time and algorithmic trading. However, the above mentioned companies (and I'm sure there are a few more, these are just the ones that most people are familiar with) are usually the data providers for those trading via intermediaries.
For reference data on stocks (as opposed to actual stock quote data), Mergent ( http://www.mergent.com ) is one of the data suppliers and has been collecting the data for decades. It has a set of APIs at http://www.mergent.com/servius
Related
I don't have problem with getting inventory items. But I can't calculate items prices efficiently.
Valve doesn't have an api for prices.
What i have tried (using "steamcommunity.com") (javascript for example):
itemHashNameArray.forEach((hashname) => {
let url = `https://steamcommunity.com/market/priceoverview/&appid=730&market_hash_name=${hashname}`;
let itemDetails = steamApi(url);
//"steamApi" function just sends get request to site and returns response as json.
let itemPrice = parseFloat(itemDetails.lowest_price.split(" ")[0].replace(",", "."));
})
"steamcommunity.com/market" allows you to get 1 item price per request.
It is very slow / not efficient. Also steam blocks you after so many request.
Third partie apis that I found, allows 1 item per request too.
And they don't even support other currencies except dollar.
I need to calculate for other currencies too.
Is there faster and better way/api?
Just to clarify what's going on:
Most webpages and APIs have limits as far as requests, this helps prevent malicious DoS attacks or simply too much of a resource drain (often costly too). I've ran into this trouble with trying to pull Wikipedia/DBPedia information.
While this is annoying, often times a company or a third-party will recognize the need for doing bulk transactions against their data, and open up an API (usually for a fee): https://partner.steamgames.com/doc/gettingstarted
Some third parties include SteamApis and Steamlytics.
There is a whole subreddit dedicated to this too, here's a relevant post for the API call to get all of a user's inventory items: https://www.reddit.com/r/SteamBot/comments/jey4sg/help_i_am_trying_to_make_a_script_that_counts_the/
Potentially another option is to roll-your-own API service, which could somehow pull inventory data at a rate that doesn't hit limits. Not sure of the legality of this or what kind of hoops third parties have had to jump through, and it would be the most costly probably unless you start charging for its usage. I think loading the full html and then writing a parser to dissect some of that information with straight Javascript or JQuery would alleviate some of the calls looking like spam, but that probably has a limit too and I'm unsure that would include all of the relevant information easily available.
I am trying to find the equivalent of FQP/FQD/FQN queries in the Amadeus SOAP service (Flight) API, but couldn't find one. I checked the API documents as well with no luck. There is command cryptic API to call the GDS commands, but the response is raw data as in the terminal, not a structured one. I need structured data response to precess data in the system.
Is there any SOAP APIs available to get the fare details and the rules?
Thanks
This is a question best sent to Amadeus directly! It's hard to know exactly what you are after without more information. Also Amadeus offer a wide variety of doing basically the same thing and your business can be better off using one or the other - its really impossible to tell without background information.
Check out these webservice calls:
Fare_PricePNRWithBookingClass
The function Fare_PricePNRWithBookingClass is used to price itineraries.
It can return one or several fare recommendations for the passenger(s) and for the itinerary of the active PNR. Only booking classes present in the flight segment of the PNR are considered.
After calling Fare_PricePNRWithBookingClass function, the system keeps the recommendations stored internally for three minutes in a dedicated context. This context can be used to create a TST by using Ticket_CreateTSTFromPricing.
Fare_PricePNRWithLowerFares
The function Fare_PricePNRWithLowerFares is used to display the lowest available fare for a given itinerary.
"Lowest available" means that this fare is applicable in a booking class where there are still enough seats available for the passengers of the PNR. This class might not be the one currently present in the flight segment of the PNR. In this case, rebooking might be necessary.
It can return one or several fare recommendations for the passenger(s) and for the itinerary of the active PNR.
After calling Fare_PricePNRWithLowerFares function, the system keeps the recommendations stored internally for three minutes in a dedicated context. This context can be used to create a TST by using Ticket_CreateTSTFromPricing. Please not that in case rebooking is required, it must be done (for example, using Air_RebookAirSegment) before TST creation.
Fare_InformativeBestPricingWithoutPNR
The InformativeBestPricingWithoutPNR function provided in the Fare interface is used to price informatively an itinerary without any PNR. If a PNR exists, it is neither taken into account nor updated. No pricing record (TST) is created to store the results.
Fare_QuoteItinerary
The QuoteItinary transaction (FQP) quotes fares for passenger types without existing reservations (PNR). Pricing is executed according to the principles of IATA resolutions as well as according to specific user requirements, if industry conform or individual.
Most likely there are more options available (that I don't know about).
Documentation available at https://webservices.amadeus.com/ - login required.
I want to maintain a database of all the products or the brands with respect to industry.
For example I need to get information about all the food supplements. How can I get them?
I am not sure all the companies have an API for their products.
Please advise
Uhm,... what kind of information? If you need prices, you can probably get information from goverment sources. At least you can here in Argentina. Other than that, I don't think it's possible, unless you somehow manage to scrape websites of all the brands you want to track.
Speaking as someone who has worked for two data-aggregation companies, aggregating data involves a lot of manual work. You find the sources, you automate the acquisition of data as best you can (APIs, file downloads and imports, even screen scraping from HTML pages), and you stay on top of it constantly. You're always looking for additional sources, updating code for sources that have changed, minding legal implications of sources who don't want you to harvest their data, etc.
Sometimes you have to buy the data, or weigh that cost against not having data from that source or scraping it manually. Sometimes a source will block you in some way and you need to either try to get around that or negotiate some terms with them. It's a viable business model, but it's not cheap.
For some products, Retailigence ( http://www.retailigence.com ) may have data in API form. They basically keep track of local stores' inventory and pricing for certain categories of products.
You should definitely check out Good Guide - an API that gives you access to details on over 60,000 household products.
http://developer.goodguide.com
DailyMed is a good service to check out if you're interested in products in the medical space.
http://dailymed.nlm.nih.gov/dailymed
I'm reviewing some code where we've had some issues with return data from a WCF web service. Currently the service makes a list of objects, serializes them (as JSON for the record) and returns the entire serialized list down the wire. Obviously when there's a lot of data users run into quota limit problems.
I'm considering changing it so the service returns one item at a time which would send a bunch of requests on a loop adding one object at a time into the list until it was done.
Obviously in scenario one we're making one request to the service that has the potential to return a massive amount of data and run up against the quota. In the other scenario we never hit the quota but the requesting app will be requesting data item after data item in a stream of separate requests.
To illustrate we have a list of items which come in a variety of item types and those types come at a variety of price points. The app might want to aggregate a number of items, the customers who want that item, the types of item and price requested by the customer and their could be maybe seventy items with between five and eighty customers each requesting on average 2 types of product at 1 price each.
Taking averages at the extreme end this could make 7000 separate (very small) data requests in a single complete job. Is that a problem? It is possible to package it up a bit so that customer types and prices requested could be bundled but that's still potentially a couple of thousand requests at one time.
Am I better off with a single huge data stream? Or a couple of thousand smaller ones?
You're better off with the optimal sized return for your scenario :) It kinda depends on the overhead on the request. Generally the less chatter back and forth to a web service the better.
Facetious answer, so here's the rub:
You're probably best off with some sort of paging system, wherein your request asks for a specific number of items, and your response returns a "n of m" in the results. That way you can tune the number of requests and size of the response to perform best in your situation.
What would be best practice for the following situation. I have an ecommerce store that pulls down inventory levels from a distributor. Should the site, for everytime a user loads a product detail page use the third party API for the most up to date data? Or, should the site using third party APIs and then store that data for a certain amount of time in it's own system and update it periodically?
To me it seems obvious that it should be updated everytime the product detail page is loaded but what about high traffic ecommerce stores? Are completely different solutions used for that case?
In this case I would definitely cache the results from the distributor's site for some period of time, rather than hitting them every time you get a request. However, I would not simply use a blanket 5 minute or 30 minute timeout for all cache entries. Instead, I would use some heuristics. If possible, for instance if your application is written in a language like Python, you could attach a simple script to every product which implements the timeout.
This way, if it is an item that is requested infrequently, or one that has a large amount in stock, you could cache for a longer time.
if product.popularityrating > 8 or product.lastqtyinstock < 20:
cache.expire(productnum)
distributor.checkstock(productnum)
This gives you flexibility that you can call on if you need it. Initially, you can set all the rules to something like:
cache.expireover("3m",productnum)
distributor.checkstock(productnum)
In actual fact, the script would probably not include the checkstock function call because that would be in the main app, but it is included here for context. If python seems too heavyweiaght to include just for this small amount of flexibilty, then have a look at TCL which was specifically designed for this type of job. Both can be embedded easily in C, C++, C# and Java applications.
Actually, there is another solution. Your distributor keeps the product catalog on their servers and gives you access to it via Open Catalog Interface. When a user wants to make an order he gets redirected in-place to the distributor's catalog, chooses items then transfers selection back to your shop.
It is widely used in SRM (Supplier Relationship Management) branch.
It depends on many factors: the traffic to your site, how often the inventory levels change, the business impact of displaing outdated data, how often the supplers allow you to call their API, their API's SLA in terms of availability and performance, and so on.
Once you have these answers, there are of course many possibilities here. For example, for a low-traffic site where getting the inventory right is important, you may want to call the 3rd-party API on every call, but revert to some alternative behavior (such as using cached data) if the API does not respond within a certain timeout.
Sometimes, well-designed APIs will include hints as to the validity period of the data. For example, some REST-over-HTTP APIs support various HTTP Cache control headers that can be used to specify a validity period, or to only retrieve data if it has changed since last request.