As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is there any sort of API that just offers a simple symbol lookup service? i.e., input a company name and it will tell you the ticker symbol? I've tried just screen-scraping Google Finance, but after a little while it rate limits you and you have to enter a CAPTCHA. I'm trying to batch-lookup about 2000 ticker symbols. Any ideas?
You can use yahoo's symbol lookup like so:
http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=yahoo&callback=YAHOO.Finance.SymbolSuggest.ssCallback
Where query is the company name.
You'll get something like this in return:
YAHOO.Finance.SymbolSuggest.ssCallback(
{
"ResultSet": {
"Query": "ya",
"Result": [
{
"symbol": "YHOO",
"name": "Yahoo! Inc.",
"exch": "NMS",
"type": "S",
"exchDisp": "NASDAQ"
},
{
"symbol": "AUY",
"name": "Yamana Gold, Inc.",
"exch": "NYQ",
"type": "S",
"exchDisp": "NYSE"
},
{
"symbol": "YZC",
"name": "Yanzhou Coal Mining Co. Ltd.",
"exch": "NYQ",
"type": "S",
"exchDisp": "NYSE"
},
{
"symbol": "YRI.TO",
"name": "YAMANA GOLD INC COM NPV",
"exch": "TOR",
"type": "S",
"exchDisp": "Toronto"
},
{
"symbol": "8046.TW",
"name": "NAN YA PRINTED CIR TWD10",
"exch": "TAI",
"type": "S",
"exchDisp": "Taiwan"
},
{
"symbol": "600319.SS",
"name": "WEIFANG YAXING CHE 'A'CNY1",
"exch": "SHH",
"type": "S",
"exchDisp": "Shanghai"
},
{
"symbol": "1991.HK",
"name": "TA YANG GROUP",
"exch": "HKG",
"type": "S",
"exchDisp": "Hong Kong"
},
{
"symbol": "1303.TW",
"name": "NAN YA PLASTIC TWD10",
"exch": "TAI",
"type": "S",
"exchDisp": "Taiwan"
},
{
"symbol": "0294.HK",
"name": "YANGTZEKIANG",
"exch": "HKG",
"type": "S",
"exchDisp": "Hong Kong"
},
{
"symbol": "YAVY",
"name": "Yadkin Valley Financial Corp.",
"exch": "NMS",
"type": "S",
"exchDisp": "NASDAQ"
}
]
}
}
)
Which is JSON and very easy to work with.
Hush... don't tell anybody.
Google Finance does let you retrieve up to 100 stock quotes at once using the following URL:
www.google.com/finance/info?infotype=infoquoteall&q=[ticker1],[ticker2],...,[tickern]
For example:
www.google.com/finance/info?infotype=infoquoteall&q=C,JPM,AIG
Someone has deciphered the available fields here:
http://qsb-mac.googlecode.com/svn/trunk/Vermilion/Modules/StockQuoter/StockQuoter.py
The current price ("l") is real-time and the delay is on par with Yahoo Finance. There are a few quirks you should be aware of. A handful of stocks require an exchange prefix. For example, if you query "BTIM", you'll get a "Bad Request" error but "AMEX:BTIM" works. A few stocks don't work even with the exchange prefix. For example, querying "FTWRD" and "NASDAQ:FTWRD" both generate "Bad Request" errors even though Google Finance does have information for this NASDAQ stock.
The "el" field, if present, tells you the current pre-market or after-hours price.
You can send an HTTP request to http://finance.yahoo.com requesting symbols, names, quotes, and all sorts of other data. Data is returned as a .CSV so you can request multiple symbols in one query.
So if you send:
http://finance.yahoo.com/d/quotes.csv?s=MSFT+F+ATT&f=sn
You'll get back something like:
"MSFT","Microsoft Corp"
"F","FORD MOTOR CO"
"ATT","AT&T"
Here is an article called Downloading Yahoo Data which includes the various tags used to request the data.
The NASDAQ site hosts separate CSV lists for ticker symbols in each stock exchange (NYSE, AMEX and NASDAQ). You need to complete the captcha and get the CSV dump.
http://www.nasdaq.com/screening/company-list.aspx
If you didn't want to sign up for a service, I'd probably go back to the exchanges themselves; most of them aren't CAPTCHAed yet...
The symbol lookup page for:
NYSE is at http://www.nyse.com/interface/html/SymbolLookup.html
NASDAQ is at http://www.nasdaq.com/asp/NasdaqSymLookup2.asp?mode=stock
London Stock Exchange is at http://www.londonstockexchange.com/en-gb/pricesnews/prices/Trigger/genericsearch.htm
ASX is at http://www.asx.com.au/asx/research/codeLookup.do
etc...
Use YQL and you don't need to worry. It's a query language by Yahoo and you can get all the stock data including the name of the company for the ticker. It's a REST API and it returns the results via XML or JSON. I have a full tutorial and source code on my site take a look: http://www.jarloo.com/yahoo-stock-symbol-lookup/
Currently, the NASDAQ web site publicly provides CSV files containing bulk listings -- it is broken up by first letter.
http://www.nasdaq.com/screening/companies-by-name.aspx?letter=A&render=download
Google Finance has an API - you probably have to apply for a developers key, but at least you'd save yourself the hassle of screen-scraping: http://code.google.com/apis/finance/reference.html
Your best bets are probably going with one of the other lookup services (still screen-scraping), and checking whether they don't require CAPTCHAs.
Yahoo Finance
MSN Money
AlphaTrade Finance
The last appears the least likely to require a CAPTCHA at any point, but it's worth checking all three.
Use YQL: a sql-like language to retrieve stuff from public api's:
YQL Console (external link)
It gives you a nice XML file to work with!
You can use the "Company Search" operation in the Company Fundamentals API here: http://www.mergent.com/servius/
Related
Context
In our project, we need to represent resources defined by the users. That is, every user can have different resources, with different fields, different validations, etc. So we have two different things to represent in our API:
Resource definition: this is just a really similar thing to a json schema, it contains the fields definitions of the resource and its limitations (like min and max value for numeric fields). For instance, this could be the resource definition for a Person:
{
"$id": "https://example.com/person.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Person",
"type": "object",
"properties": {
"firstName": {
"type": "string",
"description": "The person's first name."
},
"lastName": {
"type": "string",
"description": "The person's last name."
},
"age": {
"description": "Age in years which must be equal to or greater than zero.",
"type": "integer",
"minimum": 0
}
}
}
Resource instance: this is just an instance of the specified resource. For instance, for the Person resource definition, we can have the following instances:
[
{
"firstName": "Elena",
"lastName": "Gomez",
},
{
"firstName": "Elena2",
"lastName": "Gomez2",
},
]
First opinion
So, it seems this kind of presents some conflicts with the Restful API approach. In particular, I think it has some problems with the Uniform Interface. When you get a resource, you should be able to handle the resource without any additional information. With this design, you need to make an additional request to first get the resource definition. Let's see this with an example:
Suppose you are our web client. And you are logged in as an user with the Person resource. To show a person in the UI, you first need to know the structure of the Person resource, that is, you to do the following request: GET /resource_definitions/person. And then, you need to request the person object: GET /resource/person/123.
Second opinion
Others seem to think that this is not a problem and that the design is still RESTful. Every time you ask for something to an API, you need to know the format previously, is not self-documented in the API, so it makes sense for this endpoint to behave the same as the others.
Question
So what do you think? Is the proposed solution compliance with the RESTful approach to API design?
The simple solution is to add a link:
{
"_links": {
"describedby": {
"href": "https://example.com/person.schema.json",
"type": "application/schema+json"
}
},
"firstName": "Elena",
"lastName": "Gomez"
}
You could also put this in a header. This is semantically equivalent:
Link: <https://example.com/person.schema.json>; rel="describedby" type="application/schema+json"
It does not violate the uniform interface if there is no standard for this kind of stuff, but there is. RDF e.g. JSON-LD and schema.org vocab can handle most of these types. Even for REST there is an RDF vocab called Hydra, though the community is not that active nowadays.
As of the actual problem, I would look around, maybe RDF technologies or graph technologies are better for it, though I am not sure how much connection there is in your graph. If it is just a few types and instances, then I would probably stick to REST.
Ohh I see meanwhile, you used an actual JSON schema. Then that part is certainly uniform interface compatible. As of the instances you need to add something like type: "https://example.com/person.schema.json" and you are ok. Maybe a vendor specific JSON derived MIME type which describes what "type" means in this context if you want to be super precise or just use JSON-LD instead. https://www.w3.org/2019/wot/json-schema Or an alternative more common solution is using RDFS and XSD with JSON-LD instead of JSON Schema.
I'm trying to structured data for producing the review like this on google search (please see the image) -
According to this link I've to write the following structured data markup -
<script type="application/ld+json">
{
"#context": "http://schema.org/",
"#type": "Review",
"itemReviewed": {
"#type": "Thing",
"name": "Super Book"
},
"author": {
"#type": "Person",
"name": "Joe"
},
"reviewRating": {
"#type": "Rating",
"ratingValue": "7",
"bestRating": "10"
},
"publisher": {
"#type": "Organization",
"name": "Washington Times"
}
}
</script>
But according to this link I've to get review from a trusted review authority. I'm wondering why we need the structured data markup (where we have static 'rating', 'bestRating' etc value definitely these shouldn't be static) or how we can combine this with trusted review authority for getting dynamic ratting that changes over time?
If I'm understanding your question correctly, I think you are confusing two issues. Google requires reviews to be created using Schema markup in order for the review to have a chance to rank directly in the SERPs.
It is the companies that provide reviews: Yelp, Angie's List, Washington Times, etc, that have to format their content management systems to upload user generated review data into the proper markup.
So if you're a web developer working for one of these companies, then it makes sense to code the CMS so that the listings are displayed using schema markup.
If you are the marketer, your job is to get reviews, not format the way they are getting displayed.
There are of course other ways to use Schema markup on your own site to boost organic traffic. Consider for example the first SERP screenshot displayed in this article.
Here the webmaster has used schema markup to list three upcoming events in their result, which gives them four links in a single listing. This causes the listing to stand out and gives increased incentive for users to click, almost guaranteeing a higher click-thru rate than if they'd have not used the markup.
I want to scrape sitelinks which are shown in the google search results(like About us Home Page etc..) . Is there any way I can retrieve them ?
enter image description here
I recently implement Google Search JSON API, and from my understanding, the only way to get the website links is through the JSON Callback where each result contains formattedUrl or htmlFormattedUrl. The query would be the site in question and hopefully the first results would give you relevant links of the site.
However, if I properly understood your question, you want to scrap the sub-links of a given website which is something that a web crawler would do. If you are the owner of the website, you can create a sitemap using many tools around the web, but if your intentions can be classified as "other", then I believe that you are barking at the wrong tree. See this question which will pinpoint you to create a simple WebCrawler.
// Example customsearch#result item in which the query was Deovandski.
"items": [
{
"kind": "customsearch#result",
"title": "Student Experience - College of Science and Mathematics (NDSU)",
"htmlTitle": "Student Experience - College of Science and Mathematics (NDSU)",
"link": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"displayLink": "www.ndsu.edu",
"snippet": "Sep 16, 2015 ... Association for Computing Machinery Student Chapter Chair: Jordan Goetze \nAdvisor: Brian Slator. Upsilon Pi Epsilon President: Deovandski ...",
"htmlSnippet": "Sep 16, 2015 \u003cb\u003e...\u003c/b\u003e Association for Computing Machinery Student Chapter Chair: Jordan Goetze \u003cbr\u003e\nAdvisor: Brian Slator. Upsilon Pi Epsilon President: \u003cb\u003eDeovandski\u003c/b\u003e ...",
"cacheId": "pyzF9XJwrXsJ",
"formattedUrl": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"htmlFormattedUrl": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"pagemap": {
"cse_image": [
{
"src": "https://www.ndsu.edu/fileadmin/_processed_/csm_080117_anatomy_03med_9dbc3c8cce.jpg"
}
],
"cse_thumbnail": [
{
"width": "184",
"height": "275",
"src": "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcTTL-GZRfSv30cyESsCnd_65BFoLMDdo8fqNS58mHfRbGiOTjSq-e-o28FE"
}
]
}
},
Below is one Json item returned from this query and this is the query:
https://www.googleapis.com/customsearch/v1?key={key}&cx={key}&q=Action+Motivation%2c+Inc.&alt=json
The "dc.type" in the Json is "Patent" and this is obviously patent data BUT I didn't specify that search engine. I've googled this to death but can't find anything re why patent data would be returned from a simple query like this. If Google "Action Motivation, Inc." on the regular google.com page, I get completely different (normal) results. Has anyone had this problem?
"items": [
{
"kind": "customsearch#result",
"title": "Patent US5622527 - Independent action stepper - Google Patents",
"htmlTitle": "Patent US5622527 - Independent \u003cb\u003eaction\u003c/b\u003e stepper - Google Patents",
"link": "https://www.google.com/patents/US5622527",
"displayLink": "www.google.com",
"snippet": "Apr 22, 1997 ... Original Assignee, Icon Health & Fitness, Inc., Proform Fitness ....",
"htmlSnippet": "Apr 22, 1997 \u003cb\u003e...\u003c/b\u003e Original Assignee, Icon Health & Fitness..."
"formattedUrl": "https://www.google.com/patents/US5622527",
"htmlFormattedUrl": "https://www.google.com/patents/US5622527",
"pagemap": {
"book": [
{
"description": "A motivational exercise stepping machine has a pair of independently operable pivoting treadles for operation..."
"url": "https://www.google.com/patents/US5622527?utm_source=gb-gplus-share",
"name": "Patent US5622527 - Independent action stepper",
"image": "https://www.google.com/patents?id=&printsec=frontcover&img=1&zoom=1"
}
],
"metatags": [
{
***"dc.type": "Patent"***,
"dc.title": "Independent action stepper",
"dc.contributor": "William T. Dalebout",
"dc.date": "1994-3-23",
"dc.description": "A motivational exercise stepping machine has a pair of independently operable pivoting treadles for operation by a user's feet. Each treadle..."
"dc.relation": "JP:S5110842"
}
]
}
},
{
When using their API, you can issue around 40 requests per hour. The results you see on the API is not what the real user sees. You are limited to what they give you, it's not really useful if you want to track ranking positions or what a real user would see. That's something you are not allowed to gather.
If you want a higher amount of API requests you need to pay.
60 requests per hour cost 2000 USD per year, more queries require a custom deal.
I have come across various situations, where I want to store some formatted data in a way such that it can be easily queried.
For example
$ cat so.txt
"question_id": 58640,
"tags": ["polls", "fun", "quotes"],
"title": "Great programming quotes"
"question_id": 184618,
"tags": ["polls", "fun", "comment"],
"title": "What is the best comment in source code you have ever encountered?"
"question_id": 3734102,
"tags": ["c++", "linux", "exit-code"],
"title": "Why cant' I return bigger values from main function ?"
"question_id": 2349378,
"tags": ["communication", "terminology", "vocabulary"],
"title": "New programming jargon you coined?"
"question_id": 3723817,
"tags": ["open-source", "project-management", "failure", "fail"],
"title": "How to make an open source project fail"
"question_id": 3699150,
"tags": ["testing", "interview-questions", "job-interview"],
"title": "Interview question please help"
$
A simple query can be displaying the titles of the questions with tags "C++".
These are the requirements
The database has to support just me.
It must be able to support all the general SQL type queries. I am familiar with SQL, so the more it is SQL-like, the better.
It has to be run locally on my Linux machine running Ubuntu 10.04.
Also, since my requirements are minimal, I expect it not to use up too much memory.
What DBMS do you suggest for this purpose?
the code you posted looks like json. if this is the format you primarily keep your data, it's possible that something like couchdb would be ideal for you? I'm a big fan of it, so I'm biased :)
otherwise, the traditional answer to "light database that only supports me" is SQLite.