Further update...
I got this working. Although Strava's documentation does not say any of the arguments in the call are mandatory it seems they all are. You need to put valid before and after arguments in epoch time and... (and this is the part that confused me a bit) you need to give a page number and items per page. The items per page default to 30 but the page number does not default. The way it works is if you say page 1 and 30 items per page you get items 1 - 30. If you say page 3 and 30 items per page you get items 31 - 60 and so on. You have to create a loop that keeps going until it gets a blank page. You then know you have retrieved all the activities. (At least that is how I think it works.)
Adrian
Question update...
After some digging and experimenting I have managed to solve some of my problem (as described below) on my own. When one creates an app on Strava listed under your settings under "My API Application" the token given has scope "read" and seems to be very, very limited.
After following the steps listed here Strava Authentication I was able to get a new token with the following scopes:
scope=read,activity:read,activity:read_all,profile:read_all,read_all
So... I thought I was "golden" as the saying goes.
Well now I am able to get individual activities using:
https://www.strava.com/api/v3/activities/2110745394?include_all_efforts="true"&access_token={{ADR_Strava_API_Key}}
But when I try to get a list of all activities I don't get any error messages but Strava simply returns
[] and this for an athlete that I know has over 1800 activities.
What I really want is to get the list of activities. Any help would be appreciated.
Thank you
Adrian
I can get athlete information back from Strava using postman using the following https request:
https://www.strava.com/api/v3/athletes/19133707?access_token={{ADR_Strava_API_Key}}
The following gets returned:
{
"id": 19133707,
"username": "adrian_geekie",
"resource_state": 2,
"firstname": "Adrian",
"lastname": "Geekie",
"city": "Gauteng, South Africa",
"state": "GP",
"country": "South Africa",
"sex": "M",
"premium": true,
"summit": true,
"created_at": "2017-01-03T16:07:37Z",
"updated_at": "2019-01-28T16:08:07Z",
"badge_type_id": 1,
"profile_medium": "https://dgalywyr863hv.cloudfront.net/pictures/athletes/19133707/5599004/2/medium.jpg",
"profile": "https://dgalywyr863hv.cloudfront.net/pictures/athletes/19133707/5599004/2/large.jpg",
"friend": null,
"follower": null
}
But when I try to get activities using this request:
https://www.strava.com/api/v3/19133707/activities?before=&after=1546293601&page=&per_page=&access_token={{ADR_Strava_API_Key}}
I get this returned:
{
"message": "Record Not Found",
"errors": [
{
"resource": "resource",
"field": "path",
"code": "invalid"
}
]
}
According to me I am asking for all records after the 1st of January 2019 i.e. epoch timestamp 1546293601. I know there are many activities for that athlete after that date. (More than 20).
I have also tried to get a single activity using:
https://www.strava.com/api/v3/activities/2110745394?include_all_efforts="true"&access_token={{ADR_Strava_API_Key}}
and I get the result:
{
"message": "Resource Not Found",
"errors": [
{
"resource": "Activity",
"field": "",
"code": "not found"
}
]
}
On the Strava developer's page the examples are given for HTTPie like this:
https://www.strava.com/api/v3/activities/{id}?include_all_efforts=" "Authorization: Bearer [[token]]
So I am replacing "Authorization: Bearer [[token]] with &access_token=
Perhaps that is my error but access_token works in the first example.
I am sorry if this is a total idiot question. I am a beginner and I would appreciate any help.
Thank you
I want to scrape sitelinks which are shown in the google search results(like About us Home Page etc..) . Is there any way I can retrieve them ?
enter image description here
I recently implement Google Search JSON API, and from my understanding, the only way to get the website links is through the JSON Callback where each result contains formattedUrl or htmlFormattedUrl. The query would be the site in question and hopefully the first results would give you relevant links of the site.
However, if I properly understood your question, you want to scrap the sub-links of a given website which is something that a web crawler would do. If you are the owner of the website, you can create a sitemap using many tools around the web, but if your intentions can be classified as "other", then I believe that you are barking at the wrong tree. See this question which will pinpoint you to create a simple WebCrawler.
// Example customsearch#result item in which the query was Deovandski.
"items": [
{
"kind": "customsearch#result",
"title": "Student Experience - College of Science and Mathematics (NDSU)",
"htmlTitle": "Student Experience - College of Science and Mathematics (NDSU)",
"link": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"displayLink": "www.ndsu.edu",
"snippet": "Sep 16, 2015 ... Association for Computing Machinery Student Chapter Chair: Jordan Goetze \nAdvisor: Brian Slator. Upsilon Pi Epsilon President: Deovandski ...",
"htmlSnippet": "Sep 16, 2015 \u003cb\u003e...\u003c/b\u003e Association for Computing Machinery Student Chapter Chair: Jordan Goetze \u003cbr\u003e\nAdvisor: Brian Slator. Upsilon Pi Epsilon President: \u003cb\u003eDeovandski\u003c/b\u003e ...",
"cacheId": "pyzF9XJwrXsJ",
"formattedUrl": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"htmlFormattedUrl": "https://www.ndsu.edu/scimath/currentstudents/student_experience/",
"pagemap": {
"cse_image": [
{
"src": "https://www.ndsu.edu/fileadmin/_processed_/csm_080117_anatomy_03med_9dbc3c8cce.jpg"
}
],
"cse_thumbnail": [
{
"width": "184",
"height": "275",
"src": "https://encrypted-tbn2.gstatic.com/images?q=tbn:ANd9GcTTL-GZRfSv30cyESsCnd_65BFoLMDdo8fqNS58mHfRbGiOTjSq-e-o28FE"
}
]
}
},
Below is one Json item returned from this query and this is the query:
https://www.googleapis.com/customsearch/v1?key={key}&cx={key}&q=Action+Motivation%2c+Inc.&alt=json
The "dc.type" in the Json is "Patent" and this is obviously patent data BUT I didn't specify that search engine. I've googled this to death but can't find anything re why patent data would be returned from a simple query like this. If Google "Action Motivation, Inc." on the regular google.com page, I get completely different (normal) results. Has anyone had this problem?
"items": [
{
"kind": "customsearch#result",
"title": "Patent US5622527 - Independent action stepper - Google Patents",
"htmlTitle": "Patent US5622527 - Independent \u003cb\u003eaction\u003c/b\u003e stepper - Google Patents",
"link": "https://www.google.com/patents/US5622527",
"displayLink": "www.google.com",
"snippet": "Apr 22, 1997 ... Original Assignee, Icon Health & Fitness, Inc., Proform Fitness ....",
"htmlSnippet": "Apr 22, 1997 \u003cb\u003e...\u003c/b\u003e Original Assignee, Icon Health & Fitness..."
"formattedUrl": "https://www.google.com/patents/US5622527",
"htmlFormattedUrl": "https://www.google.com/patents/US5622527",
"pagemap": {
"book": [
{
"description": "A motivational exercise stepping machine has a pair of independently operable pivoting treadles for operation..."
"url": "https://www.google.com/patents/US5622527?utm_source=gb-gplus-share",
"name": "Patent US5622527 - Independent action stepper",
"image": "https://www.google.com/patents?id=&printsec=frontcover&img=1&zoom=1"
}
],
"metatags": [
{
***"dc.type": "Patent"***,
"dc.title": "Independent action stepper",
"dc.contributor": "William T. Dalebout",
"dc.date": "1994-3-23",
"dc.description": "A motivational exercise stepping machine has a pair of independently operable pivoting treadles for operation by a user's feet. Each treadle..."
"dc.relation": "JP:S5110842"
}
]
}
},
{
When using their API, you can issue around 40 requests per hour. The results you see on the API is not what the real user sees. You are limited to what they give you, it's not really useful if you want to track ranking positions or what a real user would see. That's something you are not allowed to gather.
If you want a higher amount of API requests you need to pay.
60 requests per hour cost 2000 USD per year, more queries require a custom deal.
I actually do not want to list each edit, but to get only the count of it.
this data is available for every article on the left panel in link:
https://en.wikipedia.org/w/index.php?title=Wikipedia&action=info
But this produces complete web page with tables, formatting etc and its exhaustive for wikipedia servers. So I ask if is there a way to only get those few numbers and ommit the whole website scraping.
Probably not the answer you want but there isn't a way to get this information yet.
As a workaround you can use the prop=revisions to get all the revisions contributed to the article. You will be able to count the rev tag from here:
http://en.wikipedia.org/w/api.php?format=xml&action=query&titles=Wikipedia&prop=revisions&rvprop=ids&rvlimit=max
Alternatively, you can ask YQL to count it for you with the following command:
SELECT * FROM xml
WHERE url="http://en.wikipedia.org/w/api.php?format=xml&action=query&titles=Wikipedia&prop=revisions&rvprop=ids&rvlimit=max"
AND itemPath="/api/query/pages/page/revisions/rev"
Example output (Link to full output):
{
"query": {
"count": 500, //This is the total amount of edits
"created": "2014-03-04T02:29:42Z",
"lang": "en-US",
"results": {
"rev": [{
"parentid": "597995345",
"revid": "598005528"
}, {
"parentid": "597994174",
"revid": "597995345"
}, {
"parentid": "597891867",
"revid": "597994174"
}]
}
}
}
Unfortunately, the upper limit for users to retrieve revision data is 500 and for bots it's 5000.
To get the exact count, you will have to set up a parser on your server to capture the exact count from the info page whenever a user queries the data from your side.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is there any sort of API that just offers a simple symbol lookup service? i.e., input a company name and it will tell you the ticker symbol? I've tried just screen-scraping Google Finance, but after a little while it rate limits you and you have to enter a CAPTCHA. I'm trying to batch-lookup about 2000 ticker symbols. Any ideas?
You can use yahoo's symbol lookup like so:
http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=yahoo&callback=YAHOO.Finance.SymbolSuggest.ssCallback
Where query is the company name.
You'll get something like this in return:
YAHOO.Finance.SymbolSuggest.ssCallback(
{
"ResultSet": {
"Query": "ya",
"Result": [
{
"symbol": "YHOO",
"name": "Yahoo! Inc.",
"exch": "NMS",
"type": "S",
"exchDisp": "NASDAQ"
},
{
"symbol": "AUY",
"name": "Yamana Gold, Inc.",
"exch": "NYQ",
"type": "S",
"exchDisp": "NYSE"
},
{
"symbol": "YZC",
"name": "Yanzhou Coal Mining Co. Ltd.",
"exch": "NYQ",
"type": "S",
"exchDisp": "NYSE"
},
{
"symbol": "YRI.TO",
"name": "YAMANA GOLD INC COM NPV",
"exch": "TOR",
"type": "S",
"exchDisp": "Toronto"
},
{
"symbol": "8046.TW",
"name": "NAN YA PRINTED CIR TWD10",
"exch": "TAI",
"type": "S",
"exchDisp": "Taiwan"
},
{
"symbol": "600319.SS",
"name": "WEIFANG YAXING CHE 'A'CNY1",
"exch": "SHH",
"type": "S",
"exchDisp": "Shanghai"
},
{
"symbol": "1991.HK",
"name": "TA YANG GROUP",
"exch": "HKG",
"type": "S",
"exchDisp": "Hong Kong"
},
{
"symbol": "1303.TW",
"name": "NAN YA PLASTIC TWD10",
"exch": "TAI",
"type": "S",
"exchDisp": "Taiwan"
},
{
"symbol": "0294.HK",
"name": "YANGTZEKIANG",
"exch": "HKG",
"type": "S",
"exchDisp": "Hong Kong"
},
{
"symbol": "YAVY",
"name": "Yadkin Valley Financial Corp.",
"exch": "NMS",
"type": "S",
"exchDisp": "NASDAQ"
}
]
}
}
)
Which is JSON and very easy to work with.
Hush... don't tell anybody.
Google Finance does let you retrieve up to 100 stock quotes at once using the following URL:
www.google.com/finance/info?infotype=infoquoteall&q=[ticker1],[ticker2],...,[tickern]
For example:
www.google.com/finance/info?infotype=infoquoteall&q=C,JPM,AIG
Someone has deciphered the available fields here:
http://qsb-mac.googlecode.com/svn/trunk/Vermilion/Modules/StockQuoter/StockQuoter.py
The current price ("l") is real-time and the delay is on par with Yahoo Finance. There are a few quirks you should be aware of. A handful of stocks require an exchange prefix. For example, if you query "BTIM", you'll get a "Bad Request" error but "AMEX:BTIM" works. A few stocks don't work even with the exchange prefix. For example, querying "FTWRD" and "NASDAQ:FTWRD" both generate "Bad Request" errors even though Google Finance does have information for this NASDAQ stock.
The "el" field, if present, tells you the current pre-market or after-hours price.
You can send an HTTP request to http://finance.yahoo.com requesting symbols, names, quotes, and all sorts of other data. Data is returned as a .CSV so you can request multiple symbols in one query.
So if you send:
http://finance.yahoo.com/d/quotes.csv?s=MSFT+F+ATT&f=sn
You'll get back something like:
"MSFT","Microsoft Corp"
"F","FORD MOTOR CO"
"ATT","AT&T"
Here is an article called Downloading Yahoo Data which includes the various tags used to request the data.
The NASDAQ site hosts separate CSV lists for ticker symbols in each stock exchange (NYSE, AMEX and NASDAQ). You need to complete the captcha and get the CSV dump.
http://www.nasdaq.com/screening/company-list.aspx
If you didn't want to sign up for a service, I'd probably go back to the exchanges themselves; most of them aren't CAPTCHAed yet...
The symbol lookup page for:
NYSE is at http://www.nyse.com/interface/html/SymbolLookup.html
NASDAQ is at http://www.nasdaq.com/asp/NasdaqSymLookup2.asp?mode=stock
London Stock Exchange is at http://www.londonstockexchange.com/en-gb/pricesnews/prices/Trigger/genericsearch.htm
ASX is at http://www.asx.com.au/asx/research/codeLookup.do
etc...
Use YQL and you don't need to worry. It's a query language by Yahoo and you can get all the stock data including the name of the company for the ticker. It's a REST API and it returns the results via XML or JSON. I have a full tutorial and source code on my site take a look: http://www.jarloo.com/yahoo-stock-symbol-lookup/
Currently, the NASDAQ web site publicly provides CSV files containing bulk listings -- it is broken up by first letter.
http://www.nasdaq.com/screening/companies-by-name.aspx?letter=A&render=download
Google Finance has an API - you probably have to apply for a developers key, but at least you'd save yourself the hassle of screen-scraping: http://code.google.com/apis/finance/reference.html
Your best bets are probably going with one of the other lookup services (still screen-scraping), and checking whether they don't require CAPTCHAs.
Yahoo Finance
MSN Money
AlphaTrade Finance
The last appears the least likely to require a CAPTCHA at any point, but it's worth checking all three.
Use YQL: a sql-like language to retrieve stuff from public api's:
YQL Console (external link)
It gives you a nice XML file to work with!
You can use the "Company Search" operation in the Company Fundamentals API here: http://www.mergent.com/servius/