I have a Web application that needs to display up to date information on REITs and tickers like AX.UN, BEI.UN, CAR.UN etc..
For example, I need to automate consumption of information on pages such as
http://ca.finance.yahoo.com/q?s=AX-UN.TO
Are there rss feeds or apis I can use to import this kind of data? I don't want to copy and paste this information into my website on a daily basis.
On the very site you link to, there's a small link that says "download data". If you have a database with the symbols you want to track, it would be pretty easy to construct that download URL on the fly, query it, parse the CSV, and load that data into your database for display on your website.
ETA: Here's the link:
http://ca.finance.yahoo.com/d/quotes.csv?s=AX-UN.TO&f=sl1d1t1c1ohgv&e=.csv
Just have your program replace the "AX-UN.TO" with whatever symbol you want, and it should grab the data for that symbol.
Take a look at http://www.mergent.com/servius - particularly the Historical Securities Data API. Not sure if it has REIT data - if it doesn't, it may be available by special arrangement from them.
Related
I am trying to get the twitter name, website, and markets for any cryptocurrency listed on coinmarketcap.
For example:
https://coinmarketcap.com/currencies/bitcoin shows all of the data that I need but how would I parse the data listed on that page to get the twitter name and website associated with bitcoin?
Don't try to parse the data on that page if you are trying to use it in an application. If you are doing it for a one time data collection then maybe that is OK. There is absolutely no guarantee that the format will remain consistent or that the information will even be there tomorrow.
You should try to find an API that gives you the information that you are looking for. An API is a contract that is expected to be honored and therefore is reliable. A quick look at CoinMarketCap's API doesn't appear to have the info you are looking for but maybe another one exists that does.
If you were to parse the HTML you could write a regex for the specific thing you are looking for. For example if you want to get the website you could write a regex that would pick up the pattern:
Website
the capture group ([^"]+) is the website in this example. You could do something like that for every element you want to get.
Is it possible to create a Kapow Robot that can search Google for the Operating hours of the Businesses from our list/database and update the timings if changes are made?
Please share if there are any other more efficient ways than the KAPOW robot that can be implemented with minimal effort and cost-effectiveness.
That's what the Google Places API is there for. While you could in theory just open Google Maps in a Load Page action, enter the query string and then parse the results, I would advise against it. Here's why:
The API will be faster, returning results in a structured manner (JSON)
Kapow has actions for calling RESTful services and parsing/modifying JSON
Google does not like robots parsing their pages, and most likely will lock you out (i.e. present you with Captchas sooner or later)
If you decide to go for the API, here's what you should do:
Get your API key first, see this page for details: https://developers.google.com/places/web-service/get-api-key. Note that the free plan allows for 1,000 requests within a 24-hours limit (https://developers.google.com/places/web-service/usage)
Maintain the place ids for all the businesses you'd like to query regularly, and update your list.
For each place, retrieve the details as described in the API documentation. The opening hours will be within the JSON response: https://developers.google.com/places/web-service/details
Update your list. I'd recommend using a definite type in Kapow for that, and using the actions Store in Database and Query Database. In case you need the data elsewhere, you may create additional robots (e.g. for Excel files, sending data per email, et cetera).
Total API newbie here. I have an Excel file of restaurants in my city. For each restaurant, I have its name, address, city, state, zip code, and coordinates. I would like to retrieve additional information about these restaurants, like their ratings and price levels, and add those variables to my datafile. Is this something I could do through Yelp Fusion API? I've tried googling my question but I am still unclear if this is possible or not. I'm also unsure on how to code this because I only know R and there is very little example code for Yelp Fusion API in R.
The answer to your question is Yes, you can use Yelp Fusion API to pull yelp information relating to those businesses in your excel file.
With that said, I have no experience with the R language, but I do know it can be done in javascript without using any fancy libraries (except for accessing the Excel file), so my guess is you could probably pull it off with R as well.
Here is my non-professional-programmer algorithm for solving your problem...
Extract each business from the spreadsheet and place them in a data structure.
Construct the parameters for your API call in JSON format.
Loop through each business and separately send each request to Yelp.
Save each response from the Yelp API to a new data structure, such as an array.
Parse the desired information from each response and then either: 1) add it to your source spreadsheet or 2) combine the existing spreadsheet data with the new yelp data and then save to a new spreadsheet.
There is probably a better way to do this, but this is the approach I would take.
I have a project that involves having public data downloaded from Google plus, can you give me a reference on how I can download like 1 GB of any type of public data from Google plus?
The data can be posts or circles information. I've tried to work with developer tools but the far I got is downloading my own profile information but what I need is public data.
Thanks !
There is no truly "public" data on Google+.
Every stream is unique to a user.
Try viewing the site without logging in, and you'll see what I mean.
Since users have the ability to block other users from viewing even their "public" posts, before Google shows you a post they check to see if you're on the blocked list. For them to be able to do that, you have to be logged in.
Your best bet would be to create a dummy account and only look at your nearby stream or What's Hot.
Otherwise you'd need to circle users, and that would create the stream. G+ is not like twitter. There's no firehose to speak of.
To programmatically cull data, you would have to use their API, but even then their HTTP API limits you to 20 results per search and you have to provide a query.
You could get up to 100 results per user if you picked individuals and got their userids, but again there's not a programmatic way to get a bulk dump.
You could randomly select users by using an activity search for a dictionary entry, and then seed that into the activity listing api... something like (in pure pseudocode)
for Random word in dictionary
group = userids from GET https://www.googleapis.com/plus/v1/activities?query=[word]
for userid in group
GET https://www.googleapis.com/plus/v1/people/[userid]/activities/collection/public
Actual code would of course depend on the language.
I'm trying to find out if there is a programmatic way to determine how far down in a search engine's search results my site shows up for given keywords. For example, my query would provide my domain name, and keywords, and the result would return a say 94 indicating that my site was the 94th result. I'm specifically interested in how to do this with google but also interested in Bing and Yahoo.
No.
There is no programmatic access to such data. People generally roll out their own version of such trackers. Get the Google search page and use regexes to find your position. But now different results are show in different geographies and results are personalize.
gl=us parameter will help you getting results from US, you can change geography accordingly to get the results.
Before creating this from scratch, you may want to save yourself some time (and money) by using a service that does exactly that [and more]: Ginzametrics.
They have a free plan (so you can test if it fits your requirements and check if it's really worth creating your own tool), an API and can even import data from Google Analytics.