I'm fairly new to programming in general but I have delved into vb.net recently. I was looking into how to grab data from a website but don't fully comprehend how to or through what means.
My main end game is to just pull data for a lack of better word. Say for instance, a website shows text, I'd like to be able to pull that text into my program using vb.net. Is that possible?
I don't know how to use javascript, php and know only little of html/css. I'm of course willing to learn but I haven't had much luck searching for this specific information or solution. Without access to the websites database, is there another way for my app to read the website for what it would be displaying and retrieve this information? I'm mainly concerned with text, no other information is really needed. From what i've gathered, the information is tagged by something like an ID, or class? Any help would be appreciated.
Web servers essentially have 2 operations, POST, which is you sending something to the website, and GET, which is you requesting something from the website.
For the most part you can just do a GET of some sort on the website and it'll return what you request in a data format, likely JSON. You can often look for fields based on CSS selectors in the web page itself. Though, there are other ways and some HTTP libraries may return the data to you in various formats, mainly JSON and XML.
I'd suggest right click->view source on a web page to learn more about how they're structured. As for how to do a GET in javascript on some website, this code would grab the website at the URL you give it.
function httpGet(theUrl)
{
var xmlHttp = new XMLHttpRequest();
xmlHttp.open( "GET", theUrl, false ); //the get request
xmlHttp.send( null );
return xmlHttp.responseText; //the website data in XML format.
}
Related
I use quicken, which can automatically download bank of America transactions. However, it truncates all the payees so I lose data. I'd like to work around this and I'm thinking of downloading the transaction data and generating my own QFX file with the full payee info.
Is there a way that I can download transactions programmatically, or download something like a .qif (available on their website) programmatically? For the latter, I could convert the gif to a QFX myself.
If anyone has other ideas to download all of the transaction information without losing the payee info, I would welcome those ideas as well.
Do they provide an api for this? but most probably not for 3p without a contract. since its bank , there must be check for browser etc along with standard sign in so it'll hard for curl. you can have a browser plugin to read all the data from the page and do auto scroll to get new transactions if not fitting in page. it's a hacky solution but good to get what we need as you told that data is available on the page and have to revisit with updates but changes in basic structure is rare.
A quick search for bank of america api yielded this BofA API. They even have many options for types of payment information you could query here as well as lots of individual account types that you can access it as.
It looks pretty comprehensive. If you don't see what you are looking for there I put another option below, just in case.
I don't use BofA. So I can't speak to what they have natively available. But you could always use a bot to scrape it if they present it anywhere in the User Interface.
I would agree with Meena that you should not be able to use curl. But selenium uses a browser to programmatically do just about anything that you would want to do with any website. They also have bindings for many languages. So you could just pick your favorite and go to town...
It seems the API will return a JSON so you may need to find a tool to convert that to a qif or qfx if that part is important. After digging further, I can't test this without having a CashPro account but it seems what you need to do is...
Step 1:
Get an access token from here. You'll need to send this in the header of any requests
Step 2:
Send an http request with a header in the following format:
{
"accounts": [
{
"accountNumber": "xxxxxxx",
"bankId": "xxxxxxx"
}
],
"fromDate": "yyyy-mm-dd",
"toDate": "yyyy-mm-dd"
}
to https://developer.bankofamerica.com/cashpro/reporting/v1/transaction-inquiries/previous-day
Step 3:
You should get a JSON as a response
As mentioned, I can't test this but here's the documentation of the specific API endpoint you need
I am writing an app that asks for some data from a web-based form and then displays the returned JSON data. But I can't seem to get a handle on how "fill in" the form and get the data returned. The website developer has made a development form for testing this so I can see how it works. If I go to that website and enter data into the fields and press "Submit", it returns JSON-formatted text and displays it in the web browser's window.
What I need to do is have my app populate those fields with data and submit them to the server. The server returns the JSON data and I capture and process it in my app.
I would appreciate any advice on how to do this. As you may have guessed, I am a very new to iOS development. Thanks in advance.
Firstly, you'll need to know what form data you're going to send in a POST request. If you don't know where to find that, take a look at this.
Consider using AFNetworking to make that POST request with JSON feedback. Here's a good example.
Good luck!
This walkthrough by Matt Long is particularly good: http://www.cimgf.com/2010/02/12/accessing-the-cloud-from-cocoa-touch/.
Essentially, you just need your fields in the post data to match the fields that are present in the html form. Those fields should be evident by posting the form in chrome and watching the request in the network tab in developer tools.
I have a product code which I need to enter into 6 different websites in order to pull different information from them about the product. Is there away to save this product code into some sort of variable and pass it into each websites input box and it return all the information from each one automatically? Really have no idea where to go/start with this so if anyone can brainstorm a few ideas to get me moving that would be great.
In order get what you are planning for:
You need a script which visits the specified web site,
then at the website, you can get the element by tag.
For instance in javascript,
var textBox = document.getElementByTag(Input);
This will give you a reference to text field to enter the text. It can be done as follows:
textBox.value = "any string";
Once you have done this, you will have to retrieve the results from the page, based on the website layout.
So if you can specify about your work in detail, you would get better response.
Assuming you're talking about using an ordinary GUI browser, the best you can do is copy it to your system clipboard, and paste it into each page on the browser.
If you're talking about a programmatic web-access like wget or curl, it depends on what language you are writing your script in.
you have to create the web request for each web site and find a way to parse the response which will be HTML
have a look at the HttpWebRequest you can find lots of example on internet that shows how you can create an HTTP POST to a website.
http://www.terminally-incoherent.com/blog/2008/05/05/send-a-https-post-request-with-c/
could someone point me in the right direction in how i can parse data from a blog to an iphone. E.g. You have a table view displaying the posts of the blog, you select a table cell and text content is displayed. Are there any tutorials/examples on this?
I have a bit of experience with parsing data using JSON (parsed data from database to iphone) but unsure on where to start with this?
Thanks for an help..
What you want to do is use the Wordpress API. The flow goes like this:
Make an API call. This is a subject unto itself, but typically you'd use NSHTTPRequest and NSURLConnection to make the request.
Parse the result. I forget if you get XML back; you probably do; in that case there are tons of solutions for parsing, including NSXMLParser and libxml2.
Populate UITables, etc. with the retrieved information.
The Wordpress API is not that big or complex, but it's a bigger subject than I can really get into in the context of an SO answer, so I'll just refer you to the aforelinked documentation and wish you happy reading.
I want to be able to retrieve dynamic data from a web page (share prices). I started out by retrieving the html code before I realised that as it is live data, the html code will be of little use. Although I am looking to capture specific data, all i wish to do is process a webpage that I specify which will return the text off that website and not the HTML code. Basically a copy and paste of the entire page would be great..
Any ideas would be really appreciated!
'Screen Scraping' by parsing HTML is so early 2000s...what I would do is read up on Amazon's Mechnical Turk. You can develop a queued architecture where you submit urls to this Mechnical Turk service. The service would automatically distribute these bits of work to users who would then do the dirty task of copying and pasting out the valuable stock quote information you require. Users around the world would anxiously await delivery of the next URL to their Mechanical Turk inbox...pinning for the opportunity to copy/paste out another share price for your application. Sure, it might take a few minutes to update your prices, but hey, they would be HAND parsed by REAL people around the globe! Just think of the possibilities!
Well, the HTML contains the text of the website, so you "just" need to parse the HTML.
EDIT: If the data is not in the HTML but loaded dynamically, the situation is different. As I can see, you have two options:
Find out how the data is loaded (i.e. read the JavaScript on the page). If it is updated via some web service, you could query the same web service in your program.
Use a web browser to get the data and then get the dynamic HTML tree of the page. Maybe the WPF Webbrowser control can help you with this, but I'm not sure since I've never done this myself.
Is it possible to find this same data provided in a ready-to-consume format rather than scraping HTML for it? It seems like there's probably public web-services for stock quotes.
For example: A quick search for "Stock price webservice" turned up http://www.webservicex.net/stockquote.asmx; an ASMX web-service that is easy to consume in .NET.
In your Visual Studio project you should be add a reference to this service via the "Add Web Reference" command; the dialog you're given varies depending on whether your project is targeting for .NET 2.0 or .NET 3.0/3.5.
I added a reference to the service named StockPriceProxy:
Public Function GetQuote(ByVal symbol As String) As String
Using quoteService As New StockPriceProxy.StockQuote
return quoteService.GetQuote(symbol)
End Using
End Function