I'm trying to find a way around using IE in my VBA code to access my portfolios in finance.yahoo.com but I'm getting the same message using a soap/rest? format as I do when I open an IE window: "Some parts of this page is not supported on your current browser version. Please upgrade the browser to the latest." It looks like my new calls are being processed by IE also. Is that true? Is there a way around this?
I tried using Selenium to open Firefox but it isn't working with the current version of Firefox (67)
here's the code I'm using now:
Dim xmlhttp As new MSMXML2.XMLHTTP60
Dim WebPgDoc As New HTMLDocument
Dim objNode As object
With xmlhttp
.Open "GET", "https://finance.yahoo.com/portfolio/pf_14/view/view_18", False
.send
WebPgDoc.body.innerHTML = .responseText
End With
Set objNode = WebPgDoc.getElementsByTagName("div")
When I look at objNode(70).innertext it has the same error message and missing stock details as when I use code to open IE to open the portfolio .
Any ideas would be greatly appreciated.
Thanks
Related
I'm trying to make a webscraping program in Visual Studio 2019 using VB.Net Framework but I'm running into an issue. I have the below code to locate an open IE window so that I can manipulate it. The issue I'm running into is when I use IE.Document. the list that appears does not show getelementsbyid. Here is a snippet of my code, it's able to find the appropriate IE window and bring it into focus but I need to be able to manipulate the elements on the page.
I have also added the Com References
Microsoft HTML Object Library
Microsoft Internet Control
Dim SWS as new SHDocVw.ShellWindows
Dim IE as SHDocVw.InternetExplorer
For Each IE in SWS
If IE.locationName = "XXXXXXXX" Then Exit For
NExt
'To get focus on IE window
IE.Visible = False
IE.Visible = True
IE.Document.
I would appreciate any insight since I'm at a loss as to why getelementbyid won't show.
As an alternative, you can try to refer to the code example below that may help you to get/ set values.
If you want to set data then refer to the line below.
IE.Document.All.Item("fname").Value = "ABC"
If you want to fetch data then refer to the line below.
Console.WriteLine(IE.Document.All.Item("fname").Value)
VBA code to interact with specific IE window that is already open
Above is a thread to find and go to an already open instance of IE using shell applications in VBA. AFTER I found the open IE instance I am looking for, I need to query the tables from that IE page without using it's URL. The reason that I cannot use it's URL is that this IE page is a generic 'result' page that opens in a separate window after doing a search on the main website, so if I use the URL of the result page, which is: https://a836-acris.nyc.gov/DS/DocumentSearch/BBLResult, it will return an error. Are there any other methods that allow querying tables without using URL connections, like a "getElements" for tables?
K.Davis, Tim William: you are correct in your assumptions. The first part of my code/project opens up a search page: objIE.navigate "https://a836-acris.nyc.gov/DS/DocumentSearch/BBL" and through it I submit a search form. The second part (outlined above in the first paragraph) opens up a result page (pop-up). I am trying to automate the retrieving of the tables from that page. I tried using QueryTables.Add method, the way I am familiar with to connect to the data/webpage requires an URL. If I use the URL from the result page it returns an error, thus I am looking for suggestions/help on how I could otherwise connect. That said I am able to retrieve elements of the page using 'getElements' method but not able to query tables. There are other ways to connect to the data source using the QueryTables.Add method, see, https://learn.microsoft.com/en-us/office/vba/api/excel.querytables.add but I am not familiar with these other methods. Hope this clarifies a bit.
I haven't experienced a problem with this as although you have an intermediate window the final IE window resolves to being the main IE window with focus. I was able to grab the results table with the following code using the indicated search parameters:
Option Explicit
Public Sub GetInfo()
Dim IE As New InternetExplorer
With IE
.Visible = True
.navigate "https://a836-acris.nyc.gov/DS/DocumentSearch/BBL"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document
.querySelector("option[value='3']").Selected = True
.querySelector("[name=edt_block]").Value = 1
.querySelector("[name=edt_lot]").Value = "0000"
.querySelector("[name=Submit2]").Click
End With
While .Busy Or .readyState < 4: DoEvents: Wend
Dim hTable As HTMLTable
Set hTable = .document.getElementsByTagName("table")(6)
'do stuff with table
.Quit
End With
End Sub
You can copy a table via clipboard. Any tick windings appear in the right place but as empty icons.
For clipboard early bound go VBE > Tools > References > Microsoft-Forms 2.0 Object Library.
If you add a UserForm to your project, the library will get automatically added.
Dim clipboard As DataObject
Set clipboard = New DataObject
clipboard.SetText hTable.outerHTML
clipboard.PutInClipboard
ThisWorkbook.Worksheets("Sheet1").Cells(1, 1).PasteSpecial
Late bound use
Dim clipboard As Object
Set clipboard = GetObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
The following Excel macro, which is making an xmlhttp request to this webpage to retrieve some values at a second stage, has worked normally in VBA until some time ago:
Sub WebReq()
Link = "http://it.finance.yahoo.com/q?s=^FCHI&ql=10" & str(rnd())
Set htm = CreateObject("htmlFile")
Set RequestWeb = CreateObject("msxml2.xmlhttp")
With RequestWeb
.Open "GET", "" & Link & "", False
.send
htm.body.innerhtml = .responsetext
End With
End Sub
Now, instead, at the call of the method:
.send
of the object msxml2.xmlhttp is raising the following error:
Run-time error '-2147024891 (80070005)'
Access is denied.
I've been looking on the web but all the similar threads are never answered. Can anyone explain me what this error means, and if there's any way I could fix it or even just work around it?
Note: the random string at the end of the variable 'Link' has been added to force the page reloading, since the script is retrieving real-time values and so it should be loaded every time.
Additional information: while looking for a solution, I'm noticing now that the random part of the link is yielding always the same value even when I end the running and restart again:
Link = http://it.finance.yahoo.com/q?s=^FCHI&ql=10 .7055475
Why is this happening? Shouldn't rnd() yield a new random value between 0 and 1 at every call?
Use
CreateObject("MSXML2.ServerXMLHTTP.6.0")
The standard request fired from a local machine forbids access to sites that aren't trusted by IE. MSXML2.ServerXMLHTTP.6.0 is the server-side object, that doesn't perform those checks.
i found that, in my case, changing http to https fixed the access denied problem. i can only assume that the website somehow made a change and didn't tell anyone
access denied is IE issue
internet options > security tab > custom security level > Miscellaneous >Access data sources across domains > enable
Update
Sub WebReq()
link = "http://it.finance.yahoo.com/q?s=^FCHI&ql=10" & Str(Rnd())
Set htm = CreateObject("htmlFile")
Dim objHttp
Set objHttp = CreateObject("Msxml2.ServerXMLHTTP")
objHttp.Open "GET", link, False
objHttp.Send
htm.body.innerhtml = objHttp.responsetext
Set objHttp = Nothing
End Sub
This works for me:
With CreateObject("MSXML2.ServerXMLHTTP.6.0")
.Open "GET", URL, False
.Send
content = .ResponseText
End With
In my case the user didn't have AD permissions to the proxy server on our corporate network. (Simple oversight when setting up the user.) Adding the missing security group fixed the problem for the user.
I was able to fix it by changing the link being passed from being "http://" to "https://"
The site I was pulling had upgraded and trying to pull the data using the unsecured link was failing. Works great now (no code change required.
add "www" after "https://" in your custom Link,
Like this:
XMLPage.Open "GET", "https://www.x-rates.com/table/?from=GBP&amount=3", False
XMLPage.send
I'm afraid I don't understand exactly why this problem occurs, but I'm guessing it is the secure "https://" versus insecure "http://". I ran into the same "access denied" message while following sample code from a VBA course. The original code was:
XMLPage.Open "GET", "http://x-rates.com/table/?from=GBP&amount=3", False
XMLPage.send
I changed the "http://" to "https://" and the error went away.
I've searched the web and can't find the exact thing that I'm looking for - this could be because it doesn't exist, but I'll ask here anyway...
I want to use a pretty simple from written using VB, I've worked with things like RSS feeds before, but only in a HTML environment. I was wondering if there is a way to have a section of the form as a USD -> GBP converter, using a live exchange rate. Is this possible? If so, does anyone know a good source to get the live feed from?
Any ideas, code, suggestions and criticism is welcome.
Thanks for your time.
Cal.
The simplest way is to query the Yahoo currency converter using this function:
Public Function GetHTML(ByVal sURL As String) As String
'Function returns the contents of the web page at sURL
'(or the contents of the 404/error info, if sURL is
'invalid, incorrectly formatted, etc.)
Dim xmlHttp As Object
Set xmlHttp = CreateObject("MSXML2.ServerXmlHttp")
xmlHttp.Open "GET", sURL, False
xmlHttp.send
GetHTML = xmlHttp.responseText
Set xmlHttp = Nothing
End Function
Notice that it's using ServerXmlHttprather than just XmlHttp. This is because the latter was returning an Access denied error when I tried it, I'm not sure why.
Call it from your form code using something like this:
lblRate.caption = GetHTML("http://finance.yahoo.com/d/quotes.csv?&f=l1&s=USDGBP=X")
Note:
AFAICT there isn't any official documentation on the Yahoo API. But there are plenty of examples of it's usage if you search online.
I am writing a macro that will scrape my company's internal SAP site for vendor information. For several reasons I have to use VBA to do so. However, I cannot figure out why I keep getting these three errors when I attempt to scrape the page. Is it possible that this has something to do with the UAC integrity model? Or is there something wrong with my code? Is it possible for a webpage using http can be handled differently in internet explorer? I am able to go to any webpage, even other internal webpages, and can scrape each of those just fine. But when i attempt to scrape the SAP page, i get these errors. The error descriptions and when they occur are:
800706B5 - The interface is unknown (occurs when I place breakpoints before running the offending code)
80004005 - Unspecified error (occurs when I don't place any errors and just let the macro run)
80010108 - The Object invoked has disconnected from its clients. (I can't seem to get a consistent occurrence of this error, it seems to happen around the time that something in excel is so corrupted that no page will load and i have to reinstall excel)
I have absolutely no idea what is going on. The Integrity page didn't make much sense to me, and all the research I found on this talked about connecting to databases and using ADO and COM references. However I am doing everything through Internet Explorer. Here is my relevant code below:
Private Sub runTest_Click()
ie.visible = True
doScrape
End Sub
'The code to run the module
Private Sub doTest()
Dim result As String
result = PageScraper.scrapeSAPPage("<some num>")
End Sub
PageScraper Module
Public Function scrapeSAPPage(num As Long) As String
'Predefined URL that appends num onto end to navigate to specific record in SAP
Dim url As String: url = "<url here>"
Dim ie as InternetExplorer
set ie = CreateObject("internetexplorer.application")
Dim doc as HTMLDocument
ie.navigate url 'Will always sucessfully open page, regardless of SAP or other
'pauses the exection of the code until the webpage has loaded
Do
'Will always fail on next line when attempting SAP site with error
If Not ie.Busy And ie.ReadyState = 4 Then
Application.Wait (Now + TimeValue("00:00:01"))
If Not ie.Busy And ie.ReadyState = 4 Then
Exit Do
End If
End If
DoEvents
Loop
Set doc = ie.document 'After implementation of Tim Williams changes, breaks here
'Scraping code here, not relevant
End Function
I am using IE9 and Excel 2010 on a Windows 7 machine. Any help or insight you can provide would be greatly appreciated. Thank you.
I do this type of scraping frequently and have found it very difficult to make IE automation work 100% reliably with errors like those you have found. As they are often timing issues it can be very frustrating to debug as they don't appear when you step through, only during live runs To minimize the errors I do the following:
Introduce more delays; ie.busy and ie.ReadyState don't necessarily give valid answers IMMEDIATELY after an ie.navigate, so introduce a short delay after ie.navigate. For things I'm loading 1 to 2 seconds normally but anything over 500ms seems to work.
Make sure IE is in a clean state by going ie.navigate "about:blank" before going to the target url.
After that you should have a valid IE object and you'll have to look at it to see what you've got inside. Generally I avoid trying to access the entire ie.document and instead use IE.document.all.tags("x") where 'x' is a suitable thing I'm looking for such as td or a.
However after all these improvements although they have increased my success rate I still have errors at random.
My real solution has been to abandon IE and instead do my work using xmlhttp.
If you are parsing out your data using text operations on the document then it will be a no-brainer to swap over. The xmlhttp object is MUCH more reliable. and you just get the "responsetext" to access the entire html of the document.
Here is a simplified version of what I'm using in production now for scraping, it's so reliable it runs overnight generating millions of rows without error.
Public Sub Main()
Dim obj As MSXML2.ServerXMLHTTP
Dim strData As String
Dim errCount As Integer
' create an xmlhttp object - you will need to reference to the MS XML HTTP library, any version will do
' but I'm using Microsoft XML, v6.0 (c:\windows\system32\msxml6.dll)
Set obj = New MSXML2.ServerXMLHTTP
' Get the url - I set the last param to Async=true so that it returns right away then lets me wait in
' code rather than trust it, but on an internal network "false" might be better for you.
obj.Open "GET", "http://www.google.com", True
obj.send ' this line actually does the HTTP GET
' Wait for a completion up to 10 seconds
errCount = 0
While obj.readyState < 4 And errCount < 10
DoEvents
obj.waitForResponse 1 ' this is an up-to-one-second delay
errCount = errCount + 1
Wend
If obj.readyState = 4 Then ' I do these on two
If obj.Status = 200 Then ' different lines to avoid certain error cases
strData = obj.responseText
End If
End If
obj.abort ' in real code I use some on error resume next, so at this point it is possible I have a failed
' get and so best to abort it before I try again
Debug.Print strData
End Sub
Hope that helps.