Script for Real Time Quotes from Fidelity - vba

By using the Macro Recorder, and then making a few modifications, I have set up a macro to navigate to the Fidelity stock quote page for multiple quotes, and download delayed quotes. This works well. However, if I am logged in to Fidelity, and access that same page manually, I can obtain real-time quotes. The "login" can take place either before or after I access the page. I have not been able to figure out how to add the login process to the VBA routine that handles everything else. The macro recorder, started before I select the "get external data from Web option, does not pick up the login process when accessing that page for real time quotes.
Here is the part of the function that actually calls the stock quote page and requests the quotes:
With Worksheets("fidotemp").QueryTables.Add(Connection:=Connection, _
Destination:=Range("a1"))
.name = Symbol
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = IIf(bSymbol, "12", "9")
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
The Connection parameter, named Connection, is the string which includes the particular page, along with a properly formatted string to obtain one or more quotes. When multiple quotes are requested, the ticker symbols are '+' separated and limited to a 100 character length.
A sample Connection string might look like:
URL;https://fastquote.fidelity.com/webxpress/get_quote?QUOTE_TYPE=D&SID_VALUE_ID=VIP+SAN+E+MT+BCE+RAI+SNP+BP+MO+PTR&submit=Quote

Is logging in from www.fidelity.com acceptable? If so, does this get you in?
Sub test()
' open IE, navigate to the Fidelity log-in webpage of interest and loop until fully loaded
Set IE = CreateObject("InternetExplorer.Application")
login_url = "https://www.fidelity.com"
With IE
.Visible = True
.navigate login_url
.Top = 50
.Left = 130
.Height = 800
.Width = 800
End With
Do Until Not IE.Busy And IE.ReadyState = 4
DoEvents
Loop
' Enter your username and pw
IE.document.getElementById("temp_id").Value = "abcde"
IE.document.getElementById("PIN").Value = "12345"
' Click the login button
IE.document.getElementById("logButton").disabled = False
IE.document.getElementById("logButton").Click
End Sub

Related

Delete connection in Excel using VBA after web scraping

I have a connection that web scrapes based on a dynamic link. So, I cannot set a fixed connection. The following macro creates the connection and then updates the worksheet.
With ThisWorkbook.Worksheets("Data").QueryTables.Add(Connection:= _
"<URL redacted>", Destination:=ThisWorkbook.Worksheets("Data").Range("$A$1"))
.Name = "DataPull"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = False
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
.Delete
End With
This macro runs every 1 minute to update the data. So, this creates a new connection every time it runs. I do not want so many connections to exist as they won't be used anymore.
How do I delete the connection once web scraping is complete?
or Is there a way to set up a single connection that can modify itself based on a variable. The variable is the time intervals which vary based on current time.
I looked at this option
For Each qr In ThisWorkbook.Queries
qr.Delete
Next qr
But there are two other fixed connections which I don't want to delete.
The new connections that are created have the names Connection, Connection1, and so on. Is there a way to delete the connections based on name?
Let's say you want to delete all connections, except Connection1 and Connection2, try...
Dim Conn As WorkbookConnection
For Each Conn In ThisWorkbook.Connections
If Conn.Name <> "Connection1" And Conn.Name <> "Connection2" Then
Conn.Delete
End If
Next Conn
Hope this helps!

Issue getting data from web table using Excel VBA

I need to connect to a website, with credentials then click a link and get the table it brings to my excel sheet.
I'´m already logged in the site and with the following code I get the table updated but this only works while the excel"from web " connection is alive. I mean, first I have to update the DATA-FROM WEB connection refreshing credentials and then running the macro.
Though the web table is always reached and updated I can't get it pasted to the excel sheet if that connection was updated more that an hour ago.
Here is the code:
Sub Descargar_tabla()
Dim IE As Object
Set IE = CreateObject("WScript.Shell")
IE.Run ("""C:\Program Files (x86)\Opera\launcher.exe"" ""https://clientes.invertirenbolsa.com.ar/cart.aspx""")
With ActiveSheet.QueryTables.Add(Connection:="URL;https://clientes.invertirenbolsa.com.ar/cart.aspx", Destination:=Range("$L$40"))
' .Name = "cart"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = False
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlOverwriteCells
.SavePassword = True
.SaveData = True
.AdjustColumnWidth = False
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "2"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=True
Set IE = Nothing
End With
End Sub
What should I add to the code to use the website open session that I already have in the browser which it doesn't expire?

Retrieve Final Webpage URL in Microsoft Excel Visual Basic

I had a question about how to retrieve the contents of a final URL in Microsoft excel, using their Visual BASIC macros.
Essentially, I have a list (ListA) full of URLs. I have code written to trawl every URL on List A one by one and retrieve the data I need, putting it into Excel.
However, a certain percentage of the URLs redirect to 404 pages. I do not have any way of knowing which ones these will be in advance currently, and am trying to write a script that will simply:
1.) Access the URL in ListA
2.) Copy the destination URL that it redirects to
3.) Paste that URL into the cell directly to the right of the original URL
That way I can see what the final URLs are, and if any go to a 404 page, I can delete them from the list before attempting to trawl it for the info I need.
I have had no luck in doing so thus far, and every tutorial I can find online seems to feature code that will not work in Microsoft Excel's limited environment. Does anyone have any idea where I should start?
If it helps, here is the code I have written to trawl the webpage for data:
For i = 1 To 500
ThisURL = "URL;" & WSD.Cells(i, 2)
ThisParcel = "P" & WSD.Cells(i, 1)
Set WSW = Worksheets.Add(After:=Worksheets(Worksheets.Count))
WSW.Name = ThisParcel
WSW.Select
' Do a web query here
With ActiveSheet.QueryTables.Add(Connection:= ThisURL, Destination:=Range("$A$1"))
.Name = "Query" & i
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
Next i

Copy table (with no id) from password protected website

I would like to retrieve a table from my bank's website, and paste it in the same format in an excel sheet.
Thanks to a few posts on this very useful website, I managed to write a vba code that automatically :
Open the Internet Explorer windows
Fills in the Log-in and Password details
Clicks on the submit Button, thus connecting me to my bank account
Clicks on the "previous day transaction report" button, thus generating the report
And finally shows this report in Internet Explorer
Now my problem is :
How can I copy this report (which is a table) and paste it in an excel sheet, with the Exact same format as it appears on the website ?
On the html code of the website page, this table has no "id". But it has a class, called "report". And it is the only one who has this class.
I suppose I have to use this :
IE.document.getElementsByClassName("report")(0).outerHTML
But I am not sure how to use it... Basically the simplest thing for me would be to write a last portion of code that copies the report to the clipboard, and paste it with the same format on the spreadsheet.
Any idea how I could do that?
i recorded a macro to get the code, using this should copy the table for you. I think its the best way for you. You have to change the URL and the webtable. This will copy it.
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;http://THE URL AFTER LOGIN WITH THE TABLE", _
Destination:=Range("$A$1"))
.CommandType = 0
.Name = "SEQUENCE OF URL"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "11"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
Lets make this work!! :) step by step.
You are right. Fill a form using it.
Try open the page, press F12 and use the console to make sure everything is right.
To fill
document.getElementById("Id").value = "xxxxx"
document.getElementById("Id2").value = "pass"
To click
document.getElementById("name").click()
Set objie = CreateObject("InternetExplorer.Application")
objie.Visible = True
With objie
.AddressBar = False
.StatusBar = False
.MenuBar = False
.Toolbar = 0
.NAVIGATE "http://www.loginpage.com"
While .BUSY
Wend
Do While .READYSTATE <> 4: DoEvents: Loop
'first you login
Set htmldoc = .document
htmldoc.getElementById("UserId").Value = "login"
htmldoc.getElementById("Password").Value = "pass"
htmldoc.getElementById("submit-btn").Click
End With
Try using this one, its first step... Makes sense to you? Im new here, so i will do my best

Using Data from Web, how to login?

I have a macro that uses the "Data from Web" function. I have logged into the website (in Internet Explorer) that I'm pulling data from - but the results I get in Excel just keep telling me I'm not logged in.
Is there a special way to login via Excel for "Data from Web"? I know it works, as I used the Macro Recorder to learn how Excel gets the data - and doing so manually, the website asked me to login in the "Excel IE Browser window"...but it's been over an hour, so I was logged out. How do I log in again to use it?
here's the applicable data pull code if it helps (the URL works fine, once logged in):
With ActiveSheet.QueryTables.Add(Connection:="URL;" & theURL, Destination:=webInfoWS.Range("$A$2"))
.name = cel.Value & " hex"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = False
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "3"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
For the mean time, I found a work around (mainly from this thread):
Adding this after Debug.Print "Opening " & theURL and just before With ActiveSheet.QueryTables.Add(...)
''' Log in to the web stuff
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.Navigate theURL
Do Until .READYSTATE = 4
DoEvents
Loop
If Left(.Document.Title, 5) <> "Welcome!" Then
.Document.all.Item("username").Value = "My User Name"
.Document.all.Item("password").Value = "MyPassword"
.Document.forms(0).submit
End If
' .Quit
End With
''''''
What this does is actually opens the IE Window, then (automated) puts in my username and password, and submits.
However, if I run the macro again (meaning that I already did log in), this gives me an error, because there is no username/password entry form.
Thoughts on how to get around - use On Error Goto Next, but I don't like using that, but it might be the best option. I think I'll try instead to get the Window title (via HTML) and check if that is the login form or not...
Edit: A note on how to know that the .Item("____") is "username" and "password". That just comes from the HTML Input ID tag:
You'll notice in the post I found this in, the text in the .Item() is different - I assume because that HTML ID is different as well.
EDIT 2: This doesn't work! I'm able to log in, see the web page in IE, but when I get to .Refresh BackgroundQuery:=False, the resulting information is the text saying I need to login :/