Retrieve Final Webpage URL in Microsoft Excel Visual Basic - vba

I had a question about how to retrieve the contents of a final URL in Microsoft excel, using their Visual BASIC macros.
Essentially, I have a list (ListA) full of URLs. I have code written to trawl every URL on List A one by one and retrieve the data I need, putting it into Excel.
However, a certain percentage of the URLs redirect to 404 pages. I do not have any way of knowing which ones these will be in advance currently, and am trying to write a script that will simply:
1.) Access the URL in ListA
2.) Copy the destination URL that it redirects to
3.) Paste that URL into the cell directly to the right of the original URL
That way I can see what the final URLs are, and if any go to a 404 page, I can delete them from the list before attempting to trawl it for the info I need.
I have had no luck in doing so thus far, and every tutorial I can find online seems to feature code that will not work in Microsoft Excel's limited environment. Does anyone have any idea where I should start?
If it helps, here is the code I have written to trawl the webpage for data:
For i = 1 To 500
ThisURL = "URL;" & WSD.Cells(i, 2)
ThisParcel = "P" & WSD.Cells(i, 1)
Set WSW = Worksheets.Add(After:=Worksheets(Worksheets.Count))
WSW.Name = ThisParcel
WSW.Select
' Do a web query here
With ActiveSheet.QueryTables.Add(Connection:= ThisURL, Destination:=Range("$A$1"))
.Name = "Query" & i
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
Next i

Related

Web Scraping - Container Tracking from Searates Website to Excel

As lacking of essential IT knowledge, I am looking for your kind help to make codes for getting data (final destination & ETA estimated time of arrival) from website tracking container numbers SEARATES (https://www.searates.com/container/tracking/) to Excel.
I have tried method of using GetElementbyID using IE Internet Explorer but this website browser is seems obsolete and useless.
This SERATES website offer API information but it is too complicated for me to understand.
I am enclosing container tracking list for your testing.
https://drive.google.com/file/d/1E0tuA4pYMDPZMgwjQC8dDs6w6BxqVdXK/view?usp=sharing
Hope someone could give me support on this project which is important for my job of logistics.
Thanks in advance.
How about recording a Macro, for starters.
Sub Macro1()
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;https://drive.google.com/file/d/1E0tuA4pYMDPZMgwjQC8dDs6w6BxqVdXK/view", _
Destination:=Range("$A$1"))
.CommandType = 0
.Name = "view"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
Range("A1").Select
Application.Left = 201.25
Application.Top = 93.25
End Sub
Notice, you have to be signed in. I don't have login creds for this link.

Issue getting data from web table using Excel VBA

I need to connect to a website, with credentials then click a link and get the table it brings to my excel sheet.
I'´m already logged in the site and with the following code I get the table updated but this only works while the excel"from web " connection is alive. I mean, first I have to update the DATA-FROM WEB connection refreshing credentials and then running the macro.
Though the web table is always reached and updated I can't get it pasted to the excel sheet if that connection was updated more that an hour ago.
Here is the code:
Sub Descargar_tabla()
Dim IE As Object
Set IE = CreateObject("WScript.Shell")
IE.Run ("""C:\Program Files (x86)\Opera\launcher.exe"" ""https://clientes.invertirenbolsa.com.ar/cart.aspx""")
With ActiveSheet.QueryTables.Add(Connection:="URL;https://clientes.invertirenbolsa.com.ar/cart.aspx", Destination:=Range("$L$40"))
' .Name = "cart"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = False
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlOverwriteCells
.SavePassword = True
.SaveData = True
.AdjustColumnWidth = False
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "2"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=True
Set IE = Nothing
End With
End Sub
What should I add to the code to use the website open session that I already have in the browser which it doesn't expire?

Copy table (with no id) from password protected website

I would like to retrieve a table from my bank's website, and paste it in the same format in an excel sheet.
Thanks to a few posts on this very useful website, I managed to write a vba code that automatically :
Open the Internet Explorer windows
Fills in the Log-in and Password details
Clicks on the submit Button, thus connecting me to my bank account
Clicks on the "previous day transaction report" button, thus generating the report
And finally shows this report in Internet Explorer
Now my problem is :
How can I copy this report (which is a table) and paste it in an excel sheet, with the Exact same format as it appears on the website ?
On the html code of the website page, this table has no "id". But it has a class, called "report". And it is the only one who has this class.
I suppose I have to use this :
IE.document.getElementsByClassName("report")(0).outerHTML
But I am not sure how to use it... Basically the simplest thing for me would be to write a last portion of code that copies the report to the clipboard, and paste it with the same format on the spreadsheet.
Any idea how I could do that?
i recorded a macro to get the code, using this should copy the table for you. I think its the best way for you. You have to change the URL and the webtable. This will copy it.
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;http://THE URL AFTER LOGIN WITH THE TABLE", _
Destination:=Range("$A$1"))
.CommandType = 0
.Name = "SEQUENCE OF URL"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "11"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
Lets make this work!! :) step by step.
You are right. Fill a form using it.
Try open the page, press F12 and use the console to make sure everything is right.
To fill
document.getElementById("Id").value = "xxxxx"
document.getElementById("Id2").value = "pass"
To click
document.getElementById("name").click()
Set objie = CreateObject("InternetExplorer.Application")
objie.Visible = True
With objie
.AddressBar = False
.StatusBar = False
.MenuBar = False
.Toolbar = 0
.NAVIGATE "http://www.loginpage.com"
While .BUSY
Wend
Do While .READYSTATE <> 4: DoEvents: Loop
'first you login
Set htmldoc = .document
htmldoc.getElementById("UserId").Value = "login"
htmldoc.getElementById("Password").Value = "pass"
htmldoc.getElementById("submit-btn").Click
End With
Try using this one, its first step... Makes sense to you? Im new here, so i will do my best

Using Data from Web, how to login?

I have a macro that uses the "Data from Web" function. I have logged into the website (in Internet Explorer) that I'm pulling data from - but the results I get in Excel just keep telling me I'm not logged in.
Is there a special way to login via Excel for "Data from Web"? I know it works, as I used the Macro Recorder to learn how Excel gets the data - and doing so manually, the website asked me to login in the "Excel IE Browser window"...but it's been over an hour, so I was logged out. How do I log in again to use it?
here's the applicable data pull code if it helps (the URL works fine, once logged in):
With ActiveSheet.QueryTables.Add(Connection:="URL;" & theURL, Destination:=webInfoWS.Range("$A$2"))
.name = cel.Value & " hex"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = False
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "3"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
For the mean time, I found a work around (mainly from this thread):
Adding this after Debug.Print "Opening " & theURL and just before With ActiveSheet.QueryTables.Add(...)
''' Log in to the web stuff
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.Navigate theURL
Do Until .READYSTATE = 4
DoEvents
Loop
If Left(.Document.Title, 5) <> "Welcome!" Then
.Document.all.Item("username").Value = "My User Name"
.Document.all.Item("password").Value = "MyPassword"
.Document.forms(0).submit
End If
' .Quit
End With
''''''
What this does is actually opens the IE Window, then (automated) puts in my username and password, and submits.
However, if I run the macro again (meaning that I already did log in), this gives me an error, because there is no username/password entry form.
Thoughts on how to get around - use On Error Goto Next, but I don't like using that, but it might be the best option. I think I'll try instead to get the Window title (via HTML) and check if that is the login form or not...
Edit: A note on how to know that the .Item("____") is "username" and "password". That just comes from the HTML Input ID tag:
You'll notice in the post I found this in, the text in the .Item() is different - I assume because that HTML ID is different as well.
EDIT 2: This doesn't work! I'm able to log in, see the web page in IE, but when I get to .Refresh BackgroundQuery:=False, the resulting information is the text saying I need to login :/

Using VBA to Grab Stock Info from the Web

I have been attempting to write a macro that grabs financial information from the internet and pastes it into my macro for further analysis. I have been trying to use a query from one of my existing connections (MSN stock quotes).
with my code (below) I have been able make the query pop up but what I can't figure out how to do is to enter anything into the box that pops up. What I'm basically looking for is how to (after the code I have listed below) tell excel to type in certain values into the box that pops up and click "OK" to run the query.
below is my code that initiates the query box asking for tickers
With ActiveSheet.QueryTables.Add(Connection:= _
"FINDER;C:\Program Files\Microsoft Office\Office12\QUERIES\MSN MoneyCentral Investor Stock Quotes.iqy" _
, Destination:=Range("$A$1"))
.Name = "MSN MoneyCentral Investor Stock Quotes"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = False
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingAll
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = True
.Refresh BackgroundQuery:=False
End With
Sean, have a looksie at VB.NET as opposed to VBA - you'll find it much more conducive to things like this. As for good libraries for financial data - check out this library:
https://code.google.com/p/yahoo-finance-managed/