I want to prepare daily reports from a particular JSON file. My plan is to download the file through VBA, then write a macro that automatically generates the report.
The problem is that the JSON file sit behind a Single-Sign-On authentication protocol. For various reasons I just do not want to VBA myself past the SSO.
What I would like to do is:
1) Have VBA open the address for the JSON file.
2) If I'm not already signed-in, open the browser window for the SSO page so I can manually put in my credentials.
3) Once the JSON file is open - if I'm already signed on and the data is there, to close the browser window and download the data to a particular location.
4) call the other macro so the report can be generated.
A: Is this possible?
B: How?
Yes it is possible and a little bit messy. As I don't know how your system indicate a successful login, I am assuming the login page is directed to an authenticated page.
Dim IE
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "Your login page" 'Go to login page
Do
DoEvents
Loop Until InStr(lcase(IE.LocationURL), lcase("Authenticated")) <> 0 ' wait until authentication is successful
IE.Navigate "your json url" ' Navigate to json url
Do
DoEvents
Loop Until IE.ReadyState = 4 ' wait until download is completed
Debug.Print IE.Document.body.innerHTML '<-- your json
IE.Quit
Set IE = Nothing
Related
Hoping someone might understand what might be going on. What I'm aiming for is using Excel VBA to open my PayPal Multi-Order Shipping page, then open the "Create Shipment" dialogue box and populate the basic name and address fields using data from a row in my excel spreadsheet. My current trouble is with clicking the "Create Shipment" button to open the dialogue box so I can populate the fields ... the code seems to ignore the GetElementByID("createNewOrder").click command and does absolutely nothing. Here's my code so far if anyone would care to take a look ... any help is much appreciated.
Sub PayPalMultiShip()
Dim IE As New SHDocVw.InternetExplorerMedium
IE.Visible = True
IE.Navigate ("https://www.paypal.com/shiplabel/createbulk")
Do While IE.ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
IE.Document.getElementById("createNewOrder").click
Debug.Print "All Done"
End Sub
Here is a screenshot of the html code for the button that I'm trying to access ...
This screenshot comes from the Event Listener tab in the DOM Explorer ...
Event Listener Screenshot
I apologize, but since I'm new to the forum it won't allow me to embed pictures unless someone else suggests the edit
Hi I am trying to write a scraper to get data from this website (www.coned.com/tcisng) and dump data into my Ms Access Backend. When I navigate to it and enter in my credentials (username and password), it does not activate the "Log In" button which I want to click next as it is the next step on my scraper. Also this "Log In" button has no name or ID. How to activate and make it click using VBA in Ms Access 2016.
Updated Code Snippet -
Dim ie As Object
Set ie = CreateObject("internetexplorer.application")
ie.Visible = True
ie.Navigate "www.coned.com/tcisng"
While ie.Busy
DoEvents
Wend
Do
DoEvents
Loop Until ie.ReadyState = READYSTATE_COMPLETE
ie.Document.all("LoginEmail").Value = "myemail"
ie.Document.all("LoginPassword").Value = "myPassword"
Do
DoEvents
Loop Until ie.ReadyState = READYSTATE_COMPLETE
While ie.Busy
DoEvents
Wend
'ie.Document.all(48).Click
'ie.Document.querySelector("[type=submit]").Click
ie.Document.querySelector("button[title*='Log in']").Click
In the last 3 line I am trying to select the "Log In" button but it is disabled, also it has no name or ID, can you also help what exact code to replace in order to activate the button and click it later.
Instead of trying to "click" the button, try submitting the form. The Javascript for that ConEdison page would be:
document.getElementById("form-login-email").form.submit()
In the "old days", with just a plain form, this would work fine.
But on that website there is a lot more going on. That email input element has nine events attached to it. By just filling in the value, you aren't triggering any of those events.
With forms built with ReactJS, for example, the user's interaction with the Input element is monitored, and the value is extracted and stored in a non-displaying object in the DOM tree. When the Submit button is clicked, the form's visible Input elements may not be used at all. In order to feed inputs into such forms, you need to understand ReactJS data structures, and manipulate them directly.
I recently built a scraper based on Chromedriver, which allows a Chrome browser window to be controlled from an external program, such as Access VBA. Once started, Chromedriver runs as a mini-webserver on localhost, and commands sent to it from Access VBA (through a ServerXMLHTTP60 object) cause it to launch Chrome, visit a URL, send keystrokes to input elements etc. Within the browser, the keystrokes fire all the events that human keystrokes would do. The target website was ReactJS-based, but I was able to ignore all of the internal complexity of ReactJS.
Since the website runs in a regular Chrome browser, I was able to use F12 Developer Tools in the development process.
The technology is Webdriver, and was developed for building website testing tools. There are Firefox and MS Edge Chromium versions as well.
I am trying to automate a process of downloading file from a website. At the moment I am able to open a link that runs a javascript which produces an output file, here's my code:
Sub WebDataExtraction()
Dim URL As String
Dim IeApp As Object
URL = "http://si3.bcentral.cl/Siete/secure/cuadros/DescargarExcel.aspx?DropDownListFrequency=&DrDwnCalculo=NONE&codCuadro=UF_IVP_DIARIO"
Set IeApp = CreateObject("InternetExplorer.Application")
IeApp.Navigate URL
End Sub
When I run the sub the IE View Downloads window pops out. To download the file I am trying to follow this approach, however while I am using the attached code I am unable to find a child window.
Can somebody advise me how to proceed from here? Or maybe any suggestion how to approach this problem differently?
I am currently using Office 2010, Visual Basic for Applications 7, and Internet Explorer 9.
The current issue I've been having is that using an IE object seems to fail on our intranet, and I was wondering if there was a fix for this.
Functioning code:
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Navigate "http://www.google.com"
WaitUntilReady IE
'Application.Wait (Now + TimeValue("00:00:10"))
'IE.Visible = True
MsgBox (IE.Document.body.innerHTML)
Broken code:
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Navigate "http://intranet/"
WaitUntilReady IE
'Application.Wait (Now + TimeValue("00:00:10"))
'IE.Visible = True
MsgBox (IE.Document.body.innerHTML)
Functioning code does exactly what it should. Makes an IE object which doesn't show up visibly on screen, and returns a message box with the HTML code contents of google.com
Broken code loads our intranet home page, in a VISIBLE IE window, and returns this error:
Run-time error '-2147417848 (80010108)'
Automation error
The object invoked has disconnected from its clients
I've read through Excel VBA Controlling IE local intranet which suggested using an IP address instead of intranet/ which doesn't work because the IP directs to a different splash screen.
I've also tried using
Set IE = New InternetExplorerMedium
But this A- didn't seem to work either, and B- would mean making sure all my fellow employees enable this Reference.
Also, I've tried loading google.com first, then having it navigate to intranet/, which also didn't help.
Anyone have any suggestions? From my understanding, loading intranet/ is causing IE to disconnect from Excel somehow?
SO! After several days of research and at a complete loss for what was going on, I stumbled on to someone talking about intranet compatibility mode in another article.
Force "Internet Explorer 8" browser mode in intranet
Things happened to click in my head when I read this, realizing that if IE loaded in compatibility, it would explain why the Excel VBA macro lost control of IE, as compatibility mode might be loading as a separate instance of IE.
Settings > Internet Options > Security > Local Intranet > Sites >
Disable "Auto detect intranet" and enable "Include all network paths (UNCs)" fixes the error I was having, perfectly.
'simply replace -seem to work
' Set ie = CreateObject("InternetExplorer.Application")
Set ie = New InternetExplorerMedium
another solution
Set IE = CreateObject("InternetExplorer.Application")
IE.Navigate ("http://URLOnMyIntranet/site.html")
IE.Visible = True
Set IE = nothing
Set objShellApp = CreateObject("Shell.Application")
For Each objWindow In objShellApp.Windows
Debug.Print objWindow.LocationName
If LCase(objWindow.LocationName) = LCase("Your windows name, use debug to find out") Then
Set IE = objWindow
End If
Next`enter code here`
also see this post
First and foremost, thank you for taking the time and interest into this question. I have been using VBA to automate manual tasks within Excel for sometime now, but just recently started exploring accessing the web using VBA.
Goal: automate file downloads (about 15-20 xlsx files daily) from a website where the file url is nowhere to be found in the page's source code.
Below are the steps that I usually take when downloading these manually.
Open login page and enter login credential to access webpage of interest (i.e. the one with all the reports)
After login in, navigate to the webpage with the report
note1: it is setup so that 1 webpage (unique URL) = displays top 55 results in the first page
note2: the same page also has a button to export/save the entire report in different formats
Download the report
Navigate to next webpage (within the same website) and repeat steps 2 and 3 (there's about 15-20 reports/webpages to navigate)
I have gotten as far as downloading the first report by clicking save using the SendKeys. Although sometimes it stops as soon as the dialogue window appears, this has worked up to this point the farthest. It is after this that I have not been able to navigate to another webpage and repeat the same steps to download. My gut feeling is that the Open/Open file/View downloads dialogue window that appears after clicking on the save button is not allowing me to repeat the download/saving process...
I tried looking at the source code of the website to see if I could find the url to the file, but could not find it (not sure if it has to do that the export only occurs after clicking on the submit button which hides the file url or something else like running a script). I'm not very familiar with WinHttpRequest, but seems to be the preferred method after doing my google research. It also looks like this would require to have a file URL, but not sure on this either...
Below is the code that I put together so far. Any help would be very very much appreciated. Thank you! :)
Sub webMacro()
Dim IE As New InternetExplorer
IE.Visible = True 'change False --> True to open the IE window
IE.navigate "https://websiteURL.net//apps/login.aspx"
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Dim Doc As HTMLDocument: Set Doc = IE.document
Doc.getElementById("username").Value = "myusername" 'login to the website
Doc.getElementById("pass").Value = "mypassword"
Doc.getElementById("Enter").Click
Sleep (1000)
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
IE.navigate "https://firstReportWebPage.net//apps/....." 'navigates to the first webpage (report) to download after login
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Doc.getElementById("##########").Click 'ID of the Export/Save button
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Doc.getElementById("###########").Click 'ID of the Submit button
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Doc.getElementById("############").Click 'ID of the field right before it enters the Open/Save/Cancel dialogue window
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Application.Wait (Now + TimeValue("0:00:02")) 'here I'm using the SendKeys to mimic what I would manually do on the keyboard to get to the "Save" button
SendKeys "{TAB}", True
SendKeys "{TAB}", True
SendKeys "{DOWN}", True
SendKeys "{ENTER}", True
Sleep (1000)
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Sleep (1000)
''***This is where it almost always gets stuck...here I'm attempting to get to the Open/Open file/View downloads dialogue window by clicking on the field right before entering the dialogue window using the tab key; same as above when trying to click on the "Save" button in the Open/Save/Cancel dialogue window.
Doc.getElementById("############").Click 'ID of the field right before it enters the Open/Open File/View Downloads dialogue window
Sleep (1000)
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Sleep (1000)
Application.Wait (Now + TimeValue("0:00:02"))
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{ENTER}", True
Sleep (1000)
'some other code to go here...
End Sub
I had always seen 'dont use sendkeys' advised by others, but i didnt truly know what they meant when i tried to do something similar to this.
SendKeys will randomly duplicate a key send sometimes (i was using it to control 16 windows at the same time), 1 set of instructions per window and 18,000 instructions that had to be processed.
It happened about 2-3 times for every 500 instructions that were parsed by the browsers, and i couldnt find a workaround.
The navigating the website, i wrote something that does that, and then i also wrote something that downloads the HTML of the page.
Are you able to download the HTML source of the page with the Open/Save/Cancel dialog, and see if the URL to the file exists on that page within the button etc?
If it does, you could perhaps automate navigating to that page, then downloading the HTML (i have code you can have IF the url is in the source), and then parsing the HTML within VBA to calculate the download URL?