Hi I am trying to write a scraper to get data from this website (www.coned.com/tcisng) and dump data into my Ms Access Backend. When I navigate to it and enter in my credentials (username and password), it does not activate the "Log In" button which I want to click next as it is the next step on my scraper. Also this "Log In" button has no name or ID. How to activate and make it click using VBA in Ms Access 2016.
Updated Code Snippet -
Dim ie As Object
Set ie = CreateObject("internetexplorer.application")
ie.Visible = True
ie.Navigate "www.coned.com/tcisng"
While ie.Busy
DoEvents
Wend
Do
DoEvents
Loop Until ie.ReadyState = READYSTATE_COMPLETE
ie.Document.all("LoginEmail").Value = "myemail"
ie.Document.all("LoginPassword").Value = "myPassword"
Do
DoEvents
Loop Until ie.ReadyState = READYSTATE_COMPLETE
While ie.Busy
DoEvents
Wend
'ie.Document.all(48).Click
'ie.Document.querySelector("[type=submit]").Click
ie.Document.querySelector("button[title*='Log in']").Click
In the last 3 line I am trying to select the "Log In" button but it is disabled, also it has no name or ID, can you also help what exact code to replace in order to activate the button and click it later.
Instead of trying to "click" the button, try submitting the form. The Javascript for that ConEdison page would be:
document.getElementById("form-login-email").form.submit()
In the "old days", with just a plain form, this would work fine.
But on that website there is a lot more going on. That email input element has nine events attached to it. By just filling in the value, you aren't triggering any of those events.
With forms built with ReactJS, for example, the user's interaction with the Input element is monitored, and the value is extracted and stored in a non-displaying object in the DOM tree. When the Submit button is clicked, the form's visible Input elements may not be used at all. In order to feed inputs into such forms, you need to understand ReactJS data structures, and manipulate them directly.
I recently built a scraper based on Chromedriver, which allows a Chrome browser window to be controlled from an external program, such as Access VBA. Once started, Chromedriver runs as a mini-webserver on localhost, and commands sent to it from Access VBA (through a ServerXMLHTTP60 object) cause it to launch Chrome, visit a URL, send keystrokes to input elements etc. Within the browser, the keystrokes fire all the events that human keystrokes would do. The target website was ReactJS-based, but I was able to ignore all of the internal complexity of ReactJS.
Since the website runs in a regular Chrome browser, I was able to use F12 Developer Tools in the development process.
The technology is Webdriver, and was developed for building website testing tools. There are Firefox and MS Edge Chromium versions as well.
Related
Hoping someone might understand what might be going on. What I'm aiming for is using Excel VBA to open my PayPal Multi-Order Shipping page, then open the "Create Shipment" dialogue box and populate the basic name and address fields using data from a row in my excel spreadsheet. My current trouble is with clicking the "Create Shipment" button to open the dialogue box so I can populate the fields ... the code seems to ignore the GetElementByID("createNewOrder").click command and does absolutely nothing. Here's my code so far if anyone would care to take a look ... any help is much appreciated.
Sub PayPalMultiShip()
Dim IE As New SHDocVw.InternetExplorerMedium
IE.Visible = True
IE.Navigate ("https://www.paypal.com/shiplabel/createbulk")
Do While IE.ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
IE.Document.getElementById("createNewOrder").click
Debug.Print "All Done"
End Sub
Here is a screenshot of the html code for the button that I'm trying to access ...
This screenshot comes from the Event Listener tab in the DOM Explorer ...
Event Listener Screenshot
I apologize, but since I'm new to the forum it won't allow me to embed pictures unless someone else suggests the edit
I have a database in which users enter a value into Textbox1 on a form; the form also has an ActiveX Web Browser control (acxWebBrowser1), plus another blank Textbox2, and finally a command button. After the user enters text into Textbox1 and clicks the command button, the VBA code will navigate to a specific web page in acxWebBrowser1, enter the Textbox1 value into the web page form, and then obtain a result in the web page in acxWebBrowser1. The result is then grabbed by the VBA code and entered into Textbox2 on the database form.
What I need to be able to do is hide the ActiveX Web Browser control to (a) prevent the user from messing with the web page, and (b) to prevent the clutter and distraction of the ActiveX Web Browser since the VBA code does all the necessary interactions with the web page behind the scenes.
Problem is, I cannot get the acxWebBrowser1 to stay hidden behind a rectangle box control on the form because the browser automatically moves itself to the top of the stack of controls. If I reduce the size of the acxWebBrowser1 control to be very tiny, the web page does not function properly. If I make the web browser control not visible, then the code cannot grab the needed values. I cannot use a POST approach to avoid using the web browser. (I might be wrong about not being able to use POST--maybe someone can point me to how to do that if that is the way to go.)
How can I hide or put something on top of the web browser control?
Three hours later, it dawned on me how to accomplish hiding the ActiveX Web Browser but still get it to be accessible to the VBA code. Simple: Don't update the screen when running the code to access the ActiveX Web Browser and use the .Visible property to "show" the Browser Control when VBA needs to access it (but it won't be visible to the user if screen updating is off) and then hide the Browser Control when VBA is done accessing it.
Here's the code I used to accomplish that:
Application.Echo False 'turn off screen updating
Me.acxWebBrowser1.Visible = True 'make web browser control "visible" to VBA code
{do stuff, like: process web page}
Me.btnClickMe.SetFocus 'set focus on the button so we can hide the web browser control
Me.acxWebBrowser1.Visible = False 'make web browser invisible so user is not distracted
Application.Echo True 'turn screen updating back on
Me.Refresh 'refresh the screen; this is probably not necessary
Just be
absolutely sure to turn screen updating (echo) back on
You should send any error handling to code that sets Application.Echo True otherwise you will not be able to see or do anything if the code crashes after setting echo to False.
I want to prepare daily reports from a particular JSON file. My plan is to download the file through VBA, then write a macro that automatically generates the report.
The problem is that the JSON file sit behind a Single-Sign-On authentication protocol. For various reasons I just do not want to VBA myself past the SSO.
What I would like to do is:
1) Have VBA open the address for the JSON file.
2) If I'm not already signed-in, open the browser window for the SSO page so I can manually put in my credentials.
3) Once the JSON file is open - if I'm already signed on and the data is there, to close the browser window and download the data to a particular location.
4) call the other macro so the report can be generated.
A: Is this possible?
B: How?
Yes it is possible and a little bit messy. As I don't know how your system indicate a successful login, I am assuming the login page is directed to an authenticated page.
Dim IE
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "Your login page" 'Go to login page
Do
DoEvents
Loop Until InStr(lcase(IE.LocationURL), lcase("Authenticated")) <> 0 ' wait until authentication is successful
IE.Navigate "your json url" ' Navigate to json url
Do
DoEvents
Loop Until IE.ReadyState = 4 ' wait until download is completed
Debug.Print IE.Document.body.innerHTML '<-- your json
IE.Quit
Set IE = Nothing
First and foremost, thank you for taking the time and interest into this question. I have been using VBA to automate manual tasks within Excel for sometime now, but just recently started exploring accessing the web using VBA.
Goal: automate file downloads (about 15-20 xlsx files daily) from a website where the file url is nowhere to be found in the page's source code.
Below are the steps that I usually take when downloading these manually.
Open login page and enter login credential to access webpage of interest (i.e. the one with all the reports)
After login in, navigate to the webpage with the report
note1: it is setup so that 1 webpage (unique URL) = displays top 55 results in the first page
note2: the same page also has a button to export/save the entire report in different formats
Download the report
Navigate to next webpage (within the same website) and repeat steps 2 and 3 (there's about 15-20 reports/webpages to navigate)
I have gotten as far as downloading the first report by clicking save using the SendKeys. Although sometimes it stops as soon as the dialogue window appears, this has worked up to this point the farthest. It is after this that I have not been able to navigate to another webpage and repeat the same steps to download. My gut feeling is that the Open/Open file/View downloads dialogue window that appears after clicking on the save button is not allowing me to repeat the download/saving process...
I tried looking at the source code of the website to see if I could find the url to the file, but could not find it (not sure if it has to do that the export only occurs after clicking on the submit button which hides the file url or something else like running a script). I'm not very familiar with WinHttpRequest, but seems to be the preferred method after doing my google research. It also looks like this would require to have a file URL, but not sure on this either...
Below is the code that I put together so far. Any help would be very very much appreciated. Thank you! :)
Sub webMacro()
Dim IE As New InternetExplorer
IE.Visible = True 'change False --> True to open the IE window
IE.navigate "https://websiteURL.net//apps/login.aspx"
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Dim Doc As HTMLDocument: Set Doc = IE.document
Doc.getElementById("username").Value = "myusername" 'login to the website
Doc.getElementById("pass").Value = "mypassword"
Doc.getElementById("Enter").Click
Sleep (1000)
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
IE.navigate "https://firstReportWebPage.net//apps/....." 'navigates to the first webpage (report) to download after login
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Doc.getElementById("##########").Click 'ID of the Export/Save button
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Doc.getElementById("###########").Click 'ID of the Submit button
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Doc.getElementById("############").Click 'ID of the field right before it enters the Open/Save/Cancel dialogue window
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Application.Wait (Now + TimeValue("0:00:02")) 'here I'm using the SendKeys to mimic what I would manually do on the keyboard to get to the "Save" button
SendKeys "{TAB}", True
SendKeys "{TAB}", True
SendKeys "{DOWN}", True
SendKeys "{ENTER}", True
Sleep (1000)
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Sleep (1000)
''***This is where it almost always gets stuck...here I'm attempting to get to the Open/Open file/View downloads dialogue window by clicking on the field right before entering the dialogue window using the tab key; same as above when trying to click on the "Save" button in the Open/Save/Cancel dialogue window.
Doc.getElementById("############").Click 'ID of the field right before it enters the Open/Open File/View Downloads dialogue window
Sleep (1000)
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Sleep (1000)
Application.Wait (Now + TimeValue("0:00:02"))
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{TAB}", True
Sleep (1000)
SendKeys "{ENTER}", True
Sleep (1000)
'some other code to go here...
End Sub
I had always seen 'dont use sendkeys' advised by others, but i didnt truly know what they meant when i tried to do something similar to this.
SendKeys will randomly duplicate a key send sometimes (i was using it to control 16 windows at the same time), 1 set of instructions per window and 18,000 instructions that had to be processed.
It happened about 2-3 times for every 500 instructions that were parsed by the browsers, and i couldnt find a workaround.
The navigating the website, i wrote something that does that, and then i also wrote something that downloads the HTML of the page.
Are you able to download the HTML source of the page with the Open/Save/Cancel dialog, and see if the URL to the file exists on that page within the button etc?
If it does, you could perhaps automate navigating to that page, then downloading the HTML (i have code you can have IF the url is in the source), and then parsing the HTML within VBA to calculate the download URL?
I have a slight issue. Randomly, every few iterations of a loop that I use to load information from a web page will pop up a "Windows Security" window that requests my login and password for the page and halts execution of the code. If I click "Cancel" the macro continues and the information on the page continues to load. The site is an internal page, and we have already logged in, and whether I enter credentials and check the "remember password" checkbox or click cancel, I get the proper information all the same.
I need a way to grab the pop up when it appears and click cancel programatically, if possible. One thing I'm having difficulty with is if the pop up is coming from the IE browser or if it is coming from some other programing in windows.
Any suggestions?
Since the pop-up is coming from the browser, you can close it using SendKeys (see the MSDN SendKeys Method Reference)
For example, to send a tab, followed by a 1-second pause and an enter (which would usually cancel a prompt), you'd do the following :
SendKeys "{Tab}"
Application.Wait Time + TimeSerial(0, 0, 1)
SendKeys "{Enter}"