I'm using Chrome to get the HTML of a webpage which is generated by javascript. The applescript which does all this needs to run every 2 minutes. Everything is working perfectly, except that I obviously need Chrome to do this completely in the background. My script contains the following uses of Chrome (as well as a block to set theTab, which doesn't seem to ever cause Chrome to come to the front):
set URL of theTab to theURL
set isLoading to (loading of theTab)
execute front window's active tab javascript javascriptLocation
set theSource to execute front window's active tab javascript "document.documentElement.outerHTML"
Putting this line:
tell application "Finder" to set visible of process "Google Chrome" to false
after each of the above lines either produces no hiding at all, or at best Chrome flashes onscreen and then goes away. I find this very distracting.
Is there any way to have an application run reliably and permanently in the backround? Or, failing this, is there an invisible way to get javascript executed server-side so an applescript can get hold of its generated source?
Chrome 66.0.3359.181 running on Mac OS 10.11.6, Applescript 2.5.
The following is not an answer to the question I posted, but it is an answer to the problem I was trying to solve which is why I posted the question.
As stated in my question, I need to get hold of the HTML which some javascript generates on a site (which is not under my control). I can't do a client-side scrape because of CORS restrictions on the site. I tried the cross-domain tools listed here and couldn't get them to work.
So I was using Chrome's applescript command, execute, to first execute the javascript (to produce the HTML), and then a second time to grab hold of the HTML with document.documentElement.outerHTML. But having Chrome flash onto the screen every 2 minutes throughout the day was doing my head in.
Turns out Chrome can also run in headless mode, from the command line, and just happens to have an option to run javascript and return the generated HTML!
So my code got a whole lot simpler and I don't have to have Chrome in my apps list all the time. Happy coder am I :-)
Here's the one line that gets me the HTML generated on the site I need:
set theSource to (do shell script ((quoted form of POSIX path of googlePath) & " --headless --dump-dom " & theURL))
Thanks, once again, #matt. I'd never heard of headless mode and would never have found this without your suggestion of PhantomJS!
Related
When starting a website, I find it difficult to click Block button on Microphone popup because I can't query this popup (and also with block btn)
How can I handle this problem with Webdriver.io in browser?
enter image description here
One option is to use chrome command line switches. You can find the list here.
The one you are looking for is use-fake-device-for-media-stream. You can set this in your config file so that every time the chrome is launched, it won't show this popup since it is already set to a fake device.
I am writing a test that verifies that a warning to the user before the user opens a document. If the user says that they still would like to open the document, the document opens in an external application (pdf or word).
But now I have an external application over the browser window, and it messes up for other tests.
So, what are the best practices around this kind of issue? Rewrite of the appliction to allow for not opening documents in test?
Added description:
The problem is twofold.
1) It starts processes (word and acrobat) that fills the desctop and requires resources from the test-slave
2) The external process seems to interfer with other tests since (guessing here) it is located over the browser window.
what i understood from your post is, the document(word/pdf) is opening in the browser window hence you are not able to proceed with further steps. If so, you can verify the Title to make sure the document is opened in browser window and can navigate back using below snippet.
driver.navigate().back();
Hope this helps.
What I understand from the line
'But now I have an external application over the browser window, and it messes up for other tests.'
is that once user clicks on open button a new window opens up (Window based application) since you have mentioned PDF or Word.
You can use robot class in such cases, below code snippet will close the current active window:
Robot rbt = new Robot();
rbt.keyPress(KeyEvent.VK_ALT);
rbt.keyPress(KeyEvent.VK_F4);
rbt.keyRelease(KeyEvent.VK_ALT);
rbt.keyRelease(KeyEvent.VK_F4);
Make sure you deal with sync issues properly so that intended window is closed instead of AUT.
I have very weird issue with some VBA code. The code is InternetExplorer automation and it's really simple: I just need to load page, enter credentials and click on the button on another page (after successful login).
But... My code is not working (but only on one of my machines on AWS). On my local machine this code works fine.
The weird thing is that it seams I have access only to the Document property of the first loaded page (where I need to enter login/password). I mean from MyBrowser.Document property I can see all INPUT fields when I load start page. But after successful login (the IE window is visible) I see same INPUT fields from MyBrowser.Document property! Also (as I said) I have no issues with this code on another maching.
IE Protected mode is disabled (this is IE 11). I think this is some kind of security issue but I can't locate it by myself...
Here is my code:
'MyBrowser is IE instanse
'Here I'm loading start page and input login/password
'Next the browser show me another page where I need to click a button
'But Debug messages show me input fields for the first Form!
Application.Wait (Now + TimeValue("0:00:05"))
Do
DoEvents
Loop Until MyBrowser.ReadyState = READYSTATE_COMPLETE
Set HTMLDoc = MyBrowser.Document
For Each myHTML_Element In HTMLDoc.getElementsByTagName("INPUT")
If myHTML_Element.Type = "submit" And myHTML_Element.Name = "BUTTONNEWJOBS" Then myHTML_Element.Click: Exit For
Debug.Print myHTML_Element.Name
Next
UPDATE
I don't know why but my browser object is always Busy that's why I get old page HTML. I tried to .Stop it but with no luck.
I don't have answer for IE11 automation, just a hint for a path you may choose to walk.
Another Stack Overflow question Getting source of a page after it's rendered in a templating engine? provides some browser automation options.
One of the mentioned tool has currently broken support for IE11 due to a IE11 behavior change, see http://docs.seleniumhq.org/about/platforms.jsp#browsers. The blocking issue has some configuration hints you may find useful.
In my opinion Internet Explorer was ALWAYS the least reliable tool both from the point of view of rendering compatibility, programmability and even the automation problems accross different versions. And although it had improved over the years Internet Explorer is still better to be avoided (from programmer's point of view). Today it is luckily not the only tool available.
So if you just need to get the work done, there are other browsers or browser-like tools that you can use.
If you MUST use IE then you should get your answer at the Internet Explorer Dev Center → Community → Developer forums
EDIT (after comments)
By the symptoms in your question it looks like the browser object is busy because of some dialog box (perhaps a security prompt) is being show to the user. Some links from Google that may help:
required IE configuration from the selenium browser automation engine - https://code.google.com/p/selenium/wiki/InternetExplorerDriver
how to disable Internet Explorer Enhanced Security Configuration - http://4sysops.com/archives/four-ways-to-disable-internet-explorer-enhanced-security-configuration-ie-esc/
Internet Explorer Security Zones and Compatibility View - https://help.blackboard.com/en-us/Learn/9.1_SP_14/Administrator/030_Browser_Support/030_Browser_Support_IE_Issue
set of dialogs that can be manipulated by the iMacros Internet Explorer automation toolkit - http://wiki.imacros.net/Browser_Automation#Dialog_Manager
Rewriting ~1000 lines of code to use another automation interface (or browser) and asking for help at the Microsoft's Internet Explorer Dev Center are IMO still valid options
Based on your following statement
I see same INPUT fields from MyBrowser.Document property!
it would seem that the page hasn't fully loaded. Perhaps it is loading data via AJAX and clicking the submit button shows the exact same page (which would explain why you were seeing the same INPUT fields) while waiting for a response from the server.
An easy way to test this is to wait for a longer period of time (e.g. 30 seconds) and ignoring the value of the browser ReadyState. However a better way would be looping until you find a element on the successful page that isn't on the first page, with a possible timeout of maybe 30 seconds.
I've used Selenium to do some automation and ran into similar problems and had to resort to using Implicit Waits but I'm not sure if VBA as such a feature (as I don't know VBA)
I started learning how to use Selenium today. I have never used it before. I downloaded the Selenium IDE (1.0.10) plugin for FireFox (3.5.16). The way it's behaving is not matching up to the docs.
When I click the record button and perform actions in my browser, nothing happens in the IDE (nothing is recorded). (Actually, initially it did record, but now it doesn't) I tried restarting FireFox and that had no effect.
Also, the main controls are now inactive. I've included a screen shot to show what I mean by that. The controls remain inactive even it I click or double click on the name of a test case in the panel on the left.
And one final question -- it appears that a Selenium test case mentions Chrome in its default configuration even though the docs say you can only record tests using FireFox. Should I do anything about that?
If anyone can shed light on any of the above mysteries I would appreciate it. Thanks!
UPDATE
I restarted FireFox again and now it's recording actions, but the controls are still greyed-out as in the screenshot, so I can't play back the test.
The issue is that you are not in the HTML runner mode. The IDE has no concept of ruby or python or c# or Java. That is up to plugins which just essentially do a find an replace.
*chrome means Firefox Chrome. Something that has been around for a lot longer than Google Chrome the browser. It means use the browser chrome which removes a couple sandboxing issues.
If you want to play back the tests you can't go out of the table mode otherwise the IDE won't understand what to do. The code in the screenshot should be stored in a .rb file and that should be executed.
Let me try to unravel mysteries -
As soon as you launch the IDE it would be in recording mode. Do you see last Red button which is enabled.
IDE Tests can be executed only when in the selenese/html format (aka table format). I guess you have changed the format to ruby (I guess so), from Option > Format.
To be able to execute tests change it back to html from option > Format > HTML
Take my words, IDE is only and only for firefox
In order to automate DOH tests during our build process, I use Selenium RC to launch different browsers (IE and Firefox) on a server placed on a different domain than the build machine. Each browser is directed to our runTests.html in order to start DOH.
Sometimes, when a test that uses doh.robot starts, the following message is shown:
"DOH has detected that the current web page is attempting to access DOH, but belongs to a different domain than the one you agreed to let DOH automate. If you did not intend to start a new DOH test by visiting this Web page, press Cancel now and leave the Web page"
but since these tests are unattended it just sits there waiting for someone to click OK, and Selenium times out (in IE 8 it seems like the pop-up disappears automatically but the robot does nothing afterward).
As I said, it doesn't always happen. After you click OK on the Pop-up, the message will stop showing, and the message can go away for several sessions, but then it will show again in which seems to be an arbitrary way.
Does anyone knows a way to prevent this pop-up from showing?
This is probably not the correct way to do it, but in util/doh/robot/DOHRobot.java, you may be able to modify the code to not check that or always simulate pressing "OK". I haven't tried it myself, but I may also need to do that for some of our automated testing.
When the DOH robot is initialized, it first tries to click in the upper left corner of the page you are trying to test. If you obscure this div (you can see it with firebug), then the message will pop up. I think the problem is that your page isn't always loading up quick enough.
It is somewhat of a challenge to fix this. I haven't used DOH in awhile, but I don't think there is any way you can use a setTimeout to fix this. (You can try using setTimeout on the doh.run command, but it might be the case that the DOH robot clicks that div before parsing any doh commands.)
Another thing you might be able to do is add a sort of "wait" command to Selenium, or whatever shell command you are using to fire up the system.