I have a project that will involve heavy automation of complicated web pages.
I realize there are Mechanize and Beautiful Soup, but don't these break when dealing with large amounts of DOM scripting and other weird stuff you find on complicated web pages?
I think I want essentially a barebones running instance of WebKit that allows me to either do "GUI scripting" or access the DOM. Ideas?
Try Sahi with PhantomJS. Sahi is a browser automation tool, and PhantomJS is a headless Webkit browser. You can find set-up instructions here: http://sahi.co.in/w/sahi-headless-execution-with-phantomjs
Disclaimer: We created the Sahi product.
What platform are you working on? And what language do you intend to use?
Adobe Air let's you embed a webkit inside an Air application and interact with the page JavaScript (there is two-way communication between the page JS and the AIR runtime).
Otherwise, if you are not bound to webkit you could take Mozilla Chromeless for a spin.
My apologies if none of this does what you need to do, I can't quite figure what exactly you are trying to do (page scraping? submitting forms?).
For testing/scraping i would try:
Selenium
EnvJS
Windmill
Watir
Sahi
WebTest
Related
The main difference is, execution on GUI bases and non GUI bases(Headless).
I am looking for difference between all Headless browsers with each other, But unfortunately I didn't find any. I go through one by one, Which makes more confusion. It would be great if someone can share short information with differences, which makes things clear.
Browser
A Browser is an application program that provides a way to look at and interact with all the information on the World Wide Web. Technically a Browser, alternatively referred as a Web Browser or Internet Browser, is a client program that uses HTTP (Hypertext Transfer Protocol) to make requests of Web servers throughout the Internet on behalf of the Browser User.
Headless Browser
A Headless Browser is also a Web Browser but without a graphical user interface (GUI) but can be controlled programmatically which can be extensively used for automation, testing, and other purposes.
Why to use Headless Browsers?
There are a lot of advantages and disadvantages in using the Headless Browsers. Using a headless browser might not be very helpful for browsing the Web, but for Automating tasks and tests it’s awesome.
Advantages of Headless Browsers
There is a lot of advantages in using Headless Browsers. Some of tham are as follows:
A definite advantage of using Headless Browsers is that they are typically faster than real browsers. The reason for being faster is because we are not starting up a Browser GUI and can bypass all the time a real browser takes to load CSS, JavaScript and open and render HTML DOM.
Performancewise you can typically see a 2x to 15x faster performance when using a headless browser.
While Scraping Websites you don’t necessarily want to have to manually start up a website. So you can access the website headlessly and just scrape the HTML. You don’t need to render a Full Browser to do that.
Lot of developers use a Headless Browser for unit testing code changes for their websites and mobile apps. Being able to do all this from a command line without having to manually refresh or start a browser saves them lots and effort.
When You Might NOT Want to Use a Headless Browser
There can be number of reasons why you may opt to use a Real Browser instead of a Headless Browser. A couple of instances:
You need to mimic real users.
You need to visually see the test run.
If you need to do lots of debugging, headless debugging can be difficult.
Which headless browsers are better?
As you rightly pointed that ...the main difference is in the execution on GUI bases and non GUI bases(Headless)..., so from Testing Perspective a lot will depend on the Browser Engine implemented under the hood by any particular browser. For example, here are some of the Browser Engines which fully render web pages or run JavaScript in a virtual DOM.
Chromium Embedded Framework: CEF is a open source project based on the Google Chromium project with JavaScript support and BSD license.
Erik: Erik is a Headless Browser on top of Kanna and WebKit with Swift support and MIT license.
jBrowserDriver: jBrowserDriver is a Selenium-compatible Headless Browser which is WebKit-based and works with Selenium Server through Java binding support and Apache License v2.0 license.
PhantomJS: PhantomJS is a headless WebKit scriptable with a JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG with JavaScript, Python, Ruby, Java, C#, Haskell, Objective-C, Perl, PHP and R(via Selenium) support and BSD 3-Clause license.
Splash: Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python using Twisted and QT with almost all the laungage binding arts and BSD 3-Clause license.
You can find a related discussion in Which drivers support “no-browser”/“headless” testing?
I am working on an automation project, in which I need to capture the activities [ application launched, data entered, input type etc.] user performs on a desktop. I came across Microsoft UI Automation framework which so far works well for native windows based applications like MS Office, .NET apps etc. However I did not find any useful information / samples of capturing the information from different web browsers [Chrome is a must], Python apps, Java Apps etc. Can someone please confirm whether MS UI Automation Framework supports such apps. Any working example to extract user activities from these apps would be highly appreciated. Thanks.
Chrome only supports UI Automation for toolbars, tabs, menu, buttons around the web page. Everything that's rendered as a web page is not seen by UIA.
For the web page content, the easiest way is to use Selenium (driven by the ChromeDriver), which is kind of a de facto standard for browsers, and has nothing to do with UIA.
To test if an app supports UIA, and how far it does, it's very easy, just run UIA's Inspect tool and check the UI tree over that application.
Some additions to Simon's answer...
Chrome page content can be seen by UIA if you run chrome --force-renderer-accessibility. Only for existing Chrome process it won't work. Though user can create a new tab chrome://accessibility manually and enable UIA for all or some chosen pages. This method also works for AT-SPI accessibility technology on Linux. Of course, Selenium WebDriver is an industry standard here. But another way exists. Both Mozilla and IE support UIA by default.
Inspect.exe can be simply downloaded from this GitHub repo.
Regarding Java apps it depends on the app type. Your chances is about 50/50.
WxPython or PyQt5 are good for UIA. TkInter or Kivy apps are not.
P.S. There is an example how to drag a file from explorer.exe and drop to Google Drive in Chrome using Python library pywinauto.
I'm a bit late to the party..
But Chromes accessibility features are only activated once something tries to access it's accessibility.
If you call AccessibleObjectFromWindow ([DllImport("oleacc.dll")]) with the window handle an existing chrome window will have its accessibility activated (and you'll see the actual web page content in UIA!).
If the chrome window is opened after your app is running - Chrome pings open processes for any open accessibility apps... for that you use AccessibleObjectFromEvent and the event you're responding to comes from the windows pipeline: EVENT_SYSTEM_ALERT = 0x0002 .
The bottom line is - you have to tell chrome that there's something installed that wants to access it's web page content.
Oh! and your application has to be signed!! Unsigned apps won't be able to access web content - I think that's the same in firefox too.
I hope this helps someone in the future.
See:
https://www.chromium.org/developers/design-documents/accessibility
Currently i am automating the Desktop Application using CodedUI tests.
I am facing some challenges, there are inside Desktop application we having Chromium browser concept. Unable to record the chromium browser element and getting the element. If normal web page in the Desktop application, i can able to record and get all the wen elements.
So can you anyone help me how to get all the element in the chromium browser using Coded Ui test
I overlooked the question date. But still I posted it as answer, Because I think, still it would be helpful for others to get rid of the problem mentioned in question.
I too had the same issue. After lot of R&D, I found it's very hard (perhaps impossible) to automate CEF using Coded UI.
However, it can be automated using selenium. Moreover, selenium is easy to integrate within coded UI test, if you wish to not leave coded UI.
Here's some links which you can refer how to automate CEF using selenium:
Chromium Embeded Frameowrk using chromedriver.
GitHub Example
Example Framework
Over 2 years I tested web application with help Selenium framework. I know the best design is testing on VM.
The only one downside of this - it's very slow testing. Why?
browser only gets so much memory, if you will run several instances.
site coud be very slow.
connections can be very slow.
Would be great if there was a framework that emulated the browser (engine/core) correctly and can provide some results (api) for surf on the page.
I don't mean to simulate just on the one browser with different version (like IE). I mean to simulate for all browsers with very popular and newest version.
Does anyone know a framework/tool that can do it?
Thank you.
You can try PhantomJS for example.
From their page:
PhantomJS is a headless WebKit with JavaScript API. It has fast and
native support for various web standards: DOM handling, CSS selector,
JSON, Canvas, and SVG.
You can use it in combination with Jasmine (as well as several other frameworks) for testing.
However the selection of available engines is limited to WebKit. I doubt that Selenium will be easy to replace. By the way it looks like Selenium will probably become a W3C standard over the next years.
You can also run Selenium with Xvfb - I use it to execute test on remote server and it is going very well.
I am looking for some start up guidelines to share their experience on XUL development in web application. How good is the option to develop the interface in XUL ?. Can IE understand XUL interface?. I have started reading about XUL and I am liked confused a lot.
Please share your development experience on XUL development.
Thanks
XUL is a Mozilla-only technology meaning that it will only work in Firefox and other browsers based on the Gecko engine. I have bad news for you though: Firefox 4 (meaning Gecko 2.0) disabled support for remote XUL for security reasons, so using it in web applications will no longer be possible. It was arguably a bad idea in the first place.
Take a look into Ample SDK UI Framework, XUL (see examples) is just one of the several XML-based technologies it enables across all browsers, also in IE6.
The ZK web framework (www.zkoss.org) is based on XUL. Actually the web pages you built, using this framework, have the extension ZUL. It produces html + ajax code capable to run in all modern browsers. We are using it in my company for two years now and i have to say it changed my view about XUL.