I've downloaded and installed the windows 8 consumer preview, and I'd like to figure out how to use the UI Automation API's to get data from metro style applications.
Background: I have a lot of automated tests which use the Microsoft UI Automation API's to interact with applications. The scripts are written in IronRuby against .NET 4
Initially, I'd simply like to check if the start menu is visible (and if so, close it)
According to Microsoft, the normal UI Automation API's should be able to access and control metro style applications, but the problem is, I can't figure out how my code can actually access the UI of metro applications? Our normal way of viewing UI automation data is using UISpy, however when I run UI spy under windows 8 and bring up a metro app, there is no mention of it... it seems like metro apps (including the start screen) are walled off from the desktop.
If I can't find the metro apps in UISpy, how can I find what their automation ID's are, and how could I find them from my automated test scripts?
A few resources on UI automation which works in Metro style apps:
Here is a sample demonstrating it.
Here is a blog post explaining it.
This is a forum thread on the subject.
There is nothing about the app that stops traditional UI automation from working correctly. You may have to get updated tools to view the Metro style apps. Some ship with the Consumer Preview. Look for inspect.exe.
UI Automation tools can automate Metro applications. Check RIATest for example.
Stating from version 6.0 a number of features in RIATest are specifically targeted at Metro style application automation, particularly:
Ability to stay on top of Metro UI to allow you to simultaneously see your Metro application and RIATest IDE and minimize switching from Desktop to Metro screens when automating a Metro application.
Recording of actions performed on native Windows GUI elements (including Metro GUI). The recorder analyses your script code and reuses your variable names to generate cleaner recorded scripts similar to how you would hand-code an automation script.
Seamless workaround for bugs like this in Windows UI Automation implementation in Metro UI.
Disclaimer: I work for Cogitek, the RIATest company.
WinRT (aka "Metro Style") applications run in an sandbox. You cannot pierce the walls between applications, including to Desktop applications. This includes network isolation (you cannot refer back to localhost for example except in a development environment). There are a couple of exceptions to this such as the Share and Search contacts which allow for very specific types of interactions between apps.
It is this sandbox that is preventing UISpy from seeing the executing WinRT application in the background. I'm not sure how UISpy could work around this issue without some kind of development environment exception to the sandbox (similar to the network isolation exception) that isn't available in the consumer preview.
I'm also unaware of any announcements regarding when/if the UI Automation API will be supported for WinRT applications at this time.
Related
I am working on an automation project, in which I need to capture the activities [ application launched, data entered, input type etc.] user performs on a desktop. I came across Microsoft UI Automation framework which so far works well for native windows based applications like MS Office, .NET apps etc. However I did not find any useful information / samples of capturing the information from different web browsers [Chrome is a must], Python apps, Java Apps etc. Can someone please confirm whether MS UI Automation Framework supports such apps. Any working example to extract user activities from these apps would be highly appreciated. Thanks.
Chrome only supports UI Automation for toolbars, tabs, menu, buttons around the web page. Everything that's rendered as a web page is not seen by UIA.
For the web page content, the easiest way is to use Selenium (driven by the ChromeDriver), which is kind of a de facto standard for browsers, and has nothing to do with UIA.
To test if an app supports UIA, and how far it does, it's very easy, just run UIA's Inspect tool and check the UI tree over that application.
Some additions to Simon's answer...
Chrome page content can be seen by UIA if you run chrome --force-renderer-accessibility. Only for existing Chrome process it won't work. Though user can create a new tab chrome://accessibility manually and enable UIA for all or some chosen pages. This method also works for AT-SPI accessibility technology on Linux. Of course, Selenium WebDriver is an industry standard here. But another way exists. Both Mozilla and IE support UIA by default.
Inspect.exe can be simply downloaded from this GitHub repo.
Regarding Java apps it depends on the app type. Your chances is about 50/50.
WxPython or PyQt5 are good for UIA. TkInter or Kivy apps are not.
P.S. There is an example how to drag a file from explorer.exe and drop to Google Drive in Chrome using Python library pywinauto.
I'm a bit late to the party..
But Chromes accessibility features are only activated once something tries to access it's accessibility.
If you call AccessibleObjectFromWindow ([DllImport("oleacc.dll")]) with the window handle an existing chrome window will have its accessibility activated (and you'll see the actual web page content in UIA!).
If the chrome window is opened after your app is running - Chrome pings open processes for any open accessibility apps... for that you use AccessibleObjectFromEvent and the event you're responding to comes from the windows pipeline: EVENT_SYSTEM_ALERT = 0x0002 .
The bottom line is - you have to tell chrome that there's something installed that wants to access it's web page content.
Oh! and your application has to be signed!! Unsigned apps won't be able to access web content - I think that's the same in firefox too.
I hope this helps someone in the future.
See:
https://www.chromium.org/developers/design-documents/accessibility
I am working on a retail based project. We have two UI. One is web portal where different types of vouchers can be created. These vouchers can be issued and redeemed from desktop application. The issued and redeemed voucher update details will be uploaded to web portal. I am using sikuli to automate desktop application. Can i do GUI testing between desktop and web using sikuli?
Thanks in advance.
As far as my investigation goes, testing end-to-end scenarios that involve both, we and desktop applications, Sikuli is the only option.
When making the decision there few limitation to keep in mind though:
Sikuli tests will only work on the same resolution screen as the one the tests were been created on.
You will need to have a physical computer switched on with a screen attached to it to work.
As it was mentioned in other post here, long scenarios with Sikuli can become brittle. So if you have an option to split your end-to-end scenarios into multiple standalone pieces, while using Selenium for the web part and Sikuli for the desktop app, that probably will be the best solution. Perhaps do some data preparation in one app and save data in a file, then run the other part and assert everything has worked as expected based on that text file.
Base of sikuli is to search the UI controls on screen, based on screenshot.
Also it provides Sikuli API for Java with which you can do a programming around the AUT.
To answer your question,
As long as it is GUI and it's expected screenshot can be taken, then Yes, you can use Sikuli to test Desktop UI & UI for the site opened in Browser.
But testing Web UI with sikuli is quite hectic process as you may need to take screenshots of every testable controls. It will be more cumbersome if site is heavily loaded with images and input controls.
We have selenium IDE/selenium webdriver to perform such testing.
You can use Sikuli on the desktop side where autoIT may become heavy job and the UI is quite static.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
We are planning to built cross platform desktop application. We found that Node-Webkit is a perfect choice for us. But GitHub developed their own framework called Electron instead of using Node-Webkit.
What is the difference between them?
Electron has a page explaining the differences with nwjs.
Like NW.js, Electron provides a platform to write desktop applications
with web technologies. Both platforms enable developers to utilize
HTML, JavaScript, and Node.js. On the surface, they seem very similar.
There are however fundamental differences between the two projects
that make Electron a completely separate product from NW.js.
Entry of Application In NW.js, the main entry point of an
application can be an HTML web page. In that case, NW.js will open the
given entry point in a browser window.
In Electron, the entry point is always a JavaScript script. Instead of
providing a URL directly, you manually create a browser window and
load an HTML file using the API. You also need to listen to window
events to decide when to quit the application.
Electron works more like the Node.js runtime. Electron's APIs are
lower level so you can use it for browser testing in place of
PhantomJS.
Node Integration In NW.js, the Node integration in web pages
requires patching Chromium to work, while in Electron we chose a
different way to integrate the libuv loop with each platform's message
loop to avoid hacking Chromium. See the node_bindings code for how
that was done.
JavaScript Contexts If you are an experienced NW.js user, you
should be familiar with the concept of Node context and web context.
These concepts were invented because of how NW.js was implemented.
By using the multi-context feature of Node, Electron doesn't introduce
a new JavaScript context in web pages.
Note: NW.js has optionally supported multi-context since 0.13.
Legacy Support NW.js still offers a "legacy release" that supports
Windows XP. It doesn't receive security updates.
Given that hardware manufacturers, Microsoft, Chromium, and Node.js
haven't released even critical security updates for that system, we
have to warn you that using Windows XP is wildly insecure and outright
irresponsible.
However, we understand that requirements outside our wildest
imagination may exist, so if you're looking for something like
Electron that runs on Windows XP, the NW.js legacy release might be
the right fit for you.
Features There are numerous differences in the amount of supported
features. Electron has a bigger community, more production apps using
it, and a large amount of userland modules available on npm.
As an example, Electron has built-in support for automatic updates and
countless tools that make the creation of installers easier. As an
example in favor of NW.js, NW.js supports more Chrome.* APIs for the
development of Chrome Apps.
Naturally, we believe that Electron is the better platform for
polished production applications built with web technologies (like
Visual Studio Code, Slack, or Facebook Messenger); however, we want to
be fair to our web technology friends. If you have feature needs that
Electron does not meet, you might want to try NW.js.
Keep in mind this may be biased- it is from Electron's wiki page.
Electron doesn't introduce
a new JavaScript context in web pages.
Source code protection
Electron is packaging its applications with asar, which contains the applications' unprotected source code. This makes it possible for application 1 to extract application 2 and inject vulnerable scripts, without the user knowing it. You can checkout this project on GitHub to see an example of how to manipulate the Slack app for an example. As for now, the Electron team don't have any plans to implement support for source code protection.
NW.js has built in support for compiling your source code to protected binaries.
How can I develop an installable desktop application on top of the Mozilla Engine or the Webkit engine.
We want to have best of both worlds, ease of development with DOM+Javascript+RenderingEngine+ContinuedImprovements in a Browser and user's control as in a desktop app
I looked at using C++ XPCOM for Mozilla but it seems to be quite complicated, Is there any other way to code like a WebApp using Javascript but burn it into the browser and dress it to give a feel of a desktop app. Also I require that javascript is compiled into native so that one cannot sneak into the source code
Are there any examples of desktop applications done this way ?
Web apps are fine but there are concerns of piracy, privacy, security and version control. The moot point is that in a web app the control lies with the software vendor, moreover the data is also with the vendor. Not only these, any changes to the application may also necessitate another around of training. What we want is that once the customer buys a version he is sure of what he owns and that he is in total control of it and we as software developer do not exposed our source code.
The issue is we have expertise in Web App development and we want to utilize that to develop a Desktop App
Your last point is that :
The issue is we have expertise in Web App development and we want to utilize that to develop a Desktop App
Well then BowLine can be an option though it requires Ruby, so you need to consider that. You can also take a look at WebKitDotNet if you are with .net Background.
Use XUL for the user interface and code your functions using JavaScript. You problably only need C++ to expose native functionality not yet available in Gecko. Examples of software that works this way: Komodo IDE, Songbird, Firefox and Thunderbird.
I spoke to the boss of a major music software company a few years ago. He told me that if they were going to start again from the ground up, they might look at WebKit for their UI. This totally surprised me. But I'm wondering if other folks are thinking and acting this way. Is webkit working its way in to truly non-web software?
RealPlayer, iTunes, and many other applications are using it, so are some non-"web" apps such as desktop widget programs:
http://trac.webkit.org/wiki/Applications%20using%20WebKit
Designing "web-apps" with HTML/Webkit UI is beneficial for Mobile users, since many devices have Webkit built in. Even if it is currently only used on a PC, you would have the possibility of hosting it on the web or local network later, with less work to convert it.
Gwibber, a Gnome twitter client that ships with Ubuntu, uses WebKit for displaying timelines (although it uses normal GTK+ widgets for the surrounding UI).
I would consider WebKit a viable option for many pieces of UI, particularly if the program shell exposes appropriate hooks into the surrounding platform to do things like launch a real browser or hook in to system notifications. You run the serious risk, however, of building an application that doesn't fit well in the UI conventions of the user's operating system.
It's not WebKit, but building a UI on a rendering engine is essentially what Mozilla does - Firefox, Thunderbird, etc. are built in XUL rendered with Gecko.
Anything you can do on webkit can be wrapped as an application easily with PhoneGap or other tools.
For example, store.sonyentertainmentnetwork.com could be wrapped as an OSX app, an Android app, and still act as a regular website very easily.
Also: https://products.sel.sony.com/opensource/source_webkit.shtml