I have made a custom virtual keyboard widget for my kiosk application, and now comes the time when I want it to produce fake keyboard events and feed them to an QLineEdit of choice.
I do the following:
// target is the QWidget to receive the events
// k is the Qt::Key (keycode) I want to send (Testing with an 'A')
Qt::Key k=Qt::Key_A;
if(0!=target){
//According to docs this will be freed once posted
QKeyEvent * press=new QKeyEvent(QKeyEvent::KeyPress, (int )k,0);
QKeyEvent * release=new QKeyEvent(QKeyEvent::KeyRelease, (int )k,0);
//Give the target focus just to be sure it is available for input
target->setFocus();
//Post the events (queue up and let the target consume them when the eventloop gets around to the target)
QCoreApplication::postEvent ( target, press) ;
QCoreApplication::postEvent ( target, release) ;
}
I see the target widget receive focus, but there are no letters typed into the input field like I would expect. What am I doing wrong? Which assumptions are wrong?
PS: I know that this could be solved by using existing virtual keyboards or at least using the platform interface as is done in this post. In our approach we have decided to build the kayboard into the application to obtain full control over the UX and keyboard design.
Thanks!
Since no-one stepped up, I will try to provide some closure.
It turns out that Qt5 comes with a library of testing facilities called testlib. It has all sorts of goodies to facilitate easy creation, management and running of unit tests for Qt application. Among these facilities there is a set of functions for sending fake events such as fake typing of text, mouse clicks etc. It is quite comprehensive and covers many use-cases. Since this is used internally by Qt developers to test Qt itself it is also production proven code.
I simply copied what I needed from there.
Related
Question
How to use these private functions on other windows? It would be nice to have this knowledge back in the wild. I am specifically trying to get CGSOrderWindow and CGSSetWindowLevel to work.
I was trying in the direction of:
temporarily register as the dock and then register the dock as the dock again immediately afterwards
or
code injection into the Dock process per this comment:
Also, the author of the above project seems determined to make all core functionality available as a framework. It seems to be implemented as code injection into the Dock process.
Reason I know this is possible
I have been doing work on trying to setLevel on window of another app, and focus window of another app if focused. I am posting this again with the info I learned because from my searching online, I know this was done in the past, its just the knowledge is not publicly out there anymore. The sourceforge pages are no longer there. So I was wondering if you could help me make this information public again.
This is the topic I read that gave me this information - http://cocoadev.com/HowtoControlOtherAppsWindows
Here you see comments like:
You cannot control an another app's windows from a user-level process, unfortunately.
SlavaKarpenko
You can, Slava, you just need to register as the Dock. It might be possible to temporarily register as the dock and then register the dock as the dock again immediately afterwards, not sure. I think the call you'd be wanting to investigate as CoreDockRegisterDockOwner in HIServices.framework.
FinlayDobbie
You could also use APE or similar to do control the windows, or (as mentioned above) register as the Dock (look for the private APIs with Universal Connection in their name). Has anyone found a polite way of getting the Dock to give up its universal connection? The only way I can find is to force quit the Dock and grab the universal connection when it's not looking (which prevents the dock reloading).
SamTaylor
There's an open source project up on sourceforge.net that looks much more like the window managers I've used on Unix boxes than Space.app (or Space.dock): http://wsmanager.sourceforge.net/
SteveCook
Verifying things work
This is what I learned, from the sources at bottom of this post, we see all these functions work with CGWindowIds, so how do I get that, this is how:
Get all windows with CGWindowListCopyWindowInfo. Then access each element from that array with CFArrayGetValueAtIndex and then get the CGWindowId with objectForKey:, kCGWindowNumber, and then integerValue.
Now if I try to focus or set level of a window that is OWNED by the app running the code, it works fantastic. For instance:
MY_TARGET_CGWINDOW_ID = 179;
rez_CGError = CGSOrderWindow(_CGSDefaultConnection, MY_TARGET_CGWINDOW_ID, kCGSOrderAbove, 0);
Will focus it, rez_CGError is 0. Even if the window is minimized, it is unminimized, without animation, and shown.
Now however, if I try this on a window of a different app we get some errors:
MY_TARGET_CGWINDOW_ID_of_other_app = 40;
rez_CGError = CGSOrderWindow(_CGSDefaultConnection, MY_TARGET_CGWINDOW_ID_of_other_app, kCGSOrderAbove, 0);
This fails and rez_CGError is 1000, which I suspect means "cid (CGSConnection) used does not have permission to modify target window". The same happens if I first do [app activateWithOptions: (NSApplicationActivateIgnoringOtherApps | NSApplicationActivateAllWindows)] before making the call above.
So I first get the cid of that owning window like this:
var rez_CGError = CGSGetWindowOwner(_CGSDefaultConnection, MY_TARGET_CGWINDOW_ID_of_other_app, &ownerCid);
This works good and I get ownerCid is set to a value. Then I do the focus command with this new connection:
rez_CGError = CGSOrderWindow(ownerCid, MY_TARGET_CGWINDOW_ID_of_other_app, kCGSOrderAbove, 0);
However this gives rez_CGError of 268435459, which I suspect means "current app does not have permission to use this ConnectionId (cid)". (Same happens if I call activateWithOptions first.
My Sources for the Private Functions
Here is the sources for some private functions I found - https://code.google.com/p/undocumented-goodness/source/browse/trunk/CoreGraphics/CGSPrivate.h
This one source here contains a function that is not in the above link - CGSGetConnectionIDForPSN - i test it and it exists - from - https://github.com/mnutt/libqxt/blob/767498816dfa1742a6f3aee787281745afec11b8/src/gui/qxtwindowsystem_mac.h#L80
I would like to create a system tool / application which has the capacity to aid in window management. I'm trying to find documentation about the following topics, if they are indeed possible given the security sandboxing of OSX.
Show a list of running applications with the name & icon, and allow the user to choose one
Manipulate the frame(s) of said application's windows (eg, resize, reposition) from my app (with animations -- though I assume this will be trivial once I can perform the actual change)
Hide or show these applications from task managers, etc.
Be able to launch (or terminate) instances of the given application
It seems to me that Quicksilver accomplishes many of these things, but the lack of AppStore availability makes me wonder if it possible to do this while remaining in the OSX sandbox.
There are a lot of pieces of software out there that do window management. You can check out a tiling window manager I've been hacking on called Amethyst. The basic idea behind software like this relies on Accessibility (which you can find documentation for here). As a quick overview the APIs work by acquiring references to accessibility elements (applications, windows, buttons, text fields, etc.) which have properties (hidden, position, size, etc.), some of which are writable.
As an example let's say that you wanted to move all windows in every running application to the upper left corner of the screen. That code might look like
for (NSRunningApplication *runningApplication in [[NSWorkspace sharedWorkspace] runningApplications]) {
AXUIElementRef applicationRef = AXUIElementCreateApplication([runningApplication processIdentifier]);
CFArrayRef applicationWindows;
AXUIElementCopyAttributeValues(applicationRef, kAXWindowsAttribute, 0, 100, &applicationWindows);
if (!applicationWindows) continue;
for (CFIndex i = 0; i < CFArrayGetCount(applicationWindows); ++i) {
AXUIElementRef windowRef = CFArrayGetValueAtIndex(applicationWindows, i);
CGPoint upperLeft = { .x = 0, .y = 0 };
AXValueRef positionRef = AXValueCreate(kAXValueCGPointType, &upperLeft);
AXUIElementSetAttributeValue(windowRef, kAXPositionAttribute, positionRef);
}
}
Which illustrates how you get references to applications and their windows, how to copy attributes from an accessibility element, and how to set attributes of an accessibility element.
There are a variety of notifications documented in NSWorkspace for the launching and termination of applications, and the accessibility framework also has a sense of notifications for things like an application creating or destroying windows, or a window miniaturizing or deminiaturizing.
Animating the window changes is non-trivial and I haven't figured out how to do it yet, though it may be possible. It may not be possible at all without hitting private APIs. But the other things you have listed should be possible. Hiding an application, for example, could be done by setting the kAXHiddenAttribute on the application accessibility element. Launching an application can actually be done via -[NSWorkspace launchApplication:].
Note that the use of accessibility necessitates that the user have Enable access for assistive devices turned on in System Preferences > Accessibility.
I am implementing a view model that is shared by applications on multiple platforms. I am using MvvmCross v3 that has its own MvxEventToCommand class, but I believe the challenge is the same for other frameworks like MVVM Light. As long as the event is used without parameters, the implementation is straightworward, and this is the case for simple interactions like tapping the control.
But when the command needs to handle event arguments things become more complicated. For example, the view model needs to act on certain scroll bar changes (and load more items in the associated list view). Here is the example of XAML:
<cmd:EventToCommand
Command="{Binding ScrollChanged}"
CommandParameter="{Binding EventArgs}" />
(MvvmCross uses MvxEventToCommand, but the principle is the same).
Then in my model I can have the following command handler:
public ICommand ScrollChanged
{
get
{
return new RelayCommand<ScrollChangedEventArgs>(e =>
{
MessageBox.Show("Change!");
});
}
}
(MvxCommand in MvvmCross).
The problem is that ScrollChangedEventArgs is platform specific and this code simply will not compile in a portable class library. This is a general problem with any command that needs not only a push when an event was fired but requires more specific event details. Moving this code in platform-specific part is silly because it more or less kills the concept of portable view models and code-behind-free views. I tried to search for projects that share view models between different platforms, but they all use simple events like "Tap" with no attached event details.
UPDATE 1 I agree with Stuart's remark that view models should only deal with higher level abstractions, so I will rephrase the original concern: how to map results of low-level interactions to a platform-neutral event that triggers a business logic command? Consider the example above: the business logic command is "load more items in a list", i.e. we deal with a list virtualization where a limited number of items from a large collection are loaded initially, and scrolling down to a bottom of a list should cause additional items to be loaded.
WinRT can take care of list virtualization by using observable collections that support ISupportIncrementalLoading interface. The runtime detects this capability and automatically requests extra items from a respective service when the user scrolls down the list. On other platforms this feature should be implemented manually and I can't find any other way than reacting on ScrollViewer ScrollChanged event. I can see then two further options:
Place OnScrollChanged handler in a code-behind file and call the portable view model higher level event (such as "OnItemsRequested");
Avoid code-behind stuff and struggle to wire the ScrollChanged event directly to a view model, then we will need to remap the platform-specific event first.
As long as there is no support for second option, putting event handler in code-behind file is OK as long as it is done for the sole purpose of event mapping. But I would like to investigate what can be done using the second option. MvvmCross has MapCommandParameter class which seems to be able to help, so I wonder if I should exploit that one.
UPDATE 2 I tried MapCommandParameter approach, and it worked allowing me to insert a platform-specific adapter that would map low-level events to view model-specific commands. So the second option worked without any struggle. Stuart also suggested listview-subclassing so there is no need to care about scrolling events. I plan to play with it later.
I agree that viewmodel commands should normally be expressed in terms of viewmodel concepts - so it would be 'strange' to send the viewmodel a command about the scrollbar value changing, but it might be ok to send the viewmodel a command about the user selecting certain list elements to be visible (which she does via scrolling)
One example where I've done this type of thing previously is in list selection.
I originally did this across multiple platforms using a cross-platform eventargs object -
https://github.com/slodge/MvvmCross/blob/vnext/Cirrious/Cirrious.MvvmCross/Commands/MvxSimpleSelectionChangedEventArgs.cs
this was then used on WindowsPhone (for example) via an EventToCommand class like https://github.com/slodge/MvvmCross/blob/vnext/Cirrious/Cirrious.MvvmCross.WindowsPhone/Commands/MvxSelectionChangedEventToCommand.cs
However... I have to admit that this code hasn't been used much... For list selection we have instead mainly used selecteditem binding, and there simply haven't been any apps that have needed more complex parameterized commands (so far) - you might even need to go back to very old v1 mvvmcross code to find any samples that use it.
I’m looking for alternative for existing tests written in QTP for my Win32 application written in Borland C++.
My candidate is White which based on UI Automation because it’s native solution,
I can create my tests using .NET/C# and easily integrate it with nUnit and Hudson.
White
http://white.codeplex.com
MS UI Automation
http://msdn.microsoft.com/en-us/library/ms747327.aspx
UI Verify
http://uiautomationverify.codeplex.com
I use UI Verify as a spy to identify properties of objects I want to find in my tests.
More or less when I can see something in the spy, I can find it using UI Automation/White.
Generally I don't have much problems with recognizing objects
but when I try to search some content inside the tab contained in Tab Panel
or try to see MenuItems of Menu bar then the problem appears.
UI Automation/UI Verify works wired. When I run UI Verify (1.0 version) I see that objects can be registered properly only then
when I set 'Focus tracking' option and click on target objects or change the keyboard cursor on them. Otherwise it's impossible to find them.
UI Verifier can show me children of my 'tab' panel then. But I can’t find them using UI Automation/White. This is example code:
Tab tab = window.Get();
ITabPage tabPage = tab.SelectedTab;
AutomationElementCollection newCol = tabPage.AutomationElement.FindAll(TreeScope.Descendants, Condition.TrueCondition);
window.Get("buttonName");
the collection is empty even though spy see the children.
Does any of you have some experience with White/UI Automation library that he/she would like to share with me?
I want to implement the tracking feature from the spy to my tests. Can you help me with that? I'm trying to study the code of UIA Verify spy. I think that there are two classes responsible for catching the objects: FocusChangeListener and FocusTracer - this is the code:
http://uiautomationverify.codeplex.com/SourceControl/changeset/view/9992#214260
http://uiautomationverify.codeplex.com/SourceControl/changeset/view/9992#214192
Requirements:
1. Windows SDK
2. .NET 3.5
3. White
4. UIA Verify code
Do you have any better alternative for White/UI Automation?
R.
Could you, the R or YoYo, put your form compiled or in source codes (preferable without the internal logic) somewhere on a file share?
I've never seen a control that'd be not caught using UI Automation if UIAVerify sees it. I saw such windows, which could be only caught with the Focus Tracking feature of UIAVerify. This case, such a window is untouchable by UI Automation search.
Regarding a control, are you sure that the controls you struggling with have the Name property? Maybe, this is a value available only by means of ValuePattern, not the Name?
I'm looking for a guide or sample code for writing Mac OS X Finder plugins? It would like to know how to do some simple actions:
adding image overlayers to icons
adding context menu items
listen to file changes
I found the following two resources:
Writing Contextual Menu Plugins for OS X: An outdated document from 2002 that uses the COM API targeting Mac OS X 8/9.
SCPlugin: Open-source SVN Mac application that includes a Finder plug-in.
I am tempted to review the SCPlugin code, but was hoping to find an easier sample to digest.
The Finder Icon Overlay example project represents a small and very basic but actually working example of the answer below.
https://github.com/lesnie/Finder-Icon-Overlay
I know this is so old, but some may be still interested in topic (?)
Here is what I have it done under Leopard (10.6). At first proper Finder's headers are needed. Use class-dump tool to get it. Then write your code as a SIMBL plugin (refer to documentation how to do it), swizzling some methods. For instance to draw something over icon in ListView, drawIconWithFrame: method of TIconAndTextCell method must be overriden.
Here's the code for method swizzling:
+ (void) Plugin_load
{
Method old, new;
Class self_class = [self class];
Class finder_class = [objc_getClass("TIconAndTextCell") class];
class_addMethod(finder_class, #selector(FT_drawIconWithFrame:),
class_getMethodImplementation(self_class, #selector(FT_drawIconWithFrame:)),"v#:{CGRect={CGPoint=dd}{CGSize=dd}}");
old = class_getInstanceMethod(finder_class, #selector(drawIconWithFrame:));
new = class_getInstanceMethod(finder_class, #selector(FT_drawIconWithFrame:));
method_exchangeImplementations(old, new);
}
I am overriding "drawIconWithFrame:" method with my method "FT_drawIconWithFrame:". Below is sample implementation for this method.
- (void) FT_drawIconWithFrame:(struct CGRect)arg1
{
[self FT_drawIconWithFrame:arg1];
if ([self respondsToSelector:#selector(node)]) {
if ([[[[NSClassFromString(#"FINode") nodeWithFENode:[(TNodeIconAndNameCell *)self node]] fullPath] lastPathComponent] hasPrefix:#"A"])
[myPrettyIconOverlayImage drawInRect:NSMakeRect(arg1.origin.x, arg1.origin.y, arg1.size.height, arg1.size.height) fromRect:NSZeroRect operation:NSCompositeSourceOver fraction:1.0];
}
}
Essentially it draws "myPrettyIconOverlayImage" over every icon for file with filename starts with letter "A". This logic is up to you.
Pay attention to this line: [self FT_drawIconWithFrame:arg1]; this is how to call 'super' in order to get normal icon and name etc. I know, looks weird, like loop, but actually it isn't. Then wrap in into SIMBL plugin, install SIMBL and ...run.
Due to changes in Lion some work have to be done from scratch (make new "Finder.h" file with all declarations needed in it, find proper classess and methods to override), but this technique still works.
Happy hacking!
For Yosemite (MacOS 10.10 & newer), you can use Apple's FinderSync framework, which allows Finder extensions to:
Express interest in specific folder hierarchies
Provide "badges" to
indicate the status of items inside those hierarchies
Provide dynamic
menu items in Finder contextual menus, when the selected items (or
the window target) are in those hierarchies
Provide a Toolbar Item
that displays a menu with dynamic items (even if the selection is
unrelated)
Sadly, programming a Finder plugin actually does still require getting your hands dirty with COM. If you look at the SCFinderPlugin subproject of the SCPlugin project, you will find that it follows exactly the same techniques outlined in your first link, including setting up a vtable for COM, writing AddRef/ReleaseRef functions, and so on. Writing a plugin, where you're simultaneously managing old-school Carbon memory management, COM-style memory management, and Cocoa/new-style Carbon memory management, can be an incredible pain—and that totally ignores the fact that you'll be interacting in three or more radically different APIs, with different naming conventions and calling semantics. Calling the situation hysterically poor would be a vast understatement.
On the bright side, the Finder in Mac OS X 10.6 Snow Leopard has been fully rewritten in Cocoa--and with that come vastly superior plugin interfaces. If you are lucky enough to be in a situation where you can actually only target Snow Leopard, you probably should grab an ADC Premier or higher membership, download the prerelease builds, and code against that. Besides, your plugin may not work on 10.6 anyway without a Cocoa rewrite, so it might make good sense to take a look at Snow Leopard before it gets released, regardless.
There is no official or supported plugin system for the Finder. Starting with OS X 10.6, you will need to inject code into the Finder process and override objective C methods in the Finder process.
I've done this for a proprietary project. I can tell you that the reason that there are no examples or tutorials for this is because it is a significantly difficult and time consuming development task. For this reason, there's plenty of incentive for individuals or organizations who have accomplished this to guard the specifics of their process closely.
If there's any way at all that you can accomplish your goal using the Services API, do it. Writing a Finder plugin will take you 1-2 solid months of painstaking development and reasonably deep knowledge of C and Objective-C internals.
If you're still convinced that you want do to this, grab mach_star. Good luck.
As far as I know, there's no official plugin architecture for the Finder. You may be able to add image overlays to icons through an external application without having to hook into the Finder, although it wouldn't be on the fly. I don't think there is a way to add contextual menu items aside from Folder Actions and Automator. You can also look into writing an external application to monitor File System changes using the FSEvents API.
Here's a completed solution for Finder icon badges and contextual menus in Lion and Mountain Lion using the techniques described by Les Nie.
Liferay Nativity provides a scripting bundle that will swizzle the relevant Finder methods and a Java client for setting the icons and context menus. It also includes equivalent projects for Windows and Linux.
The project is open source under LGPL, so feel free to contribute any bug fixes or improvements!
The pickings are slim; it's never been really clear to me whether Finder Plugins are actually supported. A few more leads, though:
SampleCMPlugIn - Carbon-based of course, since so is Finder. Note that almost any Finder plugin is probably going to stop working with 10.6.
Automator can save things as a "Finder plugin." It's a more supported version of what you're discussing, but of course less flexible.
To add Finder/File browser icon overlays and context menus, in a cross-platform manner, from Java, take a look at the Liferay Nativity library.
I also make mention of this in another SO post, which also contains links to Apple's 'Finder Sync' docs and API.