I need to read all Shortcuts of the frontmost Application in MAC OS. Is there a API or Class in Cocoa,Objective-c who provides this?
The Accessibility API is what you need. Unfortunately, it is rather convoluted, and it would be good to read through the UIElementInspector source code to see how it's used, as well as the sections relevant to assistive applications in the Accessibility Programming Guidelines for Mac.
What you want would take a good bit of code so I will just outline the steps.
Use [[NSWorkspace sharedWorkspace] runningApplications] to get a list of applications and get the application whose active property is YES.
Get the PID of that application using the NSRunningApplication's processIdentifier property.
Now we get to Accessibility, accessed through the Application Services API
Create an AXUIElement representing the active Application using AXUIElementCreateApplication, which takes pid as the argument.
Now you have an AXUIElement with the Application role, or AXApplication pseudo-class. You'll want to proceed down the hierarch of elements AXApplication -> AXMenuBar -> AXMenuBarItem -> AXMenuItem. Note that AXMenuItems can have other AXMenuItems nested under them.
To traverse the hierarchy, use AXUIElementCopyAttributeValues to get the values of kAXChildrenAttribute. This will return an array of children AXUIElements.
Finally, when you get to AXMenuItem elements, examine their Menu Item Cmd Char, Glyph, Modifiers, and Virtual Key attributes for the actual shortcuts. The constants for the attribute names, like kAXMenuItemCmdCharAttribute, are listed here.
Not trying to compete #woody :)
But I found this, and it might help too:
UI Browser
Related
I am creating a window switcher for macOS.
I am using CGWindowListCopyWindowInfo to get the list of open windows.
let windows = CGWindowListCopyWindowInfo([.excludeDesktopElements], kCGNullWindowID)
I am not using the optionOnScreenOnly option along with CGWindowListCopyWindowInfo because that excludes off-space windows.
However, this leads to certain dummy windows also getting included in the list of windows. For example, there is a window present for each open tab on Xcode. There is also an additional invisible window for VLC player apart from the actual window. These windows aren't present if I use optionOnScreenOnly.
Is there a way to separate these dummy windows from actual windows?
I have explored the kCGWindowLayer and kCGWindowStoreType keys in the dictionaries returned by CGWindowListCopyWindowInfo but couldn't tell the windows apart.
Edit:
I am open to both Swift and Objective-C solutions, as well as private API calls.
Edit 2:
I know this is possible to do because the app Witch handles these cases correctly.
I'm not sure if this captures all the cases you're interested in, but if we take the array of dictionaries returned by CGWindowListCopyWindowInfo() and group the entries by kCGWindowNumber (the window ID), it looks like we'll end up with one of three cases:
Only one dictionary entry has that window number (it's a simple widow, which may be on or off screen)
Several dictionaries have that window id, and only one of them in the group has the key kCGWindowIsOnscreen set to 1: the others either lack the key or have it set to 0. This means we have a window with several tabs, and the entry with kCGWindowIsOnscreen set is the visible tab.
Several dictionaries have that window id, and none have the key kCGWindowIsOnscreen set to 1 (it is missing or set to 0 in all cases). This means we have a window with several tabs that is currently off-screen.
I'm assuming you're probably already filtering out anything that isn't in layer 0 (kCGWindowLayer = 0). It seems that most of the things that we visually perceive as a 'window' are in layer 0 (status menu items seem to be in layers with single or low double digits; widgets seem to be up around layer 100...).
I don't see any simple, direct way to get at this, but you should be able to build this logic into something workable.
How about using a different approach, by checking what apps are running and visible using the following:
let runningApps = NSWorkspace.shared.runningApplications.filter{ $0.activationPolicy == .regular }
If you print(runningApps) you will get a list of apps currently ran in your workspace.
Hope this helps.
Question
How to use these private functions on other windows? It would be nice to have this knowledge back in the wild. I am specifically trying to get CGSOrderWindow and CGSSetWindowLevel to work.
I was trying in the direction of:
temporarily register as the dock and then register the dock as the dock again immediately afterwards
or
code injection into the Dock process per this comment:
Also, the author of the above project seems determined to make all core functionality available as a framework. It seems to be implemented as code injection into the Dock process.
Reason I know this is possible
I have been doing work on trying to setLevel on window of another app, and focus window of another app if focused. I am posting this again with the info I learned because from my searching online, I know this was done in the past, its just the knowledge is not publicly out there anymore. The sourceforge pages are no longer there. So I was wondering if you could help me make this information public again.
This is the topic I read that gave me this information - http://cocoadev.com/HowtoControlOtherAppsWindows
Here you see comments like:
You cannot control an another app's windows from a user-level process, unfortunately.
SlavaKarpenko
You can, Slava, you just need to register as the Dock. It might be possible to temporarily register as the dock and then register the dock as the dock again immediately afterwards, not sure. I think the call you'd be wanting to investigate as CoreDockRegisterDockOwner in HIServices.framework.
FinlayDobbie
You could also use APE or similar to do control the windows, or (as mentioned above) register as the Dock (look for the private APIs with Universal Connection in their name). Has anyone found a polite way of getting the Dock to give up its universal connection? The only way I can find is to force quit the Dock and grab the universal connection when it's not looking (which prevents the dock reloading).
SamTaylor
There's an open source project up on sourceforge.net that looks much more like the window managers I've used on Unix boxes than Space.app (or Space.dock): http://wsmanager.sourceforge.net/
SteveCook
Verifying things work
This is what I learned, from the sources at bottom of this post, we see all these functions work with CGWindowIds, so how do I get that, this is how:
Get all windows with CGWindowListCopyWindowInfo. Then access each element from that array with CFArrayGetValueAtIndex and then get the CGWindowId with objectForKey:, kCGWindowNumber, and then integerValue.
Now if I try to focus or set level of a window that is OWNED by the app running the code, it works fantastic. For instance:
MY_TARGET_CGWINDOW_ID = 179;
rez_CGError = CGSOrderWindow(_CGSDefaultConnection, MY_TARGET_CGWINDOW_ID, kCGSOrderAbove, 0);
Will focus it, rez_CGError is 0. Even if the window is minimized, it is unminimized, without animation, and shown.
Now however, if I try this on a window of a different app we get some errors:
MY_TARGET_CGWINDOW_ID_of_other_app = 40;
rez_CGError = CGSOrderWindow(_CGSDefaultConnection, MY_TARGET_CGWINDOW_ID_of_other_app, kCGSOrderAbove, 0);
This fails and rez_CGError is 1000, which I suspect means "cid (CGSConnection) used does not have permission to modify target window". The same happens if I first do [app activateWithOptions: (NSApplicationActivateIgnoringOtherApps | NSApplicationActivateAllWindows)] before making the call above.
So I first get the cid of that owning window like this:
var rez_CGError = CGSGetWindowOwner(_CGSDefaultConnection, MY_TARGET_CGWINDOW_ID_of_other_app, &ownerCid);
This works good and I get ownerCid is set to a value. Then I do the focus command with this new connection:
rez_CGError = CGSOrderWindow(ownerCid, MY_TARGET_CGWINDOW_ID_of_other_app, kCGSOrderAbove, 0);
However this gives rez_CGError of 268435459, which I suspect means "current app does not have permission to use this ConnectionId (cid)". (Same happens if I call activateWithOptions first.
My Sources for the Private Functions
Here is the sources for some private functions I found - https://code.google.com/p/undocumented-goodness/source/browse/trunk/CoreGraphics/CGSPrivate.h
This one source here contains a function that is not in the above link - CGSGetConnectionIDForPSN - i test it and it exists - from - https://github.com/mnutt/libqxt/blob/767498816dfa1742a6f3aee787281745afec11b8/src/gui/qxtwindowsystem_mac.h#L80
I have made a custom virtual keyboard widget for my kiosk application, and now comes the time when I want it to produce fake keyboard events and feed them to an QLineEdit of choice.
I do the following:
// target is the QWidget to receive the events
// k is the Qt::Key (keycode) I want to send (Testing with an 'A')
Qt::Key k=Qt::Key_A;
if(0!=target){
//According to docs this will be freed once posted
QKeyEvent * press=new QKeyEvent(QKeyEvent::KeyPress, (int )k,0);
QKeyEvent * release=new QKeyEvent(QKeyEvent::KeyRelease, (int )k,0);
//Give the target focus just to be sure it is available for input
target->setFocus();
//Post the events (queue up and let the target consume them when the eventloop gets around to the target)
QCoreApplication::postEvent ( target, press) ;
QCoreApplication::postEvent ( target, release) ;
}
I see the target widget receive focus, but there are no letters typed into the input field like I would expect. What am I doing wrong? Which assumptions are wrong?
PS: I know that this could be solved by using existing virtual keyboards or at least using the platform interface as is done in this post. In our approach we have decided to build the kayboard into the application to obtain full control over the UX and keyboard design.
Thanks!
Since no-one stepped up, I will try to provide some closure.
It turns out that Qt5 comes with a library of testing facilities called testlib. It has all sorts of goodies to facilitate easy creation, management and running of unit tests for Qt application. Among these facilities there is a set of functions for sending fake events such as fake typing of text, mouse clicks etc. It is quite comprehensive and covers many use-cases. Since this is used internally by Qt developers to test Qt itself it is also production proven code.
I simply copied what I needed from there.
I would like to create a system tool / application which has the capacity to aid in window management. I'm trying to find documentation about the following topics, if they are indeed possible given the security sandboxing of OSX.
Show a list of running applications with the name & icon, and allow the user to choose one
Manipulate the frame(s) of said application's windows (eg, resize, reposition) from my app (with animations -- though I assume this will be trivial once I can perform the actual change)
Hide or show these applications from task managers, etc.
Be able to launch (or terminate) instances of the given application
It seems to me that Quicksilver accomplishes many of these things, but the lack of AppStore availability makes me wonder if it possible to do this while remaining in the OSX sandbox.
There are a lot of pieces of software out there that do window management. You can check out a tiling window manager I've been hacking on called Amethyst. The basic idea behind software like this relies on Accessibility (which you can find documentation for here). As a quick overview the APIs work by acquiring references to accessibility elements (applications, windows, buttons, text fields, etc.) which have properties (hidden, position, size, etc.), some of which are writable.
As an example let's say that you wanted to move all windows in every running application to the upper left corner of the screen. That code might look like
for (NSRunningApplication *runningApplication in [[NSWorkspace sharedWorkspace] runningApplications]) {
AXUIElementRef applicationRef = AXUIElementCreateApplication([runningApplication processIdentifier]);
CFArrayRef applicationWindows;
AXUIElementCopyAttributeValues(applicationRef, kAXWindowsAttribute, 0, 100, &applicationWindows);
if (!applicationWindows) continue;
for (CFIndex i = 0; i < CFArrayGetCount(applicationWindows); ++i) {
AXUIElementRef windowRef = CFArrayGetValueAtIndex(applicationWindows, i);
CGPoint upperLeft = { .x = 0, .y = 0 };
AXValueRef positionRef = AXValueCreate(kAXValueCGPointType, &upperLeft);
AXUIElementSetAttributeValue(windowRef, kAXPositionAttribute, positionRef);
}
}
Which illustrates how you get references to applications and their windows, how to copy attributes from an accessibility element, and how to set attributes of an accessibility element.
There are a variety of notifications documented in NSWorkspace for the launching and termination of applications, and the accessibility framework also has a sense of notifications for things like an application creating or destroying windows, or a window miniaturizing or deminiaturizing.
Animating the window changes is non-trivial and I haven't figured out how to do it yet, though it may be possible. It may not be possible at all without hitting private APIs. But the other things you have listed should be possible. Hiding an application, for example, could be done by setting the kAXHiddenAttribute on the application accessibility element. Launching an application can actually be done via -[NSWorkspace launchApplication:].
Note that the use of accessibility necessitates that the user have Enable access for assistive devices turned on in System Preferences > Accessibility.
I’m looking for alternative for existing tests written in QTP for my Win32 application written in Borland C++.
My candidate is White which based on UI Automation because it’s native solution,
I can create my tests using .NET/C# and easily integrate it with nUnit and Hudson.
White
http://white.codeplex.com
MS UI Automation
http://msdn.microsoft.com/en-us/library/ms747327.aspx
UI Verify
http://uiautomationverify.codeplex.com
I use UI Verify as a spy to identify properties of objects I want to find in my tests.
More or less when I can see something in the spy, I can find it using UI Automation/White.
Generally I don't have much problems with recognizing objects
but when I try to search some content inside the tab contained in Tab Panel
or try to see MenuItems of Menu bar then the problem appears.
UI Automation/UI Verify works wired. When I run UI Verify (1.0 version) I see that objects can be registered properly only then
when I set 'Focus tracking' option and click on target objects or change the keyboard cursor on them. Otherwise it's impossible to find them.
UI Verifier can show me children of my 'tab' panel then. But I can’t find them using UI Automation/White. This is example code:
Tab tab = window.Get();
ITabPage tabPage = tab.SelectedTab;
AutomationElementCollection newCol = tabPage.AutomationElement.FindAll(TreeScope.Descendants, Condition.TrueCondition);
window.Get("buttonName");
the collection is empty even though spy see the children.
Does any of you have some experience with White/UI Automation library that he/she would like to share with me?
I want to implement the tracking feature from the spy to my tests. Can you help me with that? I'm trying to study the code of UIA Verify spy. I think that there are two classes responsible for catching the objects: FocusChangeListener and FocusTracer - this is the code:
http://uiautomationverify.codeplex.com/SourceControl/changeset/view/9992#214260
http://uiautomationverify.codeplex.com/SourceControl/changeset/view/9992#214192
Requirements:
1. Windows SDK
2. .NET 3.5
3. White
4. UIA Verify code
Do you have any better alternative for White/UI Automation?
R.
Could you, the R or YoYo, put your form compiled or in source codes (preferable without the internal logic) somewhere on a file share?
I've never seen a control that'd be not caught using UI Automation if UIAVerify sees it. I saw such windows, which could be only caught with the Focus Tracking feature of UIAVerify. This case, such a window is untouchable by UI Automation search.
Regarding a control, are you sure that the controls you struggling with have the Name property? Maybe, this is a value available only by means of ValuePattern, not the Name?