How would I go about programming a HUD type overlay in OSX.
I want to be able to have an application that will display text at a certain point over a different application's window.
And thus if the (other applications) window moves the HUD part will stay at the same coordinate of the other window.
For the window itself, use a borderless, transparent window (plenty of examples) with your own custom view into which to draw your overlaid elements.
For the "other applications' windows" part, there's no public API that's going to let you do this smoothly. You use Universal Access and its window location/navigation API, but it requires your users to turn on "Enable access for assistive devices" (I think it still can't be done programmatically). I don't believe it "lets you know" when a window moves, but I could be wrong. If it does, it'd likely be a one-shot "here's where I am now", so your overlay would likely not keep up. I also don't think it gives you the "window level" to allow you to make sure you're "above" any given window/sheet/palette.
The only other option (to move with other apps' windows) is a system-wide, invasive hack a la Application Enhancer (which is quite controversial). It's easy to get this wrong and destabilize a user's system (hence the controversy).
You could use undocumented CoreGraphics Functions in order to track a window, see http://code.google.com/p/undocumented-goodness/source/browse/trunk/CoreGraphics/CGSPrivate.h
Related
When screen sharing a specific window on macOS with Zoom or Skype/Teams, they draw a red or green highlight border around that window (which belongs to a different application) to indicate it is being shared. The border is following the target window in real time, with resizing, z-order changes etc.
See example:
What macOS APIs and techniques might be used to achieve this effect?
You can find the location of windows using CGWindowListCopyWindowInfo and related API, which is available to Sandboxed apps.
This is a very fast and efficient API, fast enough to be polled. The SonOfGrab sample code is great platform to try out this stuff.
You can also install a global event tap using +[NSEvent addGlobalMonitorForEventsMatchingMask:handler:] (available in sandbox) to track mouse down, drag and mouse up events and then you can respond immediately whenever the user starts or releases a drag. This way your response will be snappy.
(Drawing a border would be done by creating your own transparent window, slightly larger than, and at the same window layer as, the window you are tracking. And then simply draw a pretty green box into it. I'm not exactly sure about setting the z-order. The details of this part would be best as a separate question.)
I am trying to display multiple key combinations to a MenuItem in cocoa. This is most commonly known as "chords".
For example I want to add a menu item that looks like:
"Action1 Control K, F" or "MenuItem2 K,L"
Would this be possible in Objective-C through the standard API? I've looked around and the closest thing to this on MacOS would be using custom views. Would it be the way to go for allowing this functionality?
The standard API does not support handling chords, thus it does not allow setting chords as key equivalent and thus it also cannot display a chord as a key equivalent.
If you need that functionality, you need to implement entirely yourself. Just make your own NSView object and assign it to the view property of NSMenuItem. As documented, you will then have to draw everything yourself:
A menu item with a view does not draw its title, state, font, or other
standard drawing attributes, and assigns drawing responsibility entirely
to the view. Keyboard equivalents and type-select continue to use the key
equivalent and title as normal.
Source: https://developer.apple.com/documentation/appkit/nsmenuitem/1514835-view?language=objc
Whether this is a normal NSView filled with subviews, created either programmatically or even loaded from a NIB file, or whether this is a subclass of NSView drawing everything itself is up to you, all these variations will actually work. Usually it's easiest to use a NIB file and build you menu look in interface builder and using autolayout.
Yet keep in mind that this breaks Apple Human Interface guidelines. It violates the users expectation as all his other apps don't offer anything comparable since in macOS a menu item has one key equivalent or it has none. It also breaks the ability of users to customize the key equivalent the way he is used to do this for all other applications (System Preferences > Keyboard > Shortcuts > App Shortcuts).
Generally you should not replace system standard UI with your own UI unless you really have a very good reason for doing so, as that always breaks users expectations, certain system functionality won't work as expected (e.g. accessibility features) and it destroys the uniform look and feel of the system. Also it breaks system automatic, as you can see in macOS 10.14 (Mojave) where all system standard UI automatically supports dark mode, so if you used only standard UI, your app supports dark mode without any modification, yet all custom UI needs to be customized again for dark mode.
Yes, you'll need to use a custom view. NSMenuItem only displays the first character of its keyEquivalent.
I like to make an overlay with the following properties:
should work at least on Windows 8.1
should be on top on everything, like a mouse cursor
should incorporate the pixels which are already on the background, like a blur filter
no flickering
Details to each of this points:
1) I assume that WDM is activated and DirectX 11.2 is used. Sure it would be nice to have it working on other Windows versions but this has no priority.
2) The problem is that with simply using the WS_EX_TOPMOST, menus from applications are over my overlay. In my case this really hurts as I like to display something with the same properties as a cursor. Imagine that a cursor suddenly is hidden if you open a menu -> unacceptable.
3) I like to read the pixels from the Windows desktop, including any effect Windows applies (like blur), and use this information for my filter. If I add my overlay, as described in 2, I should be able to get a fresh unobstructed copy of the background in the next frame and not read out my own overlay.
4) If I just write something into the Windows desktop directly, it gets overwritten immediately on the next frame by Windows itself. This is not acceptable.
One example of such an application is a magnifying glass, which exactly has all the properties I need. But for this case Windows 8.1 has an API. In contrast I like to write a program which displays a hand on the desktop (which is controlled by Leap Motion) which influences the Windows desktop, so you almost "feel" how you move your hand over the desktop.
If I write a tiny DirectX and/or OpenGL application for myself this is all very easy:
render all the regular stuff to a texture
use this texture for a post processing filter and add all my stuff on top of it
render just a quad to the back buffer
But I like to do that for the whole Windows desktop.
I found many different application, but they are to no use for me:
application which claim to be on top, are still behind menus. This normally doesn't really hurt, but is unacceptable for a cursor-alike thing
screen capturing programs which hook them self in all running programs are nice, but I want to hook myself into WDM
normally screen capturing programs do not draw anything into the back buffer, so they get every frame a new unobstructed back buffer
My questions can be boiled down to: How can I write my own magnifying glass for Windows 8.1.
I fear that my only serious option is to hook myself into WDM, what I try to avoid.
I'm happy to hear any idea how to achieve this, or hints to application which are doing what I describe.
As the title said, I want to move the screen up, so that my app's window can come in from the bottom. What I currently have now is a floating window, whose background has been set to a screenshot of the screen. Like this:
It looks fine to the user, except now any objects that work in the background don't appear above the window, it's basically frozen.
Can I do this? This effect is similar to what Notification Center does in 10.8.
You can do this, for a certain definition of "can" - there's no API for it, but you can likely use the same SPI that the Finder/Dock/etc uses. The only complication may lie in needing special privileges, or needing your code to be specially signed - I'm not sure what checks are in place.
It's not too tricky to figure this out; you can use tools like nm, otool and even class-dump.
Is there anyway, we can identify if the process has user interface? Anything which deals with drawing on screen.
I looked at this question, but it does not give proper answer. I am inside the process. Is there some api where i can query, or some technique?
EDIT: I am inside the process, I am doing code injection. I want to behave different for non gui apps than gui ones. I cannot use CFBundleGetValueForInfoDictionaryKey() or NSApplicationMain as I want to keep injection module light and thus I cannot link against these frameworks.
This depends heavily on what you mean by "has a graphical interface." Do you mean, "is displaying a window?" Or "has a dock icon?" Or "has access to the Aqua context such that it could display a window if it wanted to?" Or "is linked with AppKit?"
An app bundle that does not have a dock icon will have the Info.plist key LSUIElement set to "1". You can fetch this using CFBundleGetValueForInfoDictionaryKey() (or the NSBundle equivalent if you're in Objective-C). This does not mean the application has no GUI, but does mean it won't show up in the dock. Many LSUIElement apps have a status item UI.
You can also check that you actually have an Info.plist at all using CFBundleCopyBundleURL() or the like. If you don't have an Info.plist, then you're not going to be a "GUI-like" program in all likelihood. (Though again, it's possible to generate GUIs without this).
You can use weak linking to test for AppKit:
if (NSApplicationMain != NULL) {
// We're linked with AppKit
}
That's a fairly good indicator that you're going to have a UI. But it won't catch older Carbon apps.
Some more background on what you mean by "I am inside the process" would be helpful. I assume you're a framework and want to behave differently for GUI apps than for non-GUI apps?
To detect an application capable of drawing on the screen, I would check for whether CoreGraphics is linked. That'll only work for programs built since 10.3, but should be fairly accurate (I believe that old QuickDraw apps still link Core Graphics in 10.3+, but I don't have one handy to check). Probably the best way is to do a weak linking check against CGColorCreate() since it's been around for a long time and is unlikely to ever go away:
extern CGColorRef CGColorCreate(CGColorSpaceRef space, const CGFloat components[])
__attribute__((weak_import));
...
if (CGColorCreate != NULL) {
// Linked with CoreGraphics. Probably a GUI
Of course things can link with CoreGraphics and be capable of drawing on the screen, but never actually draw on the screen. They might link with CoreGraphics in order to do image processing. It might be more accurate by looking for ApplicationServices. You could test ApplicationServicesVersionNumber against NULL, but that's not documented.
This will not catch X apps, since they don't technically draw on the screen. They send commands to X11.app, which is a GUI. Do you consider /usr/X11/bin/xterm a GUI for this purpose? /usr/X11/bin/xeyes?
There is no particular flag I'm aware of in any app that says unambiguously "I draw on the screen." Drawing on the screen is not something you need to pre-declare. Apps that draw on the screen don't have to do so every time they run. Apps that are generally invisible might optionally or occasionally create a status item. It's hard to come up with a single description that includes GrowlHelperApp.app, Pages.app, and /usr/X11/bin/xeyes.