I am currently trying to wrap my head around the hole NSTask, NSPipe, NSFileHandle business. So I thought I write a little tool, which can compile and run C code. I also wanted to be able to redirect my stdout and stdin to a text view.
Here is what I got so far.
I used code from this post to redirect my stdio: What is the best way to redirect stdout to NSTextView in Cocoa?
NSPipe *inputPipe = [NSPipe pipe];
// redirect stdin to input pipe file handle
dup2([[inputPipe fileHandleForReading] fileDescriptor], STDIN_FILENO);
// curInputHandle is an instance variable of type NSFileHandle
curInputHandle = [inputPipe fileHandleForWriting];
NSPipe *outputPipe = [NSPipe pipe];
NSFileHandle *readHandle = [outputPipe fileHandleForReading];
[readHandle waitForDataInBackgroundAndNotify];
// redirect stdout to output pipe file handle
dup2([[outputPipe fileHandleForWriting] fileDescriptor], STDOUT_FILENO);
// Instead of writing to curInputHandle here I would like to do it later
// when my C program hits a scanf
[curInputHandle writeData:[#"123" dataUsingEncoding:NSUTF8StringEncoding]];
NSTask *runTask = [[[NSTask alloc] init] autorelease];
[runTask setLaunchPath:target]; // target was declared earlier
[runTask setArguments:[NSArray array]];
[runTask launch];
NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserver:self selector:#selector(stdoutDataAvailable:) name:NSFileHandleReadCompletionNotification object:readHandle];
And here the stdoutDataAvailable method
- (void)stdoutDataAvailable:(NSNotification *)notification
{
NSFileHandle *handle = (NSFileHandle *)[notification object];
NSString *str = [[NSString alloc] initWithData:[handle availableData] encoding:NSUTF8StringEncoding];
[handle waitForDataInBackgroundAndNotify];
// consoleView is an NSTextView
[self.consoleView setString:[[self.consoleView string] stringByAppendingFormat:#"Output:\n%#", str]];
}
This Program is working just fine. It is running the C program printing the stdout to my text view and reading "123" from my inputPipe. Like indicated in my comment above I would like to provide the input while the task is running once it is needed.
So there are two questions now.
Is there a way to get a notification as soon as somebody tries to read data from my inputPipe?
If the answer to 1 is no, is there a different approach I can try? Maybe using a class other than NSTask?
Any help, sample code, links to other resources are highly appreciated!
I'm not sure whether you can detect a "pull" on an NSPipe. I do have a vague sense that polling for write-availability with select() or using kqueue to look for I/O availability events on the underlying file descriptor of your NSFileHandle might do the trick, but I'm not very familiar with using those facilities in this way.
Do you have to support arbitrary C programs, or is it a special daemon or something you've developed?
If it's your own program, you could watch for requests for feedback on outputPipe, or just blast input onto the inputPipe as you find out what it is you want to send, and let the C program consume it when it's ready; if it's somebody else's code, you may be able to hook scanf and friends using a link-time method (since it's code you're compiling) like the one described in Appendix A-4 of:
http://www.cs.umd.edu/Library/TRs/CS-TR-4585/CS-TR-4585.pdf
The gist of it is to make a .dylib with your custom I/O functions (which may send some sigil to your app indicating that they need input), link that into the built program, set an environment variable (DYLD_BIND_AT_LAUNCH=YES) for the launched task, and run it. Once you've got those hooks in, you can provide whatever conveniences you want for your host program.
Related
Basic Setup
I use NSTask to run a process that optimizes images. This process writes output data to stdout. I use the readabilityHandler property of NSTask to capture that data. Here is the abbreviated setup:
NSTask *task = [[NSTask alloc] init];
[task setArguments:arguments]; // arguments defined above
NSPipe *errorPipe = [NSPipe pipe];
[task setStandardError:errorPipe];
NSFileHandle *errorFileHandle = [errorPipe fileHandleForReading];
NSPipe *outputPipe = [NSPipe pipe];
[task setStandardOutput:outputPipe];
NSFileHandle *outputFileHandle = [outputPipe fileHandleForReading];
NSMutableData *outputData = [[NSMutableData alloc] init];
NSMutableData *errorOutputData = [[NSMutableData alloc] init];
outputFileHandle.readabilityHandler = ^void(NSFileHandle *handle) {
NSLog(#"Appending data for %#", inputPath.lastPathComponent);
[outputData appendData:handle.availableData];
};
errorFileHandle.readabilityHandler = ^void(NSFileHandle *handle) {
[errorOutputData appendData:handle.availableData];
};
I then call NSTask like this:
[task setLaunchPath:_pathToJPEGOptim];
[task launch];
[task waitUntilExit];
(This is all done on a background dispatch queue). Next I examine the return values of NSTask:
if ([task terminationStatus] == 0)
{
newSize = outputData.length;
if (newSize <= 0)
{
NSString *errorString = [[NSString alloc] initWithData:errorOutputData encoding:NSUTF8StringEncoding];
NSLog(#"ERROR string: %#", errorString);
}
// Truncated for brevity...
}
The Problem
Approximately 98% of the time, this works perfectly. However, it appears that -waitUntilExit CAN fire before the readabilityHandler block is run. Here is a screenshot showing that the readability handler is running AFTER the task has exited:
So this is clearly a race condition between the dispatch queue running the readabilityHandler and the dispatch queue where I've fired off my NSTask. My question is: how the hell can I determine that the readabilityHandler is done? How do I beat this race condition if, when NSTask tells me it's done, it may not be done?
NOTE:
I am aware that NSTask has an optional completionHandler block. But the docs state that this block is not guaranteed to run before -waitUntilExit returns, which implies that it CAN begin running even SOONER than -waitUntilExit. This would make the race condition even more likely.
Update: Modern macOS:
availableData no longer has the issues I describe below. I'm unsure precisely when they were resolved, but at least Monterey works correctly. The approach described below is for older releases of macOS.
Additionally, with the modern Swift concurrency system in place and the new paradigm of "threads can always make forward progress", using semaphores like below should be a last resort. If you can, use NSTask's completionHandler API. I have no FORMAL guarantee that the readability handlers will complete before the completionHandler is called, but they seem to in practice, at least on modern macOS. Your mileage may vary.
Old Advice:
Ok, after much trial-and-error, here's the correct way to handle it:
1. Do not Use -AvailableData
In your readability handler blocks, do not use the -availableData method. This has weird side effects, will sometimes not capture all available data, and will interfere with the system's attempt to call the handler with an empty NSData object to signal the closing of the pipe because -availableData blocks until data is actually available.
2. Use -readDataOfLength:
Instead, use -readDataOfLength:NSUIntegerMax in your readability handler blocks. With this approach, the handler correctly receives an empty NSData object that you can use to detect the closing of the pipe and signal a semaphore.
3. Beware macOS 10.12!
There is a bug that Apple fixed in 10.13 that is absolutely critical here: on old versions of macOS, the readability handlers are never called if there is no data to read. That is, they never get called with zero-length data to indicate that they’re finished. That results in a permanent hang using the semaphore approach because the semaphore is never incremented. To combat this, I test for macOS 10.12 or below and, if I’m running on an old OS, I use a single call to dispatch_semaphore_wait() that is paired with a single call to dispatch_semaphore_signal() in NSTask’s completionHandler block. I have that completion block sleep for 0.2 seconds to allow the handlers to execute. That’s obviously a godawfully ugly hack, but it works. If I’m on 10.13 plus, I have different readability handlers that signal the semaphore (once from the error handler and once from the normal output handler) and I still signal the semaphore from the completionHandler block. These are paired with 3 calls to dispatch_semaphore_wait() after I launch the task. In this case, no delay is required in the completion block because macOS correctly calls the readability handlers with zero-length data when the fileHandle is done.
Example:
(Note: assume stuff is defined as in my original question example. This code is shortened for readability.)
// Create the semaphore
dispatch_semaphore_t sema = dispatch_semaphore_create(0);
// Define a handler to collect output data from our NSTask
outputFileHandle.readabilityHandler = ^void(NSFileHandle *handle)
{
// DO NOT use -availableData in these handlers.
NSData *newData = [handle readDataOfLength:NSUIntegerMax];
if (newData.length == 0)
{
// end of data signal is an empty data object.
outputFileHandle.readabilityHandler = nil;
dispatch_semaphore_signal(sema);
}
else
{
[outputData appendData:newData];
}
};
// Repeat the above for the 'errorFileHandle' readabilityHandler.
[task launch];
// two calls to wait because we are going to signal the semaphore once when
// our 'outputFileHandle' pipe closes and once when our 'errorFileHandle' pipe closes
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);
// ... do stuff when the task is done AND the pipes have finished handling data.
// After doing stuff, release the semaphore
dispatch_release(sema);
sema = NULL;
Create a semaphore with an initial value of 0. In the readability handlers, check if the data object returned from availableData has length 0. If it does, that means end of file. In that case, signal the semaphore.
Then, after waitUntilExit returns, wait on the semaphore twice (once for each pipe you're reading). When those waits return, you've got all of the data.
OK, let's say I'm creating a (Bash) Terminal emulator - I'm not actually, but it's pretty close in terms of description.
I've managed to get (almost) everything working, however I'm facing one simple issue: maintaining the current directory.
I mean... let's say the user runs pwd and we execute this via NSTask and /usr/bin/env bash. This outputs the current app's directory. That's fine.
Now, let's say the user enters cd ... The path is changing right? (OK, even for that particular session, but it is changing, nope?)
So, I though of storing the task's currentDirectoryPath when the Task is terminated and then re-set it when starting another Bash-related task.
However, it keeps getting the very same path (the one the app bundle is in).
What am I missing?
Any ideas on how to get this working?
The code
- (NSString*)exec:(NSArray *)args environment:(NSDictionary*)env action:(void (^)(NSString*))action completed:(void (^)(NSString*))completed
{
_task = [NSTask new];
_output = [NSPipe new];
_error = [NSPipe new];
_input = [NSPipe new];
NSFileHandle* outputF = [_output fileHandleForReading];
NSFileHandle* errorF = [_error fileHandleForReading];
__block NSString* fullOutput = #"";
NSMutableDictionary* envs = [NSMutableDictionary dictionary];
envs[#"PATH"] = [[APP environment] envPath];
[_task setLaunchPath:#"/usr/bin/env"];
if (env)
{
for (NSString* key in env)
{
envs[key] = env[key];
}
}
[_task setEnvironment:envs];
[_task setArguments:args];
[_task setStandardOutput:_output];
[_task setStandardError:_error];
[_task setStandardInput:_input];
void (^outputter)(NSFileHandle*) = ^(NSFileHandle *file){
NSData *data = [file availableData];
NSString* str = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
action(str);
fullOutput = [fullOutput stringByAppendingString:str];
};
[outputF setReadabilityHandler:outputter];
[errorF setReadabilityHandler:outputter];
[_task setTerminationHandler:^(NSTask* task){
completed(fullOutput);
dispatch_async(dispatch_get_main_queue(), ^{
[[APP environment] setPwd:[task currentDirectoryPath]];
});
[task.standardOutput fileHandleForReading].readabilityHandler = nil;
[task.standardError fileHandleForReading].readabilityHandler = nil;
[task.standardInput fileHandleForWriting].writeabilityHandler = nil;
[task terminate];
task = nil;
}];
if (![[[APP environment] pwd] isEqualToString:#""])
[_task setCurrentDirectoryPath:[[APP environment] pwd]];
[_task launch];
return #"";
}
As a general matter, it is difficult to impossible to modify another process's environment and other properties from the outside. Similarly, it is not generally possible to query those from the outside. Debuggers and ps can do it by using special privileges.
The parent process which created the process in question has the opportunity to set the initial environment and properties at the point where it spawns the subprocess.
The cd command is necessarily a shell built-in command precisely because it has to modify the shell process's state. That change does not directly affect any other existing process's state. It will be inherited by subprocesses that are subsequently created.
The currentDirectoryPath property of NSTask is only meaningful at the point where the task is launched. It is the current directory that the new process will inherit. It does not track the subprocess's current directory, because it can't. Querying it only returns the value that the NSTask object was configured to use (or the default value which is the current directory of the process which created the NSTask object).
If you're trying to write something like a terminal emulator, you will need to create a long-running interactive shell subprocess with communication pipes between the parent and the shell. Don't run individual commands in separate processes. Instead, write the commands to the interactive shell over the pipe and read the subsequent output. It probably doesn't make sense to try to interpret that output since it can be general in form and not easily parsable. Just display it directly to the user.
Alternatively, you will have to interpret some commands locally in the parent process. This will be analogous to shell built-ins. So, you would have to recognize a cd command and, instead of launching an NSTask to execute it, you would just modify the state of the parent process in such a way that the new current directory will be used for subsequent tasks. That is, you could track the new current directory in a variable and set currentDirectoryPath for all subsequent NSTask objects before launching them or you could modify the parent process's current directory using -[NSFileManager changeCurrentDirectoryPath:] and that will automatically be inherited by future subprocesses.
I am fairly certain that running cd in NSTask does not change the value of task.currentDirectoryPath. Have you tried setting a break point in your dispatch call to see if that value is actually being set correctly?
Edit:
From your termination handler try doing [[APP environment] setPwd:[[[NSProcessInfo processInfo]environment]objectForKey:#"PATH"]];
this is my first post, so let me send me many thanks to all the posting guys
outside there (I use SO extensively passively - great!)
I'm working on an video exporting tool for Mac OS X using the good old Quicktime API.
Brief:
I cut frames from multiple input movies an arrange them (scaled) to a new output
movie (kind of media-kiosk).
As many of the needed QT functionality (e.g. writing timecode ...) need to be
nested in a 32-bit Application, I decided to do this offline using a 32 bit command line
tool. The tool renders frame by frame (offline) and prints the current progress
in values between 0.0 and 1.0
It is invoked by the main application (Cocoa, GUI, the modern stuff) via
NSTask. The stout is caught by a NSPipe.
I took a look at some examples and 'll give you quick overview over my code:
NSTask *task;
NSPipe *pipe;
float progress;
// prepare the offline process
//
//
-(void) prepareOfflineExport {
task = [[NSTask alloc] init];
pipe = [[NSPipe alloc] init];
[task setLaunchPath:pathToRenderer];
[task setStandardOutput:pipe];
}
// arguments are passed outside
// invoke the process
//
-(void) startOfflineExport {
progress = 0.0f;
NSArray *argv = [NSArray arrayWithObjects: /* command line args */, nil];
[task setArguments:argv];
NSFileHandle *fh = [pipe fileHandleForReading];
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(dataReady:) name:NSFileHandleReadCompletionNotification object:fh];
[[NSNotificationCenter defaultCenter] addObserver:self selector:#selector(taskTerminated:) name:NSTaskDidTerminateNotification object:task];
[task launch];
[fh readInBackgroundAndNotify];
}
// called when data ready
//
//
-(void) dataReady:(NSNotification*)n {
NSData *d = [[n userInfo] valueForKey:NSFileHandleNotificationData];
if([d length]) {
NSString *s = [[NSString alloc] initWithData:d encoding:NSUTF8StringEncoding];
progress = [s floatValue];
}
}
// called when process exits
//
//
-(void) taskTerminated:(NSNotification*)n {
task = nil;
progress = 1.0f;
}
Now the Problem:
When launching the application inside Xcode (via "run"), everything works fine.
The Invocation is done proper, the process is visible in the activity monitor and
the NSLevelIndicator (on the guy of the innovating app) is proceeding well according
the (float) progress variable.
BUT: if i "archive" the application and execute it outside of Xcode, the stdout of my
Command Line Tool never seem to reach the application. I tried writing a debug file
in
-(void) dataReady:(NSNotification*)n
No chance, it is never called! I tested the issue on several Macs, same problem...
Did I make an obvious mistake or is there some preferences to configure (Sandboxing is off),
maybe known issues that I overlooked?
Thank you for help
Greetings
Mat
I'm trying to figure out how to set up IPC between my custom app and a pre-made program.
I'm using MacOSX Lion 10.7.2 and Xcode 4.2.1.
It doesn't matter actually what program exactly, since I believe that a similar reasoning may be applied to any kind of external process.
For testing purposes I'm using a simple bash script:
#test.sh
echo "Starting"
while read out
do
echo $out
done
What I would like to achieve is to redirect input and output of this script, using my app to send inputs to it and read its outputs.
I tried to use NSTask,NSPipe and NSFileHandle as follows:
-(void)awakeFromNib {
task = [[NSTask alloc] init];
readPipe = [NSPipe pipe];
writePipe = [NSPipe pipe];
[task setStandardOutput:readPipe];
[task setStandardInput:writePipe];
[task setLaunchPath:#"/path/test.sh"];
[[NSNotificationCenter defaultCenter] addObserver:self
selector:#selector(read:)
name:NSFileHandleReadCompletionNotification
object:nil];
[[readPipe fileHandleForReading] readInBackgroundAndNotify];
[task launch];
}
-(IBAction)write:(id)sender {
NSLog(#"Write called: %d %#\n",[task isRunning],writePipe);
NSFileHandle *writeHandle = [writePipe fileHandleForWriting];
NSString *message = #"someString";
[writeHandle writeData:[message dataUsingEncoding:NSUTF8StringEncoding] ];
}
-(void)read:(NSNotification*)notification {
NSString *output = [[NSString alloc] initWithData:[[notification userInfo] valueForKey: NSFileHandleNotificationDataItem]
encoding:NSUTF8StringEncoding];
NSLog(#"%#",output);
[output release];
[[notification object] readInBackgroundAndNotify];
}
but I'm able only to read the output of test.sh, not to send it any input.
Actually any other example I saw on the web is pretty similar to my code, so I'm not sure if this issue is due to some mistake(s) of mine or to other issues (like app's sandboxing of MacOS Lion).
I've checked XPC documentation, but, according to my researches, in order to use XPC API to IPC, both sides should connect to the same service.
That's not what I'm looking for since I don't want to alter the script in any way, I just want redirect its input and output.
Is my issue due to the lack of XPC and/or to app's sandboxing?
If yes, is there a way to use XPC without modifying the script?
If no, then may somebody explain me what I'm doing wrong?
You don't need XPC for this. Won't make any difference.
Is your script / external process able to read input when you pipe something to it on the command line
% echo "foobar" | /path/test.sh
?
How much data are you sending to it. Writing will be buffered. IIRC -synchronizeFile will flush buffers -- same as fsync(2).
I need to read the last added line to a log file, in realtime, and capture that line being added.
Something similar to Tail -f.
So my first attempt was to use Tail -f using NSTask.
I can't see any output using the code below:
NSTask *server = [[NSTask alloc] init];
[server setLaunchPath:#"/usr/bin/tail"];
[server setArguments:[NSArray arrayWithObjects:#"-f", #"/path/to/my/LogFile.txt",nil]];
NSPipe *outputPipe = [NSPipe pipe];
[server setStandardInput:[NSPipe pipe]];
[server setStandardOutput:outputPipe];
[server launch];
[server waitUntilExit];
[server release];
NSData *outputData = [[outputPipe fileHandleForReading] readDataToEndOfFile];
NSString *outputString = [[[NSString alloc] initWithData:outputData encoding:NSUTF8StringEncoding] autorelease];
NSLog (#"Output \n%#", outputString);
I can see the output as expected when using:
[server setLaunchPath:#"/bin/ls"];
How can i capture the output of that tail NSTask?
Is there any alternative to this method, where I can open a stream to file and each time a line is added, output it on screen? (basic logging functionality)
This is a little tricky to do your way, as readDataToEndOfFile will wait until tail closes the output stream before returning, but tail -f never closes the output stream (stdout). However, this is actually pretty simple to do with basic C I/O code, so I whipped up a simple FileTailer class that you can check out. It's not anything fancy, but it should show you how it's done. Here're the sources for FileTailer.h, FileTailer.m, and a test driver.
The meat of the class is pretty simple. You pass it a block, and it reads a character from the stream (if possible) and passes it to the block; if EOF has been reached, it waits a number of seconds (determined by refresh) and then tries to read the stream again.
- (void)readIndefinitely:(void (^)(int ch))action
{
long pos = 0L;
int ch = 0;
while (1) {
fseek(in, pos, SEEK_SET);
int ch = fgetc(in);
pos = ftell(in);
if (ch != EOF) {
action(ch);
} else {
[NSThread sleepForTimeInterval:refresh];
}
}
}
You can call it pretty simply, like this:
FileTailer *tail = [[[FileTailer alloc] initWithStream:stdin refreshPeriod:3.0] autorelease];
[tail readIndefinitely:^ void (int ch) { printf("%c", ch); }];
(Caveat: I wrote the FileTailer class pretty fast, so it's kind of ugly right now and should be cleaned up a bit, but it should serve as a decent example on how to read a file indefinitely, à la tail -f.)
Here's a way to use "tail -f logfile" via NSTask in Objective-C:
asynctask.m -- sample code that shows how to implement asynchronous stdin, stdout & stderr streams for processing data with NSTask
...
Being a GUI-less application (i.e. a Foundation-based command line tool), asynctask.m runs an NSRunLoop manually
to enable the use of asynchronous "waitForDataInBackgroundAndNotify" notifications. In addition, asynctask.m
uses pthread_create(3) and pthread_detach(3) for writing more than 64 KB to the stdin of an NSTask.
Source code of asynctask.m available at: http://www.cocoadev.com/index.pl?NSPipe