I use the NSTask to run shell command and output the data via NSPipe. At first, I using bellow method to read output data, it is no any problem.
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData = [[NSMutableData alloc] init];
NSData *taskData = nil;
if((taskData = [readHandle availableData]) && [taskData length])
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
The problem about the method is that it only output 4096 bytes data. So I using while loop to get more data, modify the method like this：
- (void)outputAvailable:(NSNotification *)aNotification {
NSString *newOutput;
NSMutableData *allData; //Added.
NSData *taskData = nil;
while ((taskData = [readHandle availableData]) && [taskData length]) {
[allData appendData:taskData];
}
newOutput = [[NSString alloc] initWithData:allData encoding:NSASCIIStringEncoding];
NSLog(#"%#", newOutput);
[readHandle readInBackgroundAndNotify];
}
Then problem occurs: the program is blocking in the while loop and can not perform the following statements. I ensure that allData is what I wanted, but after appending the last data chunk, it is blocking.
Could you give me some solutions? Thanks.
Your while() loop effectively blocks further notifications, causing the whole program to block waiting for something to flush the buffer.
You should readInBackgroundAndNotify, then pull off availableBytes on each notification, appending it to your NSMutableData (which is likely held in an instance variable). When you handle the notification, don't attempt to wait for more data or do any kind of a while loop. The system will notify you when more data is available.
I.e. the system pushes data to you, you do not pull data from the system.
Ahh... OK. You should still only pull data when there is data available. Your while() loop is doing that. Not enough coffee. My bad.
The final block is most likely because your external process is not closing the pipe; no EOF is received and, thus, the program is waiting forever for more data that never arrives.
Either:
make sure the background task exits
detect when you've received enough data and terminate the process
If you are doing some kind of conversion program (say, tr) where you write data on the processes standard input, then you might need to close the standard input pipe.
Related
Basic Setup
I use NSTask to run a process that optimizes images. This process writes output data to stdout. I use the readabilityHandler property of NSTask to capture that data. Here is the abbreviated setup:
NSTask *task = [[NSTask alloc] init];
[task setArguments:arguments]; // arguments defined above
NSPipe *errorPipe = [NSPipe pipe];
[task setStandardError:errorPipe];
NSFileHandle *errorFileHandle = [errorPipe fileHandleForReading];
NSPipe *outputPipe = [NSPipe pipe];
[task setStandardOutput:outputPipe];
NSFileHandle *outputFileHandle = [outputPipe fileHandleForReading];
NSMutableData *outputData = [[NSMutableData alloc] init];
NSMutableData *errorOutputData = [[NSMutableData alloc] init];
outputFileHandle.readabilityHandler = ^void(NSFileHandle *handle) {
NSLog(#"Appending data for %#", inputPath.lastPathComponent);
[outputData appendData:handle.availableData];
};
errorFileHandle.readabilityHandler = ^void(NSFileHandle *handle) {
[errorOutputData appendData:handle.availableData];
};
I then call NSTask like this:
[task setLaunchPath:_pathToJPEGOptim];
[task launch];
[task waitUntilExit];
(This is all done on a background dispatch queue). Next I examine the return values of NSTask:
if ([task terminationStatus] == 0)
{
newSize = outputData.length;
if (newSize <= 0)
{
NSString *errorString = [[NSString alloc] initWithData:errorOutputData encoding:NSUTF8StringEncoding];
NSLog(#"ERROR string: %#", errorString);
}
// Truncated for brevity...
}
The Problem
Approximately 98% of the time, this works perfectly. However, it appears that -waitUntilExit CAN fire before the readabilityHandler block is run. Here is a screenshot showing that the readability handler is running AFTER the task has exited:
So this is clearly a race condition between the dispatch queue running the readabilityHandler and the dispatch queue where I've fired off my NSTask. My question is: how the hell can I determine that the readabilityHandler is done? How do I beat this race condition if, when NSTask tells me it's done, it may not be done?
NOTE:
I am aware that NSTask has an optional completionHandler block. But the docs state that this block is not guaranteed to run before -waitUntilExit returns, which implies that it CAN begin running even SOONER than -waitUntilExit. This would make the race condition even more likely.
Update: Modern macOS:
availableData no longer has the issues I describe below. I'm unsure precisely when they were resolved, but at least Monterey works correctly. The approach described below is for older releases of macOS.
Additionally, with the modern Swift concurrency system in place and the new paradigm of "threads can always make forward progress", using semaphores like below should be a last resort. If you can, use NSTask's completionHandler API. I have no FORMAL guarantee that the readability handlers will complete before the completionHandler is called, but they seem to in practice, at least on modern macOS. Your mileage may vary.
Old Advice:
Ok, after much trial-and-error, here's the correct way to handle it:
1. Do not Use -AvailableData
In your readability handler blocks, do not use the -availableData method. This has weird side effects, will sometimes not capture all available data, and will interfere with the system's attempt to call the handler with an empty NSData object to signal the closing of the pipe because -availableData blocks until data is actually available.
2. Use -readDataOfLength:
Instead, use -readDataOfLength:NSUIntegerMax in your readability handler blocks. With this approach, the handler correctly receives an empty NSData object that you can use to detect the closing of the pipe and signal a semaphore.
3. Beware macOS 10.12!
There is a bug that Apple fixed in 10.13 that is absolutely critical here: on old versions of macOS, the readability handlers are never called if there is no data to read. That is, they never get called with zero-length data to indicate that they’re finished. That results in a permanent hang using the semaphore approach because the semaphore is never incremented. To combat this, I test for macOS 10.12 or below and, if I’m running on an old OS, I use a single call to dispatch_semaphore_wait() that is paired with a single call to dispatch_semaphore_signal() in NSTask’s completionHandler block. I have that completion block sleep for 0.2 seconds to allow the handlers to execute. That’s obviously a godawfully ugly hack, but it works. If I’m on 10.13 plus, I have different readability handlers that signal the semaphore (once from the error handler and once from the normal output handler) and I still signal the semaphore from the completionHandler block. These are paired with 3 calls to dispatch_semaphore_wait() after I launch the task. In this case, no delay is required in the completion block because macOS correctly calls the readability handlers with zero-length data when the fileHandle is done.
Example:
(Note: assume stuff is defined as in my original question example. This code is shortened for readability.)
// Create the semaphore
dispatch_semaphore_t sema = dispatch_semaphore_create(0);
// Define a handler to collect output data from our NSTask
outputFileHandle.readabilityHandler = ^void(NSFileHandle *handle)
{
// DO NOT use -availableData in these handlers.
NSData *newData = [handle readDataOfLength:NSUIntegerMax];
if (newData.length == 0)
{
// end of data signal is an empty data object.
outputFileHandle.readabilityHandler = nil;
dispatch_semaphore_signal(sema);
}
else
{
[outputData appendData:newData];
}
};
// Repeat the above for the 'errorFileHandle' readabilityHandler.
[task launch];
// two calls to wait because we are going to signal the semaphore once when
// our 'outputFileHandle' pipe closes and once when our 'errorFileHandle' pipe closes
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER);
// ... do stuff when the task is done AND the pipes have finished handling data.
// After doing stuff, release the semaphore
dispatch_release(sema);
sema = NULL;
Create a semaphore with an initial value of 0. In the readability handlers, check if the data object returned from availableData has length 0. If it does, that means end of file. In that case, signal the semaphore.
Then, after waitUntilExit returns, wait on the semaphore twice (once for each pipe you're reading). When those waits return, you've got all of the data.
I know that people have asked this but I have not found satisfactory answers. I have one method that I send all my URLRequests through. I return the response of the request as a string when the method completes. I have recently added ssl to my program. This means that I can no longer use a synchronous request because I need to take advantage of the didReceiveAuthenticationChallenge function as my credentials are currently self-signing. The program needs the response from the URL in order to continue so there is not harm in waiting for the response. However, I cannot seem to find a way to just hold the code up and continue once completed. I can alert the original function that called to request function but I would like the program to pick up right after that call. And it has unique code below such calls so I cannot specialize the connectionDidFinishLoading: function because each method who calls this is different.
How can I pause the program so I can return the nsdata from the connection to the methods that called it?
Here is some pseudo-code to show you what I mean:
- (void) login:(NSString *)username :(NSString *)password {
NSString *str = [NSString stringWithFormat:%#"%#:::%#",username,password];
NSURL *url = [NSURL urlWithString:#"https://blahblahblah"];
NSString *result = [self connectToUrl:str:url];
if ([result isEqualToString:#"valid"]) {
//this would be more complex in here
NSLog(#"hooray");
} else {
NSLog(#"bummer");
}
}
- (NSString *)connectToUrl:(NSURL *)url :(NSString *)str {
NSData *FileData = [str dataUsingEncoding: NSUTF8StringEncoding];
NSMutableData *data = [[NSMutableData alloc] initWithCapacity:100];
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:url];
[request setHTTPMethod:#"POST"];
//set up the rest of the request...
...
connection = [NSURLConnection connectionWithRequest:request delegate:self];
[connection start];
//WOULD LIKE TO PAUSE HERE UNTIL COMPLETE! THEN CONTINUE
// received data is assigned in didReceiveData: method
return [[[NSString alloc] initWithData:receivedData encoding:NSUTF8StringEncoding] autorelease];
}
But alas, I cannot do this because I cannot make the final line wait until the connection is complete... Please help me!
Very appreciative!
R
iOS and OS X and much of the Cocoa/Cocoa touch frameworks are built on an event model. You don't pause your app. That's not the proper approach. You need to start the connection and then move on. When the connection completes, you act on that event.
In other words, your login method can't sit and wait for the result. It should start the connection and return.
When you get the result of the connection you call some method to process the login result.
Making use of blocks can make things like this easier but there are other ways. You just need to stop thinking about such things in a linear fashion. Dealing with asynchronous processing requires a different approach.
I've been trying to make do (see this and this) with the recent NSUserScriptTask class and its subclasses and so far I've solved some problems, but some others remain to be solved. As you can see from the docs, NSUserScriptTask does not allow for the cancellation of tasks. So, I decided to create a simple executable that takes as arguments the path to the script and runs the script. That way, I can launch the helper from my main app using NSTask and call [task terminate] when necessary. However, I require:
The main app to receive output and errors from the helper it launched
The helper only terminating when the NSUserScriptTask is done
The code for the main app is simple: just launch an NSTask with the proper info. Here's what I have now (for the sake of simplicity I ignored the code for security-scoped bookmarks and the like, which are out of the problem. But don't forget this is running sandboxed):
// Create task
task = [NSTask new];
[task setLaunchPath: [[NSBundle mainBundle] pathForResource: #"ScriptHelper" ofType: #""]];
[task setArguments: [NSArray arrayWithObjects: scriptPath, nil]];
// Create error pipe
NSPipe* errorPipe = [NSPipe new];
[task setStandardError: errorPipe];
// Create output pipe
NSPipe* outputPipe = [NSPipe new];
[task setStandardOutput: outputPipe];
// Set termination handler
[task setTerminationHandler: ^(NSTask* task){
// Save output
NSFileHandle* outFile = [outputPipe fileHandleForReading];
NSString* output = [[NSString alloc] initWithData: [outFile readDataToEndOfFile] encoding: NSUTF8StringEncoding];
if ([output length]) {
[output writeToFile: outputPath atomically: NO encoding: NSUTF8StringEncoding error: nil];
}
// Log errors
NSFileHandle* errFile = [errorPipe fileHandleForReading];
NSString* error = [[NSString alloc] initWithData: [errFile readDataToEndOfFile] encoding: NSUTF8StringEncoding];
if ([error length]) {
[error writeToFile: errorPath atomically: NO encoding: NSUTF8StringEncoding error: nil];
}
// Do some other stuff after the script finished running <-- IMPORTANT!
}];
// Start task
[task launch];
Remember, I need the termination handler to only run when: (a) the task was cancelled (b) the task terminated on its own because the script finished running.
Now, on the helper side things start to get hairy, at least for me. Let's imagine for the sake of simplicity that the script is an AppleScript file (so I use the NSUserAppleScriptTask subclass - on the real world I'd have to accomodate for the three types of tasks). Here's what I got so far:
int main(int argc, const char * argv[])
{
#autoreleasepool {
NSString* filePath = [NSString stringWithUTF8String: argv[1]];
__block BOOL done = NO;
NSError* error;
NSUserAppleScriptTask* task = [[NSUserAppleScriptTask alloc] initWithURL: [NSURL fileURLWithPath: filePath] error: &error];
NSLog(#"Task: %#", task); // Prints: "Task: <NSUserAppleScriptTask: 0x1043001f0>" Everything OK
if (error) {
NSLog(#"Error creating task: %#", error); // This is not printed
return 0;
}
NSLog(#"Starting task");
[task executeWithAppleEvent: nil completionHandler: ^(NSAppleEventDescriptor *result, NSError *error) {
NSLog(#"Finished task");
if (error) {
NSLog(#"Error running task: %#", error);
}
done = YES;
}];
// Wait until (done == YES). How??
}
return 0;
}
Now, I have three questions (which are the ones I want to ask with this SO entry). Firstly, "Finished task" never gets printed (the block never gets called) because the task never even starts executing. Instead, I get this on my console:
MessageTracer: msgtracer_vlog_with_keys:377: odd number of keys (domain: com.apple.automation.nsuserscripttask_run, last key: com.apple.message.signature)
I tried running the exact same code from the main app and it completes without a fuss (but from the main app I lose the ability to cancel the script).
Secondly, I only want to reach the end of main (return 0;) after the completion handler is called. But I have no idea how to do that.
Thridly, whenever there's an error or output from the helper I want to send that error/output back to the app, which will receive them through the errorPipe/outputPipe. Something like fprintf(stderr/stdout, "string") does the trick, but I'm not sure if it is the right way to do it.
So, in short, any help regarding the first and second problems is appreciated. The third one I just want to make sure that's how I'm supposed to do it.
Thanks
Question 1: The sub-task doesn't run because its parent exits immediately. (The log message about "odd number of keys" is a bug in NSUserScriptTask, and happens because your helper doesn't have a bundle identifier, but is otherwise harmless and irrelevant to your problem.) It exits immediately because it's not waiting for the completion block to fire, which brings us to...
Question 2: How do you wait for an asynchronous completion block? This has been answered elsewhere, including Wait until multiple networking requests have all executed - including their completion blocks, but to recap, use dispatch groups, something like this:
dispatch_group_t g = dispatch_group_create();
dispatch_group_enter(g);
[task executeWithAppleEvent:nil completionHandler:^(NSAppleEventDescriptor *result, NSError *e) {
...
dispatch_group_leave(g);
}];
dispatch_group_wait(g, DISPATCH_TIME_FOREVER);
dispatch_release(g);
This same pattern works for any call that has a completion block you want to wait for. If you wanted another notification when the group finishes instead of waiting for it, use dispatch_group_notify instead of dispatch_group_wait.
As a side note, the way you’re testing error after allocating the NSUserAppleScriptTask is incorrect. The value of error is defined if and only if the function result is nil (or NO, or whatever indicates failure). If the function succeeds (which you know if it returns non-nil), then error may be anything -- the function may set it to nil, it may leave it undefined, it may even fill it in with a real object. (See also What's the Point of (NSError**)error?)
I have a situation that I receive a byte data through Web Services request and want to write it to a file on my iOS device. I used to append all data (till end of data) in a memory variable and at the end writing the data using NSStream to a file in my iOS device using method:
stream:(NSStream *)theStream handleEvent:(NSStreamEvent)streamEvent
It works fine for small size of data, but the problem is if I am receiving data via web services it could be a big chunk (couple MBs) and I don't want to collect all in memory to write it to the file, to make it efficent I think I have to switch to NSFileHandle to write data in a small chunk size to the same file in several times. Now my question is what is the best approach to do this? I mean how can I do write to the file in BACKGROUND using NSFileHandle? I use code like this:
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
if(![[NSFileManager defaultManager] fileExistsAtPath:newFilePath ]) {
[[NSFileManager defaultManager] createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
}
but with passing a data size of 1-2 Mb to above method, do I need to make it running in background? FYI I'm writing in main thread.
Maybe you can try Grand Central Dispatch.
I spent some time trying it, bellow is my way to do it.
According to Apple's document, if our program need executing only one task at a time, we should create a "Serial Dispatch Queue".So, first declare a queue as iVar.
dispatch_queue_t queue;
create a serial dispatch queue in init or ViewDidLoad using
if(!queue)
{
queue = dispatch_queue_create("yourOwnQueueName", NULL);
}
When data occurs, call your method.
- (void) setUpAsynchronousContentSave:(NSData *) data {
NSString *newFilePath = [NSHomeDirectory() stringByAppendingPathComponent:#"/Documents/MyFile.xml"];
NSFileManager *fileManager = [[NSFileManager alloc] init];
if(![fileManager fileExistsAtPath:newFilePath ]) {
[fileManager createFileAtPath:newFilePath contents:nil attributes:nil];
}
if(!fileHandle_writer) {
self.fileHandle_writer = [NSFileHandle fileHandleForWritingAtPath:newFilePath];
}
dispatch_async( queue ,
^ {
// execute asynchronously
[fileHandle_writer seekToEndOfFile];
[fileHandle_writer writeData:data];
});
}
At last, we need to release the queue in ViewDidUnload or dealloc
if(queue)
{
dispatch_release(queue);
}
I combine these code with ASIHttp, and it works.
Hope it helps.
I am trying to read a text file containing characters in billions. Using the function
contentOfFile is not working, as my application get crashed due to it.
So anybody please send me the sample code so that I get the chunks according to my requirement.Whichever i need i wanna get that one only.
please reply as soon as possible.
I'm guessing this is an iOS app. In that case, you are likely hitting the memory limit by calling contentsOfFile: because that method is trying to read the entire contents of the file into a variable (memory). Remember that on iOS your app must play nice and if it decides to consume too much memory, then the watchdog process will kill your app to save the device from rebooting (which happens because there is no disk to swap to on iOS devices).
Have you had a look at NSFileHandle? NSFileHandle supports seeking within a text a file. With some simple iteration you can use the following to methods to seek within the file and read chunks of data:
- (NSData *)readDataOfLength:(NSUInteger)length;
- (void)seekToFileOffset:(unsigned long long)offset;
It might look something like this. Assume pathToFile is an NSString containing the path to the text file to be read in.
uint64 offset = 0;
uint32 chunkSize = 1024; //Read 1KB chunks.
NSFileHandle *handle = [NSFileHandle fileHandleForReadingAtPath:pathToFile];
NSData *data = [handle readDataOfLength:chunkSize];
while ([data length] > 0)
{
//Make sure for the next line you choose the appropriate string encoding.
NSString *dataString = [[[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding] autorelease];
/* PERFORM STRING PROCESSING HERE */
/* END STRING PROCESSING */
offset += [data length];
[handle seekToFileOffset:offset];
data = [handle readDataOfLength:chunkSize];
}
[handle closeFile];
A good idea is to look at the textedit source because I've opened massive files with it before and there should be a way to do it. Not sure why your app is crashing though. It shouldn't have a problem.