I recently upgraded to MacOS Ventura and discovered that software I'd written in the past to scrape data from a webpage would no longer work. That code used initWithContentsOfURL to access and download a sequence of pages which my code would then process (yes, I know synchronous networking is bad practice, but this is a research app only I use, not commercial software).
I then modified to code to use NSURLSessionTask and dataTaskWithURL to grab the pages asynchronously, which worked great, except... since my software is now making 10 requests in rapid order, the website is generating Error 1015 informing me that I'm being rate limited and temporarily banned.
I guess this means I now have to slow down my code when making the requests? My code looks like this (cutting it down from 10 requests to 3, removing error handling, and NSURL initialization):
NSURLSession *session = [NSURLSession sharedSession];
__block NSURLSessionTask *task1 = nil;
__block NSURLSessionTask *task2 = nil;
__block NSURLSessionTask *task3 = nil;
task1 = [session dataTaskWithURL:pageURL1 completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (data) {
task1Source = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
}
[task2 resume];
}];
task2 = [session dataTaskWithURL:pageURL2 completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (data) {
task2Source = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
}
[task3 resume];
}];
task3 = [session dataTaskWithURL:pageURL3 completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (data) {
task3Source = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
}
}];
[task1 resume];
The site I'm hitting obviously does not like this, which has generated the rate limit error. Is there a preferred way to distribute the requests or otherwise generate an acceptable delay that will be acceptable to the site (I also have a request out to the site's tech support)?
Thanks!
Related
I am new to Mac application development and our existing Mac application contains the following line of code
[NSURLConnection sendSynchronousRequest:request returningResonse:response error:Error
A warning message getting displayed as
sendSynchronousRequest is deprecated in macOS 10.11
and suggesting to use [NSURLSession dataTaskWithRequest:completionHandler:]
I have implemented the following code changes to use NSURLSession as suggested but it is returning the value of data as "nil". Can you please suggest what needs to be done in order to get the required data in response?
__block NSError *WSerror;
__block NSURLResponse *WSresponse;
__block NSData *myData;
NSURLSession *session =[NSURLSession sharedSession];
[[session completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
myData = data;
WSresponse =response;
WSerror = error;
}] resume];
NSString *theXml = [[NSString alloc] initWithData:myData encoding:NSUTF8StringEncoding];
return theXml;
The completion handler is asynchronous. You have to do your work inside the completion handler.
The __block declarations are pointless.
NSURLSession *session =[NSURLSession sharedSession];
[[session completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (error) {
NSLog(#"%#", error);
} else {
NSString *theXml = [[NSString alloc] initWithData: data encoding:NSUTF8StringEncoding];
// do something with `theXml`
}
}] resume];
I must be making this harder than it is... or implementing the solutions I see online incorrectly.
I have an array of URLs which I would like to loop through and push the results to a dictionary in order or the array. How can I make it wait for the dictionary to be updated before running the next request? Basically I want to make the calls synchronously in a background thread.
Here is where I call the download:
for (NSString *path in paths) {
NSURLSession *session = [NSURLSession sessionWithConfiguration [NSURLSessionConfiguration defaultSessionConfiguration]];
NSURLRequest *request = [[NSURLRequest alloc] initWithURL:[NSURL URLWithString:path]
cachePolicy:NSURLRequestUseProtocolCachePolicy
timeoutInterval:10];
NSURLSessionDataTask *task = [session dataTaskWithRequest:request
completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (error)
{
}
else
{
NSError *parsingError = nil;
NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data
options:NSJSONReadingAllowFragments
error:&error];
if (parsingError)
{
}
else
{
[myDictionary addObject:dict];
}
}
}];
[task resume];
}
Unless one request really requires the results of the prior request before being issued (which is not the case here), you should not run them sequentially. It may feel more logical to issue the sequentially, but you pay a huge performance penalty to do so. Issue them concurrently, save the results in some unordered structure (like a dictionary), and then when all done, build your ordered structure.
NSMutableDictionary *results = [NSMutableDictionary dictionaryWithCapacity:[paths count]];
// don't create new session for each request ... initialize this outside of the loop
NSURLSession *session = [NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]]; // or since you're not doing anything special here, use `sharedSession`
// since we're going to block a thread in the process of controlling the degree of
// concurrency, let's do this on a background queue; we're still blocking
// a GCD worker thread as these run (which isn't ideal), but we're only using
// one worker thread.
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
// only do for requests at a time, so create queue a semaphore with set count
dispatch_semaphore_t semaphore = dispatch_semaphore_create(4); // only do four requests at a time
// let's keep track of when they're all done
dispatch_group_t group = dispatch_group_create();
// now let's loop through issuing the requests
for (NSString *path in paths) {
dispatch_group_enter(group); // tell the group that we're starting another request
dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER); // wait for one of the four slots to open up
NSURLRequest *request = [[NSURLRequest alloc] initWithURL:[NSURL URLWithString:path]
cachePolicy:NSURLRequestUseProtocolCachePolicy
timeoutInterval:10];
NSURLSessionDataTask *task = [session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (error) {
// ...
} else {
NSError *parsingError = nil;
NSDictionary *dict = [NSJSONSerialization JSONObjectWithData:data options:0 error:&error];
if (parsingError) {
} else {
// synchronize updating of dictionary
dispatch_async(dispatch_get_main_queue(), ^{
results[path] = dict;
});
}
}
dispatch_semaphore_signal(semaphore); // when done, flag task as complete so one of the waiting ones can start
dispatch_group_leave(group); // tell the group that we're done
}];
[task resume];
}
dispatch_group_notify(group, dispatch_get_main_queue(), ^{
// trigger whatever you want when they're all done
// and if you want them in order, iterate through the paths and pull out the appropriate result in order
for (NSString *path in paths) {
// do something with `results[path]`
}
});
});
I tried to reduce the amount of extra dependencies here, so I used dispatch groups and semaphores. In the above, I use semaphores to constrain the degree of concurrency and I use dispatch group to identify when it's all done.
Personally, I wouldn't use semaphores and groups, but rather I'd wrap these requests in asynchronous NSOperation subclass, but I was trying to limit the changes I made to your code. But the NSOperation idea is logically the same as the above: Run them concurrently, but constrain the degree of concurrency so you don't end up with them timing out on you, and trigger the retrieval of the results only when all the results are retrieved.
Having an app written in Objective-C targeting iOS8/9 there are real vantange in performance or stability updating code using NSURLSession
NSURLSession *session = [NSURLSession sharedSession];
[[session dataTaskWithRequest:request
completionHandler:^(NSData *data,
NSURLResponse *response,
NSError *error) {
dispatch_sync(dispatch_get_main_queue(), ^{
//UPDATE UI
});
}] resume];
in place of code like the following that use NSURLConnection:
[NSURLConnection sendAsynchronousRequest:request
queue:[CMRequestManager connectionQueue]
completionHandler:^(NSURLResponse *response,
NSData *data, NSError *connectionError)
{
dispatch_sync(dispatch_get_main_queue(), ^{
//UPDATE UI
});
}];
If you decide to run the code in a WatchKit extension or on tvOS at some point in the future, then yes. Otherwise, if the snippet above is representative of the way you're using the API, then I probably wouldn't bother rewriting it. With that said, this is very much a matter of opinion.
My code calls HTTP post call to remote server and obtains results in JSON format, I have extracted this JSON, but now I need to store these results to SQLite. Based on my reading NSURLSessionDataTask is background thread, so my confusion is, should I call SQL open and insert inside completionHandler (or) is there any other best practice to handle this type of requirements?
EDIT 1
The point I am struggling more is, is it valid to write SQLite operations inside "completionHandler"? does "completionHandler" will be considered as method on separate thread (which is executing SessionDataTask) or main thread?
EDIT 2
I am open to CoreData related solutions too.
Any help would be appreciated.
NSURL *loginUrl = [NSURL URLWithString:#"myurl"];
NSURLSession *session = [NSURLSession sharedSession];
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:loginUrl];
request.HTTPMethod = #"POST";
NSString *ipData = [NSString stringWithFormat:#"uName=%#&pwd=%#",self.userName.text,self.userPwd.text];
request.HTTPBody = [ipData dataUsingEncoding:NSUTF8StringEncoding];
NSURLSessionDataTask *postDataTask = [session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *jsonError) {
NSLog(#"Inside post data task......");
NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)response;
if(httpResp.statusCode == 200)
{
NSLog(#"Response succesfull.........");
NSDictionary *jsonDict = [NSJSONSerialization JSONObjectWithData:data options:NSJSONReadingMutableContainers error:&jsonError];
if(!jsonError)
{
//No Json error
NSString *uName = jsonDict[#"userName"];
NSString *uID = jsonDict[#"uID"];
//HOW CAN I INSERT THESE DETAILS TO SQLITE DB BEFORE CALLING SEGUE TO MOVE TO NEXT SCREEN?
dispatch_async(dispatch_get_main_queue(), ^{
[self performSegueWithIdentifier:#"mysegueID" sender:self];
});
}
}else
{
NSLog(#"Response is not succesfulll...");
}
}];
[postDataTask resume];
A lot of people use FMDB as objective-c wrapper around sqlite.
In case of NSURLSession, the block of the completion handler will be executed on the "delegate queue" (see delegateQueue property of NSURLSession).
It is valid to do SQLite in completion handler as long as you follow SQLite threading rules. I recommend FMDB her again because it has helpers for this. See Using FMDatabaseQueue and Thread Safety.
So your example would look like:
FMDatabaseQueue *queue = [FMDatabaseQueue databaseQueueWithPath:aPath];
NSURL *loginUrl = [NSURL URLWithString:#"myurl"];
NSURLSession *session = [NSURLSession sharedSession];
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:loginUrl];
request.HTTPMethod = #"POST";
NSString *ipData = [NSString stringWithFormat:#"uName=%#&pwd=%#",self.userName.text,self.userPwd.text];
request.HTTPBody = [ipData dataUsingEncoding:NSUTF8StringEncoding];
NSURLSessionDataTask *postDataTask = [session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *jsonError) {
NSLog(#"Inside post data task......");
NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)response;
if(httpResp.statusCode == 200)
{
NSLog(#"Response succesfull.........");
NSDictionary *jsonDict = [NSJSONSerialization JSONObjectWithData:data options:NSJSONReadingMutableContainers error:&jsonError];
if(!jsonError)
{
//No Json error
NSString *uName = jsonDict[#"userName"];
NSString *uID = jsonDict[#"uID"];
//HOW CAN I INSERT THESE DETAILS TO SQLITE DB BEFORE CALLING SEGUE TO MOVE TO NEXT SCREEN?
[queue inDatabase:^(FMDatabase *db) {
NSDictionary *argsDict = #{ #"uName" : uName, #"uID" : uID};
[db executeUpdate:#"INSERT INTO myTable (name) VALUES (:name)" withParameterDictionary:argsDict];
}];
dispatch_async(dispatch_get_main_queue(), ^{
[self performSegueWithIdentifier:#"mysegueID" sender:self];
});
}
}
else
{
NSLog(#"Response is not succesfulll...");
}
}];
[postDataTask resume];
A SQLite DB can be accessed from any thread in an app. The only restriction is that SQLite does not happily tolerate simultaneous access from multiple threads (and "simultaneous" here applies to the duration of a "transaction", not simply the duration of a call to SQLite methods).
So you must somehow assure that there is never simultaneous access. A simple way to do this is to always use the same thread (eg, the main thread) for access. Or you can implement "soft" protocols such that you know that two actions are not simultaneously trying to use the DB because they are separated in time. Or you can make use of Objective-C lock or other synchronization mechanisms in the software/OS.
I am trying to pull some data from my local node server. The server is getting the get request and logging it, but for some reason my iOS app will not execute any of the code that I have in the completion handler. Here is the code:
- (IBAction) buttonPressed{
NSURL *url = [NSURL URLWithString:#"http://127.0.0.1:3000/"];
NSURLSessionDataTask *dataTask =
[self.session dataTaskWithURL:url
completionHandler:^(NSData *data,
NSURLResponse *response,
NSError *error){
nameLabel.text = #"yay!";
/*
if (!error){
nameLabel.text = #"noerr";
NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)response;
if (httpResp.statusCode == 200){
NSError *jsonErr;
NSDictionary *usersJSON =
[NSJSONSerialization JSONObjectWithData:data
options:NSJSONReadingAllowFragments
error:&jsonErr];
if (!jsonErr){
// nameLabel.text = usersJSON[#"username"];
nameLabel.text = #"nojerr";
}
else{
nameLabel.text = #"jsonErr";
}
}
}
else{
nameLabel.text = #"Err";
}
*/
}];
[dataTask resume];
}
When the program is run, the nameLabel is not changed to "yay". However if I try to change the nameLabel before the NSURLSessionDataTask line, it changes.
NSURLSessionDataTask runs in a background thread. To update anything in the user interface such as labels, buttons, table views, etc, you must do so on the main thread. If you want to update the label text from the completionHandler block then you need to update the label in the main thread like so:
dispatch_sync(dispatch_get_main_queue(), ^{
nameLabel.text = #"yay!";
});
try this magic:
static NSURLSession* sharedSessionMainQueue = nil;
if(!sharedSessionMainQueue){
sharedSessionMainQueue = [NSURLSession sessionWithConfiguration:nil delegate:nil delegateQueue:[NSOperationQueue mainQueue]];
}
NSURLSessionDataTask *dataTask =
[sharedSessionMainQueue dataTaskWithURL:url completionHandler:^(NSData *data,
NSURLResponse *response,
NSError *error){
//now will be on main thread
}];
[dataTask resume];
This gives you the original behavior of NSURLConnection with the completing handler on the main thread so you are safe to update the UI. However, say you would like to parse the download or do some heavy processing, in that case you might benefit from the completion handler on the operation queue's background thread and then using dispatch_sync to the main thread as a final step.