use case of process_spider_input in spidermiddleware - scrapy

Does anyone know the difference between process_spider_input(response, spider) in spidermiddleware and process_response(request, response, spider) in Downloadermiddleware.
And how to choose one over another, because I see they do quite the same work, they handle response.

According to the source, they do have difference
return value
spider_mw.process_spider_input() returns None, you can check or modify the Response. Basically it supposes the response has been accepted and you can't refuse it.
downloader_mw.process_response() returns Response or Request. You can refuse the response from download handler and generate a new request. (e.g. the RetryMiddleware)

Related

ASP.NET Core Web API - Return a string message with a NoContent Response?

I have some records in the database that may or may not have some information for a certain column, and that's OK.
If a consumer of the endpoint runs into one of these records, I'd like to return "NoContent", because, well, there's No Content for that record that we can execute on. BUT ... I'd like to give them a message why, as there are two distinct reasons.
When looking at other status codes, this works:
return Accepted("I SEE THIS STRING!");
It actually shows up in the Response Header as the field location (I think, don't make me run this project again!)
This works for say Internal Server error:
return StatusCode(500, "I SEE THIS TOO!");
But, for NoContent there is no constructor like Accepted, where you can pass an object. Also, this shows me no string whatsoever:
return StatusCode(204, "WHY CAN'T I SEE THIS STRING?!");
Help!
204 (No Content) returns an HttpResponseMessage with, well, No Content.
If you do have content to return that the user can consume (the two reasons), then you don't have a No Content scenario.
Perhaps an Ok (200) with a custom ContentNotProvided class that holds a message, which can be consumed by the client, will do the trick?
Alternatively, return a BadRequest (400) with a message.

Making the same call twice in a row doesn't update the response

What I'm trying to do is hit a service twice in a row with a post of the same data. The idea here is to confirm that I can't have duplicate data, so I should get a 422 and a specific message in the response.
The test looked something like this:
When javaClient.createFoo(parameters)
And javaClient.createFoo(parameters)
Then status 422
And match $.message == "This is a duplicate."
It doesn't have a problem matching the status of the second call, but when I try to match the message (or any part of the response) it compares to the response from the first call.
I've tried making the second call a When, tried checking the status of the first call in between, but there must be something else that I'm missing.
Any ideas? Is there something special I need to do to clear the response?
Unless the method step is called, Karate does not do anything. The use of javaClient.createFoo() (whatever that is) suggests that there are some fundamental problems in your use and understanding of Karate.
I suggest keep it simple and just repeat the method post with the same payload and it should work as you expect.
Also please read this: https://stackoverflow.com/a/54126724/143475

HTTP status code response when there is not matched data with DB

I am building an API about email auth code.
Its process is simple.
input random code (client browser)
request with input code. (client browser)
receive the request (server)
scan the code from DB (server)
there is no code matched (server)
return a response with status code.
There are many status code, (2xx, 4xx, 5xx);
but I don't know which status code number is the most proper for this case.
It depends on the semantics you want to give your request. E.g.:
The API should search for items matching the query and return a list of results, like GET /codes?q=4ba69g. Think a "search page". In this case, an empty result [] is a perfectly valid result; nothing was wrong with the query, it just didn't return any matches. That's a 200 OK, or maybe a 204 No Content if you want to omit the empty response body entirely.
The code is treated like a resource, e.g. GET /codes/4ba69g. In this case a missing code would result in a 404 Not Found.
It's an action you want to perform which may fail, e.g. POST /login. If the user supplied the wrong credentials/code and hence the action cannot complete, that's a client-side error and therefore a 400 Bad Request.

How to set http response code in Parse Server cloud function?

A parse server cloud function is defined via
Parse.Cloud.define("hello", function(request, response) {..});
on the response, I can call response.success(X) and response.error(Y), and that sets the http response code and the body of the response.
But how do I define a different code, like created (201)?
And how do I set the headers of the response?
thanks, Tim
You are allowed to return any valid JSON from response.success(). Therefore, you could create an object with fields such as code, message, and value, so you can set the code, give it a string descriptor, and pass back the value you normally would, if there is one. This seems to accomplish what you need, though you will have to keep track of those codes across your platforms. I recommend looking up standard http response codes and make sure you don't overlap with any standards.

getContentServiceInfo1 returns truncated data regardless of reqSpecifier

Calling the getContentServiceInfo1 REST API seems to return the same data regardless of provided reqSpecifier and notrim values. I am using this call as part of the refresh account flow to determine whether the service requires MFA. Is there another call that I can make to achieve the same thing?
I noticed that the values accepted for reqSpecifier are essentially binary bit masks but providing the flag as 10000000, or even 0b10000000 for that matter, instead of 128 does not change the results. What value and value type should I be providing for reqSpecifier to get more than the basic level information? Relevant request and return information when using this API call for E*Trade is below.
Request (cobrand token is omitted):
{"cobSessionToken":"",
"contentServiceId":24,
"reqSpecifier":128,
"notrim":true}
Response (this is always the response I get unless I provide arguments that result in an error):
{"contentServiceId":24,
"siteId":744,
"containerInfo":{"containerName":"stocks","assetType":1}}
This response comes when you pass "notrim" value as "false", if you are passing it "false" then please pass it as true. It should work.