I am trying to HTTP POST an item to an API using Scrapy. In my pipeline I have:
Request( url, method='POST',
body=json.dumps(item),
headers={'Content-Type':'application/json'} )
This does not work. The error is:
{ some JSON } is not JSON serializable
Any idea on what I am doing wrong?
As stated in paul trmbrth's comment, instead of
body=json.dumps(item)
use
body=json.dumps(dict(item))
So your code would become:
Request( url, method='POST',
body=json.dumps(dict(item)),
headers={'Content-Type':'application/json'} )
I have used JsonRequest from from scrapy.http as follows:
JsonRequest(
url=<url>,
method='POST',
# Replace with your item or dict(item)
data=item
)
Behind it calls json.dumps on the dictionary you pass to the data parameter so you first might want to make sure that your item that you're trying to send is json serializable.
Related
I am trying to create Karate feature file for API Testing.
It is a post request and we have to pass random number for 1 field i.e. order in request payload every time.
Now, I am trying to use this function in the same feature file to pass that random generated value in json payload before sending post request.
Could someone please have a look at my feature file and help.
Thanks
Also, Is there a way if i want to pass this same random value for a json payload created as a separate request.json payload file
Your requestPayload is within double quotes, so it became a string.
Here's an example that should get you going. Just paste it into a new Scenario and run it and see what happens.
* def temp1 = 'bar'
* url 'https://httpbin.org/anything'
* def payload = { foo: '#(temp1)' }
* request payload
* method post
And please read the documentation and examples, it will save a you a lot of time.
I’m having issue with xml post request where post method is not executed. When I try to post same request body in post man it worked.My test is success with 200 but actual request is not executed.
Please let me know if I’m missing
To pass the request body,I’m calling through java object and payload is correctly constructed and printed.In execution test is success and doesn’t print response.But actually test is not executed.
Only headers are printed.
***************** create-user.feature*****************
Feature: create ims user for provided country
Requires country code,
Background:
# load secrets from json
* def createuser = Java.type('com.user.JavaTestData')
* def create = createuser.createUser("US")
Scenario: get service token
Given url imscreateuserurl
And request create
When method post
Then status 200
* print response
***************** create-user.feature*****************
Here is java class
public class JavaTestData {
private static final Logger logger = LoggerFactory.getLogger(JavaTestData.class);
public static String createUser(String countryCodeInput) {
logger.debug("create user for country code input", countryCodeInput);
Unless you post a full working example, no one can help you. Pretty clear that the value of create is null or empty.
Also I personally think you are wasting your time using Java. The whole point of Karate is to avoid using Java as far as possible.
Look at these examples for ideas: https://github.com/intuit/karate/blob/master/karate-junit4/src/test/java/com/intuit/karate/junit4/xml/xml.feature
Edit: also refer to the doc on type-conversion: https://github.com/intuit/karate#type-conversion
#Peter, here is my feature file
Feature: create ims user for provided country
Requires country code,
Background:
# load secrets from json
* def createuser = Java.type('com.adobe.imscreateuser.JavaTestData')
* def create = createuser.createUser("US")
Scenario: get service token
Given url imscreateuserurl
And header Content-Type = 'application/xml; charset=utf-8'
And request create
When method post
Then status 200
* print response
I have performed print for create and showing complete payload.At when method post -> statement its going as null or empty...
Not sure where it is missing
I have a question regarding the use of the getBandwidthDataByDate request using the SoftLayer REST API.
In the documentation it lists 3 parameters for this request, but it's a GET request. Does anyone know how to make this request and/or have an example?
https://api.softlayer.com/rest/v3/SoftLayer_Virtual_Guest/getBandwidthDataByDate/'device_id'.json
I'm not sure where to add the parameters here.(startDateTime, endDateTime, networkType)
And what does the dateTime object look like?
Thanks
This is a POST request, so you need to pass the parameters in "Payload" (I'm using Advanced REST client for Chrome).
Try the following REST request:
https://$user:$apiKey#api.softlayer.com/rest/v3/SoftLayer_Virtual_Guest/$device_id/getBandwidthDataByDate
Method: Post (Copy the below code in "Payload")
{
"parameters":[
"2016-03-10T00:00:00",
"2016-03-15T00:00:00",
"public"
]
}
Note: Replace $user, $apiKey and $device_id with your own information
References:
SoftLayer_Virtual_Guest::getBandwidthDataByDate
I am using scrapy to scrape some data, I am wondering how much data of Request the Response stores.
My specific question is shown in the following code:
def parse(self,response):
r = FormRequest(url=url1, formdata={somedata}, callback=parse2)
#is this line necessary if I want the formdata being attached?
r.meta['formdata'] = formdata
yield r
def parse2(self,response):
#can I access to the formdata here without that line of code?
Any suggestion would be appreciated.
Yes, however formdata is already converted into body and won't be the dictionary but a string as far as I remember.
so try r.body also quick reminder you can check the attributes of an object by:
dir(r)
How do I Invoke an HTTP request with a particular URL and process the body of the response as XML?
Information Provided by ShopStyle:
HOW TO USE THE API:
Choose the method that returns the data your application needs. For example, the /products method is used to get products that match a given category or brand.Construct a URL for that method with the appropriate host, method name, and query parameters. Invoke the URL as an HTTP GET.
When the HTTP response arrives, extract the required data elements from the response's body.The rest of this document describes the details of constructing the right URL for each of the API methods. The XML format of the responses may be seen by clicking on the sample URLs shown for each method. The responses in JSON format contain identical information, just in a different format.
SHOPSTYLE API URLS
All ShopStyle API URLs have the following form:
http://api.shopstyle.com/api/VERSION/METHOD_NAMEpid=YOUR_API_KEY&format=FORMAT&...
The METHOD_NAME is taken from the list of methods in the various API shown at left (Press Link To View List of Methods-https://www.shopstylecollective.com/api/overview).
COMMON API PARAMETERS
All methods in the API accept these parameters:
API_KEY (my unique key is ******************)
pid Unique API_KEY string that is assigned to the caller. You can find your API Key on the Account Settings page.
FORMAT
The format of the response. Supported values are:
json - The response is in JSON format with UTF-8 encoding. This is the default if the parameter is absent.
xml - The response is in XML format with UTF-8 encoding.
jsonp - The response is in JSON format with UTF-8 encoding wrapped in a JavaScript method called padding. The padding must be specified with the query parameter 'callback'. Only single expressions (function reference, or object property function reference) are accepted as valid padding.
When set to 1 or 'true' the HTTP status will always be 200. Use the field "errorCode" in the response to detect whether the API Call was successful. By default, when an error occur the HTTP Status of the response will be different than 200
Again I am a beginner, so please provide detailed information on how to retrieve CATEGORY data (Examples: Dresses, Tops, Buttoms, etc) in XML format.**
Thank you!!!
Here's a simple example to get your started. Copy the code below and put it into a file, say "cat.php". Then run it by typing "php cat.php" at a command prompt (assumes you have php on your machine):
<?php
// don't show dom parse errors
libxml_use_internal_errors(true);
// grab the xml from the api
$response = file_get_contents("http://api.shopstyle.com/api/v2/categories?pid=TEST&format=xml");
$doc = new DOMDocument();
$doc->loadHTML($response);
// grab all the categories
$elements = $doc->getElementsByTagName('categories');
foreach($elements as $node){
foreach($node->childNodes as $child) {
// get the name out of the category
$nodes = $child->getElementsByTagName("name");
foreach ($nodes as $name) {
echo $name->nodeValue . PHP_EOL;
}
}
}