I'm using JsonLinesItemExporter to export some data and instead of
{"name": "Color TV", "price": "1200"}
{"name": "DVD player", "price": "200"}
scrapy is writing the following to file:
{"name": ["Color TV"], "price": ["1200"]}
{"name": ["DVD player"], "price": ["200"]}
(From debug) it seems I'm passing a correct value (not a list) and that both item.add_value and item.replace_value are replacing my strings by a single string element list.
Is this configurable?
If not, how to get a different behaviour? Extend JsonLinesItemExporter or is there a better approach?
Are you sure that you're using properly configured ItemLoader?
I'de recommend to use TakeFirst (documentation here: https://docs.scrapy.org/en/latest/topics/loaders.html)
Example of usage:
class YourItemLoader(ItemLoader):
default_output_processor = TakeFirst()
Related
I have a use case where I want to assert on a API response and compare it with the csv data.
Step1:
Csv file: *test.csv*
id,date,fullname,cost,country,code
1,02-03-2002,user1,$200,Canada,CAN
2, 04-05-2016,user2,$1500,United States, USA
I read the csv file and store it in a variable
def var1 = read(test.csv)
So now, var1 is a list of jsons based on my csv
var1 = [
{
"id":1,
"date":"02-03-2002",
"fullname": "user1",
"cost": "$200",
"country": "Canada",
"code": "CAN"
},
{
"id":2,
"date":"04-05-2016",
"fullname": "user2",
"cost": "$1500",
"country": "United States",
"code": "USA"
}
]
Step2:
I hit my api and get a response
Given url "https://dummyurl.com
Given path "/userdetails"
When method get
Then status 200
* def apiResponse = response
Step 3:
My api returns a list response which is:
{
"id":1,
"date":"02-03-2002",
"fullname": "user1",
"cost": "$200",
"country": {
"name": "Canada",
"code": "CAN"
}
},
{
"id":2,
"date":"05-04-2012",
"fullname": "user2",
"cost": "$1500",
"country": {
"name": "United States",
"code": "USA"
}
},
...and more 100 records..
]
Step 4:
So there are two assertions now which I wanted to perform
Get the count of csvresponse and apiresponse and compare which I did using the .length operator
Secondly, I want to confirm if each csv records are matching with each api response.
And if possible in my case id key from csv and apiresponse is primary key, so if I can iterate on id and match the api response for any discrepancy.
Let me know if this is readable for you and if I was able to explain my use case.
Thanks for your earlier response.
Please read up on the match contains syntax, that's all you need: https://github.com/intuit/karate#match-contains
So this one line should be enough:
* match var1 contains response
Also look at this answer in case the new contains deep helps: https://stackoverflow.com/a/63103746/143475
Try to avoid iterating, it is not needed for most API tests. But you can certainly do it. Look at these answers:
https://stackoverflow.com/a/62567262/143475
Also read this - because I suspect you are trying to over-complicate your tests. Please don't. Write tests where your are 100% sure of the "shape" of the response as far as possible: https://stackoverflow.com/a/54126724/143475
And please please read the docs. It is worth it.
In jmeter, I want to pass dynamic parameters. For simple json its easy to put ${value1} but if json structure is complex like array or with multiple values then what is the proper method to pass parameter dynamically. Please refer below json.
Below is json with parameter :
{
"squadName": "Super hero squad",
"homeTown": "Metro City",
"formed": 2016,
"secretBase": "Super tower",
"active": true,
"members": [
{
"name": "Molecule Man",
"age": 29,
"secretIdentity": "Dan Jukes",
"powers": [
"Radiation resistance",
"Turning tiny",
"Radiation blast"
]
},
{
"name": "Madame Uppercut",
"age": 39,
"secretIdentity": "Jane Wilson",
"powers": [
"Million tonne punch",
"Damage resistance",
"Superhuman reflexes"
]
},
{
"name": "Eternal Flame",
"age": 1000000,
"secretIdentity": "Unknown",
"powers": [
"Immortality",
"Heat Immunity",
"Inferno",
"Teleportation",
"Interdimensional travel"
]
}
]
}
=======
Now I have used below method to send parameter through csv config file.
Is there any other simple method to pass parameter through variables in Jmeter for complex json (5-6 level with array data) ?
CSV DATA config is the best to parameterize your test data.
If you want to customize the way you want to pick values from CSV you can use BeanShell /JSR223 sampler
here is one article that shows how to pick random values from CSV data config.
I assigned a geojson object to the fit property of the projection as described in the documentation: https://vega.github.io/vega/docs/projections/
I am always getting the error "Unsupported parameter object: {"type": "FeatureCollection"..."
I am assigning the geojson object as following: (my source data is topojson format)
spec.projections[0].fit = topojson.feature(
mapData,
mapData.objects.topology
);
The documentation explicitly says that this parameter should be a GeoJSON Feature or FeatureCollection. How am I supposed to use the fit property?
You have to use a reference to a data object.
"data": [
{
"name": "counties",
"url": "data/us-10m.json",
"format": {"type": "topojson", "feature": "counties" }
}
],
"projections": [
{
"name": "projection",
"type": "mercator",
"fit": {"signal": "data('counties')"},
"size": {"signal": "[width, height]"}
}
]
Here is an example from VEGA that you can test in their edit.
https://github.com/vega/vega/blob/master/test/specs-valid/map-fit.vg.json
Also notice the "size" property. The fit does not get reflected without it.
According to the docs, the "option set" field is read-only:
https://developer.bigcommerce.com/api/stores/v2/products#update-product
Does this mean you are not able to apply an Option Set to a Product via the API?
Does anyone know how to apply an Option Set to a Product?
you're looking for a sub resource of products, you can set the ID by posting the ID to the option_sets sub resource https://developer.bigcommerce.com/api/objects/v2/option_set
"option_set" is read-only, "option_set_id" is writable. I.E. you must retrieve the id of the desired option set and set it.
PUT /products/11
{
"name": "Plain T-Shirt",
"type": "physical",
"description": "This timeless fashion staple will never go out of style!",
"price": "29.99",
"categories": [18],
"availability": "available",
"weight": "0.5",
**"option_set_id":25**
}
my problem is: I have a JsonObject like this:
{
"success": true,
"type": "message",
"body": {
"_id": "5215bdd32de81e0c0f000005",
"id": "411c79eb-a725-4ad9-9d82-2db54dfc80ee",
"type": "metaModel",
"title": "testchang",
"authorId": "5215bd552de81e0c0f000001",
"drawElems": [
{
"type": "App.draw.metaElem.ModelStartPhase",
"id": "27re7e35-550j",
"x": 60,
"y": 50,
"width": 50,
"height": 50,
"title": "problem engagement",
"isGhost": true,
"pointTo": "e88e2845-37a4-4c45-a030-d02a3c3e03f9",
"bindingId": "90f79d70-0afc-11e3-98d2-83967d2ad9a6",
"model": "meta",
"entityType": "phase",
"domainId": "411c79eb-a725-4ad9-9d82-2db54dfc80ee",
"authorId": "5215bd552de81e0c0f000001",
"userData": {},
"_id": "5215f4c5d89f629c1700000d"
},
{...}
]
}
}
And I tried to define a mapping as follows to index only parts of this object.
String mapping = XContentFactory.jsonBuilder()
.startObject()
.startObject("domaindata").field("dynamic","false")
.startObject("properties")
.startObject("id").field("type","string").field("store","yes").endObject()
.startObject("type").field("type","string").field("store","yes").endObject()
.startObject("title").field("type","integer").field("store","yes").endObject()
.startObject("drawElems")
.startObject("properties")
.startObject("type").field("store","yes").field("type","string").endObject()
.startObject("title").field("store","yes").field("type","string").endObject()
.endObject().endObject().endObject().endObject().endObject().string();
after adding this mapping into my type with:
node.client().admin()
.indices().prepareCreate("test")
.addMapping("domaindata", mapping)
.execute().actionGet();
I still got all of the jsonobject in my indexresponse, it seems that my mapping does not work.
Could anybody help me? Thanks a lot!
The problem here is that using static mapping only means that fields that are not already present in the mapping won't be added to it, thus won't be indexed either. But as they are part of the source document that you sent, they are returned as part of the _source field.
Same goes if you disable a specific object in the mapping ("enable":false) as mentioned here. That object won't be parsed nor indexed, but will still be part of the stored _source field.
If you want to avoid storing part of the _source you can use the source includes/excludes feature as described here.