pymongo: Document must be an instance of dict - pymongo

Afternoon,
I'm facing a problem with pymongo, I'm not able to set up correctly the parameter to insert into MongoDB via insert_many(). I came across with de following error:
TypeError: document must be an instance of dict, bson.son.SON,
bson.raw_bson.RawBSONDocument, or a type that inherits from
collections.MutableMapping [while running 'Insere no MongoDB']
What am I doing wrong ?
class InsertMongoDB(beam.DoFn):
def process(self, element):
arqJson=json.loads(element)
client = MongoClient("mongodb://user:password#mkp-cr-marketplace-core.lcr88.gcp.mongodb.net/db-poc-base360?retryWrites=true&w=majority%20")
db = client['db-poc-base360']
db.tbPropostaSucesso.insert_many(arqJson)
# tbPropostaErro = db['tbPropostaErro']
# tbPropostaErro
resultado = 0
yield resultado
I receive a messagem from Google PubSub and forward it to a method called InsertMongoDB().
I don't know how to suit my massage, whose value is in a json format, to use it correctly in the insert_many().
When I debug my variable "arqJson" has :
The json that I'm using is :
{
"Status": "Sucesso ",
"Documento": {
"Apolice": [{
"ItemAuto": [{
"nmTipo": "FOX",
"nrItem": "000001",
"nmMarca": "VOLKSWAGEN",
"aaModelo": "2017",
"cdModelo": "0017664",
"nmModelo": "TRENDLINE 1.0 FLEX 12V 5P",
"aaFabricacao": "2016",
"nmTipoVeiculo": "Hatch"
}, {
"nmTipo": "FOX",
"nrItem": "000001",
"nmMarca": "VOLKSWAGEN",
"aaModelo": "2017",
"cdModelo": "0017664",
"nmModelo": "TRENDLINE 1.0 FLEX 12V 5P",
"aaFabricacao": "2016",
"nmTipoVeiculo": "Hatch"
}],
"ItemProp": [{
"dsUF": "MG",
"idLocal": "000001",
"dsCidade": "BELO HORIZONTE",
"dsEndereco": "RUA RUA RUA",
"dsComplemento": "CASA"
}],
"cdEmpresa": "1",
"idApolice": "501741",
"idEndosso": "000000",
"cdCarteira": "431",
"cdSucursal": "010",
"cdPatrimonio": "1",
"nrItemContrato": "2",
"dsTipoDocumento": "A",
"cdVeiculoSegurado": "1"
}],
"Cliente": [{
"cdCliente": "1",
"nmCliente": "Lucas",
"nrCpfCnpj": "4355582833",
"icRegistroAtivo": "1",
"cdAcaoInformacao": "A",
"dtAcaoInformacao": "2020-02-02",
"cdServicoAcaoInformacao": "cdServicoAcao",
"cdUsuarioAcaoInformacao": "cdUsuarioAcao"
}, {
"cdCliente": "2",
"nmCliente": "Lucas",
"nrCpfCnpj": "43331971",
"icRegistroAtivo": "1",
"cdAcaoInformacao": "A",
"dtAcaoInformacao": "2020-02-01",
"cdServicoAcaoInformacao": "cdServicoAcao2",
"cdUsuarioAcaoInformacao": "cdUsuarioAcao2"
}],
"Mensagem": [{
"cdMensagem": "1",
"dsMensagem": "Teste de mensagem"
}],
"EnderecoCobranca": [{
"dsUF": "RS",
"dsBairro": "INTEGRAÇÃO",
"dsCidade": "PAROBE",
"cdEndereco": 1,
"dsEndereco": "RUA RUA RUA",
"nrEndereco": "280",
"dsComplemento": "",
"icRegistroAtivo": "1",
"cdAcaoInformacao": "A",
"dtAcaoInformacao": "2020-02-02",
"cdServicoAcaoInformacao": "cdServicoAcao",
"cdUsuarioAcaoInformacao": "cdUsuarioAcao"
}, {
"dsUF": "SP",
"dsBairro": "INTEGRAÇÃO2",
"dsCidade": "POC2",
"cdEndereco": 2,
"dsEndereco": "RUA B",
"nrEndereco": "222",
"dsComplemento": "CASA 2",
"icRegistroAtivo": "1",
"cdAcaoInformacao": "A",
"dtAcaoInformacao": "2020-02-01",
"cdServicoAcaoInformacao": "cdServicoAcao2",
"cdUsuarioAcaoInformacao": "cdUsuarioAcao2"
}]
}
}
2020/11/20:
At the moment I'm struggling with the format of arqJson that I need to use in the insert_one(arqJson).
I forgot to mention that my method InsertMongoDB receives the arqJson from another method called InsertPostgreSQL.
InsertPostgreSQL does:
Receive the message from Pubsub;
-Transform the element : json.dumps(json.loads(element))
Save it into arqJson. After that, InsertMongoDB is called.
At this moment, I don't know how to format "element" (whose type is list) and save it into arqJson, because I have this error:
raise TypeError("%s must be an instance of dict, bson.son.SON, "
TypeError: document must be an instance of dict, bson.son.SON,
bson.raw_bson.RawBSONDocument, or a type that inherits from
collections.MutableMapping [while running 'Insere no MongoDB']
Thank you,
Juliano

The solution is :
The first error is because your JSON contains a single document not
multiple docs for an insert many. If you use brackets like this
db.tbPropostaSucesso.insert_many([arqJson]) and convert it to a list
with a single element it will work. Or you can try
insert_one(arqJson). – DaveStSomeWhere 5 hours ago
Thank you DaveStSomeWhere

I had the same problem, and what worked for me was adding retryWrites=false to the connection URL:
mongodb+srv://user:pass#server/etc...etc?retryWrites=false

Related

Why one entity extratcted different values in different forms in one intent

I have my Dialogflow es agent working weid now.
Previously, I was training one intent understanding how many bedrooms the enduser mentioned, so I have an entity relating to that called $sc_bed It should be a list as a checked the "is list" in action and parameters
action and parametes I have for this Intent
Also I have 3 output contexts with this intent and store the key parameters mentioned
contexts
For my understanding, the sc_bed entity should be stored in all 3 contexts I mentioned above, with same value and format(which should be list rather than string)
However, the log shows that the entity in different contexts are different in both format and value, I will shoe the log here.
"name": "projects/xxxx/locations/global/agent/sessions/f4xxxxxxxx/contexts/search_criteria",
"lifespanCount": 49,
"parameters": {
"is_visitor": "yes",
"lead_first_name": "Test Lead",
"sc_location": [],
"clarify_renter.original": "",
"lead_email.original": "",
"buyer_reply_message2": "I’d like to get a little more information from you if that's ok.",
"sc_timeframe.original": [],
"buyer_interested_listing.original": "",
"conversation.original": "",
"lead_type": "renter",
"buyer_reply_message": "I've got your preferences down here and I'll set up a search for places you might like.",
"seller_market_snapshot.original": "",
"lead_first_name.original": "",
"assigned_agent_name.original": "",
**"sc_bed": {
"amount": 3
},**
"assigned_agent_name": "Test Assigned Agent",
"reply_message_timeframe": "We would love to assist you with the renting process!",
"sc_maxPrice.original": [],
"reminder_price_reduced_link": "chime.me/pricereduced",
"sc_timeframe_range": [],
"search_listing_result.original": "",
"sc_propertyType": [],
"sc_priceRange": [],
"sc_maxPrice": [],
}
},
{
"name": "projects/newagent-mxiwbg/locations/global/agent/sessions/f47970c6-49b2-c8a9-7aff-60316c185007/contexts/general_info",
"lifespanCount": 49,
"parameters": {
"chime_action_add_tag": "AI: Interested",
"buyer_reply_message2.original": "",
"follow_up_event.original": "",
"conversation": "renter_home_search",
"conversation.original": "",
"sc_feature.original": [],
"sc_location": [],
"lead_type": "renter",
"reply_message_timeframe.original": "",
"sc_priceRange": [],
"buyer_reply_message.original": "",
"sc_approxPrice": "",
"sc_propertyType": [],
"sc_propertyType.original": [],
"clarify_renter.original": "",
"sc_timeframe_range": [],
"clarify_renter": "Yes",
"sc_maxPrice.original": [],
**"sc_bed": [
{
"amount.original": [
"2",
"3"
],
"amount": [
2,
3
]
}
],
"sc_bed.original": [
"2 or 3 bedroom"
],**
}
},
What I am expecting are 2 points:
The entity in different contexts should be the same one at least for the entity extracted in this exactly same one intent
The entity info should keep the format I checked which is a list not a string.

Array of objects to a single object containing all said objects Dataweave

I am trying to transform the below Array of Objects input:
[
{
"Id": "3",
"Code": "4190484",
"Expense": "Huge Expense "
},
{
"Id": "4",
"Code": "271",
"Expense": "Big Expense"
},
{
"Id": "3",
"Code": "433",
"Expense": "No Expense"
}
]
to this Output of a single object:
{
"Id": "3",
"Code": "4190484",
"Expense": "Huge Expense ",
"Id": "4",
"Code": "271",
"Expense": "Big Expense",
"Id": "3",
"Code": "433",
"Expense": "No Expense"
}
How would you accomplish this in Dataweave?
You can also use the dynamic elements feature of the language:
%dw 2.0
output application/json
---
{(payload)}
Like #aled explained in his answer, you should not be using duplicate keys in JSON.
You can use the reduce() function but be warned that using duplicate keys in JSON is implementation dependent. I think it is a bad design to use duplicate keys in JSON. It might lead to unexpected behaviors. Some implementations might ignore the duplicates. For example DataWeave will return only one Id of the resulting object with payload.Id.
If even after what I mentioned you still want to go ahead it this is an example:
%dw 2.0
output application/json
---
// I don't recommend to use duplicate keys
payload reduce ((item, acc = {}) -> acc ++ item)

How to validate Nested JSON Response

I am facing issue while validate Nested JSON response in API Testing using Karate Framework.
JSON Response:
Feed[
{ "item_type": "Cake" ,
"title": "Birthday Cake",
"Services":
[
{
"id": "1",
"name": {
"first_name": "Rahul",
"last_name": "Goyal"
}
},
{
"id": "2",
"name":{
"first_name": "Hitendra",
"last_name": "garg"
}
}
]
},
{
"item_type":"Cycle",
"title": "used by"
},
{
"item_type": "College"
"dept":
[
{"branch": "EC"},
{"branch": "CSE"},
{"branch": "CIVIL"}
]
},
]
}
Now i need to validate response based on Item type. as we can see nested JSON is different for different item_type.
I have tried with below solution
Schema Design for Item_type value cake
def Feed_Cake_Service_name={first_name: '#string',last_name: '#string'}
def Feed_Cake_Services= {id: '#string',name:#(Feed_Cake_Service_name)}
def Feed_Cake={item_type:'#string',title: '#string',Services: '#[] Feed_Cake_Services'}
def Feed_Cake_Response= {Feed: '#[] Feed_Cake'}
Schema Design for item_type Cycle
def Feed_Cycle={item_type:'#string',title:'#string'}
Schema Design for item type College
def Feed_College_Dept_Branch={branch:'#string'}
def Feed_College={item_type:'#string',dept: '[] Feed_College_Dept_Branch'}
now if i want to verify only item type Cake then i have written match like below
match response contains Feed_Cake_Response
but here my test case is getting failed. because it is comparing for all item type.
so here i have two question
1.) How we can compare particular item type schema
2.) How we can include all item type in one match equation since any item type can come in JSON response , and i want to validate all
Thanks
I'll just give you one hint. For the rest, read the documentation please:
* def item = { item_type: '#string', title: '##string', dept: '##[]', Services: '##[]' }
* match each response == item

how to query embedded document using mongodb

Need help constructing this mongo query.
So far I can query on the first level, but unable to do so at the next embedded level ("labels" > 2")
For example, the document structure looks like this:
> db.versions_20170420.findOne();
{
"_id" : ObjectId("54bf146b77ac503bbf0f0130"),
"account" : "foo",
"labels" : {
"1" : {
"name" : "one",
"color" : "color1"
},
"2" : {
"name" : "two",
"color" : "color2"
},
"3" : {
"name" : "three",
"color" : "color3"
}
},
"profile" : "bar",
"version" : NumberLong("201412192106")
This query I can filter at the first level (account, profile).
db.profile_versions_20170420.find({"account":"foo", "profile": "bar"}).pretty()
However, given this structure, I'm looking for documents where the "label" > "2". It doesn't look like "2" is a number, but a string. Is there a way to construct the mongo query to do that? Do I need to do some conversion?
If I correctly understand you and your data structure, "label" > "2" means that object labels must contain property labels.3, and it is easy to check with next code:
db.profile_versions_20170420.find(
{"account": "foo", "profile": "bar", "labels.3": {$exists: true}}
).pretty();
But it doesn't mean that your object contains at least 3 properties, because it is not $size function which calculates count of elements in array, and we cannot use $size because labels is object not array. Hence in our case, we only know that labels have property 3 even it is the only one property which labels contains.
You can improve find criteria:
db.profile_versions_20170420.find({
"account": "foo",
"profile": "bar",
"labels.1": {$exists: true},
"labels.2": {$exists: true},
"labels.3": {$exists: true}
}).pretty();
and ensure that labes contains elements 1, 2, 3, but in this case, you have to care about object structure on application level during insert/update/delete data in document.
As another option, you can update your db and add extra field labelsConut and after that you will be able to run query like this:
db.profile_versions_20170420.find(
{"account": "foo", "profile": "bar", "labelsConut": {$gt: 2}}
).pretty();
btw, it will work faster...

Mule:Dataweave Iteration not working

I am trying to take output from Salesforce & transform it to a json. here is my code:
%dw 1.0
%output application/json
payload map {
headerandlines:{ id : $.Id,
agreementLineID : $.LineItems__r.Id,
netPrice : $.LineItems__r.Price__c,
volume : $.Volume__c,
name : $.Name,
StartDate : $.Start_Date__c,
EndDate : $.End_Date__c,
poField : $.PO_Field__c,
ConsoleNumber : $.Console_Number__c,
Term : $.Term__c,
ownerID : $.OwnerId,
Unit : $.Unit__c,
siteNumber : $.Site_Num__c,
customerNumber : $.Customer_Num__c
}
}
input payload looks like this.. it is a collection of objects. Somehow after the transformation only the first object is sent & rest is clobbered.
[
{
"id": "DA0YAAW",
"LineID": [
"jGEAU",
"jBEAU",
"j6EAE"
],
"Price": [
"50000.0",
"12000.0",
"45000.0"
],
"netPrice": null,
"volume": null,
"name": " Test 2.24",
"StartDate": "2017-02-17",
"EndDate": "2018-02-17",
"poField": "123456",
"ConsoleNumber": "8888888",
"PaymentTerm": "thirty (30)",
"ownerID": “abcd”,
"OperatingUnit": " International Company",
"siteNumber": null,
"customerNumber": null
},
{
"id": "a37n0000000DAMAAA4",
"LineID": [
"JunEAE",
"JuiEAE",
"KdMEAU",
"JuYEAU"
],
"Price": [
"5000.0",
"8000.0",
"5000.0",
"5000.0"
],
"netPrice": null,
"volume": null,
"name": " Test 3.6",
"StartDate": "2017-03-06",
"EndDate": "2018-03-16",
"poField": "12345",
"ConsoleNumber": "123456-",
"PaymentTerm": "30 NET",
"ownerID": “dfgh”,
"OperatingUnit": ", inc.",
"siteNumber": null,
"customerNumber": null
},
….
]
When I call this code from the browser (using API testing) I get the complete payload with multiple objects. When I call this from another API I get only one 1 object indicating it is not looping through. I can confirm that the payload has multiple objects . Is there anything I am missing in terms of looping through this code to extract multiple objects? I assume that '$' notation is good enough for iteration.
#insaneyogi, your input is either incorrect or your dataweave is incorrect.
Here in the input you have specified id in the small. but in dataweave, it is mentioned in capital.
I think the problem here is with your Lineitem and Price type elements. They are collection within and element. In your data mapping $. will take care of the outer object. However, i think the mapping like LineItems__r.Price__c is not correct. It should have proper index , probably LineItems__r.Price__c[0]. Please try that and it should work. First change the input with single element for price or line-item and test.
It looks like the agreementLineID and netPrice are arrays and you need to loop through them with a map operator within the bigger outer map to get all the line items. That should work.