I have a MEAN-Stack setup in which i have Devices and Servicecases saved in the MongoDB-Database.
Devices can be the content of a Servicecase
If a new Case should be created, my Frontend will deliver the following form data:
content: [
{
"device": 012345678909876,
"errorDesc": "lorem"
},
{
"device": 012345678909876,
"errorDesc": "ipsum"
}
]
There could be a device document with the submitted device number in the Database. If yes, the received doc should be populated with its ObjectId to look like this:
content: [
{
device: { type: Schema.Types.ObjectId, ref: 'Device' },
errorDesc: String
},
...
]
If not, it should stay as it is
I could iterate through each device of the array and use the findOne() query and, if a doc was found, replace it, but is there a more efficient way to use the populate() transformation?
Related
I`m new to nifi and I want to convert big amount of json data to csv format.
This is what I am doing at the moment but it is not the expected result.
These are the steps:
processes to create access_token and send request body using InvokeHTTP(This part works fine I wont name the processes since this is the expected result) and getting the response body in json.
Example of json response:
[
{
"results":[
{
"customer":{
"resourceName":"customers/123456789",
"id":"11111111"
},
"campaign":{
"resourceName":"customers/123456789/campaigns/222456422222",
"name":"asdaasdasdad",
"id":"456456546546"
},
"adGroup":{
"resourceName":"customers/456456456456/adGroups/456456456456",
"id":"456456456546",
"name":"asdasdasdasda"
},
"metrics":{
"clicks":"11",
"costMicros":"43068982",
"impressions":"2079"
},
"segments":{
"device":"DESKTOP",
"date":"2021-11-22"
},
"incomeRangeView":{
"resourceName":"customers/456456456/incomeRangeViews/456456546~456456456"
}
},
{
"customer":{
"resourceName":"customers/123456789",
"id":"11111111"
},
"campaign":{
"resourceName":"customers/123456789/campaigns/222456422222",
"name":"asdasdasdasd",
"id":"456456546546"
},
"adGroup":{
"resourceName":"customers/456456456456/adGroups/456456456456",
"id":"456456456546",
"name":"asdasdasdas"
},
"metrics":{
"clicks":"11",
"costMicros":"43068982",
"impressions":"2079"
},
"segments":{
"device":"DESKTOP",
"date":"2021-11-22"
},
"incomeRangeView":{
"resourceName":"customers/456456456/incomeRangeViews/456456546~456456456"
}
},
....etc....
]
}
]
Now I am using:
===>SplitJson ($[].results[])==>JoltTransformJSON with this spec:
[{
"operation": "shift",
"spec": {
"customer": {
"id": "customer_id"
},
"campaign": {
"id": "campaign_id",
"name": "campaign_name"
},
"adGroup": {
"id": "ad_group_id",
"name": "ad_group_name"
},
"metrics": {
"clicks": "clicks",
"costMicros": "cost",
"impressions": "impressions"
},
"segments": {
"device": "device",
"date": "date"
},
"incomeRangeView": {
"resourceName": "keywords_id"
}
}
}]
==>> MergeContent( here is the problem which I don`t know how to fix)
Merge Strategy: Defragment
Merge Format: Binary Concatnation
Attribute Strategy Keep Only Common Attributes
Maximum number of Bins 5 (I tried 10 same result)
Delimiter Strategy: Text
Header: [
Footer: ]
Demarcator: ,
What is the result I get?
I get a json file that has parts of the json data
Example: I have 50k customer_ids in 1 json file so I would like to send this data into big query table and have all the ids under the same field "customer_id".
The MergeContent uses the split json files and combines them but I will still get 10k customer_ids for each file i.e. I have 5 files and not 1 file with 50k customer_ids
After the MergeContent I use ==>>ConvertRecord with these settings:
Record Reader JsonTreeReader (Schema Access Strategy: InferSchema)
Record Writer CsvRecordWriter (
Schema Write Strategy: Do Not Write Schema
Schema Access Strategy: Inherit Record Schema
CSV Format: Microsoft Excel
Include Header Line: true
Character Set UTF-8
)
==>>UpdateAttribute (custom prop: filename: ${filename}.csv) ==>> PutGCSObject(and put the data into the google bucket (this step works fine- I am able to put files there))
With this approach I am UNABLE to send data to big query(After MergeContent I tried using PutBigQueryBatch and used this command in bq sheel to get the schema I need:
bq show --format=prettyjson some_data_set.some_table_in_that_data_set | jq '.schema.fields'
I filled all the fields as needed and Load file type: I tried NEWLINE_DELIMITED_JSON or CSV if I converted it to CSV (I am not getting errors but no data is uploaded into the table)
)
What am I doing wrong? I basically want to map the data in such a way that each fields data will be under the same field name
The trick you are missing is using Records.
Instead of using X>SplitJson>JoltTransformJson>Merge>Convert>X, try just X>JoltTransformRecord>X with a JSON Reader and a CSV Writer. This skips a lot of inefficiency.
If you really need to split (and you should avoid splitting and merging unless totally necessary), you can use MergeRecord instead - again with a JSON Reader and CSV Writer. This would make your flow X>Split>Jolt>MergeRecord>X.
I'm adding a list called 'tourlocation' to my Keystone 5 project. In my mongo database my tourlocations collection has an object called 'coordinates', with two values: 'lat' and 'long'. Example:
"coordinates" : {
"lat" : 53.343761,
"long" : -6.24953
},
In the previous version of keystone, I could define my tourlocation list coordinates object like this:
coordinates: {
lat: {
type: Number,
noedit: true
},
long: {
type: Number,
noedit: true
}
Now unfortunately, when I try to define the list this way it gives the error: The 'tourlocation.coordinates' field doesn't specify a valid type. (tourlocation.coordinates.type is undefined)'
Is there any way to represent objects in keystone 5?
#Alex Hughes I believe your error says "type" which you may need to add it like this
keystone.createList('User', {
fields: {
name: { type: Text }, // Look at the type "Text" even in the MongoDB you can choose the type but it will be better to choose it here from the beginning.
email: { type: Text },
},
});
Note that noedit: true is not supported in version 5 of KeystoneJS.
For more info look at this page https://www.keystonejs.com/blog/field-types#core-field-types
I have followed the doc for the ReferenceArrayInput (https://marmelab.com/react-admin/Inputs.html#common-input-props) but it does not seem to be working with relationship fields.
For example, I have this many-to-many relation for my Users (serialized version) :
Coming from (raw response from my API):
I have setup the ReferenceArrayInput as followed :
<ReferenceArrayInput source="profiles" reference="profiles" >
<SelectArrayInput optionText="label" />
</ReferenceArrayInput>
I think it's making the appropriate calls :
But here is my result :
Any idea what I'm doing wrong ?
Thanks in advance for your help !
On docs, ReferenceArrayInput is said to expect a source prop pointing to an array os ids, array of primitive types, and not array of objects with id. Looks like you are already transforming your raw response from api, so if you could transform a bit more, mapping [{id}] to [id], it could work.
If other parts of your app expects profiles to be an array of objects, just create a new object entry like profilesIds or _profiles.
As gstvg said, ReferenceArrayInput expects an array of primitive type, not array of objects.
If your current record is like below:
{
"id": 1,
"tags": [
{ id: 'programming', name: 'Programming' },
{ id: 'lifestyle', name: 'Lifestyle' }
]
}
And you have a resource /tags, which returns all tags like:
[
{ id: 'programming', name: 'Programming' },
{ id: 'lifestyle', name: 'Lifestyle' },
{ id: 'photography', name: 'Photography' }
]
Then you can do something like this (it will select the tags of current record)
<ReferenceArrayInput
reference="tags"
source="tags"
parse={(value) => value && value.map((v) => ({ id: v }))}
format={(value) => value && value.map((v) => v.id)}
>
<AutocompleteArrayInput />
</ReferenceArrayInput>
I resolve most of my problem only few left out of which this one is preventing me to submit the form. I am using Nativescript + vue and without Typescript. how to display the Valueproviders with array list? Here is the code
https://play.nativescript.org/?template=play-vue&id=2oWObE
There was the problem with your data type. As per the documentation Array should have key and label properties. But still if you want id and name then you should try like below.
'valuesProvider': {
key: 'id',
label: 'name',
items: [
{ id: 1121, name: 'Shanghai' },
{ id: 3651, name: 'Lagos' },
{ id: 5213, name: 'Moscow' },
{ id: 6214, name: 'São Paulo' },
{ id: 6985, name: 'Sydney' }
]
};
https://docs.nativescript.org/vuejs/ns-ui/DataForm/dataform-editors-providers
Anyway, I tried that and that was not working for me either then searched for it and relaised that there is an open feature request to support the valuesProvider for picker from JSON metadata. You can vote to support the same feature.
https://github.com/NativeScript/nativescript-ui-feedback/issues/369
Solution
Just get you cityList out of vue data and map your data
https://play.nativescript.org/?template=play-vue&id=2oWObE&v=6
more detailed version with groups
https://play.nativescript.org/?template=play-vue&id=rqK7wO&v=3
using extraction query (which used url decoded for reading):
https://api.keen.io/3.0/projects/xxx/queries/extraction?api_key=xxxx&event_collection=dispatched-orders&filters=[{"property_name":"features.tradeId","operator":"eq","property_value":8581}]&timezone=28800
return
{
result: [
{
mobile: "13185716746",
keen : {
timestamp: "2015-02-10T07:10:07.816Z",
created_at: "2015-02-10T07:10:08.725Z",
id: "54d9aed03bc6964a7d311f9e"
},
data : {
itemId: 2130,
num: 1
},
features: {
communityId: 2000,
dispatcherId: 39,
tradeId: 8581
}
}
]
}
but if i use the same filters in my delete query url (which used url decoded for reading):
https://api.keen.io/3.0/projects/xxxxx/events/dispatched-orders?api_key=xxxxxx&filters=[{"property_name":"features.tradeId","operator":"eq","property_value":8581}]&timezone=28800
return
{
properties: {
data.num: "num",
keen.created_at: "datetime",
mobile: "string",
keen.id: "string",
features.communityId: "num",
features.dispatcherId: "num",
keen.timestamp: "datetime",
features.tradeId: "num",
data.itemId: "num"
}
}
plz help me ...
It looks like you are issuing a GET request for the delete comment. If you perform a GET on a collection you get back the schema that Keen has inferred for that collection.
You'll want to issue the above as a DELETE request. Here's the cURL command to do that:
curl -X DELETE "https://api.keen.io/3.0/projects/xxxxx/events/dispatched-orders?api_key=xxxxxx&filters=[{"property_name":"features.tradeId","operator":"eq","property_value":8581}]&timezone=28800"
Note that you'll probably need to URL encode that JSON as you mentioned in your above post!