Related
I need to set datatype for Additional Column with Dynamic Content in Sink in ADF
By default its taking nvarchar(max) from Json obj but I need bigInt
Below is a Json Obj which create table with Additional column
{
"source": {
"type": "SqlServerSource",
"additionalColumns": [
{
"name": "ApplicationId",
"value": 3604509277250831000
}
],
"sqlReaderQuery": "SELECT * from Table A",
"queryTimeout": "02:00:00",
"isolationLevel": "ReadUncommitted",
"partitionOption": "None"
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": "autoCreate",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
}
ADF Configuration
After create table Database - column with datatype
If I convert Dynamic content into Int
#int(pipeline().parameters.application.applicationId)
Then getting below warning
Please let me know how can I set Datatype in ADF
I also tried the same and getting same result.
By default its taking nvarchar(max) from Json obj but I need bigInt
To resolve this when you add additional column in your source data set and in Mapping click onimport schema it will import the schema of the source and also give you additional column in schema you have to change the type of the column as Int64 as shown in below image. in below image you can see after name there is additional means it is an additional column.
After this run your pipeline, It will create additional column with data type bigint .
{
"name": "pipeline2",
"properties": {
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "JsonSource",
"additionalColumns": [
{
"name": "name",
"value": {
"value": "#pipeline().parameters.demo.age",
"type": "Expression"
}
}
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"sink": {
"type": "AzureSqlSink",
"writeBehavior": "insert",
"sqlWriterUseTableLock": false,
"tableOption": "autoCreate",
"disableMetricsCollection": false
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"mappings": [
{
"source": {
"path": "$['taskId']"
},
"sink": {
"name": "taskId",
"type": "String"
}
},
{
"source": {
"path": "$['taskObtainedScore']"
},
"sink": {
"name": "taskObtainedScore",
"type": "String"
}
},
{
"source": {
"path": "$['multiInstance']"
},
"sink": {
"name": "multiInstance",
"type": "String"
}
},
{
"source": {
"path": "$['name']"
},
"sink": {
"name": "name",
"type": "Int64"
}
}
],
"collectionReference": ""
}
},
"inputs": [
{
"referenceName": "Json1",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "AzureSqlTable1",
"type": "DatasetReference"
}
]
}
],
"parameters": {
"demo": {
"type": "object",
"defaultValue": {
"name": "John",
"age": 30,
"isStudent": true
}
}
},
"annotations": []
}
}
OUTPUT:
I feel compelled to specify from the very beginning that I am a total newbie to Snowflake and the idea of JSON data tables.
This being said I find myself faced with the following challenge:
I have a Snowflake table called ANIMALS which holds a JSON object with a semi irregular structure on the conditions part:
{
"animals": [
{
"name": "TIGER",
"id": 101,
"conditions": {
"and": [
{
"operator": "=",
"attribute": "ANIMAL_FAMILY",
"value": "feline"
},
{
"operator": "IN",
"attribute": "SUBSPECIES",
"value": ["Bengal", "Sumatran", "Siberian"]
},
{
"operator": "=",
"attribute": "ORDER",
"value": "carnivorous"
}
]
}
},
{
"name": "CAT",
"id": 102,
"conditions": {
"and": [
{
"operator": "=",
"attribute": "ANIMAL_FAMILY",
"value": "feline"
},
{
"or": [
{
"operator": "IN",
"attribute": "SUBSPECIES",
"value": ["Abyssinian", "Manx", "Siamese", "Sphynx"]
},
{
"operator": "=",
"attribute": "ENVIRONMENT",
"value": "domestic"
}
]
}
]
}
},
{
"name": "DOG",
"id": 103,
"conditions": {
"or": [
{
"operator": "=",
"attribute": "IS_MOST_BELOVED_PET",
"value": true
},
{
"and": [
{
"operator": "=",
"attribute": "ANIMAL_FAMILY",
"value": "canine"
},
{
"operator": "=",
"attribute": "ENVIRONMENT",
"value": "domestic"
}
]
},
{
"and": [
{
"operator": "=",
"attribute": "ENVIRONMENT",
"value": "domestic"
},
{
"or": [
{
"operator": "C",
"attribute": "ANIMAL_FAMILY",
"value": "%feline%"
},
{
"operator": "IN",
"attribute": "SUBSPECIES",
"value": ["Akita", "Beagle", "Border Collie", "Cane Corso", "Chihuahua"]
}
]
}
]
},
]
}
},
{
"name": "HORSE",
"id": 104,
"conditions": {
"operator": "=",
"attribute": "ANIMAL_FAMILY",
"value": "equine"
}
}
]
}
There is absolutely no limit for the degree of nesting the AND and OR operators, which unfortunately makes the conditions very irregular.
Given this JSON I need to dynamically generate a CASE WHEN [condition] THEN [value] clause which will look like this:
CASE WHEN ((ANIMAL_FAMILY = 'feline') AND (SUBSPECIES IN ('Bengal', 'Sumatran', 'Siberian')) AND (ORDER = 'carnivorous')) THEN 'TIGER'
WHEN ((ANIMAL_FAMILY = 'feline') AND ((SUBSPECIES IN ('Abyssinian', 'Manx', 'Siamese', 'Sphynx') OR (ENVIRONMENT = 'domestic')))) THEN 'CAT'
WHEN ((IS_MOST_BELOVED_PET = TRUE) OR ((ANIMAL_FAMILY = 'canine') AND (ENVIRONMENT = 'domestic')) OR ((ENVIRONMENT = 'domestic') AND ((ANIMAL_FAMILY NOT LIKE %feline%) OR (SUBSPECIES IN ('Akita', 'Beagle', 'Border Collie', 'Cane Corso', 'Chihuahua'))))) THEN 'DOG'
WHEN ANIMAL_FAMILY = 'equine' THEN 'HORSE'
END AS animal
Is this doable solely relying on Snowflake queries and no other third-party programming languages?
Any hints and pointers would be greatly appreciated.
Thank you in advance for your replies.
I have three collections
posts=[
{
"id": "p1",
"title": "title 1"
},
{
"id": "p2",
"title": "title 2"
}]
users = [
{
"id": "u1",
"name": "name1"
},
{
"id": "u2",
"name": "name2"
}]
comments = [
{
"userId": "u1",
"postId": "p1",
"comment": "comment 1"
}]
I want to get all collection posts and comments in each post by userId(u1) as:
posts=[
{
"id": "p1",
"title": "title 1",
"comments":[
"userId": "u1",
"comment": "comment 1"
]
},
{
"id": "p2",
"title": "title 2",
"comments":[]
}]
I used aggregate function and $lookup operator but I don't know using the $match operator to filter userId. I used aggregate bellow:
self.db.posts.aggregate([
{
"$lookup":{
"from": "comments",
"localField": "id",
"foreignField": "postId",
"as": "comments",
}
},
{
"$match":{
"comments.userId": {"$eq": param.objectUserId}
},
},
{"$skip": (param.page - 1) * param.pageSize},
{"$limit": param.pageSize},
{"$sort": {"unixDate": pymongo.DESCENDING}}
])
It only return one post in array corresponding with userId="u1"
Please help me!
Thank all!
You have to make use of the pipeline option of $lookup stage and pass the additional conditions that you want to apply.
db.posts.aggregate([
{
"$lookup": {
"from": "comments",
"let": {
"pId": "$id"
},
"pipeline": [
{
"$match": {
"$expr": {
"$eq": [
"$postId",
"$$pId"
],
},
"userId": "u1",
},
},
{
"$project": {
"_id": 0,
"userId": 1,
"comment": 1,
},
},
],
"as": "comments"
}
}
])
Mongo Playground Sample Execution
self.db.posts.aggregate([
{
"$lookup": {
"from": "comments",
"let": {
"pId": "$id"
},
"pipeline": [
{
"$match": {
"$expr": {
"$eq": [
"$postId",
"$$pId"
],
},
"userId": param.objectUserId,
},
},
{
"$project": {
"_id": 0,
"userId": 1,
"comment": 1,
},
},
],
"as": "comments"
}
},
{"$skip": (param.page - 1) * param.pageSize},
{"$limit": param.pageSize},
{"$sort": {"unixDate": pymongo.DESCENDING}}
])
I am currently building a web application where you can create setlists (arrays) with an array of lyric objectId's inside, that you can then sort / order into how you want it. So if you would like the 3rd list item to become the first, then you simply drag and drop it to the first line.
I now have a problem in my mongoose schema. I am looking for a way to implement an order attribute or something that would allow me to add a order value such as 0 or 1 depending on the position of the lyrics. Does any of you know how to best implement such order?
Here is a copy of my schema. Currently lyrics is an array of lyric objectId's. But in there i would need an "Order" as well, so that i can sort the array according to the order value.
const mongoose = require("mongoose");
const SetlistSchema = new mongoose.Schema({
setlistName: { type: String, required: true },
lastEdited: { type: Date },
createdAt: { type: Date, default: Date.now },
lyrics: [{
type: mongoose.Schema.Types.ObjectId,
ref: 'Lyric'
}],
author: {
id: {
type: mongoose.Schema.Types.ObjectId,
ref: "User"
},
username: String
}
});
module.exports = mongoose.model("Setlist", SetlistSchema);
Here is the Lyrics schema.
const mongoose = require("mongoose");
const LyricSchema = new mongoose.Schema({
lyricName: { type: String, required: true },
lyricContent: { type: String, required: true },
lastEdited: { type: Date },
createdAt: { type: Date, default: Date.now },
author: {
id: {
type: mongoose.Schema.Types.ObjectId,
ref: "User"
},
username: String
}
});
module.exports = mongoose.model("Lyric", LyricSchema);
If adding an order number isn't the best practice, what can you then recommend as a way of keeping track of which order the user would like the lyrics to show up?
You can use aggregation framework to sort lyrics by order field. You first need to add a sort field with Number type.
Setlist.aggregate([
{
$unwind: "$lyrics"
},
{
$lookup: {
from: "lyrics", // MUST be the PHYSICAL collection name
localField: "lyrics",
foreignField: "_id",
as: "lyrics"
}
},
{
$sort: {
"lyrics.order": 1
}
},
{
"$group": {
"_id": "$_id",
"lyrics": {
"$push": "$lyrics"
},
"allFields": {
"$first": "$$ROOT"
}
}
},
{
"$replaceRoot": {
"newRoot": {
"$mergeObjects": [
"$allFields",
{
"lyrics": "$lyrics"
}
]
}
}
}
])
Playground
Sample documents:
db={
"lists": [
{
"_id": ObjectId("5a934e000102030405000000"),
"setlistName": "list1",
"lastEdited": ISODate("2020-03-18T23:11:56.443+03:00"),
"createdAt": ISODate("2020-03-15T23:11:56.443+03:00"),
"lyrics": [
ObjectId("6a934e000102030405000000"),
ObjectId("6a934e000102030405000001"),
ObjectId("6a934e000102030405000002")
]
},
{
"_id": ObjectId("5a934e000102030405000001"),
"setlistName": "list2",
"lastEdited": ISODate("2020-03-11T23:11:56.443+03:00"),
"createdAt": ISODate("2020-03-11T23:11:56.443+03:00"),
"lyrics": [
ObjectId("6a934e000102030405000003"),
ObjectId("6a934e000102030405000004")
]
}
],
"lyrics": [
{
"_id": ObjectId("6a934e000102030405000000"),
"name": "Lyric 1",
"order": 3
},
{
"_id": ObjectId("6a934e000102030405000001"),
"name": "Lyric 2",
"order": 1
},
{
"_id": ObjectId("6a934e000102030405000002"),
"name": "Lyric 3",
"order": 2
},
{
"_id": ObjectId("6a934e000102030405000003"),
"name": "Lyric 4",
"order": 2
},
{
"_id": ObjectId("6a934e000102030405000004"),
"name": "Lyric 5",
"order": 1
}
]
}
Output: (as you see lyrics are sorted by order field value)
[
{
"_id": ObjectId("5a934e000102030405000000"),
"createdAt": ISODate("2020-03-15T20:11:56.443Z"),
"lastEdited": ISODate("2020-03-18T20:11:56.443Z"),
"lyrics": [
[
{
"_id": ObjectId("6a934e000102030405000001"),
"name": "Lyric 2",
"order": 1
}
],
[
{
"_id": ObjectId("6a934e000102030405000002"),
"name": "Lyric 3",
"order": 2
}
],
[
{
"_id": ObjectId("6a934e000102030405000000"),
"name": "Lyric 1",
"order": 3
}
]
],
"setlistName": "list1"
},
{
"_id": ObjectId("5a934e000102030405000001"),
"createdAt": ISODate("2020-03-11T20:11:56.443Z"),
"lastEdited": ISODate("2020-03-11T20:11:56.443Z"),
"lyrics": [
[
{
"_id": ObjectId("6a934e000102030405000004"),
"name": "Lyric 5",
"order": 1
}
],
[
{
"_id": ObjectId("6a934e000102030405000003"),
"name": "Lyric 4",
"order": 2
}
]
],
"setlistName": "list2"
}
]
Problem with using mapObject function properly.
Trying to retain existing array structure but calculate number of vehicles and properties and update the existing array that contains the value.
GENERAL data comes from one source, VEHICLE data comes from another source, PROPERTY data comes from another source. So when merging, I have to update GENERAL data with count of other source data.
Also GENERAL is an array object, it will always have 1. So using GENERAL[0] is safe and fine.
Original Payload
[
{
"commId": "1",
"GENERAL": [
{
"ID": "G1",
"VEHICLE_COUNT": "TODO",
"PROPERTY_COUNT": "TODO"
}
],
"VEHICLE": [
{
"ID": "V1-1"
},
{
"ID": "V1-2"
}
],
"PROPERTY": [
{
"ID": "P1-1"
}
]
},
{
"commId": "2",
"GENERAL": [
{
"ID": "G2",
"VEHICLE_COUNT": "TODO",
"PROPERTY_COUNT": "TODO"
}
],
"VEHICLE": [
{
"ID": "V2-1"
}
],
"PROPERTY": [
{
"ID": "P2-1"
},
{
"ID": "P2-2"
}
]
},
{
"commId": "3",
"GENERAL": [
{
"ID": "G3",
"VEHICLE_COUNT": "TODO",
"PROPERTY_COUNT": "TODO"
}
],
"VEHICLE": [
{
"ID": "V3-1"
},
{
"ID": "V3-2"
},
{
"ID": "V3-3"
}
]
}
]
Tried using map to loop through the payload and tried modifying 2 attribute but only managed to map one but even that is showing wrong output.
test map (item, index) -> {
(item.GENERAL[0] mapObject (value, key) -> {
(key): (value == sizeOf (item.VEHICLE)
when (key as :string) == "VEHICLE_COUNT"
otherwise value)
})
}
Expected output:
[
{
"commId": "1",
"GENERAL": [
{
"ID": "G1",
"VEHICLE_COUNT": "2",
"PROPERTY_COUNT": "1"
}
],
"VEHICLE": [
{
"ID": "V1-1"
},
{
"ID": "V1-2"
}
],
"PROPERTY": [
{
"ID": "P1-1"
}
]
},
{
"commId": "2",
"GENERAL": [
{
"ID": "G2",
"VEHICLE_COUNT": "1",
"PROPERTY_COUNT": "2"
}
],
"VEHICLE": [
{
"ID": "V2-1"
}
],
"PROPERTY": [
{
"ID": "P2-1"
},
{
"ID": "P2-2"
}
]
},
{
"commId": "3",
"GENERAL": [
{
"ID": "G3",
"VEHICLE_COUNT": "3",
"PROPERTY_COUNT": "0"
}
],
"VEHICLE": [
{
"ID": "V3-1"
},
{
"ID": "V3-2"
},
{
"ID": "V3-3"
}
]
}
]
Getting totally wrong output so far:
[
{
"ID": "G1",
"VEHICLE_COUNT": false,
"PROPERTY_COUNT": "TODO"
},
{
"ID": "G2",
"VEHICLE_COUNT": false,
"PROPERTY_COUNT": "TODO"
},
{
"ID": "G3",
"VEHICLE_COUNT": false,
"PROPERTY_COUNT": "TODO"
}
]
Edited: Update for dynamic transform
The below dataweave transform is not particularly attractive, but it might work for you.
Thanks to Christian Chibana for helping me find a dynmaic answer by answering this question: Why does Mule DataWeave array map strip top level objects?
%dw 1.0
%output application/json
---
payload map ((item) ->
(item - "GENERAL") ++
GENERAL: item.GENERAL map (
$ - "VEHICLE_COUNT"
- "PROPERTY_COUNT"
++ { VEHICLE_COUNT: sizeOf (item.VEHICLE default []) }
++ { PROPERTY_COUNT: sizeOf (item.PROPERTY default []) }
)
)
It is dynamic, so everything should be copied across as it comes in, with only the two fields you want being updated.
The output for this transform with the input you supplied is below. Only difference from your desired is that the counts are shown as numbers rather than strings. If you really need them as strings you can cast them like (sizeOf (comm.VEHICLE default [])) as :string,
[
{
"commId": "1",
"VEHICLE": [
{
"ID": "V1-1"
},
{
"ID": "V1-2"
}
],
"PROPERTY": [
{
"ID": "P1-1"
}
],
"GENERAL": [
{
"ID": "G1",
"VEHICLE_COUNT": 2,
"PROPERTY_COUNT": 1
}
]
},
{
"commId": "2",
"VEHICLE": [
{
"ID": "V2-1"
}
],
"PROPERTY": [
{
"ID": "P2-1"
},
{
"ID": "P2-2"
}
],
"GENERAL": [
{
"ID": "G2",
"VEHICLE_COUNT": 1,
"PROPERTY_COUNT": 2
}
]
},
{
"commId": "3",
"VEHICLE": [
{
"ID": "V3-1"
},
{
"ID": "V3-2"
},
{
"ID": "V3-3"
}
],
"GENERAL": [
{
"ID": "G3",
"VEHICLE_COUNT": 3,
"PROPERTY_COUNT": 0
}
]
}
]