I am new to ADF so please excuse if this pretty obvious.
I am using Azure ADF to orchestrate data ingestion and using ADF connector to pull data from another cloud.
I have currently hard coded the values in the where clause and looking to manage the list dynamically.
My initial thought was to have the list as a txt/csv file in blob storage and then using lookup activity I can pass the value to the where clause.
The problem is I am getting values as Prop_0 in an array format and I am not sure how do I pass all the values of the array to the where clause.
I tried Prop[0] and it filtered based on the 1st value of the array but I want the query to filter records for all the values from the array.
Is there any way to specify all values of an array with Prop_0[]?
from view1
where col1 in ('A BC', 'AB C', 'ABC')
Output of lookup activity:
{
"count": 3,
"value": [
{
"Prop_0": "'A BC'"
},
{
"Prop_0": "'AB C'"
},
{
"Prop_0": "'ABC'"
}
]
}
I tried to use the Append variable activity but did not make much progress either.
Appreciate any help on getting this resolved?
Another alternative I am thinking of is storing the list in Azure SQL table and then using the column heading in a lookup activity.
Is there any other easier/simpler way to handle this?
Related
Our database has a field with JSON data that we'd like to use in reports. E.g.
{
owner_type: "USER",
updated_at: 1641996749092389600,
version_no: 1,
entity_type: "INDIVIDUAL",
country:"ES",
}
How can one create dynamic fields in Superset, e.g. to expose owner_type as its own field?
I'm coming from tools like Snowflake and Zoho Analytics where you could build Views, Dynamic Tables and Formula Fields based on aggregated raw data.
You can add columns to your table in Superset. Hover on 'Sources' on the header and select 'Tables'. Then from there, choose the option to edit the record of your table. In that you can add a calculated column/custom column.
To add a column for owner_type, lets name the custom column as owner_type. Fill the datatype for the new column as VARCHAR(100). Choose the table from the dropdown. In the expression, put json_column->"$.owner_type" and then hit save. This expression is for MySQL database. You can find the expression to parse JSON in your particular DB.
I am using IICS to do ETL. My source is a JSON file. I am using Hierarchy parser to parse that JSON and load the output into database. Everything is going fine except one thing. Within that source JSON file there is one field which is array like this:
label:[Red, Green, Blue, Yellow].
Now in table which I am loading data into, there is a label column of varchar2(510 byte). When data is loaded into target by running the job, label column in table gets only one value which is at last index in source array, in above case it gets only 'Yellow' value while other 3 values are lost.
Can anybody tell me how to store entire array into the table column?
PS: We have no flexibility to use any programming language in IICS, all that we need to do is using IICS transformations.
Please let me know if additional information is needed.
Thanks in advance!
I have a Data Factory Pipeline that I want to have iterate through the rows of a SQL Lookup activity. I have narrowed the query down to three columns and 500 rows.
I understand that to reference a value in the table I use:
#{activity('lookupActivity').output.value[row#].colname}
However, the for each needs to have something to iterate over. My first guess is to set some array variable to the rows of the returned sql query. So what do I set that variable to?
#{activity('lookupActivity').output.value?
Lastly, it looks like almost all data is represented as a json in ADF, is this true? And how could I view the output of this look up as a json so I can understand what my dynamic content needs to look like?
You're right that everything (nearly) is JSON. (Exception: Azure Data Factory v2: Activity execute pipeline output
So you can put your #activity('lookupActivity').output.value which is an array into the foreach activity on the settings tab, like this
Then inside your foreach loop, you reference the current value of one of the columns as #item().colname.
You can use the output value to for each activity and go through one at a time. You can do sequential or parallel depending on your needs.
Using bigQuery UI I've created new table free_schem_table and haven't set any schema, then I tried to execute:
insert into my_dataset.free_schema_table (chatSessionId, chatRequestId,senderType,senderFriendlyName)
values ("123", "1234", "CUSTOMER", "Player")
But BigQuery UI demonsrtrated me the popup where written:
Column chatSessionId is not present in table my_dataset.free_schema_table at [1:43]
I expected that BiqQuery is a NoSql storage and I should be able to insert rows with different columns.
How could I achieve it ?
P.S.
schema:
BigQuery requires a schema with strong type.
If you need free schema, similar thing in BigQuery is to define a single column in STRING type and store JSON inside.
JSON functions will help you extract field from JSON string later, but you don't benefit from BigQuery's optimization if you predefine your schema and save data in different columns.
I want to read the resultset of a table using the following stmt:
Select col1 as A,col2 as B from tablename;
Then, I want to read each row of the result set into local variables of the SSIS package and for each row I have to pass the values to the script task.
I want to use foreach loop in SSIS. I took Foreach Item Enumerator.
The question: How to read the values into the variable using the For each Item enumerator and how can i iterator can i use select count(*) from table; pass that value to a variable and asssign the count value in the foreach loop.
I'm stuck at how to assign the count value and read columns to variables. Can anyone help with these?
Thanks in advance.
I'm not exactly sure what it is that you're trying to do, but it would seem that you're trying to process data in your control flow. The foreach iterator is not made for processing data sets, it's made for iterating over multiple data sets and doing something to each of them, usually passing them to a data flow.
You might find it more useful to create a data flow. Start with a data source component that gets the data that you want and then pass the data to a Script Component to do the processing.