I have a lookup function that returns a list of valid GUID IDs in ADF. I then have a foreach process which runs a stored procedure for each GUID ID and assigns an ID column to it.
What I want to do is then have another lookup run which will run the below query to bring me the GUID and also the newly assigned ID. It is very simple to write in SQL
SELECT GUID, Identifier from DBO.GuidLOAD
WHERE GUID in ('GUIDID','GUIDID','GUIDID')
However I am struggling to translate this in ADF.. I have got as far as the #Concat part and most of the help I find online only refers to dynamic queries with single values as input parameters.. where mine is a list of GUIDs where they may be 1, more or none at all..
Can someone advise the best way of writing this dynamic query?
first 2 run fine, I just need the third lookup to run the query based on the output of the first lookup
You can use string interpolation (#{...}) instead of concat(). I have a sample table with with 2 records in my demo table as shown below:
Now, I have sample look up which returns 3 guid records. The following is debug output of lookup activity.
Now, I have used a for loop to create an array of these guid's returned by lookup activity using append variable activity. The items value for each activity is #activity('get guid').output.value. The following is the configuration of append variable inside for each.
#item().guids
I have used join function on the above array variable to create a string which can be used in the required query.
"#{join(variables('req'),'","')}"
Now, the query accepts guid's wrapped inside single quotes i.e., WHERE GUID in ('GUIDID','GUIDID','GUIDID'). So, I created 2 parameters with following values. I used them in order to replace double quotes from the above final variable with single quotes.
singlequote: '
doublequote: "
Now in the look up where you want to use your query, you can build it using the below dynamic content:
SELECT guid, identifier from dbo.demo WHERE GUID in (#{replace(variables('final'),pipeline().parameters.doublequote,pipeline().parameters.singlequote)})
Now, when I debug the pipeline, the following query would be executed which can be seen in the debug input of the final lookup.
The output would be as below. Only one row should be returned from the sample I have taken and output is as expected:
Related
I have a dag set up in the following manner:
Task 1: Creates a table in snowflake with columns (run_id, A, B, C) using SnowflakeConnectorOperator
Task 2:Python Operator that performs some functionality and also produces a unique string for the run called 'run_id'. I use ti.xcom_push(key="run_id", value=run_id)to store this string.I produce the string with the uuid library and then type cast it to str.
Task 3: Queries the data from Snowflake and populates a google sheet. In this task I want it to populate the google sheet with the data from this run only i.e where RUN_ID = '{{ti.xcom_pull(task_ids='Task-2')}}' however, when I add this where clause the google sheet is blank. It seems to be receiving an empty string here in the xcom. Without the WHERE filter I have a google sheet that has all my data.
I've tried to play around with the way I write \"{{ ti.xcom_pull(task_ids='Task-2') }}\" instead of
where RUN_ID = '{{ti.xcom_pull(task_ids='Task-2')}}'. I'm not able to see what the xcom_pull returns. I've checked airflow ui and can confirm that it is successfully pushing the string. However, the pull statement seems to read the string as NONE in the query. Does anyone have input as to how to use xcom in a sql query's where clause where I want to match type string? I've tested the query out in Snowflake with a particular string value and have received a response. I suspect that in my third task there is a type mismatch when filtering with the where clause and that is causing the sheet to be blank.
You are trying to pull from XCOM the return value from Task-2 which I think doesn't have a return value. You should use xcom_pull(key='run_id') in this case
My SQL is limited and I have inherited the schema so please go easy!
The upper tables are populated, the lower section is to be completed.
The customer wants a simple reference, C_SCAFFOLD_DOMAIN_COMBO, for every C_SEQUENCE_RESULT of the same TYPE having the same C_PROTEIN_REGIONs (only ever two) of the same REGION_NAME.
My intention is to read each C_SEQUENCE_RESULT getting the (2) C_PROTEIN_REGION.SEQUENCEs with their REGION_NAME using a cursor.
I can then check if they are referenced by a C_SCAFFOLD_DOMAIN_COMBO already.
If so, then simply add the correct C_SCAFFOLD_DOMAIN_NAME FK to C_SEQUENCE_RESULT.
If not, create a new C_SCAFFOLD_DOMAIN_NAME entry with the appropriate C_PROTEIN_REGION refs and add the FK to C_SEQUENCE_RESULT.
My SQL query returns two results for each C_SEQUENCE_RESULT e.g.
REGION_NAME------SEQUENCE
Loop 1------ABCDEFG
Loop 2------HIJKLMN
Running as a simple query and can INSERT INTO a temp table and get the values. As a cursor I don’t seem to be able.
How can I read the data returned above using a cursor?
Thanks
I need to make a dimension for a datawarehouse using pentaho.
I need to compare a number in a table with the number I get from a REST call.
If the number is not in the table, I need to set it to a default (999). I was thinking to use table input step with a select statement, and a javascript step that if the result is null to set it to 999. The problem is if there is no result, there is nothing passed through. How can this be done? Another idea was to get all values from that table and somehow convert it to something so I can read id as an array in javascript. I'm very new to pentaho DI but I've did some research but couldn't find what I was looking for. Anyone know how to solve this? If you need information, or want to see my transformation let me know!
Steps something like this:
Load number from api
Get Numbers from table
A) If number not in table -> set number to value 999
B) If number is in table -> do nothing
Continue with transformation with that number
I have this atm:
But the problem is if the number is not in the table, it returns nothing. I was trying to check in javascript if number = null or 0 then set it to 999.
Thanks in advance!
Replace the Input rain-type table by a lookup stream.
You read the main input with a rest, and the dimension table with an Input table, then make a Stream Lookup in which you specify that the lookup step is the dimension input table. In this step you can also specify a default value of 999.
The lookup stream works like this: for each row coming in from the main stream, the steps looks if it exists in the reference step and adds the reference fields to the row. So there is always one and exactly one passing by.
Is there a way to retrieve the field names from an actual query, similar to how you can retrieve field names from a table using the INFORMATION_SCHEMA? In essence, I'm wanting to accept an entire query as a parameter, and be able to build an empty table with the field names from said query.
Right now, I'm using a STUFF() function to replace the SELECT with SELECT TOP 1 and replacing the FROM with a INTO tablename FROM, and then truncating tablename. It works, but I need this to be able to handle complex queries where the first occurrence of FROM might not be the one I need (in case the user is using a subquery for a field, for example).
Hi I have a table which was designed by a lazy developer who did not created it in 3rd normal form. He saved the arrays in the table instead of using MM relation . And the application is running so I can not change the database schema.
I need to query the table like this:
SELECT * FROM myTable
WHERE usergroup = 20
where usergroup field contains data like this : 17,19,20 or it could be also only 20 or only 19.
I could search with like:
SELECT * FROM myTable
WHERE usergroup LIKE 20
but in this case it would match also for field which contain 200 e.g.
Anybody any idea?
thanx
Fix the bad database design.
A short-term fix is to add a related table for the correct structure. Add a trigger to parse the info in the old field to the related table on insert and update. Then write a script to [parse out existing data. Now you can porperly query but you haven't broken any of the old code. THen you can search for the old code and fix. Once you have done that then just change how code is inserted or udated inthe orginal table to add the new table and drop the old column.
Write a table-valued user-defined function (UDF in SQL Server, I am sure it will have a different name in other RDBMS) to parse the values of the column containing the list which is stored as a string. For each item in the comma-delimited list, your function should return a row in the table result. When you are using a query like this, query against the results returned from the UDF.
Write a function to convert a comma delimited list to a table. Should be pretty simple. Then you can use IN().