I'm a complete beginner with Azure Data Factory and I'm struggling to set a parameter using a lookup output, and then use the parameter within my query to import data.
I'm using an Azure SQL database as the source for the lookup and an Oracle database as the source for the copy data
My lookup query works ok and my copy data query works without a parameter so I know I'm part way there, but I'm struggling with the parameter part.
My lookup details are below with the output query
The copy data query is below with the lookup activity as the parameter added but as you can see it's not correct.
I'm not sure if I should be using the lookup as the parameter or if I should be using a parameter with the value generated from the lookup. In which case I have no idea how to do this.
I'm also pretty sure the parameter in the query needs some work as even hardcoding a parameter doesn't seem to work.
If anyone can help to get this working I'd be very grateful.
There doesn't seem to be much help with turning lookups into parameters on the internet.
Thanks in advance for any help anyone can give.
Try enclosing the lookup output as follows. Select column from table where rowstamp> '#{activity('Lookup1').output.firstrow.RowStamp}'
Related
I am doing something that seems like it should be very easy yet I have yet to figure this out. I have read countless posts and tried everything I can think of and still no success.
Here goes:
I created a Pipeline Parameter pplLastWritten with a default value of 2022-08-20 12:19:08 (I have tried without the time for troubleshooting and still get errors)
Then I create a Data Flow Parameter ptblTableName
I have tried to convert to a Date, keeping as is and converting later...you name it still errors out.
In the expression builder I tried this and many more ways to build out to a sql statement:
"SELECT * FROM xxxxxx."+$ptblTableName+"where Lastwritten>='{$ptblLastWritten}'"
This is the post I got the idea from: ADF data flow concat expression with single quote
This is the error I got most of the time.
Operation on target df_DynamicSelect failed: {"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed due to reason: at Source 'RptDBTEST'(Line 5/Col 0): The store configuration is not defined. This error is potentially caused by invalid parameter assignment in the pipeline.","Details":""}
I have tried so many things but in the end nothing has worked. I am new to Data Factory and come from the SSIS world which was so much easier. I would greatly appreciate someone helping. Please explain this like I'm a kindergartener because this tool is making me feel like it. :) Thank you in advanced.
I have tried various ways to format
Using different ideas in the expression builder
the ideas in this post: ADF data flow concat expression with single quote
You can use concat() function in the Data flow dynamic expression like below.
Here is the sample data in SQL.
I have created two dataflow parameters mytable and mydate.
Passed the values like below. Check the expression checkbox. For date you can also pass like this '2022-11-07T00:00:00.0000000'.
In the Query option use below Expression.
concat('select * from dbo.',$table_name,' where mydate >=','\'',$mydate,'\'')
Values inserted in Target table.
I've got this doubt in Azure Data Factory. My pipeline has a copy data activity, and after loading the information in the table I need to update a field in that destination based on a parameter. It is a simple update, but given that we do not have a SQL task (present in SSIS) I do not what to use. Create a SP for this does not seem to be the most appropriate solution, besides, modify the database is complicated. I thought the option "Use Query" in the Lookup activity could be a solution, but this does not allow me to create a SQL query with a parameter, just like in a Source.
What could be a possible workaround?
You are on the right track with the Lookup. That is definitely the way to go. The query field there will allow you to create dynamic SQL just like you did within the copy activity. You just need to reference the variable/parameter properly.
Also, with the Lookup, it will always expect something returned. You don't have to do anything with that returned value. Just ignore it, but the Lookup will not work without returning something. So, that query field would contain something like:
UPDATE dbo.MyTable SET IsComplete = 1 WHERE RunId = #{pipeline().parameters.runId};
SELECT 0 AS DummyValue; -- Necessary for Lookup to work
I have been tasked with creating a method to copy the contents of an entire database to a central database. There are a number of source databases, all in Access. I've managed to copy the majority of the tables properly, 1:1. I'm using VBScript and ADO to copy the data. It actually works surprisingly well, considering that it's Access.
However
I have 3 tables that include subdatasheets (to those that don't know, a subdatasheet is a visual representation of a 1 to many relationship. You can see related records in another table inside the main table). When My script runs, I get an error. "No value given for one or more required parameters." When I open Access and try to run the same query that I've written in SQL, It pops up message boxes asking for parameters.
If I use the query wizard inside Access to build the select query, no parameters are required and I get no subdatasheet in the result set.
My question is this: how do I write a vanilla SQL query in my VBScript that does not require parameters and just gives me the data that I want?
I've tried copying the SQL from Access and running it through my VBScript and that doesn't seem to do the trick.
Any help is greatly appreciated!
As it turns out, you need to make sure that you've spelled all of the field names properly in your source query. If you've included additional fields that aren't actually in the source or destination table they'll need to be removed too.
I am retrieving some data from sql server and trying to write it to a text file, I am getting error in retrieving and passing it to variable Could you please help me in this.
You don't give a lot of detail regarding the error. Assuming your query is working and returning data to the table, Streamwriter.Write(dt) won't return the data in the table, it will return the name of the table or something like that. to get all the data you need to either loop through the rows and columns and print each one as you like or use dt.WriteXML(myIOStream), obWRiter.(myIOstream). You can also try streamwriter.write(dt.writexml()). I haven't tried anything out so you'll need to work on the code.
I'm looking how to run a local query with date filters on a saved pass-through queries in Access. I'm trying to leverage the pass through query as basically a View in my Access database... the local query is constructed dynamically in VBA, and intended to be a clean way to filter my pass through query. I'd like to avoid creating another pass through query, or altering the original, every time i run my Sub Procedure.
My problem is that the normal access date filter format #m/d/yyyy# doesn't seem to work. I've tried both altering the date format as well in the pass through query with 1. Convert(varchar(12),p.startDate,101); 2. Convert(date,p.StartDate,101);
but neither will work when the pass through query is queried against locally.
Does anyone know how to do this?
UPDATE - I just checked and Access is reading the field as Text... does anyone know how it can read it as a date? As i mentioned the CONVERT functions don't seem to be working to do this
In a passthru you MUST use the backend's syntax. If the BE is SQL Server then I'd use a syntax like this:
WHERE DocDate = '2015-03-17'