I am trying to parameterize the pre-copy script to DROP existing tables of the same schema and table name from a SQL Server sink. I have tried variations of the above, what's the correct syntax to use the dataset properties in the pre-copy script?
Additionally, is there a good resource on using the dynamic content in ADF?
Reviewing other questions with parameterized query steps, the parameters are incorrectly prefixed, the following was executed successfully:
DROP TABLE IF EXISTS #{item().TABLE_SCHEMA}.#{item().table_name}
Related
Our challenge is the following one :
in an Azure SQL database, we have multiple tables with the following table names : table_num where num is just an integer. These tables are created dynamically so the number of tables can vary. (from table_1, table_2 to table_N) All tables have the same columns.
As part of a U-SQL script file, we would like to execute the same query on all of these tables and generate an output csv file with the combined results of all these queries.
We tried several things :
U-SQL does not allow looping so we were thinking creating a View in our Azure SQL database that would combine all the tables using a cursor of some sort. Then, the U-SQL file would query this View (using external source). However, a View in Azure SQL database can only be created via a function and a function cannot execute dynamic SQL or even call a stored procedure...
We did not find a way to call a stored procedure of the external data source directly from U-SQL
we dont want to update our U-SQL job each time a new table is added...
Is there a way to do that in U-SQL through a custom extractor for instance? Any other ideas?
One solution I can think of is to use Azure Data Factory (v2) to assist in this.
You could create a pipeline with the following activities:
Lookup activity configured to execute the stored procedure
For Each activity that uses the output of the lookup activity as a source
As a child item use a U-Sql Activity that executes your U-Sql script which writes the output of a single table (the item of the For Each activity) to blob or datalake
Add a Copy Activity that merges the blobs from step 2.1 to one final blob.
If you have little or no experience working with ADF v2 do mind that it takes some time to get to know it but once you do, you won't regret it. Having a GUI to create the pipeline is a nice bonus.
Edit: as #wBob mentions another (far easier) solution is to somehow create a single table with all rows since all dynamically generated table have the same schema. You can create a stored procedure for populating this table for example.
I got a bunch of .csv files each containing a script that would create a certain table.
I want to create tables using these scripts in said files (each table to be created using one file).
I got a foreach loop container that specifies the path and which files to use.
I don't know how to configure the Execute SQL Task to execute the script in each one of these files in order to create a table.
You can use the Execute SQL Task with an input parameter of the table name (I would use the table name that the 'for each' container provides. I would first drop the table if it exists and then recreate it with a create table command (in the Execute SQL Task).
As other people have noted you may want to be careful with tasks that drop tables but I have created plenty of SSIS packages that involve truncating and/or creating tables.
I have a scenario where I would like to run an update script after a table input and table output job, can anyone assist? I have tried these four but I can't seem to figure out how to make them work.
My Current Transformation
Here's the scenario...
Table Input: MySQL Database Table1 (*Select * from Table1*)
Table Output: Oracle Database (Create Table 1)
(this runs well to completion but then I have to execute the update script manually. I am looking for a way to automate this)
The update query I would like to run:
*update odb.table1 set colum1='New Value1' where column1='Old Value1'
update odb.table1 set colum1='New Value2' where column1='Old Value2'*
Thank you in advance.
I used the Execute SQL Script tool. I just added the two update queries separated by a semicolon ;.
I created two transformations. One for the table input and table output and another for the Execute SQL Script Tool.
I then created a Kettle Job and placed my query after the table output transformation.
I am writing an ETL to extract data from HANA table and load into SQL Server in BODS.
My job is to create a new table on SQL Server every time I run my job with name as date of that day. I know we can do that for flat files by using global variable but not sure how we can declare similar variable in template table to get desired results?
Why you want to use template tables. You can do the same as below:
Load the data in a standard staging table using BODS
Using DS scripting mechanism generate a query to create a table
Execute the query using SQL transform
Generate another query to copy data from staging table to the table created above
Several other ways also like you can write a DB procedure to create a table with the desired name and copy over the data from stage to that table. This procedure you can call from DS.
Hope this helps.
Cheers.
Shaz
I tried to append data from a query to a bigquery table.
Job ID job_i9DOuqwZw4ZR2d509kOMaEUVm1Y
Error: Job failed while writing to Bigquery. invalid: Illegal Schema update. Cannot add fields (field: debug_data) at null
I copy and paste the query executed in above jon, run it in web console and choose the same dest table to append, it works.
The job you listed is trying to append query results to a table. That query has a field named 'debug_data'. The table you're appending to does not have that field. This behavior is by design, in order to prevent people from accidentally modifying the schema of their tables.
You can run a tables.update() or tables.patch() operation to modify the table schema to add this column (see an example using bq here: Bigquery add columns to table schema), and then you'll be able to run this query successfully.
Alternately, you could use truncate instead of append as the write disposition in your query job; this would overwrite the table, and in doing so, will allow schema changes.
See this post for how to have bigquery automatically add new fields to a schema while doing an append.
The code in python is:
job_config.schema_update_options = ['ALLOW_FIELD_ADDITION']