I have a SSIS job that contains 4 packages to perform ETL job
in the first package we use "select newid()" function to create an unique ID for that ETL process
wondering how can I pass that variable value to all ETL package so all 4 package can use the same ID
Execute package task is out of the picture cause we want the job to have 4 steps (perform by 4 packages)
Can anyone point me a direction ?
Thanks
I would create a "Master Control Package" to generate the ID, store it in a variable, then use Execute Package Tasks to call the sub-packages.
This makes your SQL Agent job definition simpler (1 step calling the MCP), and you can leverage out-of-the-box features of SSIS like parallel execution, conditional execution and restart control via checkpoints.
Related
So I'm working with a professor who wants me to create a SQL Database containing information from csv files (from the New York City Department of Transportation). I've written program that takes the csv file and converts it into the appropriate sql commands. So my question is, how do i automate the database so that every 5 minutes or so a program downloads the new csv file, runs it through my csv-to-SQL command program, and then enters the output from my csv-to-SQL command program into terminal (what I use to interface with my SQL database)? Is there a specific language I should look into, I've seen people talk about cron?
cron is a reference to making a scheduled task under Unix; the Windows equivalent is to set up a task to run using a Task Scheduler.
I don't know that there's a pure SQL answer to your problem--probably the way I'd approach it is by writing a simple Import program in the language of your choice, compiling it down to an .EXE, then setting up a Task Scheduler command to run the program every 5 minutes. (Alternately, you could leave the app up all the time and simply let a timer execute every 5 minutes to trigger the import).
The simplest thing you could do is loop your script.
If you're running PHP you could do something like:
$running = true;
while ($running)
{
// Your code that gets and converts CSV
// and then saves it in SQL.
$running = $some_error ? false : true;
sleep(5000);
}
I don't know what you're using but mind the logic, not the language.
What you can do is use Sql Server Integration Services (SSIS). It's basically a workflow package built into Sql that will let you handle tasks like this. You should be able to import the CSV into a temp table and then run your appropriate queries against it.
This is Azure specific but working against hosted SQL should be similar.
https://learn.microsoft.com/en-us/sql/integration-services/import-export-data/import-data-from-excel-to-sql
I have two different scripts, one created by me and one by my collegue, that is using the same snapshots.
16.00 (or 4 PM) Coded by me.
Script 1 - deletes snapshots if they are there, creates new snapshots - executes code.
04.00 (or 4 AM) Coded by Collegue
Script 2 - deletes snapshots if they are there, creates new snapshots - executes code.
Both of these scripts are SSIS scripts that are just holders for Stored Procedures (the SSIS scripts actually don't do much more than executes a bunch of Stored Procedures in a chain).
Script 2 works without problem.
Script 1 get's 'snapshot cannot be deleted, you don't have access or the snapshot is not present in the database'
If I run script 1 in SQL Studio it works perfectly so I have not spelled anything incorrectly.
Both scrips are running under the same user both in the SSIS engine and in the JOBS engine.
I don't even know where I should start looking for errors for this?? Any suggestions?
------------- Edit: Script added ----------------
IF EXISTS(select NULL from sys.databases where name='Citybase_Snapshot')
BEGIN
DROP DATABASE Citybase_Snapshot;
END
CREATE DATABASE CityBase_Snapshot ON
( NAME = FastighetsBok_Data, FILENAME = 'S:\Snapshots\Citybase_Snapshot.ss' )
AS SNAPSHOT OF Citybase;
---------------- Edit: Error message added ----------------------
As far as I know this is a normal error message from SQL server.
EXEC proc_..." failed with the following error: "Cannot drop the
database 'Citybase_Snapshot', because it does not exist or you do not
have permission.".
The answer was more simple than I imagined.
You set an active user on the SSIS script that's running when you create a job in the SQL server for the SSIS job, but that's not the only place you might have a security user on.
You also need to check the connection you actually establish inside the SSIS script to make sure that the user that you use to connect to the database is allowed to drop the snapshot.
Easy as pie (when you know what's wrong).
I have an SSIS package that takes in (through a package parameter) a value, passes it into a script task via a script variable (readonly variable), converts it to another value inside the script task, and finally writes that value out to another script variable (readwrite variable). There are no other SSIS modules in the package aside from the one script task.
What I would like to do, from outside the package (via SQL) is:
Call the SSIS package, passing in a value for my parameter and variable
Get the value of the read/write variable that is determined at the end of the script task execution
I've got step #1 working, just can't figure out #2.
How do I get the value of package variable in an SSIS package after it has executed? Is it accessible? Is it stored anywhere or can I store it somewhere in the SSIS catalog? I've tried to see if it's stored in the SSISDB.[catalog].executions table somewhere, but it doesn't seem to be.
Do I need to write that script variable to a package parameter in order to see it from SQL after execution? Could I then perhaps see it by using EXEC [SSISDB].[catalog].get_parameter_values, or does that only show parameter values before package execution? Am I going about this completely the wrong way?
Thanks in advance!
What I would do is add one last step to the package to write the value of the variable to a table.
Then you can retrieve the value from the table via SQL.
You can either truncate the table every time, or keep a permanent history associated with each time the package runs.
How do I create a package that extracts multiple databases(and all tables in each database) from another server based on a lookup table (also found on the other server) that contains a column where all the databases I need to extract is listed ?
I need to use the lookup table because new databases is created from time to time on the source, so I cannot just create a job that extracts a "static set" of databases to a destination. It needs to be a bit dynamic...
Furthermore I also need to extract the databases incremental where I can use a timestamp that exists in all databases/tables.
Im new to SSIS, so an "easy" guide would be appriciated.
Thanks
As a rough idea, you could work with SSIS Package Configurations and executing packages from within packages, and then use the Transfer SQL Server Objects Task:
Make a "Main package" that iterates over the column in your lookup table.
For each entry, it should UPDATE the Package Config entry of your second SSIS package accordingly. Use the "SQL Server" configuration for that second package.
The Main package should then execute the second package - there is a also a task for this.
The second package looks at its config to find out which server to get databases from and uses the Transfer SQL Server Objects Task to do so.
then the Main package continues with the next entry from your lookup table.
Ideally you would want to have your "second SSIS package" inside SQL Server's MSDB rather than the file system. Its easier to execute.
I Want To Convert Data From OldDatabase To NewDatabase Each Day By SSIS.
for this:
1- Create Script From Exists New DataBase
2- Change DatabaseName OF Generated Script and Then Execute Script To Create NewDatabase2 whit no data.
3- Set SSIS Configuration Parameters.
4- Execute SSIS Package to convert Data.
I Want to do this action by Job.
How I want to do?
your requiremets are not very clear but there are two SSIS tasks that can help you.
If you want to transfer all you can use a "Transfer Database Task"
If you need to specify objects, you can use the "Transfer SQL Server Objects Task" where you can specify things like if you are dropping the objects first and how to deal with the data: