I am new to SSIS packages and just require assistance on how to transfer data from one data source onto my own database.
Below is my data flow:
Now I have a ODBC Source (Http_Requests Source) where I take data from a PostgreSQL database table (see screenshot below for table columns and data):
Below is the OLE DB destination where it has the table I want to transfer the data to (this table is currently blank):
Now I tried to start debugging to extract the data but I get a few errors (displayed below):
I am a complete novice so I would like some guidance on what I need to include in order to get this SSIS package to transfer data across. Would I need to include a merge statement and how do I apply it. I heard you can write a merge as a proc and call on the proc as a sql command. Does that mean I will need to write a proc in SSMS and then call on it within the OLE DB Destination?
If somebody can provide an example and screenshot then that would be very helpful as I am really new to SSIS.
Thank you,
Check constraint on destination table or disable them before running it.
Below are query you can use.
-- Disable all table constraints
ALTER TABLE YourTableName NOCHECK CONSTRAINT ALL
-- Enable all table constraints
ALTER TABLE YourTableName CHECK CONSTRAINT ALL
Tick keep identity
box or drop primary key on the table. After you apply the changes do not forget to refresh metadata by opening the mappings in sis.
the error means that PerformanceId is an IDENTITY column on your destination table. IDENTITY columns are read only unless you tell it otherwise. So if we were in tSQL to be able to insert IDENTITY we would turn on IDENTITY_INSERT. Because you are in SSIS you can accomplish the same thing by checking the "keep identity" box.
HOWEVER when ever you get an error like this it is usually a sign that you should NOT be mapping ID to Performance ID. The question you have to ask is the Identity from your source supposed to be the identity of the destination table? Usually not, most of the time it would be another column as a surrogate key. Then you have to understand if it is even possible. because if there is a unique constraint or primary key then the identity cannot repeat which means you have to know that your source's id column will not cause a duplicate primary key violation.
More than likely the actual fix if for you to uncheck ID from the source and ignore the value.
The column PerformanceID (in the target) is almost certainly an identity column and that is why it is not working. You may not want to transfer it (and have SQL Server generate values for PerformanceID or you can check 'Keep Identity.'
Related
The database that I am tasked with fixing has a table with an identity column/PK that has a datatype of BigInt. This causes problem with the Access front end in that a datasheet to this linked table will not allow edits to the records. (This is a known issue with ODBC drivers and Access)
The table's Id column should never have been created as a bigint in the first place but that is a moot point now. I need to convert or recreate this column with a datatype of int, without losing the existing data.
There are ~2 million records in this table.
There are an unknown number of apps and Access apps that access this table so I am trying to do this as smoothly/stealthily as possible since the likelihood of finding all of those apps and modifying them before I make the change is slim.
Any thoughts or ideas?
I'm assuming the IDENTITY column is your PRIMARY KEY, and it's probably clustered :) MY advice below is based on those assumptions.
If you've only got a few indexes on the table, and the PRIMARY KEY is only referenced by a few FOREIGN keys, you should be able to change the datatype by:
Dropping any nonclustered indexes which contain the IDENTITY value.
Dropping the FOREIGN KEY constraints which point to the PRIMARY KEY.
Drop the PRIMARY KEY
ALTER TABLE tablename ALTER COLUMN columnname INT;
REcreate the PRIMARY KEY
Re-enable the FOREIGN KEY constraints with CHECK.
Recreate your nonclustered indexes.
As RBarryYoung pointed out, a lot of this can be scripted out by the SSMS GUI (if it's configured to allow saving changes), but the difference is that the GUI will create a temporary table, move your data, rename the new table to the old name, and drop the original.
Here's how to do it from Management Studio(SSMS):
First, make a backup copy of your database. If you make a mistake, or something unexpected happens, the easiest way to fix it is to restore from backup.
In the SSMS Explorer Pane, navigate to the table, then right-click on it and click "Design".
Select the Identity column's row and change it's datatype to "INT".
Save your changes, ignore the warning.
If you need a script instead, then replace step (4) above with:
Click the Script Changes button. Ignore the warning and then copy the script into you paste buffer. Make a new query window and paste the script into it. Then close the design window, cancelling any changes.
As Stuart Ainsworth points out, in later versions of SQL Server, it may prevent you from doing this, with a warning about "Dropping a Table". To fix this in SSMS, click the Tools..Options menu entry, then go to the "Tables and Designers" pane under "Designers" and uncheck the "Prevent saving changes that require table recreation" option.
I have an issue related to data in sql server. In my database some of the constraint were not enabled i.e. they were not checked , After some time working on it we found this issue that a parent rows can be deleted without deleting child, which was an issue. I enabled all the constraint in the database using query
ALTER TABLE tbl_name CHECK CONSTRAINT ALL
above query was executed on all the tables of that database without any error . But my concern is whether it will work or not , if it will work on the existing data then what will happen to that data whose parent table data has been deleted.
I want to know is there any way such that I can validate such data data whose parent record doesn't exist in the entire database. There are about 270 constraint containing FOREIGN KEY AND UNIQUE KEY . I don't want to go for manual option.
Please help me out.
ALTER TABLE tbl_name CHECK CONSTRAINT ALL
only re-enables the constraints. Importantly, the constraints are not checked against the existing data in the database (nor are they trusted by the optimizer). If you want that to occur, you need to specify WITH CHECK as well:
ALTER TABLE tbl_name WITH CHECK CHECK CONSTRAINT ALL
(And yes, the word CHECK appears twice)
If you execute this, and there are orphaned child rows (or other invalid constraints), then the ALTER TABLE will fail with an error message. There's nothing SQL Server can do to fix this issue - it's for you to decide whether to a) remove the orphaned rows, or b) to re-create, in some manner, a suitable parent row for them.
You can also add the 'ON DELETE CASCADE' code to the end of foreign keys to prevent orphaned child rows from persisting.
This is more of a 'better practice' going forward than a solution, but I believe Damien_The_Unbeliever has answered your main question.
sometimes i face the following case in my database design,, i wanna to know what is the best practice to handle this case:::
for example i have a specific table and after a while ,, when the database in operation and some real data are already entered.. i need to add some required fields (that supposed not to accept null)..
what is the best practice in this situation..
make the field accept null as (some data already entered in the table ,, and scarify the important constraint )and try to force the user to enter this field through some validation in the code..
truncate all the entered data and reentered them again (tedious work)..
any other suggestions about this issue...
It depends on requirements. If the data to populate existing rows for the new column isn't available immediately then I would generally prefer to create a new table and just populate new rows when the data exists. If and when you have all the data for every row then put the new column into the original table.
If possible i would set a default value for the new column.
e.g. For Varchar
alter table table_name
add column_name varchar(10) not null
constraint column_name_default default ('Test')
After you have updated you could then drop the default
alter table table_name
drop constraint column_name_default
A lot will come down to your requirements.
It depends on your application, your database scheme, your entities.
The best way to go about it is to truncate the data and re - enter it again, but it need not be too tedious an item. Temporary tables and table variables could assist a great deal with this issue. A simple procedure comes to mind to go about it:
In SQL Server Management Studio, Right - click on the table you wish to modify and select Script Table As > CREATE To > New Query Editor Window.
Add a # in front of the table name in the CREATE statement.
Move all records into the temporary table, using something to the effect of:
INSERT INTO #temp SELECT * FROM original
Then run the script to keep all your records into the temporary table.
Truncate your original table, and make any changes necessary.
Right - click on the table and select Script Table As > INSERT To > Clipboard, paste it into your query editor window and modify it to read records from the temporary table, using INSERT .. SELECT.
That's it. Admittedly not quite straightforward, but a well - kept database is almost always worth a slight hassle.
I have FK and PK all over my db and table data needs to be specified in a certain order or else I get FK/PK insertion errors. I'm tired of executing the wizard again and again to transfer data one table at a time.
In the SQL Server export data wizard there is an option to "Write a query to specify the data to transfer". I'd like to write the query myself and specify the correct order.
Will this solve my problem?
How do I do this? Can you provide a sample query (or link to one)
The databases are on two different servers - SQL Server 2008 on each ; The database names & permissions are the same ; each table name & col is the same ; I need Identity Insert for each table.
Disable foreign keys before importing, enable them after the import:
ALTER TABLE tablename NOCHECK CONSTRAINT ALL
ALTER TABLE tablename WITH CHECK CHECK CONSTRAINT ALL
Update: Thanks for the comments, I fixed the syntax.
You could always save the package and then open and edit the package to put things in the right order (you might have to copy the data flow several times and put dependencies between them)
You can use 3rd party tools to transfer a data; these tools disable/enable constraints automatically.
I wanted to modify a column in a sql server 2005 table to IDENTITY(1,1)
Incidentally this table is empty and the column to be changed is a primary key.
This column is also a foreign key for two other tables.
After googling I found that you cannot use Alter table syntax to modify a column and make it an indentity column.
Link #1 : How do I add the identity property to an existing column in SQL Server
Link #2 : Adding an identity to an existing column -SQL Server
I ended up checking the dependent tables (2 of them) removing the foreign keys (generated the script from SSMS) then dropping the main table then re-creating with identity. (could try the rename option here as well)
Then re-created the foreign keys for the earlier dependent two tables.
But all this was manual work, any scripts or SPs out there to make this easier.
Ideally all these steps would be done by such a script/tool/utility:
Check dependent tables keys
Generate Create and drop foreign key scripts for this
Generate create script for the main table
drop the main table (or rename the table if the table has data)
re-create the table with identity column enabled
re-create foreign keys
You can use SSMS to generate a script (Edit a table, save script), but otherwise it's a manual process as you identified.
The SSMS scripts will pick up dependencies etc. For this kind of work, I tend to use SSMS to generate a basic script, pimp it a bit, run it carefully, then use a comparison tool (such as Red Gate compare) to generate a safer version.
Edit: The SSMS error is not an error, it's a safety check that can be switched off
(This is merely a follow-up to gbn's post with more details -- it isn't all that easy to figure this stuff out.)(
It isn't impossible to write a utility to do this, just very complex and very hard. Fortunately, Microsoft has already done it -- its called SSMS (or SMO?). To generate such a script:
In the Object Explorer, drill down to the database and table that you want to modify
Right click and select Design
Make the desired changes to the one table in the design screen. It's reasonably intuitive.
To add/remove the identity property, select the column in the upper pane, and in the lower pane/"Column Properties" tab, expand and configure the settings under "Identity Specification".
To generate a script to implement all your changes, incorporating all the dependent key changes, click on the "Generate Change Script" toolbar button. This is also an option under the "Table Designer" menu.
I also do this to generate scripts (that I later modify--SSMS doesn't always produce the most efficient code.) Once done, you can exit out without saving your changes -- leaving you a DB you can test your new script on.
drop the pk and build the same datatype column
copy the data of the column which you want to set identity to the new column.
drop the old column
reset primary key
ALTER TABLE UserRole
DROP CONSTRAINT PK_XX
ALTER TABLE XX
ADD newX int not null identity(1,1) primary key
update XX set newX = oldX
alter table XX
DROP COLUMN oldX
this is the simplest way to set identity column.
if you don't want to use the long generated script.