I need to transfer some data from 3 spreadsheets (Which I got from another DB) in to DB tables. The three spreadsheets contain feedback data, customer data and person data. When moving I need to map foreign key relationships between the data. Is there any way I can achieve this without the use of SSIS?
use openrowset
We can transfer the data using OPENROWSET. But I dont think we can mark FK relation ship while transferring the data. That you have to do after transfer.
See MSDN link here
OPENROWSET
Related
I have about 100 tables to which we replicate data, e.g. from the Oracle database.
I would like to quickly check that the data replicated to the tables in db2 is the same as in the source system.
Does anyone have a way to do this? I can create 100 transformations, but that's monotonous and time consuming. I would prefer to process this in a loop.
I thought I would keep the queries in a table and reach into it for records.
I read the data from Table input (sql_db2, sql_source, table_name) and write do copy rows to result. Next I read single record and I read a single record and put it into a loop.
But here came a problem because I don't know how to dynamically compare the data for the tables. Each table has different columns and here I have a problem.
I don't know if this is also possible?
You can inject metadata (in this case your metadata would be the column and table names) to a lot of steps in Pentaho, you create a transformation to collect the metadata to inject to another transformation that has only the steps and some basic information, but the bulk of the information of the columns affected by the different steps is in the transformation injecting the metadata.
Check Pentaho official documentation about Metadata Injection (MDI) and the sample with a basic example of metadata injection available in your PDI installation.
I want to copy data from a table in our test server to a table in our production server. The table has 50 million rows in it. One of the ways I was planning on doing it was to have two tasks:
Use the data flow task to export the data from the source table to a CSV file.
Use the Bulk Insert flow task to read from the CSV file and insert it into the destination table.
Is there a better way to do this in SSIS? Thanks!
There are many approaches to transfer data between two servers. SSIS is not always the preferred one. Noting that 50 million rows are not always considered a large data set; It depends on the server resources, columns data types, and other factors.
The simplest way to import/ export data is to use the SSMS Import/Export wizard. Another approach is to use BCP as #Nick.McDermaid mentioned in the comments.
If you have limited physical resources, and you need to do this using SSIS, you can try loading data in batches as explained in the following article:
SQL OFFSET FETCH Feature: Loading Large Volumes of Data Using Limited Resources with SSIS
How can I sync two databases and do a manual refresh on the entities on either of the database whenever I want?
Let's say I have two databases DB1(prod) and DB2(dev). I want to update/insert only a few tables from prod DB to dev DB. How could I achieve this? Is this possible instead of DBlink since I do not have privileges to create a database link?
If you only want to do a manual refresh set up an import/export/datapump script to copy the data across if there is not too much data involved. If there is a large amount of data you could write some pl/sql as described above to only move the new/changed rows. This will be easier if your data has fields such as created/updated_on
I am new to SSIS.I got the task have according to the scenario as explained.
Scenario:
I have two databases A and B on different machines and have around 25 tables and 20 columns with relationships and dependencies. My task is to create a database C with selected no of tables and in each table I don't require all the columns but selected some. Conditions to be met are that the relationships should be intact and created automatically in new database.
What I have done:
I have created a package using the transfer SQL Server object task to transfer the tables and relationships.
then I have manually edited the columns that are not required
and then I transferred the data using the data source and destination
My question is: can I achieve all these things in one package? Also after I have transferred the data how can I schedule the package to just transfer the recently inserted rows in the database to the new database?
Please help me
thanks in advance
You can schedule the package by using a SQL Server Agent Job - one of the options for a job step is run SSIS package.
With regard to transferring new rows, I would either:
Track your current "position" in another table, assumes you have either an ascending key or a time stamp column - load the current position into an SSIS variable, use this variable in the WHERE statement of your data source queries.
Transfer all data across into "dump" copies of each table (no relationships/keys etc required just the same schema) & use a T-SQL MERGE statement to load new rows in, then truncate "dump" tables.
Hope this makes sense - its a bit difficult to get across in writing.
I have a 5 database with same schema, i want to copy all data in one database with same schema
or how can i copy data from *.mdf files in database.
i am using sql server 2005
Copy Database with T-SQL:
sqlauthority
http://blog.sqlauthority.com/2009/07/29/sql-server-2008-copy-database-with-data-generate-t-sql-for-inserting-data-from-one-table-to-another-table/
Copy Database with Wizard:
kodyaz
http://www.kodyaz.com/sql-server-tools/sql-server-copy-database-wizard.aspx
I'd suggest taking a look at Red Gate SQL Data Compare. That will enable you to merge the data between the two databases and directly control which one wins in any given situation.
As mentioned above you need to deal with the Primary Keys as well...
One way to deal with this to add a "Database ID" to all the tables in the single central version. The central PKs become the PK from the source table, plus the "Database ID". This way you have unique PKs in the central version AND you can tell which database the row came from. This is what sql-hub does - there is a free licence which will let you do this as a one-off task - or you could do the inserts for each database and table in SQL.