We have an old CRM running on a SQL 2005 database that I have been tasked with linking to the corporate website. At the moment the data in it is managed internally by staff. The problem is that the database is so poorly design in terms of how it stores relational data that using it with something like the entity framework makes it an unenviable task.
So with this in mind I planned to extract the information from the old 2005 database and transforming it into relational tables in SQL 2012 that work with the entity framework and use this to power the website. I’d set this up as a SQL agent job that runs once a day as there is quite a lot of data to be synchronized.
I have also in my business logic for the website written some code to handle inserts, updates and deletes in a transaction so that if the user updates their details on the website, it will update the SQL 2012 data table and, via an API update the 2005 database to have some sort of data concurrency. I have written a test application and this works satisfactorily.
The problem now comes when internal staff update the data in the CRM as using this system it will take up to 24 hours before the data will be updated on the website. So far I have found SQL Data Compare 10 and dbForce Data Compare software which can sync the changes across databases, but this will need to be done manually.
Is there an automated way to update the record in the SQL 2012 table when it is changed in the SQL 2005 database?
Is it possible to use database first for the entity framework and then alter the structure of the models it creates to it makes more sense, but it still work with the legacy database?
Thanks in advance
Related
Basically I need something to generate SQL Server change scripts for data differences only, based on the data differences between two tables with the same schema.
We will have a table with approx 250,000-330,000 rows and 10-12 columns, and two instances of the table:
The Master table, populated with records from the production system.
the Sandpit table, also populated from the production system - but in which the user can add / remove rows and edit cell contents.
Once the user is happy with their edits they need to generate a change script which makes the necessary changes to an instance of the Master table in a variety of servers (test, pre-prod, prod), so it needs to be reliable. It's safe to assume that all versions of the master data will be the same when the script is eventually run.
They also need to be able to re-run the change script for self-testing (restoring the master back to it's original state would be a separate process, out of scope for this question).
Design of the table schema is not yet done, and can be tailored to suit this purpose.
SQL Server 2008 Standard edition, upgrade likely (but still standard edition).
I understand RedGate is pretty much the industry standard / leading choice for generating SQL change scripts but their website focuses a lot on managing schema changes so not sure if it's appropriate to use here. I'm familiar with using SQL Server myself but it's been a few years - not sure if the inbuilt functionality is up to it or not (both being technically capable + user friendly enough). The end user will be a competent SQL user but comes from the business side not IT (not SQL Admin grade).
You should be able to do this from Visual Studio using SSDT (SQL Server Data Tools). You need to do a data comparison between the source and target tables, which will then generate a change script.
We have and old database with a poorly thought out table structure, virtually no relationships setup and no naming schemes. I've created a new database with a clean relational data structure that implements proper design practices.
I'm looking for advice on different methods to migrate the old data over to the new format. This will require a lot of data re-shaping which won't be fun. The data is heavily accessed and the challenge will be to keep both databases in sync for all relevant data (accounts, important services etc).
I thought triggers might be the way to go here - but maybe there is a different method that I am unaware of (maybe MS Sync Framework, or a code-level data adapter which will be more work because there is so much data access code spread all over the place, classic ASP and .Net over dozens of projects). The database in question is SQL Server 2005, running in SQL Server 2000 compatibility mode.
I think the way to go is to write a stored procedure in the new database, which will actually pull your delta changes (only the modifications that were done from the last run to the instant the stored proc is run), and put this stored procedure in the sql agent job.
Configure the sql agent job to run for every 15 minutes and let the data sync in.
disadvantages of using triggers in this scenario
triggers will reduce the performance, as the sql server will execute the trigger code as well along with the update/ insert /delete statements and includes these as part of the execution at every time, i.e. if your trigger code takes 2 seconds to execute and the update statement with no trigger takes 2 seconds to execute, then the update time will be increased to 4 seconds with trigger in place. So employing triggers in this case might result in huge performance bottle neck.
I'm dealing with the same situation at my work, and I'm currently writing an application to do the migration. The original database has no established relationships, so it's really like a set of disconnected spreadsheets. By building my own application, I'm able to migrate the data using newly-established foreign keys, and assign data-specific defaults in place of nulls.
I have an application which has two database servers: one is the development database server and the other one is a production database server. Accidentaly, a couple of months back I published the application for use and it was pointing towards the development database.
Since it was pointing to the development database, all the information inserted or restored went in to the development database.
I want to transfer the data which went into the development database from july to september to the production database.
Is there a way to do this?
Create an SSIS package that uses the development database as the source and the production database as the destination and run a query to pull the data from development into production.
By the way, we have had this happen to us once.
The big issue here is to identify what changed. If you have datetime stamps in all your tables, this should be fairly easy.
If you do not have date time stamps, you will have to compare data. So let's call your database ConsumerData. If you have a backup of the DEV environment from the day before the production instance of the app pointed to it, you can restore it as ConsumerDataNew.
There are multiple database comparison utilities (Redgate SQL Compare) that will let you compare tables across the databases (ConsumerData and ConsumerDataNew) and give out results as SQL statements that you can then run in Production.
You will, of course, have to identify which records in Dev were made by development ppl vs. which ones were made by the Production Application.
You can use DTSWizard to transfer data between two database. Write a SQL query to pull only the data you need.
Edit: I didn't realize it had fancy new name, SSIS.
Here's a good tutotrial for SQL Server 2005 Import / Export Wizard.
Red-Gate has a tool called SQL Compare to compare the schema and SQL Data compare to actually compare the data.
This might be helpful.
I'm working on a legacy project, written for the most part in Delphi 5 before it was upgraded to Delphi 2007. A lot has changed after this upgrade, except the database that's underneath. It still uses MS-Access for data storage.
Now we want to support SQL Server as an alternate database. Still just for single-user situations, although multi-user support will be a feature for the future. And although there won't be many migration problems (see below) when it needs to use a different database, keeping two database structures synchronized is a bit of a problem.
If I would create an SQL script to generate the SQL Server database then I would need a second script to keep the Access database up-to-date too. They don't speak the same dialect. (At least, not for our purposes.) So I need a way to maintain the database structure in a simple way, making sure it can generate both a valid SQL Server database as an Access database. I could write my own tool where I store the database structure inside an XML file, which combined with some smart code and ADOX would generate both database types.
But isn't there already a good tool that can do this?
Note: the application also uses ADO and all queries are just simple select statements. Although it has 50+ tables, there's one root "Document" table and the user selects one of the "documents" in this table. It then collects all records from all tables that are related to this document record and stores them in an in-memory structure. When the user saves the data, it just writes the document record and all changed data back to the database again. Basically, this read/write mechanism of documents is the only database interaction in the whole application. So using a different database is not a big problem.
We will drop the MS-Access database in the future but for now we have 4000 customers using this application. We first need to make sure the whole thing works with SQL Server and we need to continue to maintain the current code. As a result, we will have to support both databases for at least a year.
Take a look at the DB Explorer, there is a trial download too.
OR
Use migration wizard from MS Access to SQL Server
After development in Access (schema changes), use the wizard again.
Use a tool to compare SQL Server schemata.
I am writing code to migrate data from our live Access database to a new Sql Server database which has a different schema with a reorganized structure. This Sql Server database will be used with a new version of our application in development.
I've been writing migrating code in C# that calls Sql Server and Access and transforms the data as required. I migrated for the first time a table which has entries related to new entries of another table that I have not updated recently, and that caused an error because the record in the corresponding table in SQL Server could not be found
So, my SqlServer productions table has data only up to 1/14/09, and I'm continuing to migrate more tables from Access. So I want to write an update method that can figure out what the new stuff is in Access that hasn't been reflected in Sql Server.
My current idea is to write a query on the SQL side which does SELECT Max(RunDate) FROM ProductionRuns, to give me the latest date in that field in the table. On the Access side, I would write a query that does SELECT * FROM ProductionRuns WHERE RunDate > ?, where the parameter is that max date found in SQL Server, and perform my translation step in code, and then insert the new data in Sql Server.
What I'm wondering is, do I have the syntax right for getting the latest date in that Sql Server table? And is there a better way to do this kind of migration of a live database?
Edit: What I've done is make a copy of the current live database. Which I can then migrate without worrying about changes, then use that to test during development, and then I can migrate the latest data whenever the new database and application go live.
I personally would divide the process into two steps.
I would create an exact copy of Access DB in SQLServer and copy all the data
Copy the data from this temporary SQLServer DB to your destination database
In that way you can write set of SQL code to accomplish second step task
Alternatively use SSIS
Generally when you convert data to a new database that will take it's place in porduction, you shut out all users of the database for a period of time, run the migration and turn on the new database. This ensures no changes to the data are made while doing the conversion. Of course I never would have done this using c# either. Data migration is a database task and should have been done in SSIS (or DTS if you have an older version of SQL Server).
If the databse you are converting to is just in development, I would create a backup of the Access database and load the data from there to test the data loading process and to get the data in so you can do the application development. Then when it is time to do the real load, you just close down the real database to users and use it to load from. If you are trying to keep both in synch wile you develop, well I wouldn't do that but if you must, make a nightly backup of the file and load first thing in the morning using your process.
You may want to look at investing in a tool like SQL Data Compare.
I believe it has support for access databases too, and you can download a trial.
I you are happy with you C# code, but it fails because of the constraints in your destination database you temporarily can disable them and then enable after you copy the whole lot.
I am assuming that your destination database is brand new DB with no data, and not used by anyone when the transfer happens
It sounds like you have two problems:
You're migrating data from one database to another.
You're changing your schema.
Doing either of these things is tricky if you are trying to migrate the data while people are using the data.
The simplest approach is to migrate the data based on a static copy of the data, and also to queue updates to that data from the moment you captured the static copy. I don't know how easy this is in Access, but in SQLServer or Oracle you can use the redo logs for this or a manual solution using triggers. The poor-man's way of doing this is to make triggers for all the relevant tables that log the primary key of the records that have changed. Then after the old database is shut off you can iterate over those keys and get those records from the old database and put them into the new database. Just copy the whole record; if the record was deleted then delete it from the new database.
Your problem is compounded by the fact that you can't simply copy the data, you have to transform it. This means you probably have to shut down both databases and re-migrate the records based on the change list. It will take a lot of planning to ensure you get things right and I'd recommend writing a testing script that can validate that the resulting data is correct.
Also I'd ensure that the code for the migration runs inside one of the databases if possible. Otherwise you are copying the data twice and this will significantly harm the performance.