I need to be able to save all the data that gets updated like so.
User inserts a car Model (Make, Type, Year). Comes back and Updates the Year. I need to be able to save both so they have a history of all the work that they did. What is the best way to do that?
There are a number of ways to do this. One way is to write some SQL triggers and do it entirely in the database. Have a look here for some clues:
Another way is to do the auditing within the Entity Framework code. There is a nuget package called AuditDbContext with the source on Codeplex.
You need to decide if you want to do the auditing in EF or in SQL. Obviously if you need to audit everything and you might sometimes access the database from different applications which don't use the same EF datalayer (e.g. different technologies, etc), then SQL triggers might well be the way to go.
Maybe (if you are facing the "history" issue more often) the CQRS pattern is of interest for you; a good primer, Microsoft on CQRS. There is a framework build on .NET for this pattern (I have not tried it yet): NCQRS.
If you really just want the requirement in your question fulfilled now and you are using SQL Server 2010 or later, then Change Tracking may be another option. I would prefer that to triggers (but in the end all such dark processing logging solutions introduce additional risk).
Related
I have been developing a network security application for several years now, as the lead developer at my company. It is a split-architecture design, where one component resides on the customer's network, and the other component in our own cloud. We have developed our own custom versioning system that keeps both sides synchronized at each patch (per customer), but until now it has only allowed incremental changes to be made, and rollbacks are not possible.
We'd like to move to a forkable git-like solution for our code, so that we can develop and test multiple features simultaneously, but the thing that's holding us back from that is our database. We use PostgreSQL (currently 9.3.12), and I've written a custom script to calculate the deltas between the "old" and "new" database structure, each time we "make a patch". It spits out a list of SQL commands necessary to update the "old" database structure to look like the "new", including tables, functions, sequences, triggers, you name it. It's very elegant and pretty much never fails anymore, even with complicated deltas.
However, I realize that in order to have a git-like solution for this (check-out, check-in, merge changes into test and production code, etc.) while also keeping database changes in sync with application code, we'll need to have something a lot more advanced than just "old" vs "new". Note that we don't need to modify database data for the most part, only table structure, which is altered in place on existing customer databases.
So my question is this: Any ideas for a git-like SQL version control system, which allows forking and merging, and can be easily kept in sync with application code changes? Our custom tool is already a bit more advanced than some open-source tools we've looked into (such as sqlt-diff), and tools like Red Gate are a bit out of our price range as a startup (not to mention that I haven't heard anybody mention forking in context with Red Gate). We're open to writing a custom tool, if that's what we need to do, but we're scratching our heads about where to start with something like that. We know how to calculate deltas, but we don't know how to manage all those things across different forks.
Free or open-source tools, frameworks we can adapt, or general guiding principles for building such tools are all appreciated!
One way of solving this problem is with migrations. A couple of lightweight tools, but there are many others:
http://sequel.jeremyevans.net/rdoc/files/doc/migration_rdoc.html
https://flywaydb.org/
Rather than calculating deltas between versions after the fact, migrations can be used to evolve the schema in a controlled way. You can create feature-specific migrations that can be tracked (and forked/merged) along with the rest of your code.
Depending on how fancy you want to get, you may need to extend the default naming/numbering schemes.
The problem
Ok, sorry that my question is somewhat abstract and subjective, but will try to make it as specific as possible. So, the situation I am in is simple - I am remaking a very old MS Access application on a new website using ASP.NET MVC. As currently the MVC site is using SQL Server 2008 (for many well known reasons) I need to find a way to migrate the tables AND the data, because the information in the old database will be used in the new application.
Alright, so far so good, however there are a few problems. The old application is written in a different language, meaning that I want to translate table names, field names, and all other names that are there to English. Furthermore, I will be making some changes on the models themselves (change the type of some fields, add additional fields to some tables, remove old unnecessary ones and more). So technically I'll be 'having my way' with everything.
Researched solutions
With those things in mind I researched for the ways to migrate data from Access database to a SQL Server. Of course, there is a lot of information on the matter, in Stack Overflow alone there are more than a few questions and solutions. So why am I struggling to find the answer ? Well I found a few solutions that will be sufficient to some extend (actually will definitely solve my problems) but I am writing to ask if someone experienced has a better perspective on it than I do. Alright, the solutions and why I am still looking for advice: /I'll be listing just a couple of the most common and popular ones that I found, many of the others share the same capabilities and/or results /
Upsize Wizzard (Access) - this is a tool devised specifically for migrating tables and data from Access. It is my most favourite one for the moment as I find it kind of straightforward to work with and it provides good overall results. I was able to migrate the tables to SQL Server (along with the data of course) which more or less is what I am intending to do. It is fast, it seems like it allows you to migrate indexes, primary keys and even to my knowledge foreign keys (table relationships). The downsides of this tool, however, include that it ignores your queries (which I don't really need honestly) and it doesn't provide a way to change the model, names or types of the properties of the table you migrate - which is the thing I kind of prefer, because I will have to make more than a few changes, adding, renaming, deleting, etc. And then continue with the development process (of the application) which will lead to a few additional minor changes. And finally I would need to apply all changes (migration + all changes) on the production server, which overall is prone to mistakes as I will be doing it by hand (and there are more than a few tables).
SQL Server Migration Assistant (SSMA) - ok, this is a separate tool (not included in Access) with again the same idea - to migrate data from Access to ... possibly everywhere, haven't researched that. Overall it offers more functionality and customizing from the Upsize Wizard, but of course it does it in a more complicated way. I haven't put enough effort to make a migration with this tool yet, as it involves a lot of installations and additional work, but according to my research it provides almost all (if not all) of the functionality I require. The downside however comes with the naming. As I mentioned it allows you to apply changes on the tables, schema, fields, indexes, keys and probably everything, but the articles advice that I change the names in Access first, as it will be easier and the migration process will run more smoothly. I am not allowed to make changes on the original Access database, as it will remain functional until the publish of the 'renewed' project, and the data inside it is being used, so a mere copy of the file is a solution I am not particularly fond of, because I might loose new records. Also I cant predict the changes I would want to make in the development process (as I said I believe I would want/need to apply some additional changes later on when I find 'weaknesses' in my data design in the development process) so I find it to be a little half baked solution.
Conclusion
The options presented, the way I see them, are two:
Use the Upsize Wizard to migrate the access tables, then write a script that applies the changes I want to make. Then in the development process add any additional changes to the script. When ready to publish on the production server, reapply the migration with the wizard, run the changes script and pray everything is fine.
Get more involved with the SSMA tool and try producing an updated version of the tables with the migration process. (See how efficient the renaming is and decide whether to use copied file to rename and then find a way to migrate only new records or do it all in the SSMA). Then again write a script for the changes that occur in the development process and re-do and apply it all on the production server when ready and then pray everything is fine.
Option I have not yet seen, apply it and then pray everything is fine.
I have researched the matter for a couple of days now, and found a few more solutions that I do not believe are better by the mentioned. However I include the possibility of missing the 'big red X on the map', a practical and easy solution which seems like it was designed specifically for me (though I doubt that a little). Anyway, reducing all the madness that I have written so far to a few simple questions will look like:
Is anyone aware if my conclusions are correct? I am leaning towards option one as it is easier to accomplish.
Has anyone experienced/found a better way to do that, or just found some 'logic-leaps' in my writings as I am overthinking the entire thing a little and may be doing some obvious miscalculation.
Very sorry for asking a trivial question and one that includes decision making that may involve deeper understanding of my project and situation, yet I am working with rather sensitive data and would appreciate feedback, even if only to improve my confidence into the chosen approach.
There is one other tool/method you might want to consider that seems to cater to your specific needs more. This would be to use the data import/export tool that ships with sqlserver to do a complete copy of all data into a temporary location within sql server and then write custom queries to reorganize the names and other changes you want to make. Is a bit more work but you could use the end product as a seed method for your migrations ;) (if you are doing code first anyway)
Is there any way to have data migrations on production database not to be with SQL?
We are using MigratorDotNet and when we build a new funcionality for the application that changes the scheme of the database and we need to do some data updates we have to do this complex and troublesome SQL statements so the data is consistent on production.
Was wondering if there was another way to do this, what are the bests practices to do this? Any ideas on other possible solutions?
We cannot use something like NHibernate because then we have to keep fixing old migrations when the scheme changes, and that can be error prone.
The trick is to use your migration tool and fold said data manipulation statements into the migrations. We use an extended version of the same thing typically for a few projects and it can definitely handle that trick.
If you're already using a migration tool like Migrator.NET then I'd say you're most of the way there. Complex schema/data changes are just a fact of life in the RDBMS world.
Try mite. It let's you do anything that you can do with sql and use sql to do it but have the ability to ensure your database is on the desired version and not risk executing a script that has already run (or miss a script), leaving your database in a consistent state.
If your developers adopt this. Deployments are a simple mite update and then you know problems are product related or data related (but not schema related).
https://github.com/soitgoes/mite
Let me know what you think. I developed this and have been using this with my team for years with great success.
The problem: we have one application that has a portion which is used by a very small subset of the total users, and that part of the application is running off of a separate database as well. In a perfect world, the schemas of the two databases would be synced up, but such is not the case. Some migrations have been run on the smaller database, most haven't; and furthermore, there is nothing such as revision number to be able to easily identify which have and which haven't. We would like to solve this quandary for future projects. During a discussion we've come up with the following possible plan of action, and I am wondering if anyone knows of any project which has already solved this problem:
What we would like to do is create an empty database from the schema of the large fully-migrated database, and then move all of the data from the smaller non-migrated database into that empty one. If it makes things easier, it can probably be assumed for the sake of this problem specifically that no migrations have ever removed anything, only added.
Else, if there are other known solutions, I'd like to hear them as well.
You could use a schema comparison tool like Red-Gate's SQL Compare. You can synchronize the changes and not lose any data. I wrote about this and many alternative tools ranging widely in price here:
http://bertrandaaron.wordpress.com/2012/04/20/re-blog-the-cost-of-reinventing-the-wheel/
The nice thing is that most tools have trial versions. So, you can try them our for 14 days (fully functional) and only buy it if it meets your expectations. I can't speak for the other tools, but I've been using RG for years and it is a very capable and reliable tool.
(Updated 2012-06-23 to help prevent link-rot.)
Red-Gate's SQL Compare as Aaron Bertrand mentions in his answer is a very good option. However, if you are not permitted to purchase something, an option is to try something like:
1) For each database, script out all the tables, constraints, indexes, views, procedures, etc.
2) run a DIFF, and go through all the differences and make sure that the small DB can accept them. If not implement any changes (including data) necessary onto the small DB so it can accept the changes.
3) create a new empty database from the schema of the large DB
4) import the data from the small DB into the nee DB.
You could also reverse engineer your database into Visual Studio as a database project. Visual Studio Team Suite Database Edition GDR R2 (I know long name) has the capability to do a schema comparison and data comparison, but the beauty of this approach is that you get all of your database into a nice database project where you can manage change and integrate with source control. This would allow you to build from a common source and deploy consistent changes.
Assume Hibernate for the ORM.
I'm not sure how to ask this. I want to build an application that can replace part of another. For example, say I have an application with various modules, called the "big" app. This application may handle HR, financial, purchases, skill sets, etc. But maybe, for whatever reason, I don't like the skill set module, but I like the rest of the application. I want to build an app that uses the same database that the rest of the "big" app uses but use my software as the front end for that piece.
I could build my app and have it hit the database directly with no ORM. My question is is there an advantage to using an ORM here. I'm thinking there is because if the "big" app goes away and another app is purchased, we could continue to use my version of skill set because I am using hibernate instead of hitting things directly. I'm still learning but I thought that my application used objects that I named and that in the case I just described I'd have to change my mapping files only or/and my code very little.
Here is another example. I have a legacy application and legacy database. It uses database X. I decide that I no longer like the old terminal emulator application that is used to get the data and that I want a graphical version. I can use hibernate with my application and when I finally decide to get rid of the legacy database and change to the latest Oracle or SQL Server, I can do so with minimal headache? Or is my database going to change so much that it wouldn't have matter anyway (I'm suggesting that upon changing to a new database more information will want to be captured)?
I was hoping for comments, if I am misunderstanding why hibernate/ORM might or might not be a benefit.
Thank you.
I do not think you will have a huge benefit frmo hibernate if the database schema changes to something completely different, you might have to change more than just your mapping - especially if more "structure" is added to the database (tables, column and such schema things). That said, if the database was structured mostly the same way, but lets say just the column names and tables names changes and a couple of tables are merged or something like that - you can get by with just changing your mapping.
But I would really recommend using hbernate for database agnosticity, that's is a pretty easy path.
AND then just because it doesn't exactly helps you if your entire database is changed, it such an incredible amount of other forces, that I would choose that over direct DB access most of the time.
Lastly you could think about using a service-layer such as the repository pattern that abstracs away the data access, so the business of your appilcation wouldn't need to change if the database changes.
Switching from one DBMS to another (ala Oracle to SQL Server) is one thing that using an ORM would certainly make much easier.
As for switching from one "big app" to another "big app", I doubt if using an ORM would help that much. It's likely that the database structure and business logic would be different enough that you would find yourself rewriting lots of code anyways.
You can generate domain objects with Hibernate Tools, if you do that than it will be painless and fast. however if you write all the objects by hand you will die. i think its good idea to rewrite part of the app and get to know hibernate better.
I think it's generally a bad idea to make any decision based on the
unknowns versus the knowns. Whether you're deciding on a data
access/persistence strategy, what car to buy, or what college to go
to, you should put the most weight on the things you know you want
today, rather than worrying about what may or may not happen tomorrow.
So when considering ORMs, I wouldn't worry too much about things such as apps
"going away" or DBMSs changing (unless that's either already been talked about, or
there's a history of this in your company). I'm not saying that these aren't things that will never happen, but rather that they should take a back seat to the generally much more important considerations of maintainability, performance, and developer productivity.
So in short, choose an ORM based on its ability to solve the problems and satisfy the requirements that you have today.