I understand the reason behind not auto refreshing your .dbml (Linq2Sql), .edmx (Linq2Entities) or .hbm.xml (NHibernate). In each of these ORM solutions you need to update these files if you have changes in the database. Is there a way to refresh these files automatically if you are 100% sure that it wont break anything?
I am reasonably familiar with Migrations in RoR. Is there a dependabale Migration tool in the .NET world ?
In general, how do you stay agile with database design and dont recreate these mapping files manually again and again?
Please provide a link if this question is a duplicate. I was not able to find one.
I'm not sure what your hesitation about updating the EDMX is. Just go ahead and update your model from the database. Chances are very good it will work. But if it doesn't, just get the old version from source control, and make the changes manually.
With LINQ to SQL, on the other hand, it's harder, because there is no update feature. You have to remove the changed objects from your model, re-add them, and re-apply any changes you have made, manually. Again, source control is vital here; you'll want to diff the XML files from the previous version to make sure you didn't forget anything.
I'm not sure what your question about migrations means. Yes, there is a tool (RikMigrations), which does a similar thing as migrations in Rails. But migrations, of course, change your database metadata. They don't change your ORM mapping. That's done elsewhere, both in Rails and in .NET.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I am trying to develop a web application that, like most of applications do, uses a database. To create the database I write a .sql-file that contains all the changes I make to the database. I don't know why exactly but in the past I always found it hard to empty my database or modify it later on when I find that a change would make sense. Since I am still learning all this database related stuff the first layout of my database will always be changed. To keep track of all those changes I grew the habit to create a this .sql file.
In this file I am always dropping all the tables that are already existing and creating all the tables new. I kind of do this to have a reference of the actual state of my database always on hand. Changes to a file are way easier for me than directly using the command line database tool. The first question would really be: Is this actually a good practice or is there another way of organizing things that I didn't hear of yet?
The real question is: Where do I store this file? Is it good practice to store it in the same git-repository as the actual code? Should I put it in git at all? I also think of git/github as a cloud storage, if my harddrive burns, all my projects will still be there since I got them on github. If I don't have the .sql file in there I would have to set up the database from new.
The general category for this sort of code is "database migrations." There are tools specialized to these tasks and various web application frameworks may support various different DB migrations tools or they may have their own features.
Probably the most popular tool/suite in this category for Python (models implemented in SQLAlchemy) would be Alembic. Some of the other options include Flyway, Liquibase, and Sqitch.
In all of these cases you manage some abstraction of your database schema (SQL, XML, YAML) and these tools generate the necessary SQL and other code to perform "migrations" from each version of the schema to the next throughout the history of your project. Usually these are generated as increments. They start with initial database schema (possibly completely empty), build and initialize the schema into that version; then build and migrate through each step to arrive that the desired version.
This can become arbitrarily complex if you're making more radical changes to your schema. Adding an extra column to a table without a "NOT NULL" constraint is trivial. Adding new M:N relations through "NOT NULL" junction tables and migrating from some denormalized schema to a higher normal form, for example, can entail intermediary stages where you might need to drop some constraints and tolerate referential integrity violations through some transitional state.
I highly recommend reading these websites, their tutorials, HOWTOs and other documentation to gain a deeper understanding of why these tools exist and how they approach this problem space.
Yu are reinventing the wheel, my friend, your solution is versioning the DB Schema. And Yes, the changes should be added to the project files as almost all framework do. I recommend that you read the following question
How do you version your database schema?
Yes you should absolutely keep this code in source control. However, the exact location of the code within the repository is up to you and or your development team or management. Some good options would be an install, setup, SQL, or ddl folder.
Yes, what you are doing is correct.
Compare with ruby on rails: a file db/schema.rb contains the whole schema. It is good too have such a complete file. This allows you, for example, to easily bootstrap a new environment (e.g. a new testing environment). This file is obviously never used in production as it would wipe out all your data.
Then there are separate small files db/migrations/20171003_add_name_to_person_table.rb or whatever with incremental changes to the schema - called migrations. Those are used to change existing environments without losing data, with some mechanism to make sure each one is only run once per DB.
At your stage it is perfectly fine to be doing all of this manually. You can try to automate it as needed, later. It is good enough that you noticed that something is going on here.
That stuff must go into your code repository, wherever seems natural. /db, /schema, /etc might be some choices.
I recently started using LINQ to SQL in my database later for a C# Windows Forms project. Until now, I have been very impressed with how fast I have been able to implement the data access layer. The problem that I am facing is similar to the post from 2008 below
Best way to update LINQ to SQL classes after database schema change
In short, I am struggling to find an efficient solution for updating the LINQ to SQL files after making minor changes to the database such as constraints, foreign keys, new columns, etc...
Thus far, I have merely been deleting the tables in the LINQ to SQL designer and dragging them back onto the designer. However, I now have the need to rename many of the associations in the designer. The problem is that each time I have to re-create the LINQ to SQL files I lose the change that I manually made to the files. Can someone tell me if there are any new solutions and/or methods for solving this problem. The post that I have included as well as many other dated sources of information mentions that SQLMetal and Huagati are good tools. Additionally, I have read that you can manually create your LINQ to SQL files rather than auto-generate them with the designer (this is what I had to do when using Hibernate with Java).
I know that manually creating the domain classes and mapping files will be consuming. I am not familiar with SQLMetal or Huagati. Can anyone recommend the most elegant or preferred way to deal with this issue? I know that I could use Entity Framework but, I have inherited this project and I am under a very tight deadline. I can refactor it to another Framework once I have this phase complete.
After much research and reading, I have determined that the best solution for updating my DBML after minor database changes is to manually edit the file. The procedure used to update the DBML is below:
Right-click on the DBML file
Open with XML Editor
Add or change the columns in the affected table
Add or change any associations
Save the DBML
Rebuild the project
This is not ideal but, once it has been done a few times it is pretty painless for the types of changes that I occasionally need to make to the database such as changing data types, adding keys, etc...
I don't touch dbml or linq2sql generated files because there is risk that my changes would be overwritten. I use only my generated partial classes. When database schema changes I remove old table from dbml-editor and pull new table to it.
I'm trying to get into using Fluent NHibernate, and I have a couple questions. I'm finding the documentation to be lacking.
I understand that Fluent NHibernate / NHibernate allows you to auto-generate a database schema. Do people usually only do this for Test/Dev databases? Or is that OK to do for a production database? If it's ok for production, how do you make sure that you're not blowing away production data every time you run your app?
Once the database schema is already created, and you have production data, when new tables/columns/etc. need to be added to the Test and/or Production database, do people allow NHibernate to do this, or should this be done manually?
Is there any REALLY GOOD documentation on Fluent NHibernate? (Please don't point me to the wiki because in following along with the "Your first project" code building it myself, I was getting run-time errors because they forget to tell you to add a reference. Not cool.)
Thanks,
Andy
I've been using Fluent NHibernate Automapping for a few months now. I'm by no means an expert, but can take a stab at your questions...
FNH Automapping does indeed create DB schemas from POCO classes, including lists of other objects (this was the reason I chose NHibernate in the first place).
When you change schemas, you have to rerun the automapping, which does drop the whole database, unfortunately. In my case, it's not a big problem because I'm importing existing binary data files, so I just have to re-import my data every time the schema changes. I've read that there's some data migration support available with NHibernate, but have no experience with this. (BTW, Subsonic will do data migration, but it's automapping functionality is far more rudimentary - at least it was when I evaluated it a few months ago)
FNH documentation is one of my pet peeves - they have not even added Intellisense help on the method names, etc. (But they get really huffy when you point that out - ask me how I know!) I've made a couple of edits to the wiki when I could, but there's so much more that could be done there. The best approach is to start with a working example (i.e.
this one from Nikola Malovic, and post questions to the support form if (when!) you run into trouble. In general, I've found the FNH community pretty helpful, and have been able to work through all my difficulties. They've also fixed a couple of bugs I've found.
Overall, using FNH has been a huge win for my project - highly recommended!
I don't use Fluent, but I can help with classic NHibernate.
yes, the creation of the schema is very recommendable for production use (Schema Export). When you do this is up to you. For instance, you could create the database by an installer. You shouldn't drop existing databases, but this is a decision of you application.
I don't understand this question. Do you mean you need to upgrade an existing database to a new database schema? This is unfortunately something you need to implement yourself. NH can't do much about this, because it is very specific to you data and the changes you made. There is also a Schema Update or something like this, which is not recommended for production use.
I don't use Fluent, so I can't help here.
There is a development_structure.sql inside my /db folder of my rails application (rails 2.3.4, ruby 1.8.7) and I am not sure exactly what it does.
Is it needed for some specific environment? (I think I read somewhere that it's used for tests)
Do I need to add it to my git repository?
This post has been used as a reference by a coworker of mine, but the two answers are not exact or informative enough.
development_structure.sql is a low-level dump of the schema, which is necessary when you start to use proprietary database features - either you want to or not, you're going to use them at some point.
Regarding the question of storing it or not, there's some debate. Here is an informative post: http://www.saturnflyer.com/blog/jim/2010/09/14/always-check-in-schema-rb/.
And my take on this follows.
The objective of the development_structure.sql is to sync, for any given commit, the database structure with the code, without having previous knowledge of the schema structure, that is, without having to rely on a pre-existing state of the schema to get the new one.
In a nutshell, by having a schema structure available, whenever you change branch/commit, you load it directly and forget it.
This is mostly valid for dynamic and "crowded" projects, where different branches have differences in the underlying schema structure.
Without having the schema structure stored, you would need to always use an existing reference schema in your database, and migrate it back or forward every time you change branch/commit; several real-world cases can make this process inefficient (e.g. when another branch doesn't have some migrations you currently have, or some migrations can't be rolled back).
Another problem is automated builds, which suffer from the same problems, and even worse, they can't apply manual changes.
The only downside is that it requires a certain habit, which is, to store it every time you run a migration. Easy to say, but also easy to forget.
I don't say you can't live without development_structure.sql - of course you can.
But if you have it, when changing branch/commit you just load-and-forget; if you don't, you [may] have to go through a series of manual steps.
You should not add it to your git repository.
It is a file created automatically by rails when you run migrations with your database.yml configured to connect against a mysql database.
You can view it as an alternative to schema.rb
I believe you can force rails to create it by adding in your environment.rb:
config.active_record.schema_format = :sql
When present this file is used for example by:
rake db:test:clone_structure
Edit
Relevant section in Ruby On Rails Guides.
http://guides.rubyonrails.org/migrations.html#schema-dumping-and-you
They recommend to check it into source control on the wiki.
I personally like to keep it out of it. I like to be able to run all migrations very quickly. It is for me a good sign. If migrations become slow I feel like I am not in total control of my environment anymore. Slowness in migrations generally means I have a lot of data in my development database which I feel wrong.
However, It seems to be a matter of personal taste nowadays.
Follow your instincts on this one.
It's created when you run a rake task to clone your development database to your test database. The development database is outputted to SQL which is then read in to your test DB. You can safely delete it.
In rails 3, you don't even have to write this line,
config.active_record.schema_format = :sql
You can generate this structure.sql file by simply running the above rake command mentioned above
Migrations are undoubtedly better than just firing up phpMyAdmin and changing the schema willy-nilly (as I did during my php days), but after using them for awhile, I think they're fatally flawed.
Version control is a solved problem. The main function of migrations is to keep a history of changes to your database. But storing a different file for each change is a clumsy way to track them. You don't create a new version of post.rb (or a file representing the delta) when you want to add a new virtual attribute -- why should you create a new migration when you want to add a new non-virtual attribute?
Put another way, just as you check post.rb into version control, why not check schema.rb into version control and make the changes to the file directly?
This is functionally the same as keeping a file for each delta, but it's much easier to work with. My mental model is "I want table X to have such and such columns (or really, I want model X to have such and such properties)" -- why should you have to infer from this how to get there from the existing schema; just open up schema.rb and give table X the right columns!
But even the idea that classes wrap tables is an implementation detail! Why can't I just open up post.rb and say:
Class Post
t.string :title
t.text :body
end
If you went with a model like this, you'd have to make a decision about what to do with existing data. But even then, migrations are overkill -- when you migrate data, you're going to lose fidelity when you use a migration's down method.
Anyway, my question is, even if you can't think of a better way, aren't migrations kind of gross?
why not check schema.rb into version control and make the changes to the file directly?
Because the database itself is not in sync with version control.
For instance, you could be using the head of the source tree. But you're connecting to a database that was defined as some past version, not the version you have checked out. The migrations allow you to upgrade or downgrade the database schema from any version and to any version, incrementally.
But to answer your last question, yes, migrations are kind of gross. They implement a redundant revision control system on top of another revision control system. However, neither of these revision control systems is really in sync with the database.
Just to paraphrase what others have said: migrations allow you to protect the data as your schema evolves. The notion of maintaining a single schema.rb file is attractive only until your app goes into production. Thereafter, you'll need a way to migrate your existing users' data as your schema changes.
There are also data-related issues that are important to consider, which migrations solve.
Say an old version of my schema has a feet and inches column. For efficiency purposes, I want to combine that into just an inches column to make sorting and searching easier.
My migration can combine all of the feet and inches data into the inches column (feet * 12 + inches) while it's updating the database (i.e. just before it removes the feet column)
Obviously this being in a migration makes it automatically work when you later apply the changes to your production database.
As it stands, they're annoying and inadequate but quite possibly the best option we have available to us at present. Quite a few smart people have spent quite a lot of time working on the problem and this, so far, is about the best they've been able to come up with. After about 20 years of mostly hand-coding database version updates, I came very rapidly to appreciate migrations as a major improvement when I found ActiveRecord.
As you say, version control is a solved problem. Up to a point I'd agree: it's very solved for text files in particular, less so for other file types and not really very much at all for resources such as databases.
How do migrations look if you view them as version control deltas for databases? They're the sum of the deltas you have to apply to get a schema from one version to another. I'm not aware that even git, for all its super-powerfulness, can take two schema files and generate the necessary DDL to do that.
As far as declaring table content in the model, I believe that's what DataMapper does (no personal experience). I think there may be some DDL inference capabilities there as well.
"even if you can't think of a better way, aren't migrations kind of gross?"
Yes. But they're less gross than anything else we have. Do please let us know when you've completed the non-gross alternative.
I suppose given "even if you can't think of a better way", then yes, in the grand scheme of things, migrations are kind of gross. So are Ruby, Rails, ORMs, SQL, web apps, ...
Migrations have the (not insignificant) advantage that they exist. Gross-but-exists tends to win out over Pleasant-but-nonexistent. I'm sure there probably are pleasant and nonexistent ways to migrate your data, but I'm not sure what that means. :-)
OK, I'm going to take a wild guess here and say that you're probably working all by yourself. In a group development project the power of each individual to take responsibility for just his/her changes to the database required for the code that developer is writing is much much more important.
The alternative is that larger groups of programmers (e.g. 10-15 Java developers where I work) end up relying on a couple of dedicated full time database administrators to do that along with their other maintenance, optimization, etc. duties.