Okay, so this is my situation. I have generated POCOs from a db and developed around them. I have also renamed and changed some of these POCOs to be more readable. I have now made changes to my original db scheme and need to migrate the changes back to my models without overriding my changes except where the scheme has changed. It there a way to generate a *.sql script to hand to by DBAs? If so, it there a way to compare two dbs and generate a change script? ie - dev db => prod db.
You can switch a project to code first and generate the scripts you need via migrations. See the link below for a guide on moving from db first to code first, but it sounds like you may be partially there already.
1) enable-migrations for the project with your context if you haven't already.
2) create a baseline migration. EF will use this as a starting point so you won't get a bunch of code to create the objects that already exist. The ignore changes flag tells EF not to create the existing objects. https://msdn.microsoft.com/en-us/data/dn579398.aspx?f=255&MSPPError=-2147217396#option1
create-migration InitialCodeFirst -IgnoreChanges
3) Now modify your schema as you normally would and create a migration:
add-migration SomeNewThing
4) Create a script for a different database (like PROD) by using -Script. This will not update your database it just creates a script, so I usually run it a second time without -Script:
update-database -Script // creates a script (in VS), but does not apply
update-database // updates the database your connect string points to
5) DBA runs script and this will add a record to __MigrationHistory to identify it as being applied.
http://devgush.com/2014/02/24/migrating-a-project-from-database-first-to-code-first/
Here is a useful link on deployment: http://cpratt.co/migrating-production-database-with-entity-framework-code-first/#at_pco=smlwn-1.0&at_si=54ad5c7b61c48943&at_ab=per-12&at_pos=0&at_tot=1
Related
I have been able to successfully create multiple DBContexts within my ASP.NET Core application that connect to multiple tables in multiple databases.
After creating my first migrations for both contexts, I'm able to generate my SQL scripts using the command:
Script-Migration -Context CustomContext1
Script-Migration -Context CustomContext2
When I looked at the result of the script generation, I noticed that both scripts for both contexts don't reference the database of either CustomContext1 or CustomContext2. I'm supposed to send this script down an automated pipeline, which means that the specif DB for each script needs to be included.
How do I get this to happen automatically using the script generation command?
After so many days of research and testing, the only way to accomplish this is to manually add the "USE Database" line to the beginning of the generated SQL script.
It makes no sense, really. With the presence of multiple DbContexts, the logical thing to do when SQL scripts are being generated would be to automatically add the "USE Database" line at the beginning of each script.
Perhaps a fix will be included in the next version of EF Core tools. Who knows.
The Ktor or Exposed frameworks do not have any built-in support for database migrations. What's the recommended way to do this?
If you are using Ktor with Gradle I would recommend using Flyway programatically inside the entry point of your Application. This way it could easily be a part of your Continuous Delivery pipeline. You can see the Flyway docs using API here: https://flywaydb.org/documentation/api/
What I essentially do though is add the dependency (using Kotlin DSL):
implementation("org.flywaydb:flyway-core:6.5.2")
And then all you need to do is create an instance of Flyway and call migrate when you load your module:
fun Application.module() {
Flyway.configure().dataSource(/*config to your DB*/).load().migrate()
//the rest of your application
routing {
}
}
You could of course extract the creation of Flyway to your DI tool (e.g. Koin) and add some logging to show progress.
This way your DB will be migrated (if necessary) every time just before your app is started.
As for writing up the actual migration the official docs are very helpful. What you essentially need to do is:
Make sure you have the required directory for migration files - src/main/resources/db/migration by default ).
Write plain SQL in separate migration files in the above directory. The filenames need to stick to the convention too (by default: start with capital V then a number, which you will increment for each new migration, double underscore - this tricked me at the beginning ;) - and a snake_case description, e.g. V1__Create_person_table.sql)
Run the app and observe magic :D
Database tables are migrated at deployment via Flyway and scripts can be added in db/migrations folder to add new tables or execute queries like inserting data etc on server startup.
(https://github.com/arun0009/kotlin-ktor-exposed-sample-api)
Here's how Flyway works: https://flywaydb.org/getstarted/how
Download, install, and configure Flyway (See https://flywaydb.org/getstarted/firststeps/commandline)
Point it to a database, which will be the location of flyway_schema_history
Write your migration files for create table or insert data following their naming convention: V<version number>__<migration description> and run with flyway migrate
Write your repeatable migration files for create view following their naming convention: R__<migration description> and run with flyway migrate
Write your undo migration files for drop table or delete data following their naming convention: V<version number>__<migration description> and run with flyway migrate
Check your migration status with flyway info and commit your files if you're happy
Make any necessary modifications are rerun migration. Repeat and commit.
In case if this is a one time activity, you can try using a Off-the shelf utility like SQL Data Compare.
To make this happen, you would need to ensure that both the databases are accessible locally from your machine so that you can create 2 DB connections and run a comparison against them.
At the end of comparison, you can get a Auto-Generated SQL Script out of it to run against your new schema and make it sychronized.
In case if you wish to compare Schema Objects as well, again Red-Gate provides a similar Schema Compare tool, which they have now started to call as SQL Compare (God knows why !!). This utility would also provide similar auto-generated script to help you.
But, again Red Gate is good for one-time migration and you can use this with their Trial version for a period of 30 days. For similar activity in regular basis, you would need to buy the Licensed version of same software.
For Data Migrantion, I use Navicat Premium which i find very easy to use, but it is not an open source. If you are looking for an Open Source Tool, then You can use SQLines Data which is an open source (Apache License 2.0), scalable, parallel high performance data transfer and schema conversion tool that you can use for database migrations and ETL processes.
SQLines
It is available for Linux, Windows, both 64-bit and 32-bit platforms.
You can also use SQLines Data for cross-platform database migration. The tool migrates table definitions, constraints, indexes and transfers data.
This how you can start with SQLines:
Download and unzip the file, no installation is required
Run sqldataw.exe on Windows to launch the GUI version
Run ./sqldata on Linux to launch the command line tool.
And There are Migration guidelines available for specific databases.Guidelines
In a new code-first asp.net-core project connecting to my SQL-server database. When I run
add-migration MyInitialMigration
a migration file is created, then executed in SQL server creating the database, table structure with fields, indexes, etc. and populating the tables with the seed data from my DBInitialiser. The table __MigrationHistory is not created in the database and the ModelSnapshot.cs file is created. At this point I haven't run
update-database
which from everything I can find, I should need to run before the database, seed data, etc. is actually created. As far as I can tell, I've not got automatic migrations enabled.
Because of this, subsequent migrations aren't applied to the database because the tables already exist, which I'm guessing it figures out by looking for the __MigrationHistory table and not finding any rows in it.
How can I either get the initial add-migration to add the __MigrationHistory table or how do I get the migration to only happen when I enter update-database?
Not sure if it's related, but another project that was migrating fine last week now doesn't work properly. The migration .cs file is created fine, but when it automatically tries to create the database, etc. (when previously I needed to run update-database ) I get the error
Invalid JSON primitive: "C:\\x\\Migrations\\20171108125832_MyInitialMigration.cs",
"metadataFile": "C:\\x\\Migrations\\20171108125832_MyInitialMigration.Designer.cs",
"snapshotFile": "C:\\x\\Migrations\\MyContextModelSnapshot.cs"
}.
which relates to the following command I the migrations .cs file
migrationBuilder.CreateIndex(
name: "IX_tablea_tableID2",
table: "tablea",
column: "tableID2");
which as far as I can tell is identical to the several blocks of code above it which all work fine. I've tried deleting the migration files and database and rerunning the add-migration with the same effect. Each time I delete everything and start again like this, a different index causes the error message, even if it worked the previous time.
Has there been an update which breaks/alters how migrations work that I've missed?
You must have something odd in your code, like calling dbContext.Database.EnsureCreatedAsync(). This will create tables and database, but no migrations.
In previous versions of ASP.NET Core / EF Core (1.x), the Configure method of Startup class would not be executed, when running the EF Core tools (either add-migration in Powershell or dotnet ef migrations add in command line).
In order for the __MigrationHistory Table to be created, the migration must be applied with either dbContext.Database.MigrateAsync() or by database-update/dotnet ef database update.
However, with the new versions the migrations (or the dbContext.Database.EnsureCreated()) will be applied on every EF Core tools command.
The new pattern for performing DbContext discovery is to have a BuildWebHost static file, which configures the whole application and have extension methods in Main(...) method to perform the migration and seeding.
My previous answer on the new way to apply migrations (and seeding) covers the code.
The ASP.NET Core Annoucement documents this changes (its always a good idea to subscribe to this GitHub repository to receive notifications about new features and breaking changes)
Our company is in the process of adapting TFS for source repository and project management. I am in charge of database part of the project. We are using SQL Server 2008 R2, Visual Studio 2012 and TFS Online. We have a database that is used by several of our applications. So far I have been the only one handling any change to this database. As the company is expending we are going to have multiple dev teams. So I am planning to save the database as as SSDT project to TFS.
At the moment I am maintaining my database like the following:
I have separate folders for UDFs, Stored Procedures, and Config.
Under these folders I have subfolders for each objects. For example, for stored procedures I have subfolders for each stored procedure which contains the SQL script to create the SP. The config folder contains any script similar to SSDT's post deployment script (for example, populating static data).
The SQL script contains code to drop the procedure and create it.
I have a c# app to concatenate all the SQL files into one single SQL file. Let's call it the FINAL script. When creating FINAL script I can specify version number which adds an update statement to update the version table on the database.
FINAL script is made available for customers to download and execute on the database. So the script mainly contains any add/edit to SPs, UDFs, and static data. It does not touch any existing data (data entered by user) in most cases.
As a newbie to TFS and SSDT I am not exactly sure how this can be done using SSDT/TFS or if there is better way of doing something similar. So far what I have understood about SSDT and TFS is:
I can import an existing database to SSDT project.
This will create scripts for all objects including tables.
I can easily do a publish of the database to a local server or to a server I have access to.
Things that seem confusing so far:
How do I supply clients with my latest update script? I am thinking of manually including the FINAL script to the SSDT project but there must be better way of doing it.
How do I publish the changes to a copy of the database without the loss of any user-entered data? My guess is when publishing the tables get created. I can take care of the static data but I am not sure how to handle data entered by users.
May be there is something fundamentally wrong in my understanding of this whole thing. That is why I am here... :)
You want to pull your DB into a SQL Project. Maintain all of your changes there. This tells your system what the schema of your database should be. From there, I'd generate the dacpac files (through building the project) and provide those to your clients along with having them install the SSDT tools that include SQLPackage. They can run SQLPackage to make changes to their database to handle the schema changes automatically. This will bring their database in line with your schema, no matter how far off it might be.
I'd also create a publish profile for them to use. This lets you control some of the settings.
You can choose to not drop any objects not in your project
You can choose to ignore users/permissions
You can set an option to not allow changes if there would be data loss.
You can wrap everything in a transaction so a failed update rolls back
If you give them a batch file to run, you can specify an output file or a Diff report, or have them generate their own script to do the update.
I blogged about this at http://schottsql.blogspot.com/2013/10/all-ssdt-articles.html
(or http://schottsql.blogspot.com/search/label/SSDT if that doesn't work well). That will take you through some basics of why you might want to use SQL Projects, creating them, maintaining them, and publishing the changes to an existing database.
I have 2 files: db structure, and db data sql files.
(we are not in part of project where we are creating alter files yet).
Is there a way to achieve running these files before build each time if any of them changed using maven?
The only idea I have is to rename the files (bring the version up) after each change to them making sure flyaway will pick it up.
Is there a smarter way, perhaps using API or some other 'trick'?
Thanks
--MB
You can use flyway migrate with the following configuration
validationMode=ALL
validationErrorMode=CLEAN
For each sql migration a CRC32 checksum is calculated when the sql script is executed. The validate mechanism checks if the sql migrations in the classpath still has the same checksum as the sql migration already executed in the database.
validationErrorMode=CLEAN is exclusively intended as a convenience for development. Even tough we strongly recommend not to change migration scripts once they have been checked into SCM and run, this provides a way of dealing with this case in a smooth manner. The database will be wiped clean automatically, ensuring that the next migration will bring you back to the state checked into SCM.
This configuration can be used from API, maven plugin or command line.
With newer Flyway, you would use
flyway.setInitOnMigrate(true);
flyway.setCleanOnValidationError(true);
flyway.setValidateOnMigrate(true);
To drop everything if the checksum of the file changes or the version table is missing.