Adding 'GO' statements to Entity Framework migrations - sql

So I have an application with a ton of migrations made by Entity framework.
We want to get a script for all the migrations at once and using the -Script tag does work fine.
However...it does not add GO statements in the SQL giving us problems like Alter view should be the first statement in a batch file...
I have been searching around and manually adding Sql("GO"); help with this problem but only for the entire script. When I use the package console manager again it returns an exception.
System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure 'GO'.
Is there a way to add these GO tags only when using the -Script tag?
If not, what is a good approach for this?
Note: we have also tried having multiple files but since we have so many migrations, this is near impossible to maintain every time.

If you are trying to alter your view using Sql("Alter View dbo.Foos As etc"), then you can avoid the should be the first statement in a batch file error without adding GO statements by putting the sql inside an EXEC command:
Sql("EXEC('Alter View dbo.Foos As etc')")

In order to change the SQL Generated by entity framework migrations you can create a new SqlServerMigrationSqlGenerator
We have done this to add a GO statement before and after the migration history:
public class MigrationScriptBuilder: SqlServerMigrationSqlGenerator
{
protected override void Generate(System.Data.Entity.Migrations.Model.InsertHistoryOperation insertHistoryOperation)
{
Statement("GO");
base.Generate(insertHistoryOperation);
Statement("GO");
}
}
then add in the Configuration constructor (in the Migrations folder of the project where you DbContext is) so that it uses this new sql generator:
[...]
internal sealed class Configuration : DbMigrationsConfiguration<PMA.Dal.PmaContext>
{
public Configuration()
{
SetSqlGenerator("System.Data.SqlClient", new MigrationScriptBuilder());
AutomaticMigrationsEnabled = false;
}
[...]
So now when you generate a script using the -Script tag, you can see that the insert into [__MigrationHistory] is surrounded by GO
Alternatively in your implementation of SqlServerMigrationSqlGenerator you can override any part of the script generation, the InsertHistoryOperation was suitable for us.

Turn out the concept exist deep in the SqlServerMigrationSqlGenerator as an optional argument for Statement(sql, batchTerminator). Here is something based on Skyp idea. It works both in -script mode or not. The GOs are for different operations than for Skyp only because our needs are a little different. You then need to register this class in the Configuration as per Skyp instructions.
public class MigrationScriptBuilder : SqlServerMigrationSqlGenerator
{
private string Marker = Guid.NewGuid().ToString(); //To cheat on the check null or empty of the base generator
protected override void Generate(AlterProcedureOperation alterProcedureOperation)
{
SqlGo();
base.Generate(alterProcedureOperation);
SqlGo();
}
protected override void Generate(CreateProcedureOperation createProcedureOperation)
{
SqlGo();
base.Generate(createProcedureOperation);
SqlGo();
}
protected override void Generate(SqlOperation sqlOperation)
{
SqlGo();
base.Generate(sqlOperation);
}
private void SqlGo()
{
Statement(Marker, batchTerminator: "GO");
}
public override IEnumerable<MigrationStatement> Generate(IEnumerable<MigrationOperation> migrationOperations, string providerManifestToken)
{
var result = new List<MigrationStatement>();
var statements = base.Generate(migrationOperations, providerManifestToken);
bool pendingBatchTerminator = false;
foreach (var item in statements)
{
if(item.Sql == Marker && item.BatchTerminator == "GO")
{
pendingBatchTerminator = true;
}
else
{
if(pendingBatchTerminator)
{
item.BatchTerminator = "GO";
pendingBatchTerminator = false;
}
result.Add(item);
}
}
return result;
}
}

The easiest way is to add /**/ before the GO statement.

Just replace the current statement with a .Replace("GO", "");

Related

How to add a semi colon ; automatically to each generated sql statement using jOOQ

I'm trying to add a semi colon ; to every jOOQ generated sql statement as I'm writing multiple DDL and insert statement to an output file.
I found a similar question here suggesting using an ExecuteListener here https://jooq-user.narkive.com/6adKecpt/adding-semicolon-at-the-end-of-sql-statement.
My setup is now as follows (using Groovy):
private DSLContext createDSLContext() {
def configuration = new DefaultConfiguration()
configuration.settings = new Settings()
.withRenderFormatted(true)
.withRenderKeywordCase(RenderKeywordCase.LOWER)
.withRenderQuotedNames(RenderQuotedNames.ALWAYS)
.withStatementType(StatementType.STATIC_STATEMENT)
configuration.set(
new DefaultExecuteListenerProvider(new DefaultExecuteListener() {
#Override
void renderEnd(ExecuteContext ctx) {
ctx.sql(ctx.sql() + ";")
}
}),
new DefaultExecuteListenerProvider(new DefaultExecuteListener() {
#Override
void start(ExecuteContext ctx) {
println "YEAH!!!"
}
}))
// return configuration.dsl();
return DSL.using(configuration)
}
but is not adding the semi colon, nor is it getting in the renderEnd method at all.
I added another execute listener to print something at the start (as I have seen in other examples) but it is also never called..
My code looks like:
file.withWriter { writer ->
// Drop schema objects.
DEFAULT_SCHEMA.tables.each {
switch (it.type) {
case TABLE:
writer.writeLine(dsl.dropTableIfExists(it).SQL)
break
case VIEW:
writer.writeLine(dsl.dropViewIfExists(it).SQL)
break
}
}
writer.writeLine("")
// Create schema objects.
def ddlStatements = dsl.ddl(DEFAULT_SCHEMA)
ddlStatements.each {
writer.writeLine(it.SQL)
writer.writeLine("")
}
// Insert data.
def insert = dsl.insertInto(Tales.CUSTOMER).columns(Tales.CUSTOMER.fields())
customers.each {insert.values(it) }
writer.writeLine(insert.SQL)
}
The ExecuteListener lifecycle is only triggered when you execute your queries with jOOQ. You're not doing that, you're just calling Query.getSQL()
You could wrap your queries into DSLContext.queries(Query...), and jOOQ will separate the statements using ; when you call Queries.getSQL() when you call Queries.toString(). Of course, that's not reliable, the behaviour of toString() might change in the future, which is why it would make sense to offer methods like Queries.getSQL() and the likes: https://github.com/jOOQ/jOOQ/issues/11755
For the time being, why not just add the semi colon manually to the writer in your code?

Custom Liquibase executor combining JdbcExecutor and LoggingExecutor

I'm looking for a way to record and write all those SQL statements to an output
file which get executed while running a Liquibase migration against an empty
target database.
The idea behind this is to speed up the initialization phase of integration tests
against a test database by simply reading and executing the SQL statements from the
generated file for subsequent tests.
I had no luck using updateSQL due to different handling of changesets with
pre-conditions (e.g. changeSetExecuted resolves to true for "update" but false for
"updateSQL").
Another approach was to run the Liquibase migration first, then writing a temporary
changelog file using GenerateChangeLogCommand which is finally used by another Liquibase
instance to produce an SQL update file.
While this approach works, it a) feels a bit hacky-ish, b) the end result is not the same
as running the migration directly.
Anyway, what I've come up with is a custom implementation of JdbcExecutor which incorporates
a LoggingExecutor. The implementation looks as follows:
#LiquibaseService(skip = true)
public class LoggingJdbcExecutor extends JdbcExecutor {
private LoggingExecutor loggingExecutor;
public LoggingJdbcExecutor(Database database, Writer writer) {
loggingExecutor = new LoggingExecutor(this, writer, database);
setDatabase(database);
}
#Override
public void execute(SqlStatement sql, List<SqlVisitor> sqlVisitors) throws DatabaseException {
super.execute(sql, sqlVisitors);
loggingExecutor.execute(sql, sqlVisitors);
}
#Override
public int update(SqlStatement sql, List<SqlVisitor> sqlVisitors) throws DatabaseException {
final int result = super.update(sql, sqlVisitors);
loggingExecutor.update(sql, sqlVisitors);
return result;
}
#Override
public void comment(String message) throws DatabaseException {
super.comment(message);
loggingExecutor.comment(message);
}
}
This executor gets injected into Liquibase before update() is invoked as follows:
final String path = configuration.getUpdateSqlExportFile();
ExecutorService.getInstance().setExecutor(liquibase.getDatabase(), new LoggingJdbcExecutor(
liquibase.getDatabase(), new FileWriter(path)
));
Is this approach reasonable and future proof ? While it seems to work I'm not sure if maybe I'm
missing something and there's a better way.
Thanks

Use MEF to compose parts but postpone the creation of the parts

As explained in these questions I'm trying to build an application that consists of a host and multiple task processing clients. With some help I have figured out how to discover and serialize part definitions so that I could store those definitions without having to have the actual runtime type loaded.
The next step I want to achieve (or next two steps really) is that I want to split the composition of parts from the actual creation and connection of the objects (represented by those parts). So if I have a set of parts then I would like to be able to do the following thing (in pseudo-code):
public sealed class Host
{
public CreationScript Compose()
{
CreationScript result;
var container = new DelayLoadCompositionContainer(
s => result = s);
container.Compose();
return script;
}
public static void Main()
{
var script = Compose();
// Send the script to the client application
SendToClient(script);
}
}
// Lives inside other application
public sealed class Client
{
public void Load(CreationScript script)
{
var container = new ScriptLoader(script);
container.Load();
}
public static void Main(string scriptText)
{
var script = new CreationScript(scriptText);
Load(script);
}
}
So that way I can compose the parts in the host application, but actually load the code and execute it in the client application. The goal is to put all the smarts of deciding what to load in one location (the host) while the actual work can be done anywhere (by the clients).
Essentially what I'm looking for is some way of getting the ComposablePart graph that MEF implicitly creates.
Now my question is if there are any bits in MEF that would allow me to implement this kind of behaviour? I suspect that the provider model may help me with this but that is a rather large and complex part of MEF so any guidelines would be helpful.
From lots of investigation it seems that is not possible to separate the composition process from the instantiation process in MEF so I have had to create my own approach for this problem. The solution assumes that the scanning of plugins results in having the type, import and export data stored somehow.
In order to compose parts you need to keep track of each part instance and how it is connected to other part instances. The simplest way to do this is to make use of a graph data structure that keeps track of which import is connected to which export.
public sealed class CompositionCollection
{
private readonly Dictionary<PartId, PartDefinition> m_Parts;
private readonly Graph<PartId, PartEdge> m_PartConnections;
public PartId Add(PartDefinition definition)
{
var id = new PartId();
m_Parts.Add(id, definition);
m_PartConnections.AddVertex(id);
return id;
}
public void Connect(
PartId importingPart,
MyImportDefinition import,
PartId exportingPart,
MyExportDefinition export)
{
// Assume that edges point from the export to the import
m_PartConnections.AddEdge(
new PartEdge(
exportingPart,
export,
importingPart,
import));
}
}
Note that before connecting two parts it is necessary to check if the import can be connected to the export. In other cases MEF does that but in this case we'll need to do that ourselves. An example of how to approach that is:
public bool Accepts(
MyImportDefinition importDefinition,
MyExportDefinition exportDefinition)
{
if (!string.Equals(
importDefinition.ContractName,
exportDefinition.ContractName,
StringComparison.OrdinalIgnoreCase))
{
return false;
}
// Determine what the actual type is we're importing. MEF provides us with
// that information through the RequiredTypeIdentity property. We'll
// get the type identity first (e.g. System.String)
var importRequiredType = importDefinition.RequiredTypeIdentity;
// Once we have the type identity we need to get the type information
// (still in serialized format of course)
var importRequiredTypeDef =
m_Repository.TypeByIdentity(importRequiredType);
// Now find the type we're exporting
var exportType = ExportedType(exportDefinition);
if (AvailableTypeMatchesRequiredType(importRequiredType, exportType))
{
return true;
}
// The import and export can't directly be mapped so maybe the import is a
// special case. Try those
Func<TypeIdentity, TypeDefinition> toDefinition =
t => m_Repository.TypeByIdentity(t);
if (ImportIsCollection(importRequiredTypeDef, toDefinition)
&& ExportMatchesCollectionImport(
importRequiredType,
exportType,
toDefinition))
{
return true;
}
if (ImportIsLazy(importRequiredTypeDef, toDefinition)
&& ExportMatchesLazyImport(importRequiredType, exportType))
{
return true;
}
if (ImportIsFunc(importRequiredTypeDef, toDefinition)
&& ExportMatchesFuncImport(
importRequiredType,
exportType,
exportDefinition))
{
return true;
}
if (ImportIsAction(importRequiredTypeDef, toDefinition)
&& ExportMatchesActionImport(importRequiredType, exportDefinition))
{
return true;
}
return false;
}
Note that the special cases (like IEnumerable<T>, Lazy<T> etc.) require determining if the importing type is based on a generic type which can be a bit tricky.
Once all the composition information is stored it is possible to do the instantiation of the parts at any point in time because all the required information is available. Instantiation requires a generous helping of reflection combined with the use of the trusty Activator class and will be left as an exercise to the reader.

NHibernate database versioning: object level schema and data upgrades

I would like to approach database versioning and automated upgrades in NHibernate from a different direction than most of the strategies proposed out there.
As each object is defined by an XML mapping, I would like to take size and checksum for each mapping file/ configuration and store that in a document database (raven or something) along with a potential custom update script. If no script is found, use the NHibernate DDL generator to update the object schema. This way I can detect changes, and if I need to make DML changes in addition to DDL, or perform a carefully ordered transformation, I can theoretically do so in a controlled, testable manner. This should also maintain a certain level of persistence-layer agnosticism, although I'd imagine the scripts would still necessarily be database system-specific.
The trick would be, generating the "old" mapping files from the database and comparing them to the current mapping files. I don't know if this is possible. I also don't know if I'm missing anything else that would make this strategy prohibitively impractical.
My question, then: how practical is this strategy, and why?
what i did to solve just that problem
version the database in a table called SchemaVersion
query the table to see if schema is up to date (required version stored in DAL), if yes goto 6.
get updatescript with version == versionFromBb from resources/webservices/...
run the script which also alters the schemaversion to the new version
goto 2.
run app
to generate the scripts i have used 2 options
support one rdbms: run SchemaUpdate to export into file and add DML statements manually
support multiple rdbms: use Nhibernate class Table to generate at runtime ddl to add/alter/delete tables and code which uses a session DML
Update:
"what method did you use to store the current version"
small example
something like this
public static class Constants
{
public static readonly Version DatabaseSchemaVersion = new Version(1, 2, 3, 4);
}
public class DBMigration
{
private IDictionary<Version, Action> _updates = new Dictionary<Version, Action>();
private Configuration _config;
private Dialect _dialect;
private IList<Action<ISession>> _actions = new List<Action<ISession>>(16);
private string _defaultCatalog;
private string _defaultSchema;
private void CreateTable(string name, Action<Table> configuretable)
{
var table = new Table(name);
configuretable(table);
string createTable = table.SqlCreateString(_dialect, _config.BuildMapping(), _defaultCatalog, _defaultSchema);
_actions.Add(session => session.CreateSQLQuery(createTable).ExecuteUpdate());
}
private void UpdateVersionTo(Version version)
{
_actions.Add(session => { session.Get<SchemaVersion>(1).Value = version; session.Flush(); });
}
private void WithSession(Action<session> action)
{
_actions.Add(action);
}
public void Execute(Configuration config)
{
_actions.Clear();
_defaultCatalog = config.Properties[NH.Environment.DefaultCatalog];
_defaultSchema = config.Properties[NH.Environment.DefaultSchema];
_config = config;
_dialect = Dialect.GetDialect(config.Properties);
using (var sf = _config.BuildSessionFactory())
using (var session = sf.OpenSession())
using (var tx = session.BeginTransaction())
{
Version dbVersion = session.Get<SchemaVersion>(1).Value;
while (dbVersion < Constants.DatabaseSchemaVersion)
{
_actions.Clear();
_updates[dbVersion].Invoke(); // init migration, TODO: error handling
foreach (var action in _actions)
{
action.Invoke(session);
}
tx.Commit();
session.Clear();
dbVersion = session.Get<SchemaVersion>(1).Value;
}
}
}
public DBMigration()
{
_updates.Add(new Version(1, 0, 0, 0), UpdateFromVersion1);
_updates.Add(new Version(1, 0, 1, 0), UpdateFromVersion1);
...
}
private void UpdateFromVersion1()
{
AddTable("Users", table => table.AddColumn(...));
WithSession(session => session.CreateSqlQuery("INSERT INTO ..."));
UpdateVersionTo(new Version(1,0,1,0));
}
...
}

How to get the sql update method generated by LinqToSql?

How to get the sql generated by LinqToSql for update method?
I use the following code to show the sql generated by LinqToSql in VS2008's debug output window, but it only gets the sql select method generated,
how can I find the sql update method that was generated by LinqToSql?
I know the Sql Server Profiler and LinqPad can get it(the sql-update generated), but I want to show them in VS2008 or Log them to a file.
public partial class Linq2 : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
DemoDataContext ctx = new DemoDataContext ();
// Then attach it to the Log property of your DataContext...
ctx.Log = new DebugTextWriter();
var product = ctx.Products.FirstOrDefault();
product.ProductName = "NewName1";
ctx.SubmitChanges();
}
}
// Add this class somewhere in your project...
public class DebugTextWriter : System.IO.TextWriter
{
public override void Write(char[] buffer, int index, int count)
{
System.Diagnostics.Debug.Write(new String(buffer, index, count));
}
public override void Write(string value)
{
System.Diagnostics.Debug.Write(value);
}
public override Encoding Encoding
{
get { return System.Text.Encoding.Default; }
}
}
And I get the sql-select query in the VS2008 debug output window:
SELECT TOP (1) [t0].[Id], [t0].[ProductName] ……
FROM [dbo].[Products] AS [t0]
-- Context: SqlProvider(Sql2005) Model: AttributedMetaModel Build: 3.5.30729.1
Your Database context object has a Log method that you can override. Your full Update statement, and every SQL command generated by Linq-To-SQL, can be captured via this Log method. I know this works because I use it to capture all our queries in our app. Just keep in mind that L2S can send a fair amount of output to the Log method, so make sure to capture it all. Your Update statement is in there somewhere.
Thank you for all answers.
I have found Linq To Sql Profiler to solve the problem.