In HSQL to change TRANSACTION CONTROL there can't be any active transactions.
Flyway, in turn, after committing migration X and before executing SQL from migration X, sets autocommitt=false and executes some of its own statements. So if the migration contains SET DATABASE TRANSACTION CONTROL statement it will wait for those uncommitted statements forever causing application to hang.
(Side note: The statements executed by flyway before migration varies from version to version e.g. in 1.7 that were pure selects so changing from LOCK to MVCC was possible but after I had MVCC any subsequent DDL statements in further migrations hanged; in flyway 2.0 it was select for update on schema_version table so any transaction control change hanged; in 2.2 select for update was changed to explicit lock with the same effect as in 2.0)
So basically it is not possible to change transaction control in flyway migrations. On the other hand flyway discourages changes outside of its migration. Any idea then how to change transaction control in with flyway/hsql?
Update
Another observation is that when database control is set to MVCC then any DDL statement in flyway migration hangs application too. So I would just set LOCKS before each migration and restore MVCC after it. Would that be clean solution from Flyway perspective?
import com.googlecode.flyway.core.util.jdbc.JdbcUtils;
public void migrate() {
setDbTransactionControl("LOCKS");
flyway.migrate();
setDbTransactionControl("MVCC");
}
private void setDbTransactionControl(String mode) {
Connection connection = null;
try {
connection = JdbcUtils.openConnection(ds);
connection.createStatement().execute("SET DATABASE TRANSACTION CONTROL " + mode);
} catch (SQLException e) {
//log it
JdbcUtils.closeConnection(connection);
} finally {
JdbcUtils.closeConnection(connection);
}
}
This is not possible inside a Flyway migration.
Before Flyway starts a migration, it opens a transaction in a separate connection to acquire a lock on its metadata table. So you will never be able to execute a statement that absolutely must be run without any other transactions.
Your best option is probably to set it on the datasource, so it can init each connection this way upon create.
Try to use the Flyway callbacks beforeMigrate and afterMigrate. Both run apart from the migration transactions. MVCC should be used for my application so the the JDBC URL contains hsqldb.tx=mvcc. I could sucessfully change the transaction model during the Flyway migration with beforeMigrate.sql SET DATABASE TRANSACTION CONTROL LOCKS; and afterMigrate.sql SET DATABASE TRANSACTION CONTROL MVCC;. There are also Java versions of the callbacks. I'm using HSQLDB 2.3.3 and Flyway 3.2.1.
Related
I wrote a script that uses sqlps to deploy a SSAS tabular cube. After the deployment, I need to perform a couple of actions on the cube but if I try to access it, I get a message saying that the cube doesn't exist. But it does, in fact, if I split the actions into two scripts (deploy -> exit sql ps -> new sqlps session), it works (for reasons that don't matter now, I cant do that).
It seems that the sqlps session doesn't see the cube it just deployed. I'm wondering if there is a refresh command I can run or if I can run sqlps in a "read uncommited" state.
Do you use the PowerShell Transactions supporting cmdlets? Looks like the transaction is being commited only after your script runs out.
Try to split the deploying logic into two transactions: creating the cube and the other actions you need to perform. For each of this parts you should use it's own transaction, like this:
Start-PSTransaction
// logic
Complete-PSTransaction
If you need to access the transactions programmatically, you should use the TransactionScope analog in the PowerShell, the Cmdlet's CurrentPsTransaction property, like this:
using(CurrentPsTransaction)
{
... // Perform transactional work here
}
... // Perform non-transacted work here
using(CurrentPsTransaction)
{
... // Perform more transactional work here
}
Update:
May be you can turn on the transaction using the declaration?
Developing for transactions – Cmdlet and Provider declarations
Cmdlets declare their support for transactions in a similar way that they declare their support for ShouldProcess:
[Cmdlet(“Get”, “Process”, SupportsTransactions=True)]
Providers declare their support for transactions through the ProviderCapabilities flag:
[CmdletProvider(“Registry”, ProviderCapabilities.Transactions)]
More about the isolation levels in Powershell:
TransactionScope with IsolationLevel set to Serializable is locking all SQL SELECTs
We have couple of data schemas and we investigate the migration to Liquibase. (One of data schemas is already migrated to Liquibase).
Important question for us is if Liquibase supports dry run:
We need to run database changes on all schemas without commit to ensure we do not have problems.
In case of success all database changes run once again with commit.
(The question similar to this SQL Server query dry run but related to Liquibase)
Added after the answer
I read documentation related to updateSQL and it is not answers the requirements of “dry run”.
It just generates the SQL (in command line, in Ant task and in Maven plugin).
I will clarify my question:
Does Liquibase support control on transactions?
I want to open transaction before executing of Liquibase changelog, and to rollback the transaction after the changelog execution.
Of course, I need to verify the result of the execution.
Is it possible?
Added
Without control on transactions (or dry run) we can not migrate to Liquibase all our schemas.
Please help.
You can try "updateSQL" mode, it will connect db (check you access rights), acquire db lock, generate / print SQL sentences to be applied (based on db state and you current liquibase change sets) also it will print chageset id's missing in current state of db and release db lock.
Unfortunately, no.
By default, Liquibase commits the transaction executing all statements of a changeset. I assume that the migration paths you have in mind usually involve more than a single changeset.
The only way you can modify the transaction behavior is the runInTransaction attribute for the <changeset> tag, as documented here. By setting it to false, you effectively disable the transaction management, i.e. it enables auto-commit mode as you can see in ChangeSet.java.
I think that this feature could be a worthwhile addition to Liquibase, so I opened a feature request: CORE-1790.
I think your answer is "it does not support dry runs" but the problem is primarily with the database and not with liquibase.
Liquibase does run each changeSet in a transaction and commits it after inserting into the DATABASECHANGELOG table so in theory you could override liquibase logic to roll back that transaction instead of committing it, but you will run into the problem where most SQL ran by liquibase is auto-committing.
For example, if you had a changeSet of:
<changeSet>
<createTable name="test">
...
</createTable>
</changeSet>
What is ran is:
START TRANSACTION
CREATE TABLE NAME ...
INSERT INTO DATABASECHANGELOG...
COMMIT
but even if you changed the last command to ROLLBACK the create table call will auto-commit when it runs and the only thing that will actually roll back is the INSERT.
NOTE: there are some databases that will rollback DDL SQL such as postgresql, but the majority do not.
INSERT/UPDATE commands would run in a transaction and could be auto-rolled back at the end, but liquibase does not have a postCondition command to do the in-transaction check of the state that would be required. That would be a useful feature (https://liquibase.jira.com/browse/CORE-1793) but even it would not be usable if there are any auto-committing change tags in the changeset. If you added a postcondition to create table example above, the postcondition would fail and the update would fail, but the table would still be there.
If your Liquibase migration is sufficiently database agnostic, you can just run it on an in-memory H2 database (or some other "throwaway database") that you can spin up easily using a few lines of code.
var info = new Properties();
info.put("user", "sa");
info.put("password", "");
try (var con = new org.h2.Driver().connect("jdbc:h2:mem:db", info)) {
var accessor = new FileSystemResourceAccessor();
var jdbc = new JdbcConnection(con);
var database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(jdbc);
Liquibase liquibase = new Liquibase("/path/to/liquibase.xml", accessor, database);
liquibase.update("");
}
I've blogged about this approach more in detail here.
I have a n-tier C# ASP .Net application server which uses stored procedures to communicate with the database.
I have a service layer which rolls back all ADO .net transactions if an exception is thrown, using TransactionScope.requiresNew.
In my stored procedure, I want to track login attempt numbers, so we want to keep the transaction framework as is, but want to have an isolated transaction which we commit.
How do I do this?
I have tried using a new TransactionScope.RequiresNew in our data layer, but this has no effect.
Strange - RequiresNew in the inner (Logging) TransactionScope should work.
In the below nested transaction, TransactionScopeOption.Suppress or TransactionScopeOption.RequiresNew both work for me - the inner transaction is committed (Dal2.x), and the outer one aborted (Dal1.x).
try
{
using (TransactionScope tsOuter = new TransactionScope(TransactionScopeOption.Required))
{
DAL1.Txn1();
using (TransactionScope tsLogging = new TransactionScope(TransactionScopeOption.Suppress))
{
DAL2.Txn2();
tsLogging.Complete();
}
throw new Exception("Big Hairy Exception");
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
Edit : Mixing TransactionScope and explicit T-SQL transactions is to be avoided - this is stated in the same link you've referenced viz http://msdn.microsoft.com/en-us/library/ms973865.aspx, quoted below
TransactionScopes manage transaction escalation quite intelligently - they will use the (e.g. DTC will only be used if the transactions span multiple databases or resources - e.g. SQL and MSMQ). They also work with the SQL 2005+ Lightweight transactions, so multiple connections to the same database will also be managed within a transaction without the overheads of DTC.
IMHO the decision as to whether to use Suppress vs RequiresNew will depend on whether you need to do your auditing within a transaction at all - RequiresNew for an isolated txn, vs Suppress for none.
When using System.Transactions,
applications should not directly
utilize transactional programming
interfaces on resource managers—for
example the T-SQL BEGIN TRANSACTION or
COMMIT TRANSACTION verbs, or the
MessageQueueTransaction() object in
System.Messaging namespace, when
dealing with MSMQ. Those mechanisms
would bypass the distributed
transaction management handled by
System.Transactions, and combining the
use of System.Transactions with these
resource manager "internal"
transactions will lead to inconsistent
results .... Never mix the two
When trying to use SQLite with System.Transactions TransactionScope with the identity generator as Increment, i noticed that i was getting an exception (given below along with code) when NHibernate was trying to retrieve the next Identity number.
This seems to be because the new SQLite connection is doing a auto enlist of the current transaction. From what i have heard SQLite only support single write transaction, but should support multiple read's, so i am surprised that i am getting a Database locked exception for a read operation. Did anybody use SQLite with Transaction Scope in this manner.
The same Code works fine if i use a NHibernate Transaction instead of TransactionScope
Code Block:
using (var scope = new TransactionScope())
{
var userRepository =
container.GetInstance<IUserRepository>();
var user = new User();
userRepository.SaveOrUpdate(user);
scope.Complete();
}
Exception:
19:34:19,126 ERROR [ 7] IncrementGenerator [(null)]- could not get
increment value
System.Data.SQLite.SQLiteException: The database file is locked
database is locked
at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
at System.Data.SQLite.SQLiteTransaction..ctor(SQLiteConnection
connection, Boolean deferredLock)
at System.Data.SQLite.SQLiteConnection.BeginTransaction(Boolean
deferredLock)
at System.Data.SQLite.SQLiteConnection.BeginTransaction()
at System.Data.SQLite.SQLiteEnlistment..ctor(SQLiteConnection cnn,
Transaction scope)
at
System.Data.SQLite.SQLiteConnection.EnlistTransaction(Transaction
transaction)
at System.Data.SQLite.SQLiteConnection.Open()
at NHibernate.Connection.DriverConnectionProvider.GetConnection()
at NHibernate.Id.IncrementGenerator.GetNext(ISessionImplementor
session)
19:34:20,063 ERROR [ 7] ADOExceptionReporter [(null)]- The database
file is locked
database is locked
There are two things at play here.
As you mentioned, System.Data.SQLite will auto-enlist in distributed transactions. This is on by default and can turn it off by adding Enlist=no.
The second is that System.Data.SQLite by default creates transactions with an automatic write lock. This is done based on the assumption that if a transaction is started, writes will be done. This can be overridden by starting the transactions with Serializable.ReadCommitted.
The default can also be specified in the connection string using the DefaultIsolationLevel key. Valid values are ReadCommitted and Serializable only. Other isolation levels are not supported by SQLite. ReadCommitted defers the write lock whereas Serializable obtains the write lock immediately.
Unspecified will use the default isolation level specified in the connection string. If no isolation level is specified in the connection string, Serializable is used. Serializable transactions are the default. In this mode, the engine gets an immediate lock on the database, and no other threads may begin a transaction. Other threads may read from the database, but not write.
With a ReadCommitted isolation level, locks are deferred and elevated as needed. It is possible for multiple threads to start a transaction in ReadCommitted mode, but if a thread attempts to commit a transaction while another thread has a ReadCommitted lock, it may timeout or cause a deadlock on both threads until both threads' CommandTimeout's are reached.
I have triggers that manipulate and insert a lot of data into a Change tracking table for audit purposes on every insert, update and delete.
This trigger does its job very well, by using it we are able to log the desired oldvalues/newvalues as per the business requirements for every transaction.
However in some cases where the source table has a lot columns, it can take up to 30 seconds for the transaction to complete which is unacceptable.
Is there a way to make the trigger run asynchronously? Any examples.
You can't make the trigger run asynchronously, but you could have the trigger synchronously send a message to a SQL Service Broker queue. The queue can then be processed asynchronously by a stored procedure.
these articles show how to use service broker for async auditing and should be useful:
Centralized Asynchronous Auditing with Service Broker
Service Broker goodies: Cross Server Many to One (One to Many) scenario and How to troubleshoot it
SQL Server 2014 introduced a very interesting feature called Delayed Durability. If you can tolerate loosing a few rows in case of an catastrophic event, like a server crash, you could really boost your performance in schenarios like yours.
Delayed transaction durability is accomplished using asynchronous log
writes to disk. Transaction log records are kept in a buffer and
written to disk when the buffer fills or a buffer flushing event takes
place. Delayed transaction durability reduces both latency and
contention within the system
The database containing the table must first be altered to allow delayed durability.
ALTER DATABASE dbname SET DELAYED_DURABILITY = ALLOWED
Then you could control the durability on a per-transaction basis.
begin tran
insert into ChangeTrackingTable select * from inserted
commit with(DELAYED_DURABILITY=ON)
The transaction will be commited as durable if the transaction is cross-database, so this will only work if your audit table is located in the same database as the trigger.
There is also a possibility to alter the database as forced instead of allowed. This causes all transactions in the database to become delayed durable.
ALTER DATABASE dbname SET DELAYED_DURABILITY = FORCED
For delayed durability, there is no difference between an unexpected
shutdown and an expected shutdown/restart of SQL Server. Like
catastrophic events, you should plan for data loss. In a planned
shutdown/restart some transactions that have not been written to disk
may first be saved to disk, but you should not plan on it. Plan as
though a shutdown/restart, whether planned or unplanned, loses the
data the same as a catastrophic event.
This strange defect will hopefully be addressed in a future release, but until then it may be wise to make sure to automatically execute the 'sp_flush_log' procedure when SQL server is restarting or shutting down.
To perform asynchronous processing you can use Service Broker, but it isn't the only option, you can also use CLR objects.
The following is an example of an stored procedure (AsyncProcedure) that asynchronous calls another procedure (SyncProcedure):
using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Runtime.Remoting.Messaging;
using System.Diagnostics;
public delegate void AsyncMethodCaller(string data, string server, string dbName);
public partial class StoredProcedures
{
[Microsoft.SqlServer.Server.SqlProcedure]
public static void AsyncProcedure(SqlXml data)
{
AsyncMethodCaller methodCaller = new AsyncMethodCaller(ExecuteAsync);
string server = null;
string dbName = null;
using (SqlConnection cn = new SqlConnection("context connection=true"))
using (SqlCommand cmd = new SqlCommand("SELECT ##SERVERNAME AS [Server], DB_NAME() AS DbName", cn))
{
cn.Open();
using (SqlDataReader reader = cmd.ExecuteReader())
{
reader.Read();
server = reader.GetString(0);
dbName = reader.GetString(1);
}
}
methodCaller.BeginInvoke(data.Value, server, dbName, new AsyncCallback(Callback), null);
//methodCaller.BeginInvoke(data.Value, server, dbName, null, null);
}
private static void ExecuteAsync(string data, string server, string dbName)
{
string connectionString = string.Format("Data Source={0};Initial Catalog={1};Integrated Security=SSPI", server, dbName);
using (SqlConnection cn = new SqlConnection(connectionString))
using (SqlCommand cmd = new SqlCommand("SyncProcedure", cn))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#data", SqlDbType.Xml).Value = data;
cn.Open();
cmd.ExecuteNonQuery();
}
}
private static void Callback(IAsyncResult ar)
{
AsyncResult result = (AsyncResult)ar;
AsyncMethodCaller caller = (AsyncMethodCaller)result.AsyncDelegate;
try
{
caller.EndInvoke(ar);
}
catch (Exception ex)
{
// handle the exception
//Debug.WriteLine(ex.ToString());
}
}
}
It uses asynchronous delegates to call SyncProcedure:
CREATE PROCEDURE SyncProcedure(#data xml)
AS
INSERT INTO T(Data) VALUES (#data)
Example of calling AsyncProcedure:
EXEC dbo.AsyncProcedure N'<doc><id>1</id></doc>'
Unfortunatelly, the assembly requires UNSAFE permission.
I wonder if you could tag a record for the change tracking by inserting into a "too process" table including who did the change etc etc.
Then another process could come along and copy the rest of the data on a regular basis.
There's a basic conflict between "does its job very well" and "unacceptable", obviously.
It sounds to me that you're trying to use triggers the same way you would use events in an OO procedural application, which IMHO doesn't map.
I would call any trigger logic that takes 30 seconds - no, more that 0.1 second - as disfunctional. I think you really need to redesign your functionality and do it some other way. I'd say "if you want to make it asynchronous", but I don't think this design makes sense in any form.
As far as "asynchronous triggers", the basic fundamental conflict is that you could never include such a thing between BEGIN TRAN and COMMIT TRAN statements because you've lost track of whether it succeeded or not.
Create history table(s). While updating (/deleting/inserting) main table, insert old values of record (deleted pseudo-table in trigger) into history table; some additional info is needed too (timestamp, operation type, maybe user context). New values are kept in live table anyway.
This way triggers run fast(er) and you can shift slow operations to log viewer (procedure).
From sql server 2008 you can use CDC feature for automatically logging changes, which is purely asynchronous. Find more details in here
Not that I know of, but are you inserting values into the Audit table that also exist in the base table? If so, you could consider tracking just the changes. Therefore an insert would track the change time, user, extra and a bunch of NULLs (in effect the before value). An update would have the change time, user etc and the before value of the changed column only. A delete has the change at, etc and all values.
Also, do you have an audit table per base table or one audit table for the DB? Of course the later can more easily result in waits as each transaction tries to write to the one table.
I suspect that your trigger is of of these generic csv/text generating triggers designed to log all changes for all table in one place. Good in theory (perhaps...), but difficult to maintain and use in practice.
If you could run asynchronously (which would still require storing data somewhere for logging again later), then you are not auditing and neither do have history to use.
Perhaps you could look at the trigger execution plan and see what bit is taking the longest?
Can you change how you audit, say, to per table? You could split the current log data into the relevant tables.