Can I populate database from Terraform script (for GCP) - sql

I am a newby in terraform and working with GCP project.
I have following terraform code that creates a postgres database for an existing database instance
resource "google_sql_database" "db" {
name = "test_db1"
instance = "already_existing_instance"
project = var.project_id
}
And I would like to understand how (if possible) to run some SQL initial script for that newly created database with terraform script. SQL script should create a table and insert few records.

Terraform is not really designed for schema management, so in the long run you might be better to seek out a tool more specialized for that task (for example, so that it can potentially modify the schema later as your needs change).
If you do still want to take these actions with Terraform, you'd need to seek out a provider which supports the protocol for whichever database engine you've selected. For example, if you're using the Postgres database engine then you could potentially use the cyrilgdn/postgresql provider. (I've not used this provider myself, so I'm just giving this as an example and I can't vouch for it's suitability.)
However, providers like these typically go only as far as creating the object which contains tables (in Postgres terms, the "schema" object). To go more granular than that, creating tables and possibly data, you may not be able to achieve your goals with Terraform alone (unless you write your own provider), and may need to resort to using external software for that final setup step.

Related

How to Create all tables of an RDS instance in a new instance in a new Region

So here is what I need. I have two RDS instances setup in a specific way (Security Groups, Naming convention, Subnet, encryption, etc). I need empty replicas of this database in a new region as the data needs to be stored locally for each region. Also each database is accessed by it's root user using a randomly generated password, but ideally they should not be the same password accross region. Final requirement is that we would like all this to be scriptable (Cloud Formation or a series of AWS CLI calls. We can obvsiously generate the random password ourselves and insert into the script at some point)
So one option is to:
Create a read replica of the DB into a new region. (Setting everything appropiately)
Promote to regular DB.
Change it's master password.
Truncate all tables.
But I would like to know if there is an option to simply create (from zero) each DB in the new region with the new password, setup and everything and then simply run a script (SQL script) that generates the empty tables. This seems like the most script-friendly solution. My problem is that I don't know how I can authomatically generate a SQL script that is simpy a bunch of create for the tables with the exact same structure as the databases already running. Is there a way to do this? Or is my first option better (from a how-easy-it-is-to-script point of view?

Use Liquibase autogenerated xml for Corda Enterprise DB migration

I switched to Corda Enterprise mainly to try how it handles automated database migration.
In the documentation here it says tools-database-manager generates only SQL version of Liquibase script for initial DB and SQL version is Database specific so should not be used for production.
But it is possible to generate the XML also with liqubase cmd using this command:
/snap/bin/liquibase --url="jdbc:h2:tcp://localhost:10039/node" --driver=org.h2.Driver --classpath=/home/corda/Downloads/h2.jar generateChangeLog
which I did, and then I had to remove all the chnagelogs which are related to corda internal tables, and left only the ones that are my own and it seems everything works.
So the question is - may this approach have some hidden dangers that I don't know. Why otherwise Corda team developed tools-database-manager, and why they don't yet support xml generation with tools-database-manager?
And this leads to another question - what if I for example forget to include one of my tables in the initial script? Seems corda does not complain about it. Won't my table be created? Will I be able to ever migrate that table if it is missing in the initial script?
Firstly tools-database-manager is a helper tool available to make it easy for developers to perform database migration.
Let’s say you have 2 nodes in your network, each using a different database. PartyA uses PostgreSQL and PartyB uses Oracle. If PartyA uses this tool to create the migration script by connecting to PostgreSQL, this will out SQL statements specific to PostgreSQL.
Hence this is not portable and hence it's said the generated script is database specific.
Also, you do not want to trust a script and fire it directly on your production database, it contains DDL statements, so it is strongly recommended that every time a script is generated, make sure you know what the script is doing by manually looking into it.
There are a lot of enhancements going on in this space, supporting XML for migration script being one of them.
As mentioned earlier, you should manually look at the migration script. If you forget to add one of your table, Corda will not complain. It will fail sometime later when from within your code you try to access this table.
Yes, you can stop the node and create the table again by adding a create table script.

Bluemix sql database recovery

When I removed my app from the Bluemix dashboard, it removed the associated SQL db as well. I have a script that creates new tables/indexes with our schema name but the free version of SQL database does not support user-defined schema names. The problem is in our code, we need to have our schema name rather than user*** schema name.
Does Bluemix still offer small version of SQL database ? If not, is there a way to recover our database, or is there a way I can rename the user*** schema created by the free version to the name I want?
Unfortunately it is not possible to use a user defined schema name. Anyway as a general rule in development, properties like schema name or connection properties should be parametric, in order to have more flexibility in your solution.
What is preventing to have your SQL to be adapted to the new db instance? You could have a simple script which load it and run on the instance, without any need to use an hardcoded schema name

Common practice to implement SQL scripts application

I have quite old application with current database (on MSSQL but it does not matter). I scripted it completely with all required static data. Now I want to introduce DB change only via update scripts. So each function, each SP will be placed in stand-alone file and all schema update scripts will be stored in files named like 'SomeProduct01_0001' what means that this script belongs to product SomeProduct, sprint 1 and it is first schema update script.
I know that each script must be absolutely re-runnable, but anyway I want to have functionality to combine these scripts into one based on DB version (stored in DB table).
What common best practices there is to handle bunches of update scripts?
What is better - implement version anylyzis in collector
(bat or exe file) or add some SQL header to each file? From other point of view I am already have version - it will consist of sprint identifier and script identifier, not sure that it is ok to duplicate this information in script header.
How to skip file content if user tries to apply it to newer database but keep
availability combine this script with any other to perform updates
of other old database?
How to avoid database conflicts if combined scripts operates columns/table which still does not exists in database but will be created byt this script (for example, in line 10 table created and in line 60 it is used in trigger or constraint, as I know script will not be validated)? Maybe wrap in EXEC('') entire script? What I need to escape besides sigle quote characters?
UPD: As David Tanzer asnwered it is better to use ready solutions for DB migrations, so it may be best solution for cases like mine. It was not and answer exactly for my question, but it is suitable for new solutions.
You don't have to implement this yourself, there are tools that do it. Take a look at dbmaintain, it provides almost exactly the functionality you described:
http://www.dbmaintain.org/overview.html
I know of and have worked with several teams who use it to manage their database schemas in all their environments: Development, Testing, Staging and Production.
http://flywaydb.org/ seems to be another tool to do this, and it has even more features. They even have a comparison of multiple tools on their homepage (including dbmaintain)

Sql Azure CompatibleWithModel not working

In my code I am trying to check if my entity framework Code First model and Sql Azure database are in sync by using the "mycontext.Database.CompatibleWithModel(true)". However when there is an incompatibility this line falls over with the following exception.
"The model backing the 'MyContext' context has changed since the database was created. Either manually delete/update the database, or call Database.SetInitializer with an IDatabaseInitializer instance. For example, the DropCreateDatabaseIfModelChanges strategy will automatically delete and recreate the database, and optionally seed it with new data."
This seems to defeat the purpose of the check as the very check itself is falling over as a result of the incompatibility.
For various reasons I don't want to use the Database.SetInitializer approach.
Any suggestions?
Is this a particular Sql Azure problem?
Thanks
Martin
Please check out the ScottGu blog below:
http://weblogs.asp.net/scottgu/archive/2010/08/03/using-ef-code-first-with-an-existing-database.aspx
Here is what is going on and what to do about it:
When a model is first created, we run a DatabaseInitializer to do things like create the database if it's not there or add seed data. The default DatabaseInitializer tries to compare the database schema needed to use the model with a hash of the schema stored in an EdmMetadata table that is created with a database (when Code First is the one creating the database). Existing databases won’t have the EdmMetadata table and so won’t have the hash…and the implementation today will throw if that table is missing. We'll work on changing this behavior before we ship the fial version since it is the default. Until then, existing databases do not generally need any database initializer so it can be turned off for your context type by calling:
Database.SetInitializer<Production>(null);
Using above code you are no recreating the database instead using the existing one so I don't think using Database.SetInitializer is a concern unless you have some serious thoughts about using it.
More info: Entity Framework Code Only error: the model backing the context has changed since the database was created