Why do migrations fail on test, but not on migrate? - django-testing

I'm using Django==1.7, and have four applications:
frontend
game
geo
people
The apps settings is like this:
INSTALLED_APPS = (
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'fandjango',
'people',
'geo',
'game',
'frontend'
)
And the database settings are:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'hoods_raising',
'USER': 'hoods_raising',
'PASSWORD': 'hr$nestor$123',
'HOST': 'localhost',
'TEST_CHARSET': 'utf8mb4'
}
}
My applications have migrations and tests:
game
migrations
0001_initial.py
geo
migrations
0001_initial.py
tests.py
people
migrations
0001_initial.py
0002_install_data.py
Many files were omitted to narrow the problem (I'll expand the question with more files if needed), e.g. models.py, views.py.
The contents of 0002_install_data.py are:
class Migration(migrations.Migration):
dependencies = [
('people', '0001_initial'),
]
operations = [
migrations.RunPython(NamesInstaller(), lambda apps, schema_editor: None)
]
If I run manage.py migrate to install the database, everything works as expected.
If I run manage.py test to run the tests, the first step will be the test database installation. Something weird happens:
The first migration to be executed is 0002_install_data. Other tables are never created (e.g. auth tables, geo tables, game tables, fandjango tables, ...) and migration 0001_initial in people is not run.
For such reason, A dependency error occurs in 0002_install_data (it says that 0001_initial does not exist).
KeyError: u"Migration people.0002_install_data dependencies references nonexistent parent node (u'people', u'0001_initial')"
Why could this be happening? Why wouldn't the test command not arrange correctly the application migrations? (this does not happen on manage.py migrate).

I solved it. That issue appeared because I messed with Squashed migrations: since I did not deploy this beforehand to a productive environment I took the freedom to delete the replaced migrations (and only keep the squashed).
When you delete the replaced migrations and keep the squashed and then do migrate, everything will work as expected. However the squashed migration will reference the replaced migrations if you run the tests and so it will fail.
Unfortunately, I named the squashed migration 0001_initial, like the first migration, which mislead me to think it was a dependency problem with an existent file.
So: If you want to squash a migration ensure you know what you're doing and don't delete previous migrations unless:
You know nobody will use them again (i.e. no instance is "in the middle" of the squashed migration path).
You DELETE the replacement directive in the squashed migration. Otherwise tests will fail because database will not be able to setup.

Related

How to specify model schema when referencing another dbt project as a package? (dbt multi-repo setup)

We're using a dbt multi-repo setup with different projects for different business areas. We have several projects, something like this:
dbt_dwh
dbt_project1
dbt_project2
The dbt_dwh project contains models which we plan to reference in projects 1 and 2 (we have ~10 projects that would reference the dbt_dwh project) by way of installing git packages. Ideally, we'd like to be able to just reference the models in the dbt_dwh project (e.g.
SELECT * FROM {{ ref('dbt_dwh', 'model_1') }}). However, each of our projects sits in it's own database schema and this causes issue upon dbt run because dbt uses the target schema from dbt_project_x, where these objects don't exist. I've included example set-up info below, for clarity.
packages.yml file for dbt_project1:
packages:
- git: https://git/repo/url/here/dbt_dwh.git
revision: master
profiles.yml for dbt_dwh:
dbt_dwh:
target: dwh_dev
outputs:
dwh_dev:
<config rows here>
dwh_prod:
<config rows here>
profiles.yml for dbt_project1:
dbt_project1:
target: project1_dev
outputs:
project1_dev:
<config rows here>
project1_prod:
<config rows here>
sf_orders.sql in dbt_dwh:
{{
config(
materialized = 'table',
alias = 'sf_orders'
)
}}
SELECT * FROM {{ source('salesforce', 'orders') }} WHERE uid IS NOT NULL
revenue_model1.sql in dbt_project1:
{{
config(
materialized = 'table',
alias = 'revenue_model1'
)
}}
SELECT * FROM {{ ref('dbt_dwh', 'sf_orders') }}
My expectation here was that dbt would examine the sf_orders model and see that the default schema for the project it sits in (dbt_dwh) is dwh_dev, so it would construct the object reference as dwh_dev.sf_orders.
However, if you use command dbt run -m revenue_model1 then the default dbt behaviour is to assume all models are located in the default target for dbt_project1, so you get something like:
11:05:03 1 of 1 START sql table model project1_dev.revenue_model1 .................... [RUN]
11:05:04 1 of 1 ERROR creating sql table model project1_dev.revenue_model1 ........... [ERROR in 0.89s]
11:05:05
11:05:05 Completed with 1 error and 0 warnings:
11:05:05
11:05:05 Runtime Error in model revenue_model1 (folder\directory\revenue_model1.sql)
11:05:05 404 Not found: Table database_name.project1_dev.sf_orders was not found
I've got several questions here:
How do you force dbt to use a specific schema on runtime when using dbt ref function?
Is it possible to force dbt to use the default parameters/settings for models inside the dbt_dwh project when this Git repo is installed as a package in another project?
Some points to note:
All objects & schemas listed above sit in the same database
I know that many people recommend mono-repo set-up to avoid exactly this type of scenario, but switching to a mono-repo structure is not feasible right now, as we are already fully invested in multi-repo setup
Although it would be feasible to create source.yml files in each of the dbt projects to reference the output objects of the dbt_dwh project, this feels like duplication of effort and could result in different versions of the same sources.yml file across projects
I appreciate it is possible to hard-code the output schema in the dbt config block, but this removes our ability to test in dev environment/schema for dbt_dwh project
I managed to find a solution so I'll answer my own question in case anybody else runs up against the same issue. Unfortunately this is not documented anywhere that I can find, however, a throw-away comment in the dbt Slack workspace sparked an idea that allowed me to find the/a solution (I'll post the message if I manage to find it again, to give credit where it's due).
To fix this is actually very simple, you just need to add the project being imported to your profiles.yml file and specify the schema. For our use case this is fine as we only have 1 schema we use.
profiles.yml for dbt_project1:
models:
db_project_1:
outputs:
project1_dev:
<configs here>
project1_prod:
<configs here>
dbt_dwh:
+schema: [[schema you want these models to run into]]
<configs here>
The advantages with this approach are:
When you generate/serve dbt docs it allows you to see the upstream lineage from the upstream project
If there are any upstream dependencies in your upstream project you can run this using dbt run -m +model_name (this can be super handy)
If you don't want this behaviour then you can use dbt run -m +model_name --exclude dbt_dwh (for example) to prevent models in your upstream project from running.
I haven't yet figured out if it is possible to use the default parameters/settings for models inside the upstream project (in this case dbt_dwh) but I will edit this answer if I find a way.

Spring boot flyway migrations

I am using flyway for migrations in my Spring boot application. I have around 5 migration scripts with names in the below fashion:
V1__initialmigrations.sql
V2__alter_message_table.sql
When the migrations run and I see the data in 'flyway_schema_history' table, the data looks good for all migration scripts except the very first one for which under the 'script' column, the value is '<< Flyway Baseline >>' rather than the name of the script unlike other rows. Also, the 'installed_by' column has the value 'null' for this very row while other have the user name that I have in my Spring boot yml file. Also, the 'checksum' is null as well.
The only flyway related properties in the spring env yml file are :
spring:
flyway:
baseline-on-migrate: true
enabled: true
I am not sure if this is the right behavior. Any inputs would be appreciated.
You only use the baseline-on-migrate property if you've created a new baseline after the initial application. Flyway ignores all script with a version below what is set in baseline-version (default 1).
For example in applications i've worked on we had years of migration script backlog. we replaced all of them with a single file with current db structure and enabled baseline-on-migrate with baseline-version: X.1 where our new baseline script was VX_0_0.
See also official documentation on baseline: https://flywaydb.org/documentation/concepts/baselinemigrations

Running manifests (classes) from a task or plan in Puppet Enterprise

TL;DR
In Puppet Enterprise, how do I run a manifest (testpp.pp) from a task or plan (not Bolt).
plan base_windows::testplan (
  TargetSpec $targets,
  Optional[String] $contents = undef,
  String $filename,
){
  $apply_prep($targets)
  $apply_results = apply($targets, '_catch_errors' => true) {
    class { 'base_windows::testpp': }
  }
  $apply_results.each | $result | {
    notice($result.report)
  }
}
apply_prep seems to succeed, but apply is failing with the following error:
{
"msg" : "Evaluation Error: Unknown function: 'report'. (file: /opt/puppetlabs/server/data/orchestration-services/code/environments/development/modules/base_windows/plans/testplan.pp, line: 16, column: 19)",
"kind" : "bolt/plan-failure",
"details" : {
"class" : "Bolt::PAL::PALError"
}
}
If I change the code to:
plan base_windows::testplan (
  TargetSpec $targets,
  Optional[String] $contents = undef,
  String $filename,
){
  apply_prep($targets)
  $apply_results = apply($targets, '_catch_errors' => true) {
# Is this how to call a class? I cannot find an example.    
class { 'base_windows::testpp': }
  }
  $apply_results.each |$result| {
$target = $result.target.name
if $result.ok {
  out::message("${target} returned a value: ${result.value}")
} else {
 out::message("${target} errored with a message: ${result.error.message}")
}
  }
}
The plan tells me it has failed, but there are no errors in the node's report. In fact, there is no entry for the time the plan was executed.
I cannot find any examples on how to call a class from a plan, so the above apply() is a guess, based on this documentation.
I have installed the puppetlabs_reboot module and successfully ran a plan using it, therefore, I conclude my system is set up correctly, it's just my code that is wrong.
Background
I may be going about this all wrong, so here is some background to the problem. Currently, I have a series of manifests that install various packages from the public Chocolatey repository depending on a node's classification. Package definitions are stored in Hiera data and each package' version is set to latest. At the end of the Package{} resource, some manifests include a reboot.
These manifests are used to provision new nodes and keep existing nodes up-to-date with the latest package version.
The Puppet agent is set to run once per hour and if the source package is updated in the Chocolatey repo, on the next Puppet run, the manifest will update the package, rebooting the node, if required.
Goal
New nodes are provisioned with the latest package version.
Prevent package updates at undetermined times on existing nodes.
Continue to allow Puppet agent runs every hour.
Make use of existing manifests.
Ideas
Split out the package{} code from the profile manifest and place them in tasks / plans, allowing packages to be updated out-of-hours.
Specify the actual package version in Hiera. Although this is more declarative and idempotent, it means keeping an eye on over 100 package version. I guess it would be fairly simple to interrogate the Chocolatey repos with code to pull the latest version number, but even so I am no better off.
Create a task with a script that runs choco upgrade all, however, the next Puppet run would revert package versions according to the version defined in Hiera, meaning Hiera still needs to be kept up-to-date.
Problems
As per the main crux of this question, how do I run manifests (classes) from plans? If I understand correctly, tasks are for ad-hoc scripts, whereas plans can run tasks and manifests. As a lot of time has been invested in writing manifests, I would prefer not to rewrite all my manifests as scripts.
I am confused by the Puppet documentation as it seems to switch between PE and Bolt syntax. I am using Puppet Enterprise where Puppet says they don't recommend using Bolt but their examples seem to site Bolt commands.
No errors in the node' report. apply_prep() reports executed successfully, albeit taking far longer to execute than puppetlabs_reboot module, but apply() results in a failure, but nothing is logged in the node's reports.
Using puppetlabs_reboot module as a reference, it appears their plan uses a bunch of tasks. It appears that they don't use apply() to run their reboot{} class. Is this not duplicating the work?
If anyone has any suggestions or ideas, I'd be grateful if you could share.
I've got it to work. The class I was trying to run, required parameters that I hadn't provided!
plan base_windows::testplan (
TargetSpec $targets,
Optional[String] $contents = undef,
String $filename,
){
apply_prep($targets)
$apply_results = apply($targets, '_catch_errors' => true) {
class { 'base_windows::testpp':
filename => $filename,
contents => $contents,
}
}
}
# Output the whole result_set in the PE console
return $apply_results
I found this out using the logs.
Turn on debug level logging in /etc/puppetlabs/puppetserver/logback.xml (root level="debug")
Tail the following logs:
tail -f /var/log/puppetlabs/bolt-server/bolt-server.log
tail -f /var/log/puppetlabs/puppetserver/puppetserver.log | grep -B 5 -A 5 'testplan'
tail -f /var/log/puppetlabs/orchestration-services/orchestration-services.log

Django unit tests ask to delete MariaDB DB: (1007, "Can't create database 'test_x_django'; database exists")

I have django unit tests creating a MariaDB database for tests.
I am accessing the database with a user and password in the settings file and keep getting this error:
Type 'yes' if you would like to try deleting the test database 'test_x_django', or 'no' to cancel: Got an error creating the test database: (1007, "Can't create database 'test_x_django'; database exists")
Which makes me type 'yes' every time that I run unit tests - specifically after stopping tests in the middle (not allowing django to delete the database?).
I have tried a lot of solutions, including: trying to use the root user, giving django_user all permissions to 'test_x_django' db, giving django_user permissions to createdb. I have not found a solution online for MariaDB, in this case.
The configuration in my settings.py file is:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'x_django',
'USER': 'django_user',
'PASSWORD': 'xxxx',
'HOST': 127.0.0.1,
'PORT': '3306'
}
}
Is there a solution to make django stop asking the question, and delete the Database automatically?
Just pass --keepdb option. python manage.py test --keepdb if you want to keep it. If you want to destroy it pass --noinput python manage.py test --noinput.
See docs https://docs.djangoproject.com/en/2.2/topics/testing/overview/#the-test-database

Running one specific Laravel migration (single file)

I don't want to run All Outstanding Migrations on laravel 4. I have 5 migrations. Now I just want to run one migration.
instead of doing : php artisan migrate
I would like to run one specific migration like : php artisan migrate MY_MIGRATION_TO_RUN
Looks like you're doing it wrong.
Migrations were made to be executed by Laravel one by one, in the exact order they were created, so it can keep track of execution and order of execution. That way Laravel will be able to SAFELY rollback a batch of migrations, without risking breaking your database.
Giving the user the power to execute them manually, make it impossible to know (for sure) how to rollback changes in your database.
If your really need to execute something in your database, you better create a DDL script and manually execute it on your webserver.
Or just create a new migration and execute it using artisan.
EDIT:
If you need to run it first, you need to create it first.
If you just need to reorder them, rename the file to be the first. Migrations are created with a timestemp:
2013_01_20_221554_table
To create a new migration before this one you can name it
2013_01_19_221554_myFirstMigration
You can put migrations in more folders and run something like:
php artisan migrate --path=/app/database/migrations/my_migrations
Just move the already run migrations out of the app/config/database/migrations/ folder . Then run the command php artisan migrate . Worked like a charm for me .
A nice little snippet to ease any fears when running Laravel 4 migrations php artisan migrate --pretend . This will only output the SQL that would have been run if you ran the actual migration.
It sounds like your initial 4 migrations have already been run. I would guess that when you php artisan migrate it will only run the new, recent migration.
Word of advice: makes sure all of your up() and down() work how you expect them to. I like to run up(), down(), up() when I run my migrations, just to test them. It would be awful for you to get 5-6 migrations in and realize you can't roll them back without hassle, because you didn't match the down() with the up() 100% percent.
Just my two cents! Hope the --pretend helps.
The only way to re run a migrattion is a dirty one. You need to open your database and delete the line in the migrations table that represents your migration.
Then run php artisan migrate again.
You can create a separate directory for your migrations from your terminal as follows:
mkdir /database/migrations/my_migrations
And then move the specific migration you want to run to that directory and run this command:
php artisan migrate --path=/database/migrations/my_migrations
Hope this helps!
If you want to run(single file) migration in Laravel you would do the following:
php artisan migrate --path=/database/migrations/migrations_file_name
eg.
C:\xampp\htdocs\laravelv3s>php artisan migrate --path=/database/migrations/2020_02_14_102647_create_blogs_table.php
I gave this answer on another post, but you can do this: run artisan migrate to run all the migrations, then the following SQL commands to update the migrations table, making it look like the migrations were run one at a time:
SET #a = 0;
UPDATE migrations SET batch = #a:=#a+1;
That will change the batch column to 1, 2, 3, 4 .. etc. Add a WHERE batch>=... condition on there (and update the initial value of #a) if you only want to affect certain migrations.
After this, you can artisan migrate:rollback as much as is required, and it'll step through the migrations one at a time.
You can use below solution:
create your migration.
check your migration status like: php artisan migrate:status.
copy the full name of new migration and do this like: php artisan migrate:rollback --path:2018_07_13_070910_table_tests.
and then do this php artisan migrate.
finally, you migrate specific table.
Goodluck.
If you want to run your latest migration file you would do the following:
php artisan migrate
You can also revert back to before you added the migration with:
php artisan migrate: rollback
There is one easy way I know to do this can only be available for you on just local host
Modify your migration file as needed
open your phpMyAdmin or whatever you use to see your database table
find the desired table and drop it
find migrations table and open it
in this table under migration field find your desired table name and delete its row
finally run the command php artisan migrate from your command line or terminal. this will only migrate that tables which not exists in the migrations table in database.
This way is completely safe and will not make any errors or problems while it looks like un-professional way but it still works perfectly.
good luck
If it's just for testing purposes, this is how i do it:
For my case, i have several migrations, one of them contains App-Settings.
While I'm testing the App and not all of the migrations are already setup i simply move them into a new folder "future". This folde won't be touched by artisan and it will only execute the migration you want.
Dirty workaround, but it works...
I have same problem.
Copy table creation codes in first migration file something like below:
public function up()
{
Schema::create('posts', function(Blueprint $table){
$table->increments('id');
// Other columns...
$table->timestamps();
});
Schema::create('users', function (Blueprint $table) {
$table->increments('id');
// Other columns...
$table->softDeletes()->nullable();
});
}
Also you can change(decrease) batch column number in migrations table ;)
And then run php artisan migrate.
Throw an exception in a migration, if you don't want to apply it, and it would stop the whole process of migration.
Using this approach you can split your bunch of migrations into steps.
Working in Laravel 8+
Run single specific migration:
php artisan migrate --path=/database/migrations/yourfilename.php
Run all migrations:
php artisan migrate
so simple...! just go to your migration folder. move all migration files into another folder. then return all migration one by one into migration folder and run migration for one of them(php artisan). when you insert bad migration file into master migration folder and run php artisan migrate in command prompt will be error.
I used return on line 1 so the previous dbs are retained as it is.
<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreateUsersTable extends Migration
{
/**
* Run the migrations.
*
* #return void
*/
public function up()
{
return; // This Line
Schema::create('users', function (Blueprint $table) {
$table->increments('id');
$table->string('name', 50);
$table->string('slug', 50)->unique();
$table->integer('role_id')->default(1);
$table->string('email', 50)->unique();
$table->timestamp('email_verified_at')->nullable();
$table->string('mobile', 10)->unique();
$table->timestamp('mobile_verified_at')->nullable();
$table->text('password');
$table->integer('can_login')->default(1);
$table->rememberToken();
$table->timestamps();
});
}
/**
* Reverse the migrations.
*
* #return void
*/
public function down()
{
return;// This Line
Schema::dropIfExists('users');
}
}
This is a bad approach of which I use.. I'll delete other migration files except the specific file I want to migrate then run PHP artisan migrate after migration is completed I'll goto my trash bin and restore the deleted files
For anybody still interested in this, Laravel 5 update: Laravel has implemented the option to run one migration file at a time (in version 5.7).
You can now run this:
php artisan migrate --path=/database/migrations/my_migration.php (as answered here)
Because the Illuminate\Database\Migrations\Migrator::getMigrationFiles() now contains this code:
return Str::endsWith($path, '.php') ? [$path] : $this->files->glob($path.'/*_*.php');
(see the source code.)
But in my usecase, I actually wanted to run a set of migrations at the same time, not just one, or all.
So I went the Laravel way and registered a different implementation of the Migrator, which decides which files to use:
/**
* A migrator that can run multiple specifically chosen migrations.
*/
class MigrationsSetEnabledMigrator extends Migrator
{
/**
* #param Migrator $migrator
*/
public function __construct(Migrator $migrator)
{
parent::__construct($migrator->repository, $migrator->resolver, $migrator->files);
// Compatibility with versions >= 5.8
if (isset($migrator->events)) {
$this->events = $migrator->events;
}
}
/**
* Get all of the migration files in a given path.
*
* #param string|array $paths
* #return array
*/
public function getMigrationFiles($paths)
{
return Collection::make($paths)->flatMap(function ($path) {
return Str::endsWith($path, ']') ? $this->parseArrayOfPaths($path) :
(Str::endsWith($path, '.php') ? [$path] : $this->files->glob($path . '/*_*.php'));
})->filter()->sortBy(function ($file) {
return $this->getMigrationName($file);
})->values()->keyBy(function ($file) {
return $this->getMigrationName($file);
})->all();
}
public function parseArrayOfPaths($path)
{
$prefix = explode('[', $path)[0];
$filePaths = explode('[', $path)[1];
$filePaths = rtrim($filePaths, ']');
return Collection::make(explode(',', $filePaths))->map(function ($filePath) use ($prefix) {
return $prefix . $filePath;
})->all();
}
}
We have to register it into the container as 'migrator' (to be accessible as $app['migrator']), because that is how Migrate command accesses it when itself is being registered into the IoC. To do so, we put this code into a service provider (in my case, it is a DatabaseServiceProvider):
public function register()
{
$this->app->extend('migrator', function ($migrator, $app) {
return new MultipleSpecificMigrationsEnabledMigrator($migrator);
});
// We reset the command.migrate bind, which uses the migrator - to
// force refresh of the migrator instance.
$this->app->instance('command.migrate', null);
}
Then you can run this:
php artisan migrate --path=[database/migrations/my_migration.php,database/migrations/another_migration.php]
Notice the multiple migration files, separated by a comma.
It is tested and working in Laravel 5.4 and should be Laravel 5.8 compatible.
Why?
For anyone interested: the usecase is updating the version of database along with it's data.
Imagine, for example, that you wanted to merge the street and house number of all users into new column, let's call it street_and_house. And imagine you wanted to do that on multiple installations in a safe and tested way - you would probably create a script for that (in my case, I create data versioning commands - artisan commands).
To do such an operation, you first have to load the users into memory; then run the migrations to remove the old columns and add the new one; and then for each user assign the street_and_house=$street . " " . $house_no and save the users. (I am simplifying here, but you can surely imagine other scenarios)
And I do not want to rely on the fact that I can run all the migrations at any given time. Imagine that you wanted to update it from let's say 1.0.0 to 1.2.0 and there were multiple batches of such updates – performing any more migrations could break your data, because those migrations must be handled by their own dedicated update command. Therefore, I want to only run the selected known migrations which this update knows how to work with, then perform operations on the data, and then possibly run the next update data command. (I want to be as defensive as possible).
To achieve this, I need the aforementioned mechanism and define a fixed set of migrations to be run for such a command to work.
Note: I would have preferred to use a simple decorator utilizing the magic __call method and avoid inheritance (a similar mechanism that Laravel uses in the \Illuminate\Database\Eloquent\Builder to wrap the \Illuminate\Database\Query\Builder), but the MigrateCommand, sadly, requires an instance of Migrator in it's constructor.
Final note: I wanted to post this answer to the question How can I run specific migration in laravel , as it is Laravel 5 - specific. But I can not - since that question is marked as a duplicate of this one (although this one is tagged as Laravel 4).
You may type the following command:
php artisan migrate --help
...
--path[=PATH] The path(s) to the migrations files to be executed (multiple values allowed)
...
If it does show an option called "--path" (like the upper example) that means your Laravel version supports this parameter. If so, you're in luck can then you can type something like:
php artisan migrate --path=/database/migrations/v1.0.0/
Where "v.1.0.0" is a directory that exists under your "/database/migrations" directory that holds those migrations you want to run for a certain version.
If not, then you can check in your migrations table to see which migrations have already been run, like this:
SELECT * FROM migrations;
And then move out of your "/database/migrations" folder those which were executed. By creating another folder "/databases/executed-migrations" and moving your executed migrations there.
After this you should be able to execute:
php artisan migrate
Without any danger to override any existing table in your schema/database.
(*) example for Windows:
php artisan migrate --path=database\migrations\2021_05_18_121604_create_service_type_table.php