I created different models with django 1.8.
Now, to enable other people to have a quickly comprehension, I would create a sql schema from the models or from the migration files (even only from the initial migration file).
Someone knows how does it?
You can squash all migrations for a moment, or delete them for a sec and generate new initial one and then run this command:
https://docs.djangoproject.com/en/1.11/ref/django-admin/#django-admin-sqlmigrate
If you just want to show the database structure to others, I would rather recommend using the graph_models command from django_extensions:
http://django-extensions.readthedocs.io/en/latest/graph_models.html
For example typing
python manage.py graph_models -a -g models.png
creates a graph with the individual models as nodes and their relations as arcs (assuming you have graphviz installed). You can also create a dot-file and render it however you like
Related
I have a DBT project that is mostly comprised of models for views over snowflake external tables.Every model view is triggered with a seperate dbt run statement concurrently.
dbt run --models model_for_view_1
I have one other model in the dbt project which materializes to a table that uses these views. I trigger this model in a separate DAG in airflow using the same DBT run statement as above. It uses no ref or source statement that connects it to the views.
I noticed recently that this table model is getting built by DBT whenever I build the view models. I thought it was because DBT was making an inference that this was a referenced model but after some experimentation in which I even set the table model SQL as something like SELECT 1+1 as column1, it was still getting built. I have placed it in a different folder in the dbt project, renamed the file etc. No joy. have no idea why running the other models is causing this unrelated model to be built. The only connection to the view models is that they share the same schema in the database. What is triggering this model to be built?
Selection syntax can be finicky, because there are many ways to select the models. From the docs:
The --select flag accepts one or more arguments. Each argument can be one of:
a package name
a model name
a fully-qualified path to a directory of models
a selection method (path:, tag:, config:, test_type:, test_name:)
(note that --models was renamed --select in v0.21, but --models has the same behavior)
So my guess is that your model_for_view_1 name isn't unique, and is either shared with your project (acting as a package in this case) or the directory that it is in.
So if your project looks like:
models
|- some_name
|- some_name.sql # the view
|- another_name.sql # the table
dbt run --models some_name will run the code in both some_name.sql and another_name.sql, since it is selecting the directory called some_name.
Would you be able to share a bit more context here.
Which version of dbt is your project on?
Would it be possible to share how the models look (while removing any sensitive information).
It is rather difficult to tell what is triggering this unexpected behaviour without these info.
I'm trying to understand the different types of migration paths we can choose when developing an ASP.NET Core 1.0 application with EF Core. When I created my first Core application I noticed it generated a ApplicationDbContextModelSnapshot class that uses a ModelBuilder to build the model.
Then I read that if I need to add a table to the database, I need to create the new model and run the command line to generate the migration file and update the database. Ok, I get it up to this point.
But when I do that, I notice that the ApplicationDbContextModelSnapshot class gets updated too.
1) Does that mean I cannot modify this ApplicationDbContextModelSnapshot class since it looks like it gets regenerated each time?
2) Should I use Data Annotations to build my model or should I use Fluent API which tells me to build my model in the ApplicationDbContext class? Huh? another file that builds the model?
I'm seeing three different ways of working with the database here, the snapshot class, data annotations, and fluent API. I'm confused because today, I made a mistake in my last migration file so I deleted the file, dropped the database and reran the database update.
But by doing that I got errors similar to:
The index 'IX_Transaction_GiftCardId' is dependent on column 'GiftCardId'.
ALTER TABLE ALTER COLUMN GiftCardId failed because one or more objects access this column.
So naturally I was wondering if I had to modify the ApplicationDbContextModelSnapshot class.
What is the path I should be taking when it comes to migrations or database updates because these three paths are confusing me.
I have run into this issue before when I create migrations, make model changes, create new migrations, and try to update the database. The root cause is when keys are being changed and relationships are not dropped and are not added back or do not exist.
You have two options
Easy Method
The easiest way is also the most destructive way and only possible in a dev environment.
Delete all migrations, drop the database, create new migrations and run 'update-database'.
Hard/Safest Method
This is the most time consuming method. I recommend do this in a local integration branch first, pushing it to a remote integration, and then production.
Open the migration file, ie 20160914173357_MyNewMigration.cs.
Drop all indexes in order
Drop/Add/Edit table schemas
Add all indexes back.
For either method, just be sure to test and test again.
Do not modify ApplicationDbContextModelSnapshot. It is a design-time artifact, and should only be modified in the case of a merge conflict.
To update the model, always use data annotations or the fluent API.
For more information on the EF Migrations workflow, see Code First Migrations. It's for EF6, but most of the information is still relevant.
I work in an Oracle instance that has hundreds of schemas and multiple developers. We have a development instance where developers can integrate their work before test or production.
We want to have source control for all the DDL run in this integrated development database. Currently this is done through a product Red Gate which we run manually after we make a change to the database. Redgate finds the changes between what is in the schema and what was last checked into source control and makes a script of the differences and puts this into source control.
The problem however is of course that running regdate can take some time and people run it infrequently or not at all for small changes. Also redgate will only look in one schema at a time and it would be VERY time consuming to manually run it against all schemas to guarantee that they are up to date. However if the source controlled code cannot be relied upon it becomes less useful...
What would seem to be ideal would be to have some software that could periodically (even once a day), or when triggered by DDL being run, update the source control (preferably github as this is used by other teams) from all the schemas.
I cannot seem to see any existing software which can be simply used to do this.
Is there a problem with doing this? (there is no need to address multiple developers overwriting each others work on the same day as we have this covered in a separate process) Is anyone doing this? Can anyone recommend a way to do this?
We do this with help of a PL/SQL function, a python script and a shell script:
The PL/SQL function can generate the DDL of a whole schema and returns this as CLOB
The python script connects to the database, fetches the DDL and stores it in files
The shell script runs the Source Control to add the modifications (we use Bazaar here).
You can see the scripts on PasteBin:
The PL/SQL function is here: http://pastebin.com/AG2Fa9zL
The python program (schema_exporter.py): http://pastebin.com/nd8Lf0gK
The shell script:
The shell script:
python schema_exporter.py
d=$(date +%Y-%m-%d__%H_%M_%S)
bzr add
bzr st | grep -q -E 'added|modified' && commit -m "Database objects on $d"
exit 0
This shell script is configured to run from cron every day.
Being in the database version control space for 5 years (as director of product management at DBmaestro) and having worked as a DBA for over two decades, I can tell you the simple fact that you cannot treat the database objects as you treat your Java, C# or other files and save the changes in simple DDL scripts.
There are many reasons and I'll name a few:
Files are stored locally on the developer’s PC and the change s/he
makes do not affect other developers. Likewise, the developer is not
affected by changes made by her colleague. In database this is
(usually) not the case and developers share the same database
environment, so any change that were committed to the database affect
others.
Publishing code changes is done using the Check-In / Submit Changes /
etc. (depending on which source control tool you use). At that point,
the code from the local directory of the developer is inserted into
the source control repository. Developer who wants to get the latest
code need to request it from the source control tool. In database the
change already exists and impacts other data even if it was not
checked-in into the repository.
During the file check-in, the source control tool performs a conflict
check to see if the same file was modified and checked-in by another
developer during the time you modified your local copy. Again there
is no check for this in the database. If you alter a procedure from
your local PC and at the same time I modify the same procedure with
code form my local PC then we override each other’s changes.
The build process of code is done by getting the label / latest
version of the code to an empty directory and then perform a build –
compile. The output are binaries in which we copy & replace the
existing. We don't care what was before. In database we cannot
recreate the database as we need to maintain the data! Also the
deployment executes SQL scripts which were generated in the build
process.
When executing the SQL scripts (with the DDL, DCL, DML (for static
content) commands) you assume the current structure of the
environment match the structure when you create the scripts. If not,
then your scripts can fail as you are trying to add new column which
already exists.
Treating SQL scripts as code and manually generating them will cause
syntax errors, database dependencies errors, scripts that are not
reusable which complicate the task of developing, maintaining,
testing those scripts. In addition, those scripts may run on an
environment which is different from the one you though it would run
on.
Sometimes the script in the version control repository does not match
the structure of the object that was tested and then errors will
happen in production!
There are many more, but I think you got the picture.
What I found that works is the following:
Use an enforced version control system that enforces
check-out/check-in operations on the database objects. This will
make sure the version control repository matches the code that was
checked-in as it reads the metadata of the object in the check-in
operation and not as a separated step done manually. This also allow
several developers to work in parallel on the same database while
preventing them to accidently override each other code.
Use an impact analysis that utilize baselines as part of the
comparison to identify conflicts and identify if a difference (when
comparing the object's structure between the source control
repository and the database) is a real change that origin from
development or a difference that was origin from a different path and
then it should be skipped, such as different branch or an emergency
fix.
Use a solution that knows how to perform Impact Analysis for many
schemas at once, using UI or using API in order to eventually
automate the build & deploy process.
An article I wrote on this was published here, you are welcome to read it.
To me it seems like your way of working is backwards: developers run DDL against the DB in an unordered fashion and then you need an automated tool for inferring the changes (and the DDL) that was run.
The process would be in better control if you did the following instead:
Developers write DDL as SQL scripts, preferably using a migration tool such as Flyway (http://flywaydb.org/documentation/migration/sql.html).
Migration scripts are checked into version control
Migration scripts are periodically run against the DB (e.g. by the migration tool)
In this workflow, the DB would only get altered through automated migration scripts and no-one is allowed to do changes manually. Could this work for you?
(I develop the Oracle tools for Redgate)
Actually using the tools you can already what I think you're asking for using Schema Compare for Oracle.
You can compare multiple schemas either in the UI or via the command line - I think what you're after is automating the command line tool which can create difference scripts, sync between source and destination (live, snapshot or scripts) and generate reports.
You can automate the command line to sync to a scripts folder which is your source code checkout and then subsequently run a command to commit the changes.
I think that's all good :)
We built a commerical tool that bridges Oracle with Git. It helps you manage your database objects with Git. Basically, the database becomes the working directory for the developer. You can perform git operations in the database such as reset, commit, branch, merge etc... and the database code is updated automatically. It might be worth taking a look: https://www.gitora.com
sometimes when working in rails, I work on several things at once using git branches
sometimes, I'd like to test new ideas by implementing them and testing how and if they work accordingly. This involves sometimes adding models and migrations.
When switching branches, however, the migrations were already migrated to the DB and they stay, causing problems later on..
Is there a way to work with several branches and each to have different migration files, and before starting to work on a branch to "soft reset" the db only to the current migration files without losing data?
Normally, in development, I need some sample data that I keep in seed.rb which enables me to recreate the db, its structure and the sample data with a rake task.
Another thing I did was to keep more than one database. I would then just manually change the entry in database.yml according to the current git branch.
My app is in beta, and I've been doing limited testing of a feature that involves a new model. After a fair amount of testing I had to make a structural change that makes the old data non-functional.
What I need to do is just drop and recreate one table. I know that I could do this in a migration, but that seems like such a hack. In a local dev copy I would just use db:reset, but in the beta app I don't want to lose data in any tables except this one.
Is this a simple way to instruct a production app to drop and recreate a single table. In my case, I'm deploying with Heroku, in case that affects how you would solve this issue.
To empty a table on Heroku without changing the schema, in your application's directory:
$ heroku run console
Ruby console for myap.heroku.com
>> ModelName.delete_all
>> exit
I know that I could do this in a migration, but that seems like such a hack.
It's not a hack. It's precisely what migrations are designed to do.
You need to rerun the migration for that table to make structural changes. I haven't used ActiveRecord before, but I'd also delete the data in the table using ModelName.delete_all from heroku console.
heroku run console
irb(main):001:0> ModelName.delete_all
And you are done.