When developing whether its Web or Desktop at which point should a developer switch from SQLite, MySQL, MS SQL, etc
It depends on what you are doing. You might switch if:
You need more scalability or better performance - say from SQLite to SQL Server or Oracle.
You need access to more specific datatypes.
You need to support a customer that only runs a particular database.
You need better DBA tools.
Your application is using a different platform where your database no longer runs, or it's libraries do not run.
You have the ability/time/budget to actually make the change. Depending on the situation, the migration could be a bigger project than everything in the project up to that point. Migrations like these are great places to introduce inconsistencies, or to lose data, so a lot of care is required.
There are many more reasons for switching and it all depends on your requirements and the attributes of the databases.
You should switch databases at milestone 2.3433, 3ps prior to the left branch of dendrite 8,151,215.
You should switch databases when you have a reason to do so, would be my advice. If your existing database is performing to your expectations, supports the load that is being placed on it by your production systems, has the features you require in your applications and you aren't bored with it, why change? However, if you find your application isn't scaling, or you are designing an application that has high load or scalability requirements and your research tells you your current database platform is weak in that area, or, as was already mentioned, you need some spatial analysis or feature that a particular database has, well there you go.
Another consideration might be taking up the use of a database agnostic ORM tool that can allow you to experiment freely with different database platforms with a simple configuration setting. That was the trigger for us to consider trying out something new in the DB department. If our application can handle any DB the ORM can handle, why pay licensing fees on a commercial database when an open source DB works just as well for the levels of performance we require?
The bottom line, though, is that with databases or any other technology, I think there are no "business rules" that will tell you when it is time to switch - your scenario will tell you it is time to switch because something in your solution won't be quite right, and if you aren't at that point, no need to change.
BrianLy hit the nail on the head, but I'd also add that you may end up using different databases at different levels of development. It's not uncommon for developers to use SQLite on their workstation when they're coding against their personal development server, and then have the staging and/or production sites using a different database tool.
Of course, if you're using extensions or capabilities specific to a certain database tool (say, PostGIS in PostGreSQL), then obviously that wouldn't work.
Related
Just a bit of background on where my question is coming from: my company has multiple databases across the globe that uses the same schema and once of my department's responsibility is to monitor and make sure all these DBs are in sync from a schema SQL change perspective.
Now, my question is if anyone knows of any Software/tool that has a a Frontend UI which is able to do the following (the lower number the more important to have):
Able to track what SQL code change was applied on which database and when. Basically, if we write a SQL query that changed the structure of a table and we need it applied to 80% or 100% percent of the DBs, either via manual input or some automatic check the tool will tell me that yes, this was indeed applied.
Code distribution tool: we give it the query or a file that contains the code and it's able to push to the Databases it needs to (and create the audit log for that)
Code/object repository: keeps track of what was custom developed and pushed to the databases
I know SSIS might be able to do some of these things, but we need a tool that also has a simple frontend interface that can be accessed by non-IT personnel. (*clarification: we are not planning on giving non-DBA people access to change things, just to the audit aspect of said tool)
I've tried searching the internet, but i have a feeling i'm not using the right vocabulary to get the results i'm looking for.
Hence i wanted to see if the community was aware of any such tool or something similar?
Try searching for one of these two types of systems:
Release/Build/Deployment Automation Complex programs like Serena that have modules for pushing, tracking, and auditing any kind of software, anywhere. These will include all the GUI bells and whistles. But you'll have to deal with extra databases, configuration, agents, workflows, consultants(?), etc. These programs are geared more towards developers.
Remote Execution/Configuration Management Simpler programs like Salt, Fabric, and Ansible that let you run operating system commands anywhere. They don't offer as many features, and you have to do more of the work yourself, but in some ways that's liberating. If you know exactly what commands you want to run you don't need some other program holding your hand. These programs are geared more towards administrators.
From a database administrator's point of view, the main problem with those types of programs is that none of them are relational. Yes they can connect to a database and run a script, but none of them really speak SQL. Their native languages are Java, XML, SSH, etc. There's nothing wrong with those technologies, but if you only care about databases you don't want to deal with all that complexity.
If you're not happy with either of those types of programs I recommend you look at my open source program Method5. It is a remote execution program built as an extension to Oracle SQL. It works entirely inside an Oracle database, so you can install it yourself and won't need any additional websites, agents, configuration files, GUIs, etc.
Based on your comment about getting bogged down by links, and my answer to your question about half a year ago, I think this is the kind of program you were gradually heading towards creating. It took my team a couple thousand hours of developing and testing to get it right so you were probably wise to give up on making your own.
To specifically answer your requirements:
Tracking Changes are stored in an audit trail. But more importantly it has the ability and a pre-built script to compare an unlimited number of schemas, all in one view. At the end of the day what you really want to know is "are my schemas the same", not necessarily "did the same thing get run everywhere?".
Code Distribution If you just have SQL or PL/SQL, deploying it through Method5 is as easy as it can possibly get. Just specify what you want to run, and where you want to run it, like this: select * from table(m5('create index ...', 'dev, qa, prodDB1, prodDB2')); The program does not (yet) run SQL*Plus scripts. But when you have the ability to run SQL and PL/SQL so easily there's little need for SQL*Plus.
Code Repository All executions are stored in a simple table, M5_AUDIT. It contains the code, who ran it, where they ran it, and how they ran it. It wasn't designed to be a repository like SVN but it's good enough for simple auditing and tracking code.
Method5 does not contain a GUI but in some ways I consider that to be a feature. Since everything is done relationally, everything is in a simple table. You can use any of your existing GUIs - Toad, PL/SQL Developer, Excel, Apex, etc. It's a robust back-end solution that will hopefully make a good foundation for easily building a simple front end.
I have been developing a network security application for several years now, as the lead developer at my company. It is a split-architecture design, where one component resides on the customer's network, and the other component in our own cloud. We have developed our own custom versioning system that keeps both sides synchronized at each patch (per customer), but until now it has only allowed incremental changes to be made, and rollbacks are not possible.
We'd like to move to a forkable git-like solution for our code, so that we can develop and test multiple features simultaneously, but the thing that's holding us back from that is our database. We use PostgreSQL (currently 9.3.12), and I've written a custom script to calculate the deltas between the "old" and "new" database structure, each time we "make a patch". It spits out a list of SQL commands necessary to update the "old" database structure to look like the "new", including tables, functions, sequences, triggers, you name it. It's very elegant and pretty much never fails anymore, even with complicated deltas.
However, I realize that in order to have a git-like solution for this (check-out, check-in, merge changes into test and production code, etc.) while also keeping database changes in sync with application code, we'll need to have something a lot more advanced than just "old" vs "new". Note that we don't need to modify database data for the most part, only table structure, which is altered in place on existing customer databases.
So my question is this: Any ideas for a git-like SQL version control system, which allows forking and merging, and can be easily kept in sync with application code changes? Our custom tool is already a bit more advanced than some open-source tools we've looked into (such as sqlt-diff), and tools like Red Gate are a bit out of our price range as a startup (not to mention that I haven't heard anybody mention forking in context with Red Gate). We're open to writing a custom tool, if that's what we need to do, but we're scratching our heads about where to start with something like that. We know how to calculate deltas, but we don't know how to manage all those things across different forks.
Free or open-source tools, frameworks we can adapt, or general guiding principles for building such tools are all appreciated!
One way of solving this problem is with migrations. A couple of lightweight tools, but there are many others:
http://sequel.jeremyevans.net/rdoc/files/doc/migration_rdoc.html
https://flywaydb.org/
Rather than calculating deltas between versions after the fact, migrations can be used to evolve the schema in a controlled way. You can create feature-specific migrations that can be tracked (and forked/merged) along with the rest of your code.
Depending on how fancy you want to get, you may need to extend the default naming/numbering schemes.
In HTML+CSS+JS world, http://jsfiddle.net/ is very helpful tool for asking / making example about web development. And I also saw several browser(javascript)-based programming language compilers and REPLs. But I can't find online / web-based test environment for database operations( especially for RDBMS ).
Is there any open/free database service with web-based interfaces for testing queries?
Added: This tool will be good for this situation; If I'm troubling with complex queries, then create a sample table via web interface and ask it on stackoverflow with the 'sample table URL'. Anyone can access to the URL and test their queries on web site. (Yes, queries are running on 'real' database system) And also the query results can be tracked, then we can even make 'ranking' for it :)
Try SQL Fiddle.
You can try your SQL query and execute/test it.
There are free "disposable" database servers like db4free and even MonoQL.
As far as the web-based interfaces and short URLs go, I don't think you'll have much luck.
To manage your data you have to stick to what is provided (usually phpMyAdmin or similar) and there is no short-URL to query mapping. One other caveat of such system is that (without the appropriate user permissions) one user could easily destroy all your test data -- and remember that (relational) database versioning is much more expensive than plain text versioning, so that's pretty much out of the question.
For non-RDBMS, I can think of try.mongodb.org -- but it suffers from the same problems.
Almost forgot, the Stack Exchange Data Explorer, lets you practice T-SQL queries (with permalinks).
PS: As a personal side-note, I think it's a cool idea and I would love to see something like that implemented, perhaps even mashed-up with SchemaBank or similar - that would be just awesome.
You can't really test a query without the right underlying dbms, schemas (or databases), tables, constraints, stored procedures, and permissions, which tend to be highly application specific. (That is, not readily reusable among multiple users.)
Instead, the database world has grown up into database management systems that you can freely download and install locally. Then you can build and populate your own tables, and test your queries however you like.
Most of these come with both a command line interface and some kind of graphical interface. It's not clear to me what a web-based interface would give you that doesn't already exist in one form or another.
I think that, to do what you want, would require commercial licenses for Oracle, DB2, SQL Server, and Sybase. That's a pretty high barrier to entry for a free web site.
Trouble with a web based query analyser is that you'd need to let it 'tunnel' on to your box to run the queries and for many making a development/test box open to the internet is not a possibility.
For a non web based tool you could look at LinqPad http://www.linqpad.net/ - it does Linq & Sql and other stuff too - very handy tool indeed
I've recently started using Kiln Source Control for all my projects VB.NET code, and I don't know how I managed without it!
I've been looking for a database source control, for all my stored procedures, UDFs etc. However, I've found that there is not as much available for database version control as there is for my web files.
Why is database version control not considered as important as my web files? Surely all the programming in my database is just as important as the code in my code-behind and .aspx files?
Version controlling database objects IS important!
However, maybe it isn't considered as important because some people see a database merely as a tool that assists them? And external tools (normally) aren't version controlled.
One thing I've found hard to manage is the release process. Right now we're using red gates source control connected to svn. When it's time for release we do the same as with the rest of the code: Merge from one branch to the other. Then to deploy it we use sql compare to create a diff script between the merged revision and the actual database. Aside from some minor quirks and beginners mistakes I think this works well in a environment where there is no downtime (purposefully ;)) and which has a high speed development process (lots of releases).
You can maintain your database artifacts in your version control system for same.
Version control system is for versioning of artifacts and Artifacts can be Program code or database. We used same VC for code and database.
VCS are designed to store versions of text. They can store binaries but it is less efficient. And the DB state itself is not a text and can't even be directly stored as a binary. You can store the SQL code though.
one solution is to store a full DB dump (SQL or binary), another - to store sequences of SQL scripts that change one DB state to the next one. The second approach can be automated in some environments and is there called migrations. if you want a separate specific VCS for a DB, you can think that migration tools are such VCS.
There are also tools that can compare two DB states and produce a diff that is able to change the first state to the second.
I suppose it depends on whether you manage the database changes (like schema changes, migrations as mentioned by wRAR) as part of source code repository, in the form of sql scripts or other formats through the use of other tools, or do you consider this as database administration, and do that using traditional methods of backup/restore.
In my experience so far, although I wouldn't consider database management as any less important, it does happen on a very low frequency as compared to actual code changes. Your case is clearly different, but a combination of script files and database tools should take care of that.
Here's the reality.
Database version control -- that is to say DDL, DML, and even data for necessary reference data required for an application to have basic functionality - is as important as all other application assets under version control. Databases should never be under any special exception where it is considered acceptable for their assets (objects and necessary reference data) to not be under version control. Ever.
So why have they been? Simple. The toolchains to keep managing those assets simple haven't always been up to snuff (in the case of SQL Server, prior to Visual Studio 2008 it wasn't shipped with first-party tools from Microsoft), and the toolchains differ on a vendor-by-vendor basis. When those toolchains are deficient, unless the organization steps up to cover that deficiency, that deficiency remains. It's technical debt, and some organizations do not prioritize it due to either time or (sadly) skill, when the tools to make it easy don't exist or require integrating third-party tools into the development workflow.
The worst is trying to bring older projects under version control, since you have to bring everyone kicking and screaming with you all at once, in addition to selling that value to the business. I won't disagree that there may be more pressing immediate needs of the business, but getting database assets under version control needs to be somewhere on that list, even if it's a lower priority.
There's no excuse. I've fought more than enough project managers, data architects, and even CIOs/CTOs on this -- I've even made it a point to have detractors kicked off of every project I'm working on. It needs to be done, and if it's not there needs to be a timeline that the business will agree to in which it will be done. Those who argue against it need to be shot in the face, and survivors need to be shot again.
Assume Hibernate for the ORM.
I'm not sure how to ask this. I want to build an application that can replace part of another. For example, say I have an application with various modules, called the "big" app. This application may handle HR, financial, purchases, skill sets, etc. But maybe, for whatever reason, I don't like the skill set module, but I like the rest of the application. I want to build an app that uses the same database that the rest of the "big" app uses but use my software as the front end for that piece.
I could build my app and have it hit the database directly with no ORM. My question is is there an advantage to using an ORM here. I'm thinking there is because if the "big" app goes away and another app is purchased, we could continue to use my version of skill set because I am using hibernate instead of hitting things directly. I'm still learning but I thought that my application used objects that I named and that in the case I just described I'd have to change my mapping files only or/and my code very little.
Here is another example. I have a legacy application and legacy database. It uses database X. I decide that I no longer like the old terminal emulator application that is used to get the data and that I want a graphical version. I can use hibernate with my application and when I finally decide to get rid of the legacy database and change to the latest Oracle or SQL Server, I can do so with minimal headache? Or is my database going to change so much that it wouldn't have matter anyway (I'm suggesting that upon changing to a new database more information will want to be captured)?
I was hoping for comments, if I am misunderstanding why hibernate/ORM might or might not be a benefit.
Thank you.
I do not think you will have a huge benefit frmo hibernate if the database schema changes to something completely different, you might have to change more than just your mapping - especially if more "structure" is added to the database (tables, column and such schema things). That said, if the database was structured mostly the same way, but lets say just the column names and tables names changes and a couple of tables are merged or something like that - you can get by with just changing your mapping.
But I would really recommend using hbernate for database agnosticity, that's is a pretty easy path.
AND then just because it doesn't exactly helps you if your entire database is changed, it such an incredible amount of other forces, that I would choose that over direct DB access most of the time.
Lastly you could think about using a service-layer such as the repository pattern that abstracs away the data access, so the business of your appilcation wouldn't need to change if the database changes.
Switching from one DBMS to another (ala Oracle to SQL Server) is one thing that using an ORM would certainly make much easier.
As for switching from one "big app" to another "big app", I doubt if using an ORM would help that much. It's likely that the database structure and business logic would be different enough that you would find yourself rewriting lots of code anyways.
You can generate domain objects with Hibernate Tools, if you do that than it will be painless and fast. however if you write all the objects by hand you will die. i think its good idea to rewrite part of the app and get to know hibernate better.
I think it's generally a bad idea to make any decision based on the
unknowns versus the knowns. Whether you're deciding on a data
access/persistence strategy, what car to buy, or what college to go
to, you should put the most weight on the things you know you want
today, rather than worrying about what may or may not happen tomorrow.
So when considering ORMs, I wouldn't worry too much about things such as apps
"going away" or DBMSs changing (unless that's either already been talked about, or
there's a history of this in your company). I'm not saying that these aren't things that will never happen, but rather that they should take a back seat to the generally much more important considerations of maintainability, performance, and developer productivity.
So in short, choose an ORM based on its ability to solve the problems and satisfy the requirements that you have today.