I'm looking for something that lets me better organize my SQL scripts.
I want to be able to include SQL code from one script into another, similar to how in C++ you can do include foo.c to import the contents of foo.c into your program.
Is that possible with SQL?
(FYI, I'm using SQL Server)
SQL is not designed to work like structured or object oriented programming languages.
In case you want to re-use scripts you built, I suggest you create functions and/or stored procedures which you will then be able to call, so you would avoid having to rewrite the code (this would be like "importing").
Functions are the basis of returning data in a custom format. You can read more about them here. You will find tips on when/where and how to use functions.
If you think functions are not enough, try reading about stored procedures here.
Related
Extract method is a common refactoring pattern when writing programming languages.
When I try to do some refactorings on my stored procedures, I am wondering if it is also a good practice to use extract method when writing stored procedures (SP)/User-defined functions (UDF) since we can call other SPs/UDFs on a SP/UDF?
Does it affect performance?
Thanks in advance.
Just my opinion (working for several years with databases now):
Stored procedures should be used for database tasks only. For example migrating data (currently I'm working on a process to transform a database structure for example), or some dynamic queries (where a sql statement is built on the fly), or maybe a procedure to build a table (for example a table that holds dates for a specific date range).
Not for anything else! For everything that gets more complicated than above examples consider to code it on application layer.
Also, you maybe heard that it's wise to put as much business logic into the database as possible. That's true for the database design, but it does not mean, that you should code almost everything in it. Databases are not good at that (talking for example about data transformation or something like that). A programming language like PHP or whatever is faster!
So, for everything that I used stored procedures for, I never felt the need to put anything in extra procedures. Apart from for example the restructuring of a database (in my case it's a ETL process (it denormalizes data into a star schema for better performance)), there I wrote a procedure for every table and these procedures are called from a procedure that manages the whole process. But again, it's nothing like a programming language.
Also, when I take this example for extract method pattern http://www.refactoring.com/catalog/extractMethod.html
having something like this in your database will become a debugging nightmare and you will spend way too much time coding. And again, the cases where a stored procedure should be used are not cases where it makes sense to apply the extract method pattern.
I would like to build arbitrary queries to a database, by allowing the user to build queries "on the fly". For every object/table, being able to select its attributes, and then "building" the query (that would translate into a SQL statement) and finally launching it, all through a web interface.
The ticketing system "rt" does that, for example, and another example would be the http://gatherer.wizards.com/Pages/Advanced.aspx webpage.
I'm currently programming in rails but any existing solution that implements this (or something similar) would be welcome.
Just be careful when creating dynamically generated queries like this that will need to be executed via sp_executesql (example: ms sql server), etc..... make sure you cover all of your bases to ensure that your application isnt vulnerable to SQL injection attacks as this type of development will essentially get one in a lot of trouble if its done incorrectly.. I would recommend storing all queries in a table and only reading queries from this table to help isolate the queries that are being ran in your application. Just identify them with a label, and allow the EU to choose the label from a dropdown list control on the frontend.
Good luck and I'm not sure of any software that will help assist
Not quite sure what your use case is here but i would say check out the
Doctrine ORM ( Object Relational Mapper )
**Edit
After reading more and looking at the example. I would only suggest Doctrine for a large website.
Then use Doctrines DQL syntax with some javascript/jquery magic for the forms.
Note that the queries you're referencing aren't arbitrary: they're on a very specific problem domain, on a specific set of sql tables.
That said, if I were you I'd look into how people are building sql queries with javascript. Something like these:
http://code.google.com/p/django-querybuilder/
http://css.dzone.com/articles/sqlike-sql-querying-engine?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+zones%2Fria+(RIA+Zone)
http://thechangelog.com/post/4914956307/rel-arel-ported-to-node-js-with-some-changes
That'll at least get you a good idea of the underlying data structures.
In the language file itself?
In a language file with all SQL statements?
In different .sql files?
Another way?
Share your code style. :)
Even if you don't use a framework, MVC helps keep you sane by separating your data access, logic and presentation into separate language files.
I guess you will get as many answers as answers. Anyway let's ask why one could
be preferrable over the other:
In the language file itself (I don't know what yoyu mean with the language file) but let me assume you meant the programming langauge itself. Well this approach was taken by Microsoft with Linq . It was taken e.g in Gemstone where the "query" langauge is Smalltalk (but not SQL)
If you put it in some .sql file then there must be a way to adress the code. I think this is what is done with stored procedures. Examples for that can e.g be found in the Postgres Database software.
If you put it in one of many files is probably open. E.g it could be that you have one query one file. Is that better or worse than having a hash table with diverse SQL statements identified by some key.
I see the following approaches every day in Access software
1) embedded in VBA as "just strings"
2) put into the queries section of access
3) I even read about putting this SQL statements in an extra SQL Statement Table.
Regards
Friedrich
It all depends - e.g:
in small DB maintenance scripts with a simple sequential control flow it's nice to see the statements where they are used
programs with loops/callbacks should prepare the statements early; then a list of all stements near a init/prepare function makes sense
a special case: I use a set of tool scripts written in different languages that do 'the same thing'; they all get their statements from .txt files containig SQL statements tagged by name
'big' applications (should) use stored procedures - then the problem vanishes
In Cobol, I put the SQL in the language file, but in separate procedures. That way, I separated the business logic from the data base logic.
In Python, I put the SQL in its own .py module. That way, I separated the business logic from the data base logic.
In Java, I put the SQL in a separate package. That way, I separated the business logic from the data base logic.
I've not used other languages, but I'd probably separate the business logic from the data base logic.
Up until now, my experience with databases has always been working with an intermediate definition layer that we have where I work. i.e. SQL wasn't directly written for the table definitions, but generated from an intermediate file which wrote out SQL scripts for creating the appropriate tables, upgrade scripts between schema changes, and helper functions for doing simple queries/updates/inserts/deletes from the database.
Now I'm in a situation where I don't have access to that, for reasons I won't get into, and I find myself somewhat lost at sea regarding what to do. I need to have a small number of tables in a database, and I'm unsure what's usually done to manage the table definitions.
Do people normally just use the SQL script that does the table creation as their definition, or does everyone just use an IDE that manages the definition in a separate file and regenerates the SQL script to create the tables?
I'd really prefer not to have to introduce a dependence on a specific IDE, because as we all know, developers are whiners that are prone to religious debates over small things.
Open your favorite text editor -> Start writing CREATE scripts -> Save -> Put in Source Control
That script now becomes the basis for you database. Anytime there are schema changes, they get put back into the scripts so that they don't get lost.
These become your definition.
I find it more reliable than depending on any specific IDE/Platform generating those scripts for you.
We write the scripts ourselves and store them in source control like any other code. Then the scripts that are appropriate for a particular version are all groupd together and promoted to prod together. Make sure to use alter table when changing existing tables becasue you don't want to drop and recreate them if they have data! I use a drop and recireate for all other objects though. If you need to add records to a particular table (usually a lookup of sometype) we do that in scripts as well. Then that too gets promoted with the rest of the version code.
For me, putting the scripts in source control however they are generated is the key step. This is how you know what you have changed for the next release. This is how you can see earlier versions and revert back easily if there is a problem. Treat database code the same wayyou treat all other code.
YOu could use one of the data modelling tools that creates scripts for you if you are starting out on a database design and the eventually want to create it for you. Some tools for that are Erwin, Fabforce etc... (though not free)
If you have access to an IDE like SQL Management studio, you can create them by using an GUI thats pretty simple.
If you are writing your own code, Its always better to write your own scripts based on a good template so that you cover all the properties of the definition of the table like the file_group, Collation & stuff. Hope this helps
Once you do create a base copy generate scripts and have a base reference copy of it so that you could do "incremental" changes on them and manage them in a source control.
Though I use TOAD for Oracle, I always write the scripts to create my database objects by hand. It gives you (and your DBA's) more control and knowledge of what's being created and how.
If your schema is too difficult to describe in SQL, you probably have other issues more pressing than which IDE. Use modelling documentation if you need a graphical representation, but yeah, you don't need an IDE.
There are multiple ways out there for what you are asking.
Old traditional way is to have a script file ready with your application that has CREATE TABLE statement.
If you are a developer, and that too a Java enterprise developer, you could generate complete schema using a persistence library called Hibernate. Here is a how to
If you are a DBA level user, you could take schema export from one environment and import that in to your current environment. This is a standard practice among DBAs. But it requires admin access as you can see. Also, the methods are dependent on the database you are using (oracle, db2 etc)
This is a problem that I come to on occasion and have yet to work out an answer that I'm happy with. I'm looking for a build system that works well for building a database - that is running all of the SQL files in the correct database instance as the correct user and in the correct order, and handling dependencies and the like properly.
I have a system that I hacked together using Gnu Make and it works, but it's not especially flexable and frankly can be a bit of a pain to work with in some situations. I've considered looking at things like SCons and CMake too, but I don't know how much better they are likely to be, or if there's a better system out there that already exists...
Just a shell script that runs all the create statements and imports in the proper order. You may also find migrations (comes with rails) interesting. It provides a make like infrastructure that let's you maintain a database the structure of which evolves over time.
Say you add a new column to some table. In migrations you'd write a snippet of code which describes the requirements for adding the column and also to rollback the change so you can switch to different versions of your schema automatically.
I'm not a big fan of the tight integration with rails, though, but the principles behind it are very interesting.
For SQL Server, I just use a batch file with SQLCMD.EXE and a bunch of .SQL files. It's not perfect, but it seems to work.
For my database, I use Migrator.NET
This is a .NET framework which allows you to create classes in where you define your DDL statements.
The framework comes with a command-line tool with which you can execute your 'migrations' in the correct order.
It also has a msbuild - task, so you can integrate it in a continuous integration build as well.
First export full DDL files describing all tables, views, source code
(procedures, functions, packages), sequences, and grants of a DB schema
See
Is there a tool to generate a full database DDL for SQL Server? What about Postgres and MySQL?
I created a database build system (part SQL-parser, part make file) to put these files together in a DB creation script using python.