I am writing a sql procedure where almost everything will be dynamic including selecting, grouping, ordering by, and where clauses using IN statements. In terms of code reuse, readability, and maintenance it makes a lot of sense to just pass in an sql query as a string and execute it. I am writing my procedure right now so that all the relevant data is joined and formatted in a static query and then inserted into a table variable. I then want to pass in sql queries to be executed against the table variable.
This opens me up to sql injection in a big way. I could create table value parameters for each of the many parameter types I am passing in but I don't want to do that. What I would really like to be able to do sandbox my procedure in a such a way that, on the procedure level, it is only possible to do these things I want to allow; ie select from certain tables, but not grant permissions or anything funny like that. Can this be done?
Of course it can be done. It's a simple matter of programming. You would keep rules in tables, and write logic in your stored procedure to query the rules tables, and follow the rules.
It will be a monumental job that will basically amount to you writing custom code to do what SQL Server already does for you if you don't use a generic, dynamic stored procedure.
I wouldn't do it, but you don't have to let that stop you.
Related
Why stored procedure cannot be used with Select, Where & Having?
I understand a function can serve the purpose - what's the reason why a stored procedure cannot be executed in select?
Blogs are answering you can use UDF. Understand we can use UDF & we are using.
Need a valid reason on stored procedure.
Biggest reasons most likely is that procedures can return any number of result sets and change data. It can have no results, or it can be 100 different results sets with 0 to n rows. It can also depend on your input parameters. Stored procedure can also affect the underlying data, so what would happen if you would use a stored procedure that changes data in the same table you're using it in the where clause? The results sets don't even necessary have names for the columns, so you couldn't refer them in any way.
So, since for procedures would be really complex to implement anything like that, why should it be possible to use them in where etc? You haven't provided any valid reason why they should.
The reason functions exist, is that you can use them for where clauses etc.
I didn't undetstood the difference between Stored Functions and Views.
Using Views in SELECT will execute the query and return the result, but Stored Functions do the same thing, don't they? So what is the difference? When I use Views and when Stored Functions?
View:
A view is a virtual table. It does not physically exist. Rather, it is created by a query joining one or more tables. View returns a table.
Stored procedure: A stored procedure is a group of Transact-SQL statements compiled into a single execution plan.
stored procedures returns Output parameters,return codes (which are always an integer value),
a result set for each SELECT statement contained in the stored procedure or any other stored procedures called by the stored procedure,a global cursor that can be referenced outside the stored procedure.
key benefits of stored procedure are Precompiled execution, reduced client/server traffic,efficient reuse of code, programming abstraction and enhanced security controls.
Update:
A stored function is a named PL/SQL Block which is similar to a procedure. The major difference between a procedure and a function is, a function must always return a value, but a procedure may or may not return a value.
1) Return Type: The header section defines the return type of the function. The return datatype can be any of the oracle datatype like varchar, number etc.
2) The execution and exception section both should return a value which is of the datatype defined in the header section
You can have a stored function return the same data a view would in most databases.
The distinction for me is that a function is executed and a view is selected from.
A view will behave as a table.
A view returns a specific pre-defined statement as exactly one result set.
A function returns a single values or a single result set. This however can differ from different types of database.
Several db implementations also have stored procedures where the result can be a single returned value, a result set or several result sets.
Getting simple (PLEASE start reading a book about SQL): A view looks like a table, so you can filter on the results and the filter will efficiently be part of the views execution, or do joins. A SP does not allow this, but a lot more logic. The rest... is in the documentation.
These can never be compares, these have totally different
approach.
A view is a output of a query ,and makes a virtual image of the table,and the input parameters are not accepted.
Main difference is that a Stored Procedure can alter your data, where
as a view only returns it and I believe from a performance point of
view, a stored procedure is better as it caches the execution plan and
will run faster as a result.
storedprocedure/function is a group of sql statements that are pre-executed and it accepts the parameters.it reduces network traffic, gives faster performance, etc.
SQL Functions in programming languages are subroutines used to encapsulate frequently performed logic. these somewhat slow down the performance.
Check these SQL View, SQL Stored Procedures and SQL User-Defined Functions
My Opinion is that SQL Stored Procedure(Stored Functions) are much better to use because it provides custom manipulations on result set also.
From my experiences I'm sharing to you my knowledge:
Don't use views
Better to use a stored procedure(it is compiled sql statement), you can use parametrized procedure as required.
Stored Function is collection of complied sql statement which is faster.
Note: Views is a SELECT statement( with/without JOIN) for a table which select data from table and if we again run a SELECT statement from VIEWS which provide slower result because the internal operation is as ( SELECT * FROM ( SELECT * FROM TargetTable ) )
So, its better to use Stored Function
Update:
Functions are computed values and cannot perform permanent environmental changed to SQL Server (i.e. no INSERT or UPDATE statements allowed).
A Function can be used inline in SQL Statements if it returns a scalar value or can be joined upon if it returns a result set.
Also please see here for performance comparison: SQL-Server Performance: What is faster, a stored procedure or a view?
As a developer mainly writing c# I have adopted some good practices when writing c# code. When I sometimes write stored procedures I have trouble applying those practices to the stored procedure code.
On several occasions I have inherited nightmare stored procedure code, first three or four layers of stored procedures setting up some temp tables and mostly calling each other. No real work done and just a few lines of code. Then at last there is a call to "the final" stored procedure, a big monster of 3000-5000 lines of SQL code. That code usually have a lot of code smells like code duplication, intricate control flows (a.k.a. spaghetti) and a method that does too many things stacked after each other with no clear separation where one chunk of work starts and where it ends (not even a comment as a divisor).
I have also noticed the use of out commented select statements that selects from intermediate temp tables. The selects can be turned back on for debug purposes, but need to be removed before any calling code expecting a specific order of the returned result sets.
Apparently my fellow team mates also share my lack of good SQL writing practices.
So... ( and here comes the real question) ... what are good practices for writing modular maintainable stored procedures?
Both home made practices and references to books/blogs are welcome. Methods as well as tools that help with certain tasks.
Lets summarize some areas where I have not found good practices
Modularization and encapsulation (is stored procedures communication via temp tables really the way to go?)
In c# I use assemblies, classes and methods decorated with access modifiers to accomplish this.
Debugging/testing (better than modifying the target of debugging?)
Debug tools?
Debug traces?
Test fixtures?
Emphasizing code/logic/data/control flow using code the structure of the code
In c# I refactor and break out smaller methods that does just one logical task each.
Code duplication
Mostly I encounter SQL Server as DBMS but DBMS agnostic answers or answers pointing out features of other DBMS:es that help in above cases are also welcome.
To give some background: Most large stored procedures I have encountered are in reporting scenarios where the base is to just create some summary values from a large table. But along the way you need to exclude some of the values that happen to be in some exception table, add some of the values in some not yet completed stuff table, compare with last year (can you imagine the ugly code that handles products changing department between years?), etc.
I write a lot of complex stored procs. Some things I would consider best practices:
Don't use dynamic SQl in a stored proc unless you are doing a search proc with lots of parameters which may or may not be needed (then it is currently one of the best solutions). If you must use dynamic SQl in a proc always have a debug input parameter and if the debug parameter is set, then print the SQL statement created rather than executing it. This will save hours of debugging time!
If you are performing more than one action query in a proc (insert/update/delete), use Try Cacth blocks and transaction processing. Add a test parameter to the input parameters and when it is is set to 1, always rollback the entire transaction. Before rolling back in test mode, I usually have a section that returns the values in the tables I'm affecting to ensure that what I think I am doing to the database is in fact what I did do. Or you could have checks as go as shown below. That is as simple as putting in the following code around your currently commented out selects (and uncommenting them) once you have the #test parameter.
If #test =1
Begin
Select * from table1 where field1 = #myfirstparameter
End
Now you don't have to go through and comment and uncomment each time time you test.
#test or #debuig should always be set with a default value of 0 and placed last in the list. That way adding them won't break existing calls of the proc.
Consider having logging and/or error logging tables for procs doing inserts/updates/deletes. If you record the the steps and or errors in table variables as you go, they are still available after a rollback to be inserted into the logging table. Knowing what part of a complex proc failed and what the error was can be invaluable later on.
Where possible do not nest stored procs. If you need to run multiple records in a loop, replace the stored proc with one that has a table-valued parameter and set up the proc to run in a set-based and not individual record fashion. This will work if the table-valued parameter has one record or many records.
If you have a complex select with a lot of subqueries or derived tables, consider using CTEs instead. Refactor any correlated subqueries or cursors to better performing set-based code. Always think in terms of sets of data not one record.
Do not, under any conceivable circumstance, nest views. The performance hit is much worse than any small amount of saved development time. And trust me, nested views do not save maintenance time when the change needs to be to the view the furthest into the chain of views.
All stored procs (and other database code) should be in source control.
Table variables are good for smaller data sets, but temp tables (real ones that start with # or ## not staging tables) can be better for performance in large data sets. If using temp tables drop them when you don't need them anymore. Try to avoid the use of global temp tables.
Learn to write performant SQL. It is usually just as easy to write SQL that will perform well than SQL which will not once you know the techiniques. If you write complex stored procs, there is no excuse for not knowing which techniques work better than which other ones. Learn how to make sure your query is sargable. Avoid cursors, correlated subqueries, scalar functions and other things which run row-by-agonizing-row.
Communication via temp tables is sometimes a huge code smell. Such procedures often cannot be run by a user without interfering with each other (if you re-use a temp table name for different procedures' ins and outs and they aren't re-created or if you use the same name with two different table schemas). They can be hard to troubleshoot - like any feature, use them when necessary and better alternatives don't exist. Using real tables temporarily can also be problematic.
Stored procs which pass data to each other in SQL Server at all (more than parameters) can be problematic. There are table-valued parameters now and many things which previously would have been done with procs can now be done with inline table-valued functions or (and usually preferred over) multi-statement table-valued functions.
In SQL Server, avoid heavy use of scalar functions and multi-statement table-valued function on large rowsets - they do not perform very well, so modular techniques which may seem obvious in C# don't really apply here.
I would recommend you look at Ken Henderson's Guru's Guide to SQL Server Stored Procedures - published in 2002, it still has a wealth of useful information on database application design.
This is such a good question. As a C# dev myself having to dabble in SQL it seems SQL by its very nature gets in the way of the best-practices I'm used to with C#.
Common Table Expresions are great to isolate queries in a stored procedure but you can only use them once! That leads you to define views but then you've lost your encapsulation.
A resultset from one stored procedure are very difficult to use in another so you might be tempted to write table-valued functions. That adds to your permissions-maintenance burden and forces you to write functions 'twice' - once as a function and another as a procedure that calls the function. Otherwise, you have different interfaces to your DAL depending on whether it's a procedure or not.
All of this has caused me, over time, to stick to simple CRUD stored procedures (that do not call each other) in the database and few, isolated, queries when the relationships are complex. More BI-stuff. Everything else is in the BLL.
Physically, SQL is isolated in seperate files by function or the table they revolve around and managed in source control.
Avoid SELECT * and favor specifying columns. That saves you from run-time problems when you change a table and don't touch all the procs. Yes, there is a recompile for procs but it WILL miss some, especially if views are involved. Plus, SELECT * almost always returns more columns than you really need and that's a waste of bandwidth.
The comments above are great advice of Do's and Dont's when it comes to SQL code writing. If I understand your questions correctly, you are asking if this is normal for SQL Developer to write hundreds even thousands of code in a single stored procedure. In C# this is a big no-no. You are to encapsulate logic into small chucks using methods, assemblies, and classes. SQL Developer tend to write the entire logic in one stored procedure to accomplish a relating task; as HLGEM mentioned above, "If possible, do not nest stored procedures". Do not nest Views.
For example: A simple Get and Insert design in C# looks like this:
Call GetData Method
Call Get Data Method
Call Transform Data Method
Call CheckAlphaNumeric Method
Call Data Enrichment Method
Call Load Transformed Data Method
A SQL developer will design it like this:
In a single stored proc:
Get Data and Transform using either temp table or table variable, then Load it into the final table.
If you are to change the way SQL is written to match the writing structure of C# Developer, you would then do this:
Execute Main Stored Procedure (which calls the below sproc.)
Execute GetData Stored Procedure and load into Stage Table
Execute Transform Stored Procedure which read the Stage Table and transform data
Execute Load Data stored procedure to load Staged or Transformed data into final table.
is it going to be faster if instead of doing
select * from users where id = 1
or
delete from users where id = 1
or
select count(*) from users
I would create a SP for it ?
Performance-wise, no it doesn't make any difference.
Security-wise, it does made a difference. Using a sproc means you only need to grant execute permissions on the sproc, whereas the non-sproc approach would require permissions to be granted directly on the underlying table(s).
Network-traffic-wise - potential, slight/negligible difference. More applicable to larger statements whereby you either send the entire SQL statement across the wire or send just the sproc call. Pretty neglible overall.
Maintenance-wise - the sproc approach would allow you to (e.g.) tune a query without having to redeploy the whole application.
Something I'd be thinking is parameterising the query instead of using "hardcoded" values within an sql statement to support execution plan reuse.
If you're concerned about efficiency, don't do:
Select * From Users
Instead do:
Select column1, column2 From Users
Otherwise, SQL Server needs to do a lookup on all the columns in the Users table.
Personally, I wouldn't put something like this in a stored proc, but some people would, if they are doing all their data access via stored procedures.
If you are using SPs for all data access then yes. Otherwise I would not use it. It's never a good idea to have inconsistent behavior especially in source code.
I guess you are asking this question because you think that SPs are faster than inline queries sent by program but starting from sql server 2005 this is no more true as all execution plans are cached.
I'm leaning more towards a no on this one but then again I can see scenarios where you may want to do this.
For example, you may wish to implement a layer of security by utilizing stored procedures.
There is no improved plan reuse by using a stored procedure when considering the sample queries you have provided.
I have some doubt regarding user defined functions. I would like to know why / when to use functions.
What are the advantages of functions over stored procedure?
Researching via google I have seen articles suggesting:
stored procedure are more advantageous than functions.
function have limited error handling
functions cannot use temporary tables
functions cannot call stored procedures.
The only advantage of function is we can use function as inline queries.
I can get the same result with stored procedure by using temporary tables, but i need to know which scenario to use functions compared to stored procedure.
I need to know why we need UDf , when most of the functionalities provided by UDF can be done by Stored procedure.
Can any one guide me over this.
The main difference (advantage) is that you can call functions inline unlike stored procedures
e.g.
SELECT dbo.fxnFormatName(FirstName, LastName) AS FormattedName
FROM MyTable
SELECT *
FROM dbo.fxnTableReturningFunction() x
User defined functions can return TABLE type data and then the function can then be called within a query as demonstrated above. With a sproc, you'd have to execute it and store the results into a temporary table in order to then manipulate/query the resultset further.
On the flip side, yes you are limited as to what you can do in a function. e.g. you can't use dynamic sql, and pre-SQL 2005 you can't use non-deterministic functions like GETDATE() within a function.
An example of when you may want to use functions, is to wrap up common "formatting" functionality as shown in the first example above - rather than repeat the logic to format a first and last name into one in every query, you wrap it in a function and call that everywhere. Typically I'd recommend leaving the formatting up to the UI but it's a simple example of where/why you might use.
Also, it can often be nicer to not have to create temp tables to hold results from a sproc in order to query it further. If the sproc changes and returns more columns, you'd also need to change everywhere that loads the results into a temp table to synch the schema of the table table it uses to hold the results with the new schema returned. You don't have this problem with the function approach as there is no temp table to be maintained.
There are three types of functions: Scalar, Inline Table and Table Valued. Generally speaking, Scalar & Table Values functions can lead to performance problems, seeing as the Query Optimiser doesn't do very well at optimisation of the use of those types of functions. The performance of Inline Table function is just fine, however.
There is a Connect request to create a new type of scalar function here: The Scalar Expression function would speed performance...
I hope that people do vote for that one, because it would improve performance greatly by allowing the query optimiser to inline functional expressions and take advantage of statistics etc just as it would for a normal query.
The main "disadvantage" of user-defined functions is that they are called for each row. So, if you have such a function in the SELECT list and you're operating on larger sets, there are good chances that your performance will suffer.
Advantage of Mysql Stored Procedure
Multiple applications are running in multiple environment and need to use the same database. By using stored procedure you can make your business logic independent of programming language.
When security is main concern use of stored procedure is vital. By doing your operation through the database you can log your all performed action. Banking site is the best example.
If you are using stored procedure then you do not have table access directly which is one more way to secure the data and transaction.
Stored procedure increases performance of your application sometime
If your application is big or your database server on remote system then by using stored procedure you can decrease the traffic between your database server and application server.
Since stored procedure is written in your database server and application call it sepratly then the degree of re-usability.