Is Inline SQL hard coding? [closed] - sql

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I have heard some strong developers say that "inline SQL in and of itself is not evil". I do not understand how inline SQL is acceptable. To me its just like hard coding. Many a developer would scoff in my face for putting a connection string in the code vs a config file. So why is it that "SELECT value1,value2 FROM TABLE" is perfectly acceptable in compiled code?


I think there is a lot of coupling between a database query and the code that surrounds it. For example, a query might fetch first name and last name for a particular user from a database, and then create an XML file containing the first name and last name.
If you put the database query somewhere else (e.g. config file) it might seem like you've increased configurability and flexibility. However, if you want to actually change it, say add a field called "age" to the XML file, you can't just change the query alone, you need to change the code writing the XML file. So in fact you've replaced the problem of having to change one piece of code, with the problem of having to change two things (code and configuration), and the danger that they might not be consistent, and you haven't gained any extra flexibility.
(If it seems like one can just write generic code to take all database fields and put them in the XML file, consider the case where age is stored as the birth year in the database, and you have to do computation to create the age.)


When I see the length programmers go to just to avoid simple queries like:
select * from products where productid in (1,2) and productactive = 1
or:
select * from products where not exists (
select * from order_lines where products.id = order_lines.productid)
Then I always wonder what their real goal is. Because it can't be simplicity.


SQL is code, it's just not the same programming language as the rest of your code. If you need to change the SQL, you usally always need to change the code dealing with that SQL, data and database schema as well. The SQL shouldn't vary between installation.
If you add a column, changing the SQL isn't enough, you'll need to change code handling that column as well. Pretty much any change to the database schema and SQL, just changing the SQL is not enough.
A connection string on the other hand is configuration, not code.
(And yes, there's exceptions to all of this, if you're creating a general purpose reporting tool, you'll need to engineer that so the SQL can be "configurable")


SQL queries are code, not configuration.
It is useful in an agile environment to use stored procedures on a system like SQL Server where they are dynamically compiled, as you can often fix or workaround an application problem by patching the stored proc.
But it is still code.


Another distinction (aside from code vs. configuration) is that a connection string is probably used all over your codebase whereas a SQL query is more likely to be coupled to a specific class or method. Therefore, if you subscribe to "Don't Repeat Yourself", you'd definitely benefit from centralizing the connection string but would likely gain nothing from moving the SQL string to a different location.
That being said, some of the most elegant solutions involve avoiding SQL strings entirely by using Hibernate or another ORM, but that's a different topic.


"SELECT value1,value2 FROM TABLE" in compiled code was the essence of so called Object-relational impedance mismatch. Since LINQ was released inline SQL is no longer acceptable in vast majority of cases...


I've made systems where all the SQL is in stored procedures and the names of the stored procedures are themselves not even stored in the program, but in the app.config.
This means then not only can stored procedures be patched, but different customers can use different procedures.
The database services interface provided to the application(s) is thus part of the system design.
A drawback of having any SQL in a client application at all (and this includes table-based ORMs and LINQ) is that you have very little explicit security interface. With stored procedures, there is a single GRANT - EXEC. With tables or even views, the ORM will need SELECT, UPDATE, INSERT, DELETE for whatever SQL is generated by the tool or stored in the application. Taking an inventory of the rights needed for the system to properly run and being able to audit is much more difficult than having a good solid layer of stored procedures (and, to some extent, read-only views) which everything goes through.
So I would say that "SELECT value1, value2 FROM TABLE" is not acceptable in client code, because every user who would potentially execute that statement needs to have SELECT on those two column from a base table, and that is hard to manage, even with roles.


One way to look at it is to think of table names and column names as part of the database's "public" interface. That's how databases implement physical data independence. And updatable views implement logical data independence, to some degree.

Related

Are user-defined SQL datatypes used much? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
My DBA told me to use a user-defined SQL datatype to represent addresses, and then use a single column of that new type in our users table instead of multiple address columns. I've never done this before and am wondering if this is a common approach.
Also, what's the best place to get information about this - is it product-specific?

As far as I can tell, at least in the SQL Server world, UDT aren't used very much.
Trouble with UDT is the fact you can't easily update them. Once created and used in databases, they're almost like set in stone.
There's no "CREATE OR ALTER (UDT)" command :-( So to change something, you have to do a lot of shuffling around - possibly copying away existing data, then dropping lots of columns from other tables, then dropping your UDT, re-creating it with the new structure and reapplying the data and everything.
That's just too much hassle - and you know : there will be change!
Right now, in SQL Server land, UDT are just a nice idea - but really badly implemented. I wouldn't recommend using them extensively.
Marc

There are a number of other questions on SO about how to represent addresses in a database. AFAICR, none of them suggest a user-defined type for the purpose. I would not regard it as a common approach; that is not to say it is not a reasonable approach. The main difficulties lie in deciding what methods to provide to manipulate the address data - those used for formatting the data to appear on an envelope, or in specific places on a printed form, or to update fields, worrying about the many ramifications of international addresses, and so on.
Defining user-defined types is very product specific. The ways you do it in Informix are different from the ways it is done in DB2 and Oracle, for example.

I would also rather avoid using User defined datatypes as their defination and usability will make your code dependant on a particular database.
Instead if you are using any object oriented language, create a composition relationship to define addresses for an employee (for example) and store the addresses in a separate table.
Eg. Employees table and Employee_Addresses table. One employee can have multiple addresses.

user-defined SQL datatype to represent addresses
User-defined types can be quite useful, but a mailing address doesn't jump out as one of those cases (to me, at least). What is a mailing address to you? Is it something you print on an envelope to mail someone? If so, text is about as good as it's going to get. If you need to know what state someone is in for legal reasons, store that separately and it's not a problem.
Other posts here have criticized UDTs, but I think they do have some amazing uses. PostgreSQL has had full text search as a plugin based on UDTs for a long time before full-text search was actually integrated into the core product. Right now PostGIS is a very successful GIS product that is entirely a plugin based on UDTs (it has GPL license, so will never be integrated into core).

Sharing Common SQL Queries Amongst a Team [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I don't even want to think about how many man hours have been spent writing the same queries to join over the exact same tables at my company.
When I first started at my job I identified this as an inefficiency and started writing views in a separate schema for the sole purpose of developer convenience.
My boss didn't like this very much and recommended I start committing my common queries to source control in a separate folder from the production SQL. This makes sense because some scripts require parameters and not all are read only.
What is a Common Query?
Script used to diagnose certain problems
Script to view relationships between several tables (doing multiple joins)
Script that we don't want in a stored procedure because it is often tweaked to diagnose the issue of the day
Issues I want to Address
Discoverability, queries will be rewritten if nobody can find them
IDE integration, want to be able to easily view queries in IDE. I've tried the SQL Server Solutions but they really suck because they lock you into only working on that set of files.
I was wondering how all the pro's out there share their common SQL queries.
Thanks

Seems like the op wants to know how to get the word out to the team about useful SQL that others can/should use so not to recreate.
How I have done this in the past is through two ways:
Create a team web wiki page that
details the SQL with examples of how
it is used.
Email the team when new SQL is
created that should be shared.
Of course, we always include the SQL code in version control, just the wiki and email are use in the "getting word out there" part.

If it is something that I would call "common" I would probably create a stored procedure that the folks with necessary permissions can run.
If the stored procedure route won't work well for your team, then the other option is to create a view. Creating a view comes with unique challenges though such as ensuring that everyone running the view has select permissions on all of the tables in the view as well.
Outside of storing the scripts in source control of some kind, maybe storing them on a Share Point site or a network file share would work OK for your team. The real challenge that you will have in sharing scripts is that people have different ways to identify what they are looking for. A wiki type of site that allows tagging the different types of things the queries do would be useful.

You create a view.

Lots of ways to do this (including some that you've mentioned already):
Table-Valued User Defined Functions
Stored Procedures
Views
Source Control
Formal, shared Data Access Layer for client code

Views are the right way to handle this sort of thing. Or, in some cases, a stored procedure.
But there's no rule that says you can't also store the DDL for a View or a Stored Procedure in source control.

Is this a valid benefit of using embedded SQL over stored procedures?

Here's an argument for SPs that I haven't heard. Flamers, be gentle with the down tick,
Since there is overhead associated with each trip to the database server, I would suggest that a POSSIBLE reason for placing your SQL in SPs over embedded code is that you are more insulated to change without taking a performance hit.
For example. Let's say you need to perform Query A that returns a scalar integer.
Then, later, the requirements change and you decide that it the results of the scalar is > x that then, and only then, you need to perform another query. If you performed the first query in a SP, you could easily check the result of the first query and conditionally execute the 2nd SQL in the same SP.
How would you do this efficiently in embedded SQL w/o perform a separate query or an unnecessary query?
Here's an example:
--This SP may return 1 or two queries.
SELECT #CustCount = COUNT(*) FROM CUSTOMER
IF #CustCount > 10
SELECT * FROM PRODUCT
Can this/what is the best way to do this in embedded SQL?

A very persuasive article
SQL and stored procedures will be there for the duration of your data.
Client languages come and go, and you'll have to re-implement your embedded SQL every time.

In the example you provide, the time saved is sending a single scalar value and a single follow-up query over the wire. This is insignificant in any reasonable scenario. That's not to say there might not be other valid performance reasons to use SPs; just that this isn't such a reason.

I would generally never put business logic in SP's, I like them to be in my native language of choice outside the database. The only time I agree SPs are better is when there is a lot of data movement that don't need to come out of the db.
So to aswer your question, I'd rather have two queries in my code than embed that in a SP, in my view I am trading a small performance hit for something a lot more clear.

How would you do this efficiently in
embedded SQL w/o perform a separate
query or an unnecessary query?
Depends on the database you are using. In SQL Server, this is a simple CASE statement.

Perhaps include the WHERE clause in that sproc:
WHERE (all your regular conditions)
AND myScalar > myThreshold

Lately I prefer to not use SPs (Except when uber complexity arises where a proc would just be better...or CLR would be better). I have been using the Repository pattern with LINQ to SQL where my query is written in my data layer in a strongly typed LINQ expression. The key here is that the query is strongly typed which means when I refactor I am refactoring properties of a class that is directly generated from the database table (which makes changes from the DB carried all the way forward super easy and accurate). While my SQL is generated for me and sent to the server I still have the option of sticking to DRY principles as the repository pattern allows me to break things down into their smallest component. I do have the issue that I might make a trip to the server and based on the results of query I may find that I need to make another trip to the server. I don't worry about this up front. If I find later that it becomes an issue then I may refactor that code into something more performant. The over all key here is that there is no one magic bullet. I tend to work on greenfield applications which allows this method of development to be most efficient for me.

Benefits of SPs:
Performance (are precompiled)
Easy to change (without compiling the application)
SQL set based features make very easy doing really difficult data tasks
Drawbacks:
Depend heavily on the database engine used
Makes deployment of upgrades a little harder (you have to deploy the App + the scripts)
My 2 cents...
About your example, it can be done like this:
select * from products where (select count(*) from customers>10)

Where do you store long/complete queries used in code? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Here's my situation. I've noticed that code gets harder to maintain when you keep embedding queries in every function that will use them. Some queries tend to grow very fast and tend to lose readability after concatenating every line. Another issue that comes with all of the concatenation is when you must test a specific query and paste it. You are forced to remove all of the " that held your query string together.
So my question is, what methods are being used to separate queries from the code? I have tried searching but it doesn't look like it's the right thing because i'm not finding anything relevant.
I'd like to note that views and stored procedure are not possible since my queries fetch data from a production database.
Thank you.

If you follow an MVC pattern, then your queries should be all in the model - i.e. the objects representing actual data.
If not, then you could just put all your repetitive queries in script files, including only those needed in each request.
However, concatenating and that kind of stuff is hard to get rid of; that's why programmers exist :)

These two words will be your best friend: Stored Procedure

I avoid this problem by wrapping queries in classes that represent the entities stored in the table. So the accounts table has an Account object. It'll have an insert/update/delete query.
I've seen places where the query is stored in a file and templates are used to replace parts of the query.
Java had something called SQLJ - don't know if it ever took off.
LINQ might provide some way around this as an issue too.

Risking being accused of not answering the question, my suggestion to you would be simply don't. Use an O/RM of your choice and you'll see this problem disappear in no time.

I usually create a data class that represents the data requirements for objects that are represented in the database. So if I have a customer class, then I create a customerData class as well that houses all the data access logic in them. This keeps data logic out of your entity classes. You would add all you CRUD methods here as well as custom data methods.
Youc an also use Stored Proceedures or an ORM tool depending on your language.
The main key is to get your data logic away from your business and entity logic.

I use stored procedures in my production environment, and we process the rows too...

Which is better: Ad hoc queries or stored procedures? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
Assuming you can't use LINQ for whatever reason, is it a better practice to place your queries in stored procedures, or is it just as good a practice to execute ad hoc queries against the database (say, SQL Server for argument's sake)?

In my experience writing mostly WinForms Client/Server apps these are the simple conclusions I've come to:
Use Stored Procedures:
For any complex data work. If you're going to be doing something truly requiring a cursor or temp tables it's usually fastest to do it within SQL Server.
When you need to lock down access to the data. If you don't give table access to users (or role or whatever) you can be sure that the only way to interact with the data is through the SP's you create.
Use ad-hoc queries:
For CRUD when you don't need to restrict data access (or are doing so in another manner).
For simple searches. Creating SP's for a bunch of search criteria is a pain and difficult to maintain. If you can generate a reasonably fast search query use that.
In most of my applications I've used both SP's and ad-hoc sql, though I find I'm using SP's less and less as they end up being code just like C#, only harder to version control, test, and maintain. I would recommend using ad-hoc sql unless you can find a specific reason not to.

I can't speak to anything other than SQL Server, but the performance argument is not significantly valid there unless you're on 6.5 or earlier. SQL Server has been caching ad-hoc execution plans for roughly a decade now.

I think this is a basic conflict between people who must maintain the database and people who develop the user interfaces.
As a data person, I would not consider working with a database that is accessed through adhoc queries because they are difficult to effectively tune or manage. How can I know what affect a change to the schema will have? Additionally, I do not think users should ever be granted direct access to the database tables for security reasons (and I do not just mean SQL injection attacks, but also because it is a basic internal control to not allow direct rights and require all users to use only the procs designed for the app. This is to prevent possible fraud. Any financial system which allows direct insert, update or delete rights to tables is has a huge risk for fraud. This is a bad thing.).
Databases are not object-oriented and code which seems good from an object-oriented perspective is can be extremely bad from a database perspective.
Our developers tell us they are glad that all our databse access is through procs becasue it makes it much faster to fix a data-centered bug and then simply run the proc on the production environment rather than create a new branch of the code and recompile and reload to production. We require all our procs to be in subversion, so source control is not an issue at all. If it isn't in Subversion, it will periodically get dropped by the dbas, so there is no resistance to using Source Control.

Stored procedures represent a software contract that encapsulates the actions taken against the database. The code in the procedures, and even the schema of the database itself can be changed without affecting compiled, deployed code, just so the inputs and outputs of the procedure remain the same.
By embedding queries in your application, you are tightly coupling yourself to your data model.
For the same reason, it is also not good practice to simply create stored procedures that are just CRUD queries against every table in your database, since this is still tight coupling. The procedures should instead be bulky, coarse grained operations.
From a security perspective, it is good practice to disallow db_datareader and db_datawriter from your application and only allow access to stored procedures.

Stored procedures are definitely the way to go...they are compiled, have execution plan before hand and you could do rights management on them.
I do not understand this whole source control issue on stored procedure. You definitely can source control them, if only you are a little disciplined.
Always start with a .sql file that is the source of your stored procedure. Put it in version control once you have written your code. The next time you want to edit your stored procedure get it from your source control than your database. If you follow this, you will have as good source control as your code.
I would like to quote Tom Kyte from Oracle here...Here's his rule on where to write code...though a bit unrelated but good to know I guess.
Start with stored procedures in PL/SQL...
If you think something can't be done using stored procedure in PL/SQL, use Java stored procedure.
If you think something can't be done using Java Stored procedure, consider Pro*c.
If you think you can't achieve something using Pro*C, you might want to rethink what you need to get done.

My answer from a different post:
Stored Procedures are MORE maintainable because:
You don't have to recompile your C# app whenever you want to change some SQL
You end up reusing SQL code.
Code repetition is the worst thing you can do when you're trying to build a maintainable application!
What happens when you find a logic error that needs to be corrected in multiple places? You're more apt to forget to change that last spot where you copy & pasted your code.
In my opinion, the performance & security gains are an added plus. You can still write insecure/inefficient SQL stored procedures.
Easier to port to another DB - no procs to port
It's not very hard to script out all your stored procedures for creation in another DB. In fact - it's easier than exporting your tables because there are no primary/foreign keys to worry about.

In our application, there is a layer of code that provides the content of the query (and is sometimes a call to a stored procedure). This allows us to:
easily have all the queries under version control
to make what ever changes are required to each query for different database servers
eliminates repetition of the same query code through out our code
Access control is implemented in the middle layer, rather than in the database, so we don't need stored procedures there. This is in some ways a middle road between ad hoc queries and stored procs.

There are persuasive arguments for both - stored procedures are all located in a central repository, but are (potentially) hard to migrate and ad hoc queries are easier to debug as they are with your code, but they can also be harder to find in the code.
The argument that stored procedures are more efficient doesn't hold water anymore.
link text
Doing a google for Stored Procedure vs Dynamic Query will show decent arguments either way and probably best for you to make your own decision...

Store procedures should be used as much as possible, if your writing SQL into code your already setting yourself up for headaches in the futures. It takes about the same time to write a SPROC as it does to write it in code.
Consider a query that runs great under a medium load but once it goes into fulltime production your badly optimized query hammers the system and brings it to a crawl. In most SQL servers you are not the only application/service that is using it. Your application has now brought a bunch of angry people at your door.
If you have your queries in SPROCs you also allow your friendly DBA to manage and optimize with out recompiling or breaking your app. Remember DBA's are experts in this field, they know what to do and not do. It makes sense to utilise their greater knowledge!
EDIT: someone said that recompile is a lazy excuse! yeah lets see how lazy you feel when you have to recompile and deploy your app to 1000's of desktops, all because the DBA has told you that your ad-hoc Query is eating up too much Server time!

someone said that recompile is a lazy excuse! yeah lets see how lazy you feel when you have to recompile and deploy your app to 1000's of desktops, all because the DBA has told you that your ad-hoc Query is eating up too much Server time!
is it good system architecture if you let connect 1000 desktops directly to database?

Some things to think about here: Who Needs Stored Procedures, Anyways?
Clearly it's a matter of your own needs and preferences, but one very important thing to think about when using ad hoc queries in a public-facing environment is security. Always parameterize them and watch out for the typical vulnerabilities like SQL-injection attacks.

Stored Procedures are great because they can be changed without a recompile. I would try to use them as often as possible.
I only use ad-hoc for queries that are dynamically generated based on user input.

Procs for the reasons mentioned by others and also it is easier to tune a proc with profiler or parts of a proc. This way you don't have to tell someone to run his app to find out what is being sent to SQL server
If you do use ad-hoc queries make sure that they are parameterized

Parametized SQL or SPROC...doesn't matter from a performance stand point...you can query optimize either one.
For me the last remaining benefit of a SPROC is that I can eliminate a lot SQL rights management by only granting my login rights to execute sprocs...if you use Parametized SQL the login withing your connection string has a lot more rights (writing ANY kind of select statement on one of the tables they have access too for example).
I still prefer Parametized SQL though...

I haven't found any compelling argument for using ad-hoc queries. Especially those mixed up with your C#/Java/PHP code.

The sproc performance argument is moot - the 3 top RDBMs use query plan caching and have been for awhile. Its been documented... Or is 1995 still?
However, embedding SQL in your app is a terrible design too - code maintenance seems to be a missing concept for many.
If an application can start from scratch with an ORM (greenfield applications are far and few between!) its a great choice as your class model drives your DB model - and saves LOTS of time.
If an ORM framework is not available we have taken a hybrid of approach of creating an SQL resource XML file to look up SQL strings as we need them (they are then cached by the resource framework). If the SQL needs any minor manipulation its done in code - if major SQL string manipulation is needed we rethink the approach.
This hybrid approach lends to easy management by the developers (maybe we are the minority as my team is bright enough to read a query plan) and deployment is a simple checkout from SVN. Also, it makes switching RDBMs easier - just swap out the SQL resource file (not as easy as an ORM tool of course, but connecting to legacy systems or non-supported database this works)

Depends what your goal is. If you want to retrieve a list of items and it happens once during your application's entire run for example, it's probably not worth the effort of using a stored procedure. On the other hand, a query that runs repeatedly and takes a (relatively) long time to execute is an excellent candidate for database storage, since the performance will be better.
If your application lives almost entirely within the database, stored procedures are a no-brainer. If you're writing a desktop application to which the database is only tangentially important, ad-hoc queries may be a better option, as it keeps all of your code in one place.
#Terrapin: I think your assertion that the fact that you don't have to recompile your app to make modifications makes stored procedures a better option is a non-starter. There may be reasons to choose stored procedures over ad-hoc queries, but in the absence of anything else compelling, the compile issue seems like laziness rather than a real reason.

My experience is that 90% of queries and/or stored procedures should not be written at all (at least by hand).
Data access should be generated somehow automaticly. You can decide if you'd like to staticly generate procedures in compile time or dynamically at run time but when you want add column to the table (property to the object) you should modify only one file.

I prefer keeping all data access logic in the program code, in which the data access layer executes straight SQL queries. On the other hand, data management logic I put in the database in the form of triggers, stored procedures, custom functions and whatnot. An example of something I deem worthy of database-ifying is data generation - assume our customer has a FirstName and a LastName. Now, the user interface needs a DisplayName, which is derived from some nontrivial logic. For this generation, I create a stored procedure which is then executed by a trigger whenever the row (or other source data) is updated.
There appears to be this somewhat common misunderstanding that the data access layer IS the database and everything about data and data access goes in there "just because". This is simply wrong but I see a lot of designs which derive from this idea. Perhaps this is a local phenomonon, though.
I may just be turned off the idea of SPs after seeing so many badly designed ones. For example, one project I participated in used a set of CRUD stored procedures for every table and every possible query they encountered. In doing so they simply added another completely pointless layer. It is painful to even think about such things.

These days I hardly ever use stored procedures. I only use them for complicated sql queries that can't easily be done in code.
One of the main reasons is because stored procedures do not work as well with OR mappers.
These days I think you need a very good reason to write a business application / information system that does not use some sort of OR mapper.

Stored procedure work as block of code so in place of adhoc query it work fast.
Another thing is stored procedure give recompile option which the best part of
SQL you just use this for stored procedures nothing like this in adhoc query.
Some result in query and stored procedure are different that's my personal exp.
Use cast and covert function for check this.
Must use stored procedure for big projects to improve the performance.
I had 420 procedures in my project and it's work fine for me. I work for last 3 years on this project.
So use only procedures for any transaction.

is it good system architecture if you
let connect 1000 desktops directly to
database?
No it's obviously not, it's maybe a poor example but I think the point I was trying to make is clear, your DBA looks after your database infrastructure this is were their expertise is, stuffing SQL in code locks the door to them and their expertise.