Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 6 years ago.
Improve this question
What is the best practical way of learning index tuning while writing tsql queries? I have VS2008 SQL Express. Could someone please provide me examples, etc? I have already found online articles and they are great in theory, but I still fail to see index tuning in real life action. Are there small easy to create examples out there?
To tune indexes, you tend to need large tables with lots of data, so small simple examples aren't easy to come by.
My experience is with the SQL 2000 tools. Query Analyser, showing the Execution Plan and looking at the types of index and joins used. Very hard to describe it here.
I can recommend a good book on the subject, particularly Chapter 9.
http://www.amazon.com/Professional-Server-Performance-Tuning-Programmer/dp/0470176393
I would discourage you from using the automated Index Tuning tools until you understand how to do it yourself manually. I think it's important when it recommends adding an index that you have the ability to sanity-check the recommendation and decide for yourself whether it is a good option. Often it will recommend you add a "covering" index with many columns in order to speed up a single query you've asked to be analysed, but this may have adverse effects on your database overall when you look at all queries against that table.
Kimberly Trip (SQL Goddess) is an expert and has talked & written lots on the subject:
http://www.sqlskills.com/BLOGS/KIMBERLY/category/Indexes.aspx
Ken England's Microsoft SQL Server 2000 Performance Optimization and Tuning Handbook is a classic. (The Amazon reviews say the 2005 version of this book is not as good)
If you have SQL Developer edition, you want to have a look at the Database engine tuning advisor.
If you use the profiler to capture a standard workload against your database, the DETA can recommend what statistics and indexes you could apply.
Beware though that tuning requires a good deal of thought, as adding new indexes may increase the speed of your workload, but you may increase the speed of an unimportant query at the detriment of a very important one.
There is a good book about TSQL Querying which has some very good advice about how queries work and how you can look at tuning them, which is MSSQL specific.
In SQL Server, if your database schema is a more simple one, you don't really need to tune much, since primary keys automatically result in a clustered index, and foreign key constraints require a unique index.
So your joins are taken care of, usually.
Where is gets tricky is with searches and filters that the queries in the application use. You will have to specifically look at these queries and identify candidates for additional indices.
Another option is the SQL Server Tuning Advisor, but I discourage it's use, because it generates so much noise. You can use it to find the worst-offending queries, though.
For a starting point of identifying your worst performing queries, please see my answer here.
Learning how to tune is something that requires practice. If you want to become skilled at it, start by buying Sajal Dam's book SQL Server Query Performance Tuning Distilled
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Can anyone direct me to an easy to understand explanation (that doesn't require me to read a 300-page tome) about how to index things and how to identify what to index, what it means to normalize things, etc? Or provide very simple examples?
This is strictly for SQL Server
for every "serious" data table, there must be a well-chosen clustering index. A good clustering index is narrow, unique, static (never changes) and ideally ever-increasing - an INT IDENTITY is as close to perfect as it can be
For any column that is a foreign key, create a nonclustered index. This helps JOINs and other operations and is a generally accepted best practice
Don't OVER-INDEX! - too many indices can be worse than none!
Let your system run for a while, observe how it performs, identify potential performance bottlenecks
if you have identified some performance issues, generate a representative workload (not just a single query; by using server-side tracing) and try to optimize that by using e.g. the Data Tuning Advisor (but don't blindly adopt everything the DTA says!)
Implement one index at a time - observe your system again - did the performance (real or perceived) improve? If so: leave the index - if not: remove it again
Which indexes exactly to use - that's a bit of a black art based on a lot of know-how, experimentation, experience - you really can't give clear, black-or-white kind of ideas here. Try something - observe its benefit (or negative impact) - learn from it. Repeat until your retirement :-)
There's really no 10-page checklist and once you know that, you're done. Either you learn these things yourself, over the years - or you hire someone who has that expertise - your choice. There's no "easy way" to mastering everything about indexing in 5 easy lessons.....
Paul Litwin's Fundamentals of Relational Database Design is worth a read if you're starting from scratch. (22 page tome...) http://sbuweb.tcu.edu/bjones/20263/Access/AC101_FundamentalsDB_Design.pdf
And as he says, database design is more art than science, so examples will only get you so far with improving your own system(s).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Introduction:
So, I have an interview tomorrow and I'm trying to review SQL and databases. The job posting says that they want someone with:
Experience with database design and development
Strong knowledge of SQL
Experience with SQL Server and/or Postgres
I've read through Questions every good database SQL developer should be able to answer, and a bunch of questions tagged with SQL and interview-questions. So I realize that I need to know about SELECT, JOIN and WHERE.
Questions:
What are essential SQL, Postgres and database concepts that I need to know in order to do well in the interview?
What do I need to know about transaction and normalization?
What are some general ways to optimize slow queries?
Should I learn about the functions, keywords or both?
It depends on how much of the role is based around database development and design. For your SQL syntax, you should also understand the difference between the types of joins, and be able to use GROUP BY, ORDER BY, HAVING as well as the aggregate functions that can be used in conjunction with them.
In terms of performance monitoring, I would be looking at execeution plans (not sure about the Postgres equivalent) and how they can provide tips on increasing performance, as well as using SQL Profiler to see what instructions the server is executing in real time.
Transactions can be useful for rolling back, well, transactions (stored procs, ad-hoc queries etc.) that require queries to complete in a certain way to maintain data consistency. Some people (myself included) have a practice of placing any statements that make any changes to data into a transaction that automatically rolls back (BEGIN TRAN ... ROLLBACK TRAN) to check that the correct amount of data is manipulated before pushing changes to a live server. Have a look at the ACID model - Atomicity, Consistency, Isolation, Durability.
Normalization is something that can take a little time to go through, but just know and partially understand up to 3rd form normalization and that will get you started.
Optimisation can be a huge topic. Just remember to try and do things like UPDATE using set based queries, rather than row based (updating in a WHILE loop is an example of row based updating, but it CAN have its uses).
I hope this helps a little.
Besides the basics of sql syntax, which you listed, you should know some things about query performance. What are some common causes of slow queries and what are the remedies for those, and how can you evaluate the performance of a query.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
There was this old sql server tool called Lectoneth or something like that, you'd put sql queries in it, and it would rewrite it for you.
I think quest bought them out, but I can't find where to download a free copy of that software.
Really helps when you have no dba, and have lots of sql queries to rewrite.
Thanks
Craig
Doesn't ring a bell, and presumably you've seen, but nothing obvious on Quest's website
Perhaps a tool like Red Gate's SQL Prompt would help - the Pro edition does SQL reformatting.
Edit
Think i've found what you're looking for, mentioned here - LECCO SQL Expert. The link to the Lecco website does indeed direct to quest, but a 404.
LECCO SQL Expert is the only complete
SQL performance tuning and
optimization solution offering
problematic SQL detection and
automatic SQL rewrite. With its
built-in Artificial Intelligence (AI)
based Feedback Searching Engine, LECCO
SQL Expert reduces the effort required
to optimize SQL and makes even the
most junior programmer an expert.
Developers use LECCO SQL Expert to
optimize SQL during application
development. DBAs eliminate
problematic SQL before users
experience application performance
problems by using LECCO SQL Expert in
production systems.
Looks like it's no longer about - all mentions of I could find indicated it supported up to SQL 2000, and stale links - looks like it wasn't a free tool. As said in my comments, I think this kind of thing is a skill well worth possessing and would benefit in the long run to not relay on a tool to try and do it for you.
I wasn't aware of this tool before now, so I have picked up something from this question - got me intrigued!
Final Update:
To confirm, that product has indeed gone as Lecco was acquired some years ago now. Thanks to Brent Ozar for confirmation.
I think you're looking for a product that's been merged into Toad for SQL Server. The commercial version of Toad has a SQL Optimizer feature that tries lots of ways to rewrite your SQL statements, then tests them to find which ways are the fastest.
You can download Toad here:
http://www.toadsoft.com/
But be aware that that feature is a paid-version-only feature.
Well rather than spending your time looking for a magic bullet, why not spend some time learning performance tuning (you will need a book, this is too complex for the Internet generally). Plus it is my belief that if you want ot write decent new code, you need to understand performance in databases. There is no reason to be unable to write code that avvoids the most common problems.
First, rewrite every query to use ANSII syntax anytime you open it up to revise it for any other reason. Code review all SQl changes and do not pass the code review unless explicit joins were used.
Your first step in performance tuning to identify which queries and procs are causing the trouble. You can use tools that will tell you the worst performing queries in terms of overall time, but don't forget to tune the queries that are run frequnetly as well. Cutting seconds off a query that runs thousands of times a day can really speed things up. Also since you are in oprod already, likely your users are complaining about certain areas, those areas should be looked at first.
Things to look for that cause performance problems:
Cursors
Correlated subqueries
Views that call views
Lack of proper indexing
Functions (especially scalar function that make the query run row by row insted of through a set)
Where clauses that aren't sargeable
EAV tables
Returning more data than you need (If you have anything with select * and a join, immediately fix that.)
Reusing sps that act on one record to loop throuhg a large group of records
Badly designed autogenerated complex queries from ORMs
Incorrect data types resulting in the need to be continually be converting data in order to use it.
Since you have the old style syntax it is highly likely you have a lot of accidental cross joins
Use of distinct when it can be replaced with a derived table instead
Use of union when Union all would work
Bad table design that requires difficult construction of queries that can never perform well. If you find yourself frequently joining to the same table multiple times to get the data you need, then look at the design of the tables.
Also since you have used implicit joins you need to be aware that even in SQL Server 2000 the left and right implicit syntax does not work correctly. Sometimes this interprets as a cross join instead of a left join or right join. I would make it a priority to find and fix all of these queries immediately as they may currently be returning an incorrect result set. Bad data results are even worse that slow data returns.
Good luck.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I work in a group of about 25 developers. I'm responsible for coming up with the database design (tables, views, etc) and am called apon for performance tuning when necessary.
There are a couple of different applications that connect. Database access is via JDBC, hibernate, and iBatis SQL maps. Developers with various levels of experience write SQL statements.
What guidelines would you give to developers to write good SQL?
By good I mean: correct, performs well, easy to understand and maintain.
These are just meant to be easy to follow guidelines - I want to get people onto the right track for the majority of situations. We will break these guidelines when it makes sense.
EDIT: We have in place code reviews for all source commits (SQL, java, etc) enforced through a jira workflow.
If you have 25 developers writing SQL queries against your database you are in quite a bit of trouble. Guidelines are not worth much when your junior developers are learning SQL and checking in a mess.
I would like to offer 4 recommendations
Use an ORM of sorts so your all your devs write less SQL.
Invest in training, buy books, send people to courses.
Have all the SQL reviewed by the senior SQL developers, by all, I mean every SQL statement, no exceptions. This way your senior guys will be able to teach the juniors over time.
Have a single person, who lives and breaths Oracle, responsible for the database. By responsible I mean knows every query, understands all the structure and is able to give expert advice.
Here are some additional things you may add to your existing guidelines/checklist.
Have you tested your queries on a large data set? How was performance?
Have you performed a quick index review on the tables that are being accessed? Are all the right indexes in place? Do you recommend and new indexes?
For high volume queries, are any covering indexes required?
Are you using "NOT IN" in cases where a "LEFT JOIN" should be used?
Is your work transactionally sound? Are you missing a transaction somewhere?
Here's what I already have in my guidelines.
Work in sets, not row by row
The best way to make something go quicker is to avoid doing work you do not have to do
Databases love to join
Fully qualify and specify column names (so SQL does not break when additional columns are added)
Select only the data you need (never select *, never more rows than you require, never every column just becaues it's there)
How to use rownum to limit resultsets
Bind Variables vs Literals (use bind variables in all but a few special cases related to skewed data)
Avoid functions or calculations on columns in the WHERE clause (except for a special case of function based index)
Use ORDER BY for all queries returning more than one row (this is mostly for testability)
Each of these points is expanded a bit in the actual guidelines I've written out with an example relevant to our database schema.
Read Tom Kyte's books. He explains how you can write fast code and how you can measure performance and scalability. If you have a problem you can probably find the answer on the "ask tom"-site.
Introduce basic style guide that covers:
naming (of everything - tables, columns, procedures, aliases, ...) .
formatting style
line width
what reserved words require new line (e.g where)
are reserved word capitalized or small caps
indenting
...
Here are some examples:
Oracle PL/SQL Programming, Fourth Edition. There is older, 2nd edition - available online
SQL and PL/SQL Coding Standards
Be very strict about naming, it will be easier for you to read other people's code.
As formatting is concerned, there are tools available that can format automatically, so maybe you don't need very detailed description here.
If you are a database developer, you need to know what an EXECUTION PLAN is. If you don't then go mine coal or something.
Before developing:
first, you think what best EXECUTION PLAN will be,
second you create tables and indexes, and
third you use hints to persuade the optimizer to come out with the plan you made.
You do use hints. Forget automatic optimization, it's a marketing myth. No optimizer knows your data better than you and never will.
There are no "programmers who create queries" and "system administrators who create indexes". Programmers program, system administrators make backups (or whatever they make).
Triggers are evil.
Prefix you columns, tables and views (SELECT prs_name FROM t_person)
Make lines and indent
An hour long presentation on some Oracle fundamentals (eg parsing, SGA vs PGA). "Do this" rules may or may not apply to your situation. Give them an understanding of what the DB side does, and they at least have a basis on which to make a decision.
Plus Code reviews.
Pair-program. Any advantange it provides for agile development in general, at least doubles for SQL development.
Second choice, code reviews for all SQL.
Along with the recommendation to have queries reviewed by senior programmers, if you can get the buy-in, have code reviews which involve as many team members as possible.
I'm by no means a guru but here are my tips:
Don't use ORDER BY unless you really need an ordered list as it incurs a performance hit.
Understand the explain plan and also recognise that the plan on your development environment is often different from your production environment. Don't expect it to accurately reflect real life performance
The pros of using hints is that you get to choose your explain plan, the cons of using hints is that the optimal plan may change over time and you might be choosing a plan that is suboptimal in the long term
Make sure the developers know when to use INNER JOIN, OUTER JOIN, [NOT] IN, [NOT] EXISTS - you can put in place a lot of processes but one or two Cartesian products will bring production performance to its knees
Ensure your developers understand indexes - what they are, when they should be used, when they should be avoided
Have a DBA monitor the most executed queries and the most expensive queries and highlight these as candidates for optimisation
Peer review
Coding standards (especially code comments on particularly long/complex queries)
Unit testing
Don't write SQL if you can help it, use HQL (or JPQL is on Java EE) whenever possible
Don't use SELECT *
Pick your internet sources wisely (e.g. asktom.oracle.com)
Don't use cursors
Don't do string concatenation in SQL
Write queries such that they use indexes (fundamentally this means base WHERE predicates on the indexes that exist)
use MERGE instead of other awkward 'upsert' type logic
When working with dates, make sure you understand how they're stored in Oracle vs. how they are stored in Java, especially when it relates to TimeZone. Depending on the Calendar/Date types, this information can be stripped out, remapped to the TZ of the default locale, etc.
Most importantly: Don't use the excuse of being a developer for not knowing how to write good SQL, and how the database works. You don't have to be a DBA, but you need to invest in your own training to make yourself suitable for the task. By the same token, your company needs to invest in that as well.
I don't mean to say that these "Don'ts" always apply. It's just that, if you're talking about a developer who is not comfortable with Oracle, they need to know what they're doing before they start deciding whether those types of things are necessary and appropriate.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
Do you know a good source to learn how to design SQL solutions?
Beyond the basic language syntax, I'm looking for something to help me understand:
What tables to build and how to link them
How to design for different scales (small client APP to a huge distributed website)
How to write effective / efficient / elegant SQL queries
I started with this book: Relational Database Design Clearly Explained (The Morgan Kaufmann Series in Data Management Systems) (Paperback) by Jan L. Harrington and found it very clear and helpful
and as you get up to speed this one was good too Database Systems: A Practical Approach to Design, Implementation and Management (International Computer Science Series) (Paperback)
I think SQL and database design are different (but complementary) skills.
I started out with this article
http://en.tekstenuitleg.net/articles/software/database-design-tutorial/intro.html
It's pretty concise compared to reading an entire book and it explains the basics of database design (normalization, types of relationships) very well.
Experience counts for a lot, but in terms of table design you can learn a lot from how ORMs like Hibernate and Grails operate to see why they do things. In addition:
Keep different types of data separate - don't store addresses in your order table, link to an address in a separate addresses table, for example.
I personally like having an integer or long surrogate key on each table (that holds data, not those that link different tables together, e,g., m:n relationships) that is the primary key.
I also like having a created and modified timestamp column.
Ensure that every column that you do "where column = val" in any query has an index. Maybe not the most perfect index in the world for the data type, but at least an index.
Set up your foreign keys. Also set up ON DELETE and ON MODIFY rules where relevant, to either cascade or set null, depending on your object structure (so you only need to delete once at the 'head' of your object tree, and all that object's sub-objects get removed automatically).
If you want to modularise your code, you might want to modularise your DB schema - e.g., this is the "customers" area, this is the "orders" area, and this is the "products" area, and use join/link tables between them, even if they're 1:n relations, and maybe duplicate the important information (i.e., duplicate the product name, code, price into your order_details table). Read up on normalisation.
Someone else will recommend exactly the opposite for some or all of the above :p - never one true way to do some things eh!
I really liked this article..
http://www.codeproject.com/Articles/359654/important-database-designing-rules-which-I-fo
Head First SQL is a great introduction.
These are questions which, in my opionion, requires different knowledge from different domains.
You just can't know in advance "which" tables to build, you have to know the problem you have to solve and design the schema accordingly;
This is a mix of database design decision and your database vendor custom capabilities (ie. you should check the documentation of your (r)dbms and eventually learn some "tips & tricks" for scaling), also the configuration of your dbms is crucial for scaling (replication, data partitioning and so on);
again, almost every rdbms comes with a particular "dialect" of the SQL language, so if you want efficient queries you have to learn that particular dialect --btw. much probably write elegant query which are also efficient is a big deal: elegance and efficiency are frequently conflicting goals--
That said, maybe you want to read some books, personally I've used this book in my datbase university course (and found a decent one, but I've not read other books in this field, so my advice is to check out for some good books in database design).
It's been a while since I read it (so, I'm not sure how much of it is still relevant), but my recollection is that Joe Celko's SQL for Smarties book provides a lot of info on writing elegant, effective, and efficient queries.