I'm developing an application which queries data from the host, and then inserts them into an SQL database. My application runs in random times on our PC's ( we have 600-700 PCs ) from a server location, so my application isn't located at all PC. It is only located at the server. So it is possible that sometimes it runs on 50 PCs.
I keep getting two types of SQL error messages. The first is time out, and the other is SSL security error. Right now my application executes about 5-10 SQL commands. So i'm thinking to rewrite my code to call stored procedures which could reduce the number of SQL calls. The only question is that is it worth it? I mean of course, it has its advantages, because if i have to change something then it is enough to change the stored procedure, and i don't have to recompile my application. But won't that cause trouble for the SQL server? I mean isn't that a problem when the same stored procedure will be executed 50 times at the same time?
So what way is better? Using stored procedures, or using SQL commands in my code?
Thanks!
If you often call several identical SQL statements in a row, the answer is clear-cut: use SPs to reduce the number of round trips. The load on your SQL server is not going to change, but the network latency will go down due to reduced number of round-trips. As an added bonus, your system architecture may become easier to understand and maintain, because the complex SQL logic would be encapsulated in your data access layer.
This may not have anything to do with your SSL errors, but the situation with timeouts has a decent chance of improving.
Related
We have an AS400 mainframe running our DB2 transactional database. We also have a SQL Server setup that gets loaded nightly with data from the AS400. The SQL Server setup is for reporting.
I can link the two database servers, BUT, there's concern about how big a performance hit DB2 might suffer from queries coming from SQL Server.
Basically, the fear is that if we start hitting DB2 with queries from SQL Server we'll bog down the transactional system and screw up orders and shipping.
Thanks in advance for any knowledge that can be shared.
Anyone who has a pat answer for a performance question is wrong :-) The appropriate answer is always 'it depends.' Performance tuning is best done via measure, change one variable, repeat.
DB2 for i shouldn't even notice if someone executes a 1,000 row SELECT statement. Take Benny's suggestion and run one while the IBM i side watch. If they want a hint, use WRKACTJOB and sort on the Int column. That represents the interactive response time. I'd guess that the query will be complete before they have time to notice that it was active.
If that seems unacceptable to the management, then perhaps offer to test it before or after hours, where it can't possibly impact interactive performance.
As an aside, the RPG guys can create Excel spreadsheets on the fly too. Scott Klement published some RPG wrappers over the Java POI/HSSF classes. Also, Giovanni Perrotti at Easy400.net has some examples of providing an Excel spreadsheet from a web page.
I'd mostly agree with Buck, a 1000 row result set is no big deal...
Unless of course the system is looking through billions of rows across hundreds of tables to get the 1000 rows you are interested in.
Assuming a useful index exists, 1000 rows shouldn't be a big deal. If you have IBM i Access for Windows installed, there's a component of System i Navigator called "Run SQL Scripts" that includes "Visual Explain" that provides a visual explanation of the query execution plan. View that you can ensure that an index is being used.
On key thing, make sure the work is being done on the i. When using a standard linked table MS SQL Server will attempt to pull back all the rows then do it's own "where".
select * from MYLINK.MYIBMI.MYLIB.MYTABE where MYKEYFLD = '00335';
Whereas this format sends the statement to the remote server for processing and just gets back the results:
select * from openquery(MYLINK, 'select * from mylib.mytable where MYKEYFLD = ''00335''');
Alternately, you could ask the i guys to build you a stored procedure that you can call to get back the results you are looking for. Personally, that's my preferred method.
Charles
I have windows server 2008 r2 with microsoft sql server installed.
In my application, I am currently designing a tool for my users, that is querying database to see, if user has any notifications. Since my users can access the application multiple times in a short timespan, i was thinking about putting some kind of a cache on my query logic. But then I thought, that my ms sql server probably does that already for me. Am I right? Or do I need to configure something to make it happen? If it does, then for how long does it keep the cache up?
It's safe to assume that MSSQL will has the caching worked out pretty well =)
Don't bother trying to build anything yourself on top of it, simply make sure that the method you use to query for changes is efficient (eg. don't query on non-indexed columns).
PS: wouldn't caching locally defeat the whole purpose of checking for changes on the database?
Internally the database does all sorts of things, including 'caching', but at all times it works incredibly hard to make sure your users see up-to-date data. So it has to do some work each time your application makes a request.
If you want to reduce the workload by keeping static data in your application then you have to implement it yourself.
The later versions of the .net framework have caching features built in so you should take a look at those (building your own caching can get very complex).
SQL Server will handle caching for you, yes. When you create a query or a stored procedure SQL Server will cache that execution plan and reuse it accordingly. From MSDN:
SQL Server execution plans have the following main components: Query
Plan The bulk of the execution plan is a re-entrant, read-only data
structure used by any number of users. This is referred to as the
query plan. No user context is stored in the query plan. There are
never more than one or two copies of the query plan in memory: one
copy for all serial executions and another for all parallel
executions. The parallel copy covers all parallel executions,
regardless of their degree of parallelism.
Execution Context, each user that is currently executing the query has a data structure that holds
the data specific to their execution, such as parameter values. This
data structure is referred to as the execution context. The execution
context data structures are reused. If a user executes a query and one
of the structures is not being used, it is reinitialized with the
context for the new user.
If you wish to clear this cache you can execute sp_recompile or DBCC FREEPROCHCACHE
I have a database with a rather large number of tables, about 3500, and an application that needs to access a table list.
On a particular server this takes over 2.5 min to return.
EXEC sp_tables #table_type="'TABLE'"
I know there are faster ways to do that but sadly I'm not in a position to modify the application and need to find a way to push it below 30 seconds so the application doesn't throw timeout errors.
So. What, if anything, can I do to improve the performance of this sp within sql server?
I have seen these stored procedures run slow if you do not have the GRANT VIEW DEFINITION permission set on your user account. From what I read, this will cause a security check to occur slowing down the query.
Maybe a SQL guru can comment on why, if this does help your problem.
Well, sp_tables is system code and can't be changed (could workaround in SQL Server 2000, not SQL Server 2005+)
Your options are
Change the SQL
Change command timeout
Bigger server
You've already said "no" to the obvious solutions...
You need to approach this just like any other performance problem. Why is it slow? Namely, where does it block? Disk IO? CPU? Network? Lock contention? The scientific method is to use a methodology like Waits and Queues, or its newer SQL 2008 equivalent Troubleshooting Performance Problems in SQL Server 2008. The lazy way is to simply check the wait_type, wait_time and wait_resource columns in sys.dm_exec_requests for the session executing the sp_tables call. Once you find out what is blocking the execution, you can proceed accordingly.
If I'd venture a guess, you'll discover contention as the cause: other sessions are locking table's metadata exclusively and thus block the execution of sp_tables, which has to wait until all operations in front of it finish.
I have a big stored procedure on a SQL Server 2008 Express SP2 database that gets run about every 200 ms. Normal execution time is about 50ms. What I am seeing is large inconsistencies in this run time. It will execute for while, say 50-100 times at 40-60ms which is expected, then seemingly at random the same stored procedure will take way longer, say 900ms or 1.5 seconds to run. Sometimes more than one call of the same procedure in a row will take longer too.
It appears that something is causing sql server to slow down dramatically every minute or so, but I can't figure out what. There is no timing pattern between the occurences.
I have the same setup on two different computers, one of which is a clean XP Pro load with no virus checking and nothing installed except SQL server.
Also, The recovery options for all the databases are set to "Simple".
I would suggest breaking out applicable sections into their own stored procedures; there is only one query plan cached per batch.
It looks like my problems happen simultaneously with the SQL Server Plan Cache Object Counts hitting 999 and resetting.
I am thinking about creating stored procedures on the fly.
ie running CREATE PROCEDURE... when the (web) application is running.
What are the risks or problems that it can cause?
I know that the database account needs to have the extra privileges.
It does NOT happen everyday. Only from time to time.
I am using sql server and interested in mysql and postgres as well.
Update1:
Thanks to comments, I am considering creating a new version of stored procedure and switching over instead of ALTERing the sp. example: sp1 -> sp2 -> sp3
Update2:
The reason:
My schema changes because of custom fields (unknown number and type of columns)
I tried dynamic sql and sp_executesql first. Of course it works. Dynamic sql works greate for 1,2,3 simple update,inserts.
But it got too ugly and a lot of work and it does not mix well with stored procedure, problems with sql parameterization because it is used inside a stored procedure and the number and type of params is not known at compile time (long story).
At least the basic scenario for this solution is not that complicated.
The logic of the sp does NOT change. For each custom field I have to add a new parameter to sp and add a column to update, insert, etc.
I also considered making stored procedure parameters dynamic like sp_executesql that accepts any number and type of params but could not find a way.
For a transactional system it's probably quite expensive. If you have a large batch job and want to use a code generator for some reason (quite a common architecture in ETL tools, notably Oracle Warehouse Builder and Wherescape Red), it's not unreasonable to do this.
You mentioned that you would be adding and/or changing the calling profile of the stored procedure when you do this alteration. How are you lock-stepping the new calling profile with the application that makes the call to this? What's your roll-back plan if you ever have to revert a change that was made?
In the past what I've done is just append an incrementing numeric suffix to the stored procedure name with the new calling profile -- then you can modify the old version of the SP to call the new one with a default value for the parameter, and then you can release your software calling the new version.
If something breaks in your new version and you have to rollback, calls to the old stored proc will still work without error, and just populate the custom fields with your default values.
Firstly, the answer to this question really depends on what exactly this stored procedure is intended to do. If it's just reading data or creating a result set for reporting and you don't mind if it's a little inconsistent, then you're probably fine. If it's doing anything remotely interesting with your data then it's a very risky thing to be doing. You should think about whether it's possible (and what would happen) for two users users (or the same user twice) to run multiple versions of the the same stored procedure at the same time. Smells like a train wreck to me. One option is to only allow this procedure alteration to take place when no other users are logged into the system, or forcibly boot them off the database if they are. Another option is to create your new stored procedure with a slightly different name and swap them over when you deem it safe to do so.
Another issue is that one of the major benefits of stored procedures is that the execution plan is cached, meaning it will execute faster. If you are creating them on the fly you lose that advantage.
If you really need to do this then you should randomise the name of the procedure to avoid clashing with other users. Remember always that other users may be doing their own thing at the same time - most database systems won't give transactional isolation for stored procedures (Postgres is the only one I know of that does).
It would be extremely rare that this would be a desirable thing to do - could you elaborate at all on what made you choose this approach?
I would not do that personally.
As you mentioned you will need extra privileges to grant access to create/alter database objects. That can create a serious security risk as nothing would stop your application from creating a malicious stored procedure if someone discovered a security hole in it.
If your schema changes, change the stored procedures with the schema.
You will not be able to alter the procedure if one or more users are running the procedure, or another procedure that references your procedure. You will block until all the dependent procedures and the procedure you want to compile (and I think the procedure s you invoke from your procedure, but I am not certain) are not in use. This may be a long time on a busy production system, and if you are unlucky, you may timeout waiting for all the dependencies to not be in use (5 minutes on Oracle).
You can also get into very ugly situations (I have). Take for example stored procedures B and C, both of which call A, the procedure that you are trying to compile. When no one is running B, the system locks B. Now any user trying to run B will stall. The system then tries to lock C, but C is generating a very lengthy report that will not be done for another 10 minutes. You will timeout waiting for the lock, and some of your users will have an unresponsive system for 5 minutes. My experience is with Oracle, I would make sure your target DBMS does not behave in the same fashion, or has quicker failures or a better lock acquisition strategy.
I guess I am cautioning that what looks like may work on a development server may fail dramatically on a busy production system.
I'm not sure that the locking discussed by Tony BanBrahim is true in SQL Server 2005.
I have some long-running SPs (a 3 hours batch process of about 30 sub-processes), and I have been able to alter the SP while it is still running. (I don't believe the changes take effect until the next run, but it doesn't cause any blocking or any error). Now the outer long-running SP does both call SPs dynamically with EXEC and statically, but I've change both the root and nested SPs while they are running without error messages or blocks.
WRT your original question, I would think that your tactic is fine if used in a controlled way.
I don't know for sure, but it sounds like one or both:
an architectural problem
is existing code locking the schema tables from the application?
I'd take a look to see what code is locking the schema tables and rewrite that code. Do you have a 3rd party something or other that is locking those tables?