I'm using MySQL and am getting the "MySQL server went away" error. I've tried setting ATTR_TIMEOUT in the PDO constructor as well as with the setAttribute() method, but neither are working. I've also set the MySQL connect_timeout variable.
No matter what time I specify PDO, within seconds, returns the timeout error.
I'm using a prepared statement for a very large insert. Is there a size restriction on PDO prepared statements that's breaking the connection? I've not been able to find any documentation referring to such a limitation.
The only length limitation is that of max_allowed_packet or net_buffer_length (most likely the former). I suggest raising max_allowed_packet if you have an INSERT of a very large row.
If you have an INSERT of many rows, you might be exceeding max_allowed_packet that way too. But it's easier to manage this by INSERTing a limited number of rows.
If you examine the output of mysqldump, for example, they don't try to insert the whole table in a single INSERT statement. They do use multi-row INSERT, but they do it in batches.
Related
Is there some configuration option for MS SQL Server which enables more verbose error messages.
Specific example: I would like to see the actual field values of the inserted record which violates a constraint during an insert, to help track down a bug in stored procedures which I haven't been able to reproduce.
I don't believe there is any such option. There are trace flags that give more information about deadlocks, but I've never heard of one that gives more information on a constraint violation.
If you control the application that is causing the crash then extending it's handling (as Jenn suggested) to include parameter values etc. Once you have the parameter values you can get a copy of live setup on a non-live server and start debugging the issue.
For more options, can any of the users affected reliably reproduce the issue? If they can then you might be able to run a profiler trace to capture the actual statements / parameter values being sent to the database. Of course, if you can figure out the steps to reproduce the issue then you can probably use more traditional debugging methods...
You don't say what the constraint is, I'm assuming it is a fairly complex constraint. If so, could it be broken down into several constraints so you can get more of a hint about the problem with the data?
You could also re-write the constraint as a trigger which could then include more information in the error that it raises. Although this would obviously need testing before being deployed to a production server!
Personally, I would go with changing the error handling of the application. It is probably the less risky change.
PS The application that I helped write, and now spend my time supporting, logs quite a lot of data when an unhandled exception occurs. If it is during a save then our data access layer attaches the complete list of all commands that were being run as part of the save transaction including parameter values. This has proved to be invaluable on many occasions, including some when tracking down constraint violations.
In a stored proc, what I do to get better informatino in a complex SP about the errors is take advantage of the fact that table variables are not affected by a rollback. So I put the information I want to use to troubleshoot into table variables at the time I create it and then if I hit the catch block and rollback, after the rollback I insert the data from the table variable into an exception table along with some meta data like the datetime.
With some thought you can design an exception table that will capture what you need from just about any proc (for instance you could concatenate all the input variables into one field, you could put in the step number that failed (of course then you have to assign stepnumbers to a variable) or you could log every step along the awy and then the last one logged is the one it failed on. Belive me when you are looking at troubleshooting a 100 line SP, this can come in handy. If I have dymanic SQl inteh proc, I can log the SQL variable that contains the dynamic code that was run.
The beauty of this is now you don't have to try to reproduce the error, you know what the input parameters were and any other information you find useful. Yes it can take a bit of time to set up once, but once you do it is relatively easy to get in the habit of putting it into any complex proc that you will want to log errors on.
You might also want to set a an nongeneralized one if you want to return spefic data values of a select used for an insert or the result set of a select that would tell you waht what wopuld have been being updated or deleted. Then you would have that only if the proc failed. This is a little more work than the general exception table but may be needed in some complex cases.
We have an SSIS package that ran in production on a SQL 2008 box with a 2005 compatibility setting. The package contains a SQL Task and it appears as though the SQL at the end of the script did not run.
The person who worked on that package noted before leaving the company that the package needed "GOs" between the individual SQL commands to correct the issue. however, when testing in development on SQL Server 2008 with 2008 compatibility, the package worked fine.
From what I know, GO's place commands in batches, where commands are sent to the database provider in a batch, for efficiency's sake. I am thinking that the only way that GO should effect the outcome is if there was an error in that script somewhere above it. I can imagine GO in that case, and only that case, effecting the outcome. However, we have seen no evidence of any errors logged.
Can someone suggest to me whether or not GO is even likely related to the problem? Assuming no error was encountered, my understanding of the "GO" command suggests that it use or lack of use is most likely unrelated to the problem.
The GO keyword is, as you say, a batch separator that is used by the SQL Server management tools. It's important to note, though, that the keyword itself is parsed by the client, not the server.
Depending on the version of SQL Server in question, some things do need to be placed into distinct batches, such as creating and using a database. There are also some operations that must take place at the beginning of a batch (like the use statement), so using these keywords means that you'll have to break the script up into batches.
A couple of things to keep in mind about breaking a script up into multiple batches:
When an error is encountered within a batch, execution of that batch stops. However, if your script has multiple batches, an error in one batch will only stop that batch from executing; subsequent batches will still execute
Variables declared within a batch are available to that batch only; they cannot be used in other batches
If the script is performing nothing but CRUD operations, then there's no need to break it up into multiple batches unless any of the above behavioral differences is desired.
All of your assumptions are correct.
One thing that I've experienced is that if you have a batch of statements that is a pre-requisite for another batch, you may need to separate them with a GO. One example may be if you add a column to a table and then update that column (I think...). But if it's just a series of DML queries, then the absence or presence of GO shouldn't matter.
I've noticed that if you set up any variables in the script their state (and maybe the variables themselves) are wiped after a 'GO' statement so they can't be reused. This was certainly the case on SQL Server 2000 and I presume it will be the case on 2005 and 2008 as well.
Yes, GO can affect outcome.
GO between statements will allow execution to continue if there is an error in between. For example, compare the output of these two scripts:
SELECT * FROM table_does_not_exist;
SELECT * FROM sys.objects;
...
SELECT * FROM table_does_not_exist;
GO
SELECT * FROM sys.objects;
As others identified, you may need to issue GO if you need changes applied before you work on them (e.g. a new column) but you can't persist local or table variables across GO...
Finally, note that GO is not a T-SQL keyword, it is a batch separator. This is why you can't put GO in the middle of a stored procedure, for example ... SQL Server itself has no idea what GO means.
EDIT however one answer stated that transactions cannot span batches, which I disagree with:
CREATE TABLE #foo(id INT);
GO
BEGIN TRANSACTION;
GO
INSERT #foo(id) SELECT 1;
GO
SELECT ##TRANCOUNT; -- 1
GO
COMMIT TRANSACTION;
GO
DROP TABLE #foo;
GO
SELECT ##TRANCOUNT; -- 0
I have a SQL Script that inserts about 8000 rows into a TABLE variable.
After inserting into that variable, I use a WHILE loop to loop over that table and perform other operations. That loop is perhaps 60 lines of code.
If I run the TABLE variable insert part of the script, without the while loop, it takes about 5 seconds. That's great.
However, if I run the entire script, it takes about 15 minutes.
Here's what is interesting and what I can't figure out:
When I run the entire script, I don't see any print statements until many minutes into the script.
Then, once it figures out what to do (presumably), it runs the inserts into the table var, does the loop, and that all goes rather fast.
Then, toward the end of the loop, or even after it, it sits and hangs for many more minutes. Finally, it chugs through the last few lines or so of the script that come after the loop.
I can account for all the time taken during the insert, and then all the time taken in the loop. But I can't figure out why it appears to be hanging for so many minutes before and at the end of the script.
for kicks, I added a GO statement after the insert into the temp table, and everything up to that point ran as you'd expect; however, I can't do that because I need that variable, and the GO statement obviously kills that variable.
I believe I'm going to stop using the table variable and go with a real table so that I can issue the GO, but I would really like to know what's going on here.
Any thoughts on what SQL Server is up to during that time?
Thanks!
You can always check what a script is doing from the Activity Monitor or from the sys.dm_exec_requests view. The script will be blocked by something, and you'll be able to see what is that is blocking in the wait_type and wait_resource columns.
There are several likely culprits, like waiting on row locks or table locks, but from the description of the problem I suspect is a database or log growth event. Those tend to be very expensive once the database is a big enough and the default 10% increase means growth of GBs. If that's the case, try to pre-size the database at the required size and make sure Instant File Initialization is enabled for data files.
PRINTs are buffered, so you can't judge performance from them.
Use RAISERROR ('Message', 0, 1) WITH NOWAIT to see the output immediately.
To understand what the process is doing, I'd begin with calling sp_who2 a few times and looking at the values for the process of interest: isn't it being blocked, what are the wait types if any, and so on. Also, just looking at the server hardware load (CPU, disk activity) might help (unless there're other active processes).
And please post some code. Table var definition and the loop will be enough, I believe, no need for INSERT stuff.
If you are using the table variable, can you try substituting it with temp table and see if there is any change in performance?
And if possible, please post the code so that it can be analysed for possible area of interest.
From the wording of your question, it sounds like you're using a cursor to loop through the table. If this is the case, issuing a "SET NOCOUNT ON" command before starting the loop will help.
The table variable was mentioned in a previous answer, but as a general rule, you should really use a temp table if you have more than a few rows.
This is a pretty specific question, albeit possibly subjective, but I've been using this pattern very frequently while not seeing others use it very often. Am I missing out on something or being too paranoid?
I wrap all my UPDATE,DELETE,INSERT operations in stored procedures, and only give EXECUTE on my package and SELECT on my tables, to my application. For the UPDATE and DELETE procedures I have an IF statement at the end in which I do the following:
IF SQL%ROWCOUNT <> 1 THEN
RAISE_APPLICATION_ERROR(-20001, 'Invalid number of rows affected: ' || SQL%ROWCOUNT);
END IF;
One could also do this check in the application code, as the number of rows affected is usually available after a SQL statement is executed.
So am I missing something or is this not the safest way to ensure you're updating or deleting exactly what you want to, nothing more, nothing less?
I think this is a fine way to go. If the pl/sql proc is expected to always update/delete/insert a row and it's considered an error otherwise, then what better place to put this check than in the pl/sql proc itself? That way, no matter what client side code (C#, JAVA, PHP, Rails, etc.) uses this proc, you have this error check centralized in one place.
However, I'm not sure you need the check for an insert. If the insert fails, you should get some sort of DB exception, so no need to check for it explicitly unless you are wrapping the error in some other error message and re-raising it.
In most cases I'd use an ORM like Hibernate, which does a similar thing in order to handle Optimistic locking. Also it will use the PK in the where clause.
So I would consider this kind of stored procedure a waste of time:
- A lot of effort for minimal benefit
- Makes usage of tools like ORMs harder, which solve more and more important problems.
A client has reported repeated instances of Very strange behaviour when executing a stored procedure.
They have code which runs off a cached transposition of a volatile dataset. A stored proc was written to reprocess the dataset on demand if:
1. The dataset had changed since the last reprocessing
2. The datset has been unchanged for 5 minutes
(The second condition stops massive repeated recalculation during times of change.)
This worked fine for a couple of weeks, the SP was taking 1-2 seconds to complete the re-processing, and it only did it when required. Then...
The SP suddenly "stopped working" (it just kept running and never returned)
We changed the SP in a subtle way and it worked again
A few days later it stopped working again
Someone then said "we've seen this before, just recompile the SP"
With no change to the code we recompiled the SP, and it worked
A few days later it stopped working again
This has now repeated many, many times. The SP suddenly "stops working", never returning and the client times out. (We tried running it through management studio and cancelled the query after 15 minutes.)
Yet every time we recompile the SP, it suddenly works again.
I haven't yet tried WITH RECOMPILE on the appropriate EXEC statments, but I don't particularly want to do that any way. It gets called hundred of times an hour and normally does Nothing (It only reprocesses the data a few times a day). If possible I want to avoid the overhead of recompiling what is a relatively complicated SP "just to avoid something which "shouldn't" happen...
Has anyone experienced this before?
Does anyone have any suggestions on how to overcome it?
Cheers,
Dems.
EDIT:
The pseduo-code would be as follows:
read "a" from table_x
read "b" from table_x
If (a < b) return
BEGIN TRANSACTION
DELETE table_y
INSERT INTO table_y <3 selects unioned together>
UPDATE table_x
COMMIT TRANSACTION
The selects are "not pretty", but when executed in-line they execute in no time. Including when the SP refuses to complete. And the profiler shows it is the INSERT at which the SP "stalls"
There are no parameters to the SP, and sp_lock shows nothing blocking the process.
This is the footprint of parameter-sniffing. Yes, first step is to try RECOMPILE, though it doesn't always work the way that you want it to on 2005.
Update:
I would try statement-level Recompile on the INSERT anyway as this might be a statistics problem (oh yeah, check that automatics statistics updating is on).
If this does not seem to fit parameter-sniffing, then compare th actual query plan from when it works correctly and from when it is running forever (use estimated plan if you cannot get the actual, though actual is better). You are looking to see if the plan changes or not.
I totally agree with the parameter sniffing diagnosis. If you have input parameters to the SP which are varying (or even if they aren't varying) - be sure to mask them with a local variable and use the local variable in the SP.
You can also use the WITH RECOMPILE if the set is changing but the query plan is no longer any good.
In SQL Server 2008, you can use the OPTIMIZE FOR UNKNOWN feature.
Also, if your process involves populating a table and then using that table in another operation, I recommend breaking the process up into separate SPs and calling them individually WITH RECOMPILE. I think the plans generated at the outset of the process can sometimes be very poor (so poor as not to complete) when you populate a table and then use the results of that table to carry out an operation. Because at the time of the initial plan, the table was a lot different than after the initial insert.
As others have said, something about the way the data or the source table statistics are changing is causing the cached query plan to go stale.
WITH RECOMPILE will probably be the quickest fix - use SET STATISTICS TIME ON to find out what the recompilation cost actually is before dismissing it out of hand.
If that's still not an acceptable solution, the best option is probably to try to refactor the insert statement.
You don't say whether you're using UNION or UNION ALL in your insert statement. I've seen INSERT INTO with UNION produce some bizarre query plans, particularly on pre-SP2 versions of SQL 2005.
Raj's suggestion of dropping and
recreating the target table with
SELECT INTO is one way to go.
You could also try selecting each of
the three source queries into their own
temporary table, then UNION those temp tables
together in the insert.
Alternatively, you could try a
combination of these suggestions -
put the results of the union into a
temporary table with SELECT INTO,
then insert from that into the target
table.
I've seen all of these approaches resolve performance problems in similar scenarios; testing will reveal which gives the best results with the data you have.
Obviously changing the stored procedure (by recompiling) changes the circumstances that led to the lock.
Try to log the progress of your SP as described here or here.
I would agree with the answer given above in a comment, this sounds like an unclosed transaction, particularly if you are still able to run the select statement from query analyser.
Sounds very much like there is an open transaction with a pending delete for table_y and the insert can't happen at this point.
When your SP locks up, can you perform an insert into table_y?
Do you have an index maintenance job?
Are your statistics up to date? One way to tell is examine the estimated and actual query plans for large variations.
As others have said, this sounds very likely to be an uncommitted transaction.
My best guess:
You'll want to make sure that table_y can be deleted completely and quickly.
If there are other stored procedures or external pieces of code that ever hold transactions on this table, you may be waiting forever. (They may error out and never close the transaction)
Another note: try using truncate if possible. it uses fewer resources than a delete with no where clause:
truncate table table_y
Also, once an error happens within your OWN transaction, it will cause all following calls (every 5 minutes apparently) to "hang", unless you handle your error:
begin tran
begin try
-- do normal stuff
end try
begin catch
rollback
end catch
commit
The very first error is what will give you information about the actual error. Seeing it hang in your own subsequent tests is just a secondary effect.
If you are doing these steps:
DELETE table_y
INSERT INTO table_y <3 selects unioned together>
You might want to try this instead
DROP TABLE table_y
SELECT INTO table_y <3 selects unioned together>