I develop some jobs with Talend, and use Tlogcatcher to record errors on a database. It seems working for all jobs except one.
Here is how it work:
The first SQL connection read SQL statement from a database table, and for each of them, the tMSSqlRow read and execute it.
But when SQL failed (i.e delete not possible cause of constraint integrity), the error is not caught by the tLogCatcher component.
How I can do that?
tLogCatcher is not supposed to be used like you did in your job (with OnComponentError/OnSubjobError triggers), it has to be the first component of an independent subjob (not linked by a trigger) which gets called whenever there is an error/warning/java exception, depending on which types you check in tLogCatcher settings.
If you want to keep the OnComponentError trigger, you can omit the tLogCatcher altogether and just do your error processing in the subjob triggered by OnComponentError.
Also, make sure you check "Die on error" option in your tMSSqlRow component, otherwise no error is thrown and the job just prints an error message to the console and continues execution, thus the tLogCatcher won't be invoked.
Edit:
Based on your requirement (continue job execution on error), a solution would be to encapsulate the processing that happens starting from tMSSqlRow onward (as well as the error handling technique I suggested above) in a child job.
This child job needs to define a context parameter (for instance QUERY) and does the processing for this single query. The parent job calls this child job with an Iterate trigger, and passing it each query to be processed via the QUERY context parameter (the global variable that is in your tFlowToIterate is passed to QUERY context parameter in the tRunJob parameters tab).
This way, if your query processing in the child job results in an error, it is handled inside the child job by the tLogCatcher, and the parent job isn't aware of this error, so it continues to the next query.
Related
I have a database that is dropped and recreated completely. I want to schedule a SSIS package that:
executes a SQL script that creates TableA
than proceeds to the dataflow task that transfers data from source table to TableA.
Problem is that I get an error that says that TableA does not exist as destination. So the validation sees that in my dataflow task the destination does nog exist.
How can I set the routine, such that step 1 gets done before step 2 in one dtsx?
Please note that solutions such as truncation are not an option for me.
In the SSIS package, under the properties for the Data Flow Task, set Delay Validation property equal to True.
This will delay the validation of the step until run-time, after the table has been created by the previous Execute SQL Task.
Careful that some errors may be suppressed when using this property.
Is there some configuration option for MS SQL Server which enables more verbose error messages.
Specific example: I would like to see the actual field values of the inserted record which violates a constraint during an insert, to help track down a bug in stored procedures which I haven't been able to reproduce.
I don't believe there is any such option. There are trace flags that give more information about deadlocks, but I've never heard of one that gives more information on a constraint violation.
If you control the application that is causing the crash then extending it's handling (as Jenn suggested) to include parameter values etc. Once you have the parameter values you can get a copy of live setup on a non-live server and start debugging the issue.
For more options, can any of the users affected reliably reproduce the issue? If they can then you might be able to run a profiler trace to capture the actual statements / parameter values being sent to the database. Of course, if you can figure out the steps to reproduce the issue then you can probably use more traditional debugging methods...
You don't say what the constraint is, I'm assuming it is a fairly complex constraint. If so, could it be broken down into several constraints so you can get more of a hint about the problem with the data?
You could also re-write the constraint as a trigger which could then include more information in the error that it raises. Although this would obviously need testing before being deployed to a production server!
Personally, I would go with changing the error handling of the application. It is probably the less risky change.
PS The application that I helped write, and now spend my time supporting, logs quite a lot of data when an unhandled exception occurs. If it is during a save then our data access layer attaches the complete list of all commands that were being run as part of the save transaction including parameter values. This has proved to be invaluable on many occasions, including some when tracking down constraint violations.
In a stored proc, what I do to get better informatino in a complex SP about the errors is take advantage of the fact that table variables are not affected by a rollback. So I put the information I want to use to troubleshoot into table variables at the time I create it and then if I hit the catch block and rollback, after the rollback I insert the data from the table variable into an exception table along with some meta data like the datetime.
With some thought you can design an exception table that will capture what you need from just about any proc (for instance you could concatenate all the input variables into one field, you could put in the step number that failed (of course then you have to assign stepnumbers to a variable) or you could log every step along the awy and then the last one logged is the one it failed on. Belive me when you are looking at troubleshooting a 100 line SP, this can come in handy. If I have dymanic SQl inteh proc, I can log the SQL variable that contains the dynamic code that was run.
The beauty of this is now you don't have to try to reproduce the error, you know what the input parameters were and any other information you find useful. Yes it can take a bit of time to set up once, but once you do it is relatively easy to get in the habit of putting it into any complex proc that you will want to log errors on.
You might also want to set a an nongeneralized one if you want to return spefic data values of a select used for an insert or the result set of a select that would tell you waht what wopuld have been being updated or deleted. Then you would have that only if the proc failed. This is a little more work than the general exception table but may be needed in some complex cases.
I'm starting to use the transient fault block provided by Microsoft for SQL Azure and I'm noticing that some of my functions which have a transient handled block may have reference to other functions with this block also.
I am guessing that the retry wait time will then compound?
What you mean by retry time will then compound?
Each query to the SQL Database is executed with its own Retry Logic. If you have a method or a function call that executes 10 Queries, each one of these 10 Queries will execute with respective retry logic.
If there is transient error while executing just one of the queries. Then wait time will be that wait time. If there is transient error during execution of more than one query, then wait time will be sum of all wait times occurred during execution.
However the transient errors are what their name suggests - transient. It is very unlikely that you will encounter a transient error for more than query executed in consecutive order. But if it happens, then yes - the wait times will sum up. However if you execute queries in parallel, or asynchronously, then you don't really sum up wait times.
And finally, retry policy for one query execution, does not affect retry policy for other query execution.
I am trying to get the SQL query to be returned from a database call, when the query fails to execute and the return cursor has a failure state. This query string will be used for logging purposes.
One way to do this is to hold a variable that contains the SQL query as a string, and then wrap each query in the PL/SQL block with the exception handler and if an exception arises, return the query string with the failure status to the UI component.
Is there a better way to do this? Does Oracle exception object or any other package support this feature?
This question seems like a duplicate for the thread: Obtain the Query/CommandText that caused a SQLException
However, I did not find a solution to the problem in that thread, and I would like to know if there are any new packages supported by Oracle that gives us the query string that caused the exception to occur.
When you are calling the database from an external language (Java, C#) you can probably do it, by implementing your own JDBC (or whatever) driver.
You would do so by delegating the real world to a normal driver, but whenever a statement is sent to the database you remember the statement and if causes an exception you create a new exception which also includes the sql statement (and possibly bind variables).
Note that this will only give you your statement. If you execute a stored procedure or cause a trigger to fire which in turn fails you only get the original statement not the one that actually failed.
Story
I have a SPROC using Snapshot Isolation to perform several inserts via MERGE. This SPROC is called with very high load and often in parallel so it occasionally throws an Error 3960- which indicates the snapshot rolled back because of change conflicts. This is expected because of the high concurrency.
Problem
I've implemented a "retry" queue to perform this work again later on, but I am having difficulty reproducing the error to verify my checks are accurate.
Question
How can I reproduce a snapshot failure (3960, specifically) to verify my retry logic is working?
Already Tried
RAISEERROR doesn't work because it doesn't allow me to raise existing errors, only user defined ones
I've tried re-inserted the same record, but this doesn't throw the same failure since it's not two different transactions "racing" another
Open two connections, start a snapshot transaction on both, on connection 1 update a record, on the connection 2 update the same record (in background because it will block), then on connection 1 commit
Or treat a user error as a 3960 ...
Why not just do this:
RAISERROR(3960, {sev}, {state})
Replacing {sev} and {state} with the actual values that you see when the error occurs in production?
(Nope, as Martin pointed out, that doesn't work.)
If not that then I would suggest trying to run your test query multiple times simultaneously. I have done this myself to simulate other concurrency errors. It should be doable as long as the test query is not too fast (a couple of seconds at least).