Automating problem query identification in Oracle 11g - sql

In our test bed, a number of test suites will be run in a row (unattended), producing reports for later consumption. I want to include in those reports queries which are candidates for further investigation, along with the data that justifies their inclusion in that list. We should be able to associate any query identified this way with the test suite that exposed it as a concern.
When we use SQL Server, this is relatively straight forward - a call to DBCC FREEPROCCACHE clears all of the counters before a suite begins, then at test end we run a query against sys.dm_exec_query_stats, which gives us access to the execution counts and min/max/total time(s) of each cached query plan, with hooks available to retrieve the parameterized SQL statement (we use FORCED parametrization in our mssql instances) and the query plan.
Ref: http://msdn.microsoft.com/en-us/library/ms189741%28SQL.90%29.aspx
My question: how do I implement an approximation for this when my target app has been connected to Oracle 11g? My reading thus far suggests that everything I'm after is available via the AWR, and that it should be possible to access the supporting views directly, but I haven't been able to close the circle on my own.

Why do you need to access the supporting views directly? It would seem to me that the simplest solution would be
Each test suite starts and ends by explicitly generating an AWR snapshot so it knows the starting and ending snapshot ID and so that you can generate AWR reports for each suite individually.
You run AWR reports for each test suite
You review the AWR reports looking in particular at the various Top SQL sections
It's absolutely possible to get all the information from the underlying views directly, but if you don't need to do so, that's obviously easier.
Just for sanity, I should point out that I am assuming you are licensed to use AWR. Technically, even querying the AWR views requires that you have licensed the Performance and Tuning Pack. If you want to hit the views directly rather than generating full AWR reports because of licensing concerns, you're not saving yourself any license headaches by hitting the views.

The Oracle equivalent of DBCC FREEPROCCACHE is
SQL> alter system flush shared_pool;

The closest to the SQL Server counters are V$SYSSTAT and V$SYSTEM_EVENT.
However Oracle actually tracks these at the session level too in v$SESSION_WAIT, V$SESSION_WAIT_CLASS and V$SESSION_EVENT so you don't need to reset them at the system level.
And you don't need the Diagnostic/Tuning pack licenses to access them.
They don't go down to the SQL level. That is available in V$SQL, though would not be specific to that session. You can use session level tracing to track down individual SQLs that may be causing problems

Justin's answer had the correct outline, but I needed more details about the implementation.
Each test suite starts and ends by explicitly generating an AWR snapshot so it knows the starting and ending snapshot ID and so that you can generate AWR reports for each suite individually.
You run AWR reports for each test suite
You review the AWR reports looking in particular at the various Top SQL sections
I explicitly generate the snapshots by calling dbms_workload_repository.create_snapshot, the result gets saved off for later.
select dbms_workload_repository.create_snapshot() as snap_id from dual
In order to get the report, I need the database id and the instance number. Those are easily obtained from v$database and v$instance.
select d.DBID, i.instance_number as inst_num from v$database d, v$instance i
The report is available as text (dbms_workload_repository.awr_report_text) or html (dbms_workload_repository.awr_report_html). The arguments are the same in both cases, including a options flag which will include information from the Automatic Diagnostic Database Monitor (ADDM). It wasn't immediately obvious to me that I could leverage the ADDM results, so I turn that off. The return value is a column of varchar, so the function call gets wrapped
select output from table(dbms_workload_repository.awr_report_html(1043611354,1,5539,5544,0))
This result is easily written to a file, which is assembled with the other artifacts of the test.
Documentation of these methods is available online

Related

SQL Server sp_clean_db_free_space

I just want to ask if I can use the stored procedure sp_clean_db_free_space as part of preventive maintenance?
What are the pros and cons of this built in stored procedure? I just want to hear the inputs of someone who already uses this.
Thank you
You would typically not need to explicitly run this, unless you have specific reasons.
Quoting from SQL Server Books Online's page for sp_clean_db_free_space:
Delete operations from a table or update operations that cause a row
to move can immediately free up space on a page by removing references
to the row. However, under certain circumstances, the row can
physically remain on the data page as a ghost record. Ghost records
are periodically removed by a background process. This residual data
is not returned by the Database Engine in response to queries.
However, in environments in which the physical security of the data or
backup files is at risk, you can use sp_clean_db_free_space to clean
these ghost records.
Notice that SQL Server already has a background process to achieve the same result as sp_clean_db_free_space.
The only reason you might wish to explicitly run sp_clean_db_free_space is if there is a risk that the underlying database files or backups can be compromised and analysed. In such cases, any data that has not yet been swept up by the background process can be exposed. Of course, if your system has been compromised in such a way, you probably also have bigger problems on your hands!
Another reason might be that you have a time-bound requirement that deleted data should not be retained in any readable form. If you have only a general requirement that is not time-bound, then it would be acceptable to wait for the regular background process to perform this automatically.
The same page also mentions:
Because running sp_clean_db_free_space can significantly affect I/O
activity, we recommend that you run this procedure outside usual
operation hours.
Before you run sp_clean_db_free_space, we recommend
that you create a full database backup.
To summarize:
You'd use the sp_clean_db_free_space stored procedure only if you have a time-bound requirement that deleted data should not be retained in any readable form.
Running sp_clean_db_free_space is IO intensive.
Microsoft recommend a full database backup prior to this, which has its own IO and space requirements.
Take a look at this related question on dba.stackexchange.com: https://dba.stackexchange.com/questions/11280/how-can-i-truly-delete-data-from-a-sql-server-table-still-shows-up-in-notepad/11281

Get ALL sql queries executed in the last several hours in Oracle

I need a way to collect all queries executed in Oracle DB (Oracle Database 11g) in the last several hours regardless of how fast the queries are (this will be used to calculate sql coverage after running tests against my application that has several points where queries are executed).
I cannot use tables like V$SQL since there's no guarantee that a query will remain there long enough. It seems I could use DBA_HIST_SQLTEXT but I didn't find a way to filter out queries executed before current test run.
So my question is: what table could I use to get absolutely all queries executed in the given period of time (up to 2 hours) and what DB configuration should I adjust to reach my goal?
"I need the query itself since I need to learn which queries out of all queries coded in the application are executed during the test"
The simplest way of capturing all the queries executed would be to enable Fine Grained Audit on all your database tables. You could use the data dictionary to generate policies on every application table.
Note that even when writing to an OS file such a number of policies would generate a high impact on the database, and will increase the length of time it takes to run the tests. Therefore you should only use these policies to assess your test coverage, and disable them for other test runs.
Find out more.
In the end I went with what Justin Cave suggested and instrumented Oracle JDBC driver to collect every executed query in a Set and them dump them all into a file after running the tests.

How to get SQL executed or transaction history on a Table (AS400) DB2 for IBM i

I have an issue in our database(AS400- DB2) in one of our tables all the rows were deleted. I do not know if it was a program or SQL that a user executed. All I know it hapend +- 3am in the morning. I did check for any scheduled jobs at that time.
We managed to get the data back from backups but I want to investigate what deleted the records or what user.
Are there any logs on die as400 on physical tables to check what SQL executed and when on a specified table? This will help me determine what caused this.
I tried checking I systems navigator but could not find any logs... Is there a way of getting transnational data on a table using i system navigator or green screen? And If I can get the SQL that executed in the timeline.
Any help would be appreciated.
There was no mention of how the time was inferred\determined, but for lack of journaling, I would suggest a good approach is immediately to gather information about the file and member; DSPOBJD for both *SERVICE and *FULL, DSPFD for *ALL, DMPOBJ, and perhaps even a copy of the row for the TABLE from the catalog [to include the LAST_ALTERED_TIMESTAMP for ALTEREDTS column of SYSTABLES or the based-on field DBXATS from the QADBXREF]. Gathering those, worthwhile almost only if done before any other activity [esp. before any recovery activity], can help establish the time of the event and perhaps allude to what was the event; most timestamps are reflective of only of the most recent activity against the object [rather than as a historical log], so any recovery activity is likely to effect loss of any timestamps that would be reflective of the prior event\activity.
Even if there was no journal for the file and nothing in the plan cache, there may have been [albeit unlikely] an active SQL Monitor. An active monitor should be available visible somewhere in the iNav GUI as well. I am not sure of the visibility of a monitor that may have been active in a prior time-frame.
Similarly despite lack of journaling, there may be some system-level object or user auditing in effect for which the event was tracked either as a command-string or as an action on the file.member; combined with the inferred timing, all audit records spanning just before until just after can be reviewed.
Although there may have been nothing in the scheduled jobs, the History Log (DSPLOG) since that time may show jobs that ended, or [perhaps soon] prior to that time show jobs that started, which are more likely to have been responsible. In my experience, often the name of the job may be indicative; for example the name of the job as the name of the file, perhaps due only to the request having been submitted from PDM. Any spooled [or otherwise still available] joblogs could be reviewed for possible reference to the file and\or member name; perhaps a completion message for a CLRPFM request.
If the action may have been from a program, the file may be recorded as a reference-object such that output from DSPPGMREF may reveal programs with the reference, and any [service] program that is an SQL program could have their embedded SQL statements revealed with PRTSQLINF; the last-used for those programs could be reviewed for possible matches. Note: module and program sources can also be searched, but there is no way to know into what name they were compiled or into what they may have been bound if created only temporarily for the purpose of binding.
Using System i Navigator, expand Databases. Right click on your system database. Select SQL Plan Cache-> Show Statements. From here, you can filter based on a variety of criteria.
This is not sure-fire, but often saves me some time. Using System i Navigator, right-click on the table and choose Index Advisor. If you're lucky, one or more indexes are advised. If so, sort by date last advised and right click on the index with the newest date and select Show Statements... In that dialog box, either sort by date to help narrow things down or just scroll through the statements to find the one you're interested in. Right-click it and select Work with SQL Statement and there you go.

Why does my SELECT query take so much longer to run on the web server than on the database itself?

I'm running the following setup:
Physical Server
Windows 2003 Standard Edition R2 SP2
IIS 6
ColdFusion 8
JDBC connection to iSeries AS400 using JT400 driver
I am running a simple SQL query against a file in the database:
SELECT
column1,
column2,
column3,
....
FROM LIB/MYFILE
No conditions.
The file has 81 columns - aplhanumeric and numeric - and about 16,000 records.
When I run the query in the emulator using the STRSQL command, the query comes back immediately.
When I run the query on my Web Server, it takes about 30 seconds.
Why is this happening, and is there any way to reduce this time?
While I cannot address whatever overhead might be involved in your web server, I can say there are several other factors to consider:
This may likely have to do primarily in the differences between the way the two system interfaces work.
Your interactive STRSQL session will start displaying results as quickly as it receives the first few pages of data. You are able to page down through that initial data, but generally at some point you will see a status message at the bottom of the screen indicating that it is now getting more data.
I assume your web server is waiting until it receives the entire result set. It wants to get all the data as it is building the HTML page, before it sends the page. Thus you will naturally wait longer.
If this is not how your web server application works, then it is likely to be a JT400 JDBC Properties issue.
If you have overridden any default settings, make sure that those are appropriate.
In some situations the OPTIMIZATION_GOAL settings might be a factor. But if you are reading the table (aka physical file or PF) directly, in its physical sequence, without any index or key, then that might not apply here.
Your interactive STRSQL session will default to a setting of *FIRSTIO, meaning that the query is optimized for returning the first pages of data quickly, which corresponds to the way it works.
Your JDBC connection will default to a "query optimize goal" of "0", which will translate to an OPTIMIZATION_GOAL setting of *ALLIO, unless you are using extended dynamic packages. *ALLIO means the optimizer will try to minimize the time needed to return the entire result set, not just the first pages.
Or, perhaps first try simply adding FOR READ ONLY onto the end of your SELECT statement.
Update: a more advanced solution
You may be able to bypass the delay caused by waiting for the entire result set as part of constructing the web page to be sent.
Send a web page out to the browser without any records, or limited records, but use AJAX code to load the remainder of the data behind the scenes.
Use large block fetches whenever feasible, to grab plenty of rows in one clip.
One thing you need to remember, the i saves the access paths it creates in the job in case they are needed again. Which means if you log out and log back in then run your query, it should take longer to run, then the second time you run the query it'll be faster. When running queries in a web application, you may or may not be reusing a job meaning the access paths have to be rebuilt.
If speed is important. I would:
Look into optimizing the query. I know there are better sources, but I can't find them right now.
Create a stored procedure. A stored procedure saves the access paths created.
With only 16000 rows and no WHERE or ORDER BY this thing should scream. Break the problem down to help diagnose where the bottleneck is. Go back to the IBM i, run your query in the SQL command line and then use the B, BOT or BOTTOM command to tell the database to show the last row. THAT will force the database to cough up the entire 16k result set, and give you a better idea of the raw performance on the IBM side. If that's poor, have the IBM administrators run Navigator and monitor the performance for you. It might be something unexpected, like the 'table' is really a view and the columns you are selecting might be user defined functions.
If the performance on the IBM side is OK, then look to what Cold Fusion is doing with the result set. Not being a CF programmer, I'm no help there. But generally, when I am tasked with solving multi-platform performance issues, the client side tends to consume the entire result set and then use program logic to choose what rows to display/work with. The server is MUCH faster than the client, and given the right hints, the database optimiser can make some very good decisions about how to get at those rows.

Sql Server locked tabled

I am thinking this is impossible but I wanted to make sure.
Is there a way for me to know when a table was locked and maybe for how long? I know that I can see whether a table is currently locked, but I would like to have a "history" of locks.
A "free" alternative to the RedGate tool mentioned in other response, is the MS-SQL Server Profiler (see in Tools menu from SQL Server Management Studio). This tool lets you define "traces" by specifying the type of event you wish to monitor and/or record. There's even [in SQL2008, maybe also in older versions] a default template for locking issues: TSQL_Locks.
Beware that analyzing the profiler's logs may require a bit of work/figuring out. It is possible to filter events based on a particular set of criteria (as well as filtering these at the source, i.e. excluding these from the log in the first place), but third party products such as RedGate's are likely to offer more ease-of-use, better aggregation features etc.
Edit: (following Metju's remark)
The solution suggested above, implies that one would start recording lock-related events in the profiler tool ahead of time. There is nothing, at least nothing publicly documented, in MS-SQL 2005 which would provide access to a complete historical info about the locks, "post facto". However, depending on one's needs, enough insight may sometimes be gathered from the SQL Activity Monitor (from 'Management' in the databases etc. tree on the left, in Management Studio), in particular the "Locks by object" view. This information, which is implicitly available (no need for any setup), can often be sufficient to identify the origins of dead-locks and similar issues.
Check this tool out from Red-Gate. I use it an awful lot for exactly this kind of thing. Plus it lets me check out long running queries and a host of other useful information. There are filters for Last hour, last day, forever etc...
RedGate SQL Response
Without running a monitoring tool (a roll your own, third party, or processing the output of SQL profiler), no there isn't.