I need a way to collect all queries executed in Oracle DB (Oracle Database 11g) in the last several hours regardless of how fast the queries are (this will be used to calculate sql coverage after running tests against my application that has several points where queries are executed).
I cannot use tables like V$SQL since there's no guarantee that a query will remain there long enough. It seems I could use DBA_HIST_SQLTEXT but I didn't find a way to filter out queries executed before current test run.
So my question is: what table could I use to get absolutely all queries executed in the given period of time (up to 2 hours) and what DB configuration should I adjust to reach my goal?
"I need the query itself since I need to learn which queries out of all queries coded in the application are executed during the test"
The simplest way of capturing all the queries executed would be to enable Fine Grained Audit on all your database tables. You could use the data dictionary to generate policies on every application table.
Note that even when writing to an OS file such a number of policies would generate a high impact on the database, and will increase the length of time it takes to run the tests. Therefore you should only use these policies to assess your test coverage, and disable them for other test runs.
Find out more.
In the end I went with what Justin Cave suggested and instrumented Oracle JDBC driver to collect every executed query in a Set and them dump them all into a file after running the tests.
Related
As BigQuery is a shared resource, it is possible that one gets different values running the same code on BigQuery. OK one option that I always use is to turn off caching in Query Settings, Cache preference. This way queries will not be cached. The problem with this setting is that if you refresh the browser or leave it idle, that Cache Preference box will be ticked again.
Anyhow I had a discussion with some developers that are optimizing the code. In a nutshell, they take slow running code, run it 5 times and get an average, then following optimization then run the code again 5 times to get an average value for optimized SQL. Details are not clear to me. However, my preference would be (all in BQ console)
create a user session
turn off sql caching
On BQ console paste the slow running code;
On the same session paste the optimized code
Run the codes (separated by ";")
This will ensure that any systematics like BQ busy/overloaded, slow connection etc will affect "BOTH" SQL piece equally and the systematics will be cancelled out. In my option one only need to run it once as caching is turned off as well. Running 5 times to get an average looks excessive and superfluous?
Appreciate any suggestions/feedback
Thanks
Measuring the time is one way, the other way to see if the query has been optimized is the understanding of the query plan and how slots are used effectively.
I've been with BigQuery more than 6 years, and what you describe was never used by me. In BigQuery actually what matters is reducing the costs, and that can be done iteratively rewriting the query, and using partitioning/clustering/materialized views, caching/temporary tables.
I have developed C# application that reads data from Google Big-query using .Net Client Library.
Query:
Select SUM(Salary), Count(Employee_ID) From Employee_Details
If i am using Non-Cached Query (JobConfig.UseCacheQuery=false) in Job Configuration then able to get the result in ~6 Seconds.
If i am using Cached Query (JobConfig.UseCacheQuery=true) in Job Configuration then able to get the same result in ~2 Seconds.
Which is the best way to use Google BigQuery whether Cached nor Non-Cached. (Cached Query Execution time is faster than Non-Cached once).
If there is any drop-backs are present in Cached Queries? Kindly Clarify this.
If you run a BigQuery query twice in a row, the query cache will allow the second query invocation to simply return the same results that the first query already computed, without actually running the query again. You get your results faster, and you don't get charged for it.
The query cache is a simple way to prevent customers from overspending by repeating the same query, which sometimes happens in automated environments.
Query caching is on by default, and I would recommend leaving it enabled unless you have a specific reason to disable it. One reason you might disable caching is if you are doing performance testing and want to actually run the query to see how long it takes. But those scenarios are rare.
Read more here:
https://cloud.google.com/bigquery/querying-data#querycaching
I have SQL job that runs every night which does various inserts/updates/deletes. The job contains 40 steps which mainly execute stored procedures.
It's been running fine up until a week ago when suddenly the run time went up from 2.5 hours to over 5 hours, sometimes even 8,9,10!
Could one you please give me any pointers?
First of all let me recommend you a valuable resource on Simple-Talk site. Is a detailed methodology of how to troubleshoot performance issues on SQL Server.
Does the insert you say was carried out was a huge bulk insert that could affect performance? Maybe if it was a huge load the query execution plans could be different and you need to re-tune your table structure, indexes, etc.
If the run time suddenlychanged and no changes where done in the queries or your database structure then I would ask myself several questions:
first, does the process is still taking so long or it was only one time it ran so slow? maybe now is running smoothly and the issue only arised once. Nevertheless, try to find what triggered that bad performance, it can happend again and take down your server
is the server a dedicated sql server? if not, check if some new tasks unrelated to the SQL engine had been configured, maybe a new tasks is doing some heavy I/O jobs and therefore your CRUD operations take longer
if it is a dedicated server, then check that no new job has been added and can take down your existing jobs. Check this SO link for details on jobs settled up from the SQL Agent
maybe low memory due to another process on same server?
And there is lot more to check, but before going deeper I would check that no external (non sql server related) was the reason of the delay on the process execution.
In our test bed, a number of test suites will be run in a row (unattended), producing reports for later consumption. I want to include in those reports queries which are candidates for further investigation, along with the data that justifies their inclusion in that list. We should be able to associate any query identified this way with the test suite that exposed it as a concern.
When we use SQL Server, this is relatively straight forward - a call to DBCC FREEPROCCACHE clears all of the counters before a suite begins, then at test end we run a query against sys.dm_exec_query_stats, which gives us access to the execution counts and min/max/total time(s) of each cached query plan, with hooks available to retrieve the parameterized SQL statement (we use FORCED parametrization in our mssql instances) and the query plan.
Ref: http://msdn.microsoft.com/en-us/library/ms189741%28SQL.90%29.aspx
My question: how do I implement an approximation for this when my target app has been connected to Oracle 11g? My reading thus far suggests that everything I'm after is available via the AWR, and that it should be possible to access the supporting views directly, but I haven't been able to close the circle on my own.
Why do you need to access the supporting views directly? It would seem to me that the simplest solution would be
Each test suite starts and ends by explicitly generating an AWR snapshot so it knows the starting and ending snapshot ID and so that you can generate AWR reports for each suite individually.
You run AWR reports for each test suite
You review the AWR reports looking in particular at the various Top SQL sections
It's absolutely possible to get all the information from the underlying views directly, but if you don't need to do so, that's obviously easier.
Just for sanity, I should point out that I am assuming you are licensed to use AWR. Technically, even querying the AWR views requires that you have licensed the Performance and Tuning Pack. If you want to hit the views directly rather than generating full AWR reports because of licensing concerns, you're not saving yourself any license headaches by hitting the views.
The Oracle equivalent of DBCC FREEPROCCACHE is
SQL> alter system flush shared_pool;
The closest to the SQL Server counters are V$SYSSTAT and V$SYSTEM_EVENT.
However Oracle actually tracks these at the session level too in v$SESSION_WAIT, V$SESSION_WAIT_CLASS and V$SESSION_EVENT so you don't need to reset them at the system level.
And you don't need the Diagnostic/Tuning pack licenses to access them.
They don't go down to the SQL level. That is available in V$SQL, though would not be specific to that session. You can use session level tracing to track down individual SQLs that may be causing problems
Justin's answer had the correct outline, but I needed more details about the implementation.
Each test suite starts and ends by explicitly generating an AWR snapshot so it knows the starting and ending snapshot ID and so that you can generate AWR reports for each suite individually.
You run AWR reports for each test suite
You review the AWR reports looking in particular at the various Top SQL sections
I explicitly generate the snapshots by calling dbms_workload_repository.create_snapshot, the result gets saved off for later.
select dbms_workload_repository.create_snapshot() as snap_id from dual
In order to get the report, I need the database id and the instance number. Those are easily obtained from v$database and v$instance.
select d.DBID, i.instance_number as inst_num from v$database d, v$instance i
The report is available as text (dbms_workload_repository.awr_report_text) or html (dbms_workload_repository.awr_report_html). The arguments are the same in both cases, including a options flag which will include information from the Automatic Diagnostic Database Monitor (ADDM). It wasn't immediately obvious to me that I could leverage the ADDM results, so I turn that off. The return value is a column of varchar, so the function call gets wrapped
select output from table(dbms_workload_repository.awr_report_html(1043611354,1,5539,5544,0))
This result is easily written to a file, which is assembled with the other artifacts of the test.
Documentation of these methods is available online
I have Java JDBC application running against an Oracle 10g Database. I set up a PreparedStatement to execute a query, and then call ps.executeQuery() to run it. Occasionally the query takes a long time, and I need to kill it. I have another thread access that PreparedStatement object, and call cancel() on it.
My question is, does this actually kill the query in the database? Or does it just sever it from the client, and the query is still running somewhere in the bowels of Oracle?
Thanks!
Please note that what I say below is based on observations and inferences of Oracle in use, and is not based on any deep understanding of Oracle's internals. None of it should be considered authoritative.
What ninesided said in their first paragraph is correct. However, beware the test suggested. Not all long-running oracle queries are the same. It seems that queries are evaluated over two phases, first a phase that combines up sufficient data to know how to return the rows in the right order, and second a phase that returns the rows filling in the gaps that it didn't compute in the first phase. The division of work between the two phases is also affected by the settings of the cost-based optimizer. e.g. First-rows vs All-rows.
Now, if the query is in phase 1, the cancel request seems at best to be queued up to be applied at the end of phase 1, which means that the query continues operating.
In phase 2, rows are returned in bunches, and after each bunch, the cancel command can take effect, so assuming the driver supports the command, the cancel request will result in the query being killed.
The specification for the JDBC cancel command does not seem to say what should happen if the query does not stop running, and therefore the command may wait for confirmation of the kill, or may timeout and return with the query still running.
The answer is that it's a quality-of-implementation issue. If you look at the javadoc for Statement.cancel(), it says it'll happen "if both the DBMS and driver support aborting an SQL statement".
In my experience with various versions of Oracle JDBC drivers, Statement.cancel() seems to do what you'd want. The query seems to stop executing promptly when cancelled.
It depends on the driver you are using, both the driver and the database need to support the cancellation of a statement in order for it to work as you want. Oracle does support this feature, but you'd need to know more about the specific driver you are using to know for sure.
Alternatively, you could run a simple test by looking at the database after initiating and canceling a long running query.