I am new in postgresql and I try to understand explain (buffers, analyse) instruction. I have a query and I execute it using explain (buffers, analyse).
The first time i execute it the performance is worse than the second time. Also, the first time i get a 'read' parameter next to 'hit' while the second time the 'read' does not exist.
Can somebody help me understand?
first time you select, pages get warm - they are loaded to cache, once they are in RAM - all next selects will be faster (RAM speed is higher).
Accordingly buffers show read, when pages are not in cache, cos postgres reads them, and no read when they are warm, so cache is hit...
Update with docs:
BUFFERS
Include information on buffer usage. Specifically, include the
number of shared blocks hit, read, dirtied, and written, the number of
local blocks hit, read, dirtied, and written, and the number of temp
blocks read and written. A hit means that a read was avoided because
the block was found already in cache when needed. Shared blocks
contain data from regular tables and indexes; local blocks contain
data from temporary tables and indexes; while temp blocks contain
short-term working data used in sorts, hashes, Materialize plan nodes,
and similar cases. The number of blocks dirtied indicates the number
of previously unmodified blocks that were changed by this query; while
the number of blocks written indicates the number of
previously-dirtied blocks evicted from cache by this backend during
query processing. The number of blocks shown for an upper-level node
includes those used by all its child nodes. In text format, only
non-zero values are printed. This parameter may only be used when
ANALYZE is also enabled. It defaults to FALSE.
And surprisingly not much about buffers here.
Related
I'm learning indexing in PostgreSQL now. I started trying to create my index and analyzing how it will affect execution time. I created some tables with such columns:
also, I filled them with data. After that I created my custom index:
create index events_organizer_id_index on events(organizer_ID);
and executed this command (events table contains 148 rows):
explain analyse select * from events where events.organizer_ID = 4;
I was surprised that the search was executed without my index and I got this result:
As far as I know, if my index was used in search there would be the text like "Index scan on events".
So, can someone explain or give references to sites, please, how to use indexes effectively and where should I use them to see differences?
From "Rows removed by filter: 125" I see there are too few rows in the events table. Just add couple of thousands rows and give it another go
from the docs
Use real data for experimentation. Using test data for setting up indexes will tell you what indexes you need for the test data, but that is all.
It is especially fatal to use very small test data sets. While
selecting 1000 out of 100000 rows could be a candidate for an index,
selecting 1 out of 100 rows will hardly be, because the 100 rows probably fit within a single disk page, and there is no plan that can
beat sequentially fetching 1 disk page.
In most cases, when database using an index it gets only address where the row is located. It contains data block_id and the offset because there might be many rows in one block of 4 or 8 Kb.
So, the database first searches index for the block adress, then it looks for the block on disk, reads it and parses the line you need.
When there are too few rows they fit into one on in couple of data blocks which makes it easier and quicker for DB to read whole table without using index at all.
See it the following way:
The database decides which way is faster to find your tuple (=record) with organizer_id 4. There are two ways:
a) Read the index and then skip to the block which contains the data.
b) Read the heap and find the record there.
The information in your screenshot show 126 records (125 skipped + your record) with a length ("width") of 62 bytes. Including overhead these data fit into two database blocks of 8 KB. As a rotating disk or SSD reads a series of blocks anyway - they read always more blocks into the buffer - it's one read operation for these two blocks.
So the database decides that it is pointless to read first the index to find the correct record (of in our case two blocks) and then read the data from the heap with the information from the index. That would be two read operations. Even with modern technology newer than rotating disks this needs more time than just scanning the two blocks. That's why the database doesn't use the index.
Indexes on such small tables aren't good for searching. Nevertheless unique indexes avoid double entries.
I want to set up a Postgres server on AWS, the biggest table will be 10GB - do I have to select 10GB of memory for this instance?
What happens when my query result is larger than 10GB?
Nothing will happen, the entire result set is not loaded into memory. The maximum available memory will be used and re-used as needed while the result is prepared and will spill over to disk as needed.
See PostgreSQL resource documentation for more info.
Specifically, look at work_mem:
work_mem (integer)
Specifies the amount of memory to be used by internal sort operations and hash tables before writing to temporary disk files.
As long as you don't run out of working memory on a single operation or set of parallel operations you are fine.
Edit: The above was an answer to the question What happens when you query a 10GB table without 10GB of memory on the server/instance?
Here is an updated answer to the updated question:
Only server side resources are used to produce the result set
Assuming JDBC drivers are used, by default, the entire result set is sent to your local computer which could cause out of memory errors
This behavior can be changed by altering the fetch size through the use of a cursor.
Reference to this behavior here
Getting results based on a cursor
On the server side, with a simple query like yours it just keeps a "cursor" which points to where it's at, as it's spooling the results to you, and uses very little memory. Now if there were some "sorts" in there or what not, that didn't have indexes it could use, that might use up lots of memory, not sure there. On the client side the postgres JDBC client by default loads the "entire results" into memory before passing them back to you (overcomeable by specifying a fetch count).
With more complex queries (for example give me all 100M rows, but order them by "X" where X is not indexed) I don't know, but probably internally it creates a temp table (so it won't run out of RAM) which, treated as a normal table, uses disk backing. If there's a matching index then it can just traverse that, using a pointer, still uses little RAM.
There was a query in production which was running for several hours(5-6) hours. I looked into its execution plan, and found that it was ignoring a parallel hint on a huge table. Reason - it was using TABLE ACCESS BY INDEX ROWID. So after I added a /*+ full(huge_table) */ hint before the parallel(huge_table) hint, the query started running in parallel, and it finished in less than 3 minutes. What I could not fathom was the reason for this HUGE difference in performance.
The following are the advantages of parallel FTS I can think of:
Parallel operations are inherently fast if you have more idle CPUs.
Parallel operations in 10g are direct I/O which bypass
buffer cache which means there is no risk of "buffer busy waits" or
any other contention for buffer cache.
Sure there are the above advantages but then again the following disadvantages are still there:
Parallel operations still have to do I/O, and this I/O would be more than what we have for TABLE ACCESS BY INDEX ROWID as the entire table is scanned and is costlier(all physical reads)
Parallel operations are not very scalable which means if there aren't enough free resources, it is going to be slow
With the above knowledge at hand, I see only one reason that could have caused the poor performance for the query when it used ACCESS BY INDEX ROWID - some sort of contention like "busy buffer waits". But it doesn't show up on the AWR top 5 wait events. The top two events were "db file sequential read" and "db file scattered read". Is there something else that I have missed to take into consideration? Please enlighten me.
First, without knowing anything about your data volumes, statistics, the selectivity of your predicates, etc. I would guess that the major benefit you're seeing is from doing a table scan rather than trying to use an index. Indexes are not necessarily fast and table scans are not necessarily slow. If you are using a rowid from an index to access a row, Oracle is limited to doing single block reads (sequential reads in Oracle terms) and that it's going to have to read the same block many times if the block has many rows of interest. A full table scan, on the other hand, can do nice, efficient multiblock reads (scattered reads in Oracle terms). Sure, an individual single block read is going to be more efficient than a single multiblock read but the multiblock read is much more efficient per byte read. Additionally, if you're using an index, you've potentially got to read a number of blocks from the index periodically to find out the next rowid to read from the table.
You don't actually need to read all that much data from the table before a table scan is more efficient than an index. Depending on a host of other factors, the tipping point is probably in the 10-20% range (that's a very, very rough guess). Imagine that you had to get a bunch of names from the phone book and that the phone book had an index that included the information you're filtering on and the page that the entry is on. You could use an index to find the name of a single person you want to look at, flip to the indicated page, record the information, flip back to the index, find the next name, flip back, etc. Or you could simply start at the first name, scan until you find a name of interest, record the information, and continue the scan. It doesn't take too long before you're better off ignoring the index and just reading from the table.
Adding parallelism doesn't reduce the amount of work your query does (in fact, adding in parallel query coordination means that you're doing more work). It's just that you're doing that work over a shorter period of elapsed time by using more of the server's available resources. If you're running the query with 6 parallel slaves, that could certainly allow the query to run 5 times faster overall (parallel query obviously scales a bit less than linearly because of overheads). If that's the case, you'd expect that doing a table scan made the query 20 times faster and adding parallelism added another factor of 5 to get your 100x improvement.
Consider Table:
Table Name:ORDER
Columns: (ID (PK), ORDER_NUM, ORDER_STATUS, etc...)
Index(ORDER_IDX) exists on (ORDER_NUM, ORDER_STATUS) together.
There are various FKs too, on which Indexes exist as well.
There are about 2 million rows in the table.
Consider SQL Query:
DELETE from ORDER where ORDER_NUM=234234;
For a particular ORDER_NUM value, the DELETE Query runs very slow first time (almost 5 seconds to delete 200 rows).
But if I rollback and run DELETE Query again for same ORDER_NUM, the DELETE QUERY now runs in 200 milliseconds.
Therefore, for ANY new ORDER_NUM supplied to this query - the query runs very slow.
What can I do to fasten the query first time itself? Do I have to rebuild indexes? Or anything else?
I am testing this from a Oracle SQL Client Tool (like TOAD/SQL-Developer) - after seeing this slow behavior within the web application where it is actually used.
EDIT>>>
Results of SET AUTOTRACE ON
FIRST TIME when QUERY is RUN
3 user calls
0 physical read total multi block requests
4915200 physical read total bytes
4915200 cell physical IO interconnect bytes
0 commit cleanout failures: block lost
0 IMU commits
1 IMU Flushes
0 IMU contention
0 IMU bind flushes
0 IMU mbu flush
SECOND TIME When Query is RUN
3 user calls
0 physical read total multi block requests
0 physical read total bytes
0 cell physical IO interconnect bytes
0 commit cleanout failures: block lost
0 IMU commits
1 IMU Flushes
0 IMU contention
0 IMU bind flushes
0 IMU mbu flush
The EXPLAIN Plans - in both FIRST and SECOND RUN is exactly same - shown below:
ID OPERATION NAME ROWS Bytes Cost(%CPU) Time<br>
=======================================================================================
0 DELETE Statement 49 2891 41 (0) 00:00:01
1 DELETE ORDER
2 INDEX RANGE SCAN ORDER_IDX 49 2891 3 (0) 00:00:01
You can see Very High Physical Reads, during the First Time.
Can I do anything at all to help with this situation?
The key to understand your problem is to understand how statements are executed. DELETE is a relatively expensive operation and often leads to performance problems. So here is how Oracle executes a DML statement:
The first step in executing DML is to find the required blocks in the database buffer cache (if they are already there) or copy them into the buffer cache from the datafiles (slow). In addition to that, an empty block of an undo segment is also copied into the buffer cache.
Then, locks are placed on the affected rows and indices.
After that, redo is generated: Change vectors describing all the changes done to the data block and undo block are generated. For a DELETE, the change vector to be written to the undo block is the entire row.
Then, the DELETE is carried out. The whole row is copied from the data block to the undo block and the row in the data block is deleted. DELETE generates much more undo data than an INSERT for example, because the contents of the whole row are copied (so other sessions can read the original data or the deletion can be rolled back).
Your query almost certainly runs faster the second time because all the relevant blocks are already in the database buffer cache. Of course, the more blocks can be held in the database buffer cache, the less I/O is needed. Make sure your SGA is sized appropriately.
So for your problem, we have to look at the following points:
Most importantly, does your query use your index? Is the index VALID? Run the EXPLAIN PLAN for your DELETE query and check if the index for ORDER_NUM is used and how your data is accessed.
Are there any CONSTRAINTS on the table? If there are CONSTRAINTS with "ON DELETE CASCADE", there might be other tables affected by your DELETE.
So for your problem, looking at the Execution Plan (EXPLAIN PLAN) might be your best bet.
Aside from the issue of buffer caching, one way of improving performance would be to promote physical clustering of the records that have the same ORDER_NUM. This would make sense only if you most commonly select groups of records by the ORDER_NUM, but could be achieved by creating the table in a hash cluster (along with any child tables that also contain the ORDER_NUM).
The benefit would be that a query with an ORDER_NUM predicate would access fewer blocks of the table segment, so even if physical i/o is required you would need less of it. Furthermore, you would be making more efficient use of the buffer cache by ensuring that each cached block contains a higher proportion of rows that you're interested in, and you'd be caching fewer blocks.
"You can see Very High Physical Reads, during the First Time. Can i do anything at all help with this situation?"
Your query has to locate the records before it can zap them. If the records are in memory that's fast, if they're on disk that will be orders of magnitude slower. Find out more. So that's what is happening. The first time you read the rows physically and that's slow, but now the rows are in the DB buffer cache so subsequent reads are faster.
What can you do to improve the performance of the first read? Well that depends on where the time goes. Despite several people asking you have not provided the explain plans. Without knowing the access paths there's not much concrete advice we can give you. So here's a couple of pointers.
physical read total multi block requests is zero so all the physical read total bytes represents single block i.e. index reads. That this takes a long time suggests the index might be a poor one. One thing to check is the clustering factor of the index: if the clustering factor is close to the number of blocks index range scans will be pretty fast. Probably the index your delete is using has a poor clustering factor. That's why you need the explain plan: so you can investigate the quality of the index. Read this article by Tom Kyte to find out more about index clustering.
If your query has a poor clustering factor there is a limit to how much you can tune it. You can't polish the proverbial. Deletion requires the retrieval of the whole row, so you have to read the table. So perhaps the simplest option is just to speed up the table read:
alter session enable parallel dml;
delete /*+ no_index (orders)
parallel */
from orders
where order_num=234234;
You're using 11g so you don't need to specify the object in the PARALLEL hint. Note you do need Enterprise Edition for parallel operations.
This may be caused by table/index fragmentation, and as you are accessing data through an index, more likely index.
For table level, you would need both of the following steps, for index only (2):
(1) Deal with the table fragmentation: alter table "ORDER" move
(2) Rebuild indexes: alter index "<YourIndex>" rebuild
If you are doing a lot of deletes & inserts, or updates that cause row migrations, this could apply to you.
When I run a certain stored procedure for the first time it takes about 2 minutes to finish. When I run it for the second time it finished in about 15 seconds. I'm assuming that this is because everything is cached after the first run. Is it possible for me to "warm the cache" before I run this procedure for the first time? Is the cached information only used when I call the same stored procedure with the same parameters again or will it be used if I call the same stored procedure with different params?
When you peform your query, the data is read into memory in blocks. These blocks remain in memory but they get "aged". This means the blocks are tagged with the last access and when Sql Server requires another block for a new query and the memory cache is full, the least recently used block (the oldest) is kicked out of memory. (In most cases - full tables scan blocks are instantly aged to prevent full table scans overrunning memory and choking the server).
What is happening here is that the data blocks in memory from the first query haven't been kicked out of memory yet so can be used for your second query, meaning disk access is avoided and performance is improved.
So what your question is really asking is "can I get the data blocks I need into memory without reading them into memory (actually doing a query)?". The answer is no, unless you want to cache the entire tables and have them reside in memory permanently which, from the query time (and thus data size) you are describing, probably isn't a good idea.
Your best bet for performance improvement is looking at your query execution plans and seeing whether changing your indexes might give a better result. There are two major areas that can improve performance here:
creating an index where the query could use one to avoid inefficient queries and full table scans
adding more columns to an index to avoid a second disk read. For example, you have a query that returns columns A, and B with a where clause on A and C and you have an index on column A. Your query will use the index for column A requiring one disk read but then require a second disk hit to get columns B and C. If the index had all columns A, B and C in it the second disk hit to get the data can be avoided.
I don't think that generating the execution plan will cost more that 1 second.
I believe that the difference between first and second run is caused by caching the data in memory.
The data in the cache can be reused by any further query (stored procedure or simple select).
You can 'warm' the cache by reading the data through any select that reads the same data. But that will even cost about 90 seconds as well.
You can check the execution plan to find out which tables and indexes your query uses. You can then execute some SQL to get the data into the cache, depending on what you see.
If you see a clustered index seek, you can simply do SELECT * FROM my_big_table to force all the table's data pages into the cache.
If you see a non-clustered index seek, you could try SELECT first_column_in_index FROM my_big_table.
To force a load of a specific index, you can also use the WITH(INDEX(index)) table hint in your cache warmup queries.
SQL server cache data read from disc.
Consecutive reads will do less IO.
This is of great help since disk IO is usually the bottleneck.
More at:
http://blog.sqlauthority.com/2014/03/18/sql-server-performance-do-it-yourself-caching-with-memcached-vs-automated-caching-with-safepeak/
The execution plan (the cached info for your procedure) is reused every time, even with different parameters. It is one of the benefits of using stored procs.
The very first time a stored procedure is executed, SQL Server generates an execution plan and puts it in the procedure cache.
Certain changes to the database can trigger an automatic update of the execution plan (and you can also explicitly demand a recompile).
Execution plans are dropped from the procedure cache based an their "age". (from MSDN: Objects infrequently referenced are soon eligible for deallocation, but are not actually deallocated unless memory is required for other objects.)
I don't think there is any way to "warm the cache", except to perform the stored proc once. This will guarantee that there is an execution plan in the cache and any subsequent calls will reuse it.
more detailed information is available in the MSDN documentation: http://msdn.microsoft.com/en-us/library/ms181055(SQL.90).aspx