We have a system accessing an Oracle 11.g underlying database.
As part of the processing flow we have about sixty processes that keep polling the database looking for messages to process every 10 milliseconds.
Each processor has a corresponding row in a PROCESSORS table (PROCESSOR_ID NUMBER, MESSAGGE_ID NUMBER)
If a message to process is found for a given processor it will update the MESSAGE_ID column for that processor.
If nothing to process it will update the MESSAGE_ID with a null value. With our current implementation it will still update the MESSAGE_ID to null even if the current value of the MESSAGE_ID is null. Updating a null with a null is what is happening in 90% of the time.
The polling process is happening inside a container managed transaction (CMT).
So we have lots of fake (null updated with null) updates followed by commits.
60 processors x 3600 seconds/hour x 100updates/second = 21600000 commits/hour
In reality they are about 16 millions transactions/hour because when something is found for processing it will take more time before looking for work again.
Ideally we should change our system not to update the PROCESSORS entry if nothing changes and we will definitely do this. However the transactional behaviour of the polling process is outside our control (the container and a third party workflow product does it for us) so it will still be lots of commits regardless we updating that record or not.
So my question would be: would you think that this “update nothing” then commit will have a performance impact on the system and if yes would you think that removing the “update nothing” will improve it.
The reason for my question is that just days before the release, as part of our performance runs we found out that the system is running OK for about ten hours and then suddenly out of nowhere its starts performing very poorly.
Thanks in advance
Definitely commits are overhead to the database, A commit consumes additional resource and runs slower. In certain cases it may cause data integrity issue as well.
In fact, you should fire a commit only when your transaction is complete, when there is no transaction at all, why going for giving extra burden to the database?
Even more, you can explore much more in a very good article by Tom Kyte. The link will help you understand more clearly.
Removing the "update nothing" will significantly improve performance. Updating a column to the same value is just as expensive as updating to a different value. The simple test case below shows a significant improvement after removing the unnecessary update.
drop table processors;
create table PROCESSORS
(
PROCESSOR_ID NUMBER primary key,
MESSAGE_ID NUMBER
);
insert into processors
select level, null
from dual connect by level <= 100;
commit;
begin
for i in 1 .. 100000 loop
update processors set message_id = null where processor_id = 1;
commit;
end loop;
end;
/
begin
for i in 1 .. 100000 loop
--update processors set message_id = null where processor_id = 1;
commit;
end loop;
end;
/
On my desktop removing the update improved run time from 8.5 seconds to 1.3 seconds.
You may also want to watch out for some weird performance issues when updating the last column to null. After removing the meaningless updates it might be worth switching the column order so that the updated column is first, to avoid heap block compress waits. But I wouldn't optimize for that problem until you know it's happening.
Related
I have a SQL server with 16130000 rows. I need to delete around 20%. When I do a simple:
delete from items where jobid=12
Takes forever.
I stopped the query after 11 minutes. Selecting data is pretty fast why is delete so slow? Selecting 850000 rows takes around 30 seconds.
Is it because of table locks? And can you do anything about it? I would expect delete rows should be faster because you dont transfer data on the wire?
Best R, Thomas
Without telling us what reservation size you are using, it is hard to give feedback on whether X records in Y seconds is expected or not. I can tell you about how the system works so that you can make this determination with a bit more investigation by yourself, however. The log commit rate is limited by the reservation size you purchase. Deletes are fundamentally limited on the ability to write out log records (and replicate them to multiple machines in case your main machine dies). When you select records, you don't have to go over the network to N machines and you may not even need to go to the local disk if the records are preserved in memory, so selects are generally expected to be faster than inserts/updates/deletes because of the need to harden log for you.
You can read about the specific limits for different reservation sizes are here:
DTU Limits and vCore Limits
One common problem customers hit is to do individual operations in a loop (like a cursor or driven from the client). This implies that each statement has a single row updated and thus has to harden each log record serially because the app has to wait for the statement to return before submitting the next statement. You are not hitting that since you are running a big delete as a single statement. That could be slow for other reasons such as:
Locking - if you have other users doing operations on the table, it could block the progress of the delete statement. You can potentially see this by looking at sys.dm_exec_requests to see if your statement is blocking on other locks.
Query Plan choice. If you have to scan a lot of rows to delete a small fraction, you could be blocked on the IO to find them. Looking at the query plan shape will help here, as will set statistics time on (I suggest you change the query to do TOP 100 or similar to get a sense of whether you are doing lots of logical read IOs vs. actual logical writes). This could imply that your on-disk layout is suboptimal for this problem. The general solutions would be to either pick a better indexing strategy or to use partitioning to help you quickly drop groups of rows instead of having to delete all the rows explicitly.
Try to use batching techniques to improve performance, minimize log usage and avoid consuming database space.
declare
#batch_size int,
#del_rowcount int = 1
set #batch_size = 100
set nocount on;
while #del_rowcount > 0
begin
begin tran
delete top (#batch_size)
from dbo.LargeDeleteTest
set #del_rowcount = ##rowcount
print 'Delete row count: ' + cast(#del_rowcount as nvarchar(32))
commit tran
end
Drop any foreign keys, delete the rows and then recreate the foreign keys can speed up things also.
I have some general queries on executing Postgres function.I recently noticed that if I store the output of any arithmetic or business operation in a variable and then call it at the time of execution in the query instead of doing the operation at time of execution it saved lot of time.
But I am not aware of any practices to be followed in general to reduce the time taken and also improve performance as I am new to Postgres.
Beware of read-modify-write cycles and transaction anomalies.
It's fine to cache values locally so long as you're careful about the scope with which you cache it and with cache invalidation. Be very careful about storing values past the lifetime of the transaction you read it in.
Read-modify-write
You must also be careful not to use that cached value as an input into a calculation that you write back to the database unless you SELECT ... FOR UPDATE the value in a transaction that stays open during the write, you use a SERIALIZABLE transaction, or you use some form of optimistic concurrency control.
If you aren't careful you can get yourself in real trouble, with classics like the banking concurrency example where account id=1 transfers $100 to accounts id=2 and id=3:
session1 session2
begin; begin;
select balance
from account
where id=1; => 100
select balance
from account
where id = 1; => 100
update account
set balance = balance + 100
where id = 2; -- this is safe
update account
set balance = balance + 100
where id = 3; -- this is safe
update account
set balance = 0 -- 100 - 100 = 0
where id = 1;
update account
set balance = 0 -- 100 - 100 = 0
where id = 1;
commit;
commit;
Whoops! You just added $100 to two people's accounts but only took $100 out of id=1's. The updates against id=2 and id=3 were OK because they did an in-place modification of the balance (balance = balance + 100). The updates to id=1 were not, because they read the value, modified it client side, and wrote a new value.
This is what I mean by a read-modify-write cycle.
It would've been safe if we'd used SELECT ... FOR UPDATE when reading the balance, because the second transaction would've got stuck until the 1st committed. But it would've been better still if we'd avoided the read-copy-write cycle and just done the updates in-place.
Caching
Caching is fine - but can introduce anomalies when the underlying data is updated but your cache doesn't get flushed and refreshed.
Cache invalidation is, in general, a hard problem, but Pg has some tools that help.
In particular, listen and notify, invoked from triggers, can be used to eagerly flush data from a cache stored in memcached/redis/whatever via a helper daemon. That means you're much less likely to have to flush large chunks of cache or drop the whole cache whenever something changes.
You also need to make decisions about how out of date it's acceptable for something to be. Sometimes you just don't care if a value is 5 seconds out of date. Or half an hour. Or a week. It depends on the application, the datum in question, etc.
There's nothing particularly wrong with storing values in variables.
If you're storing values so you can write SQL in a procedural, step-by-step way instead of a set-oriented way, then you'd probably be better off not doing that. SQL is a set-oriented language; it usually performs better if you write set-oriented SQL.
The risk of storing values and using them later is that the data underlying those values might have changed since you stored them. Whether that's a real problem is application-specific, but it's usually a problem best avoided.
We're very frustratingly getting deadlocks in MySQL. It isn't because of exceeding a lock timeout as the deadlocks happen instantly when they do happen. Here's the SQL code that is executing on 2 separate threads (with 2 separate connections from the connection pool) that produces a deadlock:
UPDATE Sequences SET Counter = LAST_INSERT_ID(Counter + 1) WHERE Sequence IS NULL
Sequences table has 2 columns: Sequence and Counter
The LAST_INSERT_ID allows us to retrieve this updated counter value as per MySQL's recommendation. That works perfect for us, but we get these deadlocks! Why are we getting them and how can we avoid them??
Thanks so much for any help with this.
EDIT: this is all in a transaction (required since I'm using Hibernate) and AUTO_INCREMENT doesn't make sense here. I should've been more clear. The Sequences table holds many sequences (in our case about 100 million of them). I need to increment a counter and retrieve that value. AUTO_INCREMENT plays no role in all of this, this has nothing to do with Ids or PRIMARY KEYs.
Wrap your sql statements in a transaction. If you aren't using a transaction you will get a race condition on LAST_INSERT_ID.
But really, you should have counter fields auto_increment, so you let mysql handle this.
Your third solution is to use LOCK_TABLES, to lock the sequence table so no other process can access it concurrently. This is the probably the slowest solution unless you are using INNODB.
Deadlocks are a normal part of any transactional database, and can occur at any time. Generally, you are supposed to write your application code to handle them, as there is no surefire way to guarantee that you will never get a deadlock. That being said, there are situations that increase the likelihood of deadlocks occurring, such as the use of large transactions, and there are things you can do to mitigate their occurrence.
First thing, you should read this manual page to get a better understanding of how you can avoid them.
Second, if all you're doing is updating a counter, you should really, really, really be using an AUTO_INCREMENT column for Counter rather than relying on a "select then update" process, which as you have seen is a race condition that can produce deadlocks. Essentially, the AUTO_INCREMENT property of your table column will act as a counter for you.
Finally, I'm going to assume that you have that update statement inside a transaction, as this would produce frequent deadlocks. If you want to see it in action, try the experiment listed here. That's exactly what's happening with your code... two threads are attempting to update the same records at the same time before one of them is committed. Instant deadlock.
Your best solution is to figure out how to do it without a transaction, and AUTO_INCREMENT will let you do that.
No other SQL involved ? Seems a bit unlikely to me.
The 'where sequence is null' probably causes a full table scan, causing read locks to be acquired on every row/page/... .
This becomes a problem if (your particular engine does not use MVCC and) there were an INSERT that preceded your update within the same transaction. That INSERT would have acquired an exclusive lock on some resource (row/page/...), which will cause the acquisition of a read lock by any other thread to go waiting. So two connections can first do their insert, causing each of them to have an exclusive lock on some small portion of the table, and then they both try to do your update, requiring each of them to be able to acquire a read lock on the entire table.
I managed to do this using a MyISAM table for the sequences.
I then have a function called getNextCounter that does the following:
performs a SELECT sequence_value FROM sequences where sequence_name = 'test';
performs the update: UPDATE sequences SET sequence_value = LAST_INSERT_ID(last_retrieved_value + 1) WHERE sequence_name = 'test' and sequence_value = last retrieved value;
repeat in a loop until both queries are successful, then retrieve the last insert id.
As it is a MyISAM table it won't be part of your transaction, so the operation won't cause any deadlocks.
Let's say that I have an article on a website, and I want to track the number of views to the article. In the Articles table, there's the PK ID - int, Name - nvarchar(50), and ViewCount - int. Every time the the page is viewed, I increment the ViewCount field. I'm worried about collisions when updating the field. I can run it in a sproc with a transaction like:
CREATE PROCEDURE IncrementView
(
#ArticleID int
)
as
BEGIN TRANSACTION
UPDATE Article set ViewCount = ViewCount + 1 where ID = #ArticleID
IF ##ERROR <> 0
BEGIN
-- Rollback the transaction
ROLLBACK
-- Raise an error and return
RAISERROR ('Error Incrementing', 16, 1)
RETURN
END
COMMIT
My fear is that I'm going to end up with PageViews not being counted in this model. The other possible solution is a log type of model where I actually log views to the articles and use a combination of a function and view to grab data about number of views to an article.
Probably a better model is to cache the number of views hourly in the app somewhere, and then update them in a batch-style process.
-- Edit:
To to elaborate more, a simple model for you may be:
Each page load, for the given page, increment a static hashmap. Also on each load, check if enough time has elapsed since 'Last Update', and if so, perform an update.
Be tricky, and put the base value in the asp.net cache (http://msdn.microsoft.com/en-us/library/aa478965.aspx) and, when it times out, [implement the cache removal handler as described in the link] do the update. Set the timeout for an hour.
In both models, you'll have the static map of pages to counts; you'll update this each view, and you'll also use this - and the cached db amount - to get the current 'live' count.
The database should be able to handle a single digit increment atomically. Queries on the queue should be handled in order in the case where there might be a conflict. Your bigger issue, if there is enough volume will be handling all of the writes to the same row. Each write will block the reads and writes behind it. If you are worried, I would create a simple program that calls SQL updates in a row and run it with a few hundred concurrent threads (increase threads until your hardware is saturated). Make sure the attempts = the final result.
Finding a mechanism to cache and/or perform batch updates as silky suggests sounds like a winner.
Jacob
You don't need to worry about concurrency within a single update statement in SQL Server.
But if you are worried about 2 users hitting a table in the same tenth of a second, keep in mind that there are 864,000 10th of a seconds in a day. Doesn't sound like something that is going to be an issue for a page that serves up articles.
Have no fear!
This update is a single (atomic) transaction - you cannot get 'collisions'. Even if 5,000,000 calls to IncrementView all hit the database at the exact same moment, they will each be processed in a serial, queue like fashion - thats what you are using a database engine for - consistency. Each call will gain an exclusive update lock on the row (at least), so no subsequent queries can update the row until the current one has committed.
You don't even need to use BEGIN TRAN...COMMIT. If the update fails, there is nothing to rollback anyway.
I don't see the need for any app caching - there's no reason why this update would take a long time adn therefore should have no impact on the performance of your app.
[Assuming it's relatively well designed!]
I have a single process that queries a table for records where PROCESS_IND = 'N', does some processing, and then updates the PROCESS_IND to 'Y'.
I'd like to allow for multiple instances of this process to run, but don't know what the best practices are for avoiding concurrency problems.
Where should I start?
The pattern I'd use is as follows:
Create columns "lockedby" and "locktime" which are a thread/process/machine ID and timestamp respectively (you'll need the machine ID when you split the processing between several machines)
Each task would do a query such as:
UPDATE taskstable SET lockedby=(my id), locktime=now() WHERE lockedby IS NULL ORDER BY ID LIMIT 10
Where 10 is the "batch size".
Then each task does a SELECT to find out which rows it has "locked" for processing, and processes those
After each row is complete, you set lockedby and locktime back to NULL
All this is done in a loop for as many batches as exist.
A cron job or scheduled task, periodically resets the "lockedby" of any row whose locktime is too long ago, as they were presumably done by a task which has hung or crashed. Someone else will then pick them up
The LIMIT 10 is MySQL specific but other databases have equivalents. The ORDER BY is import to avoid the query being nondeterministic.
Although I understand the intention I would disagree on going to row level locking immediately. This will reduce your response time and may actually make your situation worse. If after testing you are seeing concurrency issues with APL you should do an iterative move to “datapage” locking first!
To really answer this question properly more information would be required about the table structure and the indexes involved, but to explain further.
DOL, datarow locking uses a lot more locks than allpage/page level locking. The overhead in managing all the locks and hence the decrease of available memory due to requests for more lock structures within the cache will decrease performance and counter any gains you may have by moving to a more concurrent approach.
Test your approach without the move first on APL (all page locking ‘default’) then if issues are seen move to DOL (datapage first then datarow). Keep in mind when you switch a table to DOL all responses on that table become slightly worse, the table uses more space and the table becomes more prone to fragmentation which requires regular maintenance.
So in short don’t move to datarows straight off try your concurrency approach first then if there are issues use datapage locking first then last resort datarows.
You should enable row level locking on the table with:
CREATE TABLE mytable (...) LOCK DATAROWS
Then you:
Begin the transaction
Select your row with FOR UPDATE option (which will lock it)
Do whatever you want.
No other process can do anything to this row until the transaction ends.
P. S. Some mention overhead problems that can result from using LOCK DATAROWS.
Yes, there is overhead, though i'd hardly call it a problem for a table like this.
But if you switch to DATAPAGES then you may lock only one row per PAGE (2k by default), and processes whose rows reside in one page will not be able to run concurrently.
If we are talking of table with dozen of rows being locked at once, there hardly will be any noticeable performance drop.
Process concurrency is of much more importance for design like that.
The most obvious way is locking, if your database doesn't have locks, you could implement it yourself by adding a "Locked" field.
Some of the ways to simplify the concurrency is to randomize the access to unprocessed items, so instead of competition on the first item, they distribute the access randomly.
Convert the procedure to a single SQL statement and process multiple rows as a single batch. This is how databases are supposed to work.