We are using Infinispan to control a distributed cache (replicated-cache) in an JEE application running on a Payara server (Enterprise v 5.22.0) with Java 8 (OpenJDK 64-Bit Server VM Vendor: Azul Systems, Inc. Version: 25.262-b19)
In order to have a controlled start up of the application when starting multiple instances we have created a PESSIMISTIC locking cache, called Mutex, that is used to "lock" the cluster to allow 1 instance to load the cache while the others wait. The first instance to the get the lock reads a database and loads many other caches which are all configured as OPTIMISTIC locking. These cache "puts" all happen inside the outer Mutex transaction. The OPTMISTIC caches are defined with state-transfer enabled=true so that when the instance loading the cache from the database is done and releases the Mutex lock by committing the outer transaction the caches are updated on all instances too.
When loading the OPTIMISTIC caches we sometimes use entries in CACHE1 to drive the loading of CACHE2 (we do have more meaningful names but that detail does not matter here). So having loaded CACHE1 we use CACHE1.values() to orchestrate entries into CACHE2.put().
Now to the problem...
At V9.4.20.Final (and below) the process above works. At V10.x (also V.11.0.5.Final) this does not work. We have debugged our code to find that at V10.x the entries written to CACHE1 (all caches are isolation="READ_COMMITTED") are not visible with CACHE1.values() when trying to load CACHE2. Just to confirm this same code works at V9 where CACHE1.values() does return the values as expected as it is in the same transaction and should be able to see the entries.
If at V10 we don't have the outer Mutex transaction or commit the outer Mutex transaction before trying to read CACHE1 then all works.
The question:
Has the transactionality changed to remove visibilty of entries written in a nested transaction to the process that wrote them?
I have tried Weblogic 12.2.13 suspecting that the containers transaction manager might behave differently, but no. It fails at V10 works with V9 on Weblogic.
I can provided full code reproducer in a zip (eclipse / gradle project) but here are code snippets:
The CacheServiceImpl has a method exclusivePutAndGetAll and locks with name LOCK_KEY which can be called with a boolean to control whether entries are read before or after the "Mutex" parent transaction is committed:
#Override
public <K, V> Collection<V> exclusivePutAndGetAll(String cacheName, Map<K, V> values, boolean insideMutex) throws Exception {
Collection<V> returnValues = null;
LOGGER.debug("mutex manager is " + mutexManager.getManagerHash());
LOGGER.debug("cache manager is " + cacheManager.getManagerHash());
LOGGER.info("Acquiring mutex lock before starting context");
mutexManager.startTransactionAndAcquireMutex("LOCK_KEY");
putAll(cacheName, values);
if (insideMutex) {
returnValues = getAll(cacheName); // this only works and returns values with V9 !!
}
mutexManager.commitTransaction();
LOGGER.info("Mutex lock released after starting context.");
if (!insideMutex) {
returnValues = getAll(cacheName);
}
return returnValues;
}
And here's the mutexManager's startTransactionAndAcquireMutex which begins the transaction and locks the cache called Mutex with the provided "LOCK_KEY"
#Override
public boolean startTransactionAndAcquireMutex(String mutexName) {
final TransactionManager transactionManager = mutexCache.getTransactionManager();
LOGGER.debug("Mutex cache TransactionManager is " + transactionManager.getClass());
try {
transactionManager.begin();
} catch (NotSupportedException | SystemException ex) {
throw new CacheException("Unable to start transaction for mutex " + mutexName, ex);
}
return acquireMutex(mutexName);
}
and here is the mutexManager aquiring the lock:
#Override
public boolean acquireMutex(String mutexName) {
final TransactionManager transactionManager = mutexCache.getTransactionManager();
boolean lockResult = false;
try {
if (transactionManager.getStatus() == Status.STATUS_ACTIVE) {
lockResult = mutexCache.lock(mutexName);
}
} catch (final SystemException ex) {
throw new CacheException("Unable to lock mutex " + mutexName, ex);
}
return lockResult;
}
and finally the cache configuration in XML
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:10.1 http://www.infinispan.org/schemas/infinispan-config-10.1.xsd"
xmlns="urn:infinispan:config:10.1">
<jgroups>
<stack-file name="tcp-cdl" path="cluster-jgroups.xml"/>
</jgroups>
<cache-container name="SampleCacheManager" statistics="true">
<transport stack="tcp-cdl"/>
<jmx duplicate-domains="true"/>
<replicated-cache-configuration name="replicated-cache-template" statistics="true" mode="SYNC" remote-timeout="120000">
<locking isolation="READ_COMMITTED" acquire-timeout="120000" write-skew="false" concurrency-level="150" striping="false"/>
<state-transfer enabled="true" timeout="240000" chunk-size="10000"/>
<transaction
transaction-manager-lookup="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
mode="NON_XA"
locking="OPTIMISTIC">
</transaction>
</replicated-cache-configuration>
<replicated-cache name="Mutex" configuration="replicated-cache-template">
<transaction locking="PESSIMISTIC" />
</replicated-cache>
</cache-container>
</infinispan>
Related
I need to run some jobs in a cluster, only one at a time.
Because my team uses Hazelcast, I ended up with a solution based on
Hazelcast ILock implementation. For the purpose of the question, I am going to make a generalisation about it. Let's suppose we have the following interfaces (that could be easily implemented e.g. by Hazelcast or Reddison (Redis)):
public interface MyDistributedLock {
boolean lock();
void unlock();
boolean isLockedByCurrentThread();
}
public interface MyLockDistributedFactory {
MyDistributedLock getLock(String name);
}
And lock method waiting if lock cannot be acquired:
private Mono<Void> lock(String name, Publisher<?> publisher, MyLockDistributedFactory myLockFactory) {
// important to release lock on the same thread as
// it was aquired
Scheduler scheduler = Schedulers.newSingle(name.toLowerCase());
return Mono.defer(() -> Mono.just(myLockFactory.getLock(name)))
publishOn(scheduler)
.doOnNext(MyDistributedLock::lock)
.doOnNext(lock -> LOGGER.info("Process acquired lock for resource {}", name))
.flatMapMany(lock -> Flux.from(publisher))
.publishOn(scheduler)
.doFinally(signalType -> {
MyDistributedLock lock = myLockFactory.getLock(name);
if (signalType == SignalType.CANCEL) {
// cancel ignores publishOn
scheduler.schedule(() -> {
lock.unlock();
LOGGER.info("Process released lock for resource {} due to signal type {}", name, signalType);
});
} else if (lock.isLockedByCurrentThread()) {
lock.unlock();
LOGGER.info("Process released lock for resource {} due to signal type {}", name, signalType);
}
})
.then();
}
And example of some job
private Mono<Void> someJobRunEveryOneHourOnEveryNodeInCluster() {
MyLockDistributedFactory hazelcast = ...;
return lock("some-job", Flux.just(1,2), hazelcast)
.repeatWhen(afterOneHour());
}
I wonder whether this is a good approach of using Project reactor (and correct implementation) or it should be done in a different way. Please advice.
it is a correct approach when using Reactor, because you took care of offsetting the blocking portion into a dedicated Scheduler/Thread.
But I'd say mutually exclusive code like this is not a very good fit for reactive programming in general: you lose one of the key benefits of doing more with less threads, you risk blocking other parts of the application should you forget to publishOn a dedicated thread, etc...
I'm using the StackeExchange.Redis project to interact with Redis in our .NET Core C# project.
Under heavy load, our Redis connections will begin to fail with the following exception:
StackExchange.Redis.RedisServerException: OOM command not allowed when used memory > 'maxmemory'
The problem is that we have a ridiculous amount of free memory left. We're using Elasticache, so it's easy to lookup:
We can also connect to Elasticache through a shell, and see that there is memory avaialable, and interact with it just fine.
This is the code I used as a layer over the Connection information.
public class RedisTimeConnectionManager : IRedisConnectionManager
{
// More info about the Lazy<> pattern https://stackoverflow.com/questions/28792196/how-does-connectionmultiplexer-deal-with-disconnects
// Additional information about the multiplexer: https://github.com/StackExchange/StackExchange.Redis/blob/master/docs/Basics.md
private static Lazy<ConnectionMultiplexer> RedisConnectionMultiplexer = new Lazy<ConnectionMultiplexer>(() =>
{
return ConnectionMultiplexer.Connect(ConnectionString);
});
private static string ConnectionString { get; set; }
public RedisTimeConnectionManager(string connectionString)
{
ConnectionString = connectionString;
}
public ConnectionMultiplexer GetConnectionMultiplexer()
{
return RedisConnectionMultiplexer.Value;
}
public IDatabase GetDatabaseConnection()
{
return RedisConnectionMultiplexer.Value.GetDatabase();
}
}
I then pass this Connection layer to my redis "time" manager. This is the code that is throwing the OOM error:
public class TimeRedisManager : ITimeRedisManager
{
private IRedisConnectionManager RedisConnectionManager { get; }
public TimeRedisManager(IRedisConnectionManager redisConnectionManager)
{
RedisConnectionManager = redisConnectionManager;
}
public async Task<RedisUserTimelineGetValueDto> GetValueAsync(string id)
{
string key = $"time:{id}";
HashEntry[] entries = await RedisConnectionManager.GetDatabaseConnection().HashGetAllAsync(key);
// Parse and return values...
}
}
Because Elasticache has over 7.5GB free of memory, and because I can interact with it through a shell, I'm assuming it's either the StackExchange.Redis library, or an issue with connection management in my code.
.NET CORE 2.1
StackExchange.Redis v 2.0.513
One last important thing - when this exception happens, it keeps happening. Restarting the services that interact with Redis does nothing. Only restarting the Elasticache nodes solve the problem.
Redis could take 2 times the memory required by data stored in it.
Read more here : https://redis.io/topics/admin
If you are using Redis in a very write-heavy application, while saving
an RDB file on disk or rewriting the AOF log Redis may use up to 2
times the memory normally used. The additional memory used is
proportional to the number of memory pages modified by writes during
the saving process, so it is often proportional to the number of keys
(or aggregate types items) touched during this time. Make sure to size
your memory accordingly.
So, if the data stored in Redis takes 8 Gb of space, under heavy load Redis may consume 16 Gbs. You may have to tune the memory accordingly if that's the case.
In the past I have done something like this
private static bool AlreadyRunning()
{
var processes = Process.GetProcesses();
var currentProc = Process.GetCurrentProcess();
logger.Info($"Current proccess: {currentProc.ProcessName}");
foreach (var process in processes)
{
if (currentProc.ProcessName == process.ProcessName && currentProc.Id != process.Id)
{
logger.Info($"Another instance of this process is already running: {process.Id}");
return true;
}
}
return false;
}
Which has worked well. In the new dotnet core world everything has a process name of dotnet so I can only run one dotnet app at a time! Not quite what I want :D
Is there an ideal way of doing this in dotnet? I see mutex suggested but I am not sure I understand the possible downsides or error states running on other systems than a windows machine.
.NET Core now supports global named mutex. From PR description, that added that functionality:
On systems that support thread process-shared robust recursive mutexes, they will be used
On other systems, file locks are used. File locks, unfortunately, don't have a timeout in the blocking wait call, and I didn't find any other sync object with a timed wait with the necessary properties, so polling is done for timed waits.
Also, there is a useful note in Named mutex not supported on Unix issue about mutex name, that should be used:
By default, names have session scope and sessions are more granular on Unix (each terminal gets its own session). Try adding a "Global" prefix to the name minus the quotes.
In the end I used a mutex and it seeeeeems okay.
I grabbed the code from here
What is a good pattern for using a Global Mutex in C#?
The version of core I am using does not seem to have some of the security settings stuff so I just deleted it. I am sure it will be fine. (new Mutex only takes 3 parameters)
private static void Main(string[] args)
{
LogManager.Configuration = new XmlLoggingConfiguration("nlog.config");
logger = LogManager.GetLogger("console");
logger.Info("Trying to start");
const string mutexId = #"Global\{{guid-guid-guid-guid-guid}}";
bool createdNew;
using (var mutex = new Mutex(false, mutexId, out createdNew))
{
var hasHandle = false;
try
{
try
{
hasHandle = mutex.WaitOne(5000, false);
if (!hasHandle)
{
logger.Error("Timeout waiting for exclusive access");
throw new TimeoutException("Timeout waiting for exclusive access");
}
}
catch (AbandonedMutexException)
{
hasHandle = true;
}
// Perform your work here.
PerformWorkHere();
}
finally
{
if (hasHandle)
{
mutex.ReleaseMutex();
}
}
}
}
We are currently using Cassandra as NoSQL Database and GemFire as In memory Database. We have been using the GemFire CacheWriter to insert the records in Cassandra. I would like your feedback on whether it’s a good engineering practice to use Concurrent threads in CacheWriter to insert/Update records. Your feedback on this would be appreciated.
public class GenericWriter<K, V> extends CacheWriterAdapter<K, V> implements Declarable {
private static Logger log = LoggerFactory.getLogger(GenericWriter.class);
#Autowired
private CassandraOperations cassandraOperations;
ExecutorService executor = null;
#Override
public void beforeCreate(EntryEvent<K, V> e) {
executor = Executors.newSingleThreadExecutor();
executor.submit(() -> {
if (eventOperation.equals("CREATE") || eventOperation.equalsIgnoreCase("PUTALL_CREATE")) {
try {
cassandraOperations.insert(e.getNewValue());
} catch (CassandraConnectionFailureException | CassandraWriteTimeoutException
| CassandraInternalException cassException) {
} catch (Exception ex) {
log.error("Exception in GenericCacheWriter->" + ExceptionUtils.getStackTrace(ex));
throw ex;
}
}
});
executor.shutdown();
}
#Override
public void init(Properties arg0) {
// TODO Auto-generated method stub
}
}
The CacheWriter handler is called synchronously, so the application does not continue until the handler returns. Therefore, is not recommended to execute long-running operations inside this listener. If a long-running operation is needed, consider processing the operation asynchronously through an AsyncEventListener instead.
Using an ExecutorService to delegate the execution to a different thread is possible but it is an anti-pattern, as it no longer implements the fail-fast property, and the handling of the event is no longer synchronous, so its timing would not be guaranteed relative to the application's completion of the event.
You can read more about this topic in the Geode Wiki, specifically in CacheWrite and CacheListener Best Practices.
Hope this helps.
Best regards.
Yes, it's a fine pattern but remove the Executor and partition your data such that all updates into GemFire go to one and only one node. Partition Cassandra the same way. Put a write lock around the Cassandra update. Use this only when your throughput is low.
If you need high throughput, use the AsyncEventListener and guarantee eventual consistency to your users. If you must use Executors in the AEL, use them in a way so as to throw an exception in the main thread. If the update fails after a number of tries, you write the failed entry to a different region with an expiration of a few seconds or a minute. When that expires, retry the operation. Keep doing this until the succeeds and then and only then, delete the expired entry.
You will need to track version numbers and what you are updating watching old values/ new values if order of updates is important to you or not.
I'm attempting to implement DLM using the locking mechanisms provided by the ServiceStack-Redis library and described here, but I'm finding that the API seems to present a race condition which will sometimes grant the same lock to multiple clients.
BasicRedisClientManager mgr = new BasicRedisClientManager(redisConnStr);
using(var client = mgr.GetClient())
{
client.Remove("touchcount");
client.Increment("touchcount", 0);
}
Random rng = new Random();
Action<object> simulatedDistributedClientCode = (clientId) => {
using(var redisClient = mgr.GetClient())
{
using(var mylock = redisClient.AcquireLock("mutex", TimeSpan.FromSeconds(2)))
{
long touches = redisClient.Get<long>("touchcount");
Debug.WriteLine("client{0}: I acquired the lock! (touched: {1}x)", clientId, touches);
if(touches > 0) {
Debug.WriteLine("client{0}: Oh, but I see you've already been here. I'll release it.", clientId);
return;
}
int arbitraryDurationOfExecutingCode = rng.Next(100, 2500);
Thread.Sleep(arbitraryDurationOfExecutingCode); // do some work of arbitrary duration
redisClient.Increment("touchcount", 1);
}
Debug.WriteLine("client{0}: Okay, I released my lock, your turn now.", clientId);
}
};
Action<Task> exceptionWriter = (t) => {if(t.IsFaulted) Debug.WriteLine(t.Exception.InnerExceptions.First());};
int arbitraryDelayBetweenClients = rng.Next(5, 500);
var clientWorker1 = new Task(simulatedDistributedClientCode, 1);
var clientWorker2 = new Task(simulatedDistributedClientCode, 2);
clientWorker1.Start();
Thread.Sleep(arbitraryDelayBetweenClients);
clientWorker2.Start();
Task.WaitAll(
clientWorker1.ContinueWith(exceptionWriter),
clientWorker2.ContinueWith(exceptionWriter)
);
using(var client = mgr.GetClient())
{
var finaltouch = client.Get<long>("touchcount");
Console.WriteLine("Touched a total of {0}x.", finaltouch);
}
mgr.Dispose();
When running the above code to simulate two clients attempting the same operation within short succession of one another, there are three possible outputs. The first one is the optimal case where the Mutex works properly and the clients proceed in the proper order. The second case is when the 2nd client times out waiting to acquire a lock; also an acceptable outcome. The problem, however, is that as arbitraryDurationOfExecutingCode approaches or exceeds the timeout for acquiring a lock, it is quite easy to reproduce a situation where the 2nd client is granted the lock BEFORE the 1st client releases it, producing output like this:
client1: I acquired the lock! (touched: 0x)
client2: I acquired the lock! (touched: 0x)
client1: Okay, I released my lock, your turn now.
client2: Okay, I released my lock, your turn now.
Touched a total of 2x.
My understanding of the API and its documentation is that the timeOut argument when acquiring a lock is meant to be just that -- the timeout for getting the lock. If I have to guess at a timeOut value that is high enough to always be longer than the duration of my executing code just to prevent this condition, that seems pretty error prone. Does anyone have a work around other than passing null to wait on locks forever? I definitely don't want to do that or I know I'll end up with ghost locks from crashed workers.
The answer from mythz (thanks for the prompt response!) confirms that the built-in AcquireLock method in ServiceStack.Redis doesn't draw a distinction between the lock acquisition period versus the lock expiration period. For our purposes, we have existing code that expected the distributed locking mechanism to fail quickly if the lock was taken, but allow for long-running processes within the lock scope. To accommodate these requirements, I derived this variation on the ServiceStack RedisLock that allows a distinction between the two.
// based on ServiceStack.Redis.RedisLock
// https://github.com/ServiceStack/ServiceStack.Redis/blob/master/src/ServiceStack.Redis/RedisLock.cs
internal class RedisDlmLock : IDisposable
{
public static readonly TimeSpan DefaultLockAcquisitionTimeout = TimeSpan.FromSeconds(30);
public static readonly TimeSpan DefaultLockMaxAge = TimeSpan.FromHours(2);
public const string LockPrefix = ""; // namespace lock keys if desired
private readonly IRedisClient _client; // note that the held reference to client means lock scope should always be within client scope
private readonly string _lockKey;
private string _lockValue;
/// <summary>
/// Acquires a distributed lock on the specified key.
/// </summary>
/// <param name="redisClient">The client to use to acquire the lock.</param>
/// <param name="key">The key to acquire the lock on.</param>
/// <param name="acquisitionTimeOut">The amount of time to wait while trying to acquire the lock. Defaults to <see cref="DefaultLockAcquisitionTimeout"/>.</param>
/// <param name="lockMaxAge">After this amount of time expires, the lock will be invalidated and other clients will be allowed to establish a new lock on the same key. Deafults to <see cref="DefaultLockMaxAge"/>.</param>
public RedisDlmLock(IRedisClient redisClient, string key, TimeSpan? acquisitionTimeOut = null, TimeSpan? lockMaxAge = null)
{
_client = redisClient;
_lockKey = LockPrefix + key;
ExecExtensions.RetryUntilTrue(
() =>
{
//Modified from ServiceStack.Redis.RedisLock
//This pattern is taken from the redis command for SETNX http://redis.io/commands/setnx
//Calculate a unix time for when the lock should expire
lockMaxAge = lockMaxAge ?? DefaultLockMaxAge; // hold the lock for the default amount of time if not specified.
DateTime expireTime = DateTime.UtcNow.Add(lockMaxAge.Value);
_lockValue = (expireTime.ToUnixTimeMs() + 1).ToString(CultureInfo.InvariantCulture);
//Try to set the lock, if it does not exist this will succeed and the lock is obtained
var nx = redisClient.SetEntryIfNotExists(_lockKey, _lockValue);
if (nx)
return true;
//If we've gotten here then a key for the lock is present. This could be because the lock is
//correctly acquired or it could be because a client that had acquired the lock crashed (or didn't release it properly).
//Therefore we need to get the value of the lock to see when it should expire
string existingLockValue = redisClient.Get<string>(_lockKey);
long lockExpireTime;
if (!long.TryParse(existingLockValue, out lockExpireTime))
return false;
//If the expire time is greater than the current time then we can't let the lock go yet
if (lockExpireTime > DateTime.UtcNow.ToUnixTimeMs())
return false;
//If the expire time is less than the current time then it wasn't released properly and we can attempt to
//acquire the lock. This is done by setting the lock to our timeout string AND checking to make sure
//that what is returned is the old timeout string in order to account for a possible race condition.
return redisClient.GetAndSetEntry(_lockKey, _lockValue) == existingLockValue;
},
acquisitionTimeOut ?? DefaultLockAcquisitionTimeout // loop attempting to get the lock for this amount of time.
);
}
public override string ToString()
{
return String.Format("RedisDlmLock:{0}:{1}", _lockKey, _lockValue);
}
public void Dispose()
{
try
{
// only remove the entry if it still contains OUR value
_client.Watch(_lockKey);
var currentValue = _client.Get<string>(_lockKey);
if (currentValue != _lockValue)
{
_client.UnWatch();
return;
}
using (var tx = _client.CreateTransaction())
{
tx.QueueCommand(r => r.Remove(_lockKey));
tx.Commit();
}
}
catch (Exception ex)
{
// log but don't throw
}
}
}
To simplify use as much as possible, I also expose some extension methods for IRedisClient to parallel the AcquireLock method, along these lines:
internal static class RedisClientLockExtensions
{
public static IDisposable AcquireDlmLock(this IRedisClient client, string key, TimeSpan timeOut, TimeSpan maxAge)
{
return new RedisDlmLock(client, key, timeOut, maxAge);
}
}
Your question highlights the behavior of Distributed Locking in ServiceStack.Redis, if the timeout specified is exceeded, the timed-out clients treats it as an invalid lock and will attempt to auto-recover the lock. If there was no auto-recovery behavior a crashed client would never release the lock and no further operations waiting on that lock would be allowed through.
The locking behavior for AcquireLock is encapsulated in the RedisLock class:
public IDisposable AcquireLock(string key, TimeSpan timeOut)
{
return new RedisLock(this, key, timeOut);
}
Which you can take a copy of and modify to suit the behavior you'd prefer:
using (new MyRedisLock(client, key, timeout))
{
//...
}