We use LeaderLatch to select leader on my cluster.
we use it like this:
leaderLatch.addListener(new LeaderLatchListener() {
#Override
public void isLeader() {
// create leader tasks runner
}
#Override
public void notLeader() {
// shutdown leader tasks runner
});
leaderLatch.start();
leaderLatch.await();
We also have a graceful shutdown process:
CloseableUtils.closeQuietly(leaderLatch);
now, the problem is when I shutdown a non-leader instance, the await() method throws a EOFException.
This is the code from LeaderLatch itself:
public void await() throws InterruptedException, EOFException
{
synchronized(this)
{
while ( (state.get() == State.STARTED) && !hasLeadership.get() )
{
wait();
}
}
if ( state.get() != State.STARTED )
{
throw new EOFException();
}
}
since I have closed it - the state is not STARTED but CLOSED so empty EOFException is thrown.
Is there a better way?
We use curator-recepies-4.2.0
Regards,
Ido
The contract for await() is to not return until it owns the lock. It has no way of indicating that you don't own the lock other than to throw an exception. I suggest you use the version of await that takes a timeout and returns a boolean. You can then close the lock and check the result of await(). Do this in a loop if you want.
Related
I have this flow that I am trying to test but nothing works as expected. The flow itself works well but testing seems a bit tricky.
This is my flow:
#Configuration
#RequiredArgsConstructor
public class FileInboundFlow {
private final ThreadPoolTaskExecutor threadPoolTaskExecutor;
private String filePath;
#Bean
public IntegrationFlow fileReaderFlow() {
return IntegrationFlows.from(Files.inboundAdapter(new File(this.filePath))
.filterFunction(...)
.preventDuplicates(false),
endpointConfigurer -> endpointConfigurer.poller(
Pollers.fixedDelay(500)
.taskExecutor(this.threadPoolTaskExecutor)
.maxMessagesPerPoll(15)))
.transform(new UnZipTransformer())
.enrichHeaders(this::headersEnricher)
.transform(Message.class, this::modifyMessagePayload)
.route(Map.class, this::channelsRouter)
.get();
}
private String channelsRouter(Map<String, File> payload) {
boolean isZip = payload.values()
.stream()
.anyMatch(file -> isZipFile(file));
return isZip ? ZIP_CHANNEL : XML_CHANNEL; // ZIP_CHANNEL and XML_CHANNEL are PublishSubscribeChannel
}
#Bean
public SubscribableChannel xmlChannel() {
var channel = new PublishSubscribeChannel(this.threadPoolTaskExecutor);
channel.setBeanName(XML_CHANNEL);
return channel;
}
#Bean
public SubscribableChannel zipChannel() {
var channel = new PublishSubscribeChannel(this.threadPoolTaskExecutor);
channel.setBeanName(ZIP_CHANNEL);
return channel;
}
//There is a #ServiceActivator on each channel
#ServiceActivator(inputChannel = XML_CHANNEL)
public void handleXml(Message<Map<String, File>> message) {
...
}
#ServiceActivator(inputChannel = ZIP_CHANNEL)
public void handleZip(Message<Map<String, File>> message) {
...
}
//Plus an #Transformer on the XML_CHANNEL
#Transformer(inputChannel = XML_CHANNEL, outputChannel = BUS_CHANNEL)
private List<BusData> xmlFileToIngestionMessagePayload(Map<String, File> xmlFilesByName) {
return xmlFilesByName.values()
.stream()
.map(...)
.collect(Collectors.toList());
}
}
I would like to test multiple cases, the first one is checking the message payload published on each channel after the end of fileReaderFlow.
So I defined this test classe:
#SpringBootTest
#SpringIntegrationTest
#ExtendWith(SpringExtension.class)
class FileInboundFlowTest {
#Autowired
private MockIntegrationContext mockIntegrationContext;
#TempDir
static Path localWorkDir;
#BeforeEach
void setUp() {
copyFileToTheFlowDir(); // here I copy a file to trigger the flow
}
#Test
void checkXmlChannelPayloadTest() throws InterruptedException {
Thread.sleep(1000); //waiting for the flow execution
PublishSubscribeChannel xmlChannel = this.getBean(XML_CHANNEL, PublishSubscribeChannel.class); // I extract the channel to listen to the message sent to it.
xmlChannel.subscribe(message -> {
assertThat(message.getPayload()).isInstanceOf(Map.class); // This is never executed
});
}
}
As expected that test does not work because the assertThat(message.getPayload()).isInstanceOf(Map.class); is never executed.
After reading the documentation I didn't find any hint to help me solved that issue. Any help would be appreciated! Thanks a lot
First of all that channel.setBeanName(XML_CHANNEL); does not effect the target bean. You do this on the bean creation phase and dependency injection container knows nothing about this setting: it just does not consult with it. If you really would like to dictate an XML_CHANNEL for bean name, you'd better look into the #Bean(name) attribute.
The problem in the test that you are missing the fact of async logic of the flow. That Files.inboundAdapter() works if fully different thread and emits messages outside of your test method. So, even if you could subscribe to the channel in time, before any message is emitted to it, that doesn't mean your test will work correctly: the assertThat() will be performed on a different thread. Therefore no real JUnit report for your test method context.
So, what I'd suggest to do is:
Have Files.inboundAdapter() stopped in the beginning of the test before any setup you'd like to do in the test. Or at least don't place files into that filePath, so the channel adapter doesn't emit messages.
Take the channel from the application context and if you wish subscribe or use a ChannelInterceptor.
Have an async barrier, e.g. CountDownLatch to pass to that subscriber.
Start the channel adapter or put file into the dir for scanning.
Wait for the async barrier before verifying some value or state.
I have a multi threaded application which keeps writing to ignite cache using write() method. At startup, it calls init() method which creates a cache and streamer object. Once all threads are done, flush() is called which close the streamer and write any keys which failed to insert in write() method. I have some queries.
In a write() method, what should i do with IgniteFuture? Should i wait on it until it is completed?
Is this future get complete when entry is written into cache or into streamer internal buffer?
In a flush() method, I am writing all the entries which failed in a write() method before closing streamer object. Is this a correct way?
code below:
public void initialize(final String cacheName) {
getOrCreateCache(cacheName, true); // create new cache or get existing if it already exists
this.streamer = ignite.dataStreamer(cacheName);
this.cacheName = cacheName;
}
public void write(K key, V value) {
try {
IgniteFuture<?> future = streamer.addData(key, value); // what to do with this future. Should i wait on that?
numberOfEntriesWrittentIntoCache.incrementAndGet();
} catch (IgniteInterruptedException | IgniteDataStreamerTimeoutException | CacheException | IllegalStateException e) {
failedMap.put(key, value);
}
}
public void flush() {
try {
if (streamer != null) {
if (failedMap.size() > 0) {
LOGGER.info("Writing " + failedMap.size() + " failed entries");
failedMap.forEach((k, v) -> writeToGrid(k, v));
}
}
} catch (IllegalStateException | CacheException | IgniteException e) {
LOGGER.error("Exception while writing/closing ignite");
} catch (Exception e) {
LOGGER.error("Exception while writing/closing ignite");
} finally {
failedMap.clear();
if (streamer != null) {
streamer.close();
streamer = null;
}
LOGGER.info("Number of entries written into cache are " + numberOfEntriesWrittentIntoCache.intValue());
}
}
Data streamer sends data to other nodes in batches. Future, returned from addData method is completed, when a batch with the provided entry is flushed into a cache.
So, if you wait for completion of each future, returned from addData(...) method, then you may never wait till it's completed, if autoFlushFrequency is not configured and flush() or close() is not called. And even if autoFlushFrequency is configured, then each call to write() method will wait till the batch is flushed.
I don't see anything bad in trying to write failed entries to data streamer again at the end of all processing. But I don't really know a case, when it may be useful.
The only thing, that I would change is wrapping writeToGrid(k, v) inside the forEach into its own try-catch block. Otherwise one exception will stop processing of all failed entries.
I know that there are many posts on this topic and i've read them and thought i also understood them. But I am still having a problem with aborting a QThread or rather a worker object in a QThread.
I have an GUI application and a library. The GUI can request the library to exec and abort worker objects (is connected to WorkerHandler slots). The WorkerHandler can create several WorkerObjects that all inherit from a base class. I tried to reduce the code for this example, but it's still some kind of verbose.
Gui.h
class Gui : public QMainWindow
{
Q_OBJECT
public:
Gui(QWidget *parent = 0);
~Gui();
private:
Ui::GuiClass ui;
QThread *workerHandlerThread;
WorkerHandler *workerHandler;
void connectActions();
signals:
void execWorker(WorkerParams _params);
void abortWorker(WorkerType type);
slots:
void buttonExecPressed();
void buttonAbortPressed();
}
Gui.cpp
void Gui::Gui()
{
ui.btnExecA->setProperty("type", QVariant::fromValue(WorkerType::A)); //WorkerType is just a enum, bin type to button
ui.btnExecB->setProperty("type", QVariant::fromValue(WorkerType::B));
ui.btnAbortA->setProperty("type", QVariant::fromValue(WorkerType::A));
ui.btnAbortB->setProperty("type", QVariant::fromValue(WorkerType::B));
connectActions();
workerHandlerThread = new QThread();
workerHandler = new WorkerHandler();
workerHandler->moveToThread(workerHandlerThread); // move worker execution to another thread
workerHandlerThread->start(); //start will call run and run will run the QEventLoop of QThread by calling exec
}
void Gui::~Gui()
{
workerHandlerThread->quit();
workerHandlerThread->wait();
delete workerHandlerThread;
delete workerHandler;
}
void Gui::connectActions()
{
connect(ui.btnExecA, &QPushButton::clicked, this, &Gui::buttonExecPressed);
connect(ui.btnExecB, &QPushButton::clicked, this, &Gui::buttonExecPressed);
connect(ui.btnAbortA, &QPushButton::clicked, this, &Gui::buttonAbortPressed);
connect(ui.btnAbortB, &QPushButton::clicked, this, &Gui::buttonAbortPressed);
connect(this, &Gui::execWorker, workerHandler, &WorkerHandler::execWorker);
connect(this, &Gui::abortWorker, workerHandler, &WorkerHandler::abortWorker);
}
void Gui::buttonExecPressed()
{
QPushButton* button = qobject_cast<QPushButton*>(sender());
if (button)
{
WorkerType type = button->property("type").value<WorkerType>(); //get worker type
WorkerParams params = WorkerParamsFactory::Get()->CreateParams(type); //WorkerParamsFactory cretes default parameters based on type
emit execWorker(params); //tell WorkerHandler to create a workerObject based on these parameters
}
}
void Gui::buttonAbortPressed()
{
QPushButton* button = qobject_cast<QPushButton*>(sender());
if (button)
{
WorkerType type = button->property("type").value<WorkerType>();
emit abortWorker(type); //tell WorkerHandler to abort a specific workerObject
}
}
WorkerHandler.h
class WorkerHandler : public QObject {
Q_OBJECT
public:
WorkerHandler(QObject * parent = Q_NULLPTR);
~WorkerHandler();
public slots:
void execWorker(WorkerParams _params);
void abortWorker(WorkerType type);
private:
QMap<WorkerType, WorkerObjectBase*> workerPool; //contains the workerobjects
};
WorkerHandler.cpp
void WorkerHandler::execWorker(WorkerParams _params)
{
QThread *thread = new QThread();
WorkerObjectBase *worker = WorkerObjectFactory::Get()->CreateWorker(_params); //Factory to create specific Worker Object based on given params
worker->moveToThread(thread);
connect(thread, &QThread::started, workerThread, &WorkerObjectBase::process);
connect(workerThread, &WorkerObjectBase::workerFinished, thread, &QThread::quit); //quit the QThread when worker is finished
connect(thread, &QThread::finished, thread, &QThread::deleteLater); //free resources when thread is finished
connect(thread, &QThread::finished, workerThread, &WorkerObjectBase::deleteLater); //free resources when thread is finished
workerPool.insert(_params.type, worker); //_params.type contains WorkerType
thread->start(); //will call run of qthread which will call exec
}
void WorkerHandler::abortWorker(WorkerType type)
{
WorkerObjectBase *worker = workerPool.value(type);
worker->requestAbort();
QThread *workerThread = worker->thread();
if (workerThread)
{
if (!workerThread->wait(10000)) //will always block the 10 seconds and terminate the thread. using just wait() will block forever
{
workerThread->terminate();
}
}
}
WorkerHandlerBase.h
class WorkerObjectBase : public QObject {
Q_OBJECT
public:
WorkerObjectBase(QObject * parent = Q_NULLPTR);
~WorkerObjectBase();
void requestAbort();
protected:
//some WorkerObject basic parameters
bool abortRequested();
public slots:
virtual void process();
signals:
void workerFinished();
private:
QMutex abortMutex;
bool abort = false;
};
WorkerHandlerBase.cpp
void WorkerObjectBase::requestAbort()
{
abortMutex.lock();
abort = true;
abortMutex.unlock();
}
bool WorkerObjectBase::abortRequested()
{
bool abortRequested;
abortMutex.lock();
abortRequested = abort;
abortMutex.unlock();
return abortRequested;
}
WorkerObjectA.h
class WorkerObjectA : public WorkerObjectBase {
Q_OBJECT
public:
WorkerObjectA(QObject * parent = Q_NULLPTR);
~WorkerObjectA();
protected:
//some WorkerObjectA parameters
public slots:
void process();
};
WorkerObjectA.cpp
void WorkerObjectA::process()
{
while(!abortRequested())
{
//do some stuff
}
emit workerFinished();
}
The problem is, when i use wait, it blocks the signal processing. workerFinished is not handled and QThread does not quit. But I still don't get why. When i create a new worker object, i move it to a different thread. When this thread is started, it runs its own QEventLoop as stated in QThread
5.5 Documentation:
void QThread::run()
The starting point for the thread. After calling start(), the newly
created thread calls this function. The default implementation simply
calls exec().
So even if my WorkerHandler thread is blocking because of calling wait, the QThread of the specific workerObject should still manage to get the workerFinished signal and call the quit slot. If i don't use wait at all, everything is fine. But when something unexpected happens in the worker object process method that keeps it from emitting workerFinished, i want to be able to kill the thread the hard way.
So, what am i doing wrong?
I was developing a sample application to test the timeout management in saga using NserviceBus.
I am tryin to achieve the following
When a saga started set it's timeout to 1 minute
Before the timeout happens if an update came to the nessage updates the timeout to 5 minutes
My code is like below
public class OrderSaga : Saga<OrderSagaData>,
IAmStartedByMessages<SampleMessage>,
IHandleMessages<UpdateMessage>
{
public override void ConfigureHowToFindSaga()
{
ConfigureMapping<UpdateMessage>(s => s.PurchaseOrderNumber, m => m.Update);
}
public void Handle(SampleMessage message)
{
this.Data.PurchaseOrderNumber = message.Name;
RequestTimeout(DateTime.Now.AddMinutes(1), message.Name);
}
private void Complete()
{
MarkAsComplete();
}
public override void Timeout(object state)
{
Complete();
}
#region IMessageHandler<UpdateMessage> Members
public void Handle(UpdateMessage message)
{
this.Data.PurchaseOrderNumber = message.NewValue;
RequestTimeout(DateTime.Now.AddMinutes(5), message.Update);
}
#endregion
}
}
But here the problem is the timeout is not getting updated to 5 minutes.The timeout still works for 1 minute.
Could you please let me know what is doing wrong here?
Thanks in advance,
Ajai
Saga timeouts can't be updated. They will fire no matter what you do. In your case you will receive both timeouts and given that you call Complete in your timeout handler your saga will end after one minute. You need to add some logic in that takes this into account.
Something like this might do it:
if(!updateReceived or state == ThisTimeoutWasRequestedByMyUpdateHandler)
Complete();
Hope this helps!
In a web application, the Session is only available in the current thread.
Does anyone have any tips for executing queries through NHibernate in a new asynchronous thread?
For example, how could I make something like this work:
public void Page_Load()
{
ThreadPool.QueueUserWorkItem(state => FooBarRepository.Save(new FooBar()));
}
You need to have a session context that's smart enough for non web context. But more importantly, the new thread should have it's own session.
You can use something like the following:
private ISession threadSession
{
get
{
if (HttpContext.Current != null)
{
return (ISession)HttpContext.Current.Items["THREAD_SESSION"];
}
return (ISession)AppDomain.CurrentDomain
.GetData("THREAD_SESSION" + Thread.CurrentThread.ManagedThreadId);
}
set
{
if (HttpContext.Current != null)
{
HttpContext.Current.Items["THREAD_SESSION"] = value;
}
else
{
AppDomain.CurrentDomain.SetData("THREAD_SESSION"
+Thread.CurrentThread.ManagedThreadId, value);
}
}
}
Sessions are not thread-safe. IOW you'll run into issues sooner or later if you create a session on one thread and use it from another. Create a new session on your background thread and close it before your background thread finishes.
How about:
public void Page_Load()
{
ThreadPool.QueueUserWorkItem(state => NHibernateSessionFactory.GetSession().Save(new FooBar()));
}