That warning is spamming the logs, it happens every few seconds. can someone tells me what does it mean, how is it affecting the raven server and how to fix it ?
Message
Concurrent merge failed
Exception
Lucene.Net.Index.MergePolicy+MergeException: Exception of type
'Lucene.Net.Index.MergePolicy+MergeException' was thrown. --->
Lucene.Net.Index.CorruptIndexException: docs out of order (0 <= 0 )
at Lucene.Net.Index.IndexWriter.HandleMergeException(Exception t,
OneMerge merge) at Lucene.Net.Index.IndexWriter.Merge(OneMerge
merge) at
Lucene.Net.Index.ConcurrentMergeScheduler.MergeThread.Run() --- End
of inner exception stack trace --- at
Lucene.Net.Index.ConcurrentMergeScheduler.HandleMergeException(Exception
exc) at
Raven.Database.Indexing.ErrorLoggingConcurrentMergeScheduler.HandleMergeException(Exception
exc) in
C:\Builds\RavenDB-Stable-3.5\Raven.Database\Indexing\ErrorLoggingConcurrentMergeScheduler.cs:line
15
Logged
3 hours ago (01/09/18, 11:57am)
Level
Warn
Logger
Raven.Database.Indexing.ErrorLoggingConcurrentMergeScheduler
Related
When I am running buildindex in my Websphere application, I have this error in buildindex log:
[2021/05/10 15:41:57:590 GMT] I Data import pre-processing completed in 0.389 seconds for table TI_CAT_EXTENDED_41060.
[2021/05/10 15:41:57:591 GMT] I /opt/IBM/WebSphere/CommerceServer80/instances/auth/search/pre-processConfig/MC_41060/DB2/wc-dataimport-preprocess-catentry-metainf.xml
[2021/05/10 15:41:57:591 GMT] I
Table name: TI_X_CATENT_META_INF_410600
Fetch size: 500
Batch size: 500
[2021/05/10 15:41:58:048 GMT] I Error for batch element #415: DB2 SQL Error: SQLCODE=-302, SQLSTATE=22001, SQLERRMC=null, DRIVER=4.19.77
[2021/05/10 15:41:58:048 GMT] I SQL: SELECT CATENTRY_ID, TITLE, TITLE_KEYWORDS, SHORT_DESC, SHORT_DESC_KEYWORDS, LONG_DESC, LONG_DESC_KEYWORDS, LOCALE FROM X_CATENT_META_INF WHERE STORE_ID = 41006
[2021/05/10 15:41:58:087 GMT] I
The program exiting with exit code: 1.
Data import pre-processing was unsuccessful. An unrecoverable error has occurred.
[2021/05/10 15:41:58:091 GMT] E com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain:handleExecutionException Exception message: CWFDIH0002: An SQL exception was caught. The following error occurred: [jcc][t4][102][10040][4.19.77] Batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null., stack trace: com.ibm.commerce.foundation.dataimport.exception.DataImportSystemException: CWFDIH0002: An SQL exception was caught. The following error occurred: [jcc][t4][102][10040][4.19.77] Batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null.
at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:1515)
at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.execute(DataImportPreProcessorMain.java:1331)
at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.main(DataImportPreProcessorMain.java:534)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:280)
Caused by: com.ibm.db2.jcc.am.BatchUpdateException: [jcc][t4][102][10040][4.19.77] Batch failure. The batch was submitted, but at least one exception occurred on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements. ERRORCODE=-4229, SQLSTATE=null
at com.ibm.db2.jcc.am.b4.a(b4.java:475)
at com.ibm.db2.jcc.am.Agent.endBatchedReadChain(Agent.java:414)
at com.ibm.db2.jcc.am.ki.a(ki.java:5342)
at com.ibm.db2.jcc.am.ki.c(ki.java:4929)
at com.ibm.db2.jcc.am.ki.executeBatch(ki.java:3045)
at com.ibm.commerce.foundation.dataimport.preprocess.AbstractDataPreProcessor.populateTable(AbstractDataPreProcessor.java:373)
at com.ibm.commerce.foundation.dataimport.preprocess.StaticAttributeDataPreProcessor.process(StaticAttributeDataPreProcessor.java:461)
at com.ibm.commerce.foundation.dataimport.preprocess.DataImportPreProcessorMain.processDataConfig(DataImportPreProcessorMain.java:1482)
... 7 more
The exception seems to be clear, but I can't identify what is the element #415 in batch. Even the log doesn't helps, because it doesn't point to another more detailed log. Do you have any suggestion for find it?
Thanks to the comment of user #mao, I have followed this link
The failing table first must be identified. Enable more detailed tracing for di-preprocess:
Navigate to :
WC_installdir/instances/instance_name/xml/config/dataimport
and open the logging.properties file. Find all instances of INFO and
change it to FINEST. Optionally increase the size of the log file and
the number of historical log files while editing this file.
Thanks to this suggestion, I had re-run the buildindex process, and found that solr was wrongly grouping fields from original table, thus generating a too long field for the destination, and generating the error.
We are currently having a few issues with our program with Redis that happen and occur randomly. We do not understand what is going on.
The code is open source as well if anyone wants to tag in and help on these two repos:
https://github.com/pythonology/BeatTogether.Core
https://github.com/pythonology/BeatTogether.MasterServer/tree/feature/dedicated-servers (using the branch at the moment for dev purposes)
2020-12-10 03:22:17.829 +00:00 [ERR] An error occurred while handling message (MessageType='BroadcastServerStatusRequest').
StackExchange.Redis.RedisServerException: ERR Error running script (call to f_dce2ce9e149bc775618a47cd07eca24d80a33664): #user_script:4: #user_script: 4: -READONLY You can't write against a read only replica.
at BeatTogether.MasterServer.Data.Implementations.Repositories.ServerRepository.AddServer(Server server) in /app/BeatTogether.MasterServer.Data/Implementations/Repositories/ServerRepository.cs:line 80
at BeatTogether.MasterServer.Kernel.Implementations.UserService.BroadcastServerStatus(MasterServerSession session, BroadcastServerStatusRequest request) in /app/BeatTogether.MasterServer.Kernel/Implementations/UserService.cs:line 156
at BeatTogether.Core.Messaging.Implementations.BaseMessageHandler.<>c__DisplayClass6_0`2.<<Register>b__0>d.MoveNext()
--- End of stack trace from previous location --
and
2020-12-10 06:20:23.198 +00:00 [ERR] An error occurred while handling message (MessageType='BroadcastServerStatusRequest').
StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: EVAL, mc: 1/1/0, mgr: 10 of 10 available, clientName: master-server, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=1024,Max=32767), v: 2.2.4.27433 at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource`1 source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2760
--- End of stack trace from previous location ---
at BeatTogether.MasterServer.Data.Implementations.Repositories.ServerRepository.AddServer(Server server) in /app/BeatTogether.MasterServer.Data/Implementations/Repositories/ServerRepository.cs:line 80
at BeatTogether.MasterServer.Kernel.Implementations.UserService.BroadcastServerStatus(MasterServerSession session, BroadcastServerStatusRequest request) in /app/BeatTogether.MasterServer.Kernel/Implementations/UserService.cs:line 156
at BeatTogether.Core.Messaging.Implementations.BaseMessageHandler.<>c__DisplayClass6_0`2.<<Register>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at BeatTogether.Core.Messaging.Implementations.BaseMessageSource.<>c__DisplayClass15_0.<<Signal>b__0>d.MoveNext()
Our redis instance doesn't have slaves, only a master and its one instance on the same machine.
Any help is greatly appreciated. Thanks.
We have this EF 6 code,
var existingProduct = _productContext.Product.FirstOrDefault(
s => s.Name.ToUpper().Trim() == productDto.Name.ToUpper().Trim()
&& s.IDReference == productDto.IDReference);
if (existingProduct != null)
{
return existingProduct;
}
_productContext.Product.Add(productDto);
_productContext.SaveChanges();
return productDto;
It works sometimes and doesn't other times, it times out and we get this error,
An error occurred while updating the entries. See the inner exception
for details. An error occurred while updating the entries. See the
inner exception for details. Execution Timeout Expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding. The statement has been terminated. The wait operation
timed out
Am I doing anything wrong ? Is it locking anything ?
It shows that its blocking something as below,
https://prnt.sc/re7h0w
I have an endpoint that works as a distributor with three other worker endpoints.
The handling endpoint of the received message opens a transaction and tries to import some xml data into an sql db. If some exception is thrown during this process, the exception is caught, the transaction is rolled back and the xml data is written to an error folder.
Simplified, it looks like that:
public void Handle(doSomethingCmd message)
{
System.Data.SqlClient.BeginTransaction();
try
{
//... some xml data import
throw new Exception();
//Commit if succeded
}
catch (Exception exception)
{
System.Data.IDbTransaction.Rollback();
//...Write file to error folder
}
}
In the first place, no retry happens after the transaction rollback. But when the message is sent again, all the worker endpoints (only the workers) get an exception (Cannot enlist transaction, failed to send msg to control queue --> see stacktrace below) and nservicebus does retry the message (this leads to the case, that the file appears several times in the error folder)
It looks like distributed transaction is in an invalid state. I could just handing over the exception (re-throw the exception), so nservicebus handles the rollback for me, but in that case the file is written to the error folder several times as well (due to retry mechanism)
Failed raising finished message processing event.|NServiceBus.Unicast.Queuing.FailedToSendMessageException: Failed to send message to address: someEndpoint.distributor.control#SRVPS01 ---> System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 49
--- End of inner exception stack trace ---
at NServiceBus.Transports.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 88
at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, Address address) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 75
at NServiceBus.Distributor.MSMQ.ReadyMessages.ReadyMessageSender.SendReadyMessage(String sessionId, Int32 capacityAvailable, Boolean isStarting) in c:\BuildAgent\work\c3100604bbd3ca20\src\NServiceBus.Distributor.MSMQ\ReadyMessages\ReadyMessageSender.cs:line 62
at NServiceBus.Distributor.MSMQ.ReadyMessages.ReadyMessageSender.TransportOnFinishedMessageProcessing(Object sender, FinishedMessageProcessingEventArgs e) in c:\BuildAgent\work\c3100604bbd3ca20\src\NServiceBus.Distributor.MSMQ\ReadyMessages\ReadyMessageSender.cs:line 50
at System.EventHandler1.Invoke(Object sender, TEventArgs e)
at NServiceBus.Unicast.Transport.TransportReceiver.OnFinishedMessageProcessing(TransportMessage msg) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 435
NServicebus version: 4.6.0.0
Queueing: MSMQ
The worker ends it unit of work by sending a message back to the distributor. This send will join the existing distributed transaction. The error you get is caused by this new transactional resource trying to join an already failing transaction. Something has marked the distributed transaction as rolling back.
This is normally caused by your code. Either your database operation is failing somehow or you probably have exceeded the transaction timeout limit handling the message. (Default one minute)
Check your logs to see if you are using above the transaction timeout limit to process the message on the worker.
crosspost: https://orchard.codeplex.com/discussions/473454
I want to start by saying I'm currently migrating from Orchard CMS 1.6 to 1.7.2. So it used to work in 1.6 but I'm now having issues with 1.7.2.
2 of my Content Types are having issues when creating items, they never finish saving and when I check the logs I get this:
Orchard.Alias.Implementation.Updater.AliasHolderUpdater - Exception during Alias refresh
NHibernate.Exceptions.GenericADOException: could not execute query
[ select aliasrecor0_.Id as Id1829_, aliasrecor0_.Path as Path1829_, aliasrecor0_.RouteValues as RouteVal3_1829_, aliasrecor0_.Source as Source1829_, aliasrecor0_.Action_id as Action5_1829_ from Orchard_Alias_AliasRecord aliasrecor0_ where aliasrecor0_.Id>#p0 order by aliasrecor0_.Id asc ]
Name:p1 - Value:48
[SQL: select aliasrecor0_.Id as Id1829_, aliasrecor0_.Path as Path1829_, aliasrecor0_.RouteValues as RouteVal3_1829_, aliasrecor0_.Source as Source1829_, aliasrecor0_.Action_id as Action5_1829_ from Orchard_Alias_AliasRecord aliasrecor0_ where aliasrecor0_.Id>#p0 order by aliasrecor0_.Id asc] ---> System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out
When I stop it and view the site (anywhere really), it's entirely wrecked with this error:
Exception Details: System.ComponentModel.Win32Exception: The wait operation timed out
[Win32Exception (0x80004005): The wait operation timed out]
[SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.]
Line 162: return criteria
Line 163: .List<ContentItemVersionRecord>()
Line 164: .Select(x => ContentManager.Get(x.ContentItemRecord.Id, _versionOptions != null && _versionOptions.IsDraftRequired ? _versionOptions : VersionOptions.VersionRecord(x.Id)))
Source File: d:\Projects\Office Ignite\Main-1.7\src\Orchard\ContentManagement\DefaultContentQuery.cs Line: 162
I don't know why this is isolated with those two CTs. They don't have parts with custom tables or anything.
Any piece of information would be highly appreciated. Thanks!
I have same error, but it seems that problem is not related directly for my code.
I found two solutions for now:
1.) Taxonomy corruption problem https://orchard.codeplex.com/workitem/20411
2.) Static is dirty and lock which is default in select statment is heavly used https://serverfault.com/questions/419997/the-wait-operation-timed-out-when-running-sql-server-in-hyper-v