Opscenter backup failure - S3 500 InternalError - amazon-s3

Question

The backup service using Opscenter is failing with below AWS S3 error - 500 InternalError. It is failing on 3-4 nodes for different files with this error code. AWS S3 documentation recommends to retry the operation (We encountered an internal error. Please try again). I can see this file name and error only once in agent log and S3 log, which means there was no retry done after this error. Is there any way to enable retry for S3 500 error code (InternalError) in opscenter? Any suggestion on how to fix this error?
Error while sending opsagent.backups.mytable.SSTable#1d6cb0c5 to
org.jclouds.http.HttpResponseException: request: HEAD https://my-backups.s3.amazonaws.com/snapshots/a3b72e7b-bd70-4f9e-aa2a-
cf6c2a5ff336/sstables/1458057843-my_ks-mytable-ka-57417-Index.db.gz HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error
at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:67)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy51.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:175)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:48)
at opsagent.backups.destinations$fn__12755.invoke(destinations.clj:185)
at opsagent.backups.destinations$fn__12385$G__12378__12396.invoke(destinations.clj:25)
at opsagent.backups.staging$start_staging_BANG_$fn__12925$state_machine__5264__auto____12926$fn__12931$fn__12962.invoke(staging.clj:61)
at opsagent.backups.staging$start_staging_BANG_$fn__12925$state_machine__5264__auto____12926$fn__12931.invoke(staging.clj:59)
at opsagent.backups.staging$start_staging_BANG_$fn__12925$state_machine__5264__auto____12926.invoke(staging.clj:56)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async.impl.ioc_macros$take_BANG_$fn__5280.invoke(ioc_macros.clj:953)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__1785.invoke(channels.clj:102)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
1
Details:
OpsCenter 5.2.4
DSE 4.8.2
data size per node ~130GB x 3 nodes x 3 dc
Compression and S3 server side encryption enabled
JVM_OPTS="$JVM_OPTS -Xmx512M -Djclouds.mpu.parts.magnitude=100000 -Djclouds.mpu.parts.size=32000000"

Answer 1

There was an issue in OpsCenter 5.2.4 that placed the blob_exists code outside of our retry loop. This has been fixed in OpsCenter 6.0.1.