I am trying to use Camel AWS2 S3 component to retrieve objects from AWS S3 service.
Since more than one instance of this route will be running, I am using Idempotent
Consumer EIP in the route to filter out duplicates. I am using hazel cast idempotent repository in the route.
When I am running one instance of the application, everything works fine.
When I am running two instances of the application, I am seeing **software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist.**
The file consumption is not a problem and it's working fine.
If application instance 1 is consuming a file from s3, sometimes I am seeing the error in application instance 2.
If application instance 2 is consuming a file from s3, sometimes I am seeing the error in application instance 1.
My route:
`from("aws2-s3://test-bucket?prefix=mypoc&moveAfterRead=true)
.routeId("myRoute") .idempotentConsumer(simple("${header.CamelAwsS3BucketName}-${header.CamelAwsS3Key}"),repository)
.skipDuplicate(true)
.log(LoggingLevel.INFO, "Message recieved{}",String.valueOf(simple("${header.CamelAwsS3Key}")));`
Exception:
Caused by: \[software.amazon.awssdk.services.s3.model.NoSuchKeyException - The specified key does not exist. (Service: S3, Status Code: 404, Request ID: WNVX1YQWZQWAFNMB, Extended Request ID: Vm1pTrjSbM71R9h/f7+ypr60/Gn4j5pzgCZDsAhtVzd9QZBGBbrq8U14DMPWf0GOmy5/pmJvbno=)\] software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: WNVX1YQWZQWAFNMB, Extended Request ID: Vm1pTrjSbM71R9h/f7+ypr60/Gn4j5pzgCZDsAhtVzd9QZBGBbrq8U14DMPWf0GOmy5/pmJvbno=)at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) \~\[sdk-core-2.17.247.jar:na\]
But I am wondering why I am seeing the exception.Looks like both the instances are trying to consume the file and one of the instance is throwing the error.
Any insights on why we are seeing the exception?
Related
I am using spring cloud config server and getting repetitive logs from NativeEnvironmentRepository class without any api call.
Says
Adding property source:Config resource ‘file [//application.properties] via location []
[application.properties]’
I want to use the copyObjectAsync of S3 SDK (.net core) in order to copy keys from one bucket to another.
In AWS documentation (https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) I found :"A copy request might return an error when Amazon S3 receives the copy request or while Amazon S3 is copying the files. If the error occurs before the copy action starts, you receive a standard Amazon S3 error. If the error occurs during the copy operation, the error response is embedded in the 200 OK response. This means that a 200 OK response can contain either a success or an error. Design your application to parse the contents of the response and handle it appropriately."
However there is no explanation on how or where to get the error. Is it will be exception?
I found similar question :
How to catch failed S3 copyObject with 200 OK result in AWSJavaScriptSDK
But I wonder if there are another explanation why the SDK not support it and other methods to assure if the copy succeed.
Thanks in advance,
Yossi
Getting below connection error when trying to validate S3 source endpoint of DMS.
Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect to database.
Followed all the steps listed in the below links but still maybe I am missing something...
https://aws.amazon.com/premiumsupport/knowledge-center/dms-connection-test-fail-s3/
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.S3.html
The role associated with the endpoint does have access to the S3 bucket of the endpoint, along with dms being listed as trusted entity.
I got this same error when trying to use S3 as a target.
The one thing not mentioned in the documentation, and which turned out to be the root cause for my error, is that the DMS Replication Instance and the Bucket need to be in the same region.
New to Amazon S3 usage.I get the following error when trying to access the file from Amazon S3 using a simple java method.
2016-08-23 09:46:48 INFO request:450 - Received successful response:200, AWS Request ID: F5EA01DB74D0D0F5
Caught an AmazonClientException, which means the client encountered an
internal error while trying to communicate with S3, such as not being
able to access the network.
Error Message: Unable to store object contents to disk: Read timed out
The exact lines of code worked yesterday.I was able to download 100% of 5GB file in 12 min. Today I'm in a better connected environment but only 2% or 3% of the file is downloaded and then the program fails.
Code that I'm using to download.
s3Client.getObject(new GetObjectRequest("mybucket", file.getKey()), localFile);
You need to set the connection timeout and the socket timeout in your client configuration.
Click here for a reference article
Here is an excerpt from the article:
Several HTTP transport options can be configured through the com.amazonaws.ClientConfiguration object. Default values will suffice for the majority of users, but users who want more control can configure:
Socket timeout
Connection timeout
Maximum retry attempts for retry-able errors
Maximum open HTTP connections
Here is an example on how to do it:
Downloading files >3Gb from S3 fails with "SocketTimeoutException: Read timed out"
Background
I have an API built with OpenRasta that serves JSON-encoded data to an HTML/AJAX enabled interface.
Problem
When hosted with IIS 6, I'm experiencing intermittent 404 errors when a resource is queried. Everything works well after an application pool restart; after a few successful requests the 404 errors start and persist until the app pool is recycled. I have confirmed ISAPI mapping is set in IIS 6, what am I missing?
Trace output for intermittent failed request
Unregistering host of type OpenRasta.Hosting.AspNet.AspNetHost
Trace output for successful request for comparison (namespace and resource URIs redacted)
Registering host of type OpenRasta.Hosting.AspNet.AspNetHost
Using dependency resolver of type OpenRasta.DI.InternalDependencyResolver.
Registering host's root dependencies..
Using dependency registrar of type OpenRasta.Configuration.DefaultDependencyRegistrar..
Ignoring constructor, following dependencies didn't have a registration:OpenRasta.TypeSystem.Surrogates.ISurrogateBuilder[].
Registering host's leaf dependencies..
Using configuration source <namespace>.api.Configuration.
Ignoring constructor, following dependencies didn't have a registration:System.Func`1[System.Collections.Generic.IEnumerable`1[OpenRasta.Configuration.MetaModel.Handlers.IMetaModelHandler]].
Ignoring constructor, following dependencies didn't have a registration:System.Diagnostics.TraceSource.
Ignoring constructor, following dependencies didn't have a registration:OpenRasta.OperationModel.MethodBased.IMethodFilter[].
Incoming host request for http://<resource URI>.
Adding communication context data.
Entering OpenRastaRewriterHandler: Rewriting to original path.
Entering OpenRastaIntegratedHandler: Request for http://<resource URI>.
Incoming host request for http://<resource URI>.
Adding communication context data.
Found 1 operation(s) with a matching name..
Found 0 operation(s) with matching [HttpOperation] attribute..
Operation <handler>::Get() selected with 0 required members and 0 optional members, without request codec.
Ignoring constructor, following dependencies didn't have a registration:OpenRasta.OperationModel.Interceptors.IOperationInterceptor[].
Executing OperationResult OperationResult: type=OK, statusCode=200..
Selected codec JsonDataContractCodec out of 1 codecs for entity of type List`1 and negotiated media type application/json; q=0.5..
Codec JsonDataContractCodec selected..
Entering ResponseEntityWriterContributor: Generating response entity..
Setting Content-Length to 133.
Exiting ResponseEntityWriterContributor.
Writing http headers..
Exiting OpenRastaIntegratedHandler.
Exiting OpenRastaRewriterHandler.
Request finished..
OpenRasta.Hosting.AspNet.dll built from GitHub master: