I am trying to use camel (3.7.5) aws2-s3 component to upload a file to AWS S3 storage. Consuming works just fine with bucket being configured using specific prefix as component's consumer option - access credentials are therefore correct (accessKey and secretKey). However, producer does not work leaving me with AccessDenied 403. I suspect that it is due to invalid prefix/path configured on producer: on consumer if I set invalid prefix I get identical error (403). On the producer I tried to use the same 'prefix' option but apparently that does not work/the docs also mention "prefix" as only consumer option. How do I set prefix for producer properly?
That works:
from("aws2-s3://MY_BUCKET?" +
"region=eu-central-1" +
"&accessKey=RAW(XXX)" +
"&secretKey=RAW(YYY)" +
"&prefix=MY_PREFIX").log("tick");
That does not work (403):
from("timer://foo?fixedRate=true&period=5000").routeId("aws-route")
.setHeader(AWS2S3Constants.KEY, simple("TEST.xml"))
.setBody(simple("Hello"))
.to("aws2-s3://MY_BUCKET?"
+ "region=eu-central-1"
+ "&prefix=MY_PREFIX"
+ "&accessKey=RAW(XXX)"
+ "&secretKey=RAW(YYY)");
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: JMCHHA87HN67C2B6, Extended Request ID: 6dEH4iuPXS4dgbJUCtHqw2gfmGuwgbw1cJcvevWpBCnZWJxjCg9oyd1MGWJh++pe2HIe1rT2dws=)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)
at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94)
at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$6(BaseClientHandler.java:252)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.s3.DefaultS3Client.listObjects(DefaultS3Client.java:5901)
at org.apache.camel.component.aws2.s3.AWS2S3Endpoint.doStart(AWS2S3Endpoint.java:114)
at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
at org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:230)
at org.apache.camel.support.service.BaseService.start(BaseService.java:115)
at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:84)
at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:101)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1487)
at org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
... 27 more
When you set the header
.setHeader(AWS2S3Constants.KEY, simple("TEST.xml"))
It should be enough to prepend the prefix
.setHeader(AWS2S3Constants.KEY, simple("<prefix>/TEST.xml"))
Related
I am trying to UNLOAD a Redshift table to an S3 bucket, but I am getting errors that I can't resolve.
When using 's3://mybucket/' as the destination (which is the documented way to specify the destination), I have an error saying S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint..
After some research I have tried to change the destination to include the full bucket url, without success.
All these destinations:
's3://mybucket.s3.amazonaws.com/',
's3://mybucket.s3.amazonaws.com/myprefix',
's3://mybucket.s3.eu-west-2.amazonaws.com/',
's3://mybucket.s3.eu-west-2.amazonaws.com/myprefix'
return this error S3ServiceException:The authorization header is malformed; the region 'eu-west-2' is wrong; expecting 'us-east-1', which is also the error returned when I use a bucket name that doesn't exist.
My Redshift cluster and my s3 buckets all exist in the same region, eu-west-2.
What am I doing wrong?
[appendix]
Full command:
UNLOAD ('select * from mytable')
to 's3://mybucket.s3.amazonaws.com/'
iam_role 'arn:aws:iam::0123456789:role/aws-service-
role/redshift.amazonaws.com/AWSServiceRoleForRedshift'
Full errors:
ERROR: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect,Rid 6ADF2C929FD2BE08,ExtRid vjcTnD02Na/rRtLvWsk5r6p0H0xncMJf6KBK
DETAIL:
-----------------------------------------------
error: S3ServiceException:The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.,Status 301,Error PermanentRedirect,Rid 6ADF2C929FD2BE08,ExtRid vjcTnD02Na/rRtLvWsk5r6p0H0xncMJf6KBK
code: 8001
context: Listing bucket=mybucket prefix=
query: 0
location: s3_unloader.cpp:226
process: padbmaster [pid=30717]
-----------------------------------------------
ERROR: S3ServiceException:The authorization header is malformed; the region 'eu-west-2' is wrong; expecting 'us-east-1',Status 400,Error AuthorizationHeaderMalformed,Rid 559E4184FA02B03F,ExtRid H9oRcFwzStw43ynA+rinTOmynhWfQJlRz0QIcXcm5K7fOmJSRcOcHuVlUlhGebJK5iH2L
DETAIL:
-----------------------------------------------
error: S3ServiceException:The authorization header is malformed; the region 'eu-west-2' is wrong; expecting 'us-east-1',Status 400,Error AuthorizationHeaderMalformed,Rid 559E4184FA02B03F,ExtRid H9oRcFwzStw43ynA+rinTOmynhWfQJlRz0QIcXcm5K7fOmJSRcOcHuVlUlhGebJK5iH2L
code: 8001
context: Listing bucket=mybucket.s3.amazonaws.com prefix=
query: 0
location: s3_unloader.cpp:226
process: padbmaster [pid=30717]
-----------------------------------------------
Bucket zone
Cluster zone
try:
conn = boto.connect_s3(access_key,secret_access_key)
bucket = conn.get_bucket(bucket_name, validate=False)
k1 = Key(bucket)
k1.key = 'Date_Table.csv'
# k = bucket.get_key('Date_Table.csv')
k1.make_public()
k1.get_contents_to_filename(tar)
except Exception as e:
print(e)
i am getting error
S3ResponseError: 403 Forbidden
AccessDeniedAccess
DeniedD9ED8BFF6D6A993Eaw0KmxskATNBTDUEo3SZdwrNVolAnrt9/pkO/EGlq6X9Gxf36fQiBAWQA7dBSjBNZknMxWDG9GI=
i tried all posibility and still getting same error .. please guide me how to solve this issue.
i tried other way as below and getting error
An error occurred (NoSuchKey) when calling the GetObject operation:
The specified key does not exist.
session = boto3.session.Session(aws_access_key_id=access_key, aws_secret_access_key=secret_access_key,region_name='us-west-2')
print ("session:"+str(session)+"\n")
client = session.client('s3', endpoint_url=s3_url)
print ("client:"+str(client)+"\n")
stuff = client.get_object(Bucket=bucket_name, Key='Date_Table.csv')
print ("stuff:"+str(stuff)+"\n")
stuff.download_file(local_filename)
ge
Always use boto3. boto is deprecated.
As long as you setup AWS CLI credential, you don't need to pass the hard-coded credential. Read boto3 credential setup throughly.
There is no reason to initiate boto3.session unless you are using different region and user profile.
Take your time and study difference between service client(boto3.client) vs service resources(boto3.resources).
Low level boto3.client is easier to use for experiments. Use high level boto3.resource if you need to pass around arbitrary object.
Here is the simple code for boto3.client("s3").download_file.
import boto3
# initiate the proper AWS services client, i.e. S3
s3 = boto3.client("s3")
s3.download_file('your_bucket_name', 'Date_Table.csv', '/your/local/path/and/filename')
I have an exchange setup amq.topic and in there I have a routing key that is pointing to my key.
I have a route defined with a rest endpoint. I make a call to that endpoint with a body as json {"sample" : "sample"}.
rest("myendpoint")
.post()
.route()
.routeId(ROUTE_ID)
.log(ROUTE_ID + " started.")
.convertBodyTo(String.class)
.log("Receiving request with body=${body}")
.to("rabbitmq://rabbiturl:port/amq.topic?connectionFactory=rabbitmqConnectionFactory&autoDelete=false&routingKey=myroutingkey&declare=false&exchangeType=topic")
.log(ROUTE_ID + " ended.")
.end();
This accomplishes what I want to: the body of my rest call will be sent to the topic exchange of which will go to my queue based on the routing key.
The problem: It also create an auto-generated queue
[ qtp86171426-28] RabbitMQProducer INFO Starting reply manager service RabbitMQReplyManagerTimeoutChecker[amq.topic]
[ qtp86171426-28] ReplyManagerSupport INFO Using temporary queue name: amq.gen-JGF0CKioQ6_MLA9uBNIOOQ
[ qtp86171426-28] CorrelationTimeoutMap INFO in putIfAbsent with key Camel-ID-L-VC-10050-58983-1467158153454-0-3
[nagerTimeoutChecker[amq.topic]] TemporaryQueueReplyHandler INFO in onTimeout with correlationId= Camel-ID-L-VC-10050-58983-1467158153454-0-3
[nagerTimeoutChecker[amq.topic]] ReplyManagerSupport WARN Timeout occurred after 20000 millis waiting for reply message with correlationID [Camel-ID-L-VC-10050-58983-1467158153454-0-3] on destination amq.gen-JGF0CKioQ6_MLA9uBNIOOQ. Setting ExchangeTimedOutException on (MessageId: ID-L-VC-10050-58983-1467158153454-0-2 on ExchangeId: ID-L-VC-10050-58983-1467158153454-0-1) and continue routing.
I have tried skipqueuedeclare & skipexchangedeclare and am running out of ideas of why it is generating the queue. Does anyone have any ideas?
Set ExchangePattern to InOnly.
The temporary and reply queue is because the route expects a reply back so creates the temporary queue to wait for a response.
So if your rest endpoint does not return anything set the ExchangePattern to Inonly at the rest call and it should be ok. Alternatively set it at .to().
I am trying to set a scheduler in order to set a cron expression.
<camel:endpoint id="sqlEndpoint" uri="sql:${sqlQuery}?scheduler=spring&scheduler.cron=0+6+8+*+*&dataSourceRef=veloxityDS&useIterator=false"/>
But when I run this as a consumer, this exception occured:
org.apache.camel.FailedToCreateConsumerException: Failed to create
Consumer for endpoint: Endpoint[sql://$select * from
dual?dataSourceRef=veloxityDS&scheduler=spring&scheduler.cron=0+6+8++&useIterator=false].
Reason: There are 1 scheduler parameters that couldn't be set on the
endpoint. Check the uri if the parameters are spelt correctly and that
they are properties of the endpoint. Unknown parameters=[{cron=0 6 8 *
*}]
Any ideas?
The endpoint you are trying to create is using parameters that don't exist. There is a full list of parameters at: http://camel.apache.org/sql-component.html
If you want your SQL procedure to run on a time interval you can either use a quartz endpoint, a polling consumer, or a route scheduler depending on your needs.
http://camel.apache.org/polling-consumer.html
http://camel.apache.org/quartz2.html
http://camel.apache.org/cronscheduledroutepolicy.html
Current parameter issues on your endpoint:
scheduler - not a supported parameter
scheduler.cron - not a supported parameter
dataSourceRef - deprecated.
Your scheduling alternatives leveraging only the sql endpoint are:
consumer.delay
consumer.initialDelay
consumer.useFixedDelay
maxMessagesPerPoll
Our FIX Engine implemented in QuickfixJ send the received quote message in the quote session to another application listening on JMS queue for further processing like persistig to DB.
The message serialized and de-serialized using Apache serializationUtils.
The problem is, the application throws FieldNotFound error when it attempts to retrieve SenderCompId header field from the de-serialized message.
setSenderlocid(header.getField(new SenderLocationID()).toString())
quickfix.FieldNotFound: Field [142] was not found in message.
at quickfix.FieldMap.getField(FieldMap.java:216)
at quickfix.FieldMap.getFieldInternal(FieldMap.java:353)
at quickfix.FieldMap.getField(FieldMap.java:349)
Does serialization work with QuickFixJ messages?