Apache Camel AWS S3 Bucket Nested Directory: - amazon-s3

Question

I am currently trying to use Apache Camel with a S3 bucket on AWS. We currently have 1 bucket for each environment that we have for our code bases (for ex: dev bucket, e2e bucket, prod bucket).
We would like to use 1 bucket with multiple directories inside to avoid the messiness and consolidate the space. We could then point camel to a folder within an endpoint to listen to (seems feasible). This has become extremely problematic while trying to configure. Camel seems to expect a bucket for each endpoint that it can connect to.
Can someone shed light on whether this is possible or if it is necessary to have 1 bucket per endpoint? The camel documentation does not specify whether it is possible.

Answer 1

Indeed as Alex Nano wrote, s3 folders are just prefixes to the object key (the "filename"), so if you want to set up multiple routes monitoring different s3 "folders" you could use the prefix option of the aws:s3 endpoint
Prefixes work in the same function you would use aws s3 cli,
e.g.
aws s3 ls s3://mybucket/myfolderInsideBucket/
A JUnit example
public class CamelAwsS3Test extends CamelTestSupport {
#EndpointInject(uri = "mock:result")
private MockEndpoint resultEndpoint;
#Before
public void setup() throws Exception {
context.start();
}
#Test
public void foo() throws Exception {
int expectedFileCount = 3
resultEndpoint.expectedMessageCount(expectedFileCount);
resultEndpoint.assertIsSatisfied();
}
#Override
protected JndiRegistry createRegistry() throws Exception {
JndiRegistry registry = super.createRegistry();
final String accessKey = "ACCESSKEY";
final String secretKey = "SECRETKEY";
AmazonS3 client = new AmazonS3Client(new BasicAWSCredentials(accessKey, secretKey));
registry.bind("amazonS3Client", client);
return registry;
}
#Override
protected RoutesBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
#Override
public void configure() throws Exception {
from("aws-s3://{BUCKET}?amazonS3Client=#amazonS3Client&prefix=${FOLDER}")
.log("hey ${body}").to("mock:result");
}
};
}
}

Answer 2

few thoughts:
S3 only looks like folders. in reality, it's a single name:value pair (bucket:key).
That is, only the top level 'buckets' are actually folders; what looks like a path, is just a key.
so - just use one bucket, and then filter out the key per predicate (http://camel.apache.org/predicate.html) case statement to process what you need.
I'm sure there are other ways, but this seems straightforward.
also note AWS s3 limits that justify your impulse to consolidate:
only 100 buckets per AWS account
all buckets must be uniquely named in a region (even across all AWS accounts)