Add folder in Amazon s3 bucket - amazon-s3

I want to add Folder in my amazon s3 bucket using coding.
Can you please suggest me how to achieve this?

There are no folders in Amazon S3. It just that most of the S3 browser tools available show part of the key name separated by slash as a folder.
If you really need that you can create an empty object with the slash at the end. e.g. "folder/" It will looks like a folder if you open it with a GUI tool and AWS Console.

As everyone has told you, in AWS S3 there aren't any "folders", you're thinking of them incorrectly. AWS S3 has "objects", these objects can look like folders but they aren't really folders in the fullest sense of the word. If you look for creating folders on the Amazon AWS S3 you won't find a lot of good results.
There is a way to create "folders" in the sense that you can create a simulated folder structure on the S3, but again, wrap your head around the fact that you are creating objects in S3, not folders. Going along with that, you will need the command "put-object" to create this simulated folder structure. Now, in order to use this command, you need the AWS CLI tools installed, go here AWS CLI Installation for instructions to get them installed.
The command is this:
aws s3api put-object --bucket your-bucket-name --key path/to/file/yourfile.txt --body yourfile.txt
Now, the fun part about this command is, you do not need to have all of the "folders" (objects) created before you run this command. What this means is you can have a "folder" (object) to contain things, but then you can use this command to create the simulated folder structure within that "folder" (object) as I discussed earlier. For example, I have a "folder" (object) named "importer" within my S3 bucket, lets say I want to insert sample.txt within a "folder" (object) structure of the year, month, and then a sample "folder" (object) within all of that.
If I only have the "importer" object within my bucket, I do not need to go in beforehand to create the year, month, and sample objects ("folders") before running this command. I can run this command like so:
aws s3api put-object --bucket my-bucket-here --key importer/2016/01/sample/sample.txt --body sample.txt
The put-object command will then go in and create the path that I have specified in the --key flag. Here's a bit of a jewel: even if you don't have a file to upload to the S3, you can still create objects ("folders") within the S3 bucket, for example, I created a shell script to "create folders" within the bucket, by leaving off the --body flag, and not specifying a file name, and leaving a slash at the end of the path provided in the --key flag, the system creates the desired simulated folder structure within the S3 bucket without inserting a file in the process.
Hopefully this helps you understand the system a little better.
Note: once you have a "folder" structure created, you can use the S3's "sync" command to syncronize the descendant "folder" with a folder on your local machine, or even with another S3 bucket.

Java with AWS SDK:
There are no folders in s3, only key/value pairs. The key can contain slashes (/) and that will make it appear as a folder in management console, but programmatically it's not a folder it is a String value.
If you are trying to structure your s3 bucket, then your naming conventions (the keys you give your files) can simply follow normal directory patterns, i.e. folder/subfolder/file.txt.
When searching (depending on language you are using), you can search via prefix with a delimiter. In Java, it would be a listObjects(String storageBucket, String prefix, String delimiter) method call.
The storageBucket is the name of your bucket, the prefix is the key you want to search, and the delimiter is used to filter your search based off the prefix.

The AWS:S3 rails gem does this by itself:
AWS::S3::S3Object.store("teaser/images/troll.png", file, AWS_BUCKET)
Will automatically create the teaser and images "folders" if they don't already exist.

With AWS SDK .Net works perfectly, just add "/" at the end of the name folder:
var folderKey = folderName + "/"; //end the folder name with "/"
AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(AWSAccessKey, AWSSecretKey);
var request = new PutObjectRequest();
request.WithBucketName(AWSBucket);
request.WithKey(folderKey);
request.WithContentBody(string.Empty);
S3Response response = client.PutObject(request);
Then refresh your AWS console, and you will see the folder

With aws cli, it is possible to copy an entire folder to a bucket.
aws s3 cp /path/to/folder s3://bucket/path/to/folder --recursive
There is also the option to sync a folder using aws s3 sync

This is a divisive topic, so here is a screenshot in 2019 of the AWS S3 console for adding folders and the note:
When you create a folder, S3 console creates an object with the above
name appended by suffix "/" and that object is displayed as a folder
in the S3 console.
Then 'using coding' you can simply adjust the object name by prepending a valid folder name string and a forward slash.

For Swift I created a method where you pass in a String for the folder name.
Swift 3:
import AWSS3
func createFolderWith(Name: String!) {
let folderRequest: AWSS3PutObjectRequest = AWSS3PutObjectRequest()
folderRequest.key = Name + "/"
folderRequest.bucket = bucket
AWSS3.default().putObject(folderRequest).continue({ (task) -> Any? in
if task.error != nil {
assertionFailure("* * * error: \(task.error?.localizedDescription)")
} else {
print("created \(Name) folder")
}
return nil
})
}
Then just call
createFolderWith(Name:"newFolder")

In iOS (Objective-C), I did following way
You can add below code to create a folder inside amazon s3 bucket programmatically. This is working code snippet. Any suggestion Welcome.
-(void)createFolder{
AWSS3PutObjectRequest *awsS3PutObjectRequest = [AWSS3PutObjectRequest new];
awsS3PutObjectRequest.key = [NSString stringWithFormat:#"%#/", #"FolderName"];
awsS3PutObjectRequest.bucket = #"Bucket_Name";
AWSS3 *awsS3 = [AWSS3 defaultS3];
[awsS3 putObject:awsS3PutObjectRequest completionHandler:^(AWSS3PutObjectOutput * _Nullable response, NSError * _Nullable error) {
if (error) {
NSLog(#"error Creating folder");
}else{
NSLog(#"Folder Creating Sucessful");
}
}];
}

Here's how you can achieve what you're looking for (from code/cli):
--create/select the file (locally) which you want to move to the folder:
~/Desktop> touch file_to_move
--move the file to s3 folder by executing:
~/Desktop> aws s3 cp file_to_move s3://<path_to_your_bucket>/<new_folder_name>/
A new folder will be created on your s3 bucket and you'll now be able to execute cp, mv, rm ... statements i.e. manage the folder as usual.
If this new file created above is not required, simply delete it. You now have an s3 bucket created.

You can select language of your choice from available AWS SDK
Alternatively you can try minio client libraries available in Python, Go, .Net, Java, Javascript for your application development environment, it has example directory with all basic operations listed.
Disclaimer: I work for Minio

In swift 2.2 you can create folder using
func createFolderWith(Name: String!) {
let folderRequest: AWSS3PutObjectRequest = AWSS3PutObjectRequest()
folderRequest.key = Name + "/"
folderRequest.bucket = "Your Bucket Name"
AWSS3.defaultS3().putObject(folderRequest).continueWithBlock({ (task) -> AnyObject? in
if task.error != nil {
assertionFailure("* * * error: \(task.error?.localizedDescription)")
} else {
print("created \(Name) folder")
}
return nil
})
}

Below creates a empty directory called "mydir1".
Below is nodejs code, it should be similar for other languages.
The trick is to have slash (/) at the end of the name of object, as in "mydir1/", otherwise a file with name "mydir1" will be created.
let AWS = require('aws-sdk');
AWS.config.loadFromPath(__dirname + '\\my-aws-config.json');
let s3 = new AWS.S3();
var params = {
Bucket: "mybucket1",
Key: "mydir1/",
ServerSideEncryption: "AES256" };
s3.putObject(params, function (err, data) {
if (err) {
console.log(err, err.stack); // an error occurred
return;
} else {
console.log(data); // successful response
return;
/*
data = {
ETag: "\"6805f2cfc46c0f04559748bb039d69ae\"",
ServerSideEncryption: "AES256",
VersionId: "Ri.vC6qVlA4dEnjgRV4ZHsHoFIjqEMNt"
}
*/
} });
Source: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property

in-order to create a directory inside s3 bucket and copy contents inside that is pretty simple.
S3 command can be used:
aws s3 cp abc/def.txt s3://mybucket/abc/
Note: / is must that makes the directory, otherwise it will become a file in s3.

I guess your query is just simply creating a folder inside folder(subfolder).
so while coping any directory data inside a bucket sub folder use command like this.
aws s3 cp mudit s3://mudit-bucket/Projects-folder/mudit-subfolder --recursive
It will create a subfolder and put ur directory contents in it. Also once your subfolder data gets empty. Your Subfolder will automatically gets deleted.

You can use copy command to create a folder while copy a file.
aws s3 cp test.xml s3://mybucket/myfolder/test.xml

Related

Confusion with AWS --resources-to-import and --template-url

I'm trying to import an existing S3 bucket into a newly created CloudFormation stack. As a reference, I'm using this site. I use a Github workflow runner to execute this, like so:
- name: Add existing S3 bucket and object to Stack
run: aws cloudformation create-change-set
--stack-name ${{ env.STACK_NAME }} --change-set-name ImportChangeSet
--change-set-type IMPORT
--resources-to-import file://ResourcesToImport.txt
--template-url https://cf-templates.s3.eu-central-1.amazonaws.com/ResourcesToImport.yaml
I'm a little confused to what exactly should the ResourcesToImport.txt and ResourcesToImport.yaml contain. I currently have:
ResourcesToImport.txt
[
{
"ResourceType":"AWS::S3::Bucket",
"LogicalResourceId":"myBucket",
"ResourceIdentifier": {
"resourceName":"myBucket",
"resourceType":"AWS::S3::Bucket"
}
}
]
NB: As a sidenote, I have just used the bucket name, but actually I just want a specific folder within that bucket.
ResourcesToImport.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: Import existing resources
Resources:
S3SourceBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
BucketName: myBucket
I'm quite sure the replication of information in both of these files is redundant and incorrect. The ResourcesToImport.yaml file is uploaded in advance to the bucket cf-templates/ResourcesToImport.yaml
What should these two files actually contain, if I am to import only an existing S3 bucket and folder?
EDIT
In addition to the template route, I also tried adding the S3 bucket via the console. However when the S3 url is added (s3://myBucket/folder1/folder2/), I get:
S3 error: Domain name specified in myBucket is not a valid S3 domain
Here's what the two file inputs to create-change-set should contain when importing:
--resources-to-import The resources to import into your stack. This identifies the to-be-imported resources. Not a template. Make sure the LogicalResourceId matches the resource id in the template below. In your case: "LogicalResourceId": "S3SourceBucket".
--template-url The [S3] location of the file that contains the revised template. This is a CloudFormation template that includes (a) the to-be-imported resources AND (b) the existing stack resources. This is what CloudFormation will deploy when you execute the change set. Note: alternatively, use --template-body with a local file template instead.
Regarding your EDIT:
Bucket names cannot contain slashes. Object Keys can. S3 does not have folders per se, although object keys with a / have some folder-like properties. The path/to/my.json together is the S3 object Key name:
Amazon S3 supports buckets and objects, and there is no hierarchy. However, by using prefixes and delimiters in an object key name, the Amazon S3 console and the AWS SDKs can infer hierarchy and introduce the concept of folders

Do triggers exist for folders in S3 bucket when all objects are deleted?

I have a S3 folder structure as below:
s3://ABC/sample-file.csv
I want to trigger a lambda function whenever sample-file lands in ABC folder. Now, the requirement is to move(copy and delete) the sample file. Now, when I delete the sample-file using python boto3, even ABC folder is deleted as it has no objects.
boto3_session = boto3.client('s3')
boto3_session.delete_object(Bucket = Bucketname, Key = 'ABC/sample-file.csv')
I want to trigger that lambda function whenever I write sample file to ABC folder again. My question is will the previously defined lambda function be triggered as the ABC folder will be created again?
Alternatively, can I retain ABC folder without any files?
Yes. Remember that there aren't actually folders in S3, but rather the folder path is just part of the key name. The folders are just a friendly way to display the information in the console and on the cli.
Once you create the S3 event notification it will stay in effect even if you deleted the last file in a "folder" and it appears there is no folder.

How to get information of S3 bucket?

Say for example I have the following bucket set up:
bucketone
…/folderone
…/text1.txt
…/text2.txt
…/foldertwo
…/file1.json
…/folderthree
…/folderthreesub
…/file2.json
…/file3.json
But it only goes down one level.
What’s the proper way of retrieving information under a bucket?
Will be sure to accept/upvote answer.
Whats wrong with just doing this from the CLI?
aws s3 cp s3://bucketing . --recursive
Contrary to the way you'd think it will work, rsplit() actually returns the splits from left-right, even though it applies it right-to-left.
Therefore, you actually want to obtain the last element of the split:
filename = obj['Key'].rsplit('/', 1)[-1]
See: Python rsplit() documentation
Also, be careful of 'pretend directories' that might be created via the console. They are actually zero-length files the make the folder appear in the UI. Therefore, skip files with no name after the final slash.
Make those fixes and it works as desired:
import boto3
import os
s3client = boto3.client('s3')
for obj in s3client.list_objects_v2(Bucket='my-bucket')['Contents']:
filename = obj['Key'].rsplit('/', 1)[-1]
localfiledir = os.path.join('/tmp', filename)
if filename != '':
s3client.download_file('my-bucket', obj['Key'], localfiledir)

Download files with extension .txt or wildcard from Amazon S3 using .net sdk?

Is there a way to download all the files with specific extension (e.g. *.txt) OR wildcard string (1234 *.*) from a folder from Amazon S3 using .Net SDK in one go rather then looping through the list of files.
You can't make one call to download all files of a certain pattern but you can use the pattern in the list object call and then download the individual files. Using the S3DirectoryInfo from the Amazon.S3.IO namespace can simplify this.
S3DirectoryInfo info = new S3DirectoryInfo(s3Client, bucketName);
foreach (var file in info.GetFiles("*.jpg", SearchOption.AllDirectories))
{
var localPath = Path.Combine(#"C:\Temp\download", file.FullName.Substring(bucketName.Length + 2));
Console.WriteLine("Downloading: {0}", localPath);
file.CopyToLocal(localPath);
}

Amazon S3 boto - how to create a folder?

How can I create a folder under a bucket using boto library for Amazon s3?
I followed the manual, and created the keys with permission, metadata etc, but no where in the boto's documentation it describes how to create folders under a bucket, or create a folder under folders in bucket.
There is no concept of folders or directories in S3. You can create file names like "abc/xys/uvw/123.jpg", which many S3 access tools like S3Fox show like a directory structure, but it's actually just a single file in a bucket.
Assume you wanna create folder abc/123/ in your bucket, it's a piece of cake with Boto
k = bucket.new_key('abc/123/')
k.set_contents_from_string('')
Or use the console
Use this:
import boto3
s3 = boto3.client('s3')
bucket_name = "YOUR-BUCKET-NAME"
directory_name = "DIRECTORY/THAT/YOU/WANT/TO/CREATE" #it's name of your folders
s3.put_object(Bucket=bucket_name, Key=(directory_name+'/'))
With AWS SDK .Net works perfectly, just add "/" at the end of the folder name string:
var folderKey = folderName + "/"; //end the folder name with "/"
AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(AWSAccessKey, AWSSecretKey);
var request = new PutObjectRequest();
request.WithBucketName(AWSBucket);
request.WithKey(folderKey);
request.WithContentBody(string.Empty);
S3Response response = client.PutObject(request);
Then refresh your AWS console, and you will see the folder
Tried many method above and adding forward slash / to the end of key name, to create directory didn't work for me:
client.put_object(Bucket="foo-bucket", Key="test-folder/")
You have to supply Body parameter in order to create directory:
client.put_object(Bucket='foo-bucket',Body='', Key='test-folder/')
Source: ryantuck in boto3 issue
Append "_$folder$" to your folder name and call put.
String extension = "_$folder$";
s3.putObject("MyBucket", "MyFolder"+ extension, new ByteArrayInputStream(new byte[0]), null);
see:
http://www.snowgiraffe.com/tech/147/creating-folders-programmatically-with-amazon-s3s-api-putting-babies-in-buckets/
Update for 2019, if you want to create a folder with path bucket_name/folder1/folder2 you can use this code:
from boto3 import client, resource
class S3Helper:
def __init__(self):
self.client = client("s3")
self.s3 = resource('s3')
def create_folder(self, path):
path_arr = path.rstrip("/").split("/")
if len(path_arr) == 1:
return self.client.create_bucket(Bucket=path_arr[0])
parent = path_arr[0]
bucket = self.s3.Bucket(parent)
status = bucket.put_object(Key="/".join(path_arr[1:]) + "/")
return status
s3 = S3Helper()
s3.create_folder("bucket_name/folder1/folder2)
It's really easy to create folders. Actually it's just creating keys.
You can see my below code i was creating a folder with utc_time as name.
Do remember ends the key with '/' like below, this indicates it's a key:
Key='folder1/' + utc_time + '/'
client = boto3.client('s3')
utc_timestamp = time.time()
def lambda_handler(event, context):
UTC_FORMAT = '%Y%m%d'
utc_time = datetime.datetime.utcfromtimestamp(utc_timestamp)
utc_time = utc_time.strftime(UTC_FORMAT)
print 'start to create folder for => ' + utc_time
putResponse = client.put_object(Bucket='mybucketName',
Key='folder1/' + utc_time + '/')
print putResponse
Although you can create a folder by appending "/" to your folder_name. Under the hood, S3 maintains flat structure unlike your regular NFS.
var params = {
Bucket : bucketName,
Key : folderName + "/"
};
s3.putObject(params, function (err, data) {});
S3 doesn't have a folder structure, But there is something called as keys.
We can create /2013/11/xyz.xls and will be shown as folder's in the console. But the storage part of S3 will take that as the file name.
Even when retrieving we observe that we can see files in particular folder (or keys) by using the ListObjects method and using the Prefix parameter.
Apparently you can now create folders in S3. I'm not sure since when, but I have a bucket in "Standard" zone and can choose Create Folder from Action dropdown.
This question is more relevant to the future, so adding this update.
I am using the upload_file method as shown below.
fold ='/my/system/filePath/tabmcq/Tables/auto/18.tsv'
s3_client.upload_file(
Filename = full/file/path/filename.extension,
Bucket = "tab-mcq-de",
Key = f"{fold.split('/')[-3]}/{fold.split('/')[-2]}/{fold.split('/')[-1]}"
)
Ideas is the "Filename" parameter requires the absolute file path of your system.
The "Key" parameter requires the relative file path from the source directory where your files are located
In case of this example, Key parameter has to contain "Tables/auto/18.tsv" value, for client to create the folders.
Hope this helps.
The following works using Python boto3
s3 = boto3.client("s3")
s3.put_object(Bucket="dest_bucket", Key='folder_name/')