This question already has answers here:
Add folder in Amazon s3 bucket
(16 answers)
Closed last month.
As per my understanding, object storage has a 'flat' structure so you cannot create folders within buckets. However, in both GCP & AWS, I am able to upload regular folders to the buckets, which also look like regular folders on their web UI console. What is the difference between the folders I am seeing on these buckets, and the folders which are there in a file-storage system (like my personal laptop)?
As far as I know Object Storage has a 'flat' structure so you cannot create folders within buckets, nor can you nest buckets in buckets.
If you need to have some form of 'folder'-like structure, then using prefixes is the way to go. You'll then end up with this structure: {endpoint}/{bucket-name}/{object-prefix}/{object-name}.
thats what you are seeing according to me
Amazon S3 has a flat structure instead of a hierarchy as you would see in a file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. It does this by using a shared name prefix for objects (that is, objects have names that begin with a common string). Object names are also referred to as key names.
For example, you can create a folder on the console named photos and store an object named myphoto.jpg in it. The object is then stored with the key name photos/myphoto.jpg, where photos/ is the prefix.
Here are two more examples:
If you have three objects in your bucket—logs/date1.txt,
logs/date2.txt, and logs/date3.txt—the console will show a folder
named logs. If you open the folder in the console, you will see three
objects: date1.txt, date2.txt, and date3.txt.
If you have an object named photos/2017/example.jpg, the console will
show you a folder named photos containing the folder 2017. The folder
2017 will contain the object example.jpg.
When you create a folder in Amazon S3, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders.
You can read more in the Amazon S3 user guide.
Related
Are (empty) prefixes also deleted by the s3 lifecycle management?
S3 is a glorious hash table of (key, value) pairs. The presence of '/' in the key gives the illusion of folder structure and the S3 web UI also organizes the keys in a hierarchy. So, if lifecycle management rules end up deleting all the keys with a certain prefix, then it essentially means the prefix is also deleted (basically, there is no key with such a prefix). HTH.
Short answer: yes
More detail: prefix are just 0-byte object. When you use the Amazon S3 console to create a folder, Amazon S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named photos in your bucket, the Amazon S3 console creates a 0-byte object with the key photos/. The console creates this object to support the idea of folders.How S3 supports folder idea
I have an AWS S3 bucket with several folders, subfolders, and files. I want to get a list of all subfolders of a folder, excluding files. I think I understand that the S3 key concept makes such distinctions iffy, but the AWS web gui allows users to create folders without files.
The listObject() method defined in com.amazonaws.services.s3.AmazonS3 returns an ObjectListing with a list of S3ObjectSummary for the actual files. Knowing the delimiter, it would be possible to split the keys into a folder hierarchy and filenames, but this appears complicated and error-prone.
Is there an API to get a list of folders without parsing the key property of S3ObjectSummary?
'Objects' in S3 (ie files) do not exist on S3 in actual folders - the 'Keys' for those objects are all at the same level of the hierarchy, and contain slashes which are then displayed to the user on the console as if they were in folders. To find only the 'folders' as opposed to the files, you will need to work out for each object in the bucket whether there are any other files with the same key but with a slash (/) and then more characters. So eg if you had objects with keys like:
a
b
b/c
b/c/d1
b/c/d2
you would only know c is a folder because there are other objects 'inside' it (ie with extra characters after the slash.
In aws s3 web interface we can select a specific folder then navigate to properties and select IFA... like this
This will start processing all existing data, but if you open the same properties page again its not selected. If you select again and apply it will show a processing bar again...
Its ambiguous does that folder remain IFA once selected and saved? will future uploads to that folder stored in IFA storage? If not how do we do that?
I know that there is migration rule like after 30 days move to IFA but i know upfront that my data is suitable for IFA storage...
"For all selected items" means the current objects, not the folders.
The folders do not exist in S3 in any meaningful sense yes, you can "create" a folder but all that does is create a placeholder for convenience in the console -- there are never any files actually "in" the folder -- so it is impossible to actually set any kind of properties on them. The folders that appear in the console are just a human-friendly representation of a hierarchy, created by splitting the keys on / delimiters.
In Amazon S3, buckets and objects are the primary resources, where objects are stored in buckets. Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects.
http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html
If you want objects stored for their lifetime as either STANDARD_IA or REDUCED_REDUNDANCY then you have to initially upload them that way. If you want them to transition to STANDARD_IA or GLACIER later, then you use lifecycle policies.
Note also that changing storage classes in the console like you are doing incurs the same cost as re-uploading the object, because changing storage classes is accomplished by the console invoking the S3 copy operation -- using the same key for source and destination. It's $0.01 per 1000 objects, so use it wisely on large collections of objects. Objects and their metadata are immutable, so modifying them (including storage class, which isn't technically metadata) requires replacing the object with an identical object.
I understand that s3 does not have "folder" but I will still use the term to illustrate what I am looking for.
I have this folder structure in s3:
my-bucket/folder-1/file-named-a
my-bucket/folder-2/...
my-bucket/folder-3/file-named-a
my-bucket/folder-4/...
I would like to find all folders containing "file-named-a", so folder-1 and folder-3 in above example will be returned. I only need to search the "top level" folders under my-bucket. There could be tens of thousands of folders to search. How to construct the ListObjectsRequest to do that?
Thanks,
Sam
An Amazon S3 bucket can be listed (ListBucket()) to view its contents, and this API call can be limited by a Prefix. However, it is not possible to put a wildcard within the prefix.
Therefore, you would need to retrieve the entire bucket listing, looking for these files. This would require repeated calls if there are a large number of objects.
Example: Listing Keys Using the AWS SDK for Java
I know this is a question that may have been asked before (at least in Python), but I am still struggling to get this right. I compare my local folder structure and content with what I have stored in my Amazon S3 bucket. The directories not exisiting on S3, but which are found locally, are to be created in my S3 bucket. It seems that Amazon S3 does not have the concept of a folder, but rather a folder is identified as an empty file of size 0. My question is, how can I easily create a folder in objective-c by putting an empty file (with name correspoding to the folder name) on S3 (I use ASIHTTP for my get and put events)? I want to create the directory explicitly and not implicitly by copying a new file to a non-exisiting folder. I appreciate your help on this.
It seems that Amazon S3 does not have the concept of a folder, but rather a folder is identified as an empty file of size 0
The / character is often used as a delimiter, when keys are used as pathnames. To make a folder called bar in the parent folder foo, create a key with the name /foo/bar/.
Amazon now has an AWS SDK for Objective C. The S3PutObjectRequest class has the method -initWithKey:inBucket:.