How to download information? - amazon-s3

Question

Currently, I have the following that downloads all files under an AWS (Amazon Web Services) S3 bucket.

Answer 1

To do this in boto3, you could do something like this:
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
for object_summary in bucket.objects.all():
print(object_summary.key)
print(object_summary.last_modified)
In this case boto3 will handle all pagination so you won't be limited to only the first page of results.
Is that what you are trying to do?

Answer 2

From Documentation:
http://boto3.readthedocs.io/en/latest/reference/services/s3.html?highlight=s3#S3.Client.list_objects_v2
Returns some or all (up to 1000) of the objects in a bucket.
response = client.list_objects_v2(
Bucket='string',
Delimiter='string',
EncodingType='url',
MaxKeys=123,
Prefix='string',
ContinuationToken='string',
FetchOwner=True|False,
StartAfter='string',
RequestPayer='requester'
)
MaxKeys (integer) -- Sets the maximum number of keys returned in the response. The response might contain fewer keys but will never contain more.
Do you know how many objects do you have in s3?

Answer 3

Limitation: boto3 list_object() and list_object_v2() will return maximum 1000 keys. To continue get object, you need to make use of "ContinuationToken" parameter. boto3.s3.resource.objects.all() will generate an iterator, currently there is no known limit.
However, you must know this :
Downloading from S3 to your local system from the internet WILL cost you, i.e. 0.09/GB. If you need frequently process on such files, maybe you should run your download inside EC2 or use lambda to do the post-processing.
same region S3 data to EC2 download is free. you can run test massive download proof (e.g. say 100GB+ ) without worry about the bill. You just pay EC2 instance.
Downloading performance/continuity of session from s3 to your local intranet , are subject to your local internet connectivity setup. This can be restriction in your own firewall policies, your router, ISP, etc