Efficient way to extract files and meta data from Amazon S3? - amazon-s3

Question

Is there a more efficient way to list files from a bucket in Amazon S3 and also extract the meta data for each of those files? I'm using the AWS PHP SDK.
if ($paths = $s3->get_object_list('my-bucket')) {
foreach($paths AS $path) {
$meta = $s3->get_object_metadata('my-bucket', $path);
echo $path . ' was modified on ' . $meta['LastModified'] . '<br />';
}
}
At the moment I need to run get_object_list() to list all the files and then get_object_metadata() for each file to get its meta data.
If I have 100 files in my bucket, it makes 101 calls to get this data. It would be good if it's possible to do it in 1 call.
E.g:
if ($paths = $s3->get_object_list('my-bucket')) {
foreach($paths AS $path) {
echo $path['FileName'] . ' was modified on ' . $path['LastModified'] . '<br />';
}
}

Answer 1

I know this is a bit old, but I encountered this problem and to solve it I extended the Aws sdk to use the batch functionality for this type of problem. It makes a lot quicker to retrieve custom meta data for lots of files.
This is my code:
/**
* Name: Steves_Amazon_S3
*
* Extends the AmazonS3 class in order to create a function to
* more efficiently retrieve a list of
* files and their custom metadata using the CFBatchRequest function.
*
*
*/
class Steves_Amazon_S3 extends AmazonS3 {
public function get_object_metadata_batch($bucket, $filenames, $opt = null) {
$batch = new CFBatchRequest();
foreach ($filenames as $filename) {
$this->batch($batch)->get_object_headers($bucket, $filename); // Get content-type
}
$response = $this->batch($batch)->send();
// Fail if any requests were unsuccessful
if (!$response->areOK()) {
return false;
}
foreach ($response as $file) {
$temp = array();
$temp['name'] = (string) basename($file->header['_info']['url']);
$temp['etag'] = (string) basename($file->header['etag']);
$temp['size'] = $this->util->size_readable((integer) basename($file->header['content-length']));
$temp['size_raw'] = basename($file->header['content-length']);
$temp['last_modified'] = (string) date("jS M Y H:i:s", strtotime($file->header['last-modified']));
$temp['last_modified_raw'] = strtotime($file->header['last-modified']);
#$temp['creator_id'] = (string) $file->header['x-amz-meta-creator'];
#$temp['client_view'] = (string) $file->header['x-amz-meta-client-view'];
#$temp['user_view'] = (string) $file->header['x-amz-meta-user-view'];
$result[] = $temp;
}
return $result;
}
}

Answer 2

You need to know that list_objects function has limit. It doesn't allows to load more than 1000 objects, even if max-keys option will be set to some large number.
To fix this you need to load data several times:
private function _getBucketObjects($prefix = '', $booOneLevelOny = false)
{
$objects = array();
$lastKey = null;
do {
$args = array();
if (isset($lastKey)) {
$args['marker'] = $lastKey;
}
if (strlen($prefix)) {
$args['prefix'] = $prefix;
}
if($booOneLevelOny) {
$args['delimiter'] = '/';
}
$res = $this->_client->list_objects($this->_bucket, $args);
if (!$res->isOK()) {
return null;
}
foreach ($res->body->Contents as $object) {
$objects[] = $object;
$lastKey = (string)$object->Key;
}
$isTruncated = (string)$res->body->IsTruncated;
unset($res);
} while ($isTruncated == 'true');
return $objects;
}
As result - you'll have a full list of the objects.
What if you have some custom headers?
They will be not returned via list_objects function. In this case this will help:
foreach (array_chunk($arrObjects, 1000) as $object_set) {
$batch = new CFBatchRequest();
foreach ($object_set as $object) {
if(!$this->isFolder((string)$object->Key)) {
$this->_client->batch($batch)->get_object_headers($this->_bucket, $this->preparePath((string)$object->Key));
}
}
$response = $this->_client->batch($batch)->send();
if ($response->areOK()) {
foreach ($response as $arrHeaderInfo) {
$arrHeaders[] = $arrHeaderInfo->header;
}
}
unset($batch, $response);
}

Answer 3

I ended up using the list_objects function which pulled out the LastModified meta I required.
All in one call :)