I uploaded a lot of files (about 5,800) to Amazon S3, which seemed to work perfectly well, but a few of them (about 30) had their filenames converted to lowercase.
The first time, I uploaded with Cyberduck. When I saw this problem, I deleted them all and re-uploaded with Transmit. Same result.
I see absolutely no pattern that would link the files that got their names changed, it seems very random.
Has anyone had this happen to them?
Any idea what could be going on?
Thank you!
Daniel
I let you know first that Amazon S3 object URLs are case sensitive. So when you upload file file with upper case and access that file with same URL, it was working. But after renaming objects in lower case and I hope you are trying same older URL so you may get access denied/NoSuchKey error message.
Can you try Bucket Explorer to generate the file URL for Amazon S3 object and then try to access that file?
Disclosure: I work for Bucket Explorer.
When I upload to Amazon servers, I always use Filezilla and STFP. I never had such a problem. I'd guess (and honestly, this is just a guess since I haven't used Cyberduck nor Transmit) that the utilities you're using are doing the filename changing. Try it with Filezilla and see what the result is.
Related
I need to store user uploaded files in Amazon S3. I'm new to S3, but as I got from docs, S3 requires of me to specify file upload path in PUT method.
I'm wondering if there is a way to send file to S3, and simply get link for http(s) access? I wish Amazon to handle all headache related to file/folder structure itself. For example, I just pipe from node.js file to S3, and on callback I get http link with no expiration date. And Amazon itself creates smth like /2014/12/01/.../$hash.jpg and just returns me the final link? Such use case looks to be quite common.
Is it possible? If no, could you suggest any options to simplify file storage/filesystem tree structure in S3?
Many thanks.
S3 doesnt' have folders, actually. In a normal filesystem, 2014/12/01/blah.jpg would mean you've got a 2014 folder with a folder called 12 inside it and so on, but in S3 the entire 2014/12/01/blah.jpg it the key - essentially a single long filename. You don't have to create any folders.
Here are my two issues with CloudFront at the moment:
Some users it seems to be connecting really slow to my CloudFront CDN. It just hangs. But it's fine for others.
Some users don't even load some files (e.g: stylesheets). But like above, others are fine.
I am regularly making updates to my CDN files and use a query string to tell the users browser that the file has been updated. However I have a query string across the whole website so if I update one files, all files get an updated query string. Could that be the issue?
Has anyone else had issues like this before?
Thanks for your help!
What I have decided to do is remove the use of Query Strings and simply rename the files each time they changed.
So for example I'd call a file:
style_23623276323.css
The "3623276323" part is the MD5 signature of the file.
More details can be found in this article.
Hope that helps somebody.
Wanting to use Orchard 1.7 with Media storage on S3 (as I'm deploying to AppHarbor)
So far I'm looking at the S3 Storage provider But its a bit out of date.
Has anyone done this ? is there a better way to use S3 with the new media manager?
I've got images uploading to s3, but they don't display when I click the folder.
here is the Gist of my updated S3Provider
Missing methods for create file, rename folder, get file, and Get storage path. any help on how to complete these would be appreciated.... however stepping through the debugger in VS this doesn't seem to be the root cause of my displaying images issue above.
Edit
Looks like the file is up loading to s3 but not to the database, due to the GetFile method throwing an error...
Edit 2
Added some code to the Get file method. Now that works; (gist updated) Can up load images. However the thumbnails are still not working, they just come back as empty tags ...Think this is because the media manager is using the Open get method - which is supposed to open a file so you can write a stream to it. Don't know how to achieve this with S3... any ideas welcome
As Part of the AWSSKD NuGet package version 1.5.28.3 you can access a S3FileInfo object. I've used this in my S3 Storage File and updated the S3 Storage provider.
This seem to work, need to do a bit more testing on it.
NOTE: I had to add some code on the GetFile Method to ensure the permissions where set correctly otherwise the updating of thumbnails overwrote permissions on the file.... I'm sure there is a better way to do this.
When I use s3cmd to pull down files (of not unreasonable size - less than 100 megabytes) I occasionally see this error:
WARNING: MD5 signatures do not match: computed=BLAH, received="NOT-BLAH"
Googling suggests that this may be caused by the way S3 segments files. Others have said to ignore it.
Does anybody know why this happens and what the right thing to do is?
Thank you for your time,
-- Henry
Looking into this deeper, it seems as though s3cmd is reading the wrong md5 sum from Amazon. It looks as though s3cmd is getting its sum from the ETAG field. Comparing the actual data of the object that was PUT with the object that was GET'ed the contents are identical and this error can be safely ignored.
The ETag of a file in S3 will not match the MD5 if the file was uploaded as "Multipart". When a file is marked multipart AWS will hash each part, concatenate the results and then hash that value.
If the file does not actually have multiple parts the result will be a hash of a hash with -1 added to the end. Try disabling multipart in the tool you use to upload files to S3. For s3cmd, the option is --disable-multipart.
ETags with a '-' in them are expected, if the file was uploaded using the S3 Multipart Upload feature (typically used for files >15MB or files read from stdin). s3cmd 1.5.2 knows this and ignores such ETags. If your s3cmd is older than 1.5.2, please upgrade.
This is a bigger problem is you are using s3cmd sync, because it causes it to re-download previously-synced files. To solve this, add the --no-check-md5 option, which causes s3cmd to only check file sizes to determine changed files (this is good for my purposes, but probably not for everyone, depending on the application).
I saw reports about an hour ago that S3 is currently having exactly this problem, e.g. this tweet:
RT #drags: #ylastic S3 returning incorrect md5s to s3cmd as well. Never seen an md5 with a '-' in it, until AWS. #AWS #S3
Though the AWS Status Page reports no issue, I expect this is a transient problem. Try again soon :-)
We're using Amazon S3 for file storage and recently found out that we need to keep some sort of directory structure. Since S3 doesn't allow that, we know we can name the files according to their structure for storage. For example...
abc/123/draft.doc
What I want to know is if I want to provide a public link to this particular file is there anyway that the file can simply be draft.doc instead of abc/123/draft.doc ?
I feel stupid. After some more investigation I realized that by creating a GET url to the resource, I get exactly what I need.