I'm going to deploy a web application with multiple Pyramid application servers and nginx as a load balancer.
This application will have a feature for uploading files which should be available for downloading afterwards.
Total size of uploaded files may be very big so I'd like to deploy a separate file webserver to serve these static files. (this is one reason why I don't like rsync solution proposed here).
What is the best solution to handle file upload and syncronization in this case? I was thinking about NFS or something like that, but I'm not sure it is a good way to solve the problem. I suppose there must be some best-practices here or even a tool or library for these purposes.
UPDATE:
I don't want use cloud services like Dropbox, it would be nicer to find some syncronization solution inside the network segment.
UPDATE2:
I finished with setting up NFS, for now it works perfectly.
not really a python or pyramid related question. But, you should investigate distributed file systems and CDN's both of which are for this kind of thing. gridfs is easy enough to get going with. But there are plenty of other options. Both Amazon and Google have similar services.
Related
We have multiple clients using our service.
Each client may create multiple projects.
Each client may upload multiple files to any of his projects.
Each file may have custom meta data associated.
Each client may "share" any of the projects to another client.
Each client may comment any of his or shared projects/files.
My question is about file storing in a cloud. What will best solution? I thought about Amazon S3 but maybe there are better alternatives?
You can explore Box.com solution. They are an advanced file management solution in the cloud and support fine-grained permission management as you explained above. Dropbox for Teams is also another option - The permission model is not as extensive as Box, but the sync client is very stable here. In one of my recent projects, I used box.com mainly due to their fine-grained permission controls
You can also build this on S3 (Dropbox and I guess Box too is behind the scenes built on S3). To achieve all the functionality as you mentioned, it is quite some programming work !
Folks:
I'm creating an app using Node Webkit. The purpose of this app is to display images and pdfs. The app needs to download those files from a central repository, and cache them locally. When the app runs offline, the files should still be available, and displayed.
On the face of it, this sounds like appcache is the answer - and that indeed is where I was heading when this was a pure webapp in a browser. However, now I've discovered node-webkit, and here we are.
node-webkit's GitHub wiki states:
"However, application cache is designed for browser use, for apps using node-webkit, it's less useful than the other two method, read HTML5 Application Cache if you want to use it."
But doesn't say why.
I've also researched node.js filesystem - but that seems like a whole magnitude of complexity above what I need.
Can anyone point me in a sensible direction?
Thanks.
It has to do with the nature of App Cache itself.
You specify a manifest file that lists all the static assets required for your app to run offline. You don't have any programmatic access to the cache to add and remove files via JS.
So for a node-webkit app, it'd make more sense to fetch these files and store them in the Application Support folder (Or AppData, depending on the platform). That's where the node.js part is really useful, the file IO stuff.
Can someone provide an example (or a link to one) illustrating how to sync system files (not database) between a local computer and a remote computer/server not on the same network?
Syncing files within the same pc and syncing files between pc's within the same network is straightforward and rather simple. I have those scenarios working nicely.
I need to sync files from "C:\FilesToSync" to a remote endpoint or an IP address. a WCF, HTTP, FTP, or TCP implementation is fine. Just need to learn how the sync needs to be set-up for any of those.
Thanks
this is a really good question. for 'remote' front-end developers / designers like me lol, kinda hard to choose a nicely recommended solution; in some places it's git (OS independent), in certain .NET projects, it's TortoiseSVN-VisualStudio; when i search online, tons and tons of dunno-what-to-choose like BitBucket, GitHub's Fork.
I also am looking for the right 'generic' soln for months now...
Example:
1. working on a soln on one pc, large heavy files, the xml and psd types for instance
2. uploading/synchronising with my remote hosting server for clients
3. backup server on another location, comparing/collaborating/synchronising
some ppl might say 'u know, different projects require different solns, don't cum up with that cr*p please, thanks;
in the end, I use Notepad++ for remote files and my localhost, and Google Drive only for small doc files; meanwhile, winning combo dinosaur-style soln for the laymen in this field ;) ...
usb and last'modified'date on MyComputer !
duh
I am starting to use Jungle Disk to upload files to an Amazon S3 bucket which corresponds to a Cloudfront distribution. i.e. I can access it via an http:// URL and I am using Amazon as a CDN.
The problem I am facing is that Jungle Disk doesn't set 'read' permissions on the files so when I go to the corresponding URL in a browser I get an Amazon 'AccessDenied' error. If I use a tool like BucketExplorer to set the ACL then that URL now returns a 200.
I really really like the simplicity of dragging files to a network drive. JungleDisk is the best program I've found to do this reliably without tripping over itself and getting confused. However it doesn't seem to have an option to make the files read-able.
I really don't want to have to go to a different tool (especially if i have to buy it) to just change the permissions - and this seems really slow anyway because they generally seem to traverse the whole directory structure.
JungleDisk provides some kind of 'web access' - but this is a paid feature and I'm not sure if it will work or not.
S3 doesn't appear to propagate permissions down which is a real pain.
I'm considering writing a manual tool to traverse my tree and set everything to 'read' but I'd rather not do this if this is a problem someone else has already solved.
Disclaimer: I am the developer of this tool, but I think it may answer your question.
If you are on Windows you can use CloudBerry Explorer Amazon S3 client. It supports most of the Amazon S3 and CloudFront features and It is freeware.
I use the Transmit Mac app to modify permissions on files I've already uploaded with JungleDisk. If you're looking for a more cross-platform solution, the S3Fox browser plugin for Firefox claims to be able to modify permissions on S3 files as well.
If you need a web based tool, you can use S3fm, free online Amazon S3 file manager.
It's a pure Ajax app that runs in your browser and doesn't require sharing your credentials with a 3rd party web site.
If you need a reliable cross-platform tool to handle permissions, you can have a look at CrossFTP Pro. It supports most of the Amazon S3 and CloudFront features as well.
We have a bunch of data on S3 (images) but just started reading about Mosso Files (rackspace). Sometime this month they are going to add CDN capabilities so any file you upload is part of the limelight CDN.
Anyone using this service, it's not as well documented or publicized at S3.
Yes, it's not well documented or publicized as S3. But dude it has CDN support which S3 is lack off (unless you willing to pay extra of course). Bad thing is you can't FTP into Mosso CloudFile, you will either have to upload it through web-based control panel or API. Yet, it's still cheap and worth especially with CDN.
I am using the service and it's pretty good and cost effective compare to S3.
We use it for all our client sites, from images to podcasts, and it's hand down, the best way to distribute content and make it highly available - especially at this price!
cheers