I am trying to build a large file uploader. Currently I am using swfupload and nginx + rails and I am able to upload a file as large as 1 GB before running into problems. After 1 GB, depending on OS, swfuploader starts locking up or just starts throwing 500 errors.
I noticed that Vimeo offers 2GB uploads using a desktop uploader. Flickr has something similar too (and I think they support pausing the uploads). What protocol can be used for such a desktop uploader? Can FTP be used for this?
Thanks
Prateek
Na, I would not use FTP for this, for the following reasons:
FTP is evil and should die (active/passive connections, unencrypted credentials)
Doing proper handling of security would be complex. You cannot use a single account for everything (as the credentials will need to be embedded in your client-side software), so you would need an FTP server which can authenticate against your web application's user database (possible, but longwinded)
You would need to write some integration between your app and your FTP server
You could handle this using a custom simple protocol. I don't know if doing it over HTTP would be efficient, but if it was, you would just need to POST parts of your file and handle it server-side. Just make sure you allow partial uploading. It should be pretty simple to implement.
Related
I am creating a mac application which encrypts a file. Now when the user is ready to open the file I have to decrypt it and pass it along to a application, let's say preview in this case.
What is the best approach to do this?
Should I decrypt the file to a location and send that location to preview? Is that's the best approach can I do any file permissions to other apps or process do not access this file?
If you have to put a cleartext file on the file system to allow another program to read it then it may be impossible to make this 100% secure. My preference would be to avoid that if security is important. Possible alternatives are:
Use an encrypted interchange format that the other tool accepts. For example, Preview can read encrypted PDFs. You can use PDF Kit or the underlying Quartz 2D library to write encrypted PDFs. Note that the default encryption is 40-bit; you would probably want to increase this with kCGPDFContextEncryptionKeyLength.
Serve the data via localhost to the other program, e.g. a browser. You could embed a loopback web server into your application and present your data as a web page. You should disable browser caching (to keep the browser from writing cleartext) and you will need to add some form of authentication (to keep an attacker from querying for the data).
If you must put cleartext on the file system, besides restricting file permissions you can unlink the file once it has been opened by the other program. This will prevent normal methods of accessing the file and will delete it when the other program closes it. This does not, however, protect against file system access before the file is unlinked or against attacks that bypass the file system.
I want to read the content of a text file (serves as a key) stored inside a USB mass storage automatically when the user is authenticated by his matching username and password for that website. It's like the textfile(key) is the extended authentication.
I think this needs to can be done by a (1) native program? or an (2) applet? What do i need to study? Can someone give me an overview for the process to make this possible?
Quite good in web tech but not with native app.
You cannot access USB mass storage devices through chrome.usb as they are claimed and handled by the host operating system.
Instead you could use the chrome.fileSystem API, but the user will need to select the file. Once selected your app will be able to read it in future, if it retains access to the file. See the API documentation for more details.
If you want this only for Internet Explorer, You can create an Active X. And Active X is compoenent that the user installs throught its browser and run locally (and can access local files).
Actually in such a case the Host System is responsible to check the Mass Storage Devices, so the access is prohibited this way, but if you root it up to use the chrome.fileSystem.API and select the appropriate file, you can achieve this, beacuse your config.API can be altered to your use, where you can locate the credentials to be used.(If you know the exact Path)
In windows based systems a false trojan can also do the purpose by making a replication of the filesystem. Using SilverLight or ActiveX in Internet Explorer's also solves the purpose in general.
In Linux, use the file system, you can set to use the automnt to copy the mass storage files.
Why not try building a .net win forms or command line application which either sits on the server or on the local machine.
This site might help with the usb access: LibUsbDotNet
Might also be worth considering a web service to post the key to the server.
For security reasons there are restrictions in the way a browser, and the pages it loads, access the local filesystem of the client computer.
Is it safe to assume you only require this to work on a specific browser? As Ben said, please share more details about your requirement for a more comprehensive solution
I'm going to deploy a web application with multiple Pyramid application servers and nginx as a load balancer.
This application will have a feature for uploading files which should be available for downloading afterwards.
Total size of uploaded files may be very big so I'd like to deploy a separate file webserver to serve these static files. (this is one reason why I don't like rsync solution proposed here).
What is the best solution to handle file upload and syncronization in this case? I was thinking about NFS or something like that, but I'm not sure it is a good way to solve the problem. I suppose there must be some best-practices here or even a tool or library for these purposes.
UPDATE:
I don't want use cloud services like Dropbox, it would be nicer to find some syncronization solution inside the network segment.
UPDATE2:
I finished with setting up NFS, for now it works perfectly.
not really a python or pyramid related question. But, you should investigate distributed file systems and CDN's both of which are for this kind of thing. gridfs is easy enough to get going with. But there are plenty of other options. Both Amazon and Google have similar services.
Is it possible for a website to automatically find a folder on usb stick and upload all the files in it to the web server by clicking only one button?
The problem is that I don't know how to make upload form automatically detect usb stick as the drive name(ie. G:, F:, etc) may vary from computer to computer, so hard coding path is not possible.
Ps. I'm using yii framework for site development, but can add a new page that will handle this in any other language as the client really wants this feature.
Web sites are not allowed to set default files to upload (it's a major security risk!). Also, web sites cannot scan the hard drive/enumerate what file systems exist on a system, again, for security purposes.
It might be possibly to do this with Flash/Silverlight/Java. Java seems the most likely to allow a web developer to do this (Java plugin seems to be quite willing to give out every permission under the Sun).
Short answer: No.
Long answer: Allowing automatic uploads in web browsers would be a huge security hole so the browsers intentionally prevent it. Even if you manage to find a hole that permits it, the browser makers will break it as soon as they find out.
However, if you have an environment where an actual separate program can be installed on the end user's computer you could easily write a program to do automated uploads of specified directories when launched.
I know you use the C based networking API to do FTP communication but I'd prefer to use something a little higher level. I've seen a few Objective-C based wrappers but I'm not sure what to use. I don't need that complex of FTP interaction. Its just the typical create/delete dirs, upload/download files... What do you recommend?
Edit:
Here is one that looks promising but I can't get it to compile for the iPhone SDK
The ConnectionKit
This may help, but you may reconsider design for reasons stated by bbum:
http://code.google.com/p/s7ftprequest/
The reason why you can't find much in the way of useful FTP client software is because FTP isn't used much any more and is generally actively discouraged from use.
Without great care, it is quite easy to create big old security holes when using FTP (when I ran a consulting company, the 3 times we had infected machines were all because of FTP server security holes or exploits -- one time, the damned HP copier's FTP server was the attack vector!).
FTP is also inefficient unless carefully configured.
I would encourage you to use an HTTP based protocol. WebDAV allows you to do basically anything FTP can do, but does so over an HTTP channel. Thus, it'll work through proxies and the like. Heck -- HTTP has become so ubiquitous that pretty much everything works with HTTP.
And, of course, there are plenty of good HTTP client APIs built for Objective-C.
Obviously, if FTP is a requirement for your project, this answer won't help you much....
After not finding anything that works well I decided I'd go ahead and follow Apple's tutorial on how to do FTP. It sure is a PITA but at least it does work. I'm defiantly going to support WebDAV in the first revision my app, and eventually perhaps some other transfer methods later on. I think I'm going to consider releasing this open source after I get FTP & WebDAV working good, since there is no reason why you should have to do this much low level work to do such a basic and ambiguous task as FTP these days.
I've implemented FTP file download and upload, directory create and directory list download through the regular FTP possibilities in the iPhone SDK. Note: you'd be passing the login name and password as part of every FTP request unsecured. Apparently no apparent connection to the FTP server is maintained at the app level, like with a real FTP client app, that I haven't been able to find for iPhone yet. If you're interested in the source code please let me know through e-mail.
GoldRaccoon isn't mentioned and can be found on GitHub. I use that library and it works very well (besides it didn't support FTP rename)