As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I am creating a web service of scheduled posts to some social network.Need help dealing with file uploads under high traffic.
Process overview:
User uploads files to SomeServer (not mine).
SomeServer then responds with a JSON string.
My web app should store that JSON response.
Option 1: Save, cURL POST, delete tmp
The stupid way I made it work:
User uploads files to MyWebApp;
MyWebApp cURL's the file further to SomeServer, getting the response.
Option 2: JS magic
The smart way it could be perfect:
User uploads the file directly to SomeServer, from within an iFrame;
MyWebApp gets the response through JavaScript.
But this is(?) impossible due to the 'Same Origin Policy', isn't it?
Option 3: nginx proxying?
The better way for a production server:
User uploads files to MyWebApp;
nginx intercepts the file uploads and sends them directly to the SomeServer;
JSON response is also intercepted by nginx and processed by MyWebApp.
Does this make any sense, and what would be the nginx config for, say, /fileupload Location to proxy it to SomeServer?
I don't have a server to use to stand in for SomeServer for me to test out my suggestions, but I'll give it a shot anyway. If I'm wrong, then I guess you'll just have to use Flash (sample code from VK).
How about using an iFrame to upload the file to SomeServer, receive the JSON response, and then use postMessage to pass the JSON response from the iFrame to your main window from your site. As I understand it, that is pretty much the motivation for creating postMessage in the first place.
Overall, I'm thinking of something like this or YUI's io() module but with postMessage added to get around the same origin policy.
Or in VK's case, using their explicit iFrame support. It looks to me like you can add a method to the global VK object and then call that method from the VK origin domain using VK.callMethod(). You can use that workaround to create a function that can read the response from the hidden iFrame.
So you use VK.api('photos.getUploadServer', ...) to get the POST URL.
Then you use JS to insert that URL as the action for your FORM that you use to upload the file. Follow the example under "Uploading Files in an HTML Form" in the io() docs and in the complete function, use postMessage to post the JSON back to your parent window. See example and docs here. (If it doesn't work with io(), you can certainly make it work using the roll-your-own example code if I'm right about VK.callMethod().)
Then in response to the postMessage you can use regular AJAX to upload the JSON response back to your server.
I can see only two major approaches to this problem: server-side proxying and javascript/client-side cross-site uploading. Your approaches 1 and 3 are the same thing. It shouldn't really matter whether you POST files with means of cURL or nginx - not performance-wise anyway. So if you already implemented approach 1 from your question, I don't see any reason to switch to 3.
In regards to javascript and Same Origin Policy, it seems there are many ways to achieve your goal, but in all of these ways, either your scenario must be supported by SomeServer's developers, or you have to have some sort of access to SomeServer. Here's an approximate list of possibilities:
CORS—your domain must be allowed to access SomeServer's domain;
Changing document.domain—this requires that your page and target page are hosted on subdomains of the same domain;
Using a flash uploader (e.g. SWFUpload)—it is still required that your domain is allowed via the cross-domain policy, in case of Flash, via a crossdomain.xml in the root of SomeServer's domain;
xdcomm (e.g. EasyXDM)—requires that you can upload at least an html page to the target domain. This page can then be used as a javascript proxy for your manipulations with SomeServer's iframe.
The last one could, actually, be a real possibility for you, since you can upload files to SomeServer. But of course, it depends on how it's implemented—for example, in case there is another domain the files are served from, or if there are some security measures which won't allow you to host html files, it may not work out.
Related
This question already has answers here:
How does the 'Access-Control-Allow-Origin' header work?
(19 answers)
Closed 6 years ago.
I've tried to understand HTTP base statements, because that is not working as I expected.
E.g. I've put Access-Control-Allow-Origin as http://www.example.com, and I tried to send POST requests from http://www.example2.com and it was with error like I expected.
It says ...request has been blocked by CORS policy.
But I was wonder when looked that actually that request was done on http://www.example.com and POST action was called.
Question then, why do we need that protection?
When a web-page is loaded into a browser, its HTML, CSS, Javascript is loaded, its session is being used. Some of the many potential problems:
The remote page inside the iframe might be a page where you are logged in (like your personal email account's web-page) and a spider could silently steal important data (like the content of your emails, including access to confidential areas, like bank account-related data, personal, private data, etc.)
Confidential CSS/Javascript could be stolen from trusted users. Example: you create some very good code in Javascript and CSS and only paid users can use their benefit. However, someone sends you a link which points to a page which loads your site silently as an iframe and extracts the CSS and Javascript goodies from there. Then the stealer will sell your product with a discount and you can work on new products and on a better security policy.
Your accounts could be hacked. A page where you have an active session could be loaded inside an iframe and then a spider could wreak havoc there, including, but by far not limited to changing your username/password and excluding you from your own account.
Malicious things could be done against others in your name.
I have a task to do where I need to make calls to an external xml api to fetch data for an event calendar in the sidebar of a site. The date will be changed with JavaScript and then i need to make another call to refresh the data. Can somebody give me an idea about how to cleanly set up an action or function somewhere that I can direct an Ajax action to? It's easy to set up a widget with the correct HTML etc but where does the Ajax connect to? Ideally when the content initially loads on the page it would use the same function that the post is going to use to generate the HTML on the server side.
Any tips would be appreciated. This is an xml api...no option for json or jsonp so credentials including a token and user I'm assuming will have to go somewhere in my widget, something like a proxy function?
It sounds as if you're asking about making cross-domain AJAX requests, AKA the "same origin policy."
The same origin policy prevents document or script loaded from one origin from getting or setting properties of a document from a different origin (domain). See http://www.mozilla.org/projects/security/components/same-origin.html for a more detailed description of the policy.
See Ways to circumvent the same-origin policy for a good description of the options available to circumvent this limitation.
The short answer is that unless you have control over the domain to which you're making AJAX requests, your best bet is probably to set up a simple proxy that lives in the same domain where your AJAX is running, which will forward requests to the destination. Doing a google search on "simple AJAX proxy" will get you a host of results, including pre-built proxies in a variety of languages.
I'm trying to achieve urls in the form of http://localhost:9294/users instead of http://localhost:9294/#/users
This seems possible according to the documentation but I haven't been able to get this working for "bookmarkable" urls.
To clarify, browsing directly to http://localhost:9294/users gives a 404 "Not found: /users"
You can turn on HTML5 History support in Spine like this:
Spine.Route.setup(history: true)
By passing the history: true argument to Spine.Route.setup() that will enable the fancy URLs without hash.
The documentation for this is actually buried a bit, but it's here (second to last section): http://spinejs.com/docs/routing
EDIT:
In order to have urls that can be navigated to directly, you will have to do this "server" side. For example, with Rails, you would have to build a way to take the parameter of the url (in this case "/users"), and pass it to Spine accordingly. Here is an excerpt from the Spine docs:
However, there are some things you need to be aware of when using the
History API. Firstly, every URL you send to navigate() needs to have a
real HTML representation. Although the browser won't request the new
URL at that point, it will be requested if the page is subsequently
reloaded. In other words you can't make up arbitrary URLs, like you
can with hash fragments; every URL passed to the API needs to exist.
One way of implementing this is with server side support.
When browsers request a URL (expecting a HTML response) you first make
sure on server-side that the endpoint exists and is valid. Then you
can just serve up the main application, which will read the URL,
invoking the appropriate routes. For example, let's say your user
navigates to http://example.com/users/1. On the server-side, you check
that the URL /users/1 is valid, and that the User record with an ID of
1 exists. Then you can go ahead and just serve up the JavaScript
application.
The caveat to this approach is that it doesn't give search engine
crawlers any real content. If you want your application to be
crawl-able, you'll have to detect crawler bot requests, and serve them
a 'parallel universe of content'. That is beyond the scope of this
documentation though.
It's definitely a good bit of effort to get this working properly, but it CAN be done. It's not possible to give you a specific answer without knowing the stack you're working with.
I used the following rewrites as explained in this article.
http://www.josscrowcroft.com/2012/code/htaccess-for-html5-history-pushstate-url-routing/
I'm having trouble connecting some dots having recently learned of JSONP. Here's my understanding:
Cross-domain XmlHttpRequests for any content (including JSON) is banned, due to the same origin policy. This protects against XSRF.
You are permitted to have a script tag with a src that returns JSONP - some JSON padded inside a call to a Javascript function (say 'Foo')
You can have some implementation of 'foo' on the page that will get called when the JSONP data is returned, and you can do things with the JSON data that function is passed
Why is it OK to receive cross-domain data if it came via JSONP, but not if it came via JSON?
Is there an assumption that JSON is prone to permitting XSRF but JSONP is not? If so, is there any reason for that other than JSONP being some de-facto data format that won't ever provide data that enables XSRF? Why JSONP and not some arbitrary root tag on XML instead?
Thank you in advance for your answers, please make my brain work again after failing to figure this one out.
I understand this question to be about why the browser considers JSONP safe, not about whether it is safe (which it isn't). I will address this question step by step.
Regular ol' AJAX
To perform a regular AJAX request, the browser creates an XHR object, points it at the URL and pulls the data. The XHR object will only trust data from the same domain. This is a hard limitation. There is no getting round it in current browsers (edit - you can now use CORS).
Solution - Don't use XHR
Since XHR is subject to the same domain poilicy, we can't use XHR to do cross domain AJAX. Luckily, there are other ways to hit a remote server. We could append an image tag to the page for example. We can also append a script tag and give it a src attribute that points to the remote server. We can pull JQuery from a CDN for example and expect it to work.
How JSONP works.
When we make a JSONP request, our code dynamically appends a script tag to the page. The script tag has a source attribute which points to the remote JSONP API url, just as though you were inserting a script from a CDN.
The JSONP script returned by the server is wrapped in a function call. When the script is downloaded, the function will be executed automatically.
This is why we have to tell the JSONP the name of the callback function we want to wrap the script in. That function will be called once the script has downloaded.
Security concerns
There are some fairly big security concerns here. The script you are downloading could take control over your page and put your users at risk. JSONP is not safe for your users, it is just not blocked by the web browsers. JSONP really is a browser exploit which we are taking advantage of. Use with caution.
Used wisely, JSONP is pretty awesome though.
I don't know how the perception that JSONP is safe came up but see
JSON-P is, for that reason, seen by many as an unsafe and hacky
approach to cross-domain Ajax, and for good reason. Authors must be
diligent to only make such calls to remote web services that they
either control or implicitly trust, so as not to subject their users
to harm.
and
The most critical piece of this proposal is that browser vendors must
begin to enforce this rule for script tags that are receiving JSON-P
content, and throw errors (or at least stop processing) on any
non-conforming JSON-P content.
both quotes from http://json-p.org/ .
other links with some useful information about JSONP/security:
http://beebole.com/en/blog/general/sandbox-your-cross-domain-jsonp-to-improve-mashup-security/
Cross Domain Limitations With Ajax - JSON
JSONP Implications with true REST
all these tell 2 things - basically it is not considered "safe" but there are ideas on how to make it "safer"... though most ideas rely on standardization AND specific check logic to be built into browsers etc.
I manage a large and active forum and we're being plagued by a very serious problem. We allow users to embed remote images, much like how stackoverflow handles image (imgur) however we don't have a specific set of hosts, images can be embedded from any host with the following code:
[img]http://randomsource.org/image.png[/img]
and this works fine and dandy... except users can embed an image that require authentication, the image causes a pop-up to appear and because authentication pop-ups can be edited they put something like "please enter your [sitename] username and password here" and unfortunately our users have been falling for it.
What is the correct response to this? I have been considering the following:
Each page load has a piece of Javascript execute that checks each image on the page and its status
Have an authorised list of image hosts
Disable remote embedding completely
The problem is I've NEVER seen this happen anywhere else, yet we're plagued with it, how do we prevent this?
Its more than the password problem. You are also allowing some of your users to carry out CSRF attacks against other users. For example, a user can set up his profile image as [img]http://my-active-forum.com/some-dangerous-operation?with-some-parameters[/img].
The best solution is to -
Download the image server side and store it on the file system/database. Keep a reasonable maximum file size, otherwise the attacker can download tons of GBs of data onto your servers to hog n/w and disk resources.
Optionally, verify the file is actually an image
Serve the image using a throw-away domain or ip address. It is possible to create images that masquerade as a jar or applet; serving all files from a throwaway domain protects you
from such malicious activity.
If you cannot download the images on the server side, create a white list of allowed url patterns (not just domains) on the server side. Discard any urls that don't match this URL pattern.
You MUST NOT perform any checks in javascript. Performing checks in JS solves your immediate problems, but does not protect your from CSRF. You are still making a request to an attacker-controlled url from your users browser, and that is risky. Besides, the performance impact of that approach is prohibitive.
I think you mostly answered your own question. Personally I would have gone for a mix between option 1 and option 2: i.e. create a client-side Javascript which first checks image embed URLs against a set of white-listed hosts. For each embedded URL which is not in that list, do something along these lines, while checking that the server does not return the 401 status code.
This way there is a balance between latency (we attempt to minimize duplicate requests via the HEAD method and domain whitelists) and security.
Having said that, option 2 is the safest one, if your users can accept it.