[UTILITY POST] CORS, JSON-P, Proxy Servers and the Same-Origin Policy

tl;dr: If you need to get around the Same-Origin Policy, CORS is probably most helpful and most services use it nowadays. See example repo for other techniques here (NodeJS required): https://github.com/valgaze/same_origin

—-

Read this and you’ll know what’s going on and what to do about it

I had a friend the other day (and for the record this isn’t some phony-baloney “I had a friend ask me the other day…” crap setup, this actually, truly came up!) who was stuck on an issue related to CORS configuration. The solution was as uninteresting as the issue, but what was interesting was that this person had been using CORS for quite a while but didn’t actually know what problem it was solving.

I wanted to fix that and also organize some thoughts on CORS and detail some of the other solutions that have been used to attack the problem in the past.

If you want to follow along at home, there’s a repo with examples of everything covered in this note (you’ll need NodeJS on your machine or running in an environment where Node can be made available like a Docker container) available here: https://github.com/valgaze/same_origin

$ git clone https://github.com/valgaze/same_origin && cd same_origin 

(If you don’t care about the Same Origin Policy or the fascinating history of the XMLHttpRequest API, skip down to “METHOD I: Use a Proxy Server” below)

The Same Origin Policy

Before we dive into CORS or JSON-P or proxy servers, we need to briefly discuss web-browsers. Douglas Crockford, the inventor of the JSON standard we’ll use throughout this note, famously remarked that “The browser is the most hostile programming environment in the world.” The modern browser is an amazingly complex and very strange beast. Almost everything is up for grabs or manipulation. As Facebook and other ad-supported platforms are re-discovering with ad-blocking extensions, a modern browser even permits the end-user to modify behavior without much in the way of technical know-how. The browser is an almost perfect vector for security vulnerabilities.

If customers are going to be asked to manipulate very sensitive/valuable information like their credit card numbers, confidential health information, business secrets, etc using a browser, every precaution must be taken to make sure that browser is as secure as possible. One important component of the browser security model is a precautionary rule known as the Same-Origin Policy.

If you consult W3C you may be surprised to discover that There is no single same-origin policy”, but a person can make some reasonable generalizations about what a same-origin policy might look like. The Same Origin Policy at heart specifies that two different “origins” cannot spy on each other or exchange data, unless they both want to. For example, if you have SITE_A on origin 1 and SITE_B on origin 2, SITE_A cannot manipulate (or retrieve) anything interesting on SITE_B and SITE _B cannot do likewise to SITE_A unless they are both from the same “origin.” (ie A script in an iFrame thankfully can’t manipulate the DOM of its parent if the two sites live on different origins.)

When two pages/resources are said to have the “same origin”, it has a very specific meaning. An origin is taken to mean exactly three things together:

PROTOCOL [ex http/https]
HOSTNAME [ex example-1.com]
PORT [ex 80]

Imagine that you are authenticated into an application like your bank’s web interface. The Same Origin Policy ensures that only the application from that initial URI is able to access whatever banking data you retrieve from the server. Any Javascript loaded onto the page can naturally also access whatever data is retrieved, however, a script on a malicious site open in another tab/window on a different origin cannot access your banking data. We like that.

To be clear, the Same Origin Policy only applies to web-browsers, it’s a web-browser-specifc “thing” and not some magic, cosmic rule of the Internet. A native mobile application has no Same Origin Policy, for example, nor does a web server (a fact we will use to our advantage in a moment.)

The upshot of the Same Origin Policy is that if you want two sites to communicate in some way from a browser, they both must share the same origin (protocol/hostname/port)

But isn’t the whole point of the Internet that you can INTER-Network between various services? How do you get around this restriction?

Before we explain how, we need to talk about XMLHttpRequests

XMLHttpRequests Were a Big Deal

If you’ve ever seen a library fetch external data from a front-end interface, odds are that under the hood it was using the browser’s built-in XMLHttpRequest API.

You can thank a nearly-two decade old Microsoft initiative designed to port their Outlook email client to a web interface for the XMLHttpRequest API used today. As Alex Hopperman recounts

XMLHTTP actually began its life out of the Exchange 2000 team. I had joined Microsoft in November 1996 and moved to Redmond in the spring of 1997 working initially on some Internet Standards stuff as related to the future of Outlook […] I don’t recall exactly when we started working on Outlook Web Access in Exchange 2000. I think it was about a year after I joined the team, probably sometime in late 1998 […] we were already a milestone or two into the Exchange 2000 (or “Platinum”) project and had been carefully ignoring the issue of OWA [Outlook Web Access] mostly because the old version was such a hack. […] The basic premise of Outlook Web Access was that you could walk up to any computer that had the browser on it and just get to your email […] The [XMLHttpRequest] beta shipped and the OWA team was able to start running forward using the beta IE5,

The XMLHttpRequest API, started as an ActiveX control and refined over time, made it possible for websites to asynchronously retrieve data long after the page finished its initial load. (Probably also to Microsoft’s surprise, XMLHttpRequest also in the long tun made technically possible the various browser-based SAAS tools and companies which liberated many future customers from a previously inevitable vendor lock-in with Microsoft.) In other words, XMLHttpRequest is what really makes credible AJAX web applications (Asynchronous JavaScript and XML) possible. With AJAX, modern web applications can have a snappy native-like “good enough” interface. Before AJAX, every request meant a round-trip to the server involving a page refresh and for most of end-users (especially the kind who might be inclined to spend money online and upend every assumption ever made about commerce) that was a non-starter. AJAX helped (along with other forces) to transform the browser from a somewhat clunky and static informational retrieval system to a more “dynamic” experience where real business and work could be done.

XMLHttpRequest and the Same Origin Policy

Given their power, XMLHttpRequests are kept on a tight leash by modern browsers. By default (though over-rideable on some browsers), an XMLHttpRequest will hard fail unless its destination is the same origin (protocol, hostname, and port) of as its originating website. To repeat, XMLHttpRequests only work if the request URL is from the same origin as the code making the XMLHttpRequests. This applies not just to “incoming” requests, you also could not for example make a vanilla POST request to an endpoint if it was located on an origin different from where you made the request in the browser.

In its $ajax implementation (which wraps XMLHttpRequest), a library like jQuery will check for Cross Origin issues by comparing the protocol and host name:

In your travels you have no doubt noticed that despite the oppressive Same Origin Policy, websites still manage to somehow overcome the odds and do cross-origin things like fetch average temperature forecasts from an external weather API. Even though this is an allegedly unbreakable browser rule, websites can obviously exchange data across origins. How do they do it? Do they disable or tamper with the Same Origin Policy somehow? No — it turns out that if you can fool the browser into thinking you’re following the Same Origin Policy, everything works pretty well. There are three main ways to to pull it off.

METHOD I: Use a Proxy Server

If you’re following along at home (https://github.com/valgaze/same_origin) CD into the directory of examples and enter the following command to turn on the server:

$ npm run proxy

When you open up the browser and inspect the console you will see one failed request and one successful request to a 3rd-party (and very dorky) API:

The first failed request is an XMLHttpRequest (mercifully abstracted away by jQuery’s tidy $.ajax method) that looks like this:

Does this request violate the Same Origin Policy? You accessed the page on http://localhost:3000/ (origin: http, localhost, 3000) and the request is sent to http://www.swapi.com/api/people/1 (origin: http, swapi.co, 80), so there is no way this request satisfies the browser’s Same Origin Policy and that is why we get our Cross-Origin error. (If you want to nitpick, the error is explicitly reporting that a CORS header is not present, but for reasons we’ll find out later that fires because the XMLHttpRequest is cross-origin.)

Later on there is another request that appears to honor to the Same Origin Policy and succeeds:

The frontend is served on http://localhost:3000 and the XMLHttpRequest is going to http://localhost:3000/proxy/1 which has the same protocol, hostname, and port as the frontend so there are accordingly no obvious Cross Origin issues and we can retrieve our data.

The server itself is of course not a browser and has no problem communicating cross origin, which is why this is possible. When the user makes a request to /proxy/1 they are really making a request, proxied through our server, to swapi.co/api/people/1. As far as the browser is concerned we are just hitting /proxy/1 and all it cares about is the protocol, hostname, and port. It works!

METHOD 2: JSONP, Hackiest of hacks

JSON-P stands for “JSON with Padding”, which tells you absolutely nothing about this dirty, kludgey, hacky, and absolutely brilliant solution to the Same Origin Policy problem.

JSON-P requires the frontend and the endpoint to conspire in order to sneak around the unbreakable Same Origin Policy. Both sides need to “opt in” the JSON-P arrangement. The JSON-P interchange strategy requires three things to be true:

  1. The frontend has a callback (myCallback) in the global scope
  2. The frontend’s XMLHttpRequest URL specifies myCallback (ex. ?callback=myCallback)
  3. The server is setup to wrap all of its responses in a myCallback function call (ie it returns a Javascript file with a named function call with the data supplied as arguments, not just the data by itself)

If all those things are true, then JSON-P can get to work. The truly amazing parts of JSON-P are in its implementation details. Depending on what library you use, a JSON-P strategy means that each external request actually gets turned into a SCRIPT tag with the src property set to the request URL, not dissimilar to the following (a library will usually take care of the details like removing expired SCRIPT tags or finding a safe mount point):

var $scriptElm = document.createElement('script');
$scriptElm.type = 'text/javascript';
$scriptElm.src = "http://localhost:3000/req/1/jsonp?callback=myCallback";
document.getElementsByTagName('head')[0].appendChild($scriptElm);

If successful, the server will return a script with myCallback invoked and the response data as the arguments which the user can then consume as if that data came from the same origin. JSON-P is able to work because it very cleverly exploits the fact that SCRIPT tags do **NOT** need to follow the Same Origin Policy. (If this were not the case, consider the implications: it would be impossible to embed a Twitter box, use Google Analytics, or load a library like jQuery via CDN on a website.) The “padding” in JSON-P, therefore, refers to the fact that the server needs to wrap the response in some “padding”, namely a named function invocation.

If using the repo, enter the following command to get started:

$ npm run jsonp

If you open the console you should find that everything works quite nicely:

Hacky, creative, insane, and WORKING!

Shame on jQuery

It’s difficult to justify shaming a tool because it is too well-written, but I am convinced that jQuery is directly responsible for the mass misunderstanding and superstitions associated with JSON-P. It makes things maybe too easy. As long as the server has a JSON-P-compliant endpoint, this is really all one needed with jQuery:

(Tip: you can even get away with less than that if you specify a “success” function in the $.ajax call, just make sure the url is ?/callback=? and jQuery will create a temporary named function for you.)

jQuery hides a lot of the heavy lifting but if you inspect the network response, it is behaving exactly as expected:

Checking if function exists, then invoking with response payload
/**/
typeof myCallback === 'function' && myCallback({
"body": {
"name": "Luke Skywalker",
"height": "172",
"mass": "77",
"hair_color": "blond",
"skin_color": "fair",
"eye_color": "blue",
"birth_year": "19BBY",
"gender": "male",
"homeworld": "http://www.swapi.co/api/planets/1/",
"films": ["http://www.swapi.co/api/films/6/", "http://www.swapi.co/api/films/3/", "http://www.swapi.co/api/films/2/", "http://www.swapi.co/api/films/1/", "http://www.swapi.co/api/films/7/"],
"species": ["http://www.swapi.co/api/species/1/"],
"vehicles": ["http://www.swapi.co/api/vehicles/14/", "http://www.swapi.co/api/vehicles/30/"],
"starships": ["http://www.swapi.co/api/starships/12/", "http://www.swapi.co/api/starships/22/"],
"created": "2014-12-09T13:50:51.644000Z",
"edited": "2014-12-20T21:17:56.891000Z",
"url": "http://www.swapi.co/api/people/1/"
},
"msg": "This is JSONP wrapped in a function invocation"
});

JSON-P is largely deprecated (and it only works with GET requests) but as long as the server reliably opts into JSON-P, it can be a very robust cross-site solution.

CORS: The nearly-universal solution

CORS stands for Cross-Origin Resource Sharing and it is probably the most common mechanism to exchange data across origins in the browser. CORS is similar to JSON-P since both the frontend and the server must opt-in to the trick (ie both must have knowledge of CORS to make a successful request) but is vastly different in the actual mechanics. While proxy servers and JSON-P can work perfectly fine for quite a while, both techniques require quite a bit of configuration and both are almost certainly losing ground to the CORS standard when it comes to adoption by 3rd-party APIs.

Preflight Options: One interesting detail of CORS data exchange is that before sending the actual request payload — if certain conditions are true (it’s complicated, but think custom headers or Content-Type), the browser will first send a preflight OPTIONS request to the target endpoint. The OPTIONS request will play a call-and-response game of “Marco-Polo” with the server to determine if the request origin is permitted and if CORS is enabled. If all goes well with the preflight OPTIONS request, the request will then actually proceed and the data finally gets exchanged. A request is compliant with CORS if at least (there are many others) the following header is returned from the server:

Access-Control-Allow-Origin: * (star means the resource can be accessed by any domain in a cross-site manner, otherwise a specific origin is returned)

(See also: Access-Control-Allow-Methods, Access-Control-Allow-Headers, Access-Control-Allow-Credentials)

To see this in action, what we’re going to do is use a CORS-enabled server on localhost:3000 to serve CORS data to a frontend on a completely different origin (different port) on localhost:3001

Turn on API server

$ npm run cors_server

Turn on Frontend

$ npm run cors_frontend

If you take a look at the /cors/server.js, the express middleware to add CORS support on all requests is quite simple:

If all went as expected, when you open localhost:3001 you should see something like this:

Bottom line: The Same Origin Policy exists for good reason and is not going anywhere. There have been many different techniques to sneak around the SOP, the most common today is CORS and it seems to get the job done

Further Reading

Browser Security and the Same Origin Policy

XMLHTTPRequest, AJAX, and Microsoft’s Biggest Strategic Error

Proxy

JSONP

CORS