Open in app

Sign in

Write

Sign in

Adventures in Shibboleth and Nginx (Part 2 of 2)

The technicalities of part 1: how to build an Nginx + Shibboleth login system atop a RESTful API.

Chris Hammond
UCL API
20 min readJan 31, 2017

--

Welcome back! I’m going to assume that if you’re here now, you have already read Part 1. If not then do take a look if you’re interested in the higher level strucuture of our enghub.io room booking system, and how the following should look, as this article follows directly on as a guide.

The techniques I’ll talk about were learned from (and poured into) Roomie McRoomface, the code for which is available on our TechSoc Github.

Furthermore, all the configuration files I have described below are available in a repository on my Github account so I recommend you copy and paste from there rather than from the page as Medium messes up formatting pretty badly. This will also ensure you get the latest and most up to date versions of the files.

Requirements

This guide will allow you to build an authentication system with the following behaviour:

  • Nginx web server on ports 80 and 443 handling all of your traffic;
  • Shibboleth Service Provider (SP) backend configured to speak to a pre-configured Shibboleth Identity Provider (IdP);
  • Some sort of Django / Node.js / Ruby backend that will host your code
    (we used Django but, since I’m super nice, I’ll point out where something may differ between backends);
  • A live stream system to notify an independant front-end, perhaps written in React like us, when authentication has succeeded on the backend;
  • Plenty of awesomeness :)

I will make the following assumptions about your setup (so please adjust accordingly!):

  • Operating System: Ubuntu Server 64-bit, latest available. At the time of writing the two latest supported versions are 16.10 and 16.04 LTS.
  • Shibboleth state: you have a working Identity Provider (IdP). This guide will not teach you how to set an IdP up. You should also have a working knowledge of how to set up a Shibboleth Service Provider (SP). If you are not an expert in this, do not worry, as I’ll cover in this guide the basics of setting this up. Again, however, this will be all from the perspective of the SP and will not teach you how to set your SP up in IdP; your system administrator should be able to do this for you.
    (In other words, we set our SP up and beg and plead UCL to accept our Metadata then, after some black voodoo magic, our SP talks to IdP. We’re not sure how they do it, so I won’t try and teach you how they do it, either. Sorry!)
  • You should have full control over the server you are working on (e.g. a root shell is a prerequisite). I personally recommend spinning up a clean Azure or AWS instance for Shib work because you’ll need to install loads of things and sometimes it’s far easier just to start over cleanly.
  • You have a way of setting up SSL. We use LetsEncrypt (because it’s free, open source and awesome!) but other methods are supported.
  • You do not already have a web server installed. If you do, remove it, as we’ll be compiling Nginx from source.
  • Your web app will expect to receive authentication at some callback endpoint. Don’t worry if you haven’t built this yet, as this guide should give you plenty of hints about how to create this.

Main Installation Guide: Let’s do it!

Firstly, we need to install some packages. You may wish to tweak this setup to suit which Nginx compilation options you plan to use.

Once installed, we need to create some folders to ensure that all the services will start properly.

Assuming you managed to install Supervisor from the package repo as described above, you should be able to drop the following configuration file into /etc/supervisor/conf.d with any name you like, so long as it ends in .conf; we used shib.conf

Supervisor is an awesome Python application that will ensure that the Shibboleth Authorizer and Responder services are set up with useful Unix sockets that Nginx will later be able to talk to. It will also allow you to start, stop and restart them from the command line.

Grab it from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Supervisor/shib.conf

At this point, you should install your Shibboleth SP configuration to /etc/shibboleth. The main files you’ll have to deal with are the following XML files:

  • attribute-map.xml
    This tells Shibboleth which attributes (bits of information) it can expect to receive from IdP, and what each bit of information is called. This should be provided to you by your administrator, and you should overwrite Shibboleth’s version with yours.
  • attribute-policy.xml
    This maps certain data points to user groups. Again, your administrator should provide this to you and you should overwrite Shibboleth’s version with your own. We actually use the default version of this file at UCL, so it might be that your administrator will suggest that you do this too.
  • idp-ucl-metadata.xml (or similar)
    In our setup, we have been provided with information by UCL on which certificates our Shibboleth SP should expect to receive data signed with. It lists our the Development, UAT (User Acceptance Testing) and Production IdPs, along with their relevant IDs. You should receive a similar XML file from your administrator that you can drop into this folder (and reference later).
  • protocols.xml
    This sets up which authentication protocols (such as SAML versions) your servers should use. We use the out-of-box version of this file, and you should be able to as well, unless your administrator provides you with an alternative.
  • security-policy.xml
    Like protocols.xml, the defaults are okay unless your admin says so.
  • shibboleth2.xml
    This is the main XML file that I will go into a lot more detail on (as we’ll need to match some values up between here and the Nginx configuration).

In addition you’ll need a certificate and private key. These are often known as sp-cert.pem and sp-key.pem respectively, and should if possible be in the Shibboleth configuration directory. They do not need to be from an official Certificate Authority and you can easily generate them yourself using the shib-keygen tool; just make sure that once you have generated these keys you do not change them, as the IdP will stop being able to talk to your SP.

shibboleth2.xml

Yes, it deserves its own section.

So, as mentioned, shibboleth2.xml is the main Shibboleth configuration file. Rather than teach you how to create a shibboleth2.xml file (which the Shibboleth Wiki is actually intended to do!) I have pasted our version of this file below, and highlighted in bold which sections need changing for your setup. Each bolded section is numbered (hey, look at me being organised!) so you can look at its explanation.

Grab a much neater version of this from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Shibboleth/shibboleth2.xml

Bolded Section 1: The Request Mapper

The request mapper is basically an instruction to Shibboleth to respond to only certain requests to certain endpoints. Change enghub.io to whatever your domain name is (as we got there first with that domain!), change the path name to something more inventive than secure and then set the long value to whichever endpoint you would expect Shibboleth to redirect to. This does not have to exist yet.

Note that you can change and update your request mapper however much you like even after going live (e.g. if you add a new callback or update your current one). Just restart shibd any time you do.

Bolded Section 2: Your Entity ID

This is a URI that identifies your Service Provider. It does not have to resolve to anything, and in fact it probably shouldn’t. The recommended default is to use https://yourdomainhere.tld/shibboleth as shown in the example.

Bolded Section 3: IdP’s Entity ID

This is the entity ID (like section 2) of the Identity Provider you would like to connect to. Note again that this doesn’t have to resolve; that is up to the file referenced in section 4. In this case we are connecting to the UAT IdP at UCL, but the live version of our code uses the production IdP.

Bolded Section 4: Metadata Provider

This connects up an XML file that describes loads more detail about IdP. It maps Entity IDs to actual URLs, and also provides public keys to match against. You should change idp-ucl-metadata.xml to the filename of your institution’s XML file.

Bolded Section 5: Credential Resolver

This is where you enter the paths to sp-cert.pem and sp-key.pem, which were mentioned above. They uniquely identify and authorise your SP, so keep this private key private!

Moving Swiftly On

Now that we have dealt with the abomination that is the configuration of Shibboleth, the next stage is to download, compile and customise Nginx. This will involve integrating a few plugins which I will explain the purposes of.

The three modules we have added are:

  • Nginx HTTP Push Stream Module
    This will allow us to push those log in notifications to our client from the server.
  • Nginx HTTP Shibboleth Module
    This allows Nginx to integrate with the Shibboleth Authorizer and Responder.
  • Nginx HTTP Clear Headers Module
    This is recommended by the creators of the Shibboleth module, as it helps to avoid header manipulation attacks that could compromise the security of your application.

Once Nginx is installed, the final stage is to configure it to speak to Shibboleth. Head to the /usr/local/nginx/conf folder and wipe out the contents. Paste the following files in:

fastcgi.conf

Grab it from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Nginx/fastcgi.conf

fastcgi_params

Grab it from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Nginx/fastcgi_params

nginx.conf

I have highlighted and annotated in bold what needs to be changed.

Grab a much neater and better annotated version from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Nginx/nginx.conf

Note above that I separated out the callback endpoint from the general API. This is for efficiency purposes, as Shibboleth’s Authorizer backend should only need calling to pass authentication details from IdP to the application. For all other endpoints there is no need to touch Shibboleth, so it saves a bunch of background processing for every other endpoint.

shib_fastcgi_params

This file actually comes bundled with the Shib Nginx plugin but I was getting crashes in Nginx without tweaking it, so the version I have that works is below. Your mileage may vary. Again, I have bolded important parts of this.

Grab it from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Nginx/shib_fastcgi_params

If there are other attributes defined in Shibboleth that you would like to gain access to, you should set them up in this file, too. Follow this pattern:

shib_clear_headers

The final main configuration file is for the Clear Headers module. This is tweaked from the original provided by the Shibboleth Nginx module.

Grab it from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Nginx/shib_clear_headers

Again, this should be updated to support your configuration. You should add new attribute names to the second list after OrganisationName such that they match up with the shib_fastcgi_params file above.

Coming Online!

Now that most things are configured, we can bring Shibboleth and Nginx online.

If all is well then those three commands should work well. The only stage left for configuration is to ensure that Nginx runs at boot. Even though it’s possible to run the command directly as shown above, it’s possibly easier to just create an Upstart script. There are instructions available on the Nginx website.

Time to Code

Now that everything is configured, I will explain how we programmatically approached the problem. To recap from the previous article, we wanted to be able to handle authentication away from the app itself and not redirect the user away from the application if possible. Instead, we open the Shib login window in a new tab, redirect that tab back to a callback URL then that endpoint creates the user account internally (if necessary), sets up the session and sends an API key to the frontend. A more formal workflow for this, from the perspective of the frontend, is as follows:

  1. Request a login URL from the backend containing a randomly generated login token. This token can be any length you like, but I recommend you stick to letters and numbers. I also recommend it’s pretty long because that token will be the name of the channel on which the API token will be sent on; therefore, we do not want the ID to be easily guessable. Our tokens are “shib” followed by sixty random characters so that it is not feasible to try and guess them.
  2. Start listening on a push channel with the ID set to the Shibboleth login token generated above.
  3. Open up the Shibboleth login URL sent in Step #1 in a new window/tab.
  4. Once the user logs in they’ll be sent to the callback URL which will take the Shib data, process it (e.g. create database entries if necessary) and generate an API token.
  5. The API token should then be sent to the frontend via the push stream chaannel associated with the Shib token.
  6. The frontend should then close the window/tab and continue the login process with the API token. This token should provide access to all the user data the callback obtained from IdP.

I realise this is quite convoluted (but it works surprisingly well) so I will explain each step with some code examples.

Getting a Login URL and Shibboleth Token

Shibboleth Login URLs should always contain at least one GET parameter: the target. This is the callback URL, and can be set to anything you, as the programmer, decide. In this case, we embed the aforementioned randomly generated shib token as a GET parameter to the callback, which is then in turn a URL Encoded target parameter. This looks something like this:

The URL Encoding here doesn’t help explain what’s going on, so here’s a version without that:

What we’ve done here is generated a token with the ID shibabcdefghijklmnopqrstuvwxyz and told the login URL to append that token to the callback URL as a parameter. That way, once login has succeeded, the callback will know which client frontend was trying to login and hence know where to send the API key to.

This generated Login URL should be sent via a JSON API request from the frontend.

Listening on the Push Channel

In order to detect when login has completed in the other tab, the client should listen on the channel specified in the Shibboleth response. The code below should help you get started with this, but note that it requires the JavaScript Client for the Push Stream Library, as well as jQuery. The docs for this library are available here.

Grab it from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Stream-Client-Sample/stream-client-sample.js

This is the point where I make a strong recommendation for your backend (which I’ll talk in more detail about later): you should base-64 encode the JSON data you send from your backend to your frontend. The reason for this is that the push stream module itself sends JSON data to the client, and will try and slot your JSON data into a string object called text. As soon as your send any quotation marks of your own you’ll trip up the JSON response making it invalid, so base-64 encoding the data makes this happen smoothly. The reason we used window.atob above is that the code is programmed to expect base-64 within the JSON.

To see how this is implemented in React, take a look at our Roomie McRoomface frontend code.

Getting the User to Login

Just open up a popup window using JavaScript to the Shibboleth Login URL your backend sends the frontend via an API call. Nice and easy!

The Callback Endpoint (Backend)

This is the final complex part of this setup. At this point I have covered how the frontend will present a login page to the user and how it’ll listen for a response, but now we need to create the callback function itself. The code we used for this in Django is as follows; I’ll bold the important parts and explain what’s going on. The full code for this file is available on our GitHub.

Medium is terrible with tabs and spacing, so grab a properly clean and indented version from: https://github.com/ChristopherHammond13/nginx-shibboleth-guide/blob/master/Django-Backend-Sample/backend-sample-view.py

Just a little aside here: an EPPN is known internally in Shibboleth as an eduPerson-principleName, or in other words represents a unique identifier for a user in the IdP. At UCL the EPPN is our unique_seven_letter_username@ucl.ac.uk, but this could be totally different in your institution.

All Done!

Hopefully you found this guide helpful and/or interesting, and do let me know if you need anything else clarifying.

-Chris

Nginx
Shibboleth
Ucl Api
Django
React

--

--

Chris Hammond
UCL API

Written by Chris Hammond

132 Followers

Computer Science student at UCL. Lunatic by day, developer all night. Outspoken, but all views are my own. InfoSec is fun. No Node.js on desktop please thanks.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams