ExpressJS Series: Understanding the request from the client with the help of middlewares

Understanding middlewares can be really easy or difficult depending on what functionality you are attempting to implement. You might just have a huge and unnecessary bloated blocking logic in middleware, and your application response times can go for a toss. This is quite possible, not just in other languages but also in ExpressJS application. We will talk about this again when we talk about structuring our application code. In the last post, we learnt how to create and work with middlewares.

In this post, we will focus on getting a server understand and parse request’s bodies (avoiding errors that happen when the buffer is not readable). We will be doing that with already available middleware packages that solve a specific problem for your application.

For a request to be understood by the application, we need to:

• Parse the request body

Apart from this, we might also need the server to:

• Parse cookies/sessions, if needed
• Protect the application in a way that it avoids cross domain requests (CORS), Cross Site Request Forgery (CSRF), Attempt to void Cross Site Scripting (XSS), Apply some basic Content Security Policy (CSP).

We will use packages to fulfill the above one by one. I promise to cover each section above in depth later.

The URL dynamic parameters and query parameters can be parsed by ExpressJS application. We have seen this during the route creation. To recollect, “req.params.” and “req.query.” is how you access them, respectively.

Let us look at parsing the request body. When a request is made to the server it can be sent as a buffer with its character set and content-type (in Header). The character set can be anything, including ‘utf8’ which is the one used for strings/text. In such cases, trying to parsing a non-text character set to Javascript string will fail and create issues. Hence, the server has to take the buffer and use the character set/content-type to understand the contents of the request body. It is somewhat similar to deciphering languages based on language code in the Header.

For this reason, we will use an npmjs.com package (where all free NodeJS packages are hosted) called body-parser. We have used this repository before while installing ExpressJS package in our NodeJS project to start the Server. We will use body-parser package in different ways while understanding some use cases of data sent by the user.

1. How do I parse a JSON file?
2. How do I parse a text or html file?
3. How do I parse a XML file?
4. How do I know if a file is an image, like gif?
5. How do I parse if the data sent is a buffer with a different character set and content-type?
6. What if the user is posting a multi-part form/form-data?

Let us use the body-parser middleware that has a JSON body parser, Raw body parser, Text body parser, URL-encoded form body parser. We will apply this to all the routes. For working with forms, we will have to use multer (alternative formidable) and then apply that to a specific route to work with forms. Lets install body-parser to our project:

npm install body-parser -S

The way we apply body-parser to our application is exactly the same as we applied our middleware function; after require-ing our body-parser package.

const bodyParser = require(‘body-parser’);

// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }));

// parse application/json
app.use(bodyParser.json());

Let us look at the code again:

That is it. The body-parser middleware will now parse all request bodies with a JSON parser.

Is it necessary that all request bodies will be of type JSON? You may have XML, HTML, plain text. Right? To ensure we are getting this right, we can also apply a text parser to our POST function at path /. Have a look at the way we do it:

const textParser = bodyParser.text({…options…});

app.post(‘/’, textParser, myRouteMiddleware, function(req, res) {…});

Code Sample:

What if the client has submitted a form with the header : content-type: application/x-www-form-urlencoded? Well, use the bodyParser.urlencoded({…options…}). This is how a x-www-form-urlencoded form submit is sent as:

firstname=Jasmine&category=testing&photo=jasmine-logo.png

Code Sample:

We applied the urlencoded parser to the PUT route handler for /. This seems more of a repetition/redundancy since we have already applied the urlencoded parser to all the routes using the common application middleware in line 7 as well. This is quite unnecessary redundancy and we can avoid it completely. This implementation was just to demonstrate this fact.

Similarly, if we have to parse any other content-type which has a different character set other than utf8 then you can use the raw parser as in the code below (lines 32–47 and line 57):

The raw parser can be the most difficult implementation if you have a custom content-type and the clients are in the public domain (because you dont know if they are trustworthy clients). Second, the implementation of verify is not compulsory all the time, especially if you trust the clients. However, if you do not verify while using the parser and the buffer is of a different content-type then you will face errors. Be ready for that. Third, implementation of verify function for a content-type will be very specific.

For XML, you can install and use a middleware like express-xml-bodyparser which takes care of XML content-type.

You can also use the options like type (type of buffer), limit (size limit of the buffer), and verify function (verification of the buffer content to be clean and trustworthy) for other body-parser types (json, urlencoded, and text).

Run all the code above (with needed packages installed) using the command below to get a feel:

node index.js

Try playing around with the code to understand the system and usage. If you want to have a look at the detailed implementation of the body-parser you can definitely have a look at the documentation here; though we covered most of it.

We have not covered form-data parsing. We will take this up when with an working example while using multer as the form library.

To summarise, along the post we looked at:

• Request body can be sent as a buffer and hence needs to be parsed to be understood. Not all buffer character set’s are simple ‘utf8’, especially when files are sent as the content.
• Body parsers basically understand the buffer, and parse it based on the content-type and character set.
• We used body-parser as the body parsing middleware.
• We looked at ways body-parser can understand json, text (utf8), html (utf8), xml(utf8 or use packages like express-xml-parser), and handle raw buffer when the buffer content type is different.
• We also learnt that since not all content or clients are trustworthy, we can also verify the content sent to the server using the body-parser middleware.
• Since body-parser is a middleware we saw that we can also apply the parsing specifically to a route and not necessarily implement this for all the routes. This gives us flexibility to work with different content types for different routes.
• We saw how not to be redundant in implementations of middlewares.

In the next post, we will implement basic security for our server.

Applying standard security to ExpressJS Server: https://medium.com/@ganeshsurfs/expressjs-series-applying-standard-security-to-expressjs-server-16e1a00c1c4f

ExpressJS Series All Blogs: https://medium.com/@ganeshsurfs/expressjs-series-links-9e038be8d78b

Let me know how I did, and if you learnt something new. Do leave your comments, and dont forget to like the article.