Oct 25, 2020
Server responses so slow the user abandoned! Trace using NEL and an example in Node.js
Network Error Logging ?
If you have never heard about Network Error Logging (NEL) before and you are working on the web, you might get really excited, as I did when I read the specification about the capabilities it provides. We will not go into an introduction of this specification (in this article) but just a few words to whet your appetite.
If you are interested in the topic and want to follow along, take a look at the specification or introductory articles first.
NEL gives web service operators visibility into problems related to network reliability that the users of those services are experiencing.
The most exciting part of NEL in my eyes is the “users“ part.
The place where the reports are stored and transmitted over to your NEL collector service is client browser.
In a more systemic language, the NEL agent is the user’s browser. That makes NEL reports, the ground truth about whether problems impact your users at any point in the network interactions of an HTTP request.
An amazing capability without requiring bespoke instrumentation, specialized infrastructure or new tools to get started.
The power to monitor “abandonment”
Except for all the TCP, DNS, TLS and HTTP specific errors being reported by NEL, for more than a year now the “abandonment” error type has been available for Chromium based browsers.
“abandonment”, as not yet well described in the official documentation, covers a significant blind spot which, if I can talk in marketing terms, is as low in the funnel as possible regarding a user reaching our product.
The “abandonment” error type is generated when the user closed the page and the requirements below are met:
- Valid, not erroneous(400–500) or redirect, headers are received in the server response.
- The browser has not managed to fully read the response body from the server.
In plain English:
For a resource request, let’s take the main document as an example which is probably the most important, this error report covers cases where all the network infrastructure has done its job but now due to the server or the CDN serving the request being slow the user has left.
This slowness can most of the times be attributed to:
1. Slow Time To First Byte (ttfb)
2. Time to generate and transmit the full body response. Server-side rendering, slow database queries are a couple of things that come to mind.
The Importance
This spot is of extreme value to web engineers, user behaviour analysts and web performance engineers to say a few.
- Client side analytics have no power there as the whole response body is not yet fully transmitted, let alone analytics scripts being run.
- Server logs depending on your infrastructure at the worst case scenario have logged a 200 status code or nothing at all. When a CDN is serving your pages, you usually don’t even have access to their logs.
Common ways of monitoring are leaving this range of behaviours not easily traceable but nevertheless crucial to profits.
Utilizing the NEL abandoned error type you can now become aware of this issue and act upon it depending on your application.
Example Node.js implementation
To showcase how you could achieve the NEL reporting functionality and understand which failures are treated as abandoned, we are gonna go about implementing an example Node.js web server using a couple of helper libraries and tools.
In this example we are going to be using:
- Express.js for the web server framework.
- nodemon to avoid restarting the local server all the time.
- ngrok for easy access to public secure URL. (NEL does not work for insecure localhost endpoints)
A Simple Web Server
Let’s start by creating our simple web server project:
$ mkdir nel-abandoned && cd nel-abandoned
$ npm init -y
$ npm install express
$ touch app.jsOpen up app.js with your favorite text editor and add the required setup to start an express web server:
const express = require("express");
const app = express();
const PORT = 3000;/*
Allow express to parse the special content type
of the NEL report.
*/
app.use(express.json({ type: "application/reports+json" }));/* Home route just sending nothing back*/
app.get("/", async (req, res) => {
res.end();
});/*
NEL collector endpoint.
In a real environment, the reporting endpoint would be
in a completely different server IP, domain and CDN.
*/
app.post("/report", (req, res) => {
// Log the reports received on the terminal
console.log(JSON.stringify(req.body));
});app.listen(PORT, () => {
console.log(`Listening on ${PORT}`);
});
Now run your server and ngrok on different terminals.
$ npx nodemon app.js
// On different terminal now
$ ngrok http 3000To let the clients know that you want to use NEL and report back to a specific endpoint, you want to send to the client the Reporting API/NEL required headers.
To do that we will create a NelMiddleware that will send to every request the NEL headers we defined back to the client.
const NelMiddleware = function (req, res, next) {
res.setHeader(
"Report-To",
JSON.stringify({
group: "network-errors",
// Expire in day
max_age: 86400,
// Here use the secure URL you are gonna get from ngrok
endpoints: [{ url: "NGROK_URL/report" }],
})
); res.setHeader(
"NEL",
JSON.stringify({
report_to: "network-errors",
// Cache the policy for a day
max_age: 86400,
})
);
next();
};/* Use the middleware before registering the routes */
app.use(NelMiddleware);
If everything went well, trying the home route of your application and inspecting the DevTools network panel you would be able to see the NEL headers included in the document request.
Simulating & monitoring an “abandonment” error
To help us in the analysis and simulation of our experiment, we can use the reporting dashboard of ngrok by opening http://localhost:4040 where we get a free request traffic inspector. We would be able to inspect the reports posted to our service by the NEL agent later on.
To simulate an abandonment case as we described earlier you can try simply adding an artificial delay to the home route. This would be the case of a slow Time to First Byte.
app.get("/", async (req, res) => {
setTimeout(() => res.end(), 10000);
});Open the secure ngrok URL on the home route and before the 10 second timeout runs, close the current tab. You can do this a few times to populate more reports.
In a few minutes you will be seeing either in the ngrok inspector or the console logger of the /report endpoint, some reports coming from the browser with the error type abandoned.
Case of not fully delivered response body
The second common case that would trigger the abandonment would be to have a part of the response body being triggered slowly and the user leaving the page before the full completion.
/* Helper for artificial delay */
const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms));app.get("/", async (req, res) => {
res.setHeader("Content-Type", "text/html");
await res.write(`
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Document</title>
</head>
<body>
`); await delay(5000);
res.write(`<p>Hello World</p></body>`); await delay(5000);
res.write(`</html>`);
res.end();
});
What this function does is it delivers parts of an HTML response on the response stream but without fully completing the body after the two delay calls of 5 seconds each.
Trying out the same trick of opening the page for a couple of seconds and then closing it down would trigger more abandonment type reports.
That’s about it 🎉
Closing notes
Thanks for reaching this far into the article. At this point I want to again stress how important this capability of the NEL proposal is in my eyes. Congratulations on the team at Google that introduced initially this idea and also the folks on the Web Performance Working Group for taking care of this specification.
Just as with all things in life, there are some caveats for now that you might need to be aware of. Currently NEL is supported only in Chromium based browsers like Chrome, Edge & Opera but hopefully this is gonna increase as time passes.
Hope you found something interesting and useful with this relatively new abandonment reporting capability, and if you liked the article it would be nice to try and spread the word!
Feel free to reach out to me on any of my social media for any questions 😊
Picture by Song Kaiyue at Pexels
