Upload and Image Compression in NodeJS with Multer and Sharp
While working on a university’s project — one that i’m the only developer responsible for the backend and the database; almost everyone is frontend! — i searched for improvements i could apply to it, and one of them was image compression.
Users must be able to send photos whose size is like 2mb, but the place that is going to store the photos — like S3, Redis or your own pc, which is the one adopted by this article — is not supposed to save their photos with the same size and quality. This would have significantly impact on the store capacity of it: A difference of 300kb per image doesn’t seem too much at first-sight, but considering a large application with millions of users that can have more than one image associated to them, well, this changes.
Before diving into the tutorial on how to “integrate” them — it’s actually just using them separately but in cooperation — , let’s address two topics, one for people that don’t know for sure what are these libraries and another for curious: What are Multer and Sharp? and How i Integrated them? But well, feel free to jump these topics and get into where you’re interested.
What are Multer and Sharp?
Multer is a middleware for Nodejs that makes handling multipart/form-data, which is used for uploading files, a breeze. Is it necessary in your node project to receive files from a client, parse, validate and save them? Then Multer is a good choice. It’d be better though to not make it work alone when handling specifically images, and that’s when Sharp comes in handy.
Sharp is a Nodejs module for handling images. Its typical use case is to convert large images into smaller ones, accepting common formats like JPEG and PNG.
How i Integrated them?
The main resources provided by the application i said i’m working on are images. I was already using Multer to get the images from the client, but i saw the long-term necessity to not only get them: to get them compressed. From that moment i decided to use Sharp.
The main problem i came across on trying to “integrate” them — as i said earlier, there’s no way to integrate them like es-lint and prettier, but we can make them work harmoniously to achieve our desired goal: handle images sent by the client — , the main problem, beside some required bugs like forgetting about importing stuff from dependencies before using them, was the following:
I realized, after searching for solutions to the error thrown by the library ‘Cannot use same file for input and output’, that i was trying to rewrite an image already saved at the server file system — it’s the beginning of the project, so no need to store in another place:
await sharp(req.file.path).jpeg({ quality: 20 }).toFile(req.file.path);At the ExpressJS code above, req.file.path would hold the path of an image already saved by Multer, and i was trying to write an image with Sharp at the same path as the already-saved image — also known as rewriting.
That wasn’t just bad because is not allowed by Sharp, but mainly for the reason that two operations involving the file system, which tend to consume more resources than simple text uploading, would be done — writing a file and rewriting it — when only one instead would be enough: write a file provided by the client but compressed.
Subsequently i noticed something else: i was using Multer for both parsing and saving. And after searching a bit more about what both NodeJs libraries can do, the conclusion i reached was that i needed Multer for image parsing and validation, but the saving part would be from now on another responsibility — besides compressing — to Sharp.
So i “copiloted”; i presented my multerConfig.js and uploadPhoto middleware to Copilot and when asked exactly this — ignore the grammar and other mistakes, i don’t care so much about avoiding them when interacting with him:
“consider this file above and the below to create a solution where multer does not create the file saved at the server, but only sharp compressed saves it”
He almost gave me everything everything i needed, just some changes were necessary.
Tutorial Time!
If any doubt come up while following the instructions, don’t avoid asking them in the comments!
Installing Packages
First of all, you’ve got to create some empty folder in your file system, access it in VSCode — i’m gonna pretend you’re using it as IDE — , press ctrl+shift+c , a shortcut for opening an external terminal with the working directory as the root of your project (when the integrated terminal is not focused), and type ‘npm init -y’. This will create a package.json file for your project. After doing that, you need to install the dependencies needed using this command — the versions specified for each dependency below are the ones i’ve used in my project; if you want to use the LTS, be aware that in long-term perspective, depending on when you’re reading this, some features might not work as expected:
npm install express@^4.19.2 multer@^1.4.5-lts.1 sharp@^0.33.3
npm install nodemon@^3.1.0 sucrase@^3.35.0 -DNow you’re going to open the package.json file and add to “scripts” key the one script command it’ll be necessary to our project — used for running it on development; i’m not include another for production — : “dev”: “nodemon ./src/index.js”. For those who don’t know, Nodemon is used in development to start a NodeJs server and watch its files for changes; after any change, it automatically refreshes it.
Files Tree
We already have everything we need to build-up the project. Before anything else, below it’s the file structure you’ll have at the end of the tutorial. By that, you can already create all directories and files since now:
├── nodemon.json
├── package-lock.json
├── package.json
├── src
│ ├── app.js
│ ├── index.js
│ ├── multerConfig.js
│ ├── public
│ │ └── images
│ ├── routes.js
│ └── uploadImageMiddleware.js
└── sucraserc.jsonThe Way the Application Runs
The nodemon.json provides a way for executing Nodemon aside with Sucrase, a transpiler such as Babel which is used for fast development builds. The main reason it exists, aside with the sucraserc.json, is just for converting the ES6 modules syntax to CommonJS before running the application.
PS: i noticed that in our case only adding the property “type”: “module” in package.json would be enough, but frequently not all packages used in our projects support ES6 modules. Well, generally you don’t want to use CommonJS syntax in all the project due to a few or just one package, so one approach to solve this issue is using Sucrase, and by force of habit i’ve chosen it.
// nodemon.json
{
"execMap": {
"js": "node -r sucrase/register"
}
}// sucraserc.json
{
"transforms": ["imports"]
}About the Entry and Static Files
Now let’s define the two main files in order to make our Express application to work: one which defines it by applying the high-level middlewares to be used by the application— the ones used directly by the Express instance. The other file just runs it.
// app.js
import path from "path";
import express from "express";
import routes from "./routes";
const app = express();
app.use(express.urlencoded({ extended: true }));
app.use(express.json());
app.use(
"/images/",
express.static(path.resolve(__dirname, "public", "images"))
);
app.use(routes);
export default app;What really is important to emphasize in this file is about the following instruction:
app.use(
"/images/",
express.static(path.resolve(__dirname, "public", "images"))
);Here we are defining how the application deals with static files — project files to be accessible publicly by end users: all files located in the path returned by path.resolve are accessible through /images/ path in the URL. That is, if we have an image 2190381237.png in src/public/images folder, we can access it, in our case, by http://localhost:3000/images/2190381237.png
Ok, that was basically the only in-deep explanation needed considering the two files: there’s no need to say anything but “it runs our application in a certain port” about the code below, for the sake of this tutorial.
// index.json
import app from "./app";
const PORT = 3000;
app.listen(PORT, () => {
console.log(`Server is running on port ${PORT}`);
});Our Multer Configuration
The first of the three files involving the libraries — that are also the last we’re going to see here in this article— holds the multer configuration, which is just an object i decided to separate into a single file.
// multerConfig.js
import multer, { MulterError } from "multer";
export default {
fileFilter: (req, file, cb) => {
if (file.mimetype !== "image/png" && file.mimetype !== "image/jpeg") {
return cb(new MulterError("LIMIT_INVALID_TYPE"));
}
return cb(null, true);
},
limits: {
fileSize: 1024 * 1024 * 2,
},
storage: multer.memoryStorage(),
};This object, as you’ll see right below, is used in Multer middleware creation. In this configuration — check out the official documentation if you need anything else — , we are defining two constraints to the file handled by it: it needs to be an image of type PNG or JPEG, and also have less than 2MB of storage size.
We’re also defining the way Multer will storage the uploaded files: multer.memoryStorage() make them be stored in the memory of our application as a Buffer object. The file data is kept in memory until the file upload is complete; until the buffered file gets compressed and transformed into an image again.
Upload Handler Middleware
In order to handle the file upload logic — processing, validating, compressing and saving — , a middleware is created. It combines Multer and Sharp together, and finally you’ll see with your owns eyes what this “integrate” mean.
// uploadImageMiddleware.js
import path from "path";
import multer from "multer";
import multerConfig from "./multerConfig";
import sharp from "sharp";
export default (req, res, next) => {
const upload = multer(multerConfig).single("image");
upload(req, res, async (err) => {
if (err) {
try {
switch (err.code) {
case "LIMIT_INVALID_TYPE":
throw new Error("Invalid file type! Only PNG and JPEG are allowed");
case "LIMIT_FILE_SIZE":
throw new Error("File size is too large! Max size is 2MB");
default:
throw new Error("Something went wrong!");
}
} catch (err) {
res.status(400).json({ message: err.message });
return;
}
}
try {
const filename = `${Date.now()}${path.extname(req.file.originalname)}`;
const saveTo = path.resolve(__dirname, "public", "images");
const filePath = path.join(saveTo, filename);
await sharp(req.file.buffer)
.resize({ width: 300, height: 300 })
.jpeg({ quality: 30 })
.toFile(filePath);
req.file.filename = filename;
next();
} catch (err) {
res.status(400).json({ message: err.message });
return;
}
});
};Ok, this code is the hugest we’ve seen until now and we need to break it down in parts.
- Multer Upload Middleware
const upload = multer(multerConfig).single("image");Here we create the Multer upload middleware and specify two things: the configuration for it and that only one file from the form-data sent by the client should be handled, and this file is available as a value associated to the key “image”.
2. Multer’s Error Handling
if (err) {
try {
switch (err.code) {
case "LIMIT_INVALID_TYPE":
throw new Error("Invalid file type! Only PNG and JPEG are allowed");
case "LIMIT_FILE_SIZE":
throw new Error("File size is too large! Max size is 2MB");
default:
throw new Error("Something went wrong!");
}
} catch (err) {
res.status(400).json({ message: err.message });
return;
}
}Inside of Multer execution, we check if any error has been received by the middleware in its creation. The reasons for Multer errors can vary, and some oh them can be related to the constraints we mentioned before. If there’s any, we thrown an error to be catched and return a response to the client providing information about it.
This validation is extremely poor and just it has been made this way for the sake of this article: it does not give the client much context about the error, such as where it happened and what could be done to solve it — the client we’re talking about could be an Axios client used by the frontend part of the application, and the frontend is mantained by a developer: you need to think about that.
3. Compressing and Saving
try {
const filename = `${Date.now()}${path.extname(req.file.originalname)}`;
const saveTo = path.resolve(__dirname, "public", "images");
const filePath = path.join(saveTo, filename);
await sharp(req.file.buffer)
.resize({ width: 300, height: 300 })
.jpeg({ quality: 30 })
.toFile(filePath);
req.file.filename = filename;
next();
} catch (err) {
res.status(400).json({ message: err.message });
return;
}Based on the the file processed by Multer (file key is appended to req by Multer), we’re defining: the filename our file will be stored at the server with; the place where the file will be stored at the server and the absolute path of the image, as determined from the directory where the current script, index.js, is running.
This data is used by Sharp: the image parsed and buffered by Multer is resized, converted into a compressed jpeg image and stored at our file system — if you want to store in another place such as S3, just convert the processed image into a Buffer using toBuffer method in the Sharp instance and use it.
The reason for req.file.filename = filename is the following: initially the property holds undefined because we’re using multer.memoryStorage, which stores the file as a Buffer without a filename. So we assign the property with the filename we’ve defined in Sharp. This allows us to access the filename of our processed and saved image in the subsequent middleware.
Applying Our Middleware to a Route
// routes.js
import { Router } from "express";
import uploadImageHandler from "./uploadImageMiddleware";
import path from "path";
const router = Router();
router.post("/images", uploadImageHandler, (req, res) => {
res
.status(200)
.send(
`<p>Image "${req.file.filename}" uploaded successfully.</p> <p>Access it <a href="http://localhost:3000/images/${req.file.filename}">here</a></p>`
);
});
export default router;Well, we already have our middleware for handling file uploading. Now we can apply it to routes involving images, and that’s exactly what’s made in the file above!
The Result
Let’s check out the image uploading testing the endpoint with Insomnia:
We are sending a request to “/image” endpoint: Its encoding type is multipart/form-data and its body has a file named image appended to it. The image is handled successfully, so the response it’s an HTML page with a link redirecting to a URL pointing to the image. If any error is thrown, then it’s handled and an error response is sent to us.
The image was compressed and we can of course compare it with its version that wasn’t:
There’s almost no difference at first sight; we can just notice that as we zoom in the pixels become less sharp, but the difference considering a web application is minimun and the compressing result is, in terms of storaging, amazing! Here’s the image size of each one of them:
The compressed version is almost 10x lighter, and as we said: considering just one image the difference is ridiculous, but thinking in the big picture — an application in production constantly growing — it’s massive.
Mission completed. If you have difficult to achieve a similar result in your projects, i suggest you to do at least one of these things: write down a comment, visit the official docs — links below — of these libraries or check the github repository below of this tutorial.
References
expressjs/multer: Node.js middleware for handling `multipart/form-data`. (github.com)
sharp — High performance Node.js image processing (pixelplumbing.com)
