Mongoose and Express

These are my notes for Jonas Schmedtmann’s Node.js Bootcamp on Udemy: https://www.udemy.com/course/nodejs-express-mongodb-bootcamp/

Setup with Atlas

To host our database, we’ll go to MongoDB Atlas, register if we haven’t already, create a new project if we want, and create a new cluster. When we’ve done this, Atlas will display a 5-item “Get Started” checklist, but we’ll shelve that for now and go at our own pace.

We’ll start establishing a connection by clicking the “Connect” button in the cluster overview. We should see this popup:

We need to whitelist our own IP address first, which we can do so by just clicking the green button. Then, we need to create a MongoDB user. The username can be whatever. We’ll let MongoDB autogenerate a password for us, which we should then store in our project’s config.env.

Since the config.env now contains a password, we should add it to our .gitignore.

Connecting the Database To Compass and the Mongo Shell

After we’ve whitelisted our IP address and created a user, we’ll connect the database to the Compass GUI, which you should download and install off the MongoDB website if you haven’t already. From the “Choose a Connection Method” tab, we’ll select “Connect to MongoDB Compass.” That’ll give us this window:

We’ll copy the connection string, open up Compass, and select Connect > Connect To… from the toolbar at the top of the window. Compass will automatically detect the connection string in our clipboard, so we let it autofill the form for us and then simply copy and paste the password from our config.env.

To connect the database to our Mongo shell, we’ll go to the same “Choose a connection method” tab from before, select the Mongo shell this time, and then copy and run the connection string in our terminal. Again, we’ll be prompted for our password.

Connecting the Database to Node and Express

To connect the database from our app, we again go to the “Choose a connection method” tab and select “Connect Your Application.” Making sure we have Node.js selected as our driver, we’ll paste the connection string into our config.env.

NOTE: we’ll need to replace test in the connection string with the name of our database.

Next, we’ll install Mongoose from npm ( npm i mongoose). Mongoose, by the way, is simply a layer of abstraction on top of MongoDB, the same way Express is a layer of abstraction on top of Node. We’ll require Mongoose at the top of our JS file that starts the server and then invoke Mongoose’s connect() method. We need to pass in our connection string, but before we do so, we’ll make sure it includes our password:

The second argument is an options object that deals with deprecation warnings. We need not worry about its specifics.

CRUD with Mongoose

Now we’re all connected and ready to start using Mongoose! We’ll do so in the Natours project from my most recent articles.

Defining Schema

Schemas outline what our documents will look like. Note how we use native Javascript types:

We can be a little more specific by making our values into options objects:

For the required key, we can simply put a boolean or an array containing true and an error message, like [true, 'A tour must have a name'].

Creating Documents with Mongoose

With our schema in place, we create a model. A model is analogous to a class in OOP.

Our instances of Tour have access to a few prototype methods, including save(). The save() method adds the document to our collection and then returns a promise, the result being equivalent to the document that was just added:

If we then save our file with nodemon running, testTour should appear in Compass.

The other (often better) option we have is to call the create() method on the Tour model itself. We’ll gut the file-reading bits from our createTour function and use async/await with the create() method:

Of course, we need to use a try/catch block whenever we work with async/await:

Now, sending a request to this endpoint will update our database, and the results will be visible both in Compass and Atlas!

Reading Documents with Mongoose

The model has a find() method, like the find() method from the Mongo shell. It returns a promise, so we’ll handle it with async/await:

To fetch a single tour, we use findById(). It’s a nice shorthand for Tour.findOne({ _id: req.params.id }).

Updating Documents

Mongoose makes PATCH requests easy with a method called findByIdAndUpdate(). It takes two arguments: an ID and our patch object. An optional third argument is the options array. Setting new to true will put the newly updated document in the response, as opposed to the original document before the patch.

Deleting Documents

DELETE requests are easy as well:

Queries in Mongoose

A query string in a URL starts with a question mark and then has any number of key-value pairs separated by ampersands. A URL with a query string might look like https://natours.dev/api/v1/tours?duration=5&difficulty=easy. If we want to return a filtered list of tours, we’ll use the same request that returns the entire list of tours. We can get easy access to the query’s parameters by using req.query, which would be this Javascript object for the URL above:

{
   duration: '5',
   difficulty: 'easy'
}

To get a filtered list of results, we can simply pass req.query as a filter object into Mongoose’s find() method:

Alas, this solution is too simple. Later on, if we implement pagination, we’ll have URLs like https://natours.dev/api/v1/tours?duration=5&page=2, and req.query will be a filter object that’ll return no results. We need to trim down req.query if we want to keep using it as a filter query.

We don’t want to directly modify req.query, so we need to create a copy of it. Since Javascript objects are passed by reference, const queryObj = req.query won’t do. Instead, we’ll use the spread operator: const queryObj = {...req.query}. Then, we’ll loop over an array of all the fields we want to exclude from our query object, deleting any matches from queryObj.

Less Than and Greater Than Queries

We can also write queries that handle < and > operators, not just equals operators. We do so by specifying the operator in brackets before the equal sign, like duration[gte]=5. With this operator, the query object now looks like this:

{ duration: { gte: '5' }, difficulty: 'easy' }

Notice that this looks almost exactly like a filter object, only with the dollar sign missing in front of gte. We add the dollar sign in server side using the replace() method and a regular expression:

Sort Queries

We excluded 'sort' from our query earlier, but now we’re going to add it back in. When we want to add methods like sorting and limits to our query, we have to save it in a variable first. Then, it’s just a simple matter of chaining the sort() method to the query and passing in the value of sort in the query string. Here’s the whole block of code up to this point:

Now, a query string of ?sort=price will sort our results by price in ascending order. To get descending order, we tell Mongoose by simply adding a negative sign: ?sort=-price.

If we want to sort by a secondary field (e.g. sort by price and then, if two tours have the same price, sort by rating), we add a comma to our query string: ?sort=-price,ratingsAverage. Problem is, the sort() method accepts a space-separated list, so we need to take care of the commas beforehand:

Field Limit Queries

Field limiting specifies only certain fields from the document to send in the response. This process is also called “projecting.” This feature can be useful if we have very data-heavy structures in our database and we want to conserve bandwidth. A field query looks similar to a sort query: ?fields=name,duration,difficulty,price. The implementation is similar to that of our sort query:

In this block, we’ve created a default so that we never send the client the __v field, which Mongo uses internally. The minus sign in front of the __v tells Mongo, “Return everything except __v."

Another way to never send a field to the client is by setting select to false in the schema:

Pagination

A pagination query has two parts: a page parameter and a limit parameter that specifies the number of documents per page. It might look like ?page=2&limit=10. Using Mongoose, we’d create this query like:

query = query.skip(10).limit(10);

In other words, if Page 1 has results 1–10 and Page 2 has results 11–20, we have to skip 10 results to get to result 11, the beginning of Page 2. We’ll implement it in code like this:

Note that we should always use the OR operator to specify default values. Otherwise, if we had a database with 1,000,000 documents and the user didn’t specify a limit, we’d return all 1,000,000 results, which isn’t very user-friendly.

The last step is to handle errors when the user requests a page beyond the number of documents we have to show.

Aliasing

Suppose we had a query that was commonly requested. For example the top 5 best and cheapest tours: ?limit=5&sort=-ratingsAverage,price. To create an alias for this route, we go to our tour routes file and make a new route:

router.route('/top-5-cheap').get(tourController.getAllTours);

We’re still going to use the getAllTours() function, just as we have for all the above queries, but we need some way to inject information into the incoming query string for this to work. We can accomplish this with middleware, which we’ll create in our tour controller file:

Then, we call this middleware in our route:

The Aggregation Pipeline

Aggregation of a Mongo database encompasses statistics like averages, sums, minimums, and maximums. To begin using the aggregation pipeline, we’ll create a new handler:

We pass an array of objects, called stages, as an argument into the aggregate() method. The $match stage is pretty much just a filter query, but the magic happens within the $group stage. In this case, we’ve set _id to null to target all the documents in the collection that match the $match stage. Then, we can perform mathematical operations on certain fields from those documents. Take note of the new '$fieldName' syntax.

We can add a couple $sum operations to our $group stage:

Here, $sum: 1 might look tricky, but it’s really just saying, “Every time a tour passes through this pipeline, add 1 to the accumulator.” Now, our response object looks like this:

If we want to group by difficulty, we can set _id to '$difficulty' in the $group object and get this result:

If we then want to sort these three objects in the "stats" array by average prices, we can add another stage:

We can even repeat a stage to filter out more data. For example, if we want to exclude the easy tours from our results, we can add another $match stage below the $sort stage:

Unwinding

Let’s say our client wants to make a monthly plan that outlines which tours start in which month. In our tours collection, each tour has three start dates. Because the startDates field is an array, we have the option to create a new document for each individual start date by using an $unwind stage:

Now we will get 27 results from our original 9 tours, each having one start date. The next step is to specify a year with a $match stage. Thankfully, date comparisons are easy with Mongo:

Then, we need to group the tours by month. Luckily, one of Mongo’s aggregate pipeline operators, $month, extracts the month from a date string for us, so we don’t have to bother with regular expressions.

This block of code warrants a breakdown. The _id field is equivalent to the month extracted to the start date. Then, every time a tour with a matching month passes through the pipeline, Mongo will add 1 to numTourStarts. Last, the value of that tour’s name field will get pushed to the tours array. Our response now looks like this:

… and so forth. It’d be nice if "_id" was "month" instead. We can’t change the key’s name, though, so we do this:

In a $project stage, we set the fields we want to hide to 0. Finally, we use a $sort stage to sort our data by numTourStarts, assigning -1 for descending:

And we’re done! Our output now looks like this:

(etc. etc.)

Virtual Properties

Virtual properties are fields on a document that will not be stored in the database. An example might be a kilometers field when we already have a miles field; since we can convert between the two easily, there’s no need to store both in the database. In this example, we’ll create a durationWeeks property derived from our duration (in days) property. We go over to our tour model file and write this code after the schema:

After specifying the property name in the virtual() method, we the chain an Express get() method and pass in a function that handles the calculation we want. We have to use get() because this property is only created when a GET request is made. Note that we can’t use an arrow function here because arrow functions use lexical this binding. We don’t want that; we want this to point to the document in question when the function is called.

At this point, we won’t yet be able to see durationWeeks in our output. We need to add a second object to our schema:

Now output shows the virtual property. Note that we do this conversion in the schema to follow MVC architecture and keep business logic as much in the model as possible. The mantra here is, “fat models, thin controllers.”

Mongoose Middleware

We can define functions in Mongoose to run before or after certain events, like saving a new document.

Document Middleware

We define document middleware right below our schema like this:

Here, this points to the current document being saved. Now, we’ll use this function to create a slug out of the tour’s name using the slugify package from npm. We’ll also give our middleware access to the next() function so that we don’t run into any problems when we add more middleware.

Now we just need to add slug: String, somewhere in our schema and our new property is ready to go!

Note: we can also run middleware after an event with tourSchema.post().

Query Middleware

We can also manipulate query objects with middleware. The use case we’ll explore here is the idea of secret tours that aren’t available to the general public. We’ll start by adding a secretTour field to our schema:

Now, we’ll create a middleware that modifies the query object. Note that it looks similar to the document middleware. The difference is that the hook is now 'find' (for the find() method) instead of 'save':

Here, this points to the query, to which we’re chaining another find() method before its execution. Problem is, this middleware works only for find(), not findOne, findOneAndRemove(), etc. We can fix this by replacing our 'find' hook with a regular expression that matches any method that begins with “find”: /^find/.

Aggregation Middleware

Let’s say we also wanted to exclude our secret tour from any of our aggregations. Rater than add a new $match stage to each of our aggregations, we’ll add this middleware:

When we console.log(this), we see the Aggregate object, which includes the _pipeline array. We can access this array with either that name or pipeline() and then add a new $match stage to the beginning of the array:

Now, the secret tour is excluded from all of our aggregations.

Data Validation

Validation means making sure each of the required fields in our schema has been populated.

Built-in Validators

We’ve used one of Mongoose’s built-in validators already: the required field in our schema. Let’s add two more: maxlength and minlength.

This validation will also run in our updateTour endpoint if we have runValidators set to true.

The validators maxlength and minlength only apply to strings. For numbers, we can use min and max.

Note: min and max also work with dates.

Next, we want to restrict our difficulty field to only three values: 'easy', 'medium', or 'difficult'. We can accomplish this with the enum validator:

Because we’re passing in an array of values, we can’t use the [value, errorMessage] shorthand like we had for the other properties.

Custom Validators

Let’s imagine we want to make sure our priceDiscount value isn’t greater than the price.

Note that we cannot use an arrow function here. If we do, this will lexically point to the schema and not to the incoming document.

An important caveat here is that this will point to the incoming document only when we are creating a new document. This validator won’t work if we attempt to update the document.

A great library for validation is validator, which we can install with npm i validator. Then, using one of that library’s functions in our code would look like this:

Hope this helps!