Building my own NodeJS MVC Framework — Part 2

In order to start building our Model.Base class to be extended in each model, we need to set up our “helpers”. They should be 3. Model.Attributes (our constant data types, they we'll be used only for reference by now but we can apply this in the future as strong types.), Model.Validators a set of small functions to validate attributes easily. And finally (and most important) Model.Client our PG.Pool client using pg package to handle the database connections and response. We'll be using all the power of query() to also sanitize our data :) security first! I've compiled all 3 in just one file (src/ModelHelpers.js).

First, let’s load the pg package and create our 3 constants. Attributes are pretty much self explanatory, Validators is a set of 2 methods (initially) and we'll export our Client as an instance from PG.Pool() which will handle async/await easily for us. We won't need to worry about opening and closing a connection to Postgres every time, pg will do this for us. Please, configure your host, database, username and password along. This tutorial will not be covering database creation and table population.

There are only 2 things missing explanation here, and it’s the both methods in Validators.
All validators method will expect 2 variables to handle,attribute which will act in the current validation (title, in our example) and model which is actually the object itself. This is pretty straight forward, allowing us to check the current model value we want to handle as model._attributes[attribute].value (as in our structure of private/defaults above). And as we have our public model.errors array, we will populate it with our handled error if we found something. before saving or updating our model, we will verify if errors are populated, if so we halt the process and raise the error. Easy enough.

Now, let’s start building our Model.Base ! Which will handle all the real magic.

First, let’s import what we already have, design our class and export our default model.

Here we import what we need and export what we want. Save this as src/ModelBase.js and we'll be using as our Model.Base in the future. We will import it as import Model from './src/ModelBase'; and be able to work with that.

The first and most important data for our model to know is the table name, we will define a default one based in a plural lower case name of the class, or we’ll be able the user to overwrite it by only setting this._tableName = 'newName'; . So, our code should be: this._tableName = this.constructor.name.toLowerCase() + "s" . this.constructor returns our class, and .name gives us the class name, in our case, Post. We get the lower case post and just append a s to pluralize it. Easy enough, we now have posts as our default table name. If in case we have a User model, it'll be users . Let's setup our extra variables now, as we described above:

We have our private data in place, our default table name and our 3 default attributes which all models should have: id, createdAt and updatedAt.
At the last point of the constructor we call this.setup() which in the base class does nothing, as we’ll use it in our model classes to put everything in place.
As you probably noticed, we didn’t used our attrs variable we're passing in the constructor. We want to use this to set attributes in mass for our model, as new Post({title: 'something', content: 'something else'}) and be able to access it easy as post.title and be sure we'll be saving it later correctly. Let's do that. After setup add this:

And after setup() {} define our _setMassAttributes method as:

Here we’re looping against all the attributes, we first check if we have the attribute defined. If the user send new Post({author: 'Leonardo Pereira'}) but we don't know that attribute, we won't be sending it to the database, that check prevent us of handling unknown data. After that we define the attribute twice, first at a class high level class, just post.title and later as our attribute data, which we'll handle to insert/update our database.

Now, before we proceed with our code, let’s start testing what we have. For this, we’ll be using jest. Let’s first start setting up our environment: run npm init and create your package.json. As entry point, define main.js and for test command, just use jest. Now, let's install jest and enable babel to handle ES6 import/export. Run: npm install --save-dev jest babel-jest @babel/preset-env . Now, create a file named babel.config.js at root and set up as:

As we talked before, we are going to use some selected packages. But let’s start manually adding one by one as we use them. The first one we need to install is pg , just run npm install pg --save .

Let’s create our first test to verify if the tableName and attributes are being defined correctly, create a new test file inside tests: ModelBase.spec.js . Now, let's also build our Post model to extends our ModelBase as we're going to simplify our tests. Create a new file at models/Post.js and define as:

Now, we will write our first test! Will be very simple, but is the beginning of our real development. Add a new file at tests/Post.spec.js , our code will be:

Run npm test and see our first tests working. Now, let's starting adding extra attributes.

Now, let’s add the setAttribute public method. In the ModelBase.js structure, we'll have 3 kinds of methods: static public and private . Below is the full code with comments organising the structure, so when I say public method we'll put inside the /* PUBLIC METHODS */ block. Here's the full code with setAttribute() :

As you can see, setAttribute is pretty straight forward, we set our private _attributes and the high level this[attribute]. Let's add our tests, inside the constructors describe block of Post.spec.js include this:

Here we are checking if our 2 new attributes are defined and set as null . If you try to check a different attribute as expect(post.something).toBe(null) you'll get an error, as the return is undefined instead of null .

Great. Now our Post.js already have custom attributes, default attributes and correct table name. Now, let's add the validation hooks, as we're going to populate the errors array and we will be able to verify if the hooks are set correctly.

In our ModelBase.js add a new public method which we'll populate our _validationHooks with 2 parameters, attribute (string) and validator (function):

Great, now edit our Post.spec.js and below import Post add our Validators, we're going to use it for comparison. import { Validators } from '../src/ModelBase';

And add another test block:

Run npm test and see we're correctly defining our private hooks. But this isn't very fun, let's start building our first version of our save() method. In this moment we'll expect post.save() to return false and populate errors with our error message. In our next version save() we'll be async, but we'll keep it simple for now. We'll update everything later.

Add a new public method save and use as:

Now, create a _checkValidations private method. This one will be able to handle not one validation each time, but multiple validations, so we can use as this.validate('title', [validation1, validation2, validation3]) . Keeping the code DRY.

This function is pretty straight forward, we run a forEach for each _validationHooks we have and check if hook.validator is an array or not. If not, we will check if hook.validator is a string. Why? If it is, it'll call an internal method in the defined class, allowing to not depend in anonymous function, as this.validate('title', 'myCustomTitle') will actually check for this.myCustomTitle() allowing you to work with the object itself. The last one will assume the hook.validator is a function and call it, sending the attribute and the model itself as this .
In the case of being an array, we'll loop through it again and just check if is a string or not and do the same. I've avoid doing recursive calls as _checkValidations() doesn't pass any argument.

Now, let’s add our tests to check if it’s actually validating. In Post.spec.js add this new describe block:

Run npm test and see our tests are passing. Awesome, right?Now, let's stop for a moment with the hooks, we still have beforeSave and afterSave to check, but let's do something more fun and assign our attributes. First, we need to understand our logic. Our future public save, update and create methods will use the this._attributes and this._changedAttributes to deal with the persisted data, but we will have a high level entry to define the attributes as just post.title = 'My new post'; , so, to handle this we'll use a private method _reloadAttributes . It'll remap the internal attributes with the changed high level attributes, and when we check that something have changed, we'll update this._changedAttributes as well. With this, we'll be able to create our first finder method.

Below _setMassAttrs add a new private method called _reloadAttributes :

Here we loop between our defined this._attributes and check if the value is different from our high level attribute and if this._changedAttributes doesn’t already include the change. If true, we increase this._changedAttributes with the changed attribute. Right after we remap this._attributes[attribute] with the high level value. This is enough for our mapping. In the save() method, call this._reloadAttributes() before this._checkValidations(); .

Now, let’s add our tests. Inside the hooks describe, add:

We have added two different checks. First using high level, which we can see we have post._changedAttributes, and the second one using mass assignment, which doesn’t change any attribute as it was originally defined in the constructor. :)

Finders. Let’s write our first finder: Post.find(1) . As I've said before, we're not covering database, so I'll roughly explain what to do.

First, create a new table posts . You should have configured your Client with the correct data. The table should contains id (integer, primary key), title (VARCHAR 255), content (TEXT), createdAt (timestamp without time zone) and updatedAt (timestamp without time zone).

CREATE TABLE "posts" (
"id" serial,
"title" varchar (255),
"content" text,
"createdAt" timestamp,
"updatedAt" timestamp,
PRIMARY KEY ("id")
);

After, insert your first post, you can run just this query to get it easy and populate our timestamps: INSERT INTO "posts" ("title", "content", "createdAt", "updatedAt") VALUES ('First post', 'My content', 'NOW()', 'NOW()');

Let’s create a config/database.js file to hold our different environments configurations. As our tests will exchange informations with PostgreSQL we will need two database. Recreate the table above in a new database, with suffix _test .

In your config/database.js add the following and fill as necessary:

Update your src/ModelHelpers.js file, add a import databaseConfig from '../config/database.js'; at the top of the file and change the Client constant to: const Client = new PG.Pool(databaseConfig); .Now our ORM responds to the correct environment. We don't have to worry about trucating our tests data (and we will truncate).

Now, we’ll add a new async static method find() inside /* STATIC METHODS */ and a new private method _find() inside /* PRIVATE METHODS */ . Why? static methods just execute a simple action, doesn't return the new object, we'll need the static to instantiate the object and call the private method to actually find the persisted object.

Add this in the static methods:

And add this in the privates method:

Now, notice two things. First, both methods are using async and await , this means we're waiting the Promise to be done before moving along. pg.Pool.query returns a promise and we can end into race conditions if not using this way. As we're going to use in the future, we will also need to couple the request into async/await, as ;(async () => { return await Post.find(1); })(); . Moving on: first we run a simple query to fetch by id, using $1 and mapping in the array as a second argument we sanitize the query avoiding SQL injections. The result of .rows is an array, so we fetch the first result and use our private function this._setMassAttributes() to map our internal attributes. We also need to tell our model that this data is persisted, when we run save() in the future we'll want to update the data not duplicate it.

Let’s write our tests. In Post.spec.js change the import { Validators } to import { Client, Validators } from '../src/ModelHelpers'; . Since we want to have directly access to our database connection. Right below our import, include this afterAll callback, so we'll be sure our connection is closed when all the tests are finished: afterAll(async () => { return Client.end(); });

Add a new describe() block at the bottom:

Here we’re going to use hooks. The first one, beforeAll , we'll be creating a new record to our posts table. Simple, we'll change this in the future to use our own Post.Create . This will be executed before every test() inside this describe() block. The second one afterAll() will truncate our table and RESTART IDENTITY will reset our id serial to 1. Meaning we'll always have a result when using Post.find(1); . In this case is very useful.

The test is straight forward again, we set our const post , wait for our Promise and all our data is available. Neat!

Now, how about we add more finders? first() last() and all() . Next, we'll move to findBy() and our chaining.

Actually, we can move our private_find() to a more general function, such as __get() and do it more DRY. Here, the improved code reusing what we can:

Replace the previous async _find(id) {...} with the code above, and let's replace our static methods too:

More easy to check and to test. But we have a last general finder to add, all() , this one will only be static code. As we are not using any replication by now. add the code at the bottom of our static methods:

The ugly one here is new this()._tableName , as setting it multiple times. We can't avoid this as we need our private table and we also need to create new objects for each result. It's not a wallflower. But it works just fine. We map the rows and reuse our methods, we just return a new array full of Posts. And what about testing this? Easy pick.

Here’s our new describe('finders') , beforeAll() inserts two posts now and we're testing all our new finders:

Not much to say here, as it’s extremely close to what we did before and it’s working just fine. Now what we’re missing? Our chaining. We’re going to keep it very simple, only having two kind of where matchers, objects {} pairing and a single string. In the case of a string, our second argument will be used as our sanitized data. What this means? basically where('name = $1', 'Leonardo') will become WHERE name = 'Leonardo' but with our security up to date. As objects pairing we'll expect to respond as where({name: 'Leonardo'}) and the result should be the same as previous, but objects should be a bit more smart. If we use multiple arguments we should compare multiple fields, as where({name: 'Leonardo', age: 29}) should result in WHERE name = 'Leonardo' AND age = 29 .
Our three next methods will be limit(n) offset(n) and orderBy(string) . As in a future pagination, our design would be Post.orderBy('id DESC').limit(10).offset(page || 0) and our query should be SELECT * FROM posts ORDER BY id DESC LIMIT 10 OFFSET 0 . I'll not include select() or like comparisons, you can include by yourself if feel to.

So, no more talk, let’s create our chaining methods. As a note, the chaining should return this . And only when calling our new all() public method it'll return the data. This means we will be changing our all() current structure for static and applying as a new public method. It'll not be that complicated. :) We'll be changing _first() and _last() as well.

Update all your static methods with the new code:

Here only find() will be the same. We have included 5 new methods, findBy() at the top, which receive properties/where combinations and return the first() result. Now first() and last() are calling public methods, this static methods have become shortcuts. all() is also using a public method, but chaining limit(100) before to avoid data overload. And we finally include our 4 chaining methods: limit() offset() orderBy() and where() . All 4 pass a new method, ref() which sets this._ref , we're going use this to be able to allocate multiple objects without doing deepcopy. By reference is a bit easier.

Include 5 new private attributes beforethis.setup(); in your constructor():

Now, in our private methods, replace _first() and _last() with the code below and include our new _whereQuery() private method:

Now, both first() and last() are using the new _whereQuery() private method. This will add chaining to both method, so we can use something like Post.orderBy('title ASC').first() and get the first post by title ascending.

The _whereQuery() is a very simple structure builder. As for security we match $1, $2, $3… with a second argument, an array. So we first define let args = [] and check if we have used where() which we'll create in a bit, if we have data, we set another array where = [] and loop through our private data. Now we just populate where and concatenate args . Finally we return a 2 size array, first with the condition WHERE = ??? and by last our concatenated arguments.
If false we return a two size array again, but empty.
Now, to our new public methods:

Below save, include 5 new public methods:

We start defining ref() which just set our this._ref and return the object. All 5 methods here are returning this to allow chaining. limit() offset() and orderBy() just map our private data and return our object itself. Easy.
Our where(condition, args) is a bit more complicated, as we have seen, _whereQuery() uses the data from here. So, as we talked before, where() can behave in two ways (actually, 3). First: where('name = $1', 'Leonardo') and we get name = 'Leonardo' , we can pass the second argument as an array and match multiple results: where('name = $1 OR name = $2', ['Leonardo', 'Dante']) and we get name = 'Leonardo' OR name = 'Dante' and by last an object matcher: where({name: 'Leonardo'}) and we get the same as first name = 'Leonardo' .
To achieve this, first we check if conditions is typeof string, if this is the case, we add a new hash into this._where with 2 arguments: {conditions, args} . In args we do a quick check, if is a string we pass as [args] and if not we assume it's already an array.
Our else is a bit tricker, because we have to find our pointer for the matching, as $1, $2 and so on. So, let pointer = ??? starts mapping this._where and fetching the current args.length . Let's image we have already 2 properties defined, each with 1 size array for arguments. as a result, our map will return [1,1] . With this information, we do a reduce of a (accumulator) and c (current): (a, c) => a + c and the second argument 0 is the default value, that will basically process: 0 + 1 + 1 and return 2. At the end we add a 1 . So, if we already have $1 and $2, we should start our pointer at $3 .

Right after we do a loop between the conditions (or properties) populating this._where again. for the conditions we do an interesting incrementing. In javascript we have two ways of incrementing an integer. integer++ or ++integer . The first will return the original value and increment the variable with + 1, the second will increment the variable with +1 and return the new value. Is an ordering difference. To display it better: let a = 1; console.log(a++); console.log(a) will print 1 and 2. If we do let a = 1; console.log(++a); console.log(a) will print 2 and 2.
Knowing this we can use pointer++ to assign $3 and our pointer is now at 4.
In args: we just map again checking if is a string and putting inside an array or assuming is an array and returning it.

Now we have our chaining methods, we just need to include our new all() public method:

It’s extremely similar to what our static all() was doing, but including the power of our new this._whereQuery() to include all our conditions. And we do a small different thing here, instead of let obj = new this(); we use let obj = new this._ref() as we can't construct this() again.

The chaining is done. How about some tests? I’ve included below a bunch of different tests, I’ll not explain what they do and will let you find by yourself. :) Include the block below in our Post.spec.js file:

If you’re following correctly what we’ve done so far, our ModelBase.js file already have 233 lines. That’s a lot, and we’ll be adding about 100 more lines, but it’ll be worth it. As we’re almost there, we need now to be able to persist our data. First, let’s allow our save() to work, it's very close to be done. Our save() will need to know if the data is persisted or not, for this will be a very simple if/else and mapping to two new private methods, as _insert() and _update() . In the end save() behaves as an upsert.

Update our public save() method to previously reload our attributes and if no error are found, we will either call _update() or _insert() based if it's persisted or not:

Now, let’s write our tests first and later we will add our _insert() private method.

If we run our tests npm test you'll see it fails, it doesn't have the _insert() method.

In the private methods, below _checkValidations() add our _insert() method, it'll be async and do a lot of magic, I'll explain everything.

The insert method starts by defining two of our default values, createdAt and updatedAt . Here we use the power of PostgreSQL, NOW() will update the table timestamp with the current time. We're using the internal attributes instead of our high level as later we're only working with our internal attributes, that's why in the save() we call this._reloadAttributes() before moving on.

Later, to mount a query that makes sense, we are going to use a hash queryConstructors handling insertFields (the field names we're going to map to our table), pointers (again, the magic of pg to sanitize data), valueFields (the values of what we want to insert) and returnFields (the data we're going to fetch right after inserting in the database, here we will receive our new timestamps and the fresh new id.

In // loop through attributes we're going to do as it says, loop along all our internal attributes. But first we set our pointer to 1, to map as $1 and be increasing as each attribute is found. We don't need any kind of sort here.
If the attribute is id we don't want to use it in any way besides the returnFields , so we just assign it and ignore the rest.

Now, we just set our queryConstructor with the values we want. Pay some attention in the way the strings are built. If we just add insertFields(attrName), the SQL will be something like INSERT INTO posts (title) VALUES ('value') , that will not be a problem, but createdAt and updatedAt have a upper case A in the middle, and PostgreSQL don't understand that as a variable name, we have to encapsulate as "createdAt" to not have any problems. And is good to do in a general way to avoid weird problems in the future, believe me, I had a lot while creating this.

With our queryConstructor in place, we build a simple insert query. Our tests do new Post({title: 'Post with save'}) . And our query will be: INSERT INTO posts ("createdAt", "updatedAt", "title") VALUES ($1, $2, $3) RETURNING "id", "createdAt", "updatedAt", "title"; . This simple query will be enough to populate the data and retrieve everything we want to use.

In the try{} catch{} we run our query, reuse _setMassAttributes for general assignment, set this._persisted to true and reset our this._changedAttributes since we have saved data we don't need that information any more, and return true! In case of any internal issue during the SQL we'll have a small feedback in the screen. Run the tests again and see they're now passing.

One done, two more to go. Let’s create our _update() private method now. Before, include the tests we're going to expect to work. Include the code above inside our new describe().

Here a few things to notice. First we create a new Post again and check if it was saved, the id here is expected to be 2 as we only truncate the table after all the tests inside the block are done, means the previous test and post is still persisted in the database. Also we memoize post.updatedAt as we're going to compare if it has changed or not.

The rest of the test is pretty self explanatory, we just check if the data matches and everything is updated as well.

Get back to our ModelBase.js file and include our _update() private method, below _insert() :

In the _insert we have used our internal attributes, _update will be using them as well but based in the _changedAttributes. In save() when we call _reloadAttributes it also assigns what we’ve changed, giving us the power to update only what we really want to. So, we need to tell our database to reflect our change in updatedAt with a new NOW() and we also force push the same field in _changedAttributes . We initialize a new queryConstructors variable, but with some different fields. We won't need a pointers here since updateFields will have the pointer as well. And we'll also ignore id completely in the queryConstructors, so if the attribute name is id we just move along in the loop.

After the loop, we assign the id to the valueFields, it’ll represent the last pointer ($3 in our tests) as we do the pointer++ having the last value correctly assigned. The query is very simple, and we have our id in place and also returning our updated data, this is useful mainly to update our object with updatedAt . Finally we do the try {} catch {} block again, assign all returned attributes and reset changedAttributes.

Run your tests again and see they’re green. :)

Now we’re able to read, insert and update our data, for a full working CRUD we just need the last piece: destroy. This will be the simplest of all, since we just need to remove the record from the database and reset everything our model have. Let’s include our new tests inside our last describe():

Here we use a lot from what we’ve done before. Create a new post (id 3), get total posts count (length 3), destroy the post we’ve created and is persisted (id 3) and recount our posts (length 2). If you run it’ll fail, we need to add a new public method and a private method. The public method below our save() is very slim:

And our private method holds our real logic, add it below _update() :

First, just check if the data is persisted, if it isn’t we don’t have to destroy anything, return false. If it is, we run our DELETE FROM posts WHERE id = 3; query, reset all our internal attributes and return true. Very simple.
Run our tests again and see they're all passing. Great! CRUD is covered.

We’re really, really close to finish our basic ORM. But I’ve promised to include two very useful hooks: beforeSave and afterSave . Also, before I forget, we also want to have a Post.create() static method to do things very easily. Imagine in our controller create action, we want to do something easy as const post = await Post.create(this.post_params()); and it should have the data persisted. 3 things and we're done, I promise. I'll be quick.

Add in your describe('save/create/update/destroy') block a new test:

Easy, using the new static method we’ll add. Now, in our static methods, add the create method:

Run npm test and see all our tests are passing. We've used what we created and made an easy helper. :)

Now, our final hooks! They’ll be extremely similar. First, update models/Post.js to include two example hooks of beforeSave and afterSave:

Our hooks will be only accepting strings, the string should be the model method name. This means this.beforeSave("prependTitle") will actually call this.prependTitle() before saving.

Let’s create our tests. Including a new describe() block for the hooks:

If you run your tests, a lot will fail, and even after we add the hooks, they’ll be still failing. Our hooks will modify our data, I’ll let you be responsible for fixing it. Let’s move on, add two public methods beforeSave() and afterSave() in our ModelBase.jsbelow our validate() :

Here we’re just populating our array hooks. Simple enough, our private methods will be responsible for the logic. But we need to update our public save() as well to call it in the right moments:

Before validating our model, we run the beforeSave hooks, and instead of returning either _update or _insert we memoize the response, run the afterSave hooks and return our boolean.

Easy, now let’s add our two new private methods: _runBeforeSaveHooks() and _runAfterSaveHooks() , put both below _checkValidations() :

Here we just loop for what we have, call our methods using this[methodNameAsString]() and return what we have. beforeSave do a small trick before, it reload the internal attributes after each beforeSave hook, this can be a bit expensive but will ensure our internal attributes are correctly update between each step. afterSave doesn't need this, we can just update the high level api and when we're saving in the future what we have will handle it.

If you run our tests now, what we’ve added will be passing but previous tests have failed. The data has been modified and what we wrote before is not working anymore, as I have previously said. Update the code as well and get all the tests passing.

Congratulations! We have a functional ORM now. :) It was a lot of work, but it paid of. We have learned about using async/await correctly, exchanging data, persisting and working with a database, using interesting ways of private and public API and much more. I hope you’ve had fun doing this as I did when learning.

Now, let’s move on to Part 3. A framework isn’t done of only an ORM.