Things that make you far better Node.js developer, part 2 (CommonJS)
In this article, I want to dive into the heart of Node.js that is responsible for managing modules. We all use modules in every Node.js application, but we might not know how it works under the hood.
In the last article, I talked about how we can add new features and syntax to javascript with the help of V8 and C++, and finally, we understood that Node.js is nothing but this. We discussed that Node.js needed to address some issues to be able to manage a server. One of these issues was managing modules.
to understand this article better, you need to know these concepts in javascript: if you already know these concepts, you can skip the prerequisite part. If not, no worries. I talk about these concepts
- IIFE
- prototype inheritance
- constructed function
- call by reference
prerequisite:
What is a module?
the simplest definition of module for me is it is a technique to reuse codes without having any code collision. That’s it.
Most programming language has their own way of managing that, but At the time Node.js was created, javascript didn’t have any feature to manage modules; however, since javascript es5, this feature is a part of JS.
What do I mean by code collision?
Let’s assume we want to reuse some codes. we can create another file, eg utils.jsand put the related codes there like:
utils.js:
var msg = "Hello, World!";
var sayHi = function() {
alert(msg);
};Now, let’s assume we have another file, eg app.jsthat have a variable named msg . as you can see in both file we have a same variable name msg
var msg = "stinky code";Let’s register these files in our HTML file and call sayHi() function there
<html>
<head>
<script src="js/utils.js"></script>
<script src="js/app.js"></script>
<script>
sayHi(); // it shows stinky code!!
</script>
</head>
<body>
</body>
</html>Based on the utils.js file, we have to get Hello, World! but we see stinky code. it seems code collision happened. We can fix this problem with a technique named IIFE in JS which we will learn it soon.
What is passing by reference vs by value?
This is a really important topic, and we might see many questions in interviews regarding this topic. When we talk about passing, we either pass this variable to another function or assign it to another variable.
When a variable is declared, it is assigned a memory address in the computer like this:
var a = 10;
In javascript, we have two kinds of value:
Primitive and reference
Primitives value are:
boolean, string, number null undefined , Symbol
reference value are:
object or array
In many programming languages, if we want to pass the variable by reference or by value, they have their syntax to do this, but in javascript, it depends on the type of variable (Primitive and references) we are working with.
By value:
var a = 1;
var b = a + 1;
console.log(a,b) // 1, 2As you can see, a and b have primitive value (number), and when we are working with Primitive types, javascript copies the value and assign it to another variable. So we have two variables that point to the different parts of memory with the same value.
By reference:
Let’s do this operation with an array
var a = [1,2];
var b = a;
b.push(3);
console.log(a)
console.log(b)We expect that the a would be [1,2] and b would be [1,2,3] but if we run this code we see both of them have the same value which is [1,2,3]
This happens because, in non-primitive values, javascript doesn’t make a copy of the value when we assign or pass them to another variable or function. It makes a copy of the address of the variables in memory. So both of these variables are pointing to the same address in the memory
What is first class function?
First class function are part of javascript that makes this language amazing. let’s talk about them.
It means whatever you can do with string, boolean, or any other types in javascript, you can do it with function.
For example we can pass a function as an argument exactly like passing a string variable to another function:
function sayHello() {
return "Hello, ";
}
function greeting(helloMessage, name) {
console.log(helloMessage() + name);
}
greeting(sayHello, "JavaScript!");or you write a function that returns another function eg,
function sayHello() {
return () => {
console.log("Hello!");
};
}or you can assign a function to a variable like this:
const foo = () => {
console.log("foobar");
};
foo();We will see how this feature can help us to make modules guys. This is something that we don’t have in many programming languages and it is so powerful.
What are IIFE (Immediately invoked function expression) and closure
IIFE is a common JavaScript pattern that executes a function immediately after it’s defined.
(function() {
// Code that runs in your function
})()you can also assign this function to a variable like this:
var func = (function() {
return () => {
console.log('Hey')
}
})()
func() // Heylet’s talk about the use case of IIFE:
1- Avoid code collision:
If you define any kind of variables (let, const, var) within a function in javascript, we can not access those variables outside of the function, and also, if we have two variables with the same name outside, and inside the function (like the one that we saw in previous example) , this technique can help to avoid code collision. let’s see an example:
As you can see we have a variable named msg outside and inside of the function but by the help of IIFE we avoid code collision
var msg = 'outside';
(function() {
var msg= 'inside'
console.log(msg); // inside
})()
console.log(msg) // outside2- closure:
closure simply means when you define a function, it has access to all variables outside of the function when it was created. let’s assume we have a loop, and we want to log the index of the loop within setTimeout.
for (var i = 0; i < 6; i++) {
setTimeout(() => {
console.log(i)
}, 100)
}if you run this code, you will see 666666 instead of 012345 .
The reason is when javascript sees an asynchronous code (in this casesetTimeout), first, it needs to execute all synchronous code. for loop here is synchronous code. so it executes for till the end and then executes all the callbacks that are provided for setTimeout. Now it’s time to execute setTimeout callbacks and at this moment, the index of the loop is 6.
That’s why it prints 666666. Now let’s fix this problem with closure:
for (var i = 0; i < 6; i++) {
var innerFunction = (index) => {
setTimeout(() => {
console.log(index)
}, 100)
}
innerFunction(i)
}if you run this code, you will see 012345 because when innerFunction is created, it captures all the variables at that moment. BTW we can write the above code with IIFE format like this
for (var i = 0; i < 6; i++) {
((index) => {
setTimeout(() => {
console.log(index)
}, 100)
})(i)
}Another use case of closure is we can define private and public variables inside the method. can you guess how? put a comment if you know :)
What is object literal?
If you’ve used javascript, you probably have used object literal and what object is, but maybe you don’t know its name. It is like a JSON. the only difference here is in JSON keys are covered in the double quote.
Object literal in javascript is a way of creating a group of different data. we can define as a name-value pairs inside of curly braces like this:
var greeting = {
fullname: "Michael Jackson",
greet: (message, name) => {
console.log(message + " " + name + "!!");
},
gender: "Male",
getFullData: () => {
return this.fullname + this.gender;
}
};now we can assign or access to any properties in the greeting object with . operator:
greeting.greet();
greeting.fullname = 'Hey this is test';what is constructed function?
It is a normal function that creates an object. in javascript there are many other ways to create an object. constructed function is one of them.
function Person(first, last, age, eye) {
this.firstName = first;
this.lastName = last;
this.age = age;
this.eyeColor = eye;
}var obj = new Person('Hossein', 'Derakhshan', 33, 'Black');as you can see with new keyword, we can define a new object in js through constructed function. now if we logobj we will see:
there is another property named [[Prototype]] let’s see what it is
What is prototype in Javascript?
By the help of this feature, we can kind of implements inheritance in javascript. I’m saying kind of because JavaScript is not a class-based object-oriented language.
when we create a constructor function, javascript will add a property named prototype to that function. Then if you assign any property or methods to the prototype there will be available in instantiate object.
let’s see this example:
function Person(first, last, age, eye) {
this.firstName = first;
this.lastName = last;
this.age = age;
this.eyeColor = eye;
this.getFullName2 = function() {
return this.firstName + this.lastName;
}
}Now let’s add a method name getFullName to the prototype of this function:
Person.prototype.getFullName = function() {
return this.firstName + this.lastName;
}now let’s initiate Person by new keyword
var hossein = new Person('hossein' ,'derakhshan', 32, 'black');
console.log(hossein.getFullName()); // hossein derakhshanas you can see when you create a new object from Person, that object has access to the getFullName method even though this method is not defined in Person function itself.
You might say that hey why not adding getFullName directly to the Person class itself just like getFullName2 . to get this answer let’s initiate another object from Person
var ali = new Person('ali' ,'rezai', 32, 'black');now let’s see compare getFullName2 which is defined inside a Person function with getFullName which is assigned to the prototype
console.log(hossein.getFullName2 === ali.getFullName2); // false
console.log(hossein.getFullName === ali.getFullName); // trueas you can see in this code when we compare getFullName2, in both initiated objects (Ali and Hossein) the result is false meaning it takes another space of the memory ,but when we compare getFullName the result is true ,which means both objects are pointed to the same memory address. So we are saving memory. we must consider when JS was invented, computers had very low memory.
PS: es6 classes, it is just a syntactic sugar or a convenient syntax over es5 syntax. under the hood, even if you use es6 classes, it is still using prototype and es5 syntax.
what is Reflect.apply?
Reflect.apply is a build-in javascript method which is used to call a function using the specified argument. It works similar to the Function.prototype.apply() method to call a function, but in an efficient manner and easy to understand.
Reflect.apply(targetFunction, thisArgument, argumentsList)Here is an example:
function func (a, b, c) {
this.x = a;
this.y = b;
this.z = c;
}const obj = {};Reflect.apply ( func , obj, [12,42,32] );
console.log( obj );
{ x: 12, y: 42, z: 32 }Now that we know the basic concepts here, let’s back to the topic :)
How Node.js fixed code collision?
CommonJS is the way that Node.js address code collision. when we run node file.js we are passing the javascript file to the node.js. we know that Node.js at the end, will pass this javascript file to the v8 and then will give us the result.
let’s assume that this is our node.js application:
utils.js
var msg = "Hello, World!";
var sayHi = function() {
console.log(msg);
};
module.exports = sayHi; // node.js has added this syntaxt to JSapp.js
var whatever = require('./utils.js') // node.js has added this syntaxt to JS
var msg = 'something'
whatever();we can run it with node app.js . we know that If node.js forward this javascript to v8 directly, we will get some errors. let’s do it in chrome console to see the error:
we are getting this error because parser in v8 , can not understand this syntax. so how do we not get any error in node.js?
in order to understand better, let’s just run node without providing any file name in a directory that these files(app.js, utils.js) exist.
If you run node in the console, you should see something like this depending on the version of your node.js
Now, if you type this command in the CLI require('./app'), you will see the same result as calling node ./app which means behind the scene node.js call the require function on the entry point.
The good news is require is a javascript function (it is not a pure C++ function), that’s why we can debug it :) let’s dive into the internal part of node.js and see what will happen after calling the require function.
On the right side, in scope section, we see some local variables like (exports, module, msg, require, __dirname, …) which is not exist in the app.js let’s step into the require function (you will get your answer).
depends on the version of your node.js, at the end you should see something like above picture.
let’s step into more:
as you can see in line 1020 require attached to the Module through prototype (line 1020). we already know what prototype do.
This line once again proves that require function has been called on app.js when we ran node app.js , and it created a Module class, and this class has access to the require function.
if we step into the Module.load we will see a lot of codes, but I tried to delete the lines that are not related to this topic. so the simpler version of module.load should be something like this:
Module._load = function(request, parent, isMain) {
const filename = Module._resolveFilename(request, parent, isMain);
const module = new Module(filename, parent);
module.load(filename);
return module.exports;
};let’s break down this function:
const filename = Module._resolveFilename(request, parent, isMain);this line is going to get the full file path on the local drive. in my case:
/Users/hossein/Tools/Learning/testt/utils.js
then let’s move to the next line:
const module = new Module(filename, parent);this is the constructed function that we are already familiar with its concept. So it should be a function with a name Module right? let’s see that
function Module(id = '', parent) {
this.id = id;
this.path = path.dirname(id);
setOwnProperty(this, 'exports', {});
moduleParentCache.set(this, parent);
updateChildren(parent, this, false);
this.filename = null;
this.loaded = false;
this.children = [];
}:if we run this function, it creates an object like this:
as you can see, it creates an empty exports: {} and also it has access to a lot of methods (like require and …) through prototype
Now let’s check this section:
module.load(filename);First of all, this line doesn’t return anything, meaning that it should add something to the module object itself. in other words, load should do some call-by-reference operation.
if we step into the load function we will see this:
On the right side of this module, there is a variable named extention and the value is .js
this gives us the idea that this class can load other types of data like json or . and if we don’t provide the .js in the require function, the default value would be .js . let’s check this line
Module._extensions[extension](this, filename);Based on this line,Module._extensions should be an object. here is its value:
what we are doing in that line is we are calling a function based on the extension. let’s dive more:
Simplified version of this function can be like this:
Module._extensions['.js'] = function(module, filename) {
var content = fs.readFileSync(filename, 'utf8');
module._compile(content, filename);
};This function is going to read the file from disk based on the filename. This is something that javascript can not do. but in Node.js there is a class named fs to work with files. fs.readFileSync is a combination of c++ and javascript, and now we don’t want to dive into this function. but all it does is it reads the file.
so content variable should be the content of the utils.js file:
‘var msg = “Hello, World!”;\nvar sayHi = function() {\n console.log(msg);\n};\nmodule.exports = sayHi;’
and finally, let’s step into the last and most interesting function guys. module._compile
this function is going to pass the compiled javascript into the v8 and execute it. I tried to simplify this function so the code should be something like this:
Module.prototype._compile = function(content, filename) {
const compiledWrapper = wrapSafe(filename, content, this);
const dirname = path.dirname(filename);
const require = makeRequireFunction(this, redirects);
let result;
const exports = this.exports;
const thisValue = exports;
const module = this;
result = ReflectApply(compiledWrapper, thisValue,
[exports, require, module, filename, dirname]);
return result;
};let’s break down this function and check the first line:
const compiledWrapper = wrapSafe(filename, content, this);based on the name (wrapSafe) , this function should add something to the currentcontent which was:
‘var msg = “Hello, World!”;\nvar sayHi = function() {\n console.log(msg);\n};\nmodule.exports = sayHi;’
I tried to simplify the wrapSafe function for you.
All it does is just add two strings into our script
let wrap = function(script) {
const wrapper = [
'(function (exports, require, module, __filename, __dirname) { ',
'\n});',
];
return wrapper[0] + script + wrapper[1];
};if we run this function, we will get this result:
(exports, require, module, __filename, __dirname) {
var msg = "Hello, World!";
var sayHi = function() {
console.log(msg);
};
module.exports = sayHi;
})See? it just wraps our code with a function to avoid the code collision. but this function is not executed yet. in fact, ReflectApply is going to execute our codes by passing this function to v8 and get back the result.
result = ReflectApply(compiledWrapper, thisValue,
[exports, require, module, filename, dirname]);let’s jump back to the image that you already saw at the beginning of this article:
All the variables (exports, module, msg, require, __dirname) we saw in local scope come from the wrapFunction:
(exports, require, module, __filename, __dirname) {})
let’s see the module argument. as you might remember module was an object like this with exports = {}
we are passing this object to this function and then we run the function with ReflectApply .
We are passing this object in the inner function because if we assign something to the module.exports in the inner function, we want access to it from outside of the wrapped function.
let’s simplify this:
var obj = {
exports: {}
}
((module) => {
var msg = "Hello, World!";
var sayHi = function() {
console.log(msg);
};
module.exports = sayHi;
})(obj)
if (obj.exports) {
obj.exports()
}
I just created an object named obj then I created an IIFE and I passed the obj object into IIFE and I put a name as a module. In the inner function, I assigned sayHi function in module.exports . since I’m changing the reference of exports, outside of the IIFE I still have access to the module.exports !
If we check the last line of require function, we will see that it returns the module.exports
Module._load = function(request, parent, isMain) {
const filename = Module._resolveFilename(request, parent, isMain);
const module = new Module(filename, parent);
module.load(filename);
return module.exports; // <<<<<<<<<< this line
};That’s pretty much it! Love you and stay tuned :)
