Getting started on NodeMCU with your own Firmware

Several times a year we have our Rabbiteer sessions at Retro Rabbit where we present cool stuff related to all areas of software development. This year I’m going to be presenting the IoT (internet of things) session, with the topic “IoT on the cheap”.

What we did is get a bunch of cheap micro controllers from china (via AliExpress). The microcontrollers we got are the popular-ish NodeMCU development boards.

Furthermore, to make things go as smooth as possible I opted to build our own firmware. There is an online build service, but I wanted to build my own for two main reasons. Firstly, I wanted to understand the firmware and select exactly the modules I wanted, and be able to inspect the source code easily if needed. Secondly, I was afraid that some part of the chinesium open source software would be broken and I at least wanted the opportunity to fix it if possible (and lo I did find bugs in the TLS/HTTP code. I did not end up fixing it though).

In this article I’ll run through the steps to compiling your own firmware and uploading it.

Prerequisites

Firstly, you’ll need Linux, and my instructions will use Ubuntu. If you’re a Windows person I recommend the Linux Subsystem for Windows, which is actually what I’ll be using.

You’re going to need a butload of stuff to compile the compiler and the firmware. This should cover everything (more info here and here):

sudo apt-get install -y git autoconf build-essential gperf bison \
flex texinfo libtool libncurses5-dev wget gawk \
libc6-devpython-dev python python-serial libexpat-dev unzip \
zip gdb make unrar-free automake gcc g++ libc-dbg \
ncurses-dev expat lua5.1 lua5.1-doc luarocks sed git bash \
help2man bzip2 libtool-bin

Jikes!

You’ll also need a bunch of Lua dependencies:

sudo luarocks install bitlib 
sudo luarocks install luafilesystem
sudo luarocks install md5
sudo luarocks install luaposix
sudo luarocks install luasocket

Compiling the Compiler

We’re going to need Xtensa toolchain to compile for the Tensilica processor inside the ESP2866 which is the brain of the NodeMCU. In order to do this we need to clone the repo:

cd ~
git clone --recursive https://github.com/pfalcon/esp-open-sdk

And build it:

cd esp-open-sdk
make STANDALONE=y

This is going to take a long time (and download a ton of stuff along the way). On my laptop it takes about an hour.

In the Meanwhile: Drivers (Windows only)

While the compiler is compiling we can use the time to install the drivers. If you’re a Linux user check out what’s what over here, but good luck, I hear people have problems (I’m guessing because drivers+linux=😭).

On Windows it was pretty easy for (and I’ve done this with two different NodeMCUs with different USB Serial interfaces).

So plug in your NodeMCU, and head over to the Device Manager. Easy way is to right click on the windows button or press windows key+X.

You’ll see that the device is there but windows doesn’t know what it is.

Double click on it and head over to the Drivers tab. Click on Update Driver:

Update Driver. Please ignore the fact that my drivers are already installed.

The be sure to select the top button to search windows update:

Searching windows update

It should search for a while and then install the drivers. No downloading dodgy drivers from some weird website.

Cool man

In the Meanwhile: Connecting with PuTTY (Again, Only Windows Users will Care)

Another bonus item while the compiler is building: connecting with PuTTY:

Note in your device manager what the port is (mine is COM8 like you see above). And open up PuTTY. Select Serial and enter the port:

Enter the baud rate of 115200 (may not work for you, but it worked for the two different kinds of NodeMCUs I have).

Next go to the serial category on the side and set the settings like this (btw I did this by trial and error):

Very specific options

Now you should be able to open the terminal and send commands:

Note: If nothing happens it’s likely that your controller does not have the NodeMCU firmware on it (again, both of mine were like this). Press the reset button while PuTTY is open and see what it spits out.

Compiling the Firmware

Once the toolchain is done compiling we need to add it to the path by adding these lines to your ~/.bashrc:

export PATH="$HOME/esp-open-sdk/xtensa-lx106-elf/bin/:$PATH"
alias xgcc="xtensa-lx106-elf-gcc"

(run these commands in the console as well so we can move on without logging out and in again)

now we can move on to compiling the firmware (with the toolchain).

Clone the firmware:

cd ~
git clone https://github.com/nodemcu/nodemcu-firmware.git
cd nodemcu-firmware

And then:

make

This should only take a minute or two. Verify that the firmware is built by checking the bin folder:

ls ./bin/
0x00000.bin 0x10000.bin

If it worked it will create those two files.

Customizing the Firmware

The above should prove whether or not the firmware builds without any modification. Typically we will want to specify the modules we need. To change this, edit the file app/config/user_modules.h. For example, if we want to enable support for a DHT11 temprature/humidity sensor, we comment out this line:

//#define LUA_USE_MODULES_CRON
//#define LUA_USE_MODULES_CRYPTO
#define LUA_USE_MODULES_DHT
//#define LUA_USE_MODULES_DS18B20
//#define LUA_USE_MODULES_ENCODER

And then run make again.

We can also modify the firmware name and build date by editing user_version.h in the same directory:

...
#define NODE_VERSION "NodeMCU 2.1.0 Rabbiteer Edition"
#ifndef BUILD_DATE
#define BUILD_DATE __DATE__ __TIME__
#endif
...

This is what I set my version to.

Uploading the Firmware

To upload the firmware we’ll use esptool. This can easily be obtained by running sudo pip install esptool (LSW Windows users: you’ll have to run this from outside Linux since it does not (currently) have access to serial ports).

To upload the firmware, run (linux):

esptool.py --port <port> write_flash -fm qio \
0x00000 0x00000.bin \
0x10000 0x10000.bin

Where <port> is the serial port device (e.g. /dev/ttyUSB0 or something). On windows CMD it will typically be something like:

esptool --port COM8 write_flash -fm qio ^
0x00000 0x00000.bin ^
0x10000 0x10000.bin

(note no .py extension)

And then your firmware will be updated!

esptool --port blah blah blah
esptool.py v2.1
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0040
Compressed 30912 bytes to 22217...
Wrote 30912 bytes (22217 compressed) at 0x00000000 in 2.0 seconds (effective 126.2 kbit/s)...
Hash of data verified.
Compressed 575040 bytes to 391216...
Wrote 575040 bytes (391216 compressed) at 0x00010000 in 34.4 seconds (effective 133.6 kbit/s)...
Hash of data verified.
Leaving...
Hard resetting...

And if we PuTTY into it and press the reset button:

There we go! Our very own firmware

And that’s all there is to it! Now you can hack your way through the firmware to your heart’s content.