Arduino Remote Uploader

11 min readMar 22, 2015

I started this project about two months ago (late January 2015). There’s still a lot of work remaining to get where I want it to be, but I feel it’s in a good state to be useful for others. As the title implies, this project is for remote programming of Arduinos. Here’s the backstory.

The requirements are minimal:

The Arduino to be programmed must be an Atmega 328/168 with the Optiboot bootloader. Optiboot is a superior bootloader: it is fast at 115 kbit/s, and has a simplified protocol. Optiboot is the bootloader on all Arduino UNOs and can be installed on any Atmega 328/168 Arduino, even the Diecimila from 2007. The Arduino Leonardo is not compatible as it uses the ATmega32u4 and relies on usb-serial for programming. Optiboot can be installed using a second Arduino with ArduinoISP, or using an AVR programmer, such as the AVR Pocket Programmer from Sparkfun.
A 256kbit Mircochip EEPROM (24LC256-I/P). These are available from a variety of suppliers for about $1. The 512kbit version also compatible.
A second Arduino. An Arduino cannot program itself so another Arduino is needed. This can be any Arduino. An Arduino can be assembled on a breadboard/protoboard for under $5, and Arduino clones are available for under $3 on Ebay, so adding another Arduino does not add much cost.

Currently this project supports XBee (Series 1 & 2), Wifi (ESP8266) and Nordic nRF24L01+. The software is available on github

How it works

There are three main software components to this solution

Host Uploader. A Java command-line application that uploads the sketch from the Arduino IDE to the programmer Arduino.
Arduino Sketch. This sketch runs on the programmer Arduino. It’s responsible for receiving the sketch over the network (XBee, Nordic, wifi, bluetooth, ethernet, RFM22B, etc). It invokes the RemoteUploader library to do the rest.
RemoteUploader Arduino library. This library stores the program on the EEPROM and flashes the other Arduino. It doesn’t know how the sketch was transferred.

The programmer Arduino receives the sketch and writes it to the EEPROM. Once the entire sketch is received the programmer Arduino resets the application Arduino, reads the sketch from the EEPROM and sends Optiboot commands to flash the application Arduino. In this case I’m using an XBee to receive the sketch but you could also use a Nordic, wifi, ethernet or any other Arduino compatible network device.

This solution has error handling to support remote programming over weak connections. The host uploader sends each packet of programming data with an id (memory address) and waits for an ACK. The ACK includes a code that is either OK, or an error code (e.g. FLASH_ERROR). If it doesn’t receive an ACK within the timeout period, it retries. It will continue to do this until it receives an ACK or reaches the max retries and gives up. I’ve tested several remote uploads at the extreme ranges of XBee and Nordic and they successfully complete each time; of course it takes longer due to retries, but as long as there is a signal, it will eventually succeed.

The RemoteUploader Arduino library is a core component of this solution. It handles interfacing with the EEPROM and flashing the other Arduino using the Optiboot protocol. It is 100% independent of the network device that receives the sketch.

Below I’m using an XBee to transmit the program to the remote Arduino. The XBee is connected to programmer Arduino via SoftSerial. This is necessary since the hardware serial port is wired to the other Arduino.

Wiring

Pay careful attention to the wiring of the Microchip EEPROM. It is an I2C device and I2C pins vary depending on the Arduino. In this case I’m using a Leonardo, which uses pins 2 (SDA) & 3 (SCL). Arduino UNO and 328 based atmegas use A4 (SDA) & A5 (SCL). Refer to the Arduino Wire documentation for more information.

The tiny dot on the chip indicates pin 1. Additionally there is a notch on the pin 1 side. Wire the pins as follows

pin 1 (A0)            -> GND
pin 2 (A1)            -> GND
pin 3 (A2)            -> GND
pin 4 (Vss)           -> GND
pin 5 (SDA)           -> Arduino SDA
pin 6 (SCL)           -> Arduino SCL
pin 7 (Write Protect) -> Unconnected
pin 8 (Vcc)           -> Arduino 5V (or 3.3V if 8Mhz)Programmer Arduino pin 9 -> Reset pin on application Arduino

Here’s a remote uploader XBee configuration with two Arduino Pros

Unfortunately the XBee shield can’t be used as designed because the serial port is wired to the other Arduino for uploading firmware.

Setup

As mentioned earlier, the Arduino being programmed (application) must be an Atmega 328/168 running the Optiboot bootloader. I use version 5.0a, but the Optiboot that ships with the Arduino IDE (v4.4) should also work. Here’s a guide for installing Optiboot. If you’re using a Arduino UNO, it already has Optiboot (v4.4).

There are several Arduino libraries required for interfacing with the EEPROM, XBee or Nordic radios and uploading Arduino sketches. Installing an Arduino library simply involves copying the folder to the Arduino libraries folder.

Install the extEEPROM Arduino library https://github.com/JChristensen/extEEPROM

If using XBee, download and install the xbee-api Arduino Library https://github.com/andrewrapp/xbee-arduino

If using Nordic, download and install the RF24 Arduino Library https://github.com/maniacbug/RF24

If using Wifi, download and install my Esp8266 Arduino Library https://github.com/andrewrapp/Esp8266

Download the source code for this project from github and install the RemoteUploader library.

Afterwards the Arduino library folder should resemble the following (minus the ds1302, LCD4x20, and Time libraries)

XBee (Series 2)

If using XBees for remote programming, first they will need to be configured. The code currently only supports Series 2 XBees in API mode. One radio will need to be configured as a coordinator and another as a router. For the router I used ZigBee XB24-ZB Router API firmware, version 23A7, and for the coordinator: ZigBee XB24-ZB Coordinator API firmware, version 21A7.

The minimal configuration to apply is as follows. Assumes the default factory settings. If not installing firmware then do a factory restore first. Here’s a helpful guide for configuring the radios https://code.google.com/p/xbee-api/wiki/XBeeConfiguration

Router:

AP=2
ID=1AAA (PAN ID. choose any, just needs to be the same on both)
NI=ROUTER (optional)

Take note of the 64-bit address (SH + SL)

Coordinator:

AP=2
ID=1AAA (PAN ID. choose any, just needs to be the same on both)
NI=COORDINATOR (optional)

Also take note of the 64-bit address (SH + SL)

Make sure you don’t have another coordinator using the same PAN ID as only one coordinate per PAN ID is supported.

Refer to the wiring diagram above and connect the router XBee to the SoftSerial ports of the programmer Arduino.

Connect the coordinator XBee to the host PC, using an XBee Explorer or similar usb-serial device.

Open the XBeeUploader sketch. There are a few changes to make. Specify the XBee 64–bit address of the coordinator. This is the SH + SL of your radio, not mine!

//SH
const uint32_t COORD_MSB_ADDRESS = 0x0013a200;
//SL
const uint32_t COORD_LSB_ADDRESS = 0x408b98fe;

The sketch needs the address to send ACKs. Of course it would be easier to get the address from the packet, but this provides a little bit of security in that it can’t be programmed from just any radio on the network.

Initialize the RemoteUploader library with the Serial port eeprom and reset pin. If using Leonardo the serial port will always be &Serial1; when using atmega 328/168, always use &Serial. Arduino Megas could use other ports. In hindsight I could have defaulted the port by detecting the Arduino hardware.

remoteUploader.setup(&Serial, &eeprom, RESET_PIN);

Upload the XBee upload sketch the the programmer Arduino

Now download the host uploader software. For simplicity I’ve provided a binary download as it’s convoluted to build currently as I haven’t yet published to Maven repositories. It uses RXTX for serial communication and since RXTX is an abandoned project, you’ll need to use a 32-bit Java version as there is no 64-bit version provided. On Mac the Apple version of Java (Java 6) is the only one that works since Java 7 & 8 from Oracle do not support 32-bit mode.

Find a sketch to upload wirelessly. Here I’ve written a simple sketch to blink the onboard Arduino LED

Select the board. Only Atmega 168/328 with the Optiboot is supported, so select that. Open Arduino preferences and select show verbose output during compilation, and might as well also check it for upload. Click Verify. This compiles the sketch. The location of the compiled sketch is displayed in the console below the sketch.

Run the uploader script, specifying the location of the compiled sketch, the serial port of the local XBee usb-serial, and the address of the Router XBee radio (SH + SL). First make scripts executable with chmod u+x *.sh

./xbee-uploader.sh --sketch /var/folders/g1/vflh_srj3gb8zvpx_r5b9phw0000gn/T/build1609364047445626536.tmp/Blink.cpp.hex --serial-port /dev/tty.usbserial-A6005uRz --baud-rate 9600 --remote-xbee-address “0013A200408B98FF”

If everything works you should see output similar to

Sending sketch to xbee radio, size 1102 bytes, md5 8e7a58576bdc732d3f9708dab9aea5b9, number of packets 18, and 64 bytes per packet, header ef,ac,10,a,4,4e,0,12,40,3c
………………Successfully flashed remote Arduino in 6s, with 0 retries

Look at your application Arduino. Is it blinking? Sweet! The Arduino IDE compiles the sketch to the same location, so to make changes just recompile run the command again.

Run with the help option to see the full list of arguments. For example

./xbee-uploader.sh --help-a,--arduino-timeout <arg> How long Arduino waits for a command before it exits programming mode (in seconds). Default is 60. Note: This is not the time that programming must complete in, only the max time between commands
-b,—-baud-rate <arg> Baud rate of local xbee (host) radio serial port. Required
-c,--ack-timeout-ms <arg> How long to wait for ACK from Arduino before retrying (in milliseconds). Default is 5000
-d,--verbose Show debug output
-h,--help Show help
-p,--serial-port <arg> Serial port of of local xbee (host radio) (e.g. /dev/tty.usbserial-A6005uRz). Required
-r,--retries <arg> How many times a command is retried before giving up. Default is 10
-s,--sketch Path to compiled sketch (compiled by Arduino IDE). Required
-x,--remote-xbee-address <arg> Address (64-bit) of remote XBee radio (e.g. 0013A21240AB9856)

Nordic nRF24L01+

The steps for remote flashing with Nordic are very similar. Refer to this guide for wiring the Nordic to the programmer Arduino. One difference is the guide uses pins 9/10 for CE/CS and this software defaults to pins 10/11, although that can be changed below. Remember that the SPI pins vary for different Arduinos, so make sure you are using the correct pins! You can find SPI pins in the Arduino SPI documentation.

The the Nordic is not a serial device so a second Arduino is needed to convert from serial to SPI. I’ve written a sketch with handles this translation. On the Raspberry Pi it should be possible to use the SPI capabilities to interface directly with the Nordic radio.

The sketch expects the Nordic CE/CS pins to be connected to the Arduino as follows:

#define NORDIC_CE 10
#define NORDIC_CS 11

Update if you have it wired differently.

I chose an arbitrary address and channel. This can be changed however it will also need to be updated in the NordicSerial2SPI sketch

uint64_t baseAddress = 0xca05cade05LL;
const uint64_t pipes[2] = { baseAddress, baseAddress + 1 };

The remaining defaults should be fine. Upload the Nordic programmer to the programmer Arduino.

Open the NordicSerial2SPI sketch. Perform the same updates (CS/CE pins), if necessary. Upload to the host connected Arduino.

To upload run the nordic-uploader script. (First make it executable with chmod u+x *.sh)

./nordic-uploader.sh --sketch /var/folders/g1/vflh_srj3gb8zvpx_r5b9phw0000gn/T/build1609364047445626536.tmp/Blink.cpp.hex --serial-port /dev/tty.usbmodemfa131 --baud-rate 19200Sending sketch to nRF24L01 radio, size 1102 bytes, md5 8e7a58576bdc732d3f9708dab9aea5b9, number of packets 43, and 26 bytes per packet, header ef,ac,10,a,4,4e,0,2b,1a,3c
…………………………………….
Successfully flashed remote Arduino in 3s, with 0 retries

Isn’t that fun! Similar to the xbee uploader to see the full list of arguments run

./nordic-uploader.sh --help

Wifi (ESP8266)

Follow this guide for wiring and to update the ESP8266 firmware version. If the ESP is not configured to connect to an access point, uncomment the following and specify your wifi ssid/password

if (esp8266.configure(“ssid”,”password”) != SUCCESS) {
 if (COMMUNICATION_TEST) {
  Serial.println(“Configure failed”);
 }
 
 for (;;) {}
}

Open the serial console to see the output. Note the ip address of the device, or you could also find from looking at the dhcp clients in your router. On linksys my device appears as

192.168.1.115 18:FE:34:9B:A7:4C

Upload a sketch, specifying the ip address of the ESP.

./wifi-uploader.sh -d — host 192.168.1.115 — port 1111 -s /var/folders/g1/vflh_srj3gb8zvpx_r5b9phw0000gn/T/build3190504561153569118.tmp/Blink.cpp.hexSending sketch to wifi radio, size 1126 bytes, md5 c571f5e98456711d6b61d1a846dfe7a1, number of packets 44, and 26 bytes per packet, header ef,ac,10,a,4,66,0,2c,1a,3c
Sending page 1 of 44, with address 0, length 32, packet ef,ac,20,20,0,0,c,94,61,0,c,94,7e,0,c,94,7e,0,c,94,7e,0,c,94,7e,0,c,94,7e,0,c,94
...
Sending flash packet ef,ac,40,6,4,66
Successfully flashed remote Arduino in 8s, with 0 retries

Fault Tolerance

This solution has handles failed transmissions with retries. Here’s an example of a XBee programming attempt that succeeds after several failed packets.

Sending sketch to xbee radio, size 1102 bytes, md5 8e7a58576bdc732d3f9708dab9aea5b9, number of packets 18, and 64 bytes per packet, header ef,ac,10,a,4,4e,0,12,40,0
…….
Failed to deliver packet [page 7 of 18] on attempt 1, reason No ACK from transport device after 5000ms.. retrying
..
Failed to deliver packet [page 8 of 18] on attempt 1, reason No ACK from transport device after 5000ms.. retrying
…
Failed to deliver packet [page 10 of 18] on attempt 1, reason No ACK from transport device after 5000ms.. retrying
……
Failed to deliver packet [page 15 of 18] on attempt 1, reason No ACK from transport device after 5000ms.. retrying
….
Failed to deliver packet [page 18 of 18] on attempt 1, reason No ACK from transport device after 5000ms.. retrying
.
Successfully flashed remote Arduino in 33s, with 5 retries

Troubleshooting

The sketch can fail to upload for a variety of reasons. First verify all the pins are wired correctly. The EEPROMs have a tendency to not sit tight on breadboards so you might want to a couple different locations. Also some breadboards have weak/broken connections so try moving the devices around and double check all the pins are seated tight.

If the failure is on the Arduino and is not network related, the host uploader will display an error code (e.g. EEPROM_WRITE_ERROR). If the error code is helpful in resolving the issue, the best bet is to enable the DEBUG preprocessor directive in RemoteUploader.h. This is only possible however when using an Arduino with a second serial port. The Leonardo is a great choice since it has a built-in usb-serial port. When using an atmega 328/168, the only debug option would be to use SoftSerial since they only have one serial port which is hardwired to the other Arduino.

Parts

The Microchip 256 KBit EEPROM 24LC256-I/P is available from Digikey for $1 http://www.digikey.com/product-detail/en/24LC256-I%2FP/24LC256-I%2FP-ND/273431 also Sparkfun carries it as well.
XBee radios are available at Sparkfun and the big distributors (Digikey, Mouser etc).
The ESP8266 is available at Sparkfun and also Chinese sellers on ebay. Expect a few weeks or more for delivery to US from China.
The Nordic nRF24L01+ is available from Addicore for about $6 and about a $1 shipped on ebay.

What’s Next

At this point the first phase of the project is feature complete. I’ll continue to exercise it some more and fix bugs as they crop up. Eventually I’d like to support uploading directly from the Arduino IDE. This could be accomplished by using virtual com ports that proxy to the host uploader, or by modifying the Arduino IDE, similar to how Teensy works. Either option would need to emulate Optiboot, to trick the IDE into thinking it’s talking directly to an Arduino.