Elixir Nerves: Setting up network, DNS, OTA upgrades, SSH access and Debugging— Part 2

This follows the previous part Building a product based on an embedded device with Elixir, Nerves & Phoenix.

The first post was mainly an introduction to the problem we had to solve and how we did it. Now it’s time to go deeper in details!

The first thing we did on the device was to try some examples. That was nice but we had to move fast to meet our first deadline, and that’s when we had to face the fact that the device wasn’t connected to the network, we couldn’t connect to the device and we had no idea what was going on once it was powered on.

On the libraries page, we found nerves init gadget which as it is said in the github page:

* Basic network initialization for USB gadget devices (Raspberry Pi Zero and B+ eaglebone) and wired and wireless Ethernet

* mDNS support to advertise a name like nerves.local or nerves-1234.local if devices have serial numbers

* Device detection, filesystem mounting, and basic device control from nerves_runtime

* Over-the-air firmware updates using nerves_firmware_ssh

* Easy setup of Erlang distribution to support remsh, Observer and other debug and tracing tools

* Access to the IEx console via ssh and transfer files with sftp

* IEx helpers for a happier commandline experience

* Logging via ring_logger

* shoehorn-aware instructions to reduce the number of SDCard reprogrammings that you need to do in regular development.

Ok great, now we could have some very nice features, so let’s go!

This is the config for nerves init gadget:

config :nerves_init_gadget,
  ifname: "eth0",
  address_method: :dhcp,
  node_name: "my_domain",
  node_host: :mdns_domain,
  ssh_console_port: 22

Nerves init gadget uses nerves network so it can be configured easily following the docs:

config :nerves_network, :default,
  eth0: [
    ipv4_address_method: :dhcp
  ]

Great! and it’s also possible to connect using a WiFi AP:

config :nerves_network, :default,
  wlan0: [
    ssid: System.get_env("NERVES_NETWORK_SSID"),
    psk: System.get_env("NERVES_NETWORK_PSK"),
    key_mgmt: String.to_atom(key_mgmt)
  ]

Don’t forget to change ifname: "eth0" to ifname: "wlan0" -or the appropiate interface name for the wlan connection- on config :nerves_init_wadget if you want to use a wireless connection. Also keep in mind that environment variables NERVES_NETWORK_SSID and NERVES_NETWORK_PSK should be set when building the firmware.

…and voila! after burning the new firmware into the device, it’s possible to connect via SSH or even upload a firmware! But wait! what is the IP of the device?

We started looking in the router, get the IP and use it:

ssh <ip_of_the_device>

or for OTA upgrades:

clear && mix firmware && mix firmware.push <ip_of_the_device>

We are going to use mDNS to expose the ip through a local domain adding mdns_domain: "<my_domain>.local" to the configuration. If you use an Apple device, the address <my_domain>.local is available without further configuration. In my case, since I use a Linux box, I had to install avahi.

Here we go! The device is running and accessible using:

ssh <my_domain>.local

and we are into an IEx console!

Note: for exiting the console, you need to use the sequence ~. that is a ssh escape sequence.

nerves-init-gadget provides Ring Logger which is an in-memory ring buffer backend for the Elixir Logger

For enabling ring logger, we need to add or edit config.exs and makse sure our logger is configured like this:

config :logger, backends: [RingLogger]

this time let’s use the OTA upgrade for pushing the new firmware with Ring Logger enabled:

clear && mix firmware && mix firmware.push <my_domain>.local

once it has been uploaded, we can ssh into the IEx shell and attach to the logger:

ssh <my_domain>.local
iex(derberos@derberos.local)1> RingLogger.attach

As you can see in the previous code, my app name and domain name is derberos , you should see your own name here.

If you want to see the previous logged messages since the app started:

iex(derberos@derberos.local)1> RingLogger.tail

Nice, we have now a pretty decent device to start working on. We could connect a screen using the HDMI port, but maybe you don’t have one available or you prefer to view the full output of the device in the console.

For this you need a USB to TTL serial cable like this one:

USB to TTL cable for Raspberry PI

One of the ends has 4 cables:

• Red: power
• Black: ground
• White: RX (receiving)
• Green: TX (transmission)

Red and black are easy (5V PWR & ground), and what you have to take into consideration is that white cable receives data, so it must be connected to the transmission pin of the device. And the green cable sends data, so it must be connected to the receiving pin on the raspberry.

For a Raspberri PI 3, this is the diagram of the GPIO ports:

Raspberry PI 3 GPIO ports

The green cable goes to pin 10, and white cable goes to pin 8. Here is a picture of my prototype device.

Photo of the connections of a Raspberry PI3 to a UART cable

Once you had connected the device to the computer, you need a program for serial communication, I usually use picocom or minicom, but there are many available for every platform.

Before connecting to the device, we need to find the USB id. Connect the cable to the device and the computer, and then, you should see the id under/dev/serial/by-id

ls /dev/serial/by-id/

It’s important to set the the baud rate for the communication to 115200.

Now you are ready to see your device output with one of these two commands:

picocom -b 115200 /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller-if00-port0

or

minicom -b 115200 -o -D /dev/serial/by-path/pci-0000:00:1d.0-usb-0:1.6:1.0-port0

If the Raspberry Pi was already connected and powered on, maybe you can’t see anything because nothing is happening, but you can restart it and check the boot-up sequence:

iex(derberos@derberos.local)1> Nerves.Runtime.reboot

Here an example of the first second of the bootup:

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.9.80 (antares@escorpion) (gcc version 7.3.0 (crosstool-NG 1.23.0.418-d590) ) #2 SMP PREEMPT Mon Aug 6 21:04:15 CEST 2018
[ 0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[ 0.000000] CPU: div instructions available: patching division code
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt:Machine model: Raspberry Pi 3 Model B Plus Rev 1.3
[ 0.000000] cma: Reserved 8 MiB at 0x33800000
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] percpu: Embedded 14 pages/cpu @b3060000 s25804 r8192 d23348 u57344
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 211120
[ 0.000000] Kernel command line: 8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 822992K/851968K available (7168K kernel code, 414K rwdata, 1588K rodata, 1024K init, 473K bss, 20784K reserved, 8192K cma-reserved)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000–0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000–0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xb4800000–0xff800000 (1200 MB)
[ 0.000000] lowmem : 0x80000000–0xb4000000 ( 832 MB)
[ 0.000000] modules : 0x7f000000–0x80000000 ( 16 MB)
[ 0.000000] .text : 0x80008000–0x80800000 (8160 kB)
[ 0.000000] .init : 0x80b00000–0x80c00000 (1024 kB)
[ 0.000000] .data : 0x80c00000–0x80c678b8 ( 415 kB)
[ 0.000000] .bss : 0x80c69000–0x80cdf628 ( 474 kB)
[ 0.000000] SLUB: HWalign=64, Order=0–3, MinObjects=0, CPUs=4, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] arm_arch_timer: Architected cp15 timer(s) running at 19.20MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[ 0.000007] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[ 0.000023] Switching to timer-based delay loop, resolution 52ns
[ 0.000303] Console: colour dummy device 80x30
[ 0.001634] console [tty1] enabled
[ 0.001690] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[ 0.001780] pid_max: default: 32768 minimum: 301
[ 0.001985] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.002042] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.003115] Disabling memory control group subsystem
[ 0.003229] CPU: Testing write buffer coherency: ok
[ 0.003306] ftrace: allocating 20926 entries in 62 pages
[ 0.048159] CPU0: update cpu_capacity 1024
[ 0.048217] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.048285] Setting up static identity map for 0x100000–0x100034
[ 0.218224] CPU1: update cpu_capacity 1024
[ 0.218232] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.288328] CPU2: update cpu_capacity 1024
[ 0.288335] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[ 0.358450] CPU3: update cpu_capacity 1024
[ 0.358457] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[ 0.358561] Brought up 4 CPUs
[ 0.358777] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[ 0.358816] CPU: All CPU(s) started in HYP mode.
[ 0.358849] CPU: Virtualization extensions available.
[ 0.359712] devtmpfs: initialized
[ 0.371864] VFP support v0.3: implementor 41 architecture 3 part 40 variant 3 rev 4
[ 0.372217] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.372293] futex hash table entries: 1024 (order: 4, 65536 bytes)
[ 0.372885] pinctrl core: initialized pinctrl subsystem
[ 0.373901] NET: Registered protocol family 16
[ 0.376168] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[ 0.385460] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[ 0.385520] hw-breakpoint: maximum watchpoint size is 8 bytes.
[ 0.385699] Serial: AMBA PL011 UART driver
[ 0.387865] bcm2835-mbox 3f00b880.mailbox: mailbox enabled
[ 0.388547] uart-pl011 3f201000.serial: could not find pctldev for node /soc/gpio@7e200000/uart0_pins, deferring probe
[ 0.389211] irq: no irq domain found for /soc/aux@0x7e215000 !
[ 0.436506] bcm2835-dma 3f007000.dma: DMA legacy API manager at b480d000, dmachans=0x1
[ 0.438466] SCSI subsystem initialized
[ 0.438666] usbcore: registered new interface driver usbfs
[ 0.438781] usbcore: registered new interface driver hub
[ 0.438920] usbcore: registered new device driver usb
[ 0.448520] raspberrypi-firmware soc:firmware: Attached to firmware from 2018–03–21 14:52
[ 0.450711] clocksource: Switched to clocksource arch_sys_counter
[ 0.492303] FS-Cache: Loaded
[ 0.492596] CacheFiles: Loaded
[ 0.506100] NET: Registered protocol family 2
[ 0.506938] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 0.507086] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 0.507306] TCP: Hash tables configured (established 8192 bind 8192)
[ 0.507432] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 0.507508] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 0.507768] NET: Registered protocol family 1
[ 0.508466] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 7 counters available
[ 0.511070] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[ 0.527511] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.532799] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[ 0.532988] io scheduler noop registered
[ 0.533025] io scheduler deadline registered
[ 0.533362] io scheduler cfq registered (default)
[ 0.539128] BCM2708FB: allocated DMA memory f3900000
[ 0.539188] BCM2708FB: allocated DMA channel 0 @ b480d000
[ 0.547185] Console: switching to colour frame buffer device 82x26
[ 0.554021] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[ 0.558670] bcm2835-rng 3f104000.rng: hwrng registered
[ 0.560998] vc-mem: phys_addr:0x00000000 mem_base=0x3ec00000 mem_size:0x40000000(1024 MiB)
[ 0.565902] vc-sm: Videocore shared memory driver
[ 0.583794] brd: module loaded
[ 0.595498] loop: module loaded
[ 0.597615] Loading iSCSI transport class v2.0–870.
[ 0.600438] spi-bcm2835 3f204000.spi: could not get clk: -517
[ 0.602858] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[ 0.832900] Core Release: 2.80a
[ 0.834869] Setting default values for core params
[ 0.836853] Finished setting default values for core params
[ 1.039191] Using Buffer DMA mode

This helps a lot when something breaks the Erlang VM and you can’t ssh into the IEx console and take a look at Ring Logger. Usually this happens with issues starting processes.

And that’s it. We have accomplished quite a few things in this post, in the next posts I’ll keep talking about how to configure the raspberry pi to be an useful device with a purpose.

Don’t miss part 3, coming soon: Using OTP Genservers on nerves to publish the device IP to a custom domain.