Ensuring stability on the Rock64 with a full Iota node

About a month ago I got the full IRI node running on the Rock64. Lots of people asked me to post statistics of the current node.

But the truth is that I still struggle to make it run stable. As I already told in my previous story, I had no Linux nor real programming experience at all when I started this journey. This makes it particularly difficult to debug what’s going on when something crashes.

But slowly but steady I’m getting closer to my goal of running a stable full node on the Rock64. To give you all an idea of why it takes so long, and share my experiences so far, I decided to write it down, so that nobody has to undertake the same tedious things I did.

The first problems

The problem I got at first was that my node froze at random times. I did not get any error message in the IRI output, and a reboot of the device was required to get it back up running again.

At first I blamed 2 things.

  • I did not install a heatsink. The CPU felt hot so overheating was surely a possibility.
  • I also used an old HDD. On boot I got some cluster errors so I figured some corrupt HDD clusters might cause the problem as well.

I installed a heatsink and a new HDD. The CPU temperature felt better and printing the CPU temperature every x seconds to a file confirmed that it didn’t overheat. The log showed about 50 °C moments before it froze. With the installation of the new HDD the corrupt cluster warning was also gone so that couldn’t be the problem anymore either.

But still, after these adjustments, the Rock64 kept freezing at random times. At this point, I started to wonder if it might be hardware related, or even worse, a Rock64 wouldn’t be strong enough to run a full node. I ran a memory test to find out if I had some bad memory in the Rock64, but everything was ok.

Then I met a user named maeck in discourt who also ran a full Iota node based on my Rock64 tutorial, on another 4GB device. It wasn’t a Rock64, but specs were almost similar so I figured the Rock64 should be powerful enough to run it.

Progress

I couldn’t figure out what was going on until my Rock64 suddenly froze while I was not running the node. The system was idle at this point. So this made me think that the problem was not related to java or IRI at all. I rebooted and left it idle for another day. I got the same result. It froze after a couple of hours.

I decided to detach everything and slowly attach hardware again.

It turned out that when the HDD was disconnected it ran fine, but with the HDD attached it crashed. Although I was sad that my HDD didn’t work well, I also was a bit relieved because I could also just use a micro SD card instead.

Get rid of the HDD

So my next try was to put a 16GB SD card in and install the Iota node on there. It installed fine and the freezing problem was gone. Since that day the rock hasn’t frozen anymore.

But a different problem surfaced: The node did not sync anymore. I tried various things, but the Latest Solid Milestone was never synced properly. Sometimes it did not sync at all, and sometimes it tried syncing but crashed after some minutes/hours. This resulted in corrupt data or the need of a rescan and start from scratch again.

The final pieces fall in place

It got to a point where I became frustrated. It feels like a jigsaw and you hold all the pieces, but you just cannot figure out where to put them. I think the problem with the SD card is read/write speed, but I can’t tell for sure. At this point I suddently realised I didn’t try to use an SD Card and an HDD, but leave the root and boot sequence at the SD card.

So I again wiped everything, and started from scratch. This time I used the SD card to hold the boot image and root. I attached the HDD and mounted it. Then I kept it running to see if it would crash again…

And it didn’t! It now runs stable for over 48 hours so I’m confident that I can now get it to work. At this moment I am syncing the database again but I have high hopes that it will be the last time.

I will keep you all updated with the results of course ☺