Under the Hood: Interviews with SPOs: Ryan [PANL]
Welcome to a series of interviews of Stake Pool Operators (SPO) in the Cardano blockchain. In today’s post, it is PANL stake pool (https://www.panl.org/) and on Twitter @panl_stakepool. Ryan is one of the 3 co-owners/operators based in the US east coast.
My aim throughout this series is to dive into who the pool is, how they are hosting or running the pool and how it was like starting off. My goal is to keep them brief but informative.
Here was my conversation below with Ryan
How did you get started as a Stake Pool Operator?
I found out about Cardano (a little bit late) from Charles’ videos like the whiteboard and Bermuda TEDx talk. Seems to be the most common answer, but the whiteboard is such a good video. I would recommend it to anyone new to the space or who may have not seen it yet. I purchased some ADA and found out how to delegate to earn rewards and secure the network with active stake. As I learned more about the pool I was staked with, I got a bit upset about the level of centralization I was seeing across pools with active stake. When I did the research to see what was required to be a pool operator, I figured it was a perfect fit.
Tell me a little about your pool, what’s your focus and mission?
So our pool is actually myself, and two others. We’re a tight-knit group of coworkers/friends over years of time with two of us pretty involved in different spaces within the ecosystem. We’re still fairly new, registered on mainnet at the end of April. I’m still in the process of calculating and making our first officially documented donation as a pool for Q4 as we did not produce many rewards until September. We had originally only planned on producing a few blocks this year as marketing is so difficult, however, we were lucky enough to be recognized by the community and IOG for their bootstrapping program. Our mission is admittedly not defined too well still, however, we each have specific causes and charities on the list that we want to donate to in the following categories: regenerative agriculture, mental health, and veterans health. You can always find more info about each of us on our website. My current focus has been to support the ecosystem in the best practices and other chat spaces while trying to promote better decentralization through participation in groups like the xSPO and CSPA as well as running SMASH servers for both those alliances plus the CMDP.
Let’s go under the hood. What hardware or hosting are you using to run your pool?
We’ve got a hybrid setup, both cloud-hosted and local bare metal servers. In our non-crypto jobs, we have an IT company, so access to hardware is pretty good. The bare-metal servers are single OS systems running Ubuntu with solid internet SLA agreements, backup power, etc. Our testnet pool runs on a single Microsoft Hyper-V server running Ubuntu VMs. The hybrid model was for redundancy; mostly the cloud was to be able to distribute geographically as well as for disaster failover. We did this in the spirit of supporting the network as best as possible and aimed for as much redundancy as we could. We have nodes at Azure, Google cloud, and a locally hosted DC with registered relays in the US, Singapore, and Ireland (soon, I just need to update the reg), as well as unregistered relays and a backup BP
How do you think your pool performs based on the choices you made? Any drawbacks?
It’s still a very small sample size, but so far I think performance has been fantastic. We have never had an assigned slot missed/ghosted/stolen (not the operator's fault) on mainnet. The primary drawback was definitely the initial setup for any of the local nodes. When you cloud host you don’t have to worry as much about the little things like wiring and physical safeguards as long as you trust the company doing the hosting. We monitor all kinds of statistics about the pool using Grafana as well as some local scripts. I believe a lot of credit for not having any block production issues goes out to the community though, there are many SPO’s providing really educational content that helped me learn so much.
What was it like building the first node? Funny mistakes that you made?
That was an adventure for sure! I was using the github source install guide, before I had found the best practices channel, and it took me quite some time to work through the errors. I was very happy to see some recent updates in there, like the change to ghcup for cabal and ghc installation. We all make mistakes right? I had the common missing libsodium error, as well as installing a packaged version of cabal which just would not work. The first time I saw db corruption (did not know that was the issue then), I just could not sync past a certain block on one testnet relay no matter what I changed, eventually I just rebuilt and it worked (of course). I’m also very glad to have started on testnet as I also lost a few tada to botched commands in the very early days. I think mistakes are really important to the learning process, but the best place to make them is testnet so you can be ready to prevent them on the mainnet.