CTO interview: Danny Fields from Avalara, the largest tech company you have never heard of
New post in Tech Captains! The latest interview with Danny Fields, the CTO of Avalara or, as Danny puts it, ‘the largest tech company you have never heard of’. Find out how to choose the best stack, how is it to be at the epicentre of cloud computing from nascent stage until today, and how to build and manage a large tech team across four continents.
How did your career prepare you for your current role at Avalara? I see you’ve spent more than 20 years at Oracle.
Oracle was my first job outside of college, where I studied engineering. During that time, I worked in the Netherlands and Ireland, and had the opportunity to. travel around the world and interact with a lot of enterprise customers (banks, governments, tech). The product we were building in 1990 when I joined was the Oracle database. Oracle is probably the best-known database vendor on the planet and over the years they expanded into building ERP and CRM applications.
I started at Oracle as a developer and worked my way up until I eventually became a vice president. It was during that journey that I learned how to build enterprise-grade software and how to serve enterprise-grade customers.
Oracle software is built to be scalable, resilient, and very secure. One of the biggest lessons I learned from Oracle was how to build scalable, resilient, and secure software long before the cloud existed. When I joined the industry 30 years ago, everything was on-premises. We even shipped software on floppy disks, which is funny because my kids don’t know what a floppy disk is. Still, the principles for building sound software were engrained in me early in my career and have stayed with me to today.
In 2007, when Oracle was on the path to becoming a cloud company, I joined a company called RightNow Technologies — one of the original cloud companies. At that time, there was no AWS, GCP, or Azure. We were hosting on our own servers in co-located data centers. We had to build high availability and disaster recovery ourselves by setting up our hardware and software to run in multiple copies within a data center, and to run across multiple co-located data centers.
Four years later, RightNow Technologies was bought by Oracle, and I made my way back to where I started. However, what was cool about going back into Oracle was that Oracle was starting to become a cloud company, it just didn’t really have many cloud customers. RightNow Technologies brought about 2,000 cloud customers with it, and it became the Oracle Service Cloud.
We were the first major product at Oracle with many enterprise grade cloud customers, servicing them, upgrading their software in our cloud, maintaining them. That was a cool experience and set me on the path to where I am today at Avalara.
You seem to have loved your time at Oracle and its products. Did you push to get Avalara on Oracle?
I’ve learned that it’s better to build on best of breed technologies. Why build when you can buy? We’re running on multiple public clouds: AWS, Azure, and GCP. We use many databases; our primary one is MongoDB Atlas, a cloud document database. We use many other third-party software components from a number of providers, including Oracle.
Could you expand on the tech stack used at Avalara? And what is the impact of legacy technologies?
We were born in the cloud. Being born in the cloud 18 years ago meant we were born on a server that was kept under a bed, in a bedroom belonging to the daughter of one of our co-founders. Then we moved our cloud into the garage belonging to another co-founder. The internet connection into the garage was a cable that ran across a footpath and that had to be kept out of reach of the gardener’s tools. We were eventually able to move out of the garage and into a colocation data center. Most recently, in 2018, we lifted and shifted our cloud into AWS. Today, we also run services across a multi-cloud environment in AWS, Azure, and GCP.
Our main tax calculation service started as a monolithic application. In 2018 we began converting it to cloud-native, to run properly in a public cloud environment. Our goal was to make our application scalable and resilient. For example, if an entire region within AWS were to go down, our software were is designed to handle those failures.
Regarding the database, we were using a legacy database that didn’t scale. One of the things we’ve had to do over the past few years was transition to a more modern document database that runs natively in the cloud. We chose Mongo DB Atlas.
I like to say that we’ve had to change the engines on our aeroplane while it’s in-flight. The tax calculation service is Avalara’s main cloud solution, and we can’t shut it down while we do upgrades.
For the migration, we had to run two versions of our database, the legacy one and the new one side by side for many, many months, move customers over, run tests, run real live data.
Four years ago, if we had a failure in one piece of our application, the whole thing went down. Today, we’ve broken our service into smaller services and if we have a failure, only that one service is impacted. Everything else stays running.
What do you think of the multi-cloud approach?
Multi-cloud is the way of the future. You may choose to run some or all your services in multiple public clouds for the following reasons:
· Your customers may demand that you run or not run in a particular public cloud.
· You may decide you want to use technologies and services from multiple public clouds.
· If one cloud has a failure, you can switch your application to run in another cloud.
· It’s always good to have options and not get locked in with one vendor and their pricing
We have multiple services running across multiple cloud environments today.
What are the programming languages you use?
Most of our services are written in C# on .NET.
We have bought many companies, which means we have inherited many different languages and databases.
We run MongoDB, SQL Server, Dynamo DB, Postgres. You name it, we have it. We use many programming languages as well. We are trying to standardise on the smallest number of languages, and the smallest number of databases, to keep things simple.
How do you structure such a big team and maintain motivation while innovating?
When you look at us from the outside, you’d think we’re a tax company, but we’re really a technology company that applies tech and innovation to the principle of tax.
We organize around the products we’re building. One team is focused on our tax calculation engines. Another team is focused on different applications to file tax returns. Another team creates licensing solutions, while another team builds our UI. And so on. We also have an architecture team, a data engineering team, a program management team, a product security team, a reliability engineering team, and a sustaining engineering team.
Our engineering teams are located on four continents, in dozens of cities. We hire great engineers, and our vision is to create the best engineering team that we have ever worked in.
And one of the things we say is that we’re probably the biggest company in technology that you have never heard of. Innovation and tax probably don’t seem like they go together, but at Avalara, I can assure you that they do.
And in terms of innovation?
We have a huge focus on innovation.
At the end of each quarter, we have a two weeks innovation sprint. We use that time for our twice a year global hackathon, and to build other innovation projects.
The hackathon is one way for engineers to innovate and come up with ideas. We are also building an innovation team made up of product managers and engineers. This team is dedicated to rapidly innovating and prototyping and coming up with new ideas.
At Avalara, we’re solving the 5,000-year-old challenge of tax, invented by the Egyptians.
We’re calculating tax and we’re filing tax returns for companies. If you’ve ever filed a personal tax return or for a business, it can be a very complicated and manual process. We’re building automation that calculates how much tax is owed, we get all that information onto the return form, and we remit the form and any taxes due to the jurisdiction. To do this efficiently we are constantly coming up with new ideas and patterns.
What advice would you give younger techies?
You know, that’s a great question. I had to answer this question to some interns a few months ago. We have a vibrant global intern program, and we hire interns all around the world.
I told them it’s okay to make mistakes. Because you can learn from your mistakes. When I started my career, everything for me was about being perfect. It was about being successful in my job, building software a certain way, always trying to be a perfectionist. No matter what you do, you will always make mistakes. Things will always go wrong.
There are things you have no control over. In our engineering process, we have a process for what to do when something goes wrong with the software. If our cloud goes down, or if we have a technical problem, we have a process that kicks in where we get engineers together and we ask questions to understand what happened. We have a no-blame policy.
We try to find out if it was something we did wrong in the code, if we didn’t run a test properly, or if somebody made a manual mistake Was it a technology or a process problem? We learn from the problem and then we put a fix in place to make sure that the same problem never happens again.
So going back to me, and the advice I would give younger techies: Accept that you will make mistakes. Accept the fact that the things will go wrong, but that’s all okay because you will learn from the mistakes, and you will learn how to be a better engineer. And for leaders, mistakes will teach you to be a better leader. You will learn how to build a better engineering organization. Learning from mistakes is part of the process. Mistakes are part of the learning experience. You can’t learn to be a great engineering leader or how to build great software just in a classroom, or from only reading a book. — You must experience the real world.
If you want to connect with Danny, click here.
To learn more about Avalara, visit their website: avalara.com
If you’re a techie working on something exciting or you simply want to have a chat, get in touch with me. I’m currently CTO at Kolleno.com
FROM THE AUTHOR
>> 3 Reasons to Get Rid of Excel for Invoice Tracking
