Domain down, everyone panic.

Saturday morning our monitoring systems started complaining that Union was not accessible to the internet. If you ever have problems waking up on a weekend, try this method. Normally, this could mean that somehow our servers were offline or that our data centers (we use Amazon Web Services) were somehow unreachable. We checked the servers themselves and the AWS status console, and everything was in the green. We then realized it was the domain name itself that was not resolving. Uh oh.

The Union platform runs on redundant infrastructure. Servers are added and removed automatically with system load so that you get consistent loading speeds. We also have a live replica of our database ready to go if the main one fails to guarantee no customer data is ever lost. We’re pretty confident in our infrastructure — but it is all in vain if there’s no domain name to access it with.

Most of you will know that Union spun out of 1776 a few months ago. We decided we wanted to focus on our core product as our friends and partners at 1776 focus on building a great community of startups. As part of that transition, bank accounts, credit cards, and email addresses changed. A few days ago, the automatic renewal for union.vc (which renews every 2 years) failed — because the credit card information had to change. But when it did, we didn’t hear about it, because the contact information was set to an email we sunset in the transition.

So the domain was put in a redemption period — a time for the owners (us) to go back to the registrar and say “hey, we still need that” (we do). 5 minutes after we got the warning about the domain name, we had renewed it. If you ever had to renew a domain, you’ll know that it is typically not an issue. But if your domain is in redemption, your registrar (the organization where you buy the domain) has to actually communicate with the registry (the organization that controls the top level domain — in our case .vc)

Enter: timezones. Timezones are a cause of high blood pressure in engineering, so if you are an engineer, tread lightly. Between our registrar in Australia and our registry in the Grenadines (don’t ask) stands the planet core and a 16 hour difference. They are quite literally on opposite sides of the globe. This made it very, very slow for the manual part of the domain renewal process to go through. And the worst part? We couldn’t really help.

So we did the best we could:

We got an alternative domain name and pointed that at our infrastructure. The servers were running anyway, and we didn’t want to disrupt the workflow of the people who depend on Union to do their job — entrepreneurs, community managers, investors. That’s when we emailed everyone about accessing Union via joinunion.us — we know it is not a great domain name, but we really were more focused on getting the platform accessible again than originality in our URL.

So for the past 36 hours, many of you have been accessing Union at that alternative domain name. Thankfully, the hassle is over, the domain issue is fixed, and the platform is back at union.vc. We did not lose any data, but we did lose a bit of sleep. Hopefully by giving you access to the platform on an alternative domain name, we didn’t disrupt your workflows too much.

But we are still quite sorry this happened, and we will do better in the future. The most obvious change we’ve made is to make sure the platform remains accessible via multiple domains so that we don’t have a single DNS point of failure in the future. But more importantly, we want to change how we communicate platform issues with our customers. We will be creating a product blog and status page in the coming weeks where you will be able to get updates if there are future issues, and report anything you find. That way it helps you, and keeps us accountable.

Thank you for reading, and it is worth repeating: if you were affected by this issue, we appreciate your patience while we worked through this. For now we are past this and excited to get back to creating a great platform. We have some very exciting things coming in the future, that I can’t wait to write about.