Spawn 2020 in review

Published in

Spawn DB

5 min readFeb 3, 2021

--

Despite the many challenges posed by 2020, Spawn had an impactful year as we moved out of a prototype stage into a research stage while delivering databases as a service for developers and CI pipelines. Here is a summary of what we’ve been up to in the Spawn team.

Growth

👆 6 data-containers created in Q1.
🛫2,825 data-containers created in Q2.
🚀 18,039 data-containers created in Q3.
🛸27,615 data-containers created in Q4.

For a total of 48,485 data-containers created in 2020.

Blogs

Helping you accelerate production-like CI pipelines or testing database migrations with and without data. Getting rid of shared databases or reducing test times by 30%, we’ve been busy writing up various use cases where Spawn can help.

Tutorials

We also made sure Spawn concepts are clear and well explained, please head to our “How to” guides to get started with Spawn.

In review

🧬 Data container lifetimes!

When creating a data container it is now possible to specify how long the container should live for using the — lifetime flag. After this duration has expired, the data container will be automatically deleted.

🤝 Organisations

See all the members of your organisation with the new spawnctl get organisation command. This lets you see exactly who has access to Spawn within your organisation.

🔍 Troubleshooting

When data image creation fails, did you know that you can run spawnctl logs data-image to see why? We’ve made this more discoverable by suggesting it whenever an image creation attempt fails. There is also an analogous spawnctl logs data-container command to see the database engine logs for your data containers.

⌨️ Shortcuts

We’ve added some aliases for the get/create/delete data-container/data-image commands. You can now use ‘di’ or ‘dc’ instead of ‘data-image’ and ‘data-container’

⌛ No more copy paste of codes

spawnctl now authenticates using the Device Authorization Grant, which means you don’t have to copy/paste a code from the browser anymore. Just log into the browser window, and spawnctl will detect you’ve authenticated.

📦 Support for custom and plain pg_dump

Support for both custom and plain pg_dump backup formats for PostgreSQL when creating data images

🙏 Cmd line friendly

Added a — no-headers flag to all spawnctl get commands to suppress table headers from being printed
Allow reset and delete commands to take multiple resource names as a parameter
Specify new image names for the graduate command with — name <name>

🔮 With auto-completion!

Can now auto-complete resource names when pressing the tab key. The resource names it will work for include data-containers, data-images, and access-tokens
Now supports the following shells: Bash, ZSH, Fish and PowerShell

📝 YAML outputs

On top of plain text and json, we now support YAML as an output for spawnctl commands, for this use the “-o yaml” at the end of the command

🔐 It better be safe!

Increase container password entropy
Ensure MSSQL data containers use TLS encryption
Unique passwords for each data-container created from data-images

⚖️ Stability

Additional stability improvements for startup of data containers
Improved deletion performance and stability
Various website fixes and improvements
Increase infrastructure capacity
Autoscaling infrastructure to handle additional capacity requirements for users

A postmortem culture

We had 34 downtimes through 2020 🎉! More importantly, we created a postmortem culture.

We made sure to review our postmortems and act on them, looking to improve the reliability of our service while learning along the way.

Off the back of this postmortem culture we have

Created a staging spawn cluster equal to production
Improved our deployment pipelines and deployment gates
Improved our tests
Improved our troubleshooting within kubernetes
Improved our infrastructure’s stability
Used Spawn to build Spawn — we test our database migrations with and without data through CI pipelines, with the help of Flyway
Created ephemeral Spawn environments to test our changes
Improved our alerting and monitoring systems to have a faster response time when incidents arise
All while still deploying changes consistently to production every day🚀

Building a culture of learning

Fostering a culture of learning is very important to us, for that we carry on various activities within our team that happen on a weekly basis. From book clubs where we read and discuss chapters on topics such as Docker, Kubernetes, or SRE, to weekly onboarding sessions where we cover internal topics or more general ones. Below are some onboarding sessions we’ve had together as a team.

Intro to Kops
Intro to Kibana
Spawn Architecture
AWS Basics
Go onboarding
Terraform and Ansible
Dex and OIDC
Docker Katacoda (part 1 and 2)
Linux networking
ZFS Intro
ZFS Exercises review
Azure DevOps Pipelines
Unix/Linux 101 & setup ZSH
What is K8s?
Intro to Kubectl
Kubernetes Services
Intro to CEPH
Intro to Helm
How to kill containers based on policy

Looking at 2021

We are currently looking for external projects and teams with CI systems that could benefit from our help setting up pipelines and tests that use Spawn. We’ll move Spawn infrastructure to be fully in-cluster by adopting CEPH and deprecating ZFS. Finally we are looking to keep growing as we research and understand retained external beta users.

See you in 2021 🚀

If you’d like to get access to Spawn to take advantage of these benefits, then sign up for beta access here.