Upstart Hackweek Highlights — 2022
The Upstart team recently completed its annual Hackweek. In this post, I’ll share some highlights from our second-ever Hackweek!
What is Hackweek?
- Hackweek is a weeklong event where we drop our team boundaries and sprint commitments to work on projects driven by us! Upstart engineers, designers, and product managers can participate.
- Hackweek starts with idea formation: anyone is allowed to contribute an idea to be worked on during Hackweek into a shared Google Doc!
- People with project ideas then give a 2-minute pitch at a “pitch party” to advertise their project and recruit members. This year, we had a few targeted categories of work, but projects are not limited to those categories. For example, two of our targeted categories were related to process improvement and performance this year.
- At any point before Hackweek starts, participants can join a team by signing up! Team size is not limited, but those with a minimum size of three were eligible for the “competition” aspect.
- During the week, participants hack together to build their project! Project goals range from building proofs of concept, mocking out design updates, to completing full fledged microservices.
- After Hackweek, all projects eligible for judging deliver an elevator-pitch format followed by quick Q&A. Post judging, winning teams are announced.
This year, we had over 200 participants. We also had several volunteer coordinators!
Why do we do Hackweek?
While there is a competition aspect of Hackweek, the primary goals of Hackweek are to:
- Encourage creativity in new ideas
- Have fun
- Collaborate with new people
- Empower teams with autonomy to drive new ideas
The goals of Hackweek are not to skew work/life balance or add stress from the idea of a competition. For that reason, participation was encouraged but not mandatory.
Here are some of the projects I found most interesting this year!
Ephemeral Environments
One project that leans heavily to the platform and devops side is called “Ephemeral Environments.” This project aims to provide the ability to optionally spin up an externally accessible temporary environment from a GitHub pull request.
Value Proposition:
This proposed solution enables anyone at any time to create a dedicated environment on the fly, available for testing or review from other engineers or product owners. Most engineers develop locally. They must reserve space in a permanent squad environment to allow external participant access. This can be challenging as our engineering team has grown at a pace faster than we can build permanent environments!
This project also addresses the present pain point of not having an environment that is up to date with the latest data model. These permanent squad environments are not always maintained and can fall behind in data model or code.
One final selling point to note is that there is less resource waste. These environments are deleted after use — there’s no reason to keep the resources around if they aren’t being used!
How:
During Hackweek, the team worked on the mechanics for creating these environments:
- First, an added GitHub label on the pull request initializes the environment provisioning.
- Next, an OpenShift project is created, along with multiple deployments, services, routes, etc. for that environment
- Upon environment creation completion, a GitHub comment is created. This comment includes links to the ephemeral environment.
- At this point, an engineer may provide access to a business owner to complete testing or review.
- The environment gets destroyed upon pull request merge or closure!
Why it was interesting to me:
This project is of interest to me because I have previous experience with an open source project that created ephemeral environments per consulting application. That solution was implemented on a much smaller scale. We spun up environments on the fly, made changes, and our business owners were able to test or validate.
That model worked quite well. I can see it being of great value to many product teams who work with external product owners. It will enable those owners to see and interact with functionality earlier in our software development life cycle. We could potentially eliminate the need for permanent squad specific environments!
Project Contact: Want to learn more? Feel free to reach out to Isaac Adams.
Harder, Better, Faster, Stronger Frontend Assets
The second project I’ll cover here is “Harder, Better, Faster, Stronger Frontend Assets.” In this project, a team of several engineers worked together to make a significant impact by moving static assets outside of our Ruby on Rails nginx-serving containers.
Value Proposition:
This Hackweek team worked to move static page content out of our nginx containers to CloudFront and Cloudflare. They presented convincing data on the impact that performance makes, both on conversion and SEO. The improvement significantly reduces the time to first byte, a metric that is referenced by many performance tools.
How:
During Hackweek, the team:
- Created and configured configured a new GitHub Action for publishing static assets
- Completed configuration of the S3 bucket, Route 53 hosted zone, certificate, Cloudfront distribution
- Created patterns that could be successfully reproduced by other teams
Why it was interesting to me:
This project fits my generalized take on performance work: with most performance work, you start by ranking opportunities or bottlenecks by impact. You consider options for the highest-impact changes. You evaluate tradeoffs (be it in complexity, maintenance, resource requirements, etc.). Then, you determine a path. The updates are applied and you see gains!
This work successfully evaluated bottlenecks and took a path towards an improved solution. The results are extremely high impact, falling under the category of “why didn’t we do this before?!” I believe there are plenty of other low-hanging fruit, high-impact changes. This hackweek team found one of these things to address. Even better, they documented their solutions to allow adoption from other services!
Project Contact: Want to learn more? Feel free to reach out to Christopher Gray or Danny Brown.
Hackweek Combo #1: Vision + Healthcheck Admin
The ideas I’m highlighting here are two separate projects titled “Vision” and “Healthcheck Admin”. Both of these projects were driven by the challenges associated with time sensitive PagerDuty responsibilities.
Many of our engineering teams serve “pager duty” in weekly cycles. Those on pager duty are responsible for bugs, health checks, and emergencies for their team during that week. Pager duty can be challenging because the engineer on call may not always have context for the subtle business logic around any one particular issue.
Value Proposition:
The Hackweek project “Vision” aimed to build a better visual tool for debugging the lifecycle of domain objects at various states. It aggregates data from multiple sources (Bugsnag, SumoLogic, Sidekiq) to paint a picture of object state for efficient debugging. In some cases, it adds interactive elements to help troubleshoot issues.
The other project, “Healthcheck Admin’’, aimed to create a dashboard for viewing health check status. In our code base, we implement sanity checks (called healthchecks) that run on scheduled intervals. They signal problems that are not immediately identified in the tools mentioned above. Adding an admin panel for these logical checks and the ability to rerun them provides immediate value to engineers trying to solve a problem.
How:
Both of these solutions introduced interfaces which provide information and strategies for resolving issues. Both aim to present the signals in a useful fashion to resolve problems quickly. ”Vision” was implemented in a new Ruby on Rails application featuring a Vue.js frontend. “Healthcheck Admin” included creating a new React-based interface in our existing primary codebase because it ties directly into existing logic.
Why it was interesting to me:
Both “Vision” and “Healthcheck Admin” were born from the challenges associated with solving urgent problems, hindered by context switching between data/code boundaries or the limitation of missing nuanced context.
My own product team owns a broad set of business requirements across our application. I don’t always have the context at the front of my brain to pull from while dealing with these urgent issues — I may not even have worked on our primary codebase in 6 months when I find myself on pager duty! I connected with the value both of these projects aim to deliver. Both tools would be valuable, especially as the scope of our work continues to grow.
Project Contact: Want to learn more? Feel free to reach out to Parikshit Sharma for Vision and Kelsey Shannahan for Healthcheck Admin.
Hackweek Combo #2: Drift + Experiment and Variant Finder
The final projects I’m sharing are “Drift” and “Experiment and Variant Finder.” Both of these projects aim to provide insight on configuration, but in different contexts. The former aims to improve visibility of configuration from a number of sources at an environment-specific level. The latter project aims to provide visibility into experiments and variants at a domain object level.
Value Proposition:
These two Hackweek projects provide value by giving visibility to settings that impact domain objects or environment behavior. Both of them will be helpful at time sensitive moments, whether for the team servicing an end user, or for the engineers who are confirming or debugging a behavior in a specific environment. The faster we can access this information, the faster we can make a decision to take action to resolve time sensitive issues.
How:
Both tools created new admin interfaces or menu panels that communicate configuration. “Drift” was written as a new NextJS application leveraging TypeScript. “Experiment and Variant Finder” was added to existing admin panels.
Why it was interesting to me:
I think a lot about signals and communication. While we are actively moving away from a monolithic codebase, we still have 10 years of hard-earned tech debt. One facet of tech debt manifests in a number of ways in which we feature flag behavior across environments.
Both of these projects aim to provide visibility around these hidden values to better serve the interested parties. This will ultimately benefit our end users, as it will enable us to iterate quickly with better visibility!
Project Contact: Want to learn more? Feel free to reach out to Alex Holmes for Drift and Akshaya Viswanathan for Experiment & Variant Finder.
Closing Thoughts
That’s a wrap for my Hackweek highlights! I included the 6 projects that I can see impacting my day to day the most. Outside of these projects, there were a number of great projects that have the potential to move our machine learning and product forward. I hope we continue our Hackweek tradition! We put out a survey at the end of Hackweek to take feedback and further refine the experience.
If you are considering Hackweek, I’d definitely suggest experimenting and collecting feedback to shape works for your team. Or you are always welcome to join the engineering team at Upstart to participate in Hackweek 2023 — we’re actively hiring remote engineers and engineering managers!
