Busy week due to travel. My weeks are usually not this eventful. :)
- Traveled to California this week, which means In N Out and Fry’s!
- Proactive vs Reactive
- CI failures (works on my machine™)
- Starting to re-read Turn the Ship Around
Travel to California
I’m from California but have been living in Raleigh, NC for almost ten years now, so it is always nice to visit the mother-state. I’m not used to the density anymore (I currently live in rural area outside Raleigh, North Carolina) but I still love visiting, if only for In N Out and Fry’s. :) I’m aware of the debate around the popularity of In N Out but I don’t really care; I love it.
Fry’s is just sooo much fun to walk around. I get some steps in and get to play with all teh gadgets. Win-win.
JS Meetup in San Jose
I look at teaching as a confirmation of mastery. Although I feel like I did OK, I know I am improving. So, master I am not, but I am comfortable with the trend.
Regardless, I really enjoyed sharing my knowledge and passion for making things. Good times.
Proactive vs Reactive
In client work, it is common for quality to suffer, if only for a short time. Decrease in quality happens through compromises in values. Compromises happen because of team instability (rotating team members, varying quality of team members, etc), management (unclear requirements, inconsistent sprint management, etc) or for other uncontrollable reasons. Sometimes (most of the time?) we do not have control over the cause but we are still responsible for deliverables.
It is a common trap to become reactive in these situations. When quality suffers, though, being reactive means only assessing and addressing the top most issue; the thing everyone sees. What we need to address is the root cause of the issue first — the bottom of the iceberg, to put it another way.
Keeping a cool head, an open mind, and remembering the goals of the project is difficult to do at this stage but arguably the most important time to focus. Sure, it could all fall apart but nobody wants that. Being diligent and powering through the to-do list is how to get back on track.
It takes time and it will not be easy. But, like all things, it can be done. Our team is currently going through changes (normal changes, nothing bad) and I like to talk to people about being reactive during big changes to help people stay focused.
We lost a day of work because our CI was failing. The same commands passed on our development machines. We ended up finding out it was Mockgoose installing a version of Mongo that would not start on the CI machine.
Getting to that piece of information took longer than we wanted, though. There were a few issues that clogged up our ability to debug the problem:
- Mocha didn’t seem to indicate that it was hanging on a promise. Although we could quickly assume that it was a promise in a
beforecall, it was still an assumption.
- Mockgoose does not make it obvious that it downloads and installs Mongo (we were not aware that it does this so could not inspect Mongoose earlier)
- We could not reproduce locally (We figured out that it was because we already had a working version of Mongo installed)
- We aren’t Docker experts (our CI uses Docker and we thought about using the same Docker image but couldn’t get it to work)
- No stack trace, since, on CI, the process would timeout and fail, there was no stack trace.
The last successful build used a slightly different set of dependencies than the failing build. We forced the deps to be the same but still got a failing build. We did this one at a time just to make sure. The hunch was that it had to be a dependency or an environment issue (the codebase didn’t change much between the last successful build and the failed build).
Having exhausted our ability to test dependencies, we tried to recreate the issue on a similar environment. Eventually we were able to recreate the issue on a version of our test environment which happen to not have Mongo installed. Running
DEBUG=* npm test -dd got us some really verbose logs that had some weird references to downloading Mongo.
At this point we were pretty certain the issue was either with Mongoose or Mockgoose — specifically a hanging promise or async call in either library. But we didn’t know it was because something was downloading Mongo until now. Once we tracked down the download call to a specific NPM package, we grepped the
node_modules folder for the package and found a couple of our dependencies were using the Mongo downloader. The only library that imported the downloader package was Mockgoose (Mongoose uses it as well but only in their own test files). From there we found an issue filed that documented the same exact problem we were experiencing.
This sucked. Mostly because we could not assume that CI was giving us a false negative (what if our local machines were giving us false positives!?). This meant that no other code contribution could move forward until this was resolved… which ended up taking a little over half a day.
On the plus side, we realized the we were running our test suite against the latest Mongo — which obviously has some problems running on certain Linux distros. So, now, our test suite runs against the same version of Mongo we use in production. CI is back up and now, technically, more accurate. Silver lining acquired.
Turn the Ship Around
I have always been at odds with the concept of “managers” and “leaders”. The teachings of others as to what made good leaders/managers never felt right. Turn the Ship Around is a great book that talks about the importance of language and mentorship.
After a few conversations this week with friends and coworkers, I feel the need to refresh what I learned from David Marquet.
I think everyone should read this book — as a consultant, though, I am in a unique position to help others be better. The work that David Marquet has done in this book is incredibly helpful for people at almost any level in an organization.
That’s it for season 2, weeknotes 1. See you next week.