Writing tests for brew during GSoC 2017

For me, this summer was a tryst with Behavior Driven Development (BDD) wherein I was writing tests for brew as part of my Google Summer of Code with Homebrew.

I feel I have come a long way from being a complete noob at writing Ruby and RSpec to being a reasonable noob. Throughout this journey, I learnt a lot, had the privilege of experiencing the joyous moments of ‘Oh! I understand it now!’ and realized at every step how cool Homebrew is as a project. The most magical part of the code for me was how a formula is loaded in the context of a module rather than being loaded into the global namespace.

This blog post attempts to summarise the work that I have done and the roadmap ahead.

All my Pull Requests (open/merged) can be accessed at: mansimarkaur

My project increases test coverage for brew through unit and integration tests. The tests were written in RSpec — a BDD testing tool for Ruby.

List of all my contributions:

Added tests for os/mac/keg and [WIP] Tests for os/mac/keg

Attempts to increase coverage from 0 to 100% for os/mac/keg.rb

Overview: This PR adds tests for methods that modify the dylib id of a dylib file and change the install name for a MachO file. These methods are instance methods of the re-opened class Keg in keg.rb.

The tests found a bug and prove that the existing methods fail to change the dylib_id and install_name for a dylib file.

Lessons: I learnt about the MachO file format, dylib files, the different regions of a MachO file and in what conditions brew changes the install name of a file.

Added tests for utils/git

Coverage increased from 0 to 100% for utils/git.rb

Overview: This PR adds tests for checking if git is available, sets git path and version if available, checks if the given url is a valid git remote, gives the last revision commit and last revision of a given file.

Lessons: Try to minimise stubbing as much as possible. For e.g., instead of stubbing @git to false to stub unavailability of git, we can make the git shim point to a location so that it fails to find git and hence git is declared unavailable. Use directories that are automatically cleaned up as parent directories to create temporary files and directories during tests, so that you don’t need to clear them up using, say, FileUtils.rm_rf .

Added tests for utils/analytics

Coverage increased from 0 to 100% for utils/analytics.rb

Overview: This PR adds tests for the module that sends data to Google Analytics. Also, added a method to clear the @anonymous_os_prefix_ci is already set.

Lessons: All ENV variables are automatically reset after every test. Always use stub_const instead of redefining a constant for a test.

Added tests for utils/svn

Coverage increased from 0 to 100% for utils/svn.rb

Overview: This PR adds tests for methods that check if svn is available and if an svn remote exists at the given url. Also, added a new method clear_svn_location_cache to clear @svn .

Lessons: Use of mktmpdir for temporary directories. On any macOS system there will always be a working svn and git . Setting up an svn remote and listing directory entries in the svn repo upto a certain depth.

Added tests for language/java

Coverage increased from 0 to 100% for language/java.rb

Overview: This PR adds tests for methods in the Java module of Language module. The methods return the JAVA_HOME ENVand also allows overriding of JAVA_HOME ENV .

Increasing test coverage for caveats.rb

Coverage increased from 46.61% to 99.15% for caveats.rb

Overview: This PR adds tests for plist caveats, python caveats, shell completion caveats, text for keg_only formulae. plist_caveats was also split into plist_path to facilitate better testing.

Lessons: It’s good to modify the code to be tested so as to facilitate better testing. Pbpaste is the system clipboard tool on macOS and fails with tmux by default, so, checking if pbpaste fails can be used to warn users about brew services failing under tmux. Tests marked with the:needs_macos tag are run only on a MacOS system. Also, learnt how to stub ENV variables and about types of shells — bash, zsh and fish.

Added tests for language/node.rb

Coverage increased from 0 to 100% for language/node

Overview: This PR adds tests for node installations.

Lessons: There was poor error handling in setup_npm_environment which was fixed through a rescue FormulaUnavailableError if thenodeformula doesn't exist. Understood how shutup worked to silence all output to stdout or stderr during tests.

Added tests for disk_cleanup and unremovable kegs and Added tests for disk_cleanup and unremovable kegs in cleanup_spec.rb

Coverage increased from 67.35% to 94.9% for cleanup.rb

Overview: This PR adds tests for cleanup of logs, java_cache, npm_cache, glide_home, files and directories that are specific days old, old VCS checkout directories, etc.

Lessons: VCS checkout directories include -- in their name and logs older than 15 days are cleaned up by cleanup_logs , --prune="all" is used to delete all files and directories.

Added test for check_access_lock_dir and diagnostic_spec.rb: assertion in check_access_lock_dir test

Coverage increased from 41.85% to 42.6% for diagnostic.rb

Overview: These PRs add a test for checking if the lock directory is accessible.

Lessons: Octal value of File Permissions in UNIX systems.

Added /Library/Homebrew/bin and /Library/Homebrew/.npmignore to gitignore

Overview: This PR adds redundant files to gitignore.

The road ahead:

Tasks left: Test coverage for diagnostic.rb is still at 43.27%. I plan to work on increasing it to at least above 60%

Future work: We need tests for files in the cmd directory. Currently, it’s coverage stands at 33.89%. I’m going to work on increasing it above 40% at least. I’d also be happy to help and collaborate with anyone willing to contribute.

Future contribution: I am definitely going to continue contributing to brew. I plan to add more tests for brew and also try to start with resolving a few easy issues on the GitHub repository. I also plan to share my experience and lessons in a few technical conferences.

I have learnt so much during this entire experience and everyone has been super helpful. I’d like to thank my mentor, Misty De Meo for being so encouraging and helpful. I’d also like to thank MikeMcQuaid, Markus Reiter, William Woodruff, Andrew Janke and Gautham Goli for helping and guiding me.

I’d also like to thank Google for giving me the opportunity to be a part of such an amazing community!