Using vcr for Faster External API Calls in RSpec

Many of the applications we build here at philosophie utilize existing open-source solutions both to tackle basic app functionality (e.g. user authentication) as well as to help us build out components of applications that could not be feasibly built from scratch in the timeframe for a given project (e.g. payment processing). We integrate with these services either through an API wrapper (a gem) or through writing our own code to directly interact with the service if there is no gem available, the gem is deprecated or poorly-maintained, etc. While each approach is valid in its own right, when it comes to testing, I prefer a slightly different structure for each, mainly in terms of stubbing out 3rd-party API responses.

When we integrate through an API wrapper (a gem), I usually stub responses in the test suite manually, on the assumption that the gem is already tested (within reason). However, when I’m writing my own code to interface with an API, I prefer to test the totality of the request/response cycle, simulating a real-world interaction with the API. While you could still stub out the response manually in these situations, I would argue that stubbing in this situation would only be preferable if you were testing your application’s response to some kind of simple component of the HTTP response, such as, let’s say, its status code. If you are actually manipulating the response data in a some way, I would argue that a more sustainable approach would be to record the response in its totality (which is still a stub, but one done automatically), diminishing the possibility of human error and preventing you from having to periodically check-in with the API to make sure nothing has changed in terms of the response data itself.

One great way to record and reuse API responses in your test suite is to use a gem called vcr, which ‘records’ each external API request/response in a test suite the first time it’s run, stores the data in a ‘cassette’ file (basically just a YAML file consisting of a little information on the HTTP request and the entirety of the response), and then ‘replays’ the cassette on each subsequent test run. This means that each subsequent run of the test suite does not make any external calls, making the suite run much, much faster. This stubs out the request in a similar way to a manual stub, but allows for a more sustainable, speedier approach.

The only drawback is that installation is a little tricky, as vcr requires some initial configuration to set up. The rest of this article will delineate the installation process and go over some of the high-level configuration options that vcr provides.

In your Gemfile, add not only vcr, but also webmock. Webmock is a Ruby library for stubbing out HTTP requests, and vcr requires it (or a similar library) to record and stub out each request and response.

# Gemfile.rb
group :test do 
  gem ‘webmock’
  gem ‘vcr’
end

And then of course run:

$ bundle install

Once these are installed, there’s some configuration required for vcr and webmock to integrate nicely with RSpec. Add the following to your spec_helper.rb file:

# spec/spec_helper.rb
require ‘vcr’
require ‘webmock/rspec’
VCR.configure do |c|
  c.hook_into :webmock
  c.cassette_library_dir = ‘spec/support/cassettes’
  c.configure_rspec_metadata!
  c.default_cassette_options = { record: :new_episodes } 
end

Fairly straightforward configuration, but let’s go through each line:

require ‘vcr’
require ‘webmock/rspec’

Includes both vcr and the webmock-rspec files in the test suite.

c.hook_into :webmock

Sets webmock as the default HTTP stubbing library for vcr.

c.cassette_library_dir = ‘spec/support/cassettes’

We need to tell VCR where store the actual YAML files that contain the information on each request/response, so this configuration variable is set to wherever you’d like to store them. I’m a fan of keeping them inside the spec/support folder, but you can put them wherever you want.

c.configure_rspec_metadata!

This option tells RSpec to invoke vcr whenever the :vcr option is passed to a code block.

c.default_cassette_options = { record: :new_episodes }

This tells RSpec to record the HTTP response of an API call when none exists by default.

And there you have it — installation is done. Now you can begin using vcr by passing in the :vcr option to code block(s) in an RSpec test file.

# spec/api/v2/widgets_spec.rb
RSpec.describe ‘API::V2::Widgets’, vcr: true do 
  # all tests making external requests in this block will use vcr
end

Not only will this significantly speed up your test suite, but it will ensure that your tests are running against a reliable response on each subsequent playback. Another advantage I’ve found with using vcr is that in environments where the continuous integration server could (should) be blocked from making HTTP calls to the internet, the test suite will obviously still run. This applies to any situation without internet connectivity as well.

One concern is that the 3rd-party API may change as you are working on your application, and while your recorded tests might pass in your test suite, in reality they could possibly be failing. Fortunately you can tell vcr to re-record cassettes at a set interval by adding it to the configuration block we saw above:

# spec/spec_helper.rb
VCR.configure do |c|
  c.default_cassette_options = { re_record_interval: (3600 * 24) } 
end

Where the interval is the number of seconds before vcr re-records each cassette. An easy way to manage this is to just multiply the number of hours you want to wait between recordings by 3600 (the number of seconds in an hour).

There’s a wealth of documentation on vcr over at Relish, but this should hopefully give you something of a ‘quick start’ on getting set up with the library. Recording makes your test suite faster, more deterministic, and more accurate. Whether you’re working in consulting, on a single-product team, or somewhere in between, those are always fantastic goals to strive for in your applications’ test coverage.