Jest is mocking me.
I spent several days this week locked in heated battle with Jest.
As I mentioned in a previous post I’m approaching my post bootcamp projects through the lens of personal growth. So naturally, when I started building the backend for Parsley I decided to build using professional best practices. For me, that meant strong typing with typescript, clean commits with git, and most importantly — test driven development.
I’d written unit tests for all of my projects during bootcamp, so most of Jest was pretty comfortable. The one squeaky wheel was mocking. I’d run into some pretty nasty mocking bugs during bootcamp, so I put off writing those tests as long as I could. Eventually though, I ran out of other parts of the code base to test, and I had to bite the bullet.
Time to dig in. Here’s a simplified version of the file I’m testing. I’ve pulled out any logic around data validation and added a few comments for anyone who hasn’t used the API before.
const OpenAI =require("openai"); //import the OpenAI library
const openai = new OpenAI(); //create an instance of the OpenAI client
/** Accepts a string containing raw text for a recipe and returns an IRecipeBase
*
* IRecipeBase format:
* {
* name: string,
* steps: [{
* step_number: number,
* ingredients: {amount: string, description: string},
* instructions: string
* },{step2},{step3}...]
* }
*
* Throws an error if chat gpt cannot format the recipe correctly
*/
async function textToRecipe(recipeText:string):Promise<IRecipeBase>{
/**The create method here is what makes the API Call, and what
we'll eventually be mocking*/
const completion = await openai.chat.completions.create({
messages: [{
role: "system",
content: recipeText
}],
model: "gpt-3.5-turbo-1106",
response_format: { type: "json_object" },
temperature: 0
});
/** The response from OpenAI comes in as a complex object.
We'll pull just the part we care about*/
const recipeData = completion.choices[0].message.content;
const recipe = JSON.parse(recipeData);
return recipe;
}
module.exports = {textToRecipe}The challenge is straight forward. I’m using the OpenAI API to parse and structure my recipe data. It’s a surprisingly simple solution, but unfortunately not a free one. Every time I hit the API I get charged a small amount. Manageable for a small app like this, but still I have no desire to pay real money every time I run my testing suite.
The solution here is to build a fake “mock” function that simulates making an API call. Every time I would make a call to OpenAI while testing, instead I’ll run the mock function. That isolates the logic in my route (producing a better test) and neatly sidesteps any fees. Simple!
But anyone who has ever written code knows that nothing is ever as easy as it looks. My mock started with something like this:
jest.mock('openai', () => {
return jest.fn().mockImplementation(() => {
return {
chat: {
completions: {
create: jest.fn().mockImplementation(async () => {
return "recipe text";
}
}
}
};
});
});
const OpenAI = require('openai');There’s a lot going on there, so let’s unpack it a bit.
jest.mock('openai', () => {...})This line mocks the entire openai module. If I’d just called jest.mock(‘openai’) this would wrap all of the exports from the openai module in mock functions. Instead, we need access to data several levels deep (OpenAI.chat.completions.create) so we need a more specific implementation. Here Jest allows us to add a callback as a second argument. This callback is a ‘module factory’ function — a higher order function that returns another function. Since the default export of the ‘openai’ module is a class (OpenAI), this module factory allows us to return a simulated constructor function for that class. In theory, that allows us to create instances of our class and track calls to their methods.
return jest.fn().mockImplementation(() => {...}This function is our mocked constructor function. This will be called whenever an instance of our class is instantiated using the new keyword eg: const openai = new OpenAI(). Here we’re also using .mockImplementation() to define the object we want our constructor to return. Whatever object we return here should imitate the structure of an actual OpenAI instance.
return {
chat: {
completions: {
create: jest.fn().mockImplementation(async () => {
return "recipe text";
}
}
}
};Here we return the mocked instance itself and define any custom behavior. Notice that since we only access chat.completions.create in our function we don’t need to create any of the properties that an OpenAI instance would normally have.
We’ve also used jest.fn().mockImplementation() here to define what we want to happen when the create() method is called. Because this is a jest.fn(), we can keep track of things like how many times create was called and what arguments it was passed.
const OpenAI = require('openai');Last but not least, we import our actual openai module. Because we have already defined our mock implementation, the actual functionality of this module is never actually imported, but we still need this line in order to access our mocks. Note that since I’m using CommonJS in this project my mocks aren’t hoisted, so it’s important to define the mock implementation BEFORE importing the module itself.
Woof. That’s a lot to learn about for what amounts to about 4 lines of code, but I suppose that’s the journey we’re on! But of course, it’s not over. Nothing ever works on the first try.
The problem:
We’re interested in testing whether the mocked create method was called, and maybe some data about what arguments it’s getting passed and how many times we’re calling it.
When you use a module factory to create mocked instances using the new keyword, Jest keeps track of them in an array located at OpenAI.mock.instances . When we access OpenAI.mock.instances[0] we should be able to reach the chat.completions.create stub we defined earlier.
Theoretically.
Instead, when I console logged the instance what I saw was this:
mockConstructor{}.
Huh?
After a lot of research, I learned that its correct for Jest to return something called a mock constructor object in this situation. After all, we’re not creating real instances of our OpenAI class as defined in the OpenAI module. What I couldn’t explain was why the object was empty. As far as I could tell, we should be logging something that looks more like this:
mockConstructor{
chat:{
completions:{
create:(our mocked function)
}Tests in insomnia showed that my code was working fine. Even more perplexing, when I instantiated the openai module in my module and imported the instance into the test file, everything logged as a mock just I would expect.
And honestly, I could’ve stopped there. I could have just imported the instance into my test files and used that for my assertions. But dammit, that was unsatisfying. It created more clutter in my exports, and it didn’t feel like following best practices.
So instead I spent nearly 2 days troubleshooting. I argued with ChatGPT, who insisted that I could just type OpenAI.mock.instances[0] no matter how many ways I tried to explain the issue. I dove deep into the Jest documentation. I rewrote my code 100 different ways.
The solution:
The source of my issue? I still have no idea. As far as I can tell, Jest is having trouble tracking my instances due to something in my environment. If you’re still reading at this point (God bless you) and know what the issue might be, please reach out. I’d love to understand.
Eventually I did find a solution that felt a little nicer than importing my instance.
const mockCreate = jest.fn().mockImplementation(async () => { return "recipe text"; });
jest.mock('openai', () => { // moduleFactory function
return jest.fn().mockImplementation(() => { // returns a constructor
return { // this is our mocked constructor
chat: {
completions: {
create: mockCreate
}
}
};
});
});
const OpenAI = require('openai');All I needed to do here was create a direct reference to the mocked Create function. With this reference, I don’t need to dig into the instances array at all. I can track calls to chat.completions.create directly through the mockCreate variable.
Viola! Everything works.
Lessons:
Even though I never fully diagnosed the source of the problem, this tricky bug taught me SO much about mocking. I dug deeper into the concept and documentation than I ever would have if everything had worked right away.
In fact, my research paid off almost immediately. When it came time to test the route that uses this function I was able to create a mock and get my tests working in under 10 minutes.
With any luck, next time I need to use mocking in my tests I won’t feel the need to put it off till the last minute.