Chocolatest, a sweet story of yet another testing framework.

a preview of an exercise after checking the solution

Weeks ago, Hexacta asked me to develop an application to challenge computer-engineering students. This application was going to be used at a convention where Hexacta would had a stand. There we’d tell about us and get in touch with young people that want to do cool stuff like us :)

The idea was simple: “let’s have a web application where they can tackle some programming challenges”. The first thing that I thought of was “it would be cool if they are presented with different functions to implement, but it would be cooler if they are given some sample tests to explain how the function should work besides the specification. If we could make it so that they can add their tests and run them, that would be very neat. And what about giving them a score running some hidden tests for each exercise, that would definitely be very nice…”

I was up to the challenge and then asked how much time I had… Only a week!

It was something really cool to build so I said to myself that I would give my best in order to make it happen but there were no guarantees, there was not much time and so I began.

I started, as always, googling around to see if there was something I could make good use of. I just had to run JavaScript tests on the browser but these were not the kind of test where you have functions and assertion. These were the tests where your implementation is a string and your tests are a strings too!!!

It is very likely that you are now asking yourself what the heck is Nico talking about?! Do you remember that I said that the user would have to implement a function? Maybe using some editor as Monaco (the one that powers Visual Studio Code)? Do you also remember that I said that the user could also add more tests to the samples? Strings and strings, now you are following me, I feel better.

I had to implement a testing framework in one week that would evaluate JavaScript as strings and also make the application. Clearly there was no time to waste!!!

I had experience with Node’s vm module and I said “I need something like that”. https://github.com/browserify/vm-browserify was the answer.

Then I said, I would need some assertions but I don’t want to reinvent the wheel. Maybe I can find something like Node’s assert module, right? And again I was lucky to find https://github.com/browserify/commonjs-assert.

I should really thank the Browserify team for the only two dependencies Chocolatest has, without them I would have ran out of time!

Now that I had the building blocks, I needed a way of intercepting the calls to the different assert methods and create a log of what was going on during the execution of the tests.

The word intercepting made me think about Aspect Oriented Programming but we all know we do not have all those fancy things in JavaScript! What we do have is meta-programming thanks to the Proxy object.

Most of the magic happens in the following lines.

import { Assertion } from './types';
import * as assert from 'assert';

type Omit<T, K extends keyof T> = Pick<T, Exclude<keyof T, K>>;

const applyWrapper = (operator: string, method: (...args: any[]) => any, thisArg: any, args: any[], log: (assertion: Assertion) => void) => {
  try {
    let result = method.apply(thisArg, args);
    log({'ok': true, 'operator': operator, 'args': args, 'type': 'assert'});
    return result;
  } catch (e) {
    log({'ok': false, 'operator': operator, 'args': args, 'type': 'assert'});
    // we don't want to fast fail, we want to run all asserts in test
    // throw e; 
  }
}

const generateProxy = (log: (assertion: Assertion) => void) => {
  return new Proxy(assert,
  {
    apply(target, thisArg, args) {
      return applyWrapper('assert', target, thisArg, args, log);
    },
    get(target, propKey: keyof Omit<typeof assert, 'AssertionError'>) {
      const origMethod = target[propKey];
      return function (...args: any[]) {
        return applyWrapper(propKey, origMethod, target, args, log);
      };
    }
  });
}

export {
  generateProxy
}

First of all, you will notice that the code is written in TypeScript. If you haven’t tried it yet I suggest you do it.

Secondly, I’m sure you noticed the creation of a Proxy of the library assert, where it says “new Proxy(assert, …” and again you probably said what the heck Nico?!

Basically, what I’m simply doing is intercepting all the calls to the methods of assert (even assert itself which is a method too) and calling the original method in isolation inside a try catch block. With this interception I’m able to log when the method is about to be called, when it was called and it threw because of an exception or when it was executed successfully.

The method “generateProxy” is called with a log as an argument, this is a collector function where I will be allocating all the events described above and some more.

Now that I explained you the framework, I’ll give you the link in case you wanna play with it https://github.com/nicoabie/chocolatest and tell you about what happened to the application.

The application was developed on Vuejs using the Monaco editor to make a good coding experience and had some exercises ready for the convention time. The convention started at 9 AM and the last commit was about 9:20 AM or so. I will not brag about its success because in the end no student got to use it, hahaha! There were so many people walking around that it wasn’t an easy task to stop on a stand and apparently they were starting the University so they were a little bit shy unfortunately.

Anyways, we are at Hexacta thinking about having it published on our website, so you can be challenged with some exercises and test your level!!!

Thanks for reading and stay tuned!

P.S. I recently found out that around 500 modules are published each day on npm according to http://www.modulecounts.com/. I just want you to know that before I started to code I searched for quite a bit if such a framework existed to avoid polluting our environment further hahaha.