Caliper Challenges: Templates, Practice Arenas, and Colab Integration
In our last post, we outlined our guiding principles towards our mission — to measure what matters for hiring in AI, at scale.
We want to enable recruiters to evaluate practical AI and data science skills through automated bite-sized challenges on real-world problems.
In this post, we describe our first product offerings to accomplish this goal.
For recruiters and hiring managers: Templates
As a recruiter and hiring manager, you shouldn’t have to spend hours setting up pre-screening tests. Sometimes a broad test of a candidate’s practical AI and data science skills is good enough. We are now making that possible in Caliper through Templates.
Templates enable you to automatically create AI challenges in a few clicks (without any humans in the loop) and invite candidates to participate in them. Behind the scenes, we take care of setting up the exact task, an appropriate dataset, the evaluation metric, and instantiating resources to allow for variable volume of participating candidates.
We are starting with two industry-standard yet bite-sized computer vision challenges as templates — image classification on CIFAR-10 and MNIST, and these are now available to all users on our Free and Standard plans.
Recruiters and hiring managers can now easily see how different candidates approach these tasks.
For candidates: Practice Arenas and Colab Integration
The CIFAR-10 and MNIST challenges are also available now as Practice Arenas for you to get familiar with. You can browse through the challenges, train models, make submissions, and see how you do. This workflow is meant to be representative of (if not identical to) a recruiting challenge.
As we mentioned in our previous post, we believe Caliper challenges should not become yet another test to be hill-climbing on, but rather to be representative of actual skill-set, to be lightweight and bite-sized.
So even though we show you your accuracy on the challenge, these will not be visible to recruiters. Recruiters only get to see a green (completed) or red (below baseline / error) binary indicator.
In addition, to keep time and computational burden of these tasks to a minimum, we are providing fill-in-the-blank style PyTorch boilerplate code as Colab notebooks. This makes sure you focus on what matters, and don’t have to worry about library and data imports. Just write a few lines of code to define your model for this task (say CIFAR-10 or MNIST classification) and don’t worry about the overhead.
So whether you’re a recruiter or candidate, you can measure or showcase practical AI and data science skills seamlessly.
Stay tuned for more challenge templates and new domains (natural language processing, autonomous driving, etc.).
If you have any feedback for us, feel free to reach out at support@caliper.ai.
