Remember the days of the old school yard?

A different lens on defining programmer performance.

Written by Paul Bracewell, CAO, CoderCred.

Anyone who has been near a classroom will be familiar with tests. The intent of tests is to identify what we do and don’t know. This is a critical part of the education process to create a feedback loop for continued personal growth. Similar thinking is applied to aptitude testing for recruitment purposes. The underlying assumption is that the testing process is an indication of likely on-the-job performance. That’s why we give prospective recruits a challenge to see if their game is as good as their talk.

Speaking of game, we’ve found over the years that stats aren’t necessarily the greatest dinner conversation. But, sports stats; now, that’s a completely different kettle of fish. Sports stats, university exams and aptitude testing are similar in that they attempt to indicate how good someone is. We often dig deeper and assume that those performances are an indication of ability.

Optimising individual and team performance through data

It’s the assumption that performance metrics are an indication of ability that has so many sports fans clamouring after statistics regarding athlete performances. Think Moneyball by Michael LewisI. Apart from the mistake in not casting Brad Pitt as the analyst in the movie adaptation of the book, the essence of the story is how Billy Beane was able to optimise team selection for the Oakland A’s baseball team using analytics ahead of gut feel. Ultimately, Oakland were able to field a highly competitive team at a fraction of the cost of the dominant clubs, such as the New York Yankees.

This is just one example. However, there is no shortage of academic literature mapping individual and team performance to outcomesII, which validates the assumption that ability is a predictor of performance. We often delve into the world of cricketIII to test and publish our thinking in this regard. And sometimes rugbyIV so we can join in the conversations at parties. However, the challenge is how to quantify ability. Armed with this body of knowledge, the business drivers and numerous cans of red bull, we developed a framework to test a range of hypotheses relating to programming performance.

CoderCred: quantifying programming ability and predicting performance

At the heart of what we are trying to achieve at CoderCred is the notion of quantifying programming ability. If we crack a meaningful measure of ability, we should be able to predict a coder’s performance on a test conducted within a reasonable time horizon. This in turn should be a strong indicator of on-the-job performance; which is where the real value of the CoderCred approach kicks in.

Industry relevant, time bound programming challenges

CoderCred measures programming performance by using a set of industry relevant, time bound programming challenges. We have found that predictive performance is characterised by accuracy and timeliness. The weightings we apply for combining accuracy and speed are the result of an heuristic optimisation algorithm designed to maximise the predictive performance for subsequent challenges. Each of these challenges is weighted by the level of difficulty, much like gymnastics (except the coders aren’t expected to raise their hands above their heads when they have finished a floor routine). The combination of accuracy, timeliness and difficulty combines elegantly together to predict performance (both better and worse) on different challenges of varied difficulty.

Identify candidates with the greatest chance of success

The CoderCred framework is constructed on the premise that performances are a manifestation of ability and ability is a predictor of performance. Simply, the performance on well-constructed, relevant tests is a strong predictor of future performance. This can be used to help identify candidates that have the greatest chance of success, allowing a programmer’s actions to speak louder than their words.

Written by

Reference

I:

Lewis, M. (2003). Moneyball: the art of winning an unfair game. New York City: W.W. Norton & Company.

II:

Croucher, J. (2010). Player ratings in one-day cricket. In Proceedings of the fifth Australian conference on mathematics and computers in sport. Sydney University of Technology, Sydney, NSW, 95–106.

Gerber, H., & Sharp, G. (2006). Selecting a limited overs cricket squad using an integer programming model. Journal of the Operations Research Society, 5.

Leitner, C., Zeileis, A., & Hornik, K. (2010). Forecasting sports tournaments by ratings of (prob) abilities: A comparison for the euro 2008. International Journal of Forecasting, 26(3), 471–481.

Manage, A. B., & Scariano, S. M. (2013). An introductory application of principal components to cricket data. Journal of Statistics Education 21, 3.

Saaty, T. (1987). The analytic hierarchy process. Math Modelling, 9, 161–176.

Sargent, J. (2013). Player ratings in continuous and discrete team sports. PhD thesis, RMIT University.

Schumaker, R. P., Solieman, O. K., & Chen, H (2010). Predictive modeling for sports and gaming. Springer, 2010.

Sharp, G., Brettenny, W., Gonsalves, J., Lourens, M., & Stretch, R. (2011). Integer optimisation for the selection of a Twenty20 cricket team. Journal of the Operational Research Society, 62(9), 1688–1694.

Sinuany-Stern, Z. (1988). Ranking of sports teams via the AHP. Journal of the Operational Research Society, 661–667.

Stefani, R. (2011). The methodology of officially recognized international sports rating systems. Journal of Quantitative Analysis in Sports, 7(4).

III:

Bracewell, P. J., & Ruggiero, K. (2009). A Parametric Control Chart for Monitoring Individual Batting Performances in Cricket. Journal of Quantitative Analysis in Sports , 5 (3), pp. 1–19.

Bracewell, P. J., Farhadieh, F., Jowett, C. A., Forbes, D. G., & Meyer, D. H. (2009). Was Bradman Denied His Prime? Journal of Quantitative Analysis in Sport , 5 (4), pp. 1–24.

Bracewell, P. J., Downs, M. C. F., & Sewell, J. W (2014). The development of a performance based rating system for limited overs cricket. In MathSport 2014.

IV:

Bracewell, P.J., Forbes, D.G., Jowett, C.A., & Kitson, H.I.J. (2009). Determining the evenness of domestic sporting competition using a generic rating engine. Journal of Quantitative Analysis in Sports 5, 1.

Bracewell, P.J. (2003). Monitoring meaningful rugby ratings. Journal of Sports Sciences, 21, 611–620.