Introduction
In this article, I will explain what the Normalized Discounted Cumulative Gain (NDCG) metric is and how it is being used to evaluate submissions to the Visual Dialog Challenge 2019.
Motivation
I first heard about the NDCG metric while reading about the current Visual Dialog Challenge’s evaluation criteria. Unfortunately, that page did not explain what NDCG is and how its calculated. Moreover, doing a web search yields results in the context of evaluating query results which is not what this challenge is about. Hence I thought an article such as this one will help newcomers to this challenge quickly get up to speed with this metric.
The content of this article is based primarily on this challenge’s 2018 slides which walks through an example calculation and some stuff I found online. The two most useful materials I found are: this blog post from Figure Eight and this slides from an Information Retrieval course at Stanford.
Context
In a nutshell, the Visual Dialog Challenge 2019 is a competition where the goal is to build an algorithm that can converse with people about the content in an image. In this competition, participants are given a set of images to work with. Each image is accompanied by a caption describing the content of that image, a set of question and answer pairs that have occurred about that image, a new question to answer and a…
