Evaluation experience: Ask me Anything with Ernst

This month’s spotlight is a little different… We know there’s a lot of curiosity surrounding the work of evaluators at Unbabel, so we invited Ernst, an evaluator in Dutch language pairs, for an AMA.

Ernst is a 34-year-old law student from Gouda, the city of cheese. Although we interviewed him because of his work as an evaluator, he also works as an annotator. If you’re confused about the difference between these two functions, we have a helpful breakdown on our blog. In very basic terms, an annotator checks the final Unbabel output quality, either MT or MT+Human. Ernst admits he much prefers to evaluate human editors, then just annotate MT, “because it’s nice to give feedback to people and see them really learn,” while a machine still makes the same mistakes.

How much time do evaluators have to prepare their reviews?

Unlike editors, evaluators don’t have a “time bomb.” Usually, Ernst spends 5 minutes per 100 words on a task.

What does he think are the most important things for an editor to keep in mind while translating?

Having a good feeling for the language, and for the mistakes the machine makes.

Can editors also give feedback to an evaluator?

When you send feedback to a Community Manager about evaluation, this is taken into consideration by our quality team. However, it’s also important that we trust evaluators and their work. Evaluators never accumulate functions as editors.