Large Language Model Operations Fundamentals — Part 2
Evaluation, test coverage, and fine-tuning
In Part 1, we discussed open-source models, proprietary models, and model evaluation. This part is going to be more focused on further evaluation, “test coverage”, and fine-tuning.
LLM Metrics
It’s challenging to determine which metrics to measure for generative models, and there’s a focus on task-specific performance for traditional machine learning compared to coherence and clarity for language models. When testing a language model, it’s important to consider what data to test it on and what metric to use. It’s not fair to summarize a model’s performance in a single metric, as performance can vary greatly depending on the subject matter.
Language models trained on the internet will always drift, and the measurement of output is qualitative, making it difficult to define success with a number. To build an evaluation set for your task, start incrementally and continuously add data as you discover new failure modes or patterns in the model’s behavior. Interesting examples should be organized into a small data set that can be tested every time there is a change to the model. Hard examples are particularly interesting. As a junior data scientist, it’s important to continuously improve your model to…
