The Challenge of Coding with Randomness

Traditional vs. Probabilistic Software

This post is an older iteration of our weekly newsletter High Learning Rate. Follow it to get more of such posts 1 month before!

When dealing with conventional software engineering, we write code that is executed in a fully predictable environment. The result doesn’t change between runs. We have full control over the logic and can anticipate the results of our programs. We also rely on unit tests to validate each part of our code, ensuring everything works as expected. Typical tests in software engineering check the correctness of rules coded, error handling, or response times.

On the other hand, working with probabilistic models is completely different. Here, we deal with systems that produce outputs based on probabilities and learned patterns from vast datasets. Nothing is fixed or completely deterministic. Unlike regular code, the results can vary even with the same input, introducing a layer of unpredictability. This unpredictability is both a challenge and a feature, as it allows models to generate creative and nuanced responses. You can easily test this on ChatGPT when you regenerate a response from the model. It will generate different outcomes each time.

You may say that you can just drop the temperature to zero. However, for various reasons, including variations of server hosting or floating-point precision limitations in neural network computations, even subtle numerical variations can…