Photo by Katya Austin on Unsplash

Machine Learning — How to keep an ‘Experiments Journal’

Machine Learning projects involve a lot of experiments. Most of them are discarded before we arrive at a point that shows some promise.

Then we work further in that direction, continuously tweaking something or the other. It could be changing anything in the pipeline: data collection, cleansing, a different learning algorithm, the cost function, the network architecture, post-processing, anything.

Along the journey, we also discard many experiments that didn’t work so well. Once we pick the one we are happy with, the history of the experiments is forgotten.

Keeping a journal of these experiments could be useful.

Why? When we involve a new member to work on the same project (or a similar one), they can easily catch up. Later if they think about improving the results, it is likely that they will bump into many of the ideas we have discarded. It need not even be a new person, the original author won’t remember all the details after a few months.

What to write in the journal:

Usually, for every experiment, we note down these points:

  • Experiment serial number: Enumerating the experiments makes it is easy to refer to them. You can just say “Experiment 8 didn’t work so well because the data cleansing had a bug, let’s retry that”.
  • Change: What did we change in this experiment? Needless to say, try to change only one thing at a time. Otherwise, you won’t know how the result is affected by a particular change. Worse, the net result might not even show any change, because one change made it better while the other made it worse.
  • Expectation: Why did we change that? What was the expectation/thinking behind this change?
  • Result: Did the results match the expectation? Keep a record of the relevant metrics and charts.
  • Remarks: Closing thoughts. Especially, if the results didn’t match the expectation, you should note down your thoughts on why they didn’t. This often leads you to the next experiment idea.
  • Link to Code: how to access the relevant code quickly. It could just commit hash or a link to the Github commit.

Hope you find this useful!

--

--

Backend engineer. Building @LearnerStash. Formerly: @Flipkart Search, @NewtonMailApp, and Webyog.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sabyasachi Ruj

Sabyasachi Ruj

108 Followers

Backend engineer. Building @LearnerStash. Formerly: @Flipkart Search, @NewtonMailApp, and Webyog.