Anki Design Study: Learning Statistics

Here’s a card that recently came up for refactoring in my biweekly design review for my Anki knowledge base:

I’ve had this card for about a year and a half. Its ease factor is at 205%, so I’ve been getting it right. Its interval is up to 1.5 years, so I won’t have to do active recall of it again for a while. By the numbers, it’s a successful flash card. And it’s useful: I made this card because I occasionally need to compute and visualize confidence intervals around data in R or Python, and knowing the equation reduces the time and cognitive effort it takes for me to Google for the right syntax.

But there’s a huge card-design smell here: I learned this equation by rote memorization, which means I can’t really explain why the equation has the form it does. This makes it difficult to recall and unpleasant to review (even when I get it right), and makes it difficult to reason about when I need this knowledge in real life.

I’m dimly aware that it has something to do with the critical value of a t-distribution, but all that stuff gets really fuzzy really fast as soon as you step out of stats class.

The basic way to recognize cards like this is by what goes through your mind when you get it wrong:

• When I get a good card wrong, it triggers me to think “oh, of course, I see why, silly me.”
• When I get a rote memorization card wrong, instead it triggers me to think “oh shoot. Guess I’ll try harder next time.”

In general, learning equations by rote memorization is a risky idea in Anki. The entire point of an equation is that it condenses a lot of really complex logic — often a whole paragraph’s worth — into a small amount of space. And complex logic calls for multiple flash cards.

So let’s fix this card. We’ll follow the two-part strategy I usually use for scientific topics:

1. Identify conceptual landmark concepts to help orient our intuition, and then
2. add well-motivated detail cards that are related to those landmarks.

Big-Picture Landmarks

Nabbing the intuition behind statistical problems is tricky, because the field is full of technical vocabulary and precise assumptions that stats writing takes for granted. Things like population, sample mean, standard error, and standard deviation, and distributions can get murky pretty quickly (and that’s before we even get to the difference between a p.d.f., p.m.f., c.d.f., p.p.f., and i.s.f., among much much more).

First, what are we even doing here? I vaguely remember from my professors’ Sisyphean efforts to teach sound experimental methods to us computer scientists (ha!) that the mean of any data sample magically follows a nice theoretical distribution (even if the data proper does not). That’s why we’re able to calculate confidence intervals in the first place. But I would struggle to explain what that means.

Google Images to the rescue (here’s the original image source). The concept handle I’m missing is the sampling distribution. In general, it’s way easier to summarize a concept like this with an image than with a text-based explanation, so I often rely on images to express the answer to a question:

With this image permanently cemented in our mental library (this is what makes “concept handles” and technical vocabulary powerful), we’re already much better prepared to remember confidence intervals. Just being able to clearly and easily remember that this is the distribution function the confidence interval comes from can have a big impact.

But before we move on, we can grab one more bang for our landmark buck:

Awesome. We’re done with the landmarks — now on to the meat!

Detail Cards

As with any complex concept, we want to hit equations with Anki by breaking them down and becoming directly familiar with each piece on its own.

For equations, I often do this by making cards that ask about each term or symbol. This helps turn each piece into its own intuitive concept handle:

This is really great, because it focuses my attention much more than the original whole-equation card. Now I can take the time to think carefully about that complex “1-α/2;n-1" subscript, for example (which is the hardest part to remember of the whole gig).

Each of these terms is still pretty complex, though, and we’re still doing a bit of rote memorization. Let’s hit them both one more time:

Now I feel prepared to explain where each part of the confidence interval equation comes from. This makes it easier to remember, but more importantly it makes it easier to reason about, because I’m relying far more on semantic intuition and less on rote memorization than I was before.

There’s a lot more than we can add here, to grow our network of semantic associations into a bigger knowledge base toward more depth (“why does the standard error of the mean decrease with the square root of n?”) or breadth (“when do we use a t-distribution vs. a z-distribution to compute confidence intervals around a mean?”). But we’ve met our goal: we refactored one card, so our design review is completed for today (okay, I’ll be honest, it took me two days to get through this one. Some refactoring tasks are too big for one sitting!).

I just want to add a couple acronym cards to cement my new concept handles, then I’m finished for today:

If you’re wondering, in this case I kept the original card as is. The whole-equation card was fine, it just needed some supporting analysis. Together, all these cards are easier to remember than any of them would be in isolation.

EDIT: If you’re using Anki for mathematical topics, you may also fine value in Michael Nielson’s post on Using spaced repetition systems to see through a piece of mathematics.