Chris Beaumont
Sep 12 · 1 min read

Hi Garrett,

Your point is valid, we are paying a 2% “opportunity tax” by not sending messages at every opportunity. It would be reasonable to remove stable, high-performing messages from the holdback to reduce this tax. However, the data can be useful even for these kinds of messages. In this case, the question is less about “is this message a net positive” (we know it is) and more along the lines of: “is the performance of this message changing suddenly?” “In retrospect, did messages last month drive higher or lower than average benefit for particular cohorts of interest?” etc. The basic bet behind a holdback like this is that the additional innovations you will derive from the holdback data will pay for the 2% tax.