Great article. Thanks for sharing.
I agree with the safe approach, but I would like to know your opinion on this situation:
In your example, the search box without auto completion could fail on A/B tests metrics, however the search box with auto completion could be a big win. In another words, on features that complete each other, is it valid to evaluate them separately?