they still can’t really guarantee much of anything if you test them in circumstances that differ significantly from the circumstances on which they were trained
Has DeepMind Really Passed Go?
Gary Marcus
51424

I’ve noticed the same thing with deep learning used for character-level prediction. The media have run away with their imagination (eg “Google AI chatbot threatens human master”) but the reality is much different. It doesn’t appear they work as well for holding topic-specific conversations.