There is a saying that you should “not cross disciplinary boundaries unescorted”, and I think that is Yoav’s main message. I wholeheartedly agree with Yoav’s points. I have a sense of deja vu, because ML folks (including me) made similar mistakes when we first started working on problems in computational biology. The early papers were pretty embarrassing.
That said, it would be a great service to the DL community for someone from NLP to write a tutorial defining good challenge problems and evaluation methodologies. The DL world is full of enthusiastic young researchers, and while they do deserve criticism for bad work, they would also benefit tremendously from guidance, mentoring, and good role models.