[Text-to-SQL] Learning to Query Tables with Natural Language

How to model a natural language interface for relational databases

The views expressed on this post are mine alone and do not reflect the views of my employer, Microsoft.

Text-to-SQL is a task to translate a user’s query spoken in natural language into SQL automatically. It is the project that I’m working on at Microsoft.

If this problem is solved, it’s going to be widely useful because the vast majority of data in our lives is stored in relational databases. In fact, healthcare, financial services, and sales all seem to use relational databases exclusively. Interestingly, industries that can’t afford to lose transactions seem to only use the relational databases. (You can risk losing social media comments here and there but you don’t want to risk losing transaction records of your credit card.) Also, writing SQL queries can be prohibitive to non-technical users. Bill Gates noticed this problem and he himself(!) wrote down 105 questions (my team is working on 70 of them) that he wants a machine to be able to answer given the enterprise databases. If we could let people directly interact with large enterprise databases using natural language or voice, it would be very useful.

In the past two years, NLP downstream tasks have come a long way thanks to contextualized embeddings on webscale data and transfer learning. This progress gave me hope that we might be able to solve this relatively…