Introducing the new, simpler way of evaluating extractive QA pipelines in Haystack — Evaluation is an essential part of every NLP project. It lets you quantify the system’s overall performance, detect any deterioration over time, and compare it to competing configurations. Evaluating your question answering (QA) and document search pipelines in Haystack is now easier than ever. Here’s what’s new:

Modern-day natural language processing (NLP) relies largely on big and powerful Transformer models. Language models like BERT have shown an almost human-like ability to process natural language. When fine-tuned to a question answering dataset, they are great at returning meaningful answers from large document collections. But for some applications —…

Use NLP to populate your knowledge graph or structured database by extracting relations from unstructured data — Transformer-based language models excel at making sense of unstructured textual data, as evidenced by recent results in areas like question answering (QA) and machine translation. But did you know that Transformers can also extract structured information from your textual databases? Haystack’s EntityExtractor node brings information retrieval to your QA pipelines…

Our guide to natural language processing in the world’s languages — At first glance, there is no shortage of resources in natural language processing (NLP). The field boasts an active research community that is constantly building and refining models for use cases such as question answering, named entity recognition, sentiment analysis, and many more. …

Learn how to use Haystack NLP framework to extract answers from your tabular data Slowly but surely, natural language is becoming the primary interface to data. You can now do semantic question answering not only on unstructured text, but also on structured tabular data. …

How to use Haystack-based semantic question answering systems to extract information from a collection of financial statements Thanks to the Transformer-based language models and semantic search, modern natural language processing systems have an almost uncanny ability to answer questions. …

A guide to the history and the role of text vectorization in semantic search systems — For as long as we have had computers, there has been the question of how to represent data in a way that machines can work with. In natural language processing (NLP), we often talk about text vectorization — representing words, sentences, or even larger units of text as vectors (or…

Let Transformer models do the annotation work for you with Haystack’s new Question Generator — Without labeled datasets there would be no supervised machine learning: They provide the ground-truth needed to build predictive models. Trained annotators spend many hours on dataset creation — a painstaking and monotonous task. But while machine learning has managed to automate many tasks that previously depended on human expertise, data…

A timely overview of the landscape and jargon of semantic search and question answering systems — The Haystack framework is centered on question answering (QA). The concept of “search” is a big part of Haystack’s functionality, but how does it tie in? Well, a QA pipeline’s task is to first find the right documents. It then goes on to find the correct answer or answers within…