New Features in Haystack v1.0

We have been investing our time and effort into building Haystack because we believe in a future where state-of-the-art search is easily implementable in any software. Search technologies have been quietly evolving in the academic space and Transformer models have ushered in a paradigm shift. Previously aspirational tasks like question answering, semantic document search and summarization are now firmly in the realm of both the possible and the practical.

Today we have taken a big step fulfilling the vision by releasing Haystack v1.0! Key features include improved evaluation, table question answering, improved debugging and more. Along with this, our our public demo is now live! Thank you so much to our community contributors whose effort and enthusiasm has propelled us to where we are now. We hope that you enjoy the ease of use that these features bring and we hope to hear of your successes in the near future!

Note that with this release, there are a few breaking changes with regards to import statements, input document format and primitive classes. Have a look at our migration guide for more details.

Demo Website

The quickest way to interact with a running Haystack system is through our new demo website! While traveling around the world is tricky these days, we have you covered. Explore facts about cities and countries with your keyboard and your favorite NLP models instead. We have indexed a slice of Wikipedia articles about countries and capital cities and in the back end, there is an Elasticsearch backed document store, a dense passage retriever and a RoBERTa based reader model.

Improved Evaluation

Haystack’s evaluation features are there to help you find out how well your system is doing on your data. It provides pipeline level evaluation to ensure that the system’s output is really what you’re after, but also node level evaluation so that you can figure out whether its your reader or retriever that is holding back the performance.

In this release, evaluation is much simpler and cleaner to perform. All the functionality is now baked into the Pipeline class and you can kick off the process by providing Label or MultiLabel objects to the Pipeline.eval() method.

eval_result = pipeline.eval(
  labels = labels,
  params = { "Retriever":
    {"top_k": 5} },
)
Copied

The output is an EvaluationResult object which stores each Node’s prediction for each sample in a Pandas DataFrame. There is a EvaluationResult.calculate_metrics() method which will return the relevant metrics for your evaluation.

metrics = eval_result.calculate_metrics()
Copied

If you’d like to start evaluating your own systems on your own data, check out our Evaluation Tutorial!

Table QA

A lot of valuable information is stored in tables — we’ve heard this again and again from our community. While they are an efficient structured data format, it hasn’t been possible to search for table contents using traditional NLP techniques. But now, with the new TableTextRetriever and TableReader our users have all the tools they need to query for relevant tables and perform question answering.

The TableTextRetriever is the result of our team’s research into table retrieval methods which you can read about in this paper that was presented at EMNLP 2021. Behind the scenes, it uses three Transformer-based encoders — one for text passages, one for tables and one for the query. However, in Haystack, you can swap it in for any other dense retrieval model and start working with tables. The TableReader is built upon the TAPAS model and when handed table containing documents, it can return a single cell as an answer or perform an aggregation operation on a set of cells to form a final answer.

retriever = TableTextRetriever(
  document_store = document_store,
  query_embedding_model = "deepset/bert-small-mm_retrieval-question_encoder",
  passage_embedding_model = "deepset/bert-small-mm_retrieval-passage_encoder",
  table_embedding_model = "deepset/bert-small-mm_retrieval-table_encoder",
  embed_meta_fields = [ "title", "section_title" ] 
)

reader = TableReader(
  model_name_or_path = "google/tapas-base-finetuned-wtq",
  max_seq_len = 512
)
Copied

Have a look at the Table QA documentation if you’d like to learn more or dive into the Table QA tutorial to start unlocking the information in your table data.

Improved Debugging

We’ve made debugging in Haystack much simpler and also more informative! As long as your node receives a boolean debug argument, it can propagate its input, output or even some custom information to the output of the pipeline. It is now a built in feature of all existing nodes and can also easily be inherited by your custom nodes.

result = pipeline.run(
  query = "Who is the father of Arya Stark?",
  params = {
    "debug": True 
  } 
)

# Result

{ 'ESRetriever': {
  'input': { 'debug': True,
    'query': 'Who is the father of Arya Stark?',
    'root_node': 'Query',
    'top_k': 1 },
  'output': {
    'documents': [ <Document: {'content': "\n===In the Riverlands===\nThe Stark army reaches the Twins, a bridge strong", ...}> ] 
    ...}
Copied

FARM Code Migration

Those of you following Haystack from its first days will know that Haystack first evolved out of the FARM framework. While FARM is designed to handle different NLP models and tasks, Haystack gives full support to search and question answering use cases with a focus on coordinating all components that take a proof-of-concept into production.

Haystack has always relied on FARM for much lower level processing. For example, the actual handling of Transformer models is done in FARM. It also deals with the text on a tokenized level, splitting text to fit a model’s maximum sequence length and picking the valid and optimal QA answer span amongst multiple chunked passages.

While the dependency of Haystack on FARM worked, it made maintenance much trickier as some features would necessitate a change in both repos and an update to the version dependency. To reduce this overhead, we have migrated the parts of FARM that are used in Haystack and put them in the haystack/modeling folder. We’ve also found that this simplifies lower level debugging as all the code is now in the one repo!

What are you waiting for?

Haystack v1.0 is out now so start using it today! See our GitHub repo if you want to install from the source code or see our Get Started page to learn about our other installation methods.