Our thesis summary delves into the groundbreaking progress made in Zero-Shot Open Domain Question Answering (ODQA), harnessing the power of state-of-the-art language models. Unleashing the Power of State-of-the-Art Language Models in Zero-Shot Question Answering for Technical Topics? We explore the latest advancements and their implications, providing a glimpse into the future of natural language processing. For a comprehensive understanding, delve into our whitepaper, where we delve deeper into our findings and their potential impact.

This thesis embarks on a mission to explore and compare the performance of State-of-the-Art Language Models in the challenging realm of Zero-Shot Open Domain Question Answering, with a focus on technical subjects, particularly in the domains of cloud technology and containerization. While traditional Question Answering has been predominantly extractive, recent years have witnessed a paradigm shift in Natural Language Processing towards a more abstract Natural Language Generation approach. We propose a novel two-step architecture that aims to answer questions from an untrained set of documents, bypassing the need for prior training or fine-tuning. Our investigation goes beyond Retriever-Reader methods (e.g., BERT, RoBERTa) and delves into the evaluation of Retriever-Generator systems (e.g., GPT, FLAN-T5) through the lens of Long Form Question Answering. We employ the Amazon Web Services dataset as a benchmark to evaluate the performance of our zero-shot Open Book Question Answering system [1].

To obtain empirical results, we employ various techniques, including the subdivision of documents into smaller sections like paragraphs or passages. Additionally, we rigorously analyze the hyperparameters involved in document splitting using a sliding window approach. Our research underscores the significance of proper document preprocessing and meticulous hyperparameter selection in achieving outstanding results.

The empirical findings reveal the exceptional capabilities of RoBERTa-large, which achieves a groundbreaking State-of-the-Art F1 score of 59.19, surpassing the baseline by 18.66 and outperforming the best results from the original study by 16.99. However, we caution that generative models and Long Form Question Answering come with their own inherent biases and risks. We also observe that in contexts where the model’s complexity greatly surpasses the evaluation metrics, the relevance and interpretability of those metrics become questionable. In this regard, Semantic Answer Similarity and METEOR prove invaluable for analyzing diverse model outputs, as they are not reliant on lexical overlap, which is the case with traditional metrics like ROUGE, BLEU, F1, and EM. Furthermore, our research demonstrates the advantages of document splitting into passages, although we acknowledge that this approach may not universally excel across all use cases, with optimal hyperparameter values expected to vary depending on specific applications.

Unlock the full depth of our research on advancing zero-shot Open Domain Question Answering with State-of-the-Art Language Models. Explore the complete thesis for a comprehensive analysis of our findings and their implications for the future of NLP and Question Answering systems.