With Vector Databases making waves in the world of technology, the question that lingers is, “Are they here for the long haul?” It seems like Large Language Models (LLMs) are often grappling in the middle ground when it comes to lengthy inputs. A recent study titled “Lost in the Middle: How Language Models Use Long Contexts,” undertaken by Stanford researchers, dives deep into this matter. Here are my major insights from this study:
Study Purpose:
The primary aim was to assess and gauge the way LLMs utilize context by pinpointing the pertinent information within it.
Approach and Methodology:
The researchers employed both open-source models (MPT-30B-Instruct, LongChat-13B(16K)) and closed-source models (OpenAI’s GPT-3.5-Turbo and Anthropic’s Claude 1.3) in their study. Their implementation involved multi-document question-answering where the context included numerous fetched documents and one accurate answer, the position of which was continuously altered. They also used a key-value pair retrieval method to analyze the impact of longer contexts on performance.
Key Takeaways:
The study concluded that the highest performance is achieved when the relevant information is positioned at the beginning. However, as the context length increases, the performance declines. Too many fetched documents also have an adverse effect on performance. The researchers suggested that improving the retrieval and prompt creation process with Cross-Encoders (ranking) could potentially enhance performance by as much as 20%. Furthermore, the extended-context models, such as GPT-3.5-Turbo vs. GPT-3.5-Turbo (16K), didn’t show better results if the prompt aligned with the original context.
To delve deeper into the research, you can access the full paper here.
When we blend Retrieval with Ranking, it seems to generate the best performance in Retrieval-Augmented Generation (RAG) for Question Answering.
