Self-Retrieval: An LLM-Driven Information Retrieval Architecture for the Era of Large Language Models

[irthomasthomas]

• Title: "Self-Retrieval: An LLM-Driven Information Retrieval Architecture for the Era of Large Language Models"

Self-Retrieval: An LLM-Driven Information Retrieval Architecture for the Era of Large Language Models

Title: "Self-Retrieval: An LLM-Driven Information Retrieval Architecture for the Era of Large Language Models"

Description:

"The rise of large language models (LLMs) has transformed the role of information retrieval (IR) systems in the way to humans accessing information. Due to the isolated architecture and the limited interaction, existing IR systems are unable to fully accommodate the shift from directly providing information to humans to indirectly serving large language models. In this paper, we propose Self-Retrieval, an end-to-end, LLM-driven information retrieval architecture that can fully internalize the required abilities of IR systems into a single LLM and deeply leverage the capabilities of LLMs during IR process. Specifically, Self-retrieval internalizes the corpus to retrieve into a LLM via a natural language indexing architecture. Then the entire retrieval process is redefined as a procedure of document generation and self-assessment, which can be end-to-end executed using a single large language model. Experimental results demonstrate that Self-Retrieval not only significantly outperforms previous retrieval approaches by a large margin, but also can significantly boost the performance of LLM-driven downstream applications like retrieval augumented generation."

Authors:

Qiaoyu Tang^1,3†, Jiawei Chen^1,3, Bowen Yu⁴, Yaojie Lu¹, Cheng Fu⁴, Haiyang Yu⁴, Hongyu Lin¹†, Fei Huang⁴, Ben He^1,3, Xianpei Han^1,2, Le Sun^1,2, Yongbin Li⁴

1. Chinese Information Processing Laboratory
2. State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China
3. University of Chinese Academy of Sciences, Beijing, China
4. Alibaba Group

Affiliations:

{tangqiaoyu2020,jiawei2020,luyaojie,hongyu,xianpei,sunle}@iscas.ac.cn
{yubowen.ybw,fucheng.fuc,yifei.yhy,f.huang,shuide.lyb}@alibaba-inc.com
[email protected]

Figures:

URL:

https://arxiv.org/html/2403.00801v1

Suggested labels

{'label-name': 'Information Retrieval Paradigm', 'label-description': 'Describes the shift from traditional information retrieval systems to LLM-driven IR architectures like Self-Retrieval.', 'confidence': 67.55}

[irthomasthomas]

Related content

#680 similarity score: 0.87
#333 similarity score: 0.87
#316 similarity score: 0.86
#363 similarity score: 0.86
#706 similarity score: 0.86
#170 similarity score: 0.86