PhD Seminar • Information Retrieval • Improving Text Retrieval via Fine-Tuning Large Language Models

Please note: This PhD seminar will take place in DC 2585.

Xueguang Ma, PhD candidate
David R. Cheriton School of Computer Science

Supervisor: Professor Jimmy Lin

While large language models (LLMs) have demonstrated impressive NLP capabilities, existing information retrieval applications mainly focus on prompting LLMs for query expansions or generating permutations for listwise reranking.

In this presentation, I will introduce our recent work on fine-tuning LLaMA for multi-stage text retrieval. Specifically, we fine-tune the open-source LLaMA-2 model as a dense retriever (repLLaMA) and a pointwise reranker (rankLLaMA). Our study shows that fine-tuned LLM retrieval models outperform smaller models, proving to be more effective and exhibiting greater generalizability with only a straightforward training strategy. Moreover, our pipeline allows for the fine-tuning of LLMs at each stage of a multi-stage retrieval pipeline, demonstrating the strong potential for optimizing LLMs to enhance a variety of retrieval tasks. Furthermore, as LLMs are naturally pre-trained with longer contexts, they can directly represent longer documents. This eliminates the need for heuristic segmenting and pooling strategies to rank long documents. Alongside our work, I will also introduce the most recent works on large embedding models that show potential for handling longer context windows and following user instructions.