«`html
Understanding the Target Audience for FEEDER
The target audience for FEEDER: A Pre-Selection Framework for Efficient Demonstration Selection in LLMs primarily consists of researchers, data scientists, and AI practitioners working with large language models (LLMs). This audience is typically engaged in developing, fine-tuning, and deploying AI models for various applications, including natural language processing, sentiment analysis, and reasoning tasks.
Pain Points
- Difficulty in selecting the most representative demonstrations from extensive training datasets.
- High computational costs associated with current demonstration selection methods.
- Challenges in maintaining LLM performance as the number of training examples increases.
Goals
- To enhance the efficiency of demonstration selection without compromising model performance.
- To reduce the size of training datasets while retaining essential information.
- To improve the stability and reliability of LLMs across various tasks.
Interests
- Innovative methods for optimizing LLM performance.
- Research on few-shot learning and in-context learning techniques.
- Applications of LLMs in real-world business scenarios.
Communication Preferences
The audience prefers clear, concise, and technical communication that includes data-driven insights and peer-reviewed statistics. They value practical examples and case studies that illustrate the application of research findings in business contexts.
Overview of FEEDER
Large language models (LLMs) have shown exceptional performance across multiple tasks through few-shot inference, also known as in-context learning (ICL). A significant challenge in this area is selecting the most representative demonstrations from large training datasets. Early methods relied on similarity scores between examples and input questions, while current approaches incorporate additional selection rules to enhance efficiency. However, these improvements often lead to increased computational overhead as the number of shots rises.
Researchers from Shanghai Jiao Tong University, Xiaohongshu Inc., Carnegie Mellon University, Peking University, University College London, and the University of Bristol have introduced FEEDER (FEw yet Essential Demonstration prE-selectoR). This method identifies a core subset of demonstrations that contain the most representative examples from training data, tailored to specific LLMs. FEEDER employs «sufficiency» and «necessity» metrics during the pre-selection stage, utilizing a tree-based algorithm to construct this subset. Notably, FEEDER reduces training data size by 20% while maintaining performance and integrates seamlessly with various downstream demonstration selection techniques in ICL across LLMs ranging from 300M to 8B parameters.
Evaluation and Results
FEEDER has been evaluated on six text classification datasets: SST-2, SST-5, COLA, TREC, SUBJ, and FPB, covering tasks from sentiment classification to textual entailment. It has also been assessed on reasoning datasets like GSM8K, semantic-parsing datasets such as SMCALFlow, and scientific question-answering datasets like GPQA. The official splits for each dataset were followed to obtain training and test data. Multiple LLM variants were utilized for performance evaluation, including GPT-2, GPT-neo (1.3B parameters), GPT-3 (6B parameters), Gemma-2 (2B parameters), Llama-2 (7B parameters), Llama-3 (8B parameters), and Qwen-2.5 (32B parameters).
Results indicate that FEEDER enables the retention of nearly half the training samples while achieving superior or comparable performance. In complex tasks, LLMs like Gemma-2 show improved performance with FEEDER, even in scenarios where LLMs typically struggle. FEEDER effectively manages larger numbers of shots, addressing performance drops that occur when increasing examples from 5 to 10 due to noisy or repeated demonstrations. By evaluating the sufficiency and necessity of each demonstration, FEEDER minimizes negative impacts on LLM performance and enhances stability.
Conclusion
In summary, FEEDER is a demonstration pre-selector designed to leverage LLM capabilities and domain knowledge to identify high-quality demonstrations through an efficient discovery approach. It reduces training data requirements while maintaining comparable performance, offering a practical solution for efficient LLM deployment. Future research directions include exploring applications with larger LLMs and extending FEEDER’s capabilities to areas such as data safety and management. FEEDER represents a significant advancement in demonstration selection, providing researchers and practitioners with an effective tool for optimizing LLM performance while reducing computational overhead.
Check out the Paper. All credit for this research goes to the researchers of this project.
Meet the AI Dev Newsletter read by 40k+ Devs and Researchers from NVIDIA, OpenAI, DeepMind, Meta, Microsoft, JP Morgan Chase, Amgen, Aflac, Wells Fargo, and hundreds more. SUBSCRIBE NOW
«`