The expansion of question-answering (QA) systems driven by artificial intelligence (AI) results from the increasing demand for financial data analysis and management. In addition to bettering customer service, these technologies aid in risk management and provide individualized stock suggestions. Accurate and useful replies to financial data necessitate a thorough understanding of the financial domain because of the data’s complexity, domain-specific terminology and concepts, market uncertainty, and decision-making processes. Due to the complex tasks involved, such as information retrieval, summarization, analysis of data, comprehension, and reasoning, long-form question answering (LFQA) scenarios have added significance in this setting.
While there are several LFQA datasets available in the public domain, such as ELI5, WikiHowQA, and WebCPM, none of them are tailored to the financial sector. This gap in the market is significant, as complex, open-domain questions often require extensive paragraph-length replies and relevant document retrievals. Current financial QA standards, which heavily rely on numerical calculation and sentiment analysis, often struggle to handle the diversity and complexity of these questions.
In light of these difficulties, the researchers from HSBC Lab, Hong Kong University of Science and Technology (Guangzhou), and Harvard University present FinTextQA, a new dataset for testing QA models on issues pertaining to general finance, regulation, or policy. This dataset is composed of LFQAs taken from textbooks in the field as well as government agencies’ websites. The 1,262 question-answer pairs and document contexts that makeup FinTextQA are of excellent quality and have the source attributed. Selected from five rounds of human screening, it includes six question categories with an average text length of 19,7k words. By incorporating financial rules and regulations into LFQA, this dataset challenges models with more complex content and represents ground-breaking work in the field.
The team introduced the dataset and benchmarked state-of-the-art (SOTA) models using FinTextQA to set standards for future studies. Many existing LFQA systems depend on pre-trained language models that have been fine-tuned, such as GPT-3.5-turbo, LLaMA2, Baichuan2, etc. However, these models aren’t always up to answering complex financial inquiries or providing thorough answers. They end up using the RAG framework as a response. The RAG system can improve LLMs’ performance and explanation capacities by pre-processing documents in various steps and providing them with the most relevant information.
The researchers highlight that FinTextQA has fewer QA pairs despite its professional curation and high quality in contrast to bigger AI-generated datasets. Because of this restriction, models trained on it may not be able to be extended to more general real-world scenarios. Acquiring high-quality data is difficult, and copyright constraints frequently hinder sharing it. Consequently, cutting-edge approaches to data scarcity and data augmentation should be the focus of future studies. It may also be useful to investigate more sophisticated RAG capabilities and retrieval methods and broaden the dataset to include more diverse sources.
Nevertheless, the team believes that this work presents a significant step forward in improving financial concept understanding and support by introducing the first LFQA financial dataset and performing extensive benchmark trials on it. FinTextQA provides a robust and thorough framework for developing and testing LFQA systems in general finance. In addition to demonstrating the effectiveness of different model configurations, the experimental research stresses the importance of improving existing approaches to make financial question-answering systems more accurate and easier to understand.
Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.
If you like our work, you will love our newsletter..
Don’t Forget to join our 42k+ ML SubReddit
The post FinTextQA: A Long-Form Question Answering LFQA Dataset Specifically Designed for the Financial Domain appeared first on MarkTechPost.