Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis

Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis

Stať ve sborníku v databázi WoS či Scopus

This work presents the development of an advanced multimodal Retrieval-Augmented Generation (MM-RAG) framework, specifically designed to integrate and process both textual and visual data for comprehensive document analysis. Unlike traditional systems that handle only text, this framework employs cutting-edge techniques to extract and embed unstructured information from PDFs containing both text and images, ensuring a more holistic understanding of complex documents. Textual data is segmented into manageable chunks and embedded using transformer-based models, such as Gemini, which operates within a 768-dimensional embedding space to capture nuanced textual information. Simultaneously, visual data is processed through sophisticated vision-language models, which generate high-level semantic summaries that encapsulate the visual content's meaning. The MM-RAG framework then seamlessly unifies these text and image embeddings into a cohesive multimodal representation, significantly enhancing the system's ability to perform complex document retrieval and question-answering tasks. This integration enables more accurate and contextually relevant responses, making it a powerful tool for detailed document analysis.

This work presents the development of an advanced multimodal Retrieval-Augmented Generation (MM-RAG) framework, specifically designed to integrate and process both textual and visual data for comprehensive document analysis. Unlike traditional systems that handle only text, this framework employs cutting-edge techniques to extract and embed unstructured information from PDFs containing both text and images, ensuring a more holistic understanding of complex documents. Textual data is segmented into manageable chunks and embedded using transformer-based models, such as Gemini, which operates within a 768-dimensional embedding space to capture nuanced textual information. Simultaneously, visual data is processed through sophisticated vision-language models, which generate high-level semantic summaries that encapsulate the visual content's meaning. The MM-RAG framework then seamlessly unifies these text and image embeddings into a cohesive multimodal representation, significantly enhancing the system's ability to perform complex document retrieval and question-answering tasks. This integration enables more accurate and contextually relevant responses, making it a powerful tool for detailed document analysis.

Multimodal Retrieval-Augmented Generation;Document Analysis;Vision-Language Models;Transformer-Based Models;Semantic Representation;Unified Embedding Space;Question-Answering System

Multimodal Retrieval-Augmented Generation;Document Analysis;Vision-Language Models;Transformer-Based Models;Semantic Representation;Unified Embedding Space;Question-Answering System

NANDI, T.; GUPTA, S.; KAUSHAL, A.; DUTTA, M.; BURGET, R.; JEŽEK, Š.

26.11.2024

Meloneras

978-3-8007-6544-7

ICUMT 2024; 16th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops

International Congress on Ultra Modern Telecommunications and Workshops

Spojené státy americké

106

110

6