A Multimodal Graph-Based Approach for Retrieval Augmented Generation

A Multimodal Graph-Based Approach for Retrieval Augmented Generation

Kosogorov D.E. (IDSTU SB RAS, Irkutsk, Russia)
Kopylov D.E. (IDSTU SB RAS, Irkutsk, Russia)
Okhotin I.I. (IDSTU SB RAS, Irkutsk, Russia)
Poddubny A.I. (IDSTU SB RAS, Irkutsk, Russia)
Tobola K.V. (IDSTU SB RAS, Irkutsk, Russia)
Dorodnykh N.O. (IDSTU SB RAS, Irkutsk, Russia)
Mikhailov A.A. (IDSTU SB RAS, Irkutsk, Russia)

Abstract

Retrieval-Augmented Generation (RAG) has become an effective solution to fundamental problems inherent in all Large Language Models (LLM). This method enhances LLM by extracting relevant external knowledge and embedding it in the model’s context. This paper proposes a new multimodal graph-based approach, RAG, for complexly structured PDF documents. Traditional RAG systems ignore the spatial structure of documents, suffer from the so-called “modal gap” problem, and do not model the relationships between individual document fragments. The developed solution combines region-oriented chunking, a specialized table encoder based on a pre-trained multimodal model, and the integration of structural and semantic graphs into a single retrieval and generation pipeline. Preliminary experimental evaluation using the MMLongBench-Doc and LongDocURL benchmarks confirmed the superiority of the region-based chunking method over the classic RAG method across all evaluation metrics at the retrieval stage. The results support the hypothesis of the need for explicit modeling of complex intermodal dependencies. This approach opens up prospects for creating reliable systems for analyzing corporate, scientific, and technical documentation.

Keywords

natural language processing; retrieval-augmented generation; large language models; large multimodal language models; complexly structured documents; embedding.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 199-226

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(6)-33

For citation

Kosogorov D.E., Kopylov D.E., Okhotin I.I., Poddubny A.I., Tobola K.V., Dorodnykh N.O., Mikhailov A.A. A Multimodal Graph-Based Approach for Retrieval Augmented Generation. Proceedings of the Institute for System Programming, vol. 38, issue 6, part 2, 2026, pp. 199-226 DOI: 10.15514/ISPRAS-2026-38(6)-33.

Full text of the paper in pdf (in Russian) Back to the contents of the volume