TFG - Redesigning the User Experience with Artificial Intelligence: Fashion Industry Specialized Chatbot

Poc II: Multimodal RAG Chatbot

Overview

A multimodal RAG (Retrieval-Augmented Generation) chatbot that processes text queries and returns responses combining text and images. The system retrieves semantically relevant visual content from a multimodal vector store built from a custom dataset of 3D footwear images. Image and text embeddings are generated using OpenCLIP and Cohere, while multimodal reasoning is performed using LLaVA 1.6 and Llama 3.2 Vision. The application is implemented with LlamaIndex and a Gradio-based web interface.

Demo

Features

• Multimodal chatbot: text input with text + image output
• Multimodal RAG architecture with unified text–image embeddings
• Semantic retrieval of images from text queries
• MultiModalVectorStoreIndex for multimodal vector storage
• Image embeddings using OpenCLIP
• Text embeddings using Cohere embed-multilingual-v3.0
• Large Multimodal Models: LLaVA 1.6 and Llama 3.2 Vision
• Interactive web UI built with Gradio

Architecture

High-level structure of the application:

Sequence Diagram

Sequence Flow:

1. User submits a query (text) via the Frontend.
2. Frontend sends the query to the Backend.
3. Backend retrieves the most relevant documents and images from the Multimodal Vector Store using semantic search (Cohere embeddings for text + CLIP embeddings for images).
4. Backend sends the query along with the retrieved context to the Ollama Multimodal LLM (llava:13b or llama3.2-vision:11b).
5. LLM generates a multimodal response (text and/or image) based on the provided context.
6. Backend returns the generated response to the Frontend.
7. Frontend displays the generated response (text and image) to the user.

Tech Stack

Backend

Multimodal / Embeddings

Vector Database

LLM

Frontend

Infrastructure | Dev Tools

Setup

# 1. Install Poetry (required) - https://python-poetry.org/docs/#installing-with-the-official-installer
# Windows (PowerShell):
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
poetry --version

# 2. Clone the repository
git clone https://github.com/estelacode/poc_II_tfg_multimodal_rag_chatbot.git
cd poc_II_tfg_multimodal_rag_chatbot

# 3. Select the Python interpreter for this project
poetry env use /full/path/to/python
# Example for Windows:
# poetry env use C:\Users\emada\AppData\Local\Programs\Python\Python312


# 4. Install dependencies
poetry install

# Check where the virtual environment is located (optional)
poetry env info

Usage

# Run the project
poetry run python src/poc_multimod_chatbot_chroma/main.py

Project Structure

poc_II_tfg_multimodal_rag_chatbot/
├── data/                     # Data files
├── datasets_chatbot/         # Chatbot datasets
├── demo_interface/           # Demo interface or media
├── notebooks/                # Jupyter notebooks for experiments
├── src/                      # Source code 
├── tests/                    # Test images used to evaluate multimodal capabilities
├── vectorstore/              # Vector database storage 
├── .env                      # Environment variables (keep secret)
├── .gitignore                # Git ignore rules
├── poetry.lock               # Poetry dependency lock file
├── pyproject.toml            # Poetry configuration and dependencies
└── README.md                 # Project README file

Roadmap

• Support multimodal queries (text + image input)
• Expand dataset with more product images and descriptions
• Improve retrieval accuracy with alternative multimodal embeddings
• Implement reranking techniques for image retrieval
• Add structured metadata extraction (brand, color, materials)
• Apply prompt engineering techniques to improve response quality
• Introduce evaluation methods and metrics for multimodal retrieval and generation
• Replace in-memory vector store with scalable vector databases
• Deploy the application as a cloud service

References

• Poetry Documentation
• LlamaIndex
• LlamaIndex RAG
• Cohere
• Cohere Docs
• Cohere LLM University
• Ollama
• Gradio

👋 Author

Estela Madariaga