U.AI – Unified AI Taskforce

Video Captioning + Semantic Search Agent (RAG + Multimodal AI POC)

by: Abhishek Kevin Gomes

---

Overview

U.AI (Unified AI Taskforce) is a multimodal AI system designed to unify information retrieval, command execution, and patch management into a single intelligent interface.

It demonstrates how multiple agents can be orchestrated together to handle:

• Video → Knowledge pipeline (transcription + semantic search)
• Audio & PDF semantic chat agent (RAG for documents)
• Voice-driven command execution (safe demo environment)
• Patch management assistant (mock showcase)
• Unified multimodal interface (voice, text, video, audio, image)

---

Tech Stack

---

Features

1. Video Captioning + Semantic Search

• Extract audio with FFmpeg
• Transcribe via OpenAI Whisper
• Embed with HuggingFace MiniLM
• Store in ChromaDB for semantic Q&A

2. PDF & Audio Chat Agent

• Parse PDFs with PyPDF
• Transcribe audio → text
• Store embeddings in ChromaDB
• Provide contextual answers with GPT

3. Voice → Command Execution

• Record or type queries
• Interpret with GPT
• Execute with mock safe outputs

4. Patch Management Agent

• AI-powered patch recommendations
• System diagnostics + remediation demo

---

Architecture

         ┌──────────────┐
         │   Frontend   │
         │ (Web + Voice)│
         └──────┬───────┘
                │
      ┌─────────▼──────────┐
      │   Backend API      │
      │   (FastAPI)        │
      └─────────┬──────────┘
                │ 
  ┌─────────────┴─────────────┐
  │         AI Agents         │
  │ • RAG Search Agent        │
  │ • Executor Agent          │
  │ • Patch Specialist Agent  │
  │ • Multimodal Orchestrator │
  └─────────────┬─────────────┘
                │
    ┌───────────▼───────────┐
    │   Knowledge Stores    │
    │ (ChromaDB, PDFs,      │
    │  Audio, Video)        │
    └───────────────────────┘

---

Real-World Use Cases

• Enterprise IT Operations – automate patching + system checks
• Knowledge Management – searchable transcripts of training/videos
• DevOps & SecOps – AI-driven patch/security suggestions
• Content Accessibility – captions + semantic search
• Unified AI Assistant – multimodal collaboration for teams

---

Screenshots

---

Setup

1. Clone Repository

git clone https://github.com/your-username/u.ai.git
cd u.ai

2. Install Dependencies

pip install -r requirements.txt

3. Configure Environment

Create a .env file:

OPENAI_API_KEY=your-key

4. Run Backend

python app.py

---

Notes

• Demo project only — executor + patching agents are mock/safe
• API keys are excluded (load them via .env)