Skip to content

TECHNICAL.md

Latest commit

 

History

History
284 lines (226 loc) · 7.86 KB

TECHNICAL.md

File metadata and controls

284 lines (226 loc) · 7.86 KB
Raw

Technical Documentation

This document provides a detailed technical explanation of how the Obsidian Vault Query Tool works, its architecture, and the concepts behind it.

Architecture Overview

The tool is built using a modular architecture with clear separation of concerns:

  1. 🔵 Core Components (src/veruca/core/)

    • Base classes and interfaces
    • Common utilities
    • Embedding functionality

  2. 🟢 Data Sources (src/veruca/sources/)

    • Obsidian integration
    • Extensible for other sources

  3. 🟡 Document Processing (src/veruca/sources/obsidian/parser.py)

  4. 🟣 Query Handling (src/veruca/sources/obsidian/query.py)

  5. 🔴 Vector Storage and Search (src/veruca/core/embeddings.py)

Here's how these components work together:

graph TD
    A[Markdown Files] --> B[Document Processing]
    B --> C[Text Chunking]
    C --> D[Embedding Generation]
    D --> E[Vector Storage]
    F[User Query] --> G[Query Processing]
    G --> H[Vector Search]
    H --> I[Response Generation]
    E --> H

    style A fill:#2196f3,stroke:#333,stroke-width:2px,color:#fff
    style B fill:#ffeb3b,stroke:#333,stroke-width:2px,color:#000
    style C fill:#ffeb3b,stroke:#333,stroke-width:2px,color:#000
    style D fill:#f44336,stroke:#333,stroke-width:2px,color:#fff
    style E fill:#f44336,stroke:#333,stroke-width:2px,color:#fff
    style F fill:#9c27b0,stroke:#333,stroke-width:2px,color:#fff
    style G fill:#9c27b0,stroke:#333,stroke-width:2px,color:#fff
    style H fill:#f44336,stroke:#333,stroke-width:2px,color:#fff
    style I fill:#9c27b0,stroke:#333,stroke-width:2px,color:#fff
Loading

Component Details

1. Core Components

1.1 Base Classes (core/base.py)

class DataSource(ABC):
    """Abstract base class for data sources."""
  • Defines interface for data sources
  • Ensures consistent behavior
  • Enables easy extension

1.2 Embedding Utilities (core/embeddings.py)

class EmbeddingGenerator:
    """Handles embedding generation and vector operations."""
  • Manages embedding models
  • Handles vector operations
  • Provides common functionality

2. Document Processing Pipeline

2.1 File Loading (sources/obsidian/parser.py:load_markdown_files)

def load_markdown_files(vault_path: str) -> List[Tuple[str, Dict[str, Any], str]]:
    """Load all Markdown files from the given vault path with Obsidian-specific processing."""
  • Recursively scans the Obsidian vault for .md files
  • Handles file encoding and error cases
  • Preserves file structure and relationships

2.2 Obsidian Feature Processing

The tool processes several Obsidian-specific features in separate functions:

Frontmatter Parsing (sources/obsidian/parser.py:parse_frontmatter)
def parse_frontmatter(content: str) -> Tuple[Dict[str, Any], str]:
    """Parse Obsidian's YAML frontmatter if present."""
  • Uses regex to identify YAML frontmatter
  • Parses using PyYAML library
  • Stores as metadata for each document
Tag Extraction (sources/obsidian/parser.py:extract_tags)
def extract_tags(content: str) -> List[str]:
    """Extract Obsidian tags from content."""
  • Uses regex #(\w+) to find tags
  • Stores as metadata for filtering and organization
Internal Link Processing (sources/obsidian/parser.py:process_obsidian_links)
def process_obsidian_links(content: str, vault_path: str) -> str:
    """Process Obsidian's internal links and convert them to readable text."""
  • Converts to readable text
  • Preserves display text when available
  • Maintains document relationships
Callout Processing (sources/obsidian/parser.py:process_callouts)
def process_callouts(content: str) -> str:
    """Process Obsidian's callouts (admonitions) to make them more readable."""
  • Converts to readable format
  • Preserves callout type and content

3. Text Chunking

The tool uses LangChain's RecursiveCharacterTextSplitter in index_vault():

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # Number of characters per chunk
    chunk_overlap=50     # Number of characters to overlap between chunks
)

Benefits:

  • Maintains context across chunk boundaries
  • Optimizes for embedding model input size
  • Preserves document structure

4. Embedding Generation

Uses Ollama's nomic-embed-text model in both indexing and querying:

embedding_model = OllamaEmbeddings(model="nomic-embed-text")

Key features:

  • Local processing (no API calls)
  • High-quality embeddings
  • Fast processing

5. Vector Storage

Uses ChromaDB for vector storage and retrieval in index_vault():

vector_store = Chroma.from_documents(
    documents,
    embedding=embedding_model,
    persist_directory=CHROMA_DB_PATH
)

Features:

  • Persistent storage
  • Efficient similarity search
  • Metadata filtering

6. Query Processing

6.1 Query Flow (sources/obsidian/query.py:query_vault)

def query_vault(question: str, filter_tags: List[str] = None) -> None:
    """Query the indexed data with a question and optional tag filtering."""
  1. User submits query
  2. Query is embedded using same model
  3. Similarity search performed
  4. Relevant documents retrieved
  5. Response generated

6.2 Tag Filtering

if filter_tags:
    search_kwargs["filter"] = {"tags": {"$in": filter_tags}}
  • Filters results by specified tags
  • Uses ChromaDB's metadata filtering
  • Supports multiple tags

6.3 Similarity Search

retriever = vector_store.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={
        "k": 4,                    # Number of results
        "score_threshold": 0.5     # Minimum similarity score
    }
)
  • Returns top 4 most relevant chunks
  • Filters by similarity threshold
  • Balances relevance and coverage

7. Response Generation

Uses Ollama's Mistral model with a custom prompt defined in sources/obsidian/query.py:

CUSTOM_PROMPT = """You are a helpful assistant that answers questions based on the provided context from an Obsidian vault.
The context comes from various notes, and each piece of information includes metadata about its source.

Context information:
{context}

Question: {question}

Please provide a detailed answer based on the context. If the information comes from specific notes, mention them by name.
If you're not sure about something, say so. Don't make up information that isn't in the context.

Answer:"""

Features:

  • Context-aware responses
  • Source attribution
  • Confidence indication

Technical Considerations

1. Performance

Memory Usage

  • Chunk size (500 chars) balances:
    • Context preservation
    • Memory efficiency
    • Processing speed

Storage

  • ChromaDB provides efficient vector storage
  • Embeddings are persisted for reuse
  • Metadata enables fast filtering

2. Scalability

The tool can handle:

  • Large vaults (thousands of files)
  • Deep directory structures
  • Various file sizes

Limitations:

  • Memory constraints for very large files
  • Processing time for initial indexing
  • Storage requirements for embeddings

3. Security

Features:

  • Local processing only
  • No external API calls
  • No data transmission

4. Extensibility

The architecture supports:

  • Adding new data sources through the DataSource interface
  • Different embedding models
  • Alternative vector stores
  • Custom response formats

Future Improvements

  1. Performance

    • Batch processing for large vaults
    • Incremental updates
    • Caching mechanisms

  2. Features

    • More Obsidian features (tables, code blocks)
    • Advanced filtering options
    • Query history and favorites

  3. User Experience

    • Interactive query interface
    • Result visualization
    • Query suggestions

  4. Integration

    • Obsidian plugin
    • API interface
    • Web interface

License

This project is licensed under the MIT License - see the LICENSE file for details.