generate_text_embeddings streaming

[dayesouza]

Streaming Performance Improvement (with .txt):

run time: -16.46%
peak memory: -55.57%
memory delta: -81.58%

This pull request introduces a major refactor of the text embedding workflow to enable streaming, table-based processing and improve modularity and maintainability. The changes replace the previous DataFrame-based approach with a streaming Table API, introduce per-field embedding configuration, and simplify the workflow logic. Additionally, the embedding operation is now more efficient and robust, supporting output snapshots and row-level transformation.

Key changes include:

Streaming Embedding Workflow and Table API

• Refactored the generate_text_embeddings workflow to operate on streaming Table objects instead of Pandas DataFrames, allowing for more scalable and memory-efficient processing. The workflow now iterates over configured embedding fields, applies optional row transformations, and writes embeddings to vector stores and optional output tables. [1] [2]
• Updated the embed_text operation to accept a streaming Table as input, process rows in batches, and flush results to the vector store and optional output tables, improving efficiency and supporting large datasets. [1] [2] [3] [4]

Embedding Field Configuration and Row Transformation

• Introduced the EmbeddingFieldConfig dataclass and the EMBEDDING_FIELDS mapping, enabling per-field configuration of source tables, columns, and optional row transformers for embedding generation.
• Added a new row transformer function, transform_entity_row_for_embedding, to concatenate entity title and description for embedding.

Workflow and API Simplification

• Simplified the API for both generate_text_embeddings and update_text_embeddings workflows, removing DataFrame manipulation and centralizing configuration and iteration logic. The update workflow now delegates directly to the streaming embedding generator. [1] [2]
• Updated example notebooks and documentation to reflect the new streaming API and simplified usage patterns.

Licensing and Metadata

• Updated copyright statements to reflect the year 2026 and added license headers to new and modified files. [1] [2] [3]
• Added a semversioner patch release note for the new streaming embedding functionality.