Add distributed data loader project and core interfaces

[robreeves]

Summary

This is the initial commit for a Python data loader library for distributed loading of OpenHouse tables. This PR establishes the project structure, core interfaces, and CI integration.

Key Components

• OpenHouseDataLoader - Main API that creates distributable splits for parallel table loading
• TableIdentifier - Identifies tables by database, name, and optional branch
• DataLoaderSplits / DataLoaderSplit - Iterable splits that can be distributed across workers
• TableTransformer / UDFRegistry - Extension points for table transformations and UDFs

Project Setup

• Python 3.12+ with uv for dependency management
• Ruff for linting and formatting
• Makefile with sync, check, test, all targets
• Integrated into build-run-tests.yml CI workflow

Not included

• Publishing the new python package to pypi. That will happen in a later PR.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

I tested by running make -C integrations/python/dataloader all. This PR is project setup and interfaces so no new functionality needs to be tested in this PR.

uv run ruff check src/ tests/
All checks passed!
uv run ruff format --check src/ tests/
10 files already formatted
uv run pytest
============================================================================ test session starts ============================================================================
platform darwin -- Python 3.14.0, pytest-9.0.2, pluggy-1.6.0
rootdir: /Users/roreeves/li/openhouse_oss/integrations/python/dataloader
configfile: pyproject.toml
collected 1 item                                                                                                                                                            

tests/test_data_loader.py .                                                                                                                                           [100%]

============================================================================= 1 passed in 0.01s =============================================================================

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[sumedhsakdeo]

Thanks Rob. Nice PR! This is really coming together. I left some comments please let me know what you think.

[cbb330]

I can't tell if the interfaces are under-specified or good but lacking implementation details. so i suspect that there are more attributes/methods that could be added to these interfaces.

for example, these specific implementations are missing and hard to derive from the interfaces:

1. loading a table from the catalog
2. applying transform and using read splits

I think it would be helpful to have a concrete implementation of pyiceberg

e.g.

# Minimal viable implementation to validate the design
  class PyIcebergDataLoader(OpenHouseDataLoader):
      def create_splits(self, table, columns, context):
          # 1. Load table from PyIceberg catalog
          # 2. Get FileScanTasks via table.scan()
          # 3. Apply transformer to get LogicalPlan
          # 4. Return DataLoaderSplits
          ...

  class PyIcebergSplit(DataLoaderSplit):
      def __call__(self):
          # 1. Create SessionContext
          # 2. Register UDFs
          # 3. Convert FileScanTask → RecordBatchReader
          # 4. Register as table
          # 5. Execute plan, yield batches
          ...

perhaps in tests or a poc implementation module

[robreeves]

I can't tell if the interfaces are under-specified or good but lacking implementation details. so i suspect that there are more attributes/methods that could be added to these interfaces.

for example, these specific implementations are missing and hard to derive from the interfaces:

1. loading a table from the catalog
2. applying transform and using read splits

I think it would be helpful to have a concrete implementation of pyiceberg

e.g.

# Minimal viable implementation to validate the design
  class PyIcebergDataLoader(OpenHouseDataLoader):
      def create_splits(self, table, columns, context):
          # 1. Load table from PyIceberg catalog
          # 2. Get FileScanTasks via table.scan()
          # 3. Apply transformer to get LogicalPlan
          # 4. Return DataLoaderSplits
          ...

  class PyIcebergSplit(DataLoaderSplit):
      def __call__(self):
          # 1. Create SessionContext
          # 2. Register UDFs
          # 3. Convert FileScanTask → RecordBatchReader
          # 4. Register as table
          # 5. Execute plan, yield batches
          ...

perhaps in tests or a poc implementation module

The goal of this is to define the public interfaces, not all internal classes. To me that falls into the implementation category. I disagree that we should include everything including an MVP implementation in a single PR. The PR already touches 17 files and has 50+ comments. Adding implementation will balloon it even more. We can iterate as needed in future PRs. Loading the table is an implementation detail the consumer should not be aware of in most cases.

[sumedhsakdeo]

Good to go from my standpoint. I expect we might iterate on internal integration points as we develop more. But ok with crossing that bridge in follow up PRs.

Public APIs look great.

[ShreyeshArangath]

LGTM, thanks for working on this!

[ShreyeshArangath]

One small discussion topic: should we bump the version to 0.6?