Extension/Plugin System Design

[hughlv]

Overview

Design and implement an extension/plugin system to enable:

• AI models (auto-annotation, segmentation, classification)
• Custom drawing tools
• Batch processors and validators
• Custom export formats

Design Goals

1. Type Safety: Full TypeScript support with strong typing
2. Framework Agnostic: Core extension API independent of React/OSD
3. Easy Integration: Simple registration and lifecycle management
4. Performance: Minimal overhead, async operation support
5. Extensibility: Support for future capabilities

Proposed Extension API

interface AnnotaExtension {
  id: string;
  name: string;
  version: string;

  // Lifecycle hooks
  onInit?(annotator: Annotator): void | Promise<void>;
  onDestroy?(): void | Promise<void>;

  // Tool extensions
  tools?: Tool[];

  // AI capabilities
  detect?(image: ImageData, options: any): Promise<Annotation[]>;
  classify?(annotation: Annotation): Promise<Classification>;
  segment?(point: Point, image: ImageData): Promise<Polygon>;
}

Example Extension

const openCVExtension: AnnotaExtension = {
  id: 'opencv-edge-detector',
  name: 'OpenCV Edge Detector',
  version: '1.0.0',

  async onInit(annotator) {
    await initOpenCV();
    console.log('OpenCV extension loaded');
  },

  async segment(point, imageData) {
    const result = await detectEdge(imageData, point, {
      threshold: 128,
      thresholdType: 'otsu',
    });

    return result.polygon;
  },
};

// Register extension
annotator.registerExtension(openCVExtension);

Use Cases

1. OpenCV Integration: Edge detection, thresholding, morphological operations
2. SAM Integration: Segment Anything Model for one-click segmentation
3. Custom Tools: Domain-specific annotation tools
4. Export Formats: Custom serialization (GeoJSON, XML, proprietary formats)
5. Validation: Quality control and annotation validation workflows

Open Questions

1. How should extensions access the viewer/canvas for rendering?
2. Should extensions be able to extend the UI (add toolbar buttons, panels)?
3. How to handle extension dependencies and conflicts?
4. Should we support remote extensions (loaded from URLs)?
5. How to manage extension state and persistence?
6. Security considerations for third-party extensions?

Related

• Current AI integration: Contour tool uses OpenCV.js directly
• See src/tools/contour.ts for existing pattern

Discussion

Please share your thoughts on:

• API design and developer experience
• Use cases and requirements
• Implementation approach
• Security and sandboxing needs