Skip to content

Windows: explore InputInjector / sparse-package capability for vision-mode UWP coverage #1552

Description

@f-trycua

Context

PR #1551 ships a layered vision-mode (x, y) click chain on Windows that mirrors trope-cua:

  1. UIA Invoke in target HWND's subtree — covers UWP / WebView2 / DirectComposition surfaces. No focus steal, occlusion-independent. Works for every app that exposes an accessible UIA tree.
  2. PostMessage(WM_LBUTTONDOWN/UP) fallback — covers plain Win32 controls without InvokePattern and apps without a UIA tree. Z-order independent, no focus steal.

The remaining gap = apps that have no useful UIA tree AND don't read WM_LBUTTONDOWN from the HWND queue:

  • DirectX / OpenGL / Vulkan-rendered surfaces (games, custom renderers)
  • Some heavily custom-drawn Electron / WPF / WinUI windows that suppress accessibility
  • UWP apps that mark sub-elements with AccessibilityView=Raw or similar

For those, the current chain falls all the way through and the click no-ops. macOS gets a clean answer via CGEventPostToPSN (per-pid event routing). Windows has no direct equivalent — see the matrix in #1551's discussion thread.

Proposal

Spike Windows.UI.Input.Preview.Injection.InputInjector as a third tier (opt-in, behind a click arg or config flag — not silent fallback).

What it provides: coordinate-based pointer injection that UWP / WinUI / DirectX surfaces accept because it goes through the same input pipeline real hardware uses. Reaches surfaces that UIA and PostMessage both miss.

Why we can't just turn it on:

  1. InputInjector::TryCreate() returns null for unpackaged Win32 binaries. The caller must declare the inputInjectionBrokered restricted capability in an app manifest. Today cua-driver.exe is plain Win32 with no manifest — we'd need either:
    • Full UWP packaging (.msix), or
    • A sparse package that grafts a Package.appxmanifest onto the existing exe via the Sparse Signed Packages flow.
  2. Even when it works, InputInjector still routes through the global input pipeline: z-order matters, target window activates, focus steal happens.
  3. Restricted capability means: signing requirements, store-style review if we go full MSIX, or manual provisioning per-machine for sparse packages.

Win32 cousin InitializeTouchInjection / InjectTouchInput doesn't need a manifest — but has the same global-pipeline / focus-steal characteristics.

Open questions for the spike

  • Can we ship cua-driver as a sparse-packaged binary without breaking the current irm | iex install UX?
  • Does sparse-package + inputInjectionBrokered actually let InputInjector.TryCreate() succeed at runtime?
  • Is the focus-steal acceptable as a 3rd-tier opt-in (e.g. click({pid, x, y, allow_transient_foreground: true}))?
  • Compare with InjectTouchInput: simpler (no manifest) but synthesizes touch — does target UWP/game handle touch identically to mouse?
  • What's the user-visible cost (signing prompts, SmartScreen warnings) of sparse packaging?

Acceptance criteria (for closing this issue)

Either:

  • (a) A decision doc / ADR explaining why the niche isn't worth the packaging cost, or
  • (b) A merged PR adding an opt-in 3rd tier behind a click arg, with the packaging mechanics solved.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions