Skip to content

fix(macos): use CGWarpMouseCursorPosition for mouse control in VMs #1163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
robotlearning123 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(macos): use CGWarpMouseCursorPosition for mouse control in VMs #1163

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions libs/python/computer-server/computer_server/handlers/macos.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,32 @@
logger.debug(f"Failed to trigger screenshot permissions prompt: {e}")


# Fix: pynput's MouseController.position setter uses CGEventPost with
# kCGEventMouseMoved internally, which silently fails in macOS VMs running
# under Apple's Virtualization.framework (Tart, Lume, or any VZVirtualMachine-
# based hypervisor). The cursor doesn't move but no error is raised.
#
# CGWarpMouseCursorPosition works reliably in both VM and bare-metal environments.
# This descriptor replaces pynput's default position property with a CGWarp-based
# implementation. See: https://github.com/trycua/cua/issues/1162
class _CGWarpPosition:
"""Descriptor that uses CGWarpMouseCursorPosition instead of pynput's default."""

def __get__(self, obj, objtype=None):
if obj is None:
return self
pos = CGEventGetLocation(CGEventCreate(None))
return (int(pos.x), int(pos.y))

def __set__(self, obj, value):
x, y = value
CGWarpMouseCursorPosition((float(x), float(y)))
CGAssociateMouseAndMouseCursorPosition(True)
Comment on lines +102 to +103
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

What do Quartz CGWarpMouseCursorPositionandCGAssociateMouseAndMouseCursorPositionreturn on macOS, and iskCGErrorSuccess the expected success value?

💡 Result:

On macOS, both Quartz functions return a CGError status code:

  • CGWarpMouseCursorPosition(CGPoint) → returns a “result code” (CGError). On success it returns kCGErrorSuccess. [1]
  • CGAssociateMouseAndMouseCursorPosition(boolean_t connected) → returns a “result code” (CGError). On success it returns kCGErrorSuccess. [2]

Yes—kCGErrorSuccess is the expected success value, and it is defined as 0 in the Quartz Display Services result codes. [3]

Sources
[1] Quartz Display Services Reference — CGWarpMouseCursorPosition return value section (leopard-adc.pepas.com)
[2] Quartz Display Services Reference — CGAssociateMouseAndMouseCursorPosition return value section (leopard-adc.pepas.com)
[3] Quartz Display Services Reference — Result codes table (kCGErrorSuccess = 0) (leopard-adc.pepas.com)

Citations:

🏁 Script executed:

head -n 110 libs/python/computer-server/computer_server/handlers/macos.py | tail -n 20

Repository: trycua/cua

Length of output: 616

🏁 Script executed:

rg -n "def (move_cursor|left_click|mouse_down)" libs/python/computer-server/computer_server/handlers/macos.py

Repository: trycua/cua

Length of output: 261

🏁 Script executed:

sed -n '1137,1150p' libs/python/computer-server/computer_server/handlers/macos.py

Repository: trycua/cua

Length of output: 497

🏁 Script executed:

sed -n '1069,1085p' libs/python/computer-server/computer_server/handlers/macos.py

Repository: trycua/cua

Length of output: 757

🏁 Script executed:

sed -n '1019,1040p' libs/python/computer-server/computer_server/handlers/macos.py

Repository: trycua/cua

Length of output: 920

Check the CoreGraphics return codes instead of assuming success.

Both CGWarpMouseCursorPosition and CGAssociateMouseAndMouseCursorPosition return CGError; they do not raise on failure. Currently, a rejected warp lets move_cursor, left_click, and mouse_down return {"success": True}, silently masking failures to the API caller.

Possible fix 
     def __set__(self, obj, value):
         x, y = value
-        CGWarpMouseCursorPosition((float(x), float(y)))
-        CGAssociateMouseAndMouseCursorPosition(True)
+        err = CGWarpMouseCursorPosition((float(x), float(y)))
+        if err != kCGErrorSuccess:
+            raise RuntimeError(f"CGWarpMouseCursorPosition failed: {err}")
+
+        err = CGAssociateMouseAndMouseCursorPosition(True)
+        if err != kCGErrorSuccess:
+            raise RuntimeError(f"CGAssociateMouseAndMouseCursorPosition failed: {err}")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
CGWarpMouseCursorPosition((float(x), float(y)))
CGAssociateMouseAndMouseCursorPosition(True)
err = CGWarpMouseCursorPosition((float(x), float(y)))
if err != kCGErrorSuccess:
raise RuntimeError(f"CGWarpMouseCursorPosition failed: {err}")
err = CGAssociateMouseAndMouseCursorPosition(True)
if err != kCGErrorSuccess:
raise RuntimeError(f"CGAssociateMouseAndMouseCursorPosition failed: {err}")
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@libs/python/computer-server/computer_server/handlers/macos.py` around lines
102 - 103, The code currently calls CGWarpMouseCursorPosition and
CGAssociateMouseAndMouseCursorPosition without checking their CGError return
values; update the handlers (move_cursor, left_click, mouse_down) to capture
each function's return, verify it equals kCGErrorSuccess (or zero), and if not
log/return a failure payload (e.g., {"success": False, "error": "<CGError code
or message>"}); ensure you short-circuit subsequent calls when a warp/associate
fails and include enough context (function name and error code) in the
logged/returned error.



MouseController.position = _CGWarpPosition()
Comment on lines +100 to +106
Copy link
Contributor

@coderabbitai coderabbitai bot Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🌐 Web query:

Does Quartz CGWarpMouseCursorPosition generate mouse moved or mouse dragged events on macOS, and what API is recommended to synthesize drag motion while a mouse button is held?

💡 Result:

No. CGWarpMouseCursorPosition moves the cursor but does not generate a mouse event, so it won’t produce either “mouse moved” or “mouse dragged” events (neither NSMouseMoved nor NSLeftMouseDragged/NSRightMouseDragged, etc.). [1]

To synthesize drag motion while a button is held, Apple’s recommended approach is to post Quartz CGEvents: create mouse events with CGEventCreateMouseEvent and post them with CGEventPost, using a dragged event type such as kCGEventLeftMouseDragged (described as “a mouse drag event with the left button down”). [2][3]

🏁 Script executed:

cat -n libs/python/computer-server/computer_server/handlers/macos.py | sed -n '80,110p'

Repository: trycua/cua

Length of output: 1434

🏁 Script executed:

cat -n libs/python/computer-server/computer_server/handlers/macos.py | sed -n '1150,1230p'

Repository: trycua/cua

Length of output: 3603

🏁 Script executed:

rg "CGWarpMouseCursorPosition|CGAssociateMouseAndMouseCursorPosition" libs/python/computer-server/computer_server/handlers/macos.py -A 2 -B 2

Repository: trycua/cua

Length of output: 676

Synthesize drag motion events instead of warping cursor during drag operations.

drag_to() and drag() press a button, then repeatedly set self.mouse.position while holding it down (lines 1182, 1220). The replacement CGWarpMouseCursorPosition warps the cursor without generating any mouse events—target applications won't receive kCGEventLeftMouseDragged or equivalent events, so drag operations will fail silently. Use CGEventCreateMouseEvent with kCGEventLeftMouseDragged (or appropriate button type) and post it with CGEventPost to synthesize drag motion while a button is held.

Additionally, CGWarpMouseCursorPosition and CGAssociateMouseAndMouseCursorPosition (lines 102–103) return error codes that are never checked, reintroducing the silent-failure behavior the PR aimed to fix.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@libs/python/computer-server/computer_server/handlers/macos.py` around lines
100 - 106, Replace the cursor warp in _CGWarpPosition.__set__ (currently calling
CGWarpMouseCursorPosition and CGAssociateMouseAndMouseCursorPosition) with
synthesized drag motion events: detect when a mouse button is held (used by
MouseController.position during drag_to/drag), create a CGEvent via
CGEventCreateMouseEvent with the appropriate event type (e.g.,
kCGEventLeftMouseDragged or the button-specific dragged event) for each position
update, and post it with CGEventPost so target apps receive kCGEvent*Dragged
events; if you still call
CGWarpMouseCursorPosition/CGAssociateMouseAndMouseCursorPosition, check and
handle their return codes and only fall back to warping when event synthesis
fails.



# Constants for accessibility API
kAXErrorSuccess = 0
kAXRoleAttribute = "AXRole"
Expand Down