fix(macos): use CGWarpMouseCursorPosition for mouse control in VMs#1163
fix(macos): use CGWarpMouseCursorPosition for mouse control in VMs#1163robotlearning123 wants to merge 1 commit intotrycua:mainfrom
Conversation
…in VMs pynput's MouseController.position setter uses CGEventPost with kCGEventMouseMoved internally, which silently fails in macOS VMs running under Apple's Virtualization.framework (Tart, Lume). The cursor doesn't move but no error is raised, causing clicks to land at wrong coordinates. Replace with a descriptor that uses CGWarpMouseCursorPosition, which works reliably in both VM and bare-metal environments. All existing mouse methods (left_click, right_click, double_click, move_cursor, drag_to, drag_path) benefit automatically with zero code changes. Fixes trycua#1162
|
@robotlearning123 is attempting to deploy a commit to the Cua Team on Vercel. A member of the Team first needs to authorize it. |
📝 WalkthroughWalkthroughA new descriptor class Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip CodeRabbit can generate a title for your PR based on the changes with custom instructions.Set the |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@libs/python/computer-server/computer_server/handlers/macos.py`:
- Around line 100-106: Replace the cursor warp in _CGWarpPosition.__set__
(currently calling CGWarpMouseCursorPosition and
CGAssociateMouseAndMouseCursorPosition) with synthesized drag motion events:
detect when a mouse button is held (used by MouseController.position during
drag_to/drag), create a CGEvent via CGEventCreateMouseEvent with the appropriate
event type (e.g., kCGEventLeftMouseDragged or the button-specific dragged event)
for each position update, and post it with CGEventPost so target apps receive
kCGEvent*Dragged events; if you still call
CGWarpMouseCursorPosition/CGAssociateMouseAndMouseCursorPosition, check and
handle their return codes and only fall back to warping when event synthesis
fails.
- Around line 102-103: The code currently calls CGWarpMouseCursorPosition and
CGAssociateMouseAndMouseCursorPosition without checking their CGError return
values; update the handlers (move_cursor, left_click, mouse_down) to capture
each function's return, verify it equals kCGErrorSuccess (or zero), and if not
log/return a failure payload (e.g., {"success": False, "error": "<CGError code
or message>"}); ensure you short-circuit subsequent calls when a warp/associate
fails and include enough context (function name and error code) in the
logged/returned error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: dc806c4f-1332-48a6-a17c-d831431cedcf
📒 Files selected for processing (1)
libs/python/computer-server/computer_server/handlers/macos.py
| def __set__(self, obj, value): | ||
| x, y = value | ||
| CGWarpMouseCursorPosition((float(x), float(y))) | ||
| CGAssociateMouseAndMouseCursorPosition(True) | ||
|
|
||
|
|
||
| MouseController.position = _CGWarpPosition() |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Does Quartz CGWarpMouseCursorPosition generate mouse moved or mouse dragged events on macOS, and what API is recommended to synthesize drag motion while a mouse button is held?
💡 Result:
No. CGWarpMouseCursorPosition moves the cursor but does not generate a mouse event, so it won’t produce either “mouse moved” or “mouse dragged” events (neither NSMouseMoved nor NSLeftMouseDragged/NSRightMouseDragged, etc.). [1]
To synthesize drag motion while a button is held, Apple’s recommended approach is to post Quartz CGEvents: create mouse events with CGEventCreateMouseEvent and post them with CGEventPost, using a dragged event type such as kCGEventLeftMouseDragged (described as “a mouse drag event with the left button down”). [2][3]
🏁 Script executed:
cat -n libs/python/computer-server/computer_server/handlers/macos.py | sed -n '80,110p'Repository: trycua/cua
Length of output: 1434
🏁 Script executed:
cat -n libs/python/computer-server/computer_server/handlers/macos.py | sed -n '1150,1230p'Repository: trycua/cua
Length of output: 3603
🏁 Script executed:
rg "CGWarpMouseCursorPosition|CGAssociateMouseAndMouseCursorPosition" libs/python/computer-server/computer_server/handlers/macos.py -A 2 -B 2Repository: trycua/cua
Length of output: 676
Synthesize drag motion events instead of warping cursor during drag operations.
drag_to() and drag() press a button, then repeatedly set self.mouse.position while holding it down (lines 1182, 1220). The replacement CGWarpMouseCursorPosition warps the cursor without generating any mouse events—target applications won't receive kCGEventLeftMouseDragged or equivalent events, so drag operations will fail silently. Use CGEventCreateMouseEvent with kCGEventLeftMouseDragged (or appropriate button type) and post it with CGEventPost to synthesize drag motion while a button is held.
Additionally, CGWarpMouseCursorPosition and CGAssociateMouseAndMouseCursorPosition (lines 102–103) return error codes that are never checked, reintroducing the silent-failure behavior the PR aimed to fix.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@libs/python/computer-server/computer_server/handlers/macos.py` around lines
100 - 106, Replace the cursor warp in _CGWarpPosition.__set__ (currently calling
CGWarpMouseCursorPosition and CGAssociateMouseAndMouseCursorPosition) with
synthesized drag motion events: detect when a mouse button is held (used by
MouseController.position during drag_to/drag), create a CGEvent via
CGEventCreateMouseEvent with the appropriate event type (e.g.,
kCGEventLeftMouseDragged or the button-specific dragged event) for each position
update, and post it with CGEventPost so target apps receive kCGEvent*Dragged
events; if you still call
CGWarpMouseCursorPosition/CGAssociateMouseAndMouseCursorPosition, check and
handle their return codes and only fall back to warping when event synthesis
fails.
| CGWarpMouseCursorPosition((float(x), float(y))) | ||
| CGAssociateMouseAndMouseCursorPosition(True) |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What do Quartz CGWarpMouseCursorPositionandCGAssociateMouseAndMouseCursorPositionreturn on macOS, and iskCGErrorSuccess the expected success value?
💡 Result:
On macOS, both Quartz functions return a CGError status code:
CGWarpMouseCursorPosition(CGPoint)→ returns a “result code” (CGError). On success it returnskCGErrorSuccess. [1]CGAssociateMouseAndMouseCursorPosition(boolean_t connected)→ returns a “result code” (CGError). On success it returnskCGErrorSuccess. [2]
Yes—kCGErrorSuccess is the expected success value, and it is defined as 0 in the Quartz Display Services result codes. [3]
Sources
[1] Quartz Display Services Reference — CGWarpMouseCursorPosition return value section (leopard-adc.pepas.com)
[2] Quartz Display Services Reference — CGAssociateMouseAndMouseCursorPosition return value section (leopard-adc.pepas.com)
[3] Quartz Display Services Reference — Result codes table (kCGErrorSuccess = 0) (leopard-adc.pepas.com)
Citations:
- 1: https://leopard-adc.pepas.com/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html
- 2: https://leopard-adc.pepas.com/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html
- 3: https://leopard-adc.pepas.com/documentation/GraphicsImaging/Reference/Quartz_Services_Ref/Reference/reference.html
🏁 Script executed:
head -n 110 libs/python/computer-server/computer_server/handlers/macos.py | tail -n 20Repository: trycua/cua
Length of output: 616
🏁 Script executed:
rg -n "def (move_cursor|left_click|mouse_down)" libs/python/computer-server/computer_server/handlers/macos.pyRepository: trycua/cua
Length of output: 261
🏁 Script executed:
sed -n '1137,1150p' libs/python/computer-server/computer_server/handlers/macos.pyRepository: trycua/cua
Length of output: 497
🏁 Script executed:
sed -n '1069,1085p' libs/python/computer-server/computer_server/handlers/macos.pyRepository: trycua/cua
Length of output: 757
🏁 Script executed:
sed -n '1019,1040p' libs/python/computer-server/computer_server/handlers/macos.pyRepository: trycua/cua
Length of output: 920
Check the CoreGraphics return codes instead of assuming success.
Both CGWarpMouseCursorPosition and CGAssociateMouseAndMouseCursorPosition return CGError; they do not raise on failure. Currently, a rejected warp lets move_cursor, left_click, and mouse_down return {"success": True}, silently masking failures to the API caller.
Possible fix
def __set__(self, obj, value):
x, y = value
- CGWarpMouseCursorPosition((float(x), float(y)))
- CGAssociateMouseAndMouseCursorPosition(True)
+ err = CGWarpMouseCursorPosition((float(x), float(y)))
+ if err != kCGErrorSuccess:
+ raise RuntimeError(f"CGWarpMouseCursorPosition failed: {err}")
+
+ err = CGAssociateMouseAndMouseCursorPosition(True)
+ if err != kCGErrorSuccess:
+ raise RuntimeError(f"CGAssociateMouseAndMouseCursorPosition failed: {err}")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| CGWarpMouseCursorPosition((float(x), float(y))) | |
| CGAssociateMouseAndMouseCursorPosition(True) | |
| err = CGWarpMouseCursorPosition((float(x), float(y))) | |
| if err != kCGErrorSuccess: | |
| raise RuntimeError(f"CGWarpMouseCursorPosition failed: {err}") | |
| err = CGAssociateMouseAndMouseCursorPosition(True) | |
| if err != kCGErrorSuccess: | |
| raise RuntimeError(f"CGAssociateMouseAndMouseCursorPosition failed: {err}") |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@libs/python/computer-server/computer_server/handlers/macos.py` around lines
102 - 103, The code currently calls CGWarpMouseCursorPosition and
CGAssociateMouseAndMouseCursorPosition without checking their CGError return
values; update the handlers (move_cursor, left_click, mouse_down) to capture
each function's return, verify it equals kCGErrorSuccess (or zero), and if not
log/return a failure payload (e.g., {"success": False, "error": "<CGError code
or message>"}); ensure you short-circuit subsequent calls when a warp/associate
fails and include enough context (function name and error code) in the
logged/returned error.
2 participants
Summary
MouseController.positionsetter with aCGWarpMouseCursorPosition-based descriptor that works reliably in macOS VMs under Apple's Virtualization.frameworkleft_click,right_click,double_click,move_cursor,drag_to,drag_path)Problem
pynput internally uses
CGEventPostwithkCGEventMouseMovedto set cursor position. This silently fails in macOS VMs running under Virtualization.framework (Tart, Lume, anyVZVirtualMachine-based hypervisor). The handler returns{"success": true}but clicks land at the wrong coordinates.CGWarpMouseCursorPositionCGDisplayMoveCursorToPointCGEventPost(kCGEventMouseMoved)mouse.position = (x, y)Fix
A Python descriptor (
_CGWarpPosition) replaces pynput's defaultpositionproperty onMouseController. It usesCGWarpMouseCursorPositionfor setting andCGEventGetLocationfor getting. Applied once at module level — all existing mouse methods benefit automatically.CGWarpMouseCursorPositionandCGAssociateMouseAndMouseCursorPositionare already available in scope via the existingfrom Quartz.CoreGraphics import *on line 48.Test Results (in macOS VM)
Test Plan
Fixes #1162
Summary by CodeRabbit
Bug Fixes
Refactor