Skip to content

restore v19 behavior: push LIMIT to each shard for non-scatter multi-shard DML - #838

Merged
tanjinx merged 10 commits into
slack-22.0from
tanjin-scatter-update-limit-optout
Apr 29, 2026
Merged

restore v19 behavior: push LIMIT to each shard for non-scatter multi-shard DML#838
tanjinx merged 10 commits into
slack-22.0from
tanjin-scatter-update-limit-optout

Conversation

@tanjinx

@tanjinx tanjinx commented Apr 9, 2026

Copy link
Copy Markdown

What's this?

In v19, multi-shard UPDATE/DELETE with LIMIT on non-scatter routes (e.g. WHERE id IN (...)) worked by pushing the LIMIT to each shard directly.

In v22, this behavior changed: all multi-shard DML with LIMIT now goes through DMLWithInput, which requires schema tracking to be enabled, does a primary key lookup select first, and enforces a global limit across shards. This breaks queries that previously worked fine and adds overhead for cases where per-shard LIMIT is acceptable.

This PR restores the v19 behavior for non-scatter routes by always pushing LIMIT to each shard. Scatter DML with LIMIT is unaffected -- it still uses DMLWithInput with global limit enforcement.

Query examples

-- IN clause on vindex: LIMIT is pushed to each shard (v19 behavior restored)
UPDATE user_extra SET val = 1 WHERE user_id IN (1, 2) LIMIT 5;
-- v19/this PR: each shard gets "update ... limit 5" directly
-- v22 (before this PR): requires schema tracking, does primary key select first, enforces global limit

DELETE FROM user_extra WHERE user_id IN (1, 2) LIMIT 5;
-- same as above

-- Scatter route: still uses DMLWithInput with global limit (unchanged)
UPDATE user SET val = 1 WHERE (name = 'foo' OR id = 1) LIMIT 1;
DELETE FROM user LIMIT 10;

Real-world example from loadtest:

mysql> vexplain queries update loadtest_users
set is_inactive=0 where id in (15003119,0,1,4569401) and is_inactive=1 limit 5;
+------+----------+-------+-------------------------------------------------------------------------------------------------+
| #    | keyspace | shard | query                                                                                           |
+------+----------+-------+-------------------------------------------------------------------------------------------------+
|    0 | loadtest | -80   | begin                                                                                           |
|    0 | loadtest | -80   | update loadtest_users set is_inactive = 0 where id in (1, 4569401) and is_inactive = 1 limit 5  |
|    1 | loadtest | 80-   | begin                                                                                           |
|    1 | loadtest | 80-   | update loadtest_users set is_inactive = 0 where id in (15003119, 0) and is_inactive = 1 limit 5 |
+------+----------+-------+-------------------------------------------------------------------------------------------------+

What changed

  • Removed --scatter-update-limit-passthru flag from vtgate (was added as an opt-in, now always-on)
  • Removed IsScatterUpdateLimitPassthruEnabled() from VSchema interface and all implementations
  • Simplified tryPushingDownLimitInRoute to always push LIMIT for non-scatter DML
  • Moved test cases into dml_cases.json, deleted standalone test file
  • Removed flag from help text files (vtgate.txt, vtcombo.txt)

Most of this was written by Claude Code -- I just provided direction.

When enabled, multi-shard UPDATE/DELETE with LIMIT on non-scatter
routes (e.g. WHERE id IN (...) on a vindex column) pushes the LIMIT
to each shard directly instead of converting to DMLWithInput which
requires schema tracking. Scatter routes are unaffected.

This restores v19 behavior for targeted multi-shard DML with LIMIT
behind an opt-in flag.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
@salesforce-cla

salesforce-cla Bot commented Apr 9, 2026

Copy link
Copy Markdown

Thanks for the contribution! Before we can merge this, we need @frouioui @vitess-bot @systay @harshit-gangal @dbussink to sign the Salesforce Inc. Contributor License Agreement.

@tanjinx
tanjinx changed the base branch from main to slack-22.0 April 9, 2026 23:19
@tanjinx
tanjinx requested a review from a team as a code owner April 9, 2026 23:19
@tanjinx tanjinx modified the milestones: v22.0.4, v24.0.0 Apr 9, 2026
@github-actions github-actions Bot modified the milestones: v22.0.4, v24.0.0 Apr 9, 2026
@tanjinx tanjinx modified the milestones: v24.0.0, v22.0.4 Apr 9, 2026
tanjinx and others added 2 commits April 9, 2026 16:36
…PassthruEnabled

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>

Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
@codecov-commenter

codecov-commenter commented Apr 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.53%. Comparing base (b1ab347) to head (c8979d4).
⚠️ Report is 70 commits behind head on slack-22.0.

Additional details and impacted files
@@              Coverage Diff               @@
##           slack-22.0     #838      +/-   ##
==============================================
+ Coverage       67.53%   69.53%   +2.00%     
==============================================
  Files            1600     1606       +6     
  Lines          261782   214319   -47463     
==============================================
- Hits           176786   149025   -27761     
+ Misses          84996    65294   -19702     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tanjinx and others added 3 commits April 10, 2026 08:39
The flag is only registered for vtgate and vtcombo, not vttestserver.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
Add one extra space to align the description column consistently
with other flags in vtgate.txt and vtcombo.txt.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
tanjinx and others added 2 commits April 29, 2026 13:43
The behavior is now always-on: non-scatter multi-shard DML with LIMIT
pushes the LIMIT to each shard directly. Scatter DML with LIMIT still
uses DMLWithInput as before.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
@tanjinx tanjinx changed the title feat: add scatter-update-limit-passthru vtgate flag remove scatter-update-limit-passthru flag, enable by default Apr 29, 2026
Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
@tanjinx tanjinx changed the title remove scatter-update-limit-passthru flag, enable by default push LIMIT to each shard for non-scatter multi-shard DML Apr 29, 2026
@tanjinx tanjinx changed the title push LIMIT to each shard for non-scatter multi-shard DML restore v19 behavior: push LIMIT to each shard for non-scatter multi-shard DML Apr 29, 2026
The user_extra table does not have a val column in the e2e schema.

Co-Authored-By: Claude <svc-devxp-claude@slack-corp.com>
Signed-off-by: Tanjin Xu <tanjin.xu@slack-corp.com>
@tanjinx
tanjinx merged commit 20c8867 into slack-22.0 Apr 29, 2026
91 checks passed
@tanjinx
tanjinx deleted the tanjin-scatter-update-limit-optout branch April 29, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants