Skip to content

Commit a3bc7a4

Browse files
wswebcreationcursoragent
wswebcreation
and
cursoragent
authored
fix: respect alwaysSaveActualImage: false for checkScreen methods (#1115) (#1116)
* fix: fix 1115 * fix: make saveAboveTolerance optional in CompareOptions interface This fixes the TypeScript error that occurred after removing saveAboveTolerance from DEFAULT_COMPARE_OPTIONS. Co-authored-by: Cursor <[email protected]> * test: fix #1115 test to properly create baseline and verify diff The test now: 1. Saves a baseline with enableLayoutTesting: true (no visible text) 2. Checks against it with enableLayoutTesting: false (text visible) 3. Verifies there's a mismatch but within threshold 4. Verifies actual image was NOT saved (the fix) Co-authored-by: Cursor <[email protected]> * chore: fix tests * chore: fix test * chore: also fix tests, not only push the images * test: add failing test for saveMethods * fix: always save images for save methods * chore: update changeset * fix: proper fix * chore: fix UTs --------- Co-authored-by: Cursor <[email protected]>
1 parent 77a6621 commit a3bc7a4

31 files changed

+505
-21
lines changed

‎.changeset/fix-always-save-actual-image.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
"@wdio/image-comparison-core": patch
3+
"@wdio/visual-service": patch
4+
---
5+
6+
## #1115 Respect `alwaysSaveActualImage: false` for `checkScreen` methods
7+
8+
When using visual matchers like `toMatchScreenSnapshot('tag', 0.9)` with `alwaysSaveActualImage: false`, the actual image was still being saved even when the comparison passed within the threshold.
9+
10+
The root cause was that the matcher's expected threshold was not being passed to the core comparison logic. The core used `saveAboveTolerance` (defaulting to 0) to decide whether to save images, while the matcher used the user-provided threshold to determine pass/fail - these were disconnected.
11+
12+
This fix ensures:
13+
- When `alwaysSaveActualImage: false` and `saveAboveTolerance` is not explicitly set, actual images are never saved (respecting the literal meaning of the option)
14+
- When `saveAboveTolerance` is explicitly set (like matchers do internally), actual images are saved only when the mismatch exceeds that threshold
15+
16+
# Committers: 1
17+
18+
- Wim Selles ([@wswebcreation](https://github.com/wswebcreation))

‎.changeset/fix-save-methods-always-save.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
"@wdio/image-comparison-core": patch
3+
"@wdio/visual-service": patch
4+
---
5+
6+
## Fix: `save*` methods now always save files regardless of `alwaysSaveActualImage` setting
7+
8+
Previously, when `alwaysSaveActualImage: false` was set in the configuration, `save*` methods (`saveScreen`, `saveElement`, `saveFullPageScreen`, `saveAppScreen`, `saveAppElement`) were not saving files to disk, causing test failures.
9+
10+
The `alwaysSaveActualImage` option is intended to control whether actual images are saved during `check*` methods (comparison operations), not `save*` methods. Since `save*` methods are explicitly designed to save screenshots, they should always save files regardless of this setting.
11+
12+
This fix ensures:
13+
- `save*` methods always save files to disk, even when `alwaysSaveActualImage: false` is set in the config
14+
- `alwaysSaveActualImage: false` continues to work correctly for `check*` methods (as intended for issue #1115)
15+
- The behavior is now consistent: `save*` = always save, `check*` = respect `alwaysSaveActualImage` setting
16+
17+
**Implementation details:**
18+
- The visual service overrides `alwaysSaveActualImage: true` when calling `save*` methods directly from the browser API
19+
- `save*` methods respect whatever `alwaysSaveActualImage` value is passed to them (no special logic needed)
20+
- `check*` methods pass through the config value (which may be `false`), so `save*` methods respect it when called internally
21+
- This clean separation ensures `save*` methods work correctly when called directly while still respecting `alwaysSaveActualImage` for `check*` methods
22+
23+
# Committers: 1
24+
25+
- Wim Selles ([@wswebcreation](https://github.com/wswebcreation))

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
"test:unit": "vitest --coverage --run",
2020
"test:unit:ui": "vitest --coverage --ui",
2121
"test:unit:watch": "vitest --coverage --watch",
22-
"test.local.init": "rimraf localBaseline && wdio ./tests/configs/wdio.local.init.conf.ts",
22+
"test.local.init": "rimraf localBaseline && SAVE_ACTUAL=true wdio ./tests/configs/wdio.local.init.conf.ts",
2323
"test.local.desktop": "wdio tests/configs/wdio.local.desktop.conf.ts",
2424
"test.local.emus.app": "wdio tests/configs/wdio.local.android.emus.app.conf.ts",
2525
"test.local.emus.web": "wdio tests/configs/wdio.local.android.emus.web.conf.ts",

‎packages/image-comparison-core/src/__snapshots__/base.test.ts.snap‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,6 @@ exports[`BaseClass > initializes default options correctly 1`] = `
2121
"ignoreNothing": false,
2222
"rawMisMatchPercentage": false,
2323
"returnAllCompareData": false,
24-
"saveAboveTolerance": 0,
2524
"scaleImagesToSameSize": false,
2625
},
2726
"disableBlinkingCursor": false,

‎packages/image-comparison-core/src/commands/__snapshots__/saveFullPageScreen.test.ts.snap‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,9 +134,55 @@ exports[`saveFullPageScreen > should handle missing dimension values with NaN fa
134134
"isNativeContext": false,
135135
"tag": "test-fullpage",
136136
"wicOptions": {
137+
"addIOSBezelCorners": false,
138+
"addressBarShadowPadding": 6,
137139
"alwaysSaveActualImage": true,
140+
"autoElementScroll": true,
141+
"autoSaveBaseline": false,
142+
"clearFolder": false,
143+
"compareOptions": {
144+
"blockOutSideBar": false,
145+
"blockOutStatusBar": false,
146+
"blockOutToolBar": false,
147+
"createJsonReportFiles": false,
148+
"diffPixelBoundingBoxProximity": 5,
149+
"ignoreAlpha": false,
150+
"ignoreAntialiasing": false,
151+
"ignoreColors": false,
152+
"ignoreLess": false,
153+
"ignoreNothing": false,
154+
"rawMisMatchPercentage": false,
155+
"returnAllCompareData": false,
156+
"saveAboveTolerance": 0,
157+
"scaleImagesToSameSize": false,
158+
},
159+
"disableBlinkingCursor": false,
160+
"disableCSSAnimation": false,
161+
"enableLayoutTesting": false,
162+
"enableLegacyScreenshotMethod": false,
138163
"formatImageName": "{tag}-{logName}-{width}x{height}-dpr-{dpr}",
164+
"fullPageScrollTimeout": 1500,
165+
"hideScrollBars": true,
166+
"isHybridApp": false,
139167
"savePerInstance": true,
168+
"tabbableOptions": {
169+
"circle": {
170+
"backgroundColor": "rgba(255, 0, 0, 0.4)",
171+
"borderColor": "rgba(255, 0, 0, 1)",
172+
"borderWidth": 1,
173+
"fontColor": "rgba(0, 0, 0, 1)",
174+
"fontFamily": "Arial",
175+
"fontSize": 10,
176+
"size": 10,
177+
},
178+
"line": {
179+
"color": "rgba(255, 0, 0, 1)",
180+
"width": 1,
181+
},
182+
},
183+
"toolBarShadowPadding": 6,
184+
"userBasedFullPageScreenshot": false,
185+
"waitForFontsLoaded": true,
140186
},
141187
}
142188
`;
@@ -557,9 +603,55 @@ exports[`saveFullPageScreen > should take full page screenshots and return resul
557603
"isNativeContext": false,
558604
"tag": "test-fullpage",
559605
"wicOptions": {
606+
"addIOSBezelCorners": false,
607+
"addressBarShadowPadding": 6,
560608
"alwaysSaveActualImage": true,
609+
"autoElementScroll": true,
610+
"autoSaveBaseline": false,
611+
"clearFolder": false,
612+
"compareOptions": {
613+
"blockOutSideBar": false,
614+
"blockOutStatusBar": false,
615+
"blockOutToolBar": false,
616+
"createJsonReportFiles": false,
617+
"diffPixelBoundingBoxProximity": 5,
618+
"ignoreAlpha": false,
619+
"ignoreAntialiasing": false,
620+
"ignoreColors": false,
621+
"ignoreLess": false,
622+
"ignoreNothing": false,
623+
"rawMisMatchPercentage": false,
624+
"returnAllCompareData": false,
625+
"saveAboveTolerance": 0,
626+
"scaleImagesToSameSize": false,
627+
},
628+
"disableBlinkingCursor": false,
629+
"disableCSSAnimation": false,
630+
"enableLayoutTesting": false,
631+
"enableLegacyScreenshotMethod": false,
561632
"formatImageName": "{tag}-{logName}-{width}x{height}-dpr-{dpr}",
633+
"fullPageScrollTimeout": 1500,
634+
"hideScrollBars": true,
635+
"isHybridApp": false,
562636
"savePerInstance": true,
637+
"tabbableOptions": {
638+
"circle": {
639+
"backgroundColor": "rgba(255, 0, 0, 0.4)",
640+
"borderColor": "rgba(255, 0, 0, 1)",
641+
"borderWidth": 1,
642+
"fontColor": "rgba(0, 0, 0, 1)",
643+
"fontFamily": "Arial",
644+
"fontSize": 10,
645+
"size": 10,
646+
},
647+
"line": {
648+
"color": "rgba(255, 0, 0, 1)",
649+
"width": 1,
650+
},
651+
},
652+
"toolBarShadowPadding": 6,
653+
"userBasedFullPageScreenshot": false,
654+
"waitForFontsLoaded": true,
563655
},
564656
}
565657
`;

‎packages/image-comparison-core/src/commands/__snapshots__/saveWebElement.test.ts.snap‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,9 +363,55 @@ exports[`saveWebElement > should call takeElementScreenshot with BiDi disabled w
363363
"isNativeContext": false,
364364
"tag": "test-element",
365365
"wicOptions": {
366+
"addIOSBezelCorners": false,
367+
"addressBarShadowPadding": 6,
366368
"alwaysSaveActualImage": true,
369+
"autoElementScroll": true,
370+
"autoSaveBaseline": false,
371+
"clearFolder": false,
372+
"compareOptions": {
373+
"blockOutSideBar": false,
374+
"blockOutStatusBar": false,
375+
"blockOutToolBar": false,
376+
"createJsonReportFiles": false,
377+
"diffPixelBoundingBoxProximity": 5,
378+
"ignoreAlpha": false,
379+
"ignoreAntialiasing": false,
380+
"ignoreColors": false,
381+
"ignoreLess": false,
382+
"ignoreNothing": false,
383+
"rawMisMatchPercentage": false,
384+
"returnAllCompareData": false,
385+
"saveAboveTolerance": 0,
386+
"scaleImagesToSameSize": false,
387+
},
388+
"disableBlinkingCursor": false,
389+
"disableCSSAnimation": false,
390+
"enableLayoutTesting": false,
391+
"enableLegacyScreenshotMethod": false,
367392
"formatImageName": "{tag}-{logName}-{width}x{height}-dpr-{dpr}",
393+
"fullPageScrollTimeout": 1500,
394+
"hideScrollBars": true,
395+
"isHybridApp": false,
368396
"savePerInstance": true,
397+
"tabbableOptions": {
398+
"circle": {
399+
"backgroundColor": "rgba(255, 0, 0, 0.4)",
400+
"borderColor": "rgba(255, 0, 0, 1)",
401+
"borderWidth": 1,
402+
"fontColor": "rgba(0, 0, 0, 1)",
403+
"fontFamily": "Arial",
404+
"fontSize": 10,
405+
"size": 10,
406+
},
407+
"line": {
408+
"color": "rgba(255, 0, 0, 1)",
409+
"width": 1,
410+
},
411+
},
412+
"toolBarShadowPadding": 6,
413+
"userBasedFullPageScreenshot": false,
414+
"waitForFontsLoaded": true,
369415
},
370416
}
371417
`;
@@ -581,9 +627,55 @@ exports[`saveWebElement > should call takeElementScreenshot with BiDi disabled w
581627
"isNativeContext": false,
582628
"tag": "test-element",
583629
"wicOptions": {
630+
"addIOSBezelCorners": false,
631+
"addressBarShadowPadding": 6,
584632
"alwaysSaveActualImage": true,
633+
"autoElementScroll": true,
634+
"autoSaveBaseline": false,
635+
"clearFolder": false,
636+
"compareOptions": {
637+
"blockOutSideBar": false,
638+
"blockOutStatusBar": false,
639+
"blockOutToolBar": false,
640+
"createJsonReportFiles": false,
641+
"diffPixelBoundingBoxProximity": 5,
642+
"ignoreAlpha": false,
643+
"ignoreAntialiasing": false,
644+
"ignoreColors": false,
645+
"ignoreLess": false,
646+
"ignoreNothing": false,
647+
"rawMisMatchPercentage": false,
648+
"returnAllCompareData": false,
649+
"saveAboveTolerance": 0,
650+
"scaleImagesToSameSize": false,
651+
},
652+
"disableBlinkingCursor": false,
653+
"disableCSSAnimation": false,
654+
"enableLayoutTesting": false,
655+
"enableLegacyScreenshotMethod": false,
585656
"formatImageName": "{tag}-{logName}-{width}x{height}-dpr-{dpr}",
657+
"fullPageScrollTimeout": 1500,
658+
"hideScrollBars": true,
659+
"isHybridApp": false,
586660
"savePerInstance": true,
661+
"tabbableOptions": {
662+
"circle": {
663+
"backgroundColor": "rgba(255, 0, 0, 0.4)",
664+
"borderColor": "rgba(255, 0, 0, 1)",
665+
"borderWidth": 1,
666+
"fontColor": "rgba(0, 0, 0, 1)",
667+
"fontFamily": "Arial",
668+
"fontSize": 10,
669+
"size": 10,
670+
},
671+
"line": {
672+
"color": "rgba(255, 0, 0, 1)",
673+
"width": 1,
674+
},
675+
},
676+
"toolBarShadowPadding": 6,
677+
"userBasedFullPageScreenshot": false,
678+
"waitForFontsLoaded": true,
587679
},
588680
}
589681
`;
@@ -969,9 +1061,55 @@ exports[`saveWebElement > should call takeElementScreenshot with correct options
9691061
"isNativeContext": false,
9701062
"tag": "test-element",
9711063
"wicOptions": {
1064+
"addIOSBezelCorners": false,
1065+
"addressBarShadowPadding": 6,
9721066
"alwaysSaveActualImage": true,
1067+
"autoElementScroll": true,
1068+
"autoSaveBaseline": false,
1069+
"clearFolder": false,
1070+
"compareOptions": {
1071+
"blockOutSideBar": false,
1072+
"blockOutStatusBar": false,
1073+
"blockOutToolBar": false,
1074+
"createJsonReportFiles": false,
1075+
"diffPixelBoundingBoxProximity": 5,
1076+
"ignoreAlpha": false,
1077+
"ignoreAntialiasing": false,
1078+
"ignoreColors": false,
1079+
"ignoreLess": false,
1080+
"ignoreNothing": false,
1081+
"rawMisMatchPercentage": false,
1082+
"returnAllCompareData": false,
1083+
"saveAboveTolerance": 0,
1084+
"scaleImagesToSameSize": false,
1085+
},
1086+
"disableBlinkingCursor": false,
1087+
"disableCSSAnimation": false,
1088+
"enableLayoutTesting": false,
1089+
"enableLegacyScreenshotMethod": false,
9731090
"formatImageName": "{tag}-{logName}-{width}x{height}-dpr-{dpr}",
1091+
"fullPageScrollTimeout": 1500,
1092+
"hideScrollBars": true,
1093+
"isHybridApp": false,
9741094
"savePerInstance": true,
1095+
"tabbableOptions": {
1096+
"circle": {
1097+
"backgroundColor": "rgba(255, 0, 0, 0.4)",
1098+
"borderColor": "rgba(255, 0, 0, 1)",
1099+
"borderWidth": 1,
1100+
"fontColor": "rgba(0, 0, 0, 1)",
1101+
"fontFamily": "Arial",
1102+
"fontSize": 10,
1103+
"size": 10,
1104+
},
1105+
"line": {
1106+
"color": "rgba(255, 0, 0, 1)",
1107+
"width": 1,
1108+
},
1109+
},
1110+
"toolBarShadowPadding": 6,
1111+
"userBasedFullPageScreenshot": false,
1112+
"waitForFontsLoaded": true,
9751113
},
9761114
}
9771115
`;
@@ -1277,9 +1415,55 @@ exports[`saveWebElement > should handle NaN dimension values correctly 3`] = `
12771415
"isNativeContext": false,
12781416
"tag": "test-element",
12791417
"wicOptions": {
1418+
"addIOSBezelCorners": false,
1419+
"addressBarShadowPadding": 6,
12801420
"alwaysSaveActualImage": true,
1421+
"autoElementScroll": true,
1422+
"autoSaveBaseline": false,
1423+
"clearFolder": false,
1424+
"compareOptions": {
1425+
"blockOutSideBar": false,
1426+
"blockOutStatusBar": false,
1427+
"blockOutToolBar": false,
1428+
"createJsonReportFiles": false,
1429+
"diffPixelBoundingBoxProximity": 5,
1430+
"ignoreAlpha": false,
1431+
"ignoreAntialiasing": false,
1432+
"ignoreColors": false,
1433+
"ignoreLess": false,
1434+
"ignoreNothing": false,
1435+
"rawMisMatchPercentage": false,
1436+
"returnAllCompareData": false,
1437+
"saveAboveTolerance": 0,
1438+
"scaleImagesToSameSize": false,
1439+
},
1440+
"disableBlinkingCursor": false,
1441+
"disableCSSAnimation": false,
1442+
"enableLayoutTesting": false,
1443+
"enableLegacyScreenshotMethod": false,
12811444
"formatImageName": "{tag}-{logName}-{width}x{height}-dpr-{dpr}",
1445+
"fullPageScrollTimeout": 1500,
1446+
"hideScrollBars": true,
1447+
"isHybridApp": false,
12821448
"savePerInstance": true,
1449+
"tabbableOptions": {
1450+
"circle": {
1451+
"backgroundColor": "rgba(255, 0, 0, 0.4)",
1452+
"borderColor": "rgba(255, 0, 0, 1)",
1453+
"borderWidth": 1,
1454+
"fontColor": "rgba(0, 0, 0, 1)",
1455+
"fontFamily": "Arial",
1456+
"fontSize": 10,
1457+
"size": 10,
1458+
},
1459+
"line": {
1460+
"color": "rgba(255, 0, 0, 1)",
1461+
"width": 1,
1462+
},
1463+
},
1464+
"toolBarShadowPadding": 6,
1465+
"userBasedFullPageScreenshot": false,
1466+
"waitForFontsLoaded": true,
12831467
},
12841468
}
12851469
`;

0 commit comments

Comments
 (0)