fix: register mangled image names to prevent export collisions - #10915
Open
Niruhqs wants to merge 2 commits into
Open
fix: register mangled image names to prevent export collisions#10915Niruhqs wants to merge 2 commits into
Niruhqs wants to merge 2 commits into
Conversation
mangle_image_name() picked a collision-free candidate name (e.g. "image_1") when the original name was already taken, but never recorded that candidate in the `names` tracking dict. A later image whose real name happened to match an earlier unrecorded candidate would then be treated as unclaimed and returned unchanged, silently overwriting the earlier file in the exported dataset. Track every name actually handed out, not just the original name's counter. Fixes cvat-ai#8076
Comment on lines
+1069
to
+1070
| image_files = generate_image_files(1, filenames=["image.jpg"]) | ||
| task_params = { |
Contributor
There was a problem hiding this comment.
Consider doing this to test the issue being fixed:
Suggested change
| image_files = generate_image_files(1, filenames=["image.jpg"]) | |
| task_params = { | |
| image_files = generate_image_files( | |
| 3, filenames=["image.jpg", "image_1.jpg", "image_1_1.jpg"] | |
| ) | |
| task_params = { |
Comment on lines
+1071
to
+1074
| "name": task_name, | ||
| "segment_size": 1, | ||
| "project_id": project.id, | ||
| } |
Contributor
There was a problem hiding this comment.
Suggested change
| "name": task_name, | |
| "segment_size": 1, | |
| "project_id": project.id, | |
| } | |
| "name": task_name, | |
| "project_id": project.id, | |
| } |
segment size is irrelevant here
Comment on lines
+1093
to
+1100
| assert len(set(exported_image_names)) == 2 | ||
|
|
||
| with zip_file.open("annotations/instances_default.json") as anno_file: | ||
| annotations = json.load(anno_file) | ||
| annotated_names = [a["file_name"] for a in annotations["images"]] | ||
| assert len(set(annotated_names)) == 2 | ||
| assert sorted(annotated_names) == sorted(exported_image_names) | ||
|
|
Contributor
There was a problem hiding this comment.
Suggested change
| assert len(set(exported_image_names)) == 2 | |
| with zip_file.open("annotations/instances_default.json") as anno_file: | |
| annotations = json.load(anno_file) | |
| annotated_names = [a["file_name"] for a in annotations["images"]] | |
| assert len(set(annotated_names)) == 2 | |
| assert sorted(annotated_names) == sorted(exported_image_names) | |
| assert len(exported_image_names) == 6 | |
| assert len(set(exported_image_names)) == 6 | |
| with zip_file.open("annotations/instances_default.json") as anno_file: | |
| annotations = json.load(anno_file) | |
| annotated_names = [a["file_name"] for a in annotations["images"]] | |
| assert len(set(annotated_names)) == 6 | |
| assert sorted(annotated_names) == sorted(exported_image_names) | |
Comment on lines
+2
to
+6
|
|
||
| - Project export could silently overwrite images across tasks when | ||
| filename collisions were resolved with a generated fallback name that | ||
| was never marked as taken | ||
| (<https://github.com/cvat-ai/cvat/pull/10915>) |
Contributor
There was a problem hiding this comment.
Suggested change
| - Project export could silently overwrite images across tasks when | |
| filename collisions were resolved with a generated fallback name that | |
| was never marked as taken | |
| (<https://github.com/cvat-ai/cvat/pull/10915>) | |
| - Project export could silently overwrite images across tasks after resolving filename collisions | |
| (<https://github.com/cvat-ai/cvat/pull/10915>) |
zhiltsov-max
left a comment
Contributor
There was a problem hiding this comment.
Hi, thank you for sending the PR. Please address the comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8076
Motivation and context
mangle_image_name()resolves filename collisions during project export by trying candidates likeimage_1,image_1_1, etc. When it finds a free candidate, it only increments the counter for the original name in thenamestracking dict, never registering the candidate itself as taken.If task order in the project causes the fallback names to be generated before the real files that happen to share those names are processed, the real files are later returned unchanged (since their name was never marked as taken) and silently overwrite the earlier export, exactly as described in the issue.
The fix registers every name actually handed out, not just the original name's counter, so a later real file with a colliding name gets re-mangled instead of overwriting an already-exported image.
How has this been tested?
Added
test_export_project_with_duplicate_image_names_across_taskstotests/python/rest_api/test_projects.py, modeled on the existingtest_export_project_with_honeypotstest: creates a project with two tasks whose images share the same original filename, exports it, and asserts both survive as distinct files with matching annotation entries.Verified the fix in isolation with a standalone script reproducing the collision scenario from the issue (5 tasks with a duplicate image.jpg, followed by a task with the real image_1.jpg to image_4.jpg): the current code produces 4 collisions, the fix produces 0.
I was not able to run the full test suite locally (no local CVAT dev/Docker environment set up), so I'd appreciate CI/reviewer confirmation that the added test passes.
Checklist
I submit my changes into the develop branch. I have added a description of my changes into the CHANGELOG file (to follow up once this PR's number is known). I have updated the documentation accordingly (not applicable, internal helper function). I have added tests to cover my changes. Related issue is linked above via "Fixes #8076".
License
I submit my code changes under the same MIT License that covers the project.