Polish public benchmark documentation#32
Merged
Conversation
Contributor
Reviewer's GuideDocumentation-only polish to the public benchmark, clarifying the benchmark scope/results, documenting CLI/asset generation flows, and updating contribution guidance and decoder registration instructions. Class diagram for decoder registration via entry points and registryclassDiagram
class BaseDecoder {
+decode(bytes data) uint8_array
+decode_path(str path) uint8_array
}
class FooDecoder {
+decode(bytes data) uint8_array
}
class DecoderEntryPoint {
+group_name
+name
+target_module
+target_attr
+load() BaseDecoder
}
class DecoderRegistry {
-decoders : dict
+load_from_entry_points()
+get(str name) BaseDecoder
+names() list
}
BaseDecoder <|-- FooDecoder
DecoderEntryPoint --> BaseDecoder : loads
DecoderRegistry "*" --> "*" DecoderEntryPoint : reads
DecoderRegistry "*" --> "*" BaseDecoder : builds REGISTRY
class PyprojectEntryPoints {
+imread_benchmark.decoders
+foo = imread_benchmark.decoders.foo_decoder:FooDecoder
}
PyprojectEntryPoints --> DecoderEntryPoint : defines
class ImreadBenchmarkCLI {
+run()
}
ImreadBenchmarkCLI --> DecoderRegistry : uses decoders
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="README.md" line_range="90" />
<code_context>
+- `torchvision` wins both AMD platforms at peak `DataLoader` throughput and is effectively tied for first on Neoverse V2.
+- `imageio` is not a single-thread leader, but wins peak `DataLoader` throughput on Neoverse V2 in the current GCP runs.
+- OpenCV is rarely the absolute winner, but is consistently close to the local winner and has successful `DataLoader` results on every platform.
+- PyVips is reported for single-thread decode only; it is skipped in fork-based `DataLoader` benchmarks because of libvips threadpool deadlocks in this harness.
+
## GitAds Sponsored
</code_context>
<issue_to_address>
**nitpick (typo):** Consider correcting "threadpool" to "thread pool" and aligning the capitalization of PyVips/`pyvips`.
Here, “threadpool” is typically written as “thread pool” in technical docs, and the library appears as “pyvips” elsewhere in the README. For consistency and searchability, consider updating this bullet to use “pyvips” and “thread pool.”
```suggestion
- `pyvips` is reported for single-thread decode only; it is skipped in fork-based `DataLoader` benchmarks because of libvips thread pool deadlocks in this harness.
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| - `torchvision` wins both AMD platforms at peak `DataLoader` throughput and is effectively tied for first on Neoverse V2. | ||
| - `imageio` is not a single-thread leader, but wins peak `DataLoader` throughput on Neoverse V2 in the current GCP runs. | ||
| - OpenCV is rarely the absolute winner, but is consistently close to the local winner and has successful `DataLoader` results on every platform. | ||
| - PyVips is reported for single-thread decode only; it is skipped in fork-based `DataLoader` benchmarks because of libvips threadpool deadlocks in this harness. |
Contributor
There was a problem hiding this comment.
nitpick (typo): Consider correcting "threadpool" to "thread pool" and aligning the capitalization of PyVips/pyvips.
Here, “threadpool” is typically written as “thread pool” in technical docs, and the library appears as “pyvips” elsewhere in the README. For consistency and searchability, consider updating this bullet to use “pyvips” and “thread pool.”
Suggested change
| - PyVips is reported for single-thread decode only; it is skipped in fork-based `DataLoader` benchmarks because of libvips threadpool deadlocks in this harness. | |
| - `pyvips` is reported for single-thread decode only; it is skipped in fork-based `DataLoader` benchmarks because of libvips thread pool deadlocks in this harness. |
There was a problem hiding this comment.
Pull request overview
Documentation-focused PR to clarify what imread-benchmark is, how to interpret benchmark outputs, and how to reproduce/regenerate public + publication assets across local and GCP runs.
Changes:
- Refines README framing (project purpose, result interpretation, recommendations) and updates output/examples + system requirements wording.
- Expands GCP benchmarking docs to emphasize reproducible cross-machine comparisons and documents regenerating public/paper assets.
- Updates CONTRIBUTING with guidance for anonymous review mirrors and clarifies decoder registration via
pyproject.tomlentry points.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| README.md | Rewrites overview/recommendations, adds “what results mean” narrative, updates output layout example, and documents asset regeneration commands. |
| docs/gcp_benchmarks.md | Clarifies purpose of GCP runner scripts and adds steps for regenerating README + paper assets from fetched JSON outputs. |
| CONTRIBUTING.md | Adds anonymous-mirror checklist and updates decoder registration instructions to match entry-point based registry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| `imread-benchmark` is a reproducible benchmark framework for JPEG decoding in Python ML pipelines. It provides: | ||
|
|
||
| - an installable `imread-benchmark` CLI for local datasets, | ||
| - isolated per-library worker environments so conflicting stacks can be benchmarked in one run, |
Comment on lines
+84
to
+88
| Current headline patterns: | ||
|
|
||
| - `simplejpeg` is a strong single-thread baseline and wins peak `DataLoader` throughput on Intel Emerald Rapids and Neoverse N1. | ||
| - `torchvision` wins both AMD platforms at peak `DataLoader` throughput and is effectively tied for first on Neoverse V2. | ||
| - `imageio` is not a single-thread leader, but wins peak `DataLoader` throughput on Neoverse V2 in the current GCP runs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Clarify and expand the public description of the imread-benchmark framework, including how results should be interpreted and how assets are generated.
Documentation:
Chores: