Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds parallel processing utilities (pjit and threadpool) ported from the AlphaTims library, enabling efficient multi-threaded execution of functions with progress tracking support. The implementation includes utilities for thread management, conditional Numba JIT compilation, and progress callbacks.
Changes:
- Added
alpharaw/utils/pjit.pywith thread pool decorators and Numba-based parallelization utilities - Removed deprecated type annotation rules from the ruff configuration
- Fixed a typo in the ruff ignore rule identifier (
TCH003→TC003)
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| alpharaw/utils/pjit.py | New module providing pjit and threadpool decorators for parallelizing functions with thread pools and progress callbacks |
| ruff-lint-strict.toml | Removed deprecated ANN101/ANN102 rules and corrected typo in TC003 rule identifier |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Default is True. | ||
| return_results : bool | ||
| If True, it returns the results in the same order as the iterable. | ||
| This can be much slower than not returning results. Iti is better to |
There was a problem hiding this comment.
Corrected spelling of 'Iti' to 'It'.
| This can be much slower than not returning results. Iti is better to | |
| This can be much slower than not returning results. It is better to |
| If True, it returns the results in the same order as the iterable. | ||
| This can be much slower than not returning results. Iti is better to | ||
| store them in a buffer results array instead | ||
| (be carefull to avoid race conditions). |
There was a problem hiding this comment.
Corrected spelling of 'carefull' to 'careful'.
| (be carefull to avoid race conditions). | |
| (be careful to avoid race conditions). |
| def wrapper(iterable, *args) -> None: # noqa: ANN001 | ||
| """A wrapper function that parallelizes the numba.njit function. | ||
|
|
||
| The first argument of the wrapped function is seperately stored as `iterable` and its elements are |
There was a problem hiding this comment.
Corrected spelling of 'seperately' to 'separately'.
| The first argument of the wrapped function is seperately stored as `iterable` and its elements are | |
| The first argument of the wrapped function is separately stored as `iterable` and its elements are |
| results = [] | ||
| for result in _progress_callback( | ||
| pool.imap(starfunc, iterable), | ||
| total=len(iterable), | ||
| include_progress_callback=include_progress_callback, | ||
| ): | ||
| results.append(result) # noqa: PERF402 |
There was a problem hiding this comment.
The # noqa: PERF402 comment suppresses a performance warning about using append in a loop. Consider using a list comprehension instead: results = list(_progress_callback(...)) to improve performance.
| results = [] | |
| for result in _progress_callback( | |
| pool.imap(starfunc, iterable), | |
| total=len(iterable), | |
| include_progress_callback=include_progress_callback, | |
| ): | |
| results.append(result) # noqa: PERF402 | |
| results = list( | |
| _progress_callback( | |
| pool.imap(starfunc, iterable), | |
| total=len(iterable), | |
| include_progress_callback=include_progress_callback, | |
| ) | |
| ) |
| while progress_bar >= progress_count: | ||
| time.sleep(0.01) # this will be done by the main thread | ||
| progress_count = granularity * np.sum(progress_counter) / len(iterable) | ||
| progress_bar += 1 # noqa: SIM113 |
There was a problem hiding this comment.
The # noqa: SIM113 comments suppress warnings about using += instead of the increment operator. These suppressions appear unnecessary since += is the preferred Python idiom. Consider removing these noqa comments.
| progress_bar += 1 # noqa: SIM113 | |
| progress_bar += 1 |
| steps = current_progress_callback.max / 1000 | ||
| progress = 0 | ||
| for element in iterable: | ||
| progress += 1 # noqa: SIM113 |
There was a problem hiding this comment.
The # noqa: SIM113 comments suppress warnings about using += instead of the increment operator. These suppressions appear unnecessary since += is the preferred Python idiom. Consider removing these noqa comments.
| progress += 1 # noqa: SIM113 | |
| progress += 1 |
GeorgWa
left a comment
GeorgWa
left a comment
There was a problem hiding this comment.
I did not review this in detail. Why did alphabase not feel like the right place for this?
I think this is fine but alphabase would be equally fine for me.
add pjit and threadpool code from alphatims b18e1553f
Adressed only basic linting,
noqaed the rest.Note: chose to put it here as the
TimsTOFclass that will also moved here from alphatims depends on it and because alphabase felt not like the right place.