Add neuron selection support to SequenceInterpolator and SpikeInterpolator#126
Add neuron selection support to SequenceInterpolator and SpikeInterpolator#126Vrittigyl wants to merge 8 commits intosensorium-competition:mainfrom
Conversation
|
Review these changes at https://app.gitnotebooks.com/sensorium-competition/experanto/pull/126 |
There was a problem hiding this comment.
Pull request overview
Adds neuron-subset selection to interpolators so callers can load/process only specific neurons by biological IDs (meta/unit_ids.npy) or by direct column/neuron indexes, reducing memory usage for large datasets (Issue #124).
Changes:
- Extend
SequenceInterpolatorto acceptneuron_ids/indexesand filter loaded data + normalization stats accordingly. - Extend
PhaseShiftedSequenceInterpolatorto filterphase_shiftsconsistently with neuron selection. - Extend
SpikeInterpolatorto acceptneuron_ids/indexesand rebuildspikes/indicesarrays for the selected neurons.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -230,20 +269,34 @@ def __init__( | |||
| else: | |||
| self._data = np.load(self.root_folder / "data.npy") | |||
|
|
|||
| # Filter selected neurons from data | |||
| if self.indexes is not None: | |||
| self._data = self._data[:, self.indexes] | |||
| self.n_signals = len(self.indexes) | |||
|
|
|||
There was a problem hiding this comment.
New neuron-selection behavior (filtering _data, updating n_signals, filtering means/stds, and filtering phase_shifts in PhaseShiftedSequenceInterpolator) is not covered by existing tests. Please add test cases exercising neuron_ids and indexes selection (including mismatch between the two, ordering behavior, and normalization stats shape after filtering).
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
experanto/interpolators.py
Outdated
| @@ -230,20 +290,34 @@ def __init__( | |||
| else: | |||
| self._data = np.load(self.root_folder / "data.npy") | |||
There was a problem hiding this comment.
IMO if data gets cached, we should only cache the indexed neurons. Downside is that if someone alters self.indexes of the instance later manually, it is still relying on the old data. But I think I still prefer this. What do you think @pollytur ?
If changing indexes later is a real use case, one could handle it with a function that takes care of it and would reload the data appropriately, but I am not sure we need it right now.
There was a problem hiding this comment.
I’ve changed the logic so caching happens after applying neuron_indices, so only the selected subset is loaded into memory. According to me this would keep memory usage efficient for large data.
can revert changes if needed
Vrittigyl
force-pushed
the
neuron-selection
branch
from
c070196 to
ff9f0b3
Compare
| else: | ||
| self._data = np.load(self.root_folder / "data.npy") | ||
|
|
||
| # Apply indexing BEFORE caching |
There was a problem hiding this comment.
Hi @pollytur
as far as I know, if one applies indexes to memmap as lists or arrays, this loads a copy into RAM (see https://stackoverflow.com/questions/18614927/how-to-slice-memmap-efficiently or https://stackoverflow.com/questions/78426050/how-to-index-a-numpy-memmap-without-creating-an-in-memory-copy)
Only regular continuous slicing creates just a view, but I don't think neuron ids will always be continuous.
Have you thought about this? We might need to set caching to True if neurons are indexed. Or we find a workaround (I haven't investigated it yet).
There was a problem hiding this comment.
great catch, no I have not thought about it tbh
we probably want to investigate the workaround (changing order of neurons as we need and save it as a temp memmap file is the first though but thats insanly memory inefficient since neuronal responses are also the heaviest part of the dataset from a memory perspective...)
There was a problem hiding this comment.
Or maybe instead of directly indexing, we could iteratively fetch only the required columns (or in chunks) and optionally cache them.
Description
This PR adds support for selecting specific neurons in both
SequenceInterpolatorandSpikeInterpolator.Users can now pass either:
neuron_ids: biological neuron IDs mapped usingmeta/unit_ids.npyindexes: direct neuron indexesIf both are provided:
ValueErroris raised if they refer to different neurons.Changes
SequenceInterpolator
neuron_idsorindexes.PhaseShiftedSequenceInterpolator
phase_shiftsto match selected neurons when neuron selection is used.SpikeInterpolator
neuron_idsorindexes.This allows loading and processing only a subset of neurons, reducing memory usage and improving flexibility when working with imaging datasets.
Closes #124