Using vtkVolumePicker with vtkMultiVolume

[AndiSchratti]

After spending some time trying to achieve this, I believe it might not even be possible.

The goal is to detect a hit on the volume using vtkVolumePicker. This works perfectly fine with a single volume. However, when used in combination with a vtkMultiVolume, I often cannot register a proper hit - or worse, the picker reports a hit at a location far away from the actual volume, where no object exists.

Below is the part where I track the mouse coordinates and print on hit detected:

@state.change("mouse_move")
def mouse_coords(**kwargs):
    if state.mouse_move:
        x_new, y_new = state.mouse_move

        y_new = renderWindow.GetSize()[1] - (y_new - 62)  # height of view - (y - height of titlebar) -> (from bot. to top)
        x_new -= 300                                      # x - drawer

        hit = picker.Pick(x_new if state.main_drawer else x_new+300, y_new, 0, renderer)
        print(hit)
        if hit:
            print(picker.GetPickPosition())
            print(picker.GetPickNormal())

state.setdefault("mouse_move", [])

with SinglePageWithDrawerLayout(server,) as layout:
    layout.title.set_text("Hello trame")
    with layout.content:
        with vuetify.VContainer(
            fluid=True,
            classes="pa-0 fill-height",
            v_on="""(() => {
  window.addEventListener('mousemove', e => { if (e.altKey) mouse_move = [e.clientX, e.clientY] })
})()""",):
            view = vtk.VtkRemoteView(renderWindow)

My render pipeline is a bit more complicated, so here is the part I believe is relevant for the problem.

    if urls[0] is None:
        self.define_reader(2, urls[1])
        self.mapper.SetInputConnection(0, self.reader2.GetOutputPort())
        self.volume.SetVolume(self.volume2, 0)
    else:
        self.define_reader(1, urls[0])
        self.mapper.SetInputConnection(0, self.reader1.GetOutputPort())
        self.volume.SetVolume(self.volume1, 0)
        if urls[1] is not None:
            self.define_reader(2, urls[1])
            self.mapper.SetInputConnection(1, self.reader2.GetOutputPort())
            self.volume.SetVolume(self.volume2, 1)

def define_reader(self, reader, url):   # This method is inside a class, so it's not representative of my full code layout
    """
    Define the image reader
    :param reader: int 1 for reader1 or 2 for reader2
    :param url: path of the image
    """
    sitk_image = sitk.ReadImage(url)
    vtk_image = sitk_to_vtk_image(sitk_image, reorient=True)
    if reader == 1:
        self.reader1 = vtk_image_data_to_importer(vtk_image)
        self.reader1.Update(0)
    elif reader == 2:
        self.reader2 = vtk_image_data_to_importer(vtk_image)
        self.reader2.Update(0)

def vtk_image_data_to_importer(vtk_image: vtkImageData) -> vtkImageImport:
    """
    Converts a vtkImageData object to a vtkImageImport instance

    :param vtk_image: The input VTK image data to convert

    :return: vtkImageImport containing the image data and all spatial metadata
    :rtype: vtkImageImport
    """
    ...

If I had to guess, I’d say the issue comes from the MultiVolume, although I couldn’t find what exactly is causing it while debugging.

[alesgenova]

If you create a plain VTK/Python example (i.e. without Trame) do you get the same issue/behavior?

[AndiSchratti]

I tested it, and yes, the behaviour is the same... I guess MultiVolume isn’t meant to be used with a picker :c

EDIT: The problem wasn't the volume itself, but the vtkGPUVolumeRayCastMapper() - is there another mapper that supports vtkMultiVolume()?