How to run the example please?

[zcy618]

dear friend:
Could you give one instruction to show how to run the example please?
Thanks.

[boeddeker]

Dear zcy618,

do you mean, how you can run the file https://github.com/fgnt/pb_bss/blob/master/examples/mixture_model_example.ipynb ?
This is a jupyter notebook.
In the documentation for jupyter, you can read, how to launch a notebook server: https://jupyterlab.readthedocs.io/en/stable/

[pfeatherstone]

Is there a simple example for inference?

[boeddeker]

I am not sure, what you mean with inference.
The mixture models are categorized as blind source separation technique.

They have no training phase like neuronal networks.
They learn the parameters on a single example, and they are applied usually only to this example.
Applying it to an independent example will not work, because the spatial properties are different.

[pfeatherstone]

ah ok got you

[pfeatherstone]

Can this be used with a microphone array for example to do source separation?

[boeddeker]

Can this be used with a microphone array for example to do source separation?

Yes, the idea of the mixture models (cACG and complex Watson) is to model spatial differences between the different sources.
In https://github.com/fgnt/pb_bss/blob/master/examples/mixture_model_example.ipynb the observation is a multichannel signal with multiple sources at different positions.

[pfeatherstone]

So if all i have is the observation, i.e. the multichannel signal, what are speech_image_0.wav, speech_image_1.wav, ..., noise_image.wav, speech_source_0.wav, speech_source_1.wav, ... ?

[pfeatherstone]

Are the others only used for metrics?

[boeddeker]

So if all i have is the observation, i.e. the multichannel signal, what are speech_image_0.wav, speech_image_1.wav, ..., noise_image.wav, speech_source_0.wav, speech_source_1.wav, ... ?

These files were used to generate the observation.

• The postfix (_0 and _1) indicate the speaker.
• speech_source: The non-reverberant speech source (Singlechannel).
• speech_image: The speech_source convolved with the room impulse response (Multichannel).
• noise_image: The noise signal that is received at the microphones (Multichannel)

Are the others only used for metrics?

Yes, inside the notebook we use these signal to get an idea of the performance.
So they are only used for metrics and visualization.

[pfeatherstone]

So if i have a multichannel audio file and i want to extract 3 candidate sources, how can i do that? I have this so far:

import  soundfile
import  numpy as np
from    nara_wpe.utils import stft, istft
from    pb_bss.distribution import CACGMMTrainer
from    pb_bss.permutation_alignment import DHTVPermutationAlignment, OraclePermutationAlignment
from    einops import rearrange

def soundfile_read(file):
    data, data_sample_rate = soundfile.read(file)
    return data_sample_rate, np.ascontiguousarray(data.T)

def bss_pb_bss(multichannel_wav):
    samplerate, data = soundfile_read(multichannel_wav)
    Observation = stft(data, 512, 128)
    trainer = CACGMMTrainer()
    Observation_mm = rearrange(Observation, 'd t f -> f t d')
    model = trainer.fit(
        Observation_mm,
        num_classes=3,
        iterations=40,
        inline_permutation_aligner=None
    )
    affiliation = model.predict(Observation_mm)
    pa = DHTVPermutationAlignment.from_stft_size(512)
    affiliation_pa = pa(rearrange(affiliation, 'f k t -> k f t'))
    Speech_image_0_est  = Observation[0, :, :].T * affiliation_pa[0, :, :]
    Speech_image_1_est  = Observation[0, :, :].T * affiliation_pa[1, :, :]
    Noise_image_est     = Observation[0, :, :].T * affiliation_pa[2, :, :]
    speech_image_0_est  = istft(Speech_image_0_est.T, 512, 128)[..., :Observation.shape[-1]]
    speech_image_1_est  = istft(Speech_image_1_est.T, 512, 128)[..., :Observation.shape[-1]]
    noise_image_est     = istft(Noise_image_est.T, 512, 128)[..., :Observation.shape[-1]]

[pfeatherstone]

It looks like OraclePermutationAlignment requires global_pa_reference which requires Noise_image which i don't have.

[pfeatherstone]

do i need to resample speech_image_0_est, speech_image_1_est and noise_image_est ? I think i'm missing something.

[boeddeker]

The OraclePermutationAlignment is used to identify the first speaker, the second speaker and the noise.
In a real application you cannot do this. Instead, you need to identify the noise.
At the moment we have no code to identify the noise.
Usually the noise class is active at the beginning and end of an utterence.

do i need to resample speech_image_0_est, speech_image_1_est and noise_image_est ? I think i'm missing something.

Resampling is not necessary. Maybe you observed, that the noise_image_est is a speaker. The noise has no fixed index.

[pfeatherstone]

So you can only use this algorithm if you know the noise? If you did a recording of the noise profile, would that work?

[boeddeker]

No, it is not necessary to know the noise.
You can use a heuristic to identify the noise or you can reduce the number of speakers to the actual number of speakers.
We observed a better performance, when the noise is modelled as a separate class, but it also works without.

The heuristic can be using a special initialization, identifying the noise mask or depend on a system before or after the mixture model. It highly depends on the actual application, that you have in mind.