What is the best state of the art method to "augment" data recorded from physical devices

[kirk86]

Hi folks,

I have some timeseries data recorded from fire and carbon monoxide detectors for different test case scenarios.

The issue is that this dataset is imbalanced such that some test case scenarios have more samples that others.

I have thought of a naive approach to balance it would be to use upsampling on the minority test cases but I'm skeptical it might lead to hidden risks of biasing the data and I would like to avoid those or any unnecessary "artificial" correlations created by the upsampling process.

That's why I thought I should ask here to see if folks have any better ideas or know what are the current state of the art methods to generate synthetic data for the test case scenarios with few samples.

Hopefully there exist some methods that can generate synthetic data similar to real recordings without introducing too many unnecessary correlations and hidden biases.

If you know any, or even better if you've used in the past or have found yourself in a similar situation, please let me know you resolved it and what you used. Thanks 🙏

[drjfloyd]

Are you saying you have FDS simulation data from multiple simulations where the number of rows in the devc.csv files have different timesteps or different numbers of rows? Or is this data from an experiment?

Why does it matter that some tests scenarios have more samples? What are you doing with the data that this is important?

[kirk86]

Thanks for the reply.
The data are recordings from real sensors/devices on real fires in fire rooms.
The data represent different fire experiments (e.g., burning wood, paper, plastic, etc.)
Some experiments have more samples than others.

Why does it matter that some tests scenarios have more samples? What are you doing with the data that this is important?

I'm trying to test and evaluate a classifier to discriminate against the different experiments. That's why the imbalance on the data can cause it to be biased on the class/experiment with the largest numbers of samples/rows per .csv file.

Hope that clarifies things a bit. Let me know if you have any suggestions on how to augment the data avoiding potential pitfalls I mentioned previously.

[drjfloyd]

Are you able to adjust the classifier so it is not sensitive to the number of rows?

Anything you do that modifies real measured data is going to introduce some degree of perturbation to that data. The more smoothly varing the data is or the closer the files are in row count, the less that perturbutation is likely to be a concern. Mathematica, Matlab, and python all have tools to define interpolation functions for data. You could define functions for each column of data and then generate a new csv file with a specified number of rows. If you don't have access/ability to use those tools then you could use a spreadsheet and generate a linear interpolation to a new set of time points.