Make AMF estimators drift resilient

[MaxHalford]

AMFClassifier and AMFRegressor are the most accurate models we have. But they are not designed to do well with drift. Indeed, I realized when working on #1825 that copying the counts from the parents to the child can hinder performance (e.g. on the Elec2 dataset), likely because it anchors the model into the past distribution. The previous bug of not copying introduced some kind of unintended regularization because the class count was reset.

One thing we could start with is attempting to use rolling counts. But it might be a hack.

[shi-zq]

Hi,

Because the Aggregated Mondrian Forest (AMF) lacks a built-in mechanism to forget historical data, it relies on the step parameter to handle concept drift, which may not be sufficient.

Assuming the number of target classes is known a priori (to prevent trees from returning a zero probability for certain predictions), we could integrate an ADWIN drift detection mechanism into the Mondrian trees. Upon detecting drift, we could either discard the obsolete tree and retrain it from scratch, or save the last N samples to train a replacement Mondrian tree. (However, saving samples might go against standard stream machine learning principles, as we generally want to avoid storing raw data due to memory constraints).

Furthermore, the current voting system in AMF relies on the unweighted sum of prediction dictionaries. I suggest having each tree store a class-specific accuracy dictionary instead. The final ensemble prediction would then be weighted according to these individual accuracies. This approach could allow certain trees to act as specialists for specific classes.

I am still a newcomer to the field of SML, so I am not sure if I explained my ideas clearly. These issues were found during the implementation of the Mondrian tree in the Rust version.

[MaxHalford]

These issues were found during the implementation of the Mondrian tree in the Rust version

Are you implementing a Rust version on your side? We eventually want to do that in River.

[shi-zq]

No, I am currently working on a thesis about it. I would be happy to contribute after my thesis is finished.

These issues were found during the implementation of the Mondrian tree in the Rust version

Are you implementing a Rust version on your side? We eventually want to do that in River.

[MaxHalford]

Ok very cool, good luck with that. Please let us know if you make other findings.

Our midterm plan is to implement the AMF methods in Rust. We will provide a Python wrapper. They are consistently more accurate than other methods, at least on stationary data, so we think the ecosystem deserves to have a very fast implementation :)