ICLRblogpost

On the computation of the Fisher Information in continual learning (2025, ICLR Blogpost)

The code in this repository is used for the experiments reported in the ICLR 2025 blog post "On the computation of the Fisher Information in continual learning" (or see the arXiv-preprint here).

This blog post compares the performance of Elastic Weight Consolidation (EWC) with various different ways of computing the diagonal elements of the Fisher Information matrix. The following options are considered:

• EXACT
  The elements of the Fisher Information are computed exactly. All training samples are used.
  To use this option: ./main.py --ewc --fisher-labels='all'

• EXACT ($n$=500)
  The elements of the Fisher Information are computed exactly. Only 500 training samples are used.
  To use this option: ./main.py --ewc --fisher-labels='all' --fisher-n=500

• SAMPLE
  The elements of the Fisher Information are estimated using a single Monte Carlo sample. All training samples are used.
  To use this option: ./main.py --ewc --fisher-labels='sample'

• EMPIRICAL
  The empirical Fisher Information is used. All training samples are used.
  To use this option: ./main.py --ewc --fisher-labels='true'

• BATCHED ($b$=128)
  The empirical Fisher Information is approximated using mini-batches (see blog post for details).
  To use this option: ./main.py --ewc --fisher-labels='true' --fisher-batch=128

To run the experiments from the blog post, the following lines of code can be used:

python3 ICLRblogpost/compare_FI.py --seed=1 --n-seeds=30 --experiment=splitMNIST --scenario=task
python3 ICLRblogpost/compare_FI.py --seed=1 --n-seeds=30 --experiment=CIFAR10 --scenario=task --reducedResNet --iters=2000 --lr=0.001

Citation

If this is useful, please consider citing the blog post:

@inproceedings{vandeven2025fisher,
  title={On the computation of the {F}isher {I}nformation in continual learning},
  author={van de Ven, Gido M},
  booktitle={ICLR Blogposts 2025},
  year={2025},
  date={April 28, 2025},
  url={https://iclr-blogposts.github.io/2025/blog/fisher/}
}

Acknowledgments

This project has been supported by a senior postdoctoral fellowship from the Resarch Foundation -- Flanders (FWO) under grant number 1266823N.