Implement Bayesian logistic regression

[MaxHalford]

Motivation

linear_model.BayesianLinearRegression maintains a full Gaussian posterior over the weights, which gives calibrated predictive uncertainty (predict_one(..., with_dist=True)) and makes it a natural fit for downstream uses like bandit.LinUCB. There is no classification counterpart yet.

A streaming Bayesian logistic regression would provide the same benefits for binary classification: a posterior over the weights, well-calibrated probabilities, and uncertainty estimates usable for Thompson-sampling / UCB-style bandits.

Approach

The logistic likelihood is non-Gaussian, so there's no exact conjugate update. The usual streaming options:

• Laplace approximation / Online Newton — maintain a Gaussian posterior, update mean + precision per observation using the gradient and Hessian of the log-likelihood (cf. the approach behind linear_model.AdPredictor, which already keeps a Gaussian belief per weight via probit ADF).
• Assumed-density filtering (ADF) — moment-match after each observation (this is essentially what AdPredictor does for the probit link).
• Variational updates (e.g. Jaakkola–Jordan bound) for a logit link.

Worth surveying how this relates to the existing AdPredictor (probit, diagonal covariance) before deciding on the link function and covariance structure (diagonal vs full).

Notes

• Should implement Classifier/MiniBatchClassifier and expose calibrated predict_proba_one, ideally with access to predictive uncertainty.
• Reuse the NumPy-backed state + Sherman-Morrison machinery from the recently reworked BayesianLinearRegression where applicable.
• Related: BayesianLinearRegression learn_many work in Add learn_many to BayesianLinearRegression #1916.

[DagaBhai]

Hi, can I work on this?

[MaxHalford]

Knock yourself out, but be careful, it's a difficult one ;)

[DagaBhai]

Got it, thanks! I'm working on this to learn, so I'll take my time to figure it out and open a PR when it's done. Hopefully, there's no immediate rush on this one.

[Whatsonyourmind]

One thing worth pinning down in the API contract before implementation: ADF / Laplace posteriors for the logit tend to run over-confident, so a "calibrated predict_proba_one" shouldn't be asserted by construction. A tiny held-out reliability check (ECE / a reliability curve) in the test suite would let the calibration claim be verified rather than assumed.

The predictive-uncertainty output is also exactly what makes this drop-in for LinUCB / Thompson downstream, so a test that the posterior variance actually widens out-of-distribution (rather than collapsing) would be a nice guard — that's the property the bandit layer relies on and the one that's easiest to lose silently in an online approximation. Happy to share a minimal ECE snippet if it helps whoever picks it up.