PyMC v5 Sampling Performance Issues

[jbrande]

Sampling performance when using PyMC v5 compared to PyMC3 seems to be significantly degraded, both in sampling speed and accuracy. Running the transit.ipynb tutorial in separate conda environments with each version shows this.

PyMC3 Results:
Sampler:

Arviz Summary:

Corner plot:

Compare to the PyMC5 Results:
Sampler:

Arviz Summary:

Corner plot:

I have no idea why this might be the case, and it'd be nice to be able to use current versions of PyMC for various applications.

[jiayindong]

Did the two runs use the same initialization, i.e., initvals and init?

[jbrande]

Those runs didn't, but I'd seen broadly internally consistent behavior over several different runs.

I just set up new runs to test this with the same initial seeds, fake transit data, fake parameters, etc, and I'm still getting the same behavior. Even copying the sampler seeds and initial map_soln values between the versions, I still get the same behavior as above.

Here's the example notebooks I'm working with (remove the .txt suffixes to run):

pymc3_test.ipynb.txt
pymc5_test.ipynb.txt

[jbrande]

And here's the environment information for the separate conda environments where I've separately installed both versions of PyMC.

conda_env_pymc3.txt
conda_env_pymc5.txt

[TylerFair]

While I can't help with the sampling speed, The problem of getting nonsensical outputs is (likely) due to these two factors:

for your test notebooks you have a prior on r as:
r = pm.Uniform( "r", lower=0.01, upper=0.1, shape=2)
with true values of radii = np.array([0.1, 0.05]), planet b's prior is on the upper boundary and you won't get convergence.

Another thing is that for transit modelling, I find pymc5 sampling to only work with the following setting:
pm.Sample(...., init='adapt_full'). In my personal experience, the default init method for pymc5 does not initialize the chains well for these problems (note: this won't work for every problem, but specifically does for transit modelling). See attached.

pymc5_test_fixed.ipynb.txt

[jiayindong]

Agree with @TylerFair's suggestions. Maybe try the 'adapt_full' init to see if that makes a difference.

[twiecki]

Sampling speed is really not the issue here but convergence. I tried the nutpie sampler with low-rank on the pymc5 version of the model but it fails completely and shows multi-modality in the posterior.

My strong hunch is that there is something quite wrong with the model specification. They are implemented the same between the two versions so that could mean the bug is in the exoplanet functions that are getting used?

Have people seen convergence issues in other places after the upgrade?

[vandalt]

Hi!

I contributed to the pymc5 update and transit is the part of the code I am least familiar with, so definitely don't rule out me making a mistake in the internals. However, here are a few hopefully useful notes:

• There is a pull request (Update case studies to pymc v5 case-studies#78) updating the case studies to PyMC5 and the transit example is here
• When I run it locally it seems to be working fine: the results are the same as with PyMC 3, from what I can tell
• Sampling speed does seem a bit slower. There is a note in the PR Update case studies to pymc v5 case-studies#78 mentioning slower sampling speed, along with some potential pointers. But in all cases the results appear to be the same as PyMC3, just slower.
• I also agree with @TylerFair that the true radius values vs prior probably explain the convergence issues in your example. In the case-studies version the data is generated in the model specification so the so they are set to initval (previously testval).