[Do Not Merge] Troubleshooting the Homebrew installs

[RRosio]

No description provided.

[github-actions]

👈 Launch a Binder on branch RRosio/notebook/troubleshoot_homebrew

[defjaf]

@RRosio Thanks for looking into this! Do the CI macOS machines run on intel?

If so, it would still be worthwhile explicitly calling /usr/local/bin/python3 in the CI script. It would then be interesting to see if the problem still appears — I think then it depends on what the equivalent to /opt/homebrew vs /opt/homebrew/opt/python@3.11/ under intel homebrew. If the problem still occurs, that implies that it's a problem with jupyter and homebrew no matter what architecture, but it's also possible that the /usr/local hierarchy is more standard than anything in `/opt' and so the generic jupyter/hatch/pip machinery correctly picks it up.

I don't have access to an intel macOS machine right now, but I might be able to some time next week.

[RRosio]

Thanks for the feedback @defjaf! Based on the Github Actions docs it appears they do run on Intel.

I will try to explicitly call /usr/local/bin/python3 as you've suggested!

Some additional notes:
I got my hands on an Intel Mac and tried installing JupyterLab, Notebook and NbClassic with a Homebrew Python install and I had no issues. All three seemed to work well! For reference:

• brew install python@3.11 installs Python in /usr/local/bin/python3.
• inside a JupyterLab notebook sys.prefix was /usr/local/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
• the jupyter files were installed in the /usr/local/share directory which is listed in the data directories when running jupyter --paths

I noticed that when working with the Apple Silicon Mac Homebrew Python installation
while Jupyter expects to find for example, the jupyter data files, in /opt/homebrew/Cellar/python@3.11/../../../../3.11/share/jupyter they are actually placed in /opt/homebrew/share/jupyter which is not one of the paths listed in jupyter --paths. And while /opt/homebrew/opt/python@3.11/ is listed, it doesn't seem to be used..? I read that the /opt/homebrew/opt should be providing symbolic links to the latest versions of packages installed in the /opt/homebrew/Cellar directory, but checking with ls -l doesn't indicate any symbolic links exist..

[defjaf]

@RRosio writes:

the jupyter files were installed in the /usr/local/share directory which is listed in the data directories when running jupyter --paths
I noticed that when working with the Apple Silicon Mac Homebrew Python installation
while Jupyter expects to find for example, the jupyter data files, in /opt/homebrew/Cellar/python@3.11/../../../../3.11/share/jupyter they are actually placed in /opt/homebrew/share/jupyter which is not one of the paths listed in jupyter --paths. And while /opt/homebrew/opt/python@3.11/ is listed, it doesn't seem to be used..? I read that the /opt/homebrew/opt should be providing symbolic links to the latest versions of packages installed in the /opt/homebrew/Cellar directory, but checking with ls -l doesn't indicate any symbolic links exist.

Yes, this seems to be the problem (or perhaps more than one):

• homebew python pip is installing share/jupyter and etc/jupyter into /opt/homebrew/ (Apple silicon) or /usr/local (intel)
  • Does this make sense? Should they be going to into directories below sys.prefix which are more version-specific than these?
• jupyter --paths has /usr/local/share and /usr/local/etc on both Apple silicon and intel homebrew, but no /opt/homebrew/share or /opt/homebrew/etc.

How does the homebrew/hatch/pip/jupyter combination decide on the location for data files? The hatch docs say that such shared-data are installed "usually under sys.prefix", but that is not happening here, presumably on purpose, but perhaps with incorrect logic. Something is going wrong somewhere, but it's not clear if the problem is in where they are being stored, or where they are being searched.

[defjaf]

I can confirm that everything works fine on macOS with homebrew on Intel as well.

It does seem that (most of?) the problem is that the shared-data files are going into /opt/homebrew/share/jupyter and /opt/homebrew/etc/jupyter but those are not added to jupyter --paths. (Should they be in addition to, not instead of, /usr/local/{etc,share}/jupyter?)

I'm not completely sure that this is 100% of the problem, since the "schema"-related console output still appears even with

JUPYTER_PATH=/opt/homebrew/share/jupyter JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter jupyter notebook

[defjaf]

Another set of data points regarding running the notebook in a virtual environment on macOS with homebrew python (python3 -m venv .venv followed by the relevant pip install in the venv):

• If you explicitly activate the venv, and then run jupyter notebook, or python -m jupyter notebook, there are no problems: the full UI seems to appear (including all menu items), and no "schema" console errors.
• If you do not activate the venv, but run jupyter directly with .venv/bin/jupyter notebook, there are no problems.
• If you do not activate the venv, but run jupyter through the venv python, i.e., with .venv/bin/python -m jupyter notebook, the UI and "schema" errors do appear.

This last bullet is surprising. What is the difference between jupyter notebook and python -m jupyter notebook?

[RRosio]

It does seem that (most of?) the problem is that the shared-data files are going into /opt/homebrew/share/jupyter and /opt/homebrew/etc/jupyter but those are not added to jupyter --paths. (Should they be in addition to, not instead of, /usr/local/{etc,share}/jupyter?)

I'm not completely sure that this is 100% of the problem, since the "schema"-related console output still appears even with

JUPYTER_PATH=/opt/homebrew/share/jupyter JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter jupyter notebook

I'd like to add here that I tried modifying the code in jupyter_core which gathers the paths for the data and config files, but that did not seem to fix the issue. I will try to make sure that I am making all the changes necessary here.
I also noticed that for JupyterLab, there is a function that checks for a system install of Jupyterlab in the /usr/local directory. However, my attempts at checking for a system install of Jupyterlab in the /opt/homebrew directory have not been successful in mitigating the issue.

This last bullet is surprising. What is the difference between jupyter notebook and python -m jupyter notebook?

This is something I will have to look into more as I am not sure about the difference. In trying this myself, I received errors with both commands, jupyter notebook and python -m jupyter notebook.

[jtpio]

What is the difference between jupyter notebook and python -m jupyter notebook?

You can start the notebook application with jupyter notebook as the command is defined here:

notebook/pyproject.toml

Line 43 in 76db867

jupyter-notebook = "notebook.app:main"

Or with python -m notebook which corresponds to running the __main__.py file:

notebook/notebook/__main__.py

Lines 4 to 6 in 76db867

	from notebook.app import main
	sys.exit(main())

[RRosio]

You can start the notebook application with jupyter notebook as the command is defined here:

notebook/pyproject.toml

Line 43 in 76db867

jupyter-notebook = "notebook.app:main"

Or with python -m notebook which corresponds to running the __main__.py file:

notebook/notebook/__main__.py

Lines 4 to 6 in 76db867

	from notebook.app import main
	sys.exit(main())

Thank you @jtpio for the clarification!

Is it then that we should expect that there would be no difference when running the two commands? My thinking was that the jupyter notebook command that is being executed as .venv/bin/jupyter notebook would be tied to the python that was installed in the virtual environment, presumably, .venv/bin/python. Is that maybe not the case?

[jtpio]

My thinking was that the jupyter notebook command that is being executed as .venv/bin/jupyter notebook would be tied to the python that was installed in the virtual environment, presumably, .venv/bin/python. Is that maybe not the case?

This is what I would also expect, although not entirely sure. Maybe leaving a print(sys.executable) somewhere can help know more about this.

[defjaf]

I've been out of contact for a few weeks; is there anything that I (a motivated and enthusiastic amateur with no direct knowledge of the jupyter-* codebase, but many years with python in general) can do to help with this set of issues?

[defjaf]

I've been looking a little more carefully at jupyter_core/paths.py and in particular the functions jupyter_core.paths.jupyter_config_path() and jupyter_core.paths.jupyter_path(). These clearly have /usr/local hard-coded, which could possibly replaced with /opt/homebrew on Apple Silicon.

But I think the underlying problem is jupyter_core.paths.ENV_CONFIG_PATH and jupyter_core.paths.ENV_JUPYTER_PATH both of which use sys.prefix which in turn is (something like) /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11. I don't think this is wrong, except that some tools seem to be putting files in /opt/homebrew/etc and /opt/homebrew/share instead of this, even without this location appearing anywhere in jupyter_core/paths.py.

So note that there are two issues which might be separate:

1. jupyter_core does not see /opt/homebrew/ by default. This on its own should be easy/possible to fix.
2. Some tools are nonetheless putting files into /opt/homebrew/{etc,share} by default on this platform. I have not yet made it to the bottom of this. Where is it even picking up this location? /opt/homebrew is not hard-coded anywhere in the repo, and it doesn't appear to be the output of anything except the full sys.prefix above. (Could it be that some parts of jupyter are using platformdirs to get information about paths?)

(Possibly relevant aside: this investigation prompted me to try setting JUPYTER_PLATFORM_DIRS=1 which seems to help -- /opt/homebrew/share now appears in jupyter --paths, although I see that share appears in both data and config where perhaps the latter should be /opt/homebrew/etc/jupyter? In any case, some of the UI issues remain in the notebook -- no "log out" or "shutdown" -- and I see an error message which I think has something to do with ipyparallel: "IPP Extension Server Error. Failed to list clusters: might the server extension not be installed/enabled?")

[defjaf]

(I'm also not sure if this is the best/right place for this discussion...)

[jtpio]

There is also the recent jupyter/jupyter_core#364 which could be related?

[defjaf]

There is also the recent jupyter/jupyter_core#364 which could be related?

Possibly, for the non-default JUPYTER_PLATFORM_DIRS=1 setting, or if platformdirs is being used by some jupyter apps. On the other hand, the default macOS filesystem setup is case-preserving but not case-sensitive so I'm not sure it really matters.

[defjaf]

Yet more information (but still no solutions) at #6974 (comment).

TL;DR: the "schemas" issue is due to a call with the incorrect schemas_dir, but locally fixing that still leaves the other issues unsolved unless you still set either

• JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter JUPYTER_PATH=/opt/homebrew/share/jupyter or
• JUPYTER_CONFIG_PATH=/opt/homebrew/etc/jupyter JUPYTER_PLATFORM_DIRS=1

[defjaf]

FWIW, and as expected, all of these errors and misbehaviours vanish for the python.org framework build (of 3.12, but I assume more generally), without requiring any special environment variables or other changes. This still lets me install packages by pip and not be wedded to using a venv, so I think I am moving back to this infrastructure from homebrew, at least for now.