-
Notifications
You must be signed in to change notification settings - Fork 1
Add EOPF JupyterHub Dask Docker image #44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Yuvraj198920
wants to merge
4
commits into
main
Choose a base branch
from
feature/issue-43-eopf-jupyterhub-dask
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+193
−0
Open
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
4df4dd8
Add Dockerfile for EOPF JupyterHub Dask image
Yuvraj198920 bd5ad2b
Add environment.yml for package reference
Yuvraj198920 f9be212
Add documentation for eopf-jupyterhub-dask image
Yuvraj198920 cda9e68
Fix PYTHONPATH to access EOPF packages and update main README
Yuvraj198920 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,34 @@ | ||
| # eopf-container-images | ||
| Container Images used in the EOPF Sentinel Zarr Samples | ||
|
|
||
| ## Available Images | ||
|
|
||
| ### eopf-jupyterhub-dask | ||
| Docker image based on the official EOPF CPM image, compatible with JupyterHub and configured for Dask cluster operations. Designed for SAFE to Zarr conversion workflows. | ||
|
|
||
| **Location**: `eopf-jupyterhub-dask/` | ||
|
|
||
| **Base Image**: `registry.eopf.copernicus.eu/cpm/eopf-cpm:2-6-3` | ||
|
|
||
| **Key Features**: | ||
| - JupyterHub single-user server support | ||
| - JupyterLab with extensions (Git, MyST) | ||
| - Dask and Dask Gateway for distributed computing | ||
| - Ready for SAFE to Zarr conversion | ||
|
|
||
| See [eopf-jupyterhub-dask/README.md](eopf-jupyterhub-dask/README.md) for details. | ||
|
|
||
| ### eopf-jupyterlab | ||
| Custom JupyterLab environment with EOPF dependencies. | ||
|
|
||
| **Location**: `eopf-jupyterlab/` | ||
|
|
||
| ### eopf-dask | ||
| Dask cluster configuration for EOPF. | ||
|
|
||
| **Location**: `eopf-dask/` | ||
|
|
||
| ### eopf-zarr-driver | ||
| Zarr driver utilities and demonstrations. | ||
|
|
||
| **Location**: `eopf-zarr-driver/` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| # Git | ||
| .git | ||
| .gitignore | ||
| .gitattributes | ||
|
|
||
| # Documentation | ||
| README.md | ||
| *.md | ||
|
|
||
| # Python | ||
| __pycache__ | ||
| *.py[cod] | ||
| *$py.class | ||
| *.so | ||
| .Python | ||
| build/ | ||
| develop-eggs/ | ||
| dist/ | ||
| downloads/ | ||
| eggs/ | ||
| .eggs/ | ||
| lib/ | ||
| lib64/ | ||
| parts/ | ||
| sdist/ | ||
| var/ | ||
| wheels/ | ||
| *.egg-info/ | ||
| .installed.cfg | ||
| *.egg | ||
|
|
||
| # IDE | ||
| .vscode/ | ||
| .idea/ | ||
| *.swp | ||
| *.swo | ||
| *~ | ||
|
|
||
| # OS | ||
| .DS_Store | ||
| Thumbs.db |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # Docker Image for EOPF compatible with JupyterHub and Dask Cluster | ||
| # This image performs SAFE to Zarr conversion using the Dask Cluster | ||
| # Base image: Official EOPF image from EOPF CPM | ||
|
|
||
| FROM registry.eopf.copernicus.eu/cpm/eopf-cpm:2-6-3 | ||
|
|
||
| LABEL maintainer="EOPF Sample Service" | ||
| LABEL description="EOPF Docker image compatible with JupyterHub and Dask Cluster for SAFE to Zarr conversion" | ||
|
|
||
| # Fix: https://github.com/hadolint/hadolint/wiki/DL4006 | ||
| # Fix: https://github.com/koalaman/shellcheck/wiki/SC3014 | ||
| SHELL ["/bin/bash", "-o", "pipefail", "-c"] | ||
|
|
||
| USER root | ||
|
|
||
| # Install system dependencies if needed | ||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends \ | ||
| git \ | ||
| curl \ | ||
| && apt-get clean -y \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # Install required Python packages using pip | ||
| # These packages are required for JupyterHub compatibility and Dask operations | ||
| # Note: dask, dask-gateway, and distributed are already in the base image installed for dask user | ||
| # Install system-wide so they're available for all users | ||
| RUN pip install --no-cache-dir \ | ||
| jupyterhub \ | ||
| jupyterlab \ | ||
| jupyterlab-myst \ | ||
| jupyterlab-git \ | ||
| nbgitpuller \ | ||
| dask-labextension | ||
|
|
||
| # Add dask user's local packages to Python path so EOPF packages are accessible | ||
| ENV PYTHONPATH="/home/dask/.local/lib/python3.11/site-packages:${PYTHONPATH}" | ||
|
|
||
| # Expose JupyterLab port | ||
| EXPOSE 8888 | ||
|
|
||
| # Default command (will be overridden by JupyterHub) | ||
| CMD ["jupyterhub-singleuser"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| # EOPF JupyterHub Dask Image | ||
|
|
||
| ## Overview | ||
|
|
||
| This Docker image is designed for performing SAFE to Zarr conversion using the Dask Cluster. It is based on the official EOPF image and is compatible with JupyterHub deployments. | ||
|
|
||
| ## Base Image | ||
|
|
||
| - **Base**: `registry.eopf.copernicus.eu/cpm/eopf-cpm:2-6-3` | ||
| - Official EOPF image from the EOPF CPM registry | ||
|
|
||
| ## Features | ||
|
|
||
| - **JupyterHub Compatible**: Includes `jupyterhub-singleuser` for seamless integration with JupyterHub | ||
| - **JupyterLab**: Full JupyterLab environment with extensions | ||
| - **Dask Support**: Configured with Dask and Dask Gateway for distributed computing | ||
| - **Git Integration**: Includes JupyterLab Git extension and nbgitpuller | ||
| - **MyST Support**: Markdown and MyST support via jupyterlab-myst | ||
|
|
||
| ## Included Packages | ||
|
|
||
| - `jupyterhub-singleuser` - JupyterHub single-user notebook server | ||
| - `jupyterlab` - JupyterLab interface | ||
| - `jupyterlab-myst` - MyST Markdown support | ||
| - `jupyterlab-git` - Git extension for JupyterLab | ||
| - `nbgitpuller` - Pull notebooks from Git repositories | ||
| - `dask` - Parallel computing library | ||
| - `dask-gateway` - Gateway for Dask clusters | ||
| - `distributed` - Distributed computing with Dask | ||
|
|
||
| ## Docker Hub | ||
|
|
||
| The image is available on Docker Hub: | ||
|
|
||
| ```bash | ||
| docker pull yuvraj1989/eopf-jupyterhub-dask:latest | ||
| # or specific version | ||
| docker pull yuvraj1989/eopf-jupyterhub-dask:1.0.0 | ||
| ``` | ||
|
|
||
| **Docker Hub Repository**: https://hub.docker.com/r/yuvraj1989/eopf-jupyterhub-dask | ||
|
|
||
| ## Building the Image | ||
|
|
||
| ```bash | ||
| docker build -t eopf-jupyterhub-dask:latest -f eopf-jupyterhub-dask/Dockerfile . | ||
| ``` | ||
|
|
||
| ## Running the Image | ||
|
|
||
| ### Standalone Mode | ||
|
|
||
| ```bash | ||
| docker run -p 8888:8888 yuvraj1989/eopf-jupyterhub-dask:latest | ||
| ``` | ||
|
|
||
| ### With JupyterHub | ||
|
|
||
| This image is designed to be used with JupyterHub. Configure your JupyterHub to use this image: | ||
|
|
||
| ```python | ||
| c.DockerSpawner.image = 'yuvraj1989/eopf-jupyterhub-dask:latest' | ||
| ``` | ||
|
|
||
| ## Related Issues and PRs | ||
|
|
||
| - Issue: [#43 - Create Docker image with eopf compatible with JupyterHub and Dask Cluster](https://github.com/EOPF-Sample-Service/eopf-container-images/issues/43) | ||
| - Related Issue: https://github.com/EOPF-Sample-Service/eopf-sample-notebooks/issues/185 | ||
| - Related PR: https://github.com/EOPF-Sample-Service/eopf-sample-notebooks/pull/186 | ||
|
|
||
| ## Usage for SAFE to Zarr Conversion | ||
|
|
||
| This image includes all necessary dependencies for performing SAFE to Zarr conversion with Dask. Users can run the conversion notebooks in a JupyterHub environment with Dask cluster support. | ||
|
|
||
| ## Maintenance | ||
|
|
||
| Maintained by: EOPF Sample Service Team | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would remove this part, not relevant for the README
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@clausmichele okay sure, thanks.