Skip to content

Add boundary integral support (BdIntegral) with PETSc ghost facet fix#70

Open
lmoresi wants to merge 7 commits intodevelopmentfrom
feature/boundary-integrals
Open

Add boundary integral support (BdIntegral) with PETSc ghost facet fix#70
lmoresi wants to merge 7 commits intodevelopmentfrom
feature/boundary-integrals

Conversation

@lmoresi
Copy link
Member

@lmoresi lmoresi commented Mar 3, 2026

Summary

BdIntegral API

# Boundary length (2D) or area (3D)
length = uw.maths.BdIntegral(mesh, fn=1.0, boundary="Top").evaluate()

# Flux through boundary using surface normal
flux = uw.maths.BdIntegral(mesh, fn=v_soln.dot(mesh.Gamma_N), boundary="Top").evaluate()

# Internal boundaries work out of the box
circ = uw.maths.BdIntegral(mesh, fn=1.0, boundary="Internal").evaluate()

Uses PETSc's DMPlexComputeBdIntegral via a new C wrapper (UW_DMPlexComputeBdIntegral) in petsc_compat.h. Follows the same pattern as the existing Integral class.

PETSc ghost facet fix

DMPlexComputeBdResidual_Internal and DMPlexComputeBdIntegral construct facetIS as all facets at depth dim-1 but do not exclude ghost facets (SF leaves). Unlike the volume residual code (which checks the ghost label), the boundary residual path has no ghost check. For internal boundaries at partition junctions, shared facets are integrated on multiple ranks, causing double-counting.

The patch filters ghost facets using ISDifference after obtaining facetIS, following the same pattern used by DMPlexComputeBdFaceGeomFVM in PETSc's own FVM code. It applies cleanly to PETSc v3.18.0 through v3.24.5.

  • Patch: petsc-custom/patches/plexfem-ghost-facet-fix.patch
  • Auto-applied by build-petsc.sh during custom PETSc builds
  • Upstream MR branch pushed to gitlab.com/lmoresi/petsc (pending GitLab availability)

Test plan

  • 14 boundary integral tests (test_0502_boundary_integrals.py) pass serial
  • External boundary tests pass at np=4
  • Internal boundary tests pass at np=4 (with PETSc patch)
  • Stokes solver with internal boundary natural BCs: serial/parallel agreement to 10+ significant figures at np=1,2,4,8
  • Volume integral tests (test_0501_integrals.py) pass at np=4 (no regression)
  • BoxInternalBoundary has a pre-existing UnboundLocalError bug in parallel (not this PR)

Files changed

File Change
src/underworld3/cython/petsc_compat.h UW_DMPlexComputeBdIntegral C wrapper
src/underworld3/cython/petsc_extras.pxi Cython declaration
src/underworld3/cython/petsc_maths.pyx BdIntegral class
src/underworld3/maths/__init__.py Export BdIntegral
tests/test_0502_boundary_integrals.py 14 tests
petsc-custom/patches/plexfem-ghost-facet-fix.patch PETSc patch
petsc-custom/build-petsc.sh Auto-apply patches
CLAUDE.md AI attribution convention

Underworld development team with AI support from Claude Code

lmoresi added 2 commits March 3, 2026 17:20
Wrap PETSc's DMPlexComputeBdIntegral to provide standalone boundary/surface
integral capability. In 2D these are line integrals over boundary edges;
in 3D they are surface integrals over boundary faces (both codimension-1).

The integrand may reference the outward unit normal via mesh.Gamma / mesh.Gamma_N.
Internal boundaries (e.g. AnnulusInternalBoundary, BoxInternalBoundary) work
out of the box through the same DMLabel infrastructure.

New API:
    bd_int = uw.maths.BdIntegral(mesh, fn=1.0, boundary="Top")
    length = bd_int.evaluate()

Tests cover: constant/coordinate/sympy/mesh-variable integrands, normal-vector
integrands, perimeter checks, BoxInternalBoundary, and AnnulusInternalBoundary
circumference.

Underworld development team with AI support from Claude Code
Use the same team attribution wording for both commits and PR descriptions.
No emoji in PR bodies.

Underworld development team with AI support from Claude Code
Copilot AI review requested due to automatic review settings March 3, 2026 06:34
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds PETSc-backed boundary (surface/line) integration to Underworld3 via a new uw.maths.BdIntegral API, enabling integrals over named external and internal mesh boundaries (Issue #47).

Changes:

  • Introduces BdIntegral in petsc_maths.pyx, compiling scalar integrands (optionally using outward normals) and evaluating via PETSc/DMPlex.
  • Adds a thin C helper UW_DMPlexComputeBdIntegral plus Cython declarations to simplify PETSc’s multi-field boundary-integral interface.
  • Adds a dedicated boundary-integral test suite covering external boundaries and internal boundaries (box + annulus), and exports BdIntegral from underworld3.maths.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/underworld3/cython/petsc_compat.h Adds UW_DMPlexComputeBdIntegral wrapper around PETSc DMPlexComputeBdIntegral.
src/underworld3/cython/petsc_extras.pxi Declares the new C wrapper for use from Cython.
src/underworld3/cython/petsc_maths.pyx Implements BdIntegral class and unit propagation logic.
src/underworld3/maths/__init__.py Exports BdIntegral in the public maths module.
tests/test_0502_boundary_integrals.py Adds test coverage for boundary integrals on external and internal boundaries.
CLAUDE.md Updates AI attribution guidance to include PR descriptions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

cdef PetscInt label_val = boundary_enum.value
cdef PetscInt num_vals = 1

c_label = mesh.dm.getLabel(self.boundary)
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mesh.dm.getLabel(self.boundary) can return None even when boundary exists in mesh.boundaries (e.g., if the DM only has the consolidated UW_Boundaries label). In that case dmlabel.dmlabel will segfault. Consider fetching the stacked label (mesh.dm.getLabel("UW_Boundaries")) and using label_val to select the stratum, or at minimum check c_label is not None and raise a ValueError with a clear message.

Suggested change
c_label = mesh.dm.getLabel(self.boundary)
c_label = mesh.dm.getLabel(self.boundary)
if c_label is None:
# Some meshes store boundary information only in the consolidated
# "UW_Boundaries" label. In that case, fall back to that label
# and use `label_val` to select the appropriate stratum.
c_label = mesh.dm.getLabel("UW_Boundaries")
if c_label is None:
raise ValueError(
f"Boundary label '{self.boundary}' not found on mesh DM, and "
"no 'UW_Boundaries' label is present. Cannot compute boundary integral."
)

Copilot uses AI. Check for mistakes.
Comment on lines +151 to +158
PetscDS ds;
PetscSection section;
PetscInt Nf;

PetscFunctionBeginUser;

PetscCall(DMGetLocalSection(dm, &section));
PetscCall(PetscSectionGetNumFields(section, &Nf));
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PetscDS ds; is declared but never used in UW_DMPlexComputeBdIntegral(). If the build treats warnings as errors (common in CI), this can fail compilation. Remove the unused variable, or use DMGetDS()/PetscDSGetNumFields() to obtain Nf instead of leaving ds unused.

Copilot uses AI. Check for mistakes.
lmoresi added 2 commits March 3, 2026 20:19
DMPlexComputeBdIntegral (unlike DMPlexComputeIntegralFEM) does not
perform an MPI reduction — it returns only the local process contribution.
Add MPIU_Allreduce in UW_DMPlexComputeBdIntegral to sum across ranks.

Found by testing with mpirun -np 5..8 on AnnulusInternalBoundary.

Note: a pre-existing PETSc internal-facet orientation issue can cause
small errors (~1.5%) on internal boundaries at certain partition counts
(e.g. np=6 with the default annulus mesh). External boundaries are
unaffected. This is upstream of our wrapper.

Underworld development team with AI support from Claude Code
DMPlexComputeBdIntegral iterates over ALL label stratum points including
ghost facets, so shared internal boundary facets get integrated on both
the owning rank and the ghost rank. External boundaries are unaffected
because external facets have only one supporting cell (no ghosting).

The fix creates a temporary DMLabel containing only owned (non-ghost)
boundary points by checking against the PetscSF leaf set, then passes
this filtered label to DMPlexComputeBdIntegral.

Also restructures tests to use lazy initialization for BoxInternalBoundary
(which has a pre-existing MPI bug) so it doesn't block other tests under
mpirun.

Verified at np=1,5,6,7,8 — all produce identical results.

Underworld development team with AI support from Claude Code
@lmoresi lmoresi changed the title Add boundary integral support (BdIntegral) Add boundary integral support (BdIntegral) — closes #47 Mar 4, 2026
DMPlexComputeBdResidual_Internal and DMPlexComputeBdIntegral do not
exclude ghost facets (SF leaves) from the facet IS, unlike the volume
residual code which checks the ghost label. For internal boundaries
at partition junctions, this causes shared facets to be integrated on
multiple ranks, producing O(1) errors after MPI summation.

The patch filters ghost facets using ISDifference, following the same
pattern used by DMPlexComputeBdFaceGeomFVM for FVM face flux. It
applies cleanly to PETSc v3.18.0 through v3.24.5.

Includes:
- petsc-custom/patches/plexfem-ghost-facet-fix.patch
- build-petsc.sh integration (auto-applies after clone)
- MR description for upstream PETSc submission

Upstream MR branch pushed to gitlab.com/lmoresi/petsc
(fix/plexfem-ghost-facet-boundary-residual)

Underworld development team with AI support from Claude Code
@lmoresi lmoresi force-pushed the feature/boundary-integrals branch from 28a06bb to 4f76375 Compare March 4, 2026 09:27
@lmoresi lmoresi changed the title Add boundary integral support (BdIntegral) — closes #47 Add boundary integral support (BdIntegral) with PETSc ghost facet fix Mar 4, 2026
lmoresi added 2 commits March 4, 2026 21:24
PETSc changed DMPlexComputeBdIntegral from void (*func)(...) to
void (**funcs)(...) in v3.22.0. CI uses PETSc 3.21.5 (pinned in
environment.yaml), so the wrapper must handle both signatures.

Underworld development team with AI support from Claude Code
…or bug)

Underworld development team with AI support from Claude Code
@lmoresi
Copy link
Member Author

lmoresi commented Mar 4, 2026

This is a long-standing request from @jcgraciosa and @gthyagi to add surface integral calculations to uw3. There is a PETSC "bug" (unanticipated use-case) that does not handle internal surfaces correctly - we have seen this in calculating kernels and other analytical benchmarks (@gthyagi) and there is a potential fix for this that is currently added to the build script as a patch when compiling PETSc. This is also pushed to the PETSc gitlab as a merge-request, so we might get this fixed and released eventually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants