Multiple test failures on ppc64le with custom semirings

[strophy]

Describe the bug
Multiple tests are failing on ppc64le arch only under Alpine Linux, an example failure looks like this:

31/75 Test #54: LAGraphX_PageRankGX ..................***Failed    1.43 sec
Test test_ranker...                             karate:   err: 4.012138e-05 (standard), sum(r): 1.000000e+00 iters: 19
[ FAILED ]
  test_PageRankGX.c:48: Check _Generic ((diff), GrB_Matrix : _Generic ((GrB_MINUS_FP32), GrB_Semiring : GrB_Matrix_eWiseAdd_Semiring , GrB_Monoid : GrB_Matrix_eWiseAdd_Monoid , GrB_BinaryOp : GrB_Matrix_eWiseAdd_BinaryOp ), GrB_Vector : _Generic ((GrB_MINUS_FP32), GrB_Semiring : GrB_Vector_eWiseAdd_Semiring , GrB_Monoid : GrB_Vector_eWiseAdd_Monoid , GrB_BinaryOp : GrB_Vector_eWiseAdd_BinaryOp)) (diff, ((void*)0), ((void*)0), GrB_MINUS_FP32, cmatlab, centrality, ((void*)0)) == 0... failed
karate:   err: -inf (Graphalytics), sum(r): 9.999999e-01 iters: 100
status: -1005 msg: LAGraph failure (file /builds/strophy/aports/community/suitesparse/src/SuiteSparse-7.12.1/LAGraph/src/algorithm/LAGr_PageRank.c, line 145): pagerank failed to converge in 2 iterations
west0067: err: 2.117455e-05 (standard), sum(r): 9.999999e-01 iters: 11
west0067: err: 2.076663e-05 (standard), sum(r): 9.999999e-01 iters: 100

=========== ldbc-directed-example, with sink nodes 3 and 9:
Graph: kind: directed, nodes: 10 entries: 17 type: double
  structural symmetry: unknown
  adjacency matrix: GrB_FP64 matrix: 10-by-10 entries: 17
    (0, 2)   0.5
    (0, 4)   0.3
    (1, 3)   0.1
    (1, 4)   0.3
    (1, 9)   0.12
    (2, 0)   0.53
    (2, 4)   0.62
    (2, 7)   0.21
    (2, 9)   0.52
    (4, 2)   0.69
    (4, 3)   0.53
    (4, 7)   0.1
    (5, 2)   0.23
    (5, 3)   0.39
    (6, 3)   0.83
    (7, 0)   0.39
    (8, 3)   0.69
  adjacency matrix transposed: GrB_FP64 matrix: 10-by-10 entries: 17
    (0, 2)   0.53
    (0, 7)   0.39
    (2, 0)   0.5
    (2, 4)   0.69
    (2, 5)   0.23
    (3, 1)   0.1
    (3, 4)   0.53
    (3, 5)   0.39
    (3, 6)   0.83
    (3, 8)   0.69
    (4, 0)   0.3
    (4, 1)   0.3
    (4, 2)   0.62
    (7, 2)   0.21
    (7, 4)   0.1
    (9, 1)   0.12
    (9, 2)   0.52
  out degree: GrB_INT64 vector: n: 10 entries: 8
    (0)   2
    (1)   3
    (2)   4
    (4)   3
    (5)   2
    (6)   1
    (7)   1
    (8)   1

with sinks handled properly:
ldbc-directed: err: 3.109872e-05 (standard), sum(r): 1.000000e+00, niters 12
This is the correct pagerank, with sinks handled properly:
GrB_FP32 vector: n: 10 entries: 10
    (0)   0.16977
    (1)   0.0361532
    (2)   0.167321
    (3)   0.166878
    (4)   0.154096
    (5)   0.0361532
    (6)   0.0361532
    (7)   0.11537
    (8)   0.0361532
    (9)   0.0819525

with sinks handled properly:
ldbc-directed: err: 3.521144e-05 (standard), sum(r): 9.999999e-01, niters 100
This is the correct pagerank, with sinks handled properly:
GrB_FP64 vector: n: 10 entries: 10
    (0)   0.169772
    (1)   0.0361501
    (2)   0.16733
    (3)   0.166874
    (4)   0.154103
    (5)   0.0361501
    (6)   0.0361501
    (7)   0.11537
    (8)   0.0361501
    (9)   0.0819501
FAILED: 1 of 1 unit tests has failed.

It seems that this is caused by an issue in custom JIT semirings on ppc64le arch, because applying a patch like this to use the pre-compiled semiring causes the test to pass:

diff --git a/LAGraph/experimental/algorithm/LAGr_PageRankGX.c b/LAGraph/experimental/algorithm/LAGr_PageRankGX.c
index 87e744965..8924c8cbf 100644
--- a/LAGraph/experimental/algorithm/LAGr_PageRankGX.c
+++ b/LAGraph/experimental/algorithm/LAGr_PageRankGX.c
@@ -156,7 +156,7 @@ int LAGr_PageRankGX
         GRB_TRY (GrB_assign (r, NULL, NULL, teleport + sink_value, GrB_ALL,
             n, NULL)) ;
         // r += A'*w
-        GRB_TRY (GrB_mxv (r, NULL, GrB_PLUS_FP64, LAGraph_plus_second_fp64,
+        GRB_TRY (GrB_mxv (r, NULL, GrB_PLUS_FP64, GxB_PLUS_SECOND_FP64,
             AT, w, NULL)) ;
     }

To Reproduce
Build and run tests on ppc64le

Expected behavior
Tests should pass

Desktop (please complete the following information):

• OS: Alpine 3.22.2
• Compiler: gcc 15.2.0
• BLAS and LAPACK: openblas 0.3.30
• Version: 7.12.1

Additional context
I'm just a package maintainer and don't really know how to debug this further, but from reading some other issues here it seems better to report the issue than patch the tests to use pre-compiled semirings. Help diagnosing this would be appreciated, if necessary please inspect the Alpine merge request showing logs before and after applying a partial patch.

[mmuetzel]

Fwiw, all tests pass for ppc64el when running with qemu (x86_64 host) on Ubuntu 24.04 in the CI:
https://github.com/DrTimothyAldenDavis/SuiteSparse/actions/runs/19304495499/job/55207996833#step:12:8659

        Start 54: LAGraphX_PageRankGX
  54/75 Test #54: LAGraphX_PageRankGX ..................   Passed    2.05 sec
[...]
  100% tests passed, 0 tests failed out of 75

[strophy]

I wonder if it could be related to gcc 15 and/or several OpenBLAS changes that were recently patched into Alpine: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/93836

Upstream refs:
OpenMathLib/OpenBLAS#5448
OpenMathLib/OpenBLAS#5463
OpenMathLib/OpenBLAS#5472

This issue specifically mentions gcc 15 on Fedora: OpenMathLib/OpenBLAS#5372

I suspect the Ubuntu 24.04 CI runner doesn't yet use gcc 15?

[mmuetzel]

I don't know if there is a reason for that. But that runner uses clang to cross-build for the powerpc64le-linux-gnu target.
https://github.com/DrTimothyAldenDavis/SuiteSparse/actions/runs/19304495499/job/55207996833#step:11:39

clang --version
Ubuntu clang version 18.1.3 (1ubuntu1)

Edit: In that run it installed the "libopenblas-openmp-dev:ppc64el (0.3.26+ds-1)" package.

[mmuetzel]

I set up a CI run that uses a chroot to Alpine Linux (latest-stable - I think that is 3.22 currently) using qemu to build and run on ppc64le (on a x86_64 host).

In that configuration, running the tests of LAGraph took quite a while (probably because of the emulation). But all of them passed:
https://github.com/mmuetzel/SuiteSparse/actions/runs/19828219039/job/56807111625#step:11:7775

  100% tests passed, 0 tests failed out of 75
  
  Total Test time (real) = 6609.51 sec

For that test, lapack-dev was installed. So, it might indeed be an issue with the version of OpenBLAS that is distributed by Alpine Linux or the GCC version. (Or emulated vs. native hardware. Or some other configuration difference.)

[strophy]

Thanks so much @mmuetzel for helping narrow this down, I really appreciate it! Do you think I should open an issue with OpenBLAS, given they pushed a number of changes to ppc64le code and were working with GCC devs on bugs recently? It's starting to seem like SuiteSparse test failures are a symptom of some lower level problem than the source of the issue. I'm happy to test debug prints and patches to any of these packages in the Alpine CI environment on native hardware.

[mmuetzel]

Did you check whether the tests pass for you if you build with the reference implementations of BLAS and LAPACK (instead of OpenBLAS)? If they do, it might be something about the OpenBLAS package that is distributed by Alpine Linux.

Afaict, SuiteSparse relies heavily on OpenMP. I seem to recall that Alpine Linux built their OpenBLAS package without OpenMP. (I guess that means that they are using pthreads instead.) Is that still the case?
Mixing OpenMP with pthreads could also be an issue. (I think that was mostly about performance. But maybe it can also affect the resulting values?)
In any case, that sounds like a potential packaging issue in Alpine Linux or some issue in other libraries or the compiler (or its runtime libraries).

[DrTimothyAldenDavis]

GraphBLAS doesn't rely on BLAS and LAPACK, so the failures in LAGraph don't have anything to do with that.

The puzzling thing about this test failure is that the LAGraph_plus_second_fp64 and GxB_PLUS_SECOND_FP64 semirings are identical, except that one is dynamically allocated and the other is a static object. They trigger the identical code paths inside GraphBLAS.

I added this line to LAGraph/experimental/test_PageRankGX, just after the call to LAGraph_Init:

LG_SET_BURBLE (true) ;

which prints diagostic info. I then ran the LAGraph tests with both semirings, with a JIT cache initially empty in both cases. The burble output is identical, and the generated JIT kernels are identical.

So I have no explanation for why one should work and the other should fail, except that a compiler bug might be treating a statically defined object (GxB_PLUS_SECOND_FP64) different from a dynamically allocated one (LAGraph_plus_second_fp64). Their contents are identical and GraphBLAS doesn't tell the difference.

[strophy]

Thanks both for your helpful comments. OpenMathLib/OpenBLAS#5372 mentions differences between gcc versions, so I'll try to see if I can pin it down to different combinations of OpenBLAS 0.3.29, 0.3.30 and 0.3.31 (when it is released) and gcc 14/15 in the Alpine environment when I get time.

[DrTimothyAldenDavis]

It has nothing to do with OpenBLAS. It must be the gcc versions.

GraphBLAS has a JIT, which gets configured at cmake build time to use the same compiler used to compile GraphBLAS. When it runs (via the LAGraph tests in particular) it compiles a suite of JIT kernels and places them in ~/.SuiteSparse. Then when you run the tests again (even after a clean build of all of SuiteSparse) it reuses the previously compiled kernels in ~/.SuiteSparse.

So if you are mixing compilers, it's possible that the *.so files in ~/.SuiteSparse have been compiled with one version of the compiler while GraphBLAS is compiled with another version of the compiler.

So try deleting the ~/.SuiteSparse folder entirely before you run your tests. GraphBLAS will recreate it if it's not there, and then it will recompile all of its JIT kernels.

I haven't seen an issue before when JIT kernels are compiled with a different version of the compiler as compared to the compiler used for GraphBLAS itself. But perhaps that's the issue you're seeing here.