Ifpack2 btds gemm

[lucbv]

Pushing some preliminary changes based of the previous work by @yasahi-hpc
The goal is to isolate the failure that ensue in Trilinos' BTDS solver.
The current changes are sufficient to make the BTDS fail which might indicate that the underlying issue is a bad usage of Kokkos View?

[yasahi-hpc]

@lucbv

Thanks a lot for your help.
I think we can update the extent computation as attempted in #2667
I am not sure the correct (or expected) behavior for the stride computation

v.stride(r) for r >= View::rank()

[yasahi-hpc]

Suggested change

	if (r == 0) {
	int V_extent_0 = V_rank == 0 ? 0 : v.extent_int(0);
	return V_extent_0;
	} else if (r == 1) {
	int V_extent_1 = V_rank == 0 ? 0 : V_rank == 1 ? 1 : v.extent_int(1);
	return V_extent_1;
	} else {
	return -1;
	}
	}
	if (r == 0) {
	int V_extent_0 = V_rank == 0 ? 1 : v.extent_int(0);
	return V_extent_0;
	} else if (r == 1) {
	int V_extent_1 = V_rank <= 1 ? 1 : v.extent_int(1);
	return V_extent_1;
	} else {
	return 1;
	}

I think extent would be computed like this.

[ndellingwood]

I tested this PR with/without Yuichi's suggestion, gcc/11.3, OpenMPI/4.1.5 Serial build with the new View enabled, I'm still seeing the test failures:

[ndellin@blake01 build]$ ctest -R Ifpack2_BlockTriDiContainer_
Test project /home/ndellin/trilinos/Trilinos-1/Build/Blake-serial-gcc-1130-ompi-415/build
    Start 700: Ifpack2_BlockTriDiContainer_BTD_UnitTests_MPI_4
1/2 Test #700: Ifpack2_BlockTriDiContainer_BTD_UnitTests_MPI_4 ......***Failed  Required regular expression not found. Regex=[End Result: TEST PASSED
]  0.94 sec
    Start 701: Ifpack2_BlockTriDiContainer_Jacobi_UnitTests_MPI_4
2/2 Test #701: Ifpack2_BlockTriDiContainer_Jacobi_UnitTests_MPI_4 ...***Failed  Required regular expression not found. Regex=[End Result: TEST PASSED
]  1.05 sec

0% tests passed, 2 tests failed out of 2

Subproject Time Summary:
Ifpack2    =   7.99 sec*proc (2 tests)

Total Test time (real) =   2.57 sec

The following tests FAILED:
	700 - Ifpack2_BlockTriDiContainer_BTD_UnitTests_MPI_4 (Failed)
	701 - Ifpack2_BlockTriDiContainer_Jacobi_UnitTests_MPI_4 (Failed)
Errors while running CTest

Tested with:
kokkos/kokkos@4747e23
kernels with PR #2829 +/- Yuichi's suggestion
trilinos/Trilinos@69d24c6

[yasahi-hpc]

@ndellingwood Thank you for the information
Then, it would be related to the stride computation
Concerning the correct implementation in #2593

The stride computation may be like this?

template <typename ViewType>
KOKKOS_INLINE_FUNCTION std::size_t get_stride(const ViewType &v, const int r) {
  static_assert(Kokkos::is_view_v<ViewType>, "KokkosBatched: ViewType is not a Kokkos::View.");
  constexpr std::size_t V_rank = ViewType::rank();
  static_assert(V_rank <= 2, "KokkosBatched: ViewType must have rank 0, 1 or 2.");

  if (r == 0) {
    std::size_t V_stride_0 = V_rank == 0 ? 1 : v.stride(0);
    return V_stride_0;
  } else if (r == 1) {
    std::size_t V_stride_1 = V_rank <= 1 ? 1 : v.stride(1);
    return V_stride_1;
  } else {
    return 1;
  }
}

[ndellingwood]

Cross-referencing issue #2622

[ndellingwood]

I retested as a sanity check

• Serial build with SKX enabled had the failures, as posted above
• Serial build with SKX disabled passed!
• Cuda build passed!

All jobs above enabled the new View impl and complex types

[ndellingwood]

I rebuilt with @yasahi-hpc suggestion (thank you!) , and the Serial job with SKX still exhibits the failures

[ndellingwood]

I retested Trilinos@develop (containing the 4.7.01 releases as packages) in the Serial build with SKX and the legacy View, and the Ifpack2_BlockTriDiContainer* tests failed in that case as well, looks like we encountered a combo of issues

• kokkos-kernels@develop: Nightly test failures with Trilinos: various ifpack2 unit test failures #2622 introduced new Ifpack2_BlockTriDiContainer* test failures on e.g. device architectures , which are resolved by this PR (* aside from pre-existing failure configs/arches)
• Trilinos@develop had preexisting test failures in the Ifpack2_BlockTriDiContainer* tests when enabling SKX arch, separate from the changes of Nightly test failures with Trilinos: various ifpack2 unit test failures #2622 ; for past occurrences, see e.g. Ifpack2: Ifpack2_BlockTriDiContainerUnitAndPerfTests_MPI_4 test fails in serial build with intel/19 on SKX trilinos/Trilinos#10207

To move forward, it looks like we need

1. Resolve merge conflicts
2. DCO signoff
3. Whether to include the extent and stride modifications from the comments
4. CI

[yasahi-hpc]

@ndellingwood @lucbv

Thanks a lot for your help.
Glad to hear that tests pass without SKX option.
Let us keep in touch

[lucbv]

@yasahi-hpc fixed the stride and extent formula, should be good to go with all the changes made yesterday and today

[yasahi-hpc]

Thanks for changes

I do not expect these changes make any difference, but it is safer to use the value that is confirmed to work correctly

[lucbv]

Yes, agreed, applying the changes

[yasahi-hpc]

LGTM
Thanks for the fix