Tests fail on PowerPC (Big-endian): on 32-bit totally broken, on 64-bit some

[barracuda156]

After spending half of the day to fix the build, which finally worked, turned out that most of tests fail on the same spot:

/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/test/support/src/helpers.cc:134: FAILED:
  REQUIRE( rc == (0) )
with expansion:
  -1 == 0
with message:
  [TileDB::Array] Error: [TileDB::StorageManager] Error: FilterCreate:
  Deserialization error; not enough data in buffer for metadata

Any idea what goes wrong and how to fix it? This on Darwin PPC.

P. S. Usual suspects are endianness, 32-bitness and 4-byte bool.

tiledb_test_10.6_ppc.log

[barracuda156]

So this fails:

shared_ptr<tiledb::sm::Filter> tiledb::sm::FilterCreate::deserialize(
    Deserializer& deserializer,
    const EncryptionKey& encryption_key,
    const uint32_t version) {
  Status st;
  uint8_t type = deserializer.read<uint8_t>();
  FilterType filtertype = static_cast<FilterType>(type);
  uint32_t filter_metadata_len = deserializer.read<uint32_t>();
  if (deserializer.size() < filter_metadata_len) {
    throw StatusException(
        "FilterCreate",
        "Deserialization error; not enough data in buffer for metadata");
  }

But why?

[barracuda156]

@Shelnutt2 @joe-maley Could someone take a look please? Presumably 32-bit is kinda supported: #1737
The code has references to endianness too, so there should be no major obstacles to fixing PPC.

[barracuda156]

Tests on 10.6.8 x86_64 pass fine, just in case, so it is not an outdated macOS to blame (or my build):

1: All tests passed (25586067 assertions in 1016 test cases)
1: 
1/2 Test #1: tiledb_unit ......................   Passed  1924.12 sec
test 2
    Start 2: test_ci_asserts

2: Test command: /opt/x86_64/var/macports/build/_opt_x86_64_SnowLeopardPorts_databases_tiledb/tiledb/work/build/tiledb/test/ci/test_assert
2: Working Directory: /opt/x86_64/var/macports/build/_opt_x86_64_SnowLeopardPorts_databases_tiledb/tiledb/work/build/tiledb/test/ci
2: Test timeout computed to be: 10000000
2: Assert did not exit!
2: Randomness seeded to: 857524962
2: retval is 0 (0x0) from /opt/x86_64/var/macports/build/_opt_x86_64_SnowLeopardPorts_databases_tiledb/tiledb/work/build/tiledb/test/ci/try_assert
2: ===============================================================================
2: All tests passed (1 assertion in 1 test case)
2: 
2/2 Test #2: test_ci_asserts ..................   Passed    0.04 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) = 1924.17 sec

[Shelnutt2]

Thanks for posting this @barracuda156. To confirm you are working on powerpc for macos with gcc 12?

[barracuda156]

Thanks for posting this @barracuda156. To confirm you are working on powerpc for macos with gcc 12?

Thank you for responding! Yes.
(I got several versions of GCC, if cross-checking will be needed.)

P. S. I ran build and tests on 10.6 x86_64 also with gcc12.

[barracuda156]

@Shelnutt2 I wanted to run tests on 10.6 for i386 to see how 32-bit Intel behaves, but against libc++ even the build fails:

[100%] Linking CXX executable tiledb_unit
Undefined symbols for architecture i386:
  "Catch::StringMaker<std::byte, void>::convert(std::byte)", referenced from:
      Catch::BinaryExpr<std::byte&, std::byte const&>::streamReconstructedExpression(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) const in unit-backwards_compat.cc.o
  "Catch::StringMaker<std::__1::basic_string_view<char, std::__1::char_traits<char> >, void>::convert(std::__1::basic_string_view<char, std::__1::char_traits<char> >)", referenced from:
      Catch::BinaryExpr<std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>::streamReconstructedExpression(std::__1::basic_ostream<char, std::__1::char_traits<char> >&) const in unit-SubarrayPartitioner-sparse.cc.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[7]: *** [test/CMakeFiles/tiledb_unit.dir/build.make:2528: test/tiledb_unit] Error 1
gmake[6]: *** [CMakeFiles/Makefile2:10054: test/CMakeFiles/tiledb_unit.dir/all] Error 2
gmake[5]: *** [CMakeFiles/Makefile2:10087: test/CMakeFiles/check.dir/rule] Error 2

I do not have readily available setup to build for i386 against libstdc++, though it is doable in principle.

[barracuda156]

This could be relevant, perhaps:

:info:build [ 45%] Building CXX object tiledb/CMakeFiles/TILEDB_CORE_OBJECTS.dir/sm/fragment/fragment_metadata.cc.o
:info:build cd /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/build/tiledb/tiledb && /opt/local/bin/g++-mp-12 -DSPDLOG_COMPILED_LIB -DTILEDB_CORE_OBJECTS_EXPORTS -DTILEDB_STATS -D_FILE_OFFSET_BITS=64 -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/.. -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/../tiledb/sm/c_api -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/../external/include -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/../external/include/bitshuffle -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/../external/include/blosc -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/external/blosc/include -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/build/tiledb/tiledb/.. -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/../tiledb/sm/cpp_api -I/opt/local/include -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/build/externals/install/include -I/opt/local/libexec/openssl3/include -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/build/tiledb/tiledb -pipe -I/opt/local/libexec/openssl3/include -Os -DNDEBUG -I/opt/local/libexec/openssl3/include -isystem/opt/local/include/LegacySupport -isystem/opt/local/include -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17 -arch ppc -isysroot /Developer/SDKs/MacOSX10.6.sdk -mmacosx-version-min=10.6 -fPIC -fvisibility=hidden -Wall -Wextra -Wno-literal-suffix -MD -MT tiledb/CMakeFiles/TILEDB_CORE_OBJECTS.dir/sm/fragment/fragment_metadata.cc.o -MF CMakeFiles/TILEDB_CORE_OBJECTS.dir/sm/fragment/fragment_metadata.cc.o.d -o CMakeFiles/TILEDB_CORE_OBJECTS.dir/sm/fragment/fragment_metadata.cc.o -c /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc
:info:build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc: In member function 'T tiledb::sm::FragmentMetadata::get_tile_min_as(const std::string&, uint64_t) [with T = std::basic_string_view<char>; std::string = std::basic_string<char>; uint64_t = long long unsigned int]':
:info:build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc:1721:18: warning: narrowing conversion of 'size' from 'long long unsigned int' to 'std::basic_string_view<char>::size_type' {aka 'long unsigned int'} [-Wnarrowing]
:info:build  1721 |     return {min, size};
:info:build       |                  ^~~~
:info:build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc:1725:38: warning: narrowing conversion of 'size' from 'long long unsigned int' to 'std::basic_string_view<char>::size_type' {aka 'long unsigned int'} [-Wnarrowing]
:info:build  1725 |     return {static_cast<char*>(min), size};
:info:build       |                                      ^~~~
:info:build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc: In member function 'T tiledb::sm::FragmentMetadata::get_tile_max_as(const std::string&, uint64_t) [with T = std::basic_string_view<char>; std::string = std::basic_string<char>; uint64_t = long long unsigned int]':
:info:build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc:1794:18: warning: narrowing conversion of 'size' from 'long long unsigned int' to 'std::basic_string_view<char>::size_type' {aka 'long unsigned int'} [-Wnarrowing]
:info:build  1794 |     return {max, size};
:info:build       |                  ^~~~
:info:build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_databases_tiledb/tiledb/work/TileDB-2.15.2/tiledb/sm/fragment/fragment_metadata.cc:1798:38: warning: narrowing conversion of 'size' from 'long long unsigned int' to 'std::basic_string_view<char>::size_type' {aka 'long unsigned int'} [-Wnarrowing]
:info:build  1798 |     return {static_cast<char*>(max), size};
:info:build       |                                      ^~~~

[barracuda156]

@Shelnutt2 Have you had a chance to take a look at this?

[barracuda156]

@Shelnutt2 I have tested tiledb on ppc64 now, and while several test cases still fail, results are much better:

1: ===============================================================================
1: test cases:     300 |     293 passed | 7 failed
1: assertions: 3166930 | 3166922 passed | 8 failed
1: 
1/2 Test #1: tiledb_unit ......................***Exception: SegFault 70.77 sec

So it looks like for the large part the issue is bitness (or maybe size of bool too), though somewhere also endianness.

[barracuda156]

Judging by this commit, it is broken now on Linux for 32-bit too (any arch): https://git.alpinelinux.org/aports/commit/?id=19ea0281c8c50d67d634d3559da0bf27bca04115

[barracuda156]

@sunpoet Could you please comment on the status on FreeBSD? I know it supports ppc (32-bit) at least in FreeBSD 13 and earlier. Does the test suite pass or everything is as broken as on macOS?

[barracuda156]

@eddelbuettel Just in case, do you happen to still have a Solaris Big-endian system? It would be nice to have this working, but unfortunately it is not clear to me what exactly is broken.

I am pretty sure the problem is not specific to macOS (i.e. neither SDK-specific nor PowerPC-specific, at least for the most part). There may be an additional complication on ppc due to 4-byte bool size, but I do not think that is a cause of the failures.
If we could have endianness fixed and 32-bit platforms working (any arch on any Unix-like OS), which in itself gonna benefit Linux and *BSD systems, I could fix whatever is macOS-specific.

[eddelbuettel]

@barracuda156 The Solaris system R used was at Oxford University but is not longer up (as you may know) as it was IIRC years past end-of-live support. I have no access to another one.