[Issue]: hsa_init()/hsa_shut_down() and PAPI rocp_sdk inconsistent behavior

[systems-assistant]

[Migrated from original issue] ROCm/rocprofiler-sdk#72

Original issue author: @adanalis

Problem Description

The behavior of the PAPI component that interfaces with ROCprofiler-SDK is inconsistent (including a segmetation fault) depending on (A) whether the component calls hsa_init()/hsa_shut_down() at the initialization stage, and (B) whether the driver program links against the static or dynamic PAPI library.

Operating System

Red Hat Enterprise Linux 8.10 (Ootpa)

CPU

AMD Instinct MI300A Accelerator

GPU

4 x AMD Instinct MI300A Accelerator

ROCm Version

6.4.0

ROCm Component

No response

Steps to Reproduce

Download PAPI from my fork and switch to branch 2025.06.rocp_sdk-debug-hsa

https://github.com/adanalis/papi/tree/2025.06.rocp_sdk-debug-hsa

Set the environment variable PAPI_ROCP_SDK_ROOT to the location of rocm (e.g. /opt/rocm-6.4.0)

change into the "src" directory
configure and build PAPI with:
$> ./configure --with-components=rocp_sdk --with-debug=yes
$> make

go into the directory: components/rocp_sdk/tests

run the following commands:

$> ./simple
$> ./simple_dyn
$> export PAPI_ROCP_SDK_SKIP_HSA=1
$> ./simple
$> ./simple_dyn
$> export PAPI_ROCP_SDK_DISPATCH_MODE=1
$> ./simple
$> ./simple_dyn

All execution of "simple", which links against the static lib libpapi.a work fine.
The first execution of simple_dyn, which links against the dynamic libpapi.so segfaults, with the following backtrace.
The second execution of simple_dyn (when the HSA calls are skipped) works fine.
The third execution of simple_dyn, where "dispatch" mode is used, reads zero values.

Backtrace:

#0 0x00007ffff555e52f in raise () from /lib64/libc.so.6
#1 0x00007ffff5531e65 in abort () from /lib64/libc.so.6
#2 0x00007fffe7046318 in google::DumpStackTraceAndExit() [clone .cold.31] () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#3 0x00007fffe74aacba in google::LogMessage::Fail() () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#4 0x00007fffe74abe43 in google::LogMessageFatal::~LogMessageFatal() () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#5 0x00007fffe708cfc3 in rocprofiler::hsa::CounterAQLPacket::CounterAQLPacket(aqlprofile_agent_handle_t, rocprofiler::hsa::CounterAQLPacket::CounterMemoryPool, std::vector<aqlprofile_pmc_event_t, std::allocator<aqlprofile_pmc_event_t> > const&) () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#6 0x00007fffe74240f0 in rocprofiler::aql::CounterPacketConstruct::construct_packet(CoreApiTable const&, AmdExtTable const&) () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#7 0x00007fffe7422593 in rocprofiler::counters::start_agent_ctx(rocprofiler::context::context const*) () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#8 0x00007fffe73f6bfd in rocprofiler::context::start_context(rocprofiler_context_id_t) () from /opt/rocm-6.4.0/lib/librocprofiler-sdk.so
#9 0x00007ffff7f6051a in papi_rocpsdk::start_counting (ctx=0xcba450) at components/rocp_sdk/sdk_class.cpp:652
#10 0x00007ffff7f62b43 in rocprofiler_sdk_start (ctx=0xcba450) at components/rocp_sdk/sdk_class.cpp:1142
#11 0x00007ffff7f5cde5 in rocp_sdk_start (ctx=0xc67dd0, ctl=0xca94c0) at components/rocp_sdk/rocp_sdk.c:360
#12 0x00007ffff7ef6456 in PAPI_start (EventSet=0) at papi.c:2938
#13 0x000000000020398e in main ()

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module version 6.12.12 is loaded

HSA System Attributes

Runtime Version: 1.14
Runtime Ext Version: 1.6
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES

==========
HSA Agents

---

Agent 1

---

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 0
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 130750668(0x7cb18cc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 130750668(0x7cb18cc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 130750668(0x7cb18cc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 130750668(0x7cb18cc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 2

---

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 1
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 1
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131809200(0x7db3fb0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 131809200(0x7db3fb0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 131809200(0x7db3fb0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131809200(0x7db3fb0) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 3

---

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 2
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 2
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131809208(0x7db3fb8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 4

---

Name: AMD Instinct MI300A Accelerator
Uuid: CPU-XX
Marketing Name: AMD Instinct MI300A Accelerator
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 3
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3700
BDFID: 0
Internal Node ID: 3
Compute Unit: 48
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131752936(0x7da63e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 131752936(0x7da63e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 131752936(0x7da63e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131752936(0x7da63e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

---

Agent 5

---

Name: gfx942
Uuid: GPU-f68e1753dc12b715
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 4
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 4096(0x1000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2100
BDFID: 512
Internal Node ID: 4
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Memory Properties: APU
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 177
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

---

Agent 6

---

Name: gfx942
Uuid: GPU-5bd3148dc76b831a
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 5
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 4096(0x1000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2100
BDFID: 16640
Internal Node ID: 5
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Memory Properties: APU
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 177
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

---

Agent 7

---

Name: gfx942
Uuid: GPU-a57d04507c371187
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 6
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 4096(0x1000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2100
BDFID: 33024
Internal Node ID: 6
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Memory Properties: APU
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 177
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

---

Agent 8

---

Name: gfx942
Uuid: GPU-cdcfb00895cff553
Marketing Name: AMD Instinct MI300A
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 7
Device Type: GPU
Cache Info:
L1: 32(0x20) KB
L2: 4096(0x1000) KB
L3: 262144(0x40000) KB
Chip ID: 29856(0x74a0)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2100
BDFID: 49408
Internal Node ID: 7
Compute Unit: 228
SIMDs per CU: 4
Shader Engines: 24
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: TRUE
Memory Properties: APU
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 2048(0x800)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 177
SDMA engine uCode:: 24
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 98647876(0x5e13f44) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 4
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx942:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***

Additional Information

No response

[systems-assistant]

Comment by @ppanchad-amd on 2025-06-16T15:52:54Z

Hi @adanalis. Issue is currently being investigated. Thanks

[systems-assistant]

Comment by @bwelton on 2025-06-16T22:50:39Z

I have explored this and reproduced the issue on my side. When viewing simple_dyn from gdb. I can see that a second version of librocprofiler-sdk is being brought in which is causing the failure (the static variables are not the same between the two). This is from PAPI's dlopen of a librocprofiler-sdk from a different path than the one that is (ultimately) being loaded.

[systems-assistant]

Comment by @bwelton on 2025-06-17T20:30:41Z

Ok, so I was able to track it down. The crash is coming from the behavior of calling hsa_init() then hsa_shutdown() and then attempting to read a sample before hsa has been re-inited by the application (you do this to get the devices). We need an active HSA instance (currently) to read counter data.

There are a couple of options to resolves this:

1. Don't call hsa_shutdown() after you read the devices.
2. Use rocprofiler to get the devices on the system and whether they are ROCM capable (it would show no devices if there are no ROCM capable devices on the system) and use rocprofiler to get the device info. This would avoid having to make any HSA calls within your code (and avoid having to use hsa_init()).
3. Wait until after hsa_init is called by the application to read samples.
4. Don't call hsa_init or hsa_shutdown in the dynamic linked case.

We may need to discuss internally as to what sdk should do in this instance other than crash (likely return a too soon error like we do in the normal case where hsa_init is not called at all).

[systems-assistant]

Comment by @jrmadsen on 2025-06-17T21:02:24Z

I recommend approach #2. rocprofiler-sdk provides all the same information per agent and it will also provide the visibility of the device with respect to HSA, HIP, etc.

[systems-assistant]

Comment by @adanalis on 2025-06-18T15:41:03Z

What is the preferred way to get the agent information from rocprofiler-sdk?
If I try to enumerate the agents on a machine with no AMD devices, I get a segfault. I'm pasting the backtrace below. I have a reproducer, but the form won't let me attach anything other than an image or video.

#0 0x00007ffff60d3a6c in __pthread_kill_implementation () from /lib64/libc.so.6
#1 0x00007ffff6086686 in raise () from /lib64/libc.so.6
#2 0x00007ffff6070833 in abort () from /lib64/libc.so.6
#3 0x00007ffff63e6b21 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /lib64/libstdc++.so.6
#4 0x00007ffff63f252c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#5 0x00007ffff63f2597 in std::terminate() () from /lib64/libstdc++.so.6
#6 0x00007ffff63f27f9 in __cxa_throw () from /lib64/libstdc++.so.6
#7 0x00007fffea903e6a in rocprofiler::agent::(anonymous namespace)::read_topology() [clone .cold] ()
from /apps/rocm/rocm-6.4afar7/lib/librocprofiler-sdk.so
#8 0x00007fffea922d6d in rocprofiler::agent::get_agents() () from /apps/rocm/rocm-6.4afar7/lib/librocprofiler-sdk.so
#9 0x00007fffea92442f in rocprofiler_query_available_agents () from /apps/rocm/rocm-6.4afar7/lib/librocprofiler-sdk.so
#10 0x0000000000203939 in get_GPU_agent_info () at test.cpp:115
#11 0x00000000002039ea in tool_init (
fini_func=0x7fffea956340 <rocprofiler::registration::(anonymous namespace)::invoke_client_finalizer(rocprofiler_client_id_t)>, tool_data=0x2681b0)
at test.cpp:130
#12 0x00007fffea9569dc in std::once_flag::_Prepare_execution::_Prepare_execution<std::call_oncerocprofiler::registration::initialize()::{lambda()#1}(std::once_flag&, rocprofiler::registration::initialize()::{lambda()#1}&&)::{lambda()#1}>(rocprofiler::registration::initialize()::{lambda()#1}&)::{lambda()#1}::_FUN() () from /apps/rocm/rocm-6.4afar7/lib/librocprofiler-sdk.so
#13 0x00007ffff60d6f38 in __pthread_once_slow () from /lib64/libc.so.6
#14 0x00007fffea9530a1 in rocprofiler::registration::initialize() () from /apps/rocm/rocm-6.4afar7/lib/librocprofiler-sdk.so
#15 0x00007fffea9536e4 in rocprofiler_force_configure () from /apps/rocm/rocm-6.4afar7/lib/librocprofiler-sdk.so
#16 0x0000000000203a6d in setup () at test.cpp:151
#17 0x0000000000203e92 in main (argc=1, argv=0x7fffffffbcf8) at test.cpp:168