Question on extending 'block' HBM to u55c

[dhjoo98]

Hello,

I can see that the recent commit to support V80 introduces 'block' HBM mode, which allows each HBM channel to be address directly. I believe this is the extension to support per Coyotev2 paper, 'For applications that require the full HBM bandwidth, it is possible to bypass the MMU and directly expose certain HBM channels.'

As of now, I can see that this feature is not supported on u55c (as well as any Ultrascale+ platforms), from

Coyote/driver/src/vfpga/vfpga_gup.c

Line 401 in cdf54c8

dbg_info("memory block specified, but UltraScale+ devices do not support block memory; ignoring...\n");

I wonder if there are any plan to bring block memory (explicting addressing per-channel data) support to u55c.

Meanwhile, we see a saturation of Read Throughput on Figure 7a of the paper to around 60 GB/s. Is it plausible to attempt a design with DEST_BITS 5 + N_AXI_CARD 31 to get closer to the documented u55c HBM2 bandwidth 460 GB/s?

Thank you!

[bo3z]

Hi @dhjoo98,

What you're describing is possible, but there are a few things required.

First, what we refer to as "MMU bypassing" in the paper does not fully match the "block" implementation. On Versal devices, we have two implementations: "unified" which allows each card AXI stream to access the entire HBM, and "block" which has a 1:1 mapping between the AXI stream in the vFPGA and an HBM pseudo-channels (PCs); i.e. each AXI stream accesses a subset of the HBM memory (512MB on the v80). The reason for this implementation is that it reduces crossing of the horizontal NoC on the V80, which has lower bandwidth and can cause collision with many transfers.

On the u55c, the situation is slightly different. There is no NoC and the implementation defaults to "unified", meaning, again, each AXI stream can access the entire memory space. But this in turn creates a large crossbar of N_CARD_AXI x N_VFPGA x 32 (HBM channels), which leads to congestion and poor timing closure.

So, yes, implementing block HBM on UltraScale+ would also be beneficial, primarily helping in timing closure. To do so, one should at the minimum:
(1) Disable striping (done through the RAMA IP in the HBM block diagram)
(2) Change the block diagram address assignment so that each channel maps precisely to one HBM PC
(3) Disable unused channels
(4) (perhaps, not sure if it applies) floorplan the channels, so that there is spacing between them.

Additionally, we notice that on vFPGA typically saturates performance at 8-10 N_CARD_AXI; after that, we suspect that the MMU can not keep up. For the HBM throughput test on the V80, we used 4 vFPGAs with 7 card AXI streams. I am unsure how this could be further improved, without disabling the MMU in hardware (which creates complexity in moving data between the host and FPGA, page faulting etc.)