Move the prefetched info to preallocated buffers

[chouxi]

Summary:
X-link: https://github.com/facebookresearch/FBGEMM/pull/2219

This change improves the performance of tracking the deltas in TBE, mainly by
replacing DtoH copy with {F1984231816}
with DtoD copy with async DtoH under stream_callback {F1984231839}

To achieve this, the following is added

• the pre-registered UVA buffer that's accessible from both GPU and CPU are reused every iteration
  • makes the lifetime of tensors the same to TBE makes it safe to async copy.
  • reuse the same buffer to avoid repeating allocation.
• trigger the CPU thread to async copy in raw_embedding_streamer.stream()
  • GPU ops don't wait on the D2H
• To avoid the D2D copy overlaps with D2H copy
  • A GPU event to track the finish of the D2D copy, make the CPU thread to wait for the D2D copy finish
  • join_stream_tensor_copy_thread to trigger a blocking wait for the copy in the next iteration in case of CPU copies take too long before overwriting the pre-registered buffer.

Differential Revision: D86888586

[meta-codesync]

@chouxi has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86888586.