HTTP/2 proxy has slower speeds compared to HTTP/1.1

[vanchaxy]

I'm exposing LibreSpeed using an Envoy gateway, and speed test results are slower with HTTP/2 and RTT ~55ms. Both gateways are without any settings changes except for externalTrafficPolicy: Cluster.

LibreSpeed is exposed via two gateways: one local (RTT ~2ms) and one remote via tunnel (RTT ~55ms). These are the results (download/upload in mbps):

• Local Envoy gateway, HTTP/1.1: 1000/1000
• Local Envoy gateway, HTTP/2: 950/950
• Remote load balancer (direct to service without proxy), HTTP/1.1: 700/700 - the maximum possible due to the tunnel bandwidth
• Remote Envoy gateway, HTTP/1.1: 700/700
• Remote Envoy gateway, HTTP/2: 380/20 - very slow upload, may be related to this issue: Slow proxy when request has large payloads envoy#35732
• Remote Envoy gateway with ClientTrafficPolicy, HTTP/2: 380/550

The last test was with ClientTrafficPolicy using the following specs:

http2:
  initialConnectionWindowSize: 64Mi
  initialStreamWindowSize: 64Mi

HTTP/2 speed is consistently slower than HTTP/1.1 with default settings (especially upload), and even with tuning, it's slower than HTTP/1.1, which works at maximum speed.

Is that expected for the gateway to be much slower using HTTP/2? Is there some tuning needed that I'm missing?