[SmartSwitch] Support maximum gnmi message size of 125k messages#643
[SmartSwitch] Support maximum gnmi message size of 125k messages#643croos12 wants to merge 4 commits intosonic-net:masterfrom
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This PR adds configurable gRPC max receive/send message sizes to the SONiC gNMI telemetry server, to better handle larger config/telemetry payloads at scale.
Changes:
- Added
MaxRecvMsgSize/MaxSendMsgSizetoTelemetryConfig. - Introduced
-max_recv_msg_sizeand-max_send_msg_sizeCLI flags. - Applied the configured limits to the gRPC server via
grpc.MaxRecvMsgSize/grpc.MaxSendMsgSize.
|
@prsunny PFA |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azpw run |
|
Retrying failed(or canceled) jobs... |
|
Retrying failed(or canceled) stages in build 1085425: ✅Stage Build:
|
Signed-off-by: Connor Roos <croos@nvidia.com>
Signed-off-by: Connor Roos <croos@nvidia.com>
dd3513c to
b983d54
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azpw run |
|
Retrying failed(or canceled) jobs... |
|
Retrying failed(or canceled) stages in build 1085623: ✅Stage BuildArm64:
✅Stage BuildAmd64:
✅Stage Build:
|
|
/azpw retry |
|
Retrying failed(or canceled) jobs... |
|
Retrying failed(or canceled) stages in build 1085623: ✅Stage BuildAmd64:
✅Stage BuildArm64:
✅Stage Build:
|
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@croos12 Just to double check do you mean 125K or 125M in title? |
It's 125k total messages per file max. In testing I found that will fit in 32MB for gnmi send/recv buffers |
|
Sorry this is still a bit confusing. I don't see any mentioning of the 125kb in the new code and isn't 125 kb < 4mb? What exactly are you changing to 125kb? |
Sorry these variables are kind of confusing. The "125k" in the title means 125,000 messages (count), not 125 kilobytes. The largest ENI-level object configuration in DASH requires up to 125,000 protobuf messages to be sent in a single gNMI call (CA-to-PA). At that scale, the serialized payload requires a maxMessage size of around ~32MB, which I am setting in my sonic-buildimage PR that requires this change: sonic-net/sonic-buildimage#26679 |
|
Thanks for the clarification. It would be nice to include some background or pointers to background in the description. Also you might want to merge latest master for the pipeline error. |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Why I did it
At max DASH scale, a single gNMI call (CA-to-PA, ENI-level object config) can contain up to ~125,000 protobuf messages, which serializes to ~32 MB. sonic-gnmi currently uses gRPC's 4 MB default for
MaxRecvMsgSize, so these calls are rejected.How I did it
MaxRecvMsgSize(server)-max_recv_msg_sizeMaxSendMsgSize(server)math.MaxInt32(default)-max_send_msg_sizeDefaults are unchanged; sonic-buildimage#26679 sets both to 32 MB for SmartSwitch.
How to verify it
Start telemetry with
-max_recv_msg_sizeabove 4 MB and send a gNMI message larger than 4 MB; confirm it is accepted.