Fix too_many_pings during DPU SetPackage by coordinating keepalive settings by sigabrtv1-ui · Pull Request #620 · sonic-net/sonic-gnmi

sigabrtv1-ui · 2026-03-17T22:58:54Z

What I did

Fixed a keepalive mismatch between the DPU proxy client and gNMI server that causes ENHANCE_YOUR_CALM / "too_many_pings" disconnections during long-running SetPackage operations on smartswitch DPUs.

How I did it

Proxy client (pkg/interceptors/dpuproxy/proxy.go): Increased keepalive ping interval from 10s → 30s and timeout from 3s → 10s.
gNMI server (telemetry/telemetry.go): Added explicit KeepaliveEnforcementPolicy with MinTime: 20s so the server tolerates the proxy ping frequency instead of using the default 5-minute minimum.

Why I did it

During DPU image installation via SetPackage, sonic-installer install can run for several minutes. The proxy client was pinging every 10s, but the DPU server's default enforcement policy (MinTime=5m) treated this as a flood, terminating the connection with too_many_pings.

The 30s client interval with 20s server minimum provides a safe margin while still detecting dead connections reasonably fast.

How to verify it

Deploy on a smartswitch with DPU
Run SetPackage gNOI RPC targeting a DPU with a valid image
Verify the installation completes without too_many_pings disconnection
Verify keepalive still detects dead connections (kill the DPU gNMI server mid-operation)

Fixes: #619
Related: sonic-net/sonic-mgmt#23054

mssonicbld · 2026-03-17T22:59:02Z

/azp run

azure-pipelines · 2026-03-17T22:59:12Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-18T18:44:32Z

/azp run

azure-pipelines · 2026-03-18T18:44:45Z

Azure Pipelines successfully started running 1 pipeline(s).

vaibhavhd · 2026-03-18T18:34:28Z

 			grpc.WithKeepaliveParams(keepalive.ClientParameters{
-				Time:                10 * time.Second, // Send keepalive ping every 10s
-				Timeout:             3 * time.Second,  // Wait 3s for ping ack before considering connection dead
+				Time:                30 * time.Second, // Send keepalive ping every 30s


Should this be a configurable knob

@sigabrtv1-ui

See the comment below. Seems the bot can't comment since it is not in the org.

vaibhavhd · 2026-03-18T18:35:18Z

+			// frequently during long-running operations like SetPackage.
+			// See: https://github.com/sonic-net/sonic-gnmi/issues/619
+			keep_alive_policy := keepalive.EnforcementPolicy{
+				MinTime:             20 * time.Second, // Allow pings as frequent as every 20s


Same as above - should this be a configurable knob?

See the comment below. Seems the bot can't comment since it is not in the org.

mssonicbld · 2026-03-18T18:53:10Z

/azp run

sigabrtv1-ui

Addressed review feedback: keepalive values are now configurable via environment variables (GNMI_CLIENT_KEEPALIVE_INTERVAL, GNMI_CLIENT_KEEPALIVE_TIMEOUT, GNMI_SERVER_KEEPALIVE_MIN_TIME). Defaults remain the same so behavior is unchanged out of the box.

azure-pipelines · 2026-03-18T18:53:24Z

Azure Pipelines successfully started running 1 pipeline(s).

sigabrtv1-ui · 2026-03-18T18:53:50Z

@vaibhavhd Good call — addressed in the latest commit (c77261e). All three keepalive values are now configurable via environment variables:

GNMI_CLIENT_KEEPALIVE_INTERVAL — client ping interval (default: 30s)
GNMI_CLIENT_KEEPALIVE_TIMEOUT — client ping ack timeout (default: 10s)
GNMI_SERVER_KEEPALIVE_MIN_TIME — server minimum allowed ping interval (default: 20s)

Defaults remain the same so behavior is unchanged out of the box.

mssonicbld · 2026-03-19T02:56:01Z

/azp run

azure-pipelines · 2026-03-19T02:56:11Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-19T14:40:30Z

/azp run

azure-pipelines · 2026-03-19T14:40:41Z

Azure Pipelines successfully started running 1 pipeline(s).

SmartSwitch DPU runs on ARM64, so we need an ARM64 deb artifact. - Add BuildArm64 stage that runs after Build succeeds - Uses sonicso1ES-arm64 pool with sonic-slave-bookworm arm64 container - New install-dependencies-arm64.yml template for arm64 deps - Publishes sonic-gnmi.arm64 artifact Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

Download libnl debs (libnl-3-200, libnl-genl-3-200, libnl-route-3-200, libnl-nf-3-200) from common-lib.arm64 artifact alongside libyang, matching the pattern used by the amd64 install-dependencies template. Signed-off-by: sigabrtv1-ui <sig.abrt.v1@gmail.com> Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

…ttings The DPU proxy client was sending keepalive pings every 10s, but the gRPC server uses the default EnforcementPolicy with MinTime=5m. During long-running SetPackage operations (sonic-installer can take minutes), this mismatch causes the server to terminate the connection with GOAWAY/ENHANCE_YOUR_CALM and debug data 'too_many_pings'. Fix: 1. Increase proxy client keepalive interval from 10s to 30s and timeout from 3s to 10s for better resilience during long operations. 2. Add explicit KeepaliveEnforcementPolicy to the gNMI server allowing pings as frequent as every 20s, with a safety margin above the client's 30s interval. Fixes: sonic-net#619 Related: sonic-net/sonic-mgmt#23054 Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-03-19T15:50:23Z

/azp run

azure-pipelines · 2026-03-19T15:50:36Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-19T23:41:24Z

/azp run

azure-pipelines · 2026-03-19T23:41:35Z

Azure Pipelines successfully started running 1 pipeline(s).

vaibhavhd · 2026-03-20T16:21:37Z

-				Time:                10 * time.Second, // Send keepalive ping every 10s
-				Timeout:             3 * time.Second,  // Wait 3s for ping ack before considering connection dead
-				PermitWithoutStream: true,             // Send pings even when no active RPCs
+				Time:                getEnvDuration("GNMI_CLIENT_KEEPALIVE_INTERVAL", 30*time.Second),


I don't like getting configurable values from OS environment variables. We have tools (config db?) that we should leverage. May be the agent isn't aware of it, and needs some SONiC specific training to understand how we expect these knobs to be implemented.

mssonicbld · 2026-03-20T16:30:45Z

/azp run

azure-pipelines · 2026-03-20T16:30:56Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-03-20T16:31:13Z

/azp run

azure-pipelines · 2026-03-20T16:31:23Z

Azure Pipelines successfully started running 1 pipeline(s).

vaibhavhd

Approving based on offline discussion about a follow up PR where the timeouts/settings will be introduced as config.

Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-03-25T17:29:30Z

/azp run

azure-pipelines · 2026-03-25T17:29:43Z

Azure Pipelines successfully started running 1 pipeline(s).

…v vars Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

mssonicbld · 2026-03-27T04:11:13Z

/azp run

azure-pipelines · 2026-03-27T04:11:23Z

Azure Pipelines successfully started running 1 pipeline(s).

- Unify install-dependencies.yml with arch parameter (amd64/arm64) instead of maintaining separate install-dependencies-arm64.yml - Add installTestDeps parameter to skip pytest/redis/.NET for build-only jobs - Extract build-deb.yml template for shared checkout + build + publish flow - ARM64 stage now uses build-deb.yml template instead of inline steps - Remove install-dependencies-arm64.yml (merged into unified template) Signed-off-by: sigabrtv1-ui <sig.abrt.v1@gmail.com>

mssonicbld · 2026-03-27T17:40:56Z

/azp run

azure-pipelines · 2026-03-27T17:41:05Z

Azure Pipelines failed to run 1 pipeline(s).

Switch amd64 from downloading the full sonic-buildimage.vs artifact to using Azure.sonic-buildimage.common_libs (artifact: common-lib), matching the pattern used by sonic-sairedis and other SONiC repos. - Both architectures now use the same download step with arch-aware artifact name - Yang wheels still downloaded separately from pipeline 142 (arch-independent) - Eliminates arch-specific branching for libyang/libnl download and install - Smaller artifact downloads for amd64 (common-lib vs full sonic-buildimage.vs) Signed-off-by: sigabrtv1-ui <sig.abrt.v1@gmail.com>

mssonicbld · 2026-03-27T17:46:58Z

/azp run

azure-pipelines · 2026-03-27T17:47:07Z

Azure Pipelines failed to run 1 pipeline(s).

Azure Pipelines doesn't allow ${{ if }} directives embedded in strings. Replace with explicit parameters (commonLibArtifact, swssCommonArtifact, publishArtifact) and separate conditional blocks for arch-specific steps. Fixes validation errors: - 'The directive if is not allowed in this context' - Directives not supported for expressions embedded within a string Signed-off-by: sigabrtv1-ui <sig.abrt.v1@gmail.com>

mssonicbld · 2026-03-27T18:21:18Z

/azp run

azure-pipelines · 2026-03-27T18:21:29Z

Azure Pipelines successfully started running 1 pipeline(s).

hdwhdw requested review from hdwhdw, ryanzhu706 and zbud-msft March 17, 2026 23:04

ryanzhu706 previously approved these changes Mar 18, 2026

View reviewed changes

sigabrtv1-ui dismissed ryanzhu706’s stale review via 436961b March 18, 2026 18:44

sigabrtv1-ui force-pushed the fix/keepalive-too-many-pings branch from 5a3d4dd to 436961b Compare March 18, 2026 18:44

vaibhavhd reviewed Mar 18, 2026

View reviewed changes

sigabrtv1-ui commented Mar 18, 2026

View reviewed changes

sigabrtv1-ui force-pushed the fix/keepalive-too-many-pings branch from c77261e to ae20fed Compare March 19, 2026 02:55

sigabrtv1-ui force-pushed the fix/keepalive-too-many-pings branch from ae20fed to 16b2092 Compare March 19, 2026 14:40

hdwhdw and others added 4 commits March 19, 2026 15:50

Run ARM64 build in parallel with amd64 Build stage

4bcea8c

Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

sigabrtv1-ui force-pushed the fix/keepalive-too-many-pings branch from 16b2092 to 883512e Compare March 19, 2026 15:50

hdwhdw requested a review from prsunny March 20, 2026 14:58

vaibhavhd reviewed Mar 20, 2026

View reviewed changes

sigabrtv1-ui force-pushed the fix/keepalive-too-many-pings branch from 447676e to 1cfdf4b Compare March 20, 2026 16:31

vaibhavhd previously approved these changes Mar 20, 2026

View reviewed changes

hdwhdw previously approved these changes Mar 20, 2026

View reviewed changes

Remove downloaded dependencies from arm64 artifacts

b29bd22

Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

sigabrtv1-ui dismissed stale reviews from hdwhdw and vaibhavhd via b29bd22 March 25, 2026 17:29

Fix ARM64 build: add ENABLE_TRANSLIB_WRITE and ENABLE_NATIVE_WRITE en…

a434d61

…v vars Signed-off-by: Dawei Huang <daweihuang@microsoft.com>

Conversation

sigabrtv1-ui commented Mar 17, 2026

What I did

How I did it

Why I did it

How to verify it

Uh oh!

mssonicbld commented Mar 17, 2026

Uh oh!

azure-pipelines Bot commented Mar 17, 2026

Uh oh!

mssonicbld commented Mar 18, 2026

Uh oh!

azure-pipelines Bot commented Mar 18, 2026

Uh oh!

vaibhavhd Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

vaibhavhd Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

hdwhdw Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

vaibhavhd Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

hdwhdw Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Mar 18, 2026

Uh oh!

sigabrtv1-ui left a comment

Choose a reason for hiding this comment

Uh oh!

azure-pipelines Bot commented Mar 18, 2026

Uh oh!

sigabrtv1-ui commented Mar 18, 2026

Uh oh!

mssonicbld commented Mar 19, 2026

Uh oh!

azure-pipelines Bot commented Mar 19, 2026

Uh oh!

mssonicbld commented Mar 19, 2026

Uh oh!

azure-pipelines Bot commented Mar 19, 2026

Uh oh!

mssonicbld commented Mar 19, 2026

Uh oh!

azure-pipelines Bot commented Mar 19, 2026

Uh oh!

mssonicbld commented Mar 19, 2026

Uh oh!

azure-pipelines Bot commented Mar 19, 2026

Uh oh!

vaibhavhd Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Mar 20, 2026

Uh oh!

azure-pipelines Bot commented Mar 20, 2026

Uh oh!

mssonicbld commented Mar 20, 2026

Uh oh!

azure-pipelines Bot commented Mar 20, 2026

Uh oh!

vaibhavhd left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Mar 25, 2026

Uh oh!

azure-pipelines Bot commented Mar 25, 2026

Uh oh!

mssonicbld commented Mar 27, 2026

Uh oh!

azure-pipelines Bot commented Mar 27, 2026

Uh oh!

mssonicbld commented Mar 27, 2026

Uh oh!

azure-pipelines Bot commented Mar 27, 2026