kv/kvserver: TestReplicaCircuitBreaker_Partial_Retry failed

[cockroach-teamcity]

kv/kvserver.TestReplicaCircuitBreaker_Partial_Retry failed on release-25.4.4-rc @ 3b602e9ede0a2153e9d0ebd5823e9d1444e0d4fc:

    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 2: to 1
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 1: to 2
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 3: to 3
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 3: to 3
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 3: to 2
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 3: to 1
    client_replica_circuit_breaker_test.go:953: raft leadership moved to n2
    client_replica_circuit_breaker_test.go:955: writes succeeded
    client_raft_helpers_test.go:401: from store 1; adding store 2 to drop list
    client_raft_helpers_test.go:401: from store 1; adding store 3 to drop list
    client_raft_helpers_test.go:401: from store 2; adding store 1 to drop list
    client_raft_helpers_test.go:401: from store 2; adding store 3 to drop list
    client_replica_circuit_breaker_test.go:964: partitioned all nodes and tripped their breakers
    client_replica_circuit_breaker_test.go:968: writes failed with ReplicaUnavailableError
    client_replica_circuit_breaker_test.go:974: expired n2 lease
    client_replica_circuit_breaker_test.go:976: writes failed with ReplicaUnavailableError
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 1: to 1
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 1: to 1
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 1: to 3
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 1: to 2
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 2: to 2
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 2: to 2
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 2: to 1
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 2: to 3
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 3: to 3
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 3: to 3
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 3: to 1
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 3: to 2
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 1: to 1
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 1: to 1
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 1: to 3
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 1: to 2
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 2: to 2
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 2: to 2
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 2: to 3
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 2: to 1
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeat from store 3: to 3
    client_raft_helpers_test.go:417: allowing StoreLiveness msg MsgHeartbeatResp from store 3: to 3
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 3: to 1
    client_raft_helpers_test.go:415: dropping StoreLiveness msg MsgHeartbeat from store 3: to 2
    client_replica_circuit_breaker_test.go:988: no raft leader
    client_replica_circuit_breaker_test.go:990: 
        	Error Trace:	pkg/kv/kvserver/client_replica_circuit_breaker_test.go:873
        	            				pkg/kv/kvserver/client_replica_circuit_breaker_test.go:990
        	            				pkg/testutils/subtest.go:23
        	Error:      	Not equal: 
        	            	expected: int(0)
        	            	actual  : int64(1)
        	Test:       	TestReplicaCircuitBreaker_Partial_Retry/lease-type=LeaseLeader
--- FAIL: TestReplicaCircuitBreaker_Partial_Retry/lease-type=LeaseLeader (10.48s)

Parameters:

• attempt=1
• run=5
• shard=35

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-59268

[arulajmani]

This was fixed by #156368, which wasn't backported.

[arulajmani]

Closing #162471. I'll see the backport through.