Skip to content

Comments

Enable huge-service tests to run on AWS#17986

Draft
ronaldngounou wants to merge 1 commit intokubernetes:masterfrom
ronaldngounou:enable-huge-service-aws
Draft

Enable huge-service tests to run on AWS#17986
ronaldngounou wants to merge 1 commit intokubernetes:masterfrom
ronaldngounou:enable-huge-service-aws

Conversation

@ronaldngounou
Copy link
Member

@ronaldngounou ronaldngounou commented Feb 19, 2026

Motivation:

In order to run huge-service tests on AWS, this PR enables the test described in

https://github.com/kubernetes/perf-tests/blob/e0b6d58341e9dd642d1a9b1fbf99651c8dbb8274/clusterloader2/testing/huge-service/config.yaml#L7

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 19, 2026
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hakuna-matatah for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from dims and hakman February 19, 2026 07:53
@ronaldngounou
Copy link
Member Author

/assign upodroid

@ronaldngounou
Copy link
Member Author

/test pull-kops-ec2-master-scale-performance-5000

@ronaldngounou ronaldngounou marked this pull request as draft February 19, 2026 17:04
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 19, 2026
@ronaldngounou
Copy link
Member Author

ronaldngounou commented Feb 19, 2026

NetworkProgrammingLatency is defined as the time it took to program the network - from the time the service or pod has changed to the time the change was propagated and the proper kube-proxy rules were synced. Exported for each endpoints object that were part of the rules sync.
See github.com/kubernetes/community/blob/master/sig-scalability/slos/network_programming_latency.md
Note that the metrics is partially based on the time exported by the endpoints controller on the master machine. The measurement may be inaccurate if there is a clock drift between the node and master machine.

{ Failure :0
[measurement call PodStartupLatency - PodStartupLatency error: pod startup: too high latency 99th percentile: got 5.540783283s expected: 5s]
:0}
{ Failure :0
[measurement call NetworkProgrammingLatency - NetworkProgrammingLatency error: Network Programming Latency: too high latency 50th percentile: got 5m0s expected: 20s]
:0}

@ronaldngounou
Copy link
Member Author

ronaldngounou commented Feb 19, 2026

From the kubelet logs:
https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kops/17986/pull-kops-ec2-master-scale-performance-5000/2024416735951589376/artifacts/i-0001f8de44c695582/kubelet.log

E0219 10:56:35.073690    1856 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = 

NotFound desc = an error occurred when try to find container

"53b97f58dab0e31cb963eb6134efe4569b556a43750b6c6c8c17244b2f24ac2a\": not found" 

containerID="53b97f58dab0e31cb963eb6134efe4569b556a43750b6c6c8c17244b2f24ac2a"
I0219 10:56:35.074006    1856 pod_container_deletor.go:53] "DeleteContainer returned error" containerID=

{"Type":"containerd","ID":"53b97f58dab0e31cb963eb6134efe4569b556a43750b6c6c8c17244b2f24ac2a"} 

err="failed to get container status 

\"53b97f58dab0e31cb963eb6134efe4569b556a43750b6c6c8c17244b2f24ac2a\": rpc error: code = NotFound 

desc = an error occurred when try to find container 

\"53b97f58dab0e31cb963eb6134efe4569b556a43750b6c6c8c17244b2f24ac2a\": not found"

@ronaldngounou ronaldngounou force-pushed the enable-huge-service-aws branch from 9fda838 to 23c3e76 Compare February 20, 2026 06:42
Motivation:

In order to run huge-service tests on AWS. this PR enable the test
described in perf-tests/clusterloader2/testing/huge-service/config.yaml

Signed-off-by: Ronald Ngounou <ronald.ngounou@yahoo.com>
@ronaldngounou ronaldngounou force-pushed the enable-huge-service-aws branch from 23c3e76 to f35a646 Compare February 20, 2026 06:43
@ronaldngounou
Copy link
Member Author

/test pull-kops-ec2-master-scale-performance-5000

@ronaldngounou
Copy link
Member Author

/test pull-kops-gce-master-scale-performance-5000

@ronaldngounou
Copy link
Member Author

/test pull-kops-ec2-master-scale-performance-5000

@ronaldngounou
Copy link
Member Author

/test pull-kops-gce-master-scale-performance-5000

@ronaldngounou
Copy link
Member Author

/retest

@k8s-ci-robot
Copy link
Contributor

@ronaldngounou: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kops-gce-master-scale-performance-5000 f35a646 link true /test pull-kops-gce-master-scale-performance-5000
pull-kops-ec2-master-scale-performance-5000 f35a646 link false /test pull-kops-ec2-master-scale-performance-5000

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants