CI integration of networkservicemesh integration tests (NSMNSE-79) by amberimam · Pull Request #23 · cisco-app-networking/networkservicemesh

amberimam · 2021-02-04T14:21:13Z

Description

Add a jenkinsfile to this repo which installs all necessary dependencies for the networkservicemesh/cloudtest tool to run golang tests in the test/integration directory, and then runs single-cluster, interdomain and benchmark tests.
These tests can be configured to run upon any push to the networkservicemesh repo

Motivation and Context

We need to harden the cisco-app-networking fork of networkservicemesh and having these automated tests in place will help us to catch bugs more quickly.

https://jira-eng-sjc4.cisco.com/jira/browse/NSMNSE-79

…ger tests from jenkins for now

… if that helps them run

tiswanso · 2021-02-05T02:40:54Z

@@ -1,36 +0,0 @@
---


Do we need to delete these files? Does it effect anything to keep them?

In general with the networkservicemesh repo we want to try to have as small a footprint as possible (within reason) due to the need to periodically rebase to newer versions of the upstream.

I can add them back in. I just thought it would be easier to understand what's going on if we didn't have unused files lying around.

It'll be easiest to maintain the upstream rebases if we stick as much to purely additive changes as possible. If these files change upstream and we've deleted them in our downstream fork then we'll likely have to resolve the conflicts manually every rebase.

I see that makes sense. I can add back in the .cloudtest.yaml file as well I guess, even though the jenkinsfile is not using that one.

tiswanso · 2021-02-05T02:44:30Z

+    timeout: 300    # 5 minutes to start cluster
+    stop-delay: 10
+    env:
+      - KIND_CLUSTER_NAME=cloudtest-kind-$(rands10)   # Generate a uniq cluster name


How does the injection of the $(rands10) actually work? Does cloudtest process the yaml file content in a way that a bash shell would exec that?

Oh I see that a makefile target is wrapping usage of the env var with $() exec... interesting

Cloudtest does do extra processing.
https://github.com/networkservicemesh/cloudtest/blob/master/docs/README.md#environment-variables-processing

you get environment variables and a handful of other useful commands, like rands10. I'm not sure if this is executed by bash or not, they may have just borrowed the syntax.

tiswanso · 2021-02-05T02:52:33Z

+//The Jenkinsfile runs entire build in a golang container and mounts the /var/run/docker.sock file to allow access to the host docker within the container
+node('wcm') {
+    def build_ok = true
+    docker.image('golang:1.15-buster').inside('-u root --net=host -v /var/run/docker.sock:/var/run/docker.sock') {


we'll possibly have an issue with using -u root in the long run. Got some feedback on that from the SRE team that it could be causing content to be left on the jenkins slave that the container runs on.

Ok, but I can leave the -u root there for now?

If you could try without -u root to see if it's still working the same that'd be good. To get kind cluster create and go get / go mod stuff to run in nsm-nse I had to add the following to env vars:

HOME = "${WORKSPACE}" GOCACHE = "/go/.cache"

Sure, I can give that a go and see what happens.

This is what it looks like without -u root: https://engci-private-sjc.cisco.com/jenkins/eti-sre/job/AppN/job/amimam/job/networkservicemesh/job/devop_jenkins/143/console

Ideally, we would have access to an ephemeral docker host, or the build agent would be ephemeral, so that we can easily guarantee that our environments are consistent. There's too many hands touching those build agents to have any sort of guarantee that they're going to be in a consistent state.

tiswanso · 2021-02-05T03:00:04Z

+            CGO_ENABLED = 0
+            GO111MODULE = on
+        }
+        stage ('Install Docker Client') {


I refactored what was in nsm-nse to use the same base for our runner container we use in circleci to avoid having to install all these type of dependencies--docker client, kubectl, helm, kind, go, git, etc. I'm looking to get a more standard set of base container images curated by the SRE team that we can use as our jenkins agent setting. We don't have to change this now but it'll be something we want to follow-up on.

See here: https://github.com/cisco-app-networking/nsm-nse/blob/devop_jenkins/.jenkins/Jenkinsfile#L14

ok thanks, good to know.

tiswanso · 2021-02-05T03:08:30Z

-	@kind get kubeconfig --name $(KIND_CLUSTER_NAME) > $(KIND_CLUSTER_NAME)-kubeconfig
+	@touch $(CONFIG_LOCATION); \
+	kind get kubeconfig --name $(KIND_CLUSTER_NAME) > $(CONFIG_LOCATION); \
+	echo "$(CONFIG_LOCATION)"


Is tempdir an executable that generates a temporary directory? If so, I don't think the echo will actually show the right CONFIG_LOCATION value because it's going to re-exec the tempdir command which will generate a new dir. Also, I don't see how you can use the KIND_CLUSTER_NAME var like that and have it generate a name used for the kind create cluster command and then use it again for this kubeconfig get call.

Good point! I will delete that echo line.

Why can't I reuse the KIND_CLUSTER_NAME var?

I found those changes to the kind.mk file in the upstream networkservicemesh/cloudtest repo: https://github.com/networkservicemesh/cloudtest/blob/master/examples/kind/Makefile#L39

I'm thinking that $(rands10) generates a random string each time it's invoked. But it's not a standard executable so maybe cloudtest does some kind of replace on it. Craziness... but I guess if it's the established pattern in cloudtest I'm ok with it for now until we figure out whether it's a problem.

tiswanso · 2021-02-05T03:20:33Z

 	github.com/networkservicemesh/networkservicemesh/test v0.0.0-00010101000000-000000000000
 	github.com/networkservicemesh/networkservicemesh/utils v0.3.0
 	github.com/onsi/gomega v1.10.3
+	github.com/packethost/packngo v0.5.1 // indirect


was this something you actually added or something added by your IDE?

My IDE added it. Should I remove it?

yes and also revert the diffs to go.sum

Sure will do!

tiswanso · 2021-02-05T13:04:22Z

I took a look at the cloudtest repo and see that you copied a lot of the .cloudtest content from there. It uses the same kind clustername and configdir env var approach with the $(rands10) and $(tempdir). I'm a bit skeptical about that approach working properly but not because of anything you did, specifically. Based on what you mentioned about the multiple kind interdomain testing, it's probably something not working for at least the cases when there's more than 1 cluster being started. A little disappointing in that hacky-ness but maybe it's something we can fix quickly and push upstream as well.

amberimam · 2021-02-05T13:39:13Z

Thanks for your feedback Tim! :)

…anges between this repo and upstream repo

tiswanso · 2021-02-05T13:57:10Z

@@ -2,23 +2,27 @@
 *



Could you comment on how these changes came about?

I basically rearranged them in alphabetical order, and then added some folders and files that were needed by cloudtest.

I was using a dockerfile for local testing on my mac before I started using the cisco-hosted-jenkins environment.

Amber Imam added 30 commits January 27, 2021 12:48

Initial commit

25366f8

Add stage for installing docker client to jenkinsfile

5159511

Add stage for installing yq in jenkinsfile

f9c29c9

Add stage for installing kubectl in jenkinsfile

b5756fa

escape dollar sign

7656f1c

Add stage to install helm to jenkinsfile

3215b2d

Add environment vars to jenkinsfile

8377e22

Add stage to install kind to jenkinsfile

2f1e4f9

Add stage to install cloudtest to jenkinsfile

8e3fa80

Add stage to run tests in jenkinsfile

aeef96b

Add docker host networking to jenkinsfile

e6d59f9

Run single cluster tests in jenkins

51afdd1

Run single cluster tests in jenkins

93ccd37

Increase cloudtest timeout to 2 hours

8239b95

Add 'wcm' label to jenkinsfile so it runs on the right nodes

9ca800f

modify the way kind is installed in jenkinsfile

38cd1d4

reduce stats prints frequency and cleanup jenkinsfile

f2520e4

create two kind clusters for interdomain tests

b8edfc1

try again to create two kind clusters for interdomain tests

0365ef6

move all config into cloudtest config file

2f52c5b

use separate cloudtest config files for now to make it easier to trig…

a355cf9

…ger tests from jenkins for now

add interdomain tests to the jenkinsfile in a separate stage

4ccfbae

add cluster selector to interdomain tests to see if that helps them run

95553fa

try adding a second kind cluster selector to interdomain tests to see…

aedaf82

… if that helps them run

fix prepare section of cloudtest config

ddfbba0

use hardcoded cluster names to see if it will run

04d69f4

change instances to 1 to see if that helps

d7007c4

try to fix path to kubeconfig

f2eba70

pass kubeconfig locations directly into go test

01ffb5b

catch errors in interdomain tests

cbbe6b9

Amber Imam added 6 commits February 3, 2021 15:36

find files recursively to archive

cd31d11

a

8fb9d7a

a

8064ba6

a

cbb7aa4

remove unused files

a15c242

remove more unused files

23b4804

amberimam requested a review from ondrej-fabry February 4, 2021 14:22

amberimam assigned tiswanso, brecode, snowbldr, sriaradhyula, ziyuguo716, CosminNechifor, panyuenlau, sfph and shanchunyang0919 Feb 4, 2021

Amber Imam added 3 commits February 4, 2021 12:17

use helm 3 instead of helm 2

3c7e2ac

use helm 3 instead of helm 2

6d85130

undo changes to helm-init-wrapper.sh

a3c03fb

tiswanso reviewed Feb 5, 2021

View reviewed changes

add back deleted .cloudtest/execution folder and files to minimize ch…

34146a0

…anges between this repo and upstream repo

tiswanso reviewed Feb 5, 2021

View reviewed changes

revert changes to go.mod and go.sum

1f6abff

Uh oh!

Conversation

amberimam commented Feb 4, 2021

Description

Motivation and Context

Uh oh!

tiswanso Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tiswanso Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amberimam Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tiswanso commented Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amberimam commented Feb 5, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

tiswanso Feb 5, 2021 •

edited

Loading

tiswanso Feb 5, 2021 •

edited

Loading

amberimam Feb 5, 2021 •

edited

Loading

tiswanso commented Feb 5, 2021 •

edited

Loading