refactor: unify images into calicovpp/vpp and flatten repo layout

- consolidate calicovpp/agent, calicovpp/vpp, calicovpp/multinet-monitor
  and calicovpp/init-eks into a single unified calicovpp/vpp image.
- restructure source directories under pkg/ for cleaner organization.
- move executable entrypoints to top-level cmd/ to allow for building
  all golang agents from the top-level Makefile.
- runtime behavior is unchanged - two separate daemons (agent and vpp)
  run in separate containers from the same image using command overrides.

Changes:
- unified image: calicovpp/vpp (contains all binaries)
- single env var: CALICO_VPP_IMAGE (replaces CALICO_AGENT_IMAGE,
  CALICO_VPP_IMAGE, MULTINET_MONITOR_IMAGE)
- unified dockerfiles: Dockerfile and Dockerfile.debug now contain all
  binaries - calico-vpp-agent, felix-api-proxy, gobgp, multinet-monitor,
  vpp-manager and init-eks scripts in images/ubuntu/bin/
- top-level Makefile delegates image build through vpp-manager
- vpp-manager Makefile builds all Go binaries into unified image
- image/push targets removed from agent and multinet-monitor Makefile
- manifests updated to calicovpp/vpp with explicit command overrides for
  agent and multinet-monitor containers
- path references in test/ and docs/ updated for pkg/ move
- image contexts moved to top-level images/
- helper scripts moved to top-level scripts/
- VPP clone/build directory relocated to top-level vpp_build

Changes for VPP kube-test integration:

CalicoVPP now owns the kube-test manifest as a set of kustomize sources
so VPP kube-test (and any other consumer) does not need to know which
image to use, where the VPP build lives in this repo or any other
CalicoVPP-internal layout detail.

- yaml/components/kube-test/ - kustomize coimponent shared by the kind
  and baremetal flavors. Defines the kube-test daemonset patch, the
  kube-test configmap defaults and the tigera-operator snippet.
- yaml/components/kube-test/kind/ - KinD-flavor adjustments
  (uplink eth0 via af_packet, no hugepages).
- yaml/components/kube-test/baremetal/ - baremetal-flavor adjustments
  (DPDK uplink driven by ${CALICOVPP_INTERFACE}, hugepages, larger
  CPU/buffer allocation, docker.io registry, IfNotPresent pull policy).
- yaml/overlays/kube-test-{kind,baremetal}/ - overlays wiring the
  base daemonset and the kube-test components together.
- yaml/generated/calico-vpp-kubetest.yaml - regenerated from the
  kind overlay; consumers see only the runtime placeholders
  ${CALICOVPP_VERSION}, ${HOME}, ${ADDITIONAL_VPP_CONFIG},
  ${CALICOVPP_ENABLE_MEMIF}, ${CALICO_NETWORK_CONFIG} and
  ${CALICOVPP_INTERFACE} (baremetal only).
- yaml/Makefile - regenerates the checked-in kubetest yaml for kind
  and sanity-checks the baremetal overlay.

Top-level Makefile targets (the public API consumed by VPP kube-test):
- make vpp - delegates to pkg/vpp-manager with VPP_DIR/BASE pass-through;
             consumers no longer need to know where VPP build sources live.
- make kube-test-template FLAVOR={kind,baremetal}
           - runs `kubectl kustomize` on the requested overlay and appends
             the tigera Installation/APIServer snippet (the snippet contains
             the ${CALICO_NETWORK_CONFIG} placeholder, so it is not a
             kustomize resource and is concatenated verbatim post-build).
- make kube-test-push-images
           - pulls/tags/pushes each image in $KUBE_TEST_IMAGES (default:
             calicovpp/vpp) into the local kind registry. Older split-image
             release branches backport this target with ${KUBE_TEST_IMAGES}
             as calicovpp/vpp calicovpp/agent calicovpp/multinet-monitor.

Images that remain separate: calicovpp/vclsidecar, calicovpp/ctl

Directory Restructure:
├── cmd
│   ├── api-proxy
│   ├── calico-vpp-agent
│   ├── calicovppctl
│   ├── multinet-monitor
│   ├── replay-trace
│   └── vpp-manager
├── common.mk
├── docs
├── images
├── Makefile
├── pkg
│   ├── calico-vpp-agent
│   ├── config
│   ├── multinet-monitor
│   ├── vpplink
│   └── vpp-manager
├── scripts
├── test
├── vpp_build
└── yaml

Benefits:
- Single image tag to manage across all dataplane components
- Reduced bandwidth and storage requirements
- No risk of image version drift between components
- Simplified CI/CD pipeline
- Easier version management and releases

Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>

Author: aritrbas

.gitignore

@@ -1,13 +1,14 @@
 # Binaries
-calico-vpp-agent/bin
-vpp-manager/images/*/vpp-manager
-vpp-manager/images/*/bin
-vpp-manager/images/*/artifacts
-vpp-manager/images/*/vppdev.sh
-vpp-manager/vpp_build/
+pkg/calico-vpp-agent/bin
+cmd/calicovppctl/bin
+images/*/vpp-manager
+images/*/bin
+images/*/artifacts
+images/*/vppdev.sh
+vpp_build/
 *.deb
 *.so*
-multinet-monitor/watcher
+pkg/multinet-monitor/watcher
 *.tgz
 *.tar
 
@@ -28,10 +29,10 @@ test/vagrant/.vagrant
 *.bkp
 
 # build artefacts
-calico-vpp-agent/version
-vpp-manager/images/ubuntu/version
+pkg/calico-vpp-agent/version
+images/ubuntu/version
 test/scripts/.buildlogs
-multinet-monitor/version
+pkg/multinet-monitor/version
 .coverage/
 bin/
 vendor/

Makefile

@@ -5,26 +5,70 @@ check-%:
 
 .PHONY: build
 build:
-	$(MAKE) -C calico-vpp-agent $@
-	$(MAKE) -C vpp-manager $@
-	$(MAKE) -C multinet-monitor $@
+	$(MAKE) -C pkg/vpp-manager $@
 	$(MAKE) -C cmd/calicovppctl $@
 
 .PHONY: image images
 images: image
 image:
-	$(MAKE) -C calico-vpp-agent $@
-	$(MAKE) -C vpp-manager $@
-	$(MAKE) -C multinet-monitor $@
+	$(MAKE) -C pkg/vpp-manager $@
 	$(MAKE) -C cmd/calicovppctl $@
 
 .PHONY: image-kind
 image-kind: image
-	@for image in vpp:$(TAG) vpp:dbg-$(TAG) vclsidecar:$(TAG) vclsidecar:dbg-$(TAG) agent:$(TAG) multinet-monitor:$(TAG); do \
+	@for image in vpp:$(TAG) vpp:dbg-$(TAG) vclsidecar:$(TAG) vclsidecar:dbg-$(TAG); do \
 		docker image tag calicovpp/$$image localhost:5000/calicovpp/$$image ; \
 		docker push localhost:5000/calicovpp/$$image ; \
 	done
 
+# vpp: build VPP from source. VPP_DIR overrides the default build location
+# (default: vpp_build/ at the repo root). BASE overrides the VPP commit.
+.PHONY: vpp
+vpp:
+	$(MAKE) -C pkg/vpp-manager $@ VPP_DIR=$(VPP_DIR) BASE=$(BASE)
+
+# kube-test-template: emit a self-contained kube-test manifest for the
+# the requested cluster flavor. All CalicoVPP-internal layout details
+# (image names, in-repo paths to the VPP build output, etc.) are
+# resolved here from the kustomize sources under
+# yaml/overlays/kube-test-* and baked into the emitted manifest.
+#
+# Only consumer-owned runtime placeholders survive in the output:
+#   ${CALICOVPP_VERSION}, ${HOME}, ${ADDITIONAL_VPP_CONFIG},
+#   ${CALICOVPP_ENABLE_MEMIF}, ${CALICO_NETWORK_CONFIG} and
+#   ${CALICOVPP_INTERFACE} (baremetal only)
+#
+# The consumer (e.g. VPP `extras/kube-test`) does NOT need to know
+# which image to use or where the VPP build lives in this repo.
+#
+# Usage:
+#   make kube-test-template FLAVOR=kind        # default
+#   make kube-test-template FLAVOR=baremetal
+FLAVOR ?= kind
+.PHONY: kube-test-template
+kube-test-template:
+	@if [ "$(FLAVOR)" != "kind" ] && [ "$(FLAVOR)" != "baremetal" ]; then \
+	  echo "kube-test-template: unsupported FLAVOR=$(FLAVOR) (expected kind or baremetal)" >&2; \
+	  exit 2; \
+	fi
+	@kubectl kustomize yaml/overlays/kube-test-$(FLAVOR) 2>/dev/null
+	@echo "---"
+	@cat yaml/components/kube-test/tigera-installation.snippet.yaml
+
+# kube-test-push-images: pull the unified calicovpp/vpp image from docker.io
+# and push to localhost:5000 for kube-test consumption. Called by the VPP
+# kube-test `release-cluster` flow. From v3.33 onwards all binaries ship in
+# the single calicovpp/vpp image; split-image release branches (v3.29-v3.32)
+# override KUBE_TEST_IMAGES to include calicovpp/agent and multinet-monitor.
+KUBE_TEST_IMAGES ?= calicovpp/vpp
+.PHONY: kube-test-push-images
+kube-test-push-images:
+	@for img in $(KUBE_TEST_IMAGES); do \
+	  docker pull docker.io/$$img:$(CALICOVPP_VERSION); \
+	  docker image tag docker.io/$$img:$(CALICOVPP_VERSION) localhost:5000/$$img:$(CALICOVPP_VERSION); \
+	  docker push localhost:5000/$$img:$(CALICOVPP_VERSION); \
+	done
+
 .PHONY: kind-cluster-name
 kind-cluster-name:
 	@echo $(CLUSTER_NAME)
@@ -46,32 +90,26 @@ kind: kind-new-cluster image-kind kind-install-cni
 
 .PHONY: push
 push:
-	$(MAKE) -C calico-vpp-agent $@
-	$(MAKE) -C vpp-manager $@
-	$(MAKE) -C multinet-monitor $@
+	$(MAKE) -C pkg/vpp-manager $@
 	$(MAKE) -C cmd/calicovppctl $@
 
 .PHONY: dev
 dev:
-	$(MAKE) -C calico-vpp-agent ALSO_LATEST=y $@
-	$(MAKE) -C vpp-manager ALSO_LATEST=y $@
-	$(MAKE) -C multinet-monitor ALSO_LATEST=y $@
+	$(MAKE) -C pkg/vpp-manager ALSO_LATEST=y $@
 
 .PHONY: clean-vpp
 clean-vpp:
-	$(MAKE) -C vpp-manager clean-vpp
+	$(MAKE) -C pkg/vpp-manager clean-vpp
 
 .PHONY: proto
 proto:
-	$(MAKE) -C calico-vpp-agent $@
+	$(MAKE) -C pkg/calico-vpp-agent $@
 
 .PHONY: dev.k3s
 dev.k3s: dev
-	@for x in agent vpp ; do \
-		docker save -o /tmp/$$x.tar calicovpp/$$x:latest ; \
-		sudo k3s ctr images import /tmp/$$x.tar ; \
-		rm -f /tmp/$$x.tar ; \
-	done
+	docker save -o /tmp/vpp.tar calicovpp/vpp:latest
+	sudo k3s ctr images import /tmp/vpp.tar
+	rm -f /tmp/vpp.tar
 
 .PHONY: dev-kind
 dev-kind: dev
@@ -111,9 +149,7 @@ start-test-cluster:
 
 .PHONY: load-images
 load-images:
-	$(MAKE) -C test/vagrant load-image -j99 IMG=calicovpp/agent:latest
 	$(MAKE) -C test/vagrant load-image -j99 IMG=calicovpp/vpp:latest
-	$(MAKE) -C test/vagrant load-image -j99 IMG=calicovpp/multinet-monitor:latest
 
 CALICO_INSTALLATION ?= installation-default
 .PHONY: test-install-calico
@@ -216,9 +252,9 @@ restart-calicovpp:
 	kubectl -n calico-vpp-dataplane rollout status ds/calico-vpp-node
 
 .PHONY: goapi
-export VPP_DIR ?= $(shell pwd)/vpp-manager/vpp_build
+export VPP_DIR ?= $(shell pwd)/vpp_build
 goapi:
-	@go generate -v ./vpplink/generated/
+	@go generate -v ./pkg/vpplink/generated/
 
 .PHONY: cherry-vpp
 cherry-vpp:
@@ -231,13 +267,13 @@ cherry-vpp:
 		echo "branch    : $(shell cd ${VPP_DIR} && git branch --show-current)"; \
 		echo "Are you sure? [y/N] " && read ans && [ $${ans:-N} = y ]; \
 	fi
-	@BASE=$(BASE) bash ./vpplink/generated/vpp_clone_current.sh ${VPP_DIR}
+	@BASE=$(BASE) bash ./pkg/vpplink/generated/vpp_clone_current.sh ${VPP_DIR}
 	@make goapi
 
 .PHONY: cherry-wipe
 cherry-wipe:
-	rm -rf ./vpplink/binapi/.cherries-cache
-	rm -rf ./vpplink/generated/.cherries-cache
+	rm -rf ./pkg/vpplink/binapi/.cherries-cache
+	rm -rf ./pkg/vpplink/generated/.cherries-cache
 
 .PHONY: yaml
 yaml:
@@ -316,14 +352,14 @@ delete-multinet:
 
 .PHONY: lint
 lint:
-	test -d ${VPP_DATAPLANE_DIR}/vpp-manager/vpp_build && touch ${VPP_DATAPLANE_DIR}/vpp-manager/vpp_build/go.mod || true
+	test -d ${VPP_DATAPLANE_DIR}/vpp_build && touch ${VPP_DATAPLANE_DIR}/vpp_build/go.mod || true
 	go mod tidy --diff || (echo -e "Please run\ngo mod tidy" && exit 1)
 	gofmt -s -l . | grep -vE '(binapi|vpp_build|vendor)' \
 		| diff -u /dev/null - \
 		|| (echo -e "Please run\ngofmt -w ." && exit 1)
 	golangci-lint run --color=never
 	markdownlint --dot \
-		--ignore vpp-manager/vpp_build \
+		--ignore vpp_build \
 		--ignore vendor .
 
 .PHONY: cov-html
@@ -374,8 +410,8 @@ builder-image: ## Make dependencies image. (Not required normally; is implied in
 test: builder-image
 	@rm -rf $(shell pwd)/.coverage/unit
 	@mkdir -p $(shell pwd)/.coverage/unit
-	$(MAKE) -C vpp-manager image
-	$(MAKE) -C vpp-manager mock-pod-image
+	$(MAKE) -C pkg/vpp-manager image
+	$(MAKE) -C pkg/vpp-manager mock-pod-image
 	# we prevent parallel test execution as test infra does not currently support parallel VPPs
 	sudo -E env "PATH=$$PATH" VPP_BINARY=/usr/bin/vpp \
 		VPP_IMAGE=calicovpp/vpp:$(TAG) \
@@ -393,8 +429,8 @@ test: builder-image
 ci-test: builder-image
 	@rm -rf $(shell pwd)/.coverage/unit
 	@mkdir -p $(shell pwd)/.coverage/unit
-	$(MAKE) -C vpp-manager image
-	$(MAKE) -C vpp-manager mock-pod-image
+	$(MAKE) -C pkg/vpp-manager image
+	$(MAKE) -C pkg/vpp-manager mock-pod-image
 	# we prevent parallel test execution as test infra does not currently support parallel VPPs
 	docker run -t --rm \
 		--privileged \
@@ -428,4 +464,3 @@ ci-%: builder-image
 .PHONY: depend-image-hash
 depend-image-hash:
 	@echo $(CI_BUILDER_IMAGE)
-

calico-vpp-agent/Dockerfile

@@ -25,7 +25,7 @@ import (
 	"github.com/sirupsen/logrus"
 	"gopkg.in/tomb.v2"
 
-	"github.com/projectcalico/vpp-dataplane/v3/config"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/config"
 )
 
 const (

calico-vpp-agent/Makefile

@@ -36,15 +36,15 @@ import (
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/client-go/rest"
 
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/cni"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/common"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/connectivity"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/felix"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/health"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/routing"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/services"
-	"github.com/projectcalico/vpp-dataplane/v3/calico-vpp-agent/watchers"
-	"github.com/projectcalico/vpp-dataplane/v3/config"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/cni"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/common"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/connectivity"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/felix"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/health"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/routing"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/services"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/calico-vpp-agent/watchers"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/config"
 )
 
 /*

…p-agent/cmd/api-proxy/felix-api-proxy.go cmd/api-proxy/main.gocalico-vpp-agent/cmd/api-proxy/felix-api-proxy.go renamed to cmd/api-proxy/main.go calico-vpp-agent/cmd/api-proxy/felix-api-proxy.go renamed to cmd/api-proxy/main.go

@@ -1,6 +1,6 @@
 include ../../common.mk
 
-BIN_DIR = $(VPP_DATAPLANE_DIR)/bin
+BIN_DIR = ./bin
 SNAPSHOTTER_ENABLED := $(shell docker info -f '{{json .DriverStatus }}' | grep snapshotter)
 
 .PHONY: bin

…co-vpp-agent/cmd/calico_vpp_dataplane.go cmd/calico-vpp-agent/main.gocalico-vpp-agent/cmd/calico_vpp_dataplane.go renamed to cmd/calico-vpp-agent/main.go calico-vpp-agent/cmd/calico_vpp_dataplane.go renamed to cmd/calico-vpp-agent/main.go

@@ -0,0 +1,24 @@
+// Copyright (C) 2019 Cisco Systems Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+// implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	multinetmonitor "github.com/projectcalico/vpp-dataplane/v3/pkg/multinet-monitor"
+)
+
+func main() {
+	multinetmonitor.Run()
+}

cmd/calicovppctl/Makefile

@@ -35,9 +35,9 @@ import (
 	"go.fd.io/govpp/codec"
 	"go.fd.io/govpp/core"
 
-	_ "github.com/projectcalico/vpp-dataplane/v3/vpplink"
-	"github.com/projectcalico/vpp-dataplane/v3/vpplink/generated/bindings/af_packet"
-	"github.com/projectcalico/vpp-dataplane/v3/vpplink/generated/bindings/tapv2"
+	_ "github.com/projectcalico/vpp-dataplane/v3/pkg/vpplink"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/vpplink/generated/bindings/af_packet"
+	"github.com/projectcalico/vpp-dataplane/v3/pkg/vpplink/generated/bindings/tapv2"
 )
 
 var (