Skip to content

Search: sharded cluster operator support#816

Open
lsierant wants to merge 15 commits intosearch/basefrom
search/sharded-cluster
Open

Search: sharded cluster operator support#816
lsierant wants to merge 15 commits intosearch/basefrom
search/sharded-cluster

Conversation

@lsierant
Copy link
Contributor

@lsierant lsierant commented Feb 20, 2026

Based on PR #806

Chain of upstream PRs as of 2026-03-03

Summary

MCK already supported deploying single instance of mongot process using MongoDBSearch resource, with the MongoDB replicaset as source. That meant the customers can run search queries/use search against a ReplicaSet deployment.
In this PR, we are adding support to have sharded cluster as source to the MongoDBSearch resource. So that search can be used with sharded clusters as well.

To achieve this we have added the new field in the ExternalMongoDBSource type that can be used to configure the details of the sharded cluster that is going to be used as source.

type ExternalMongoDBSource struct {
	// ShardedCluster contains configuration for external sharded MongoDB clusters.
	// Mutually exclusive with HostAndPorts.
	// +optional
	ShardedCluster *ExternalShardedClusterConfig `json:"shardedCluster,omitempty"`

The ExternalShardedClusterConfig is mainly used to generate the mongot config, so that mongot correctly knows how to talk to the mongod processes. Rest of the changes revolve around this.

Proof of Work

TBD

// Watch for MongoDBSearch resources that reference ShardedCluster MongoDB resources
// Only enqueue reconciliation requests for ShardedCluster resources, not ReplicaSet or Standalone
shardedKubeClient := mgr.GetClient()
err = c.Watch(source.Kind(mgr.GetCache(), &searchv1.MongoDBSearch{},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have now two mechanisms for watching related resources. I'm wondering if we could align it with controllers/operator/watch/resource_watcher.go one way or another

}
}

func TestValidatePerShardTLSSecretsAllExist(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should have more tests testing full reconcile, checking all the expected resource names, mongot configs, etc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those individual tests checking only validatePerShardTLSSecrets, validatePerShardTLSSecrets should be removed and covered in a bigger test case

}

// CreateShardSearchStatefulSetFunc creates a StatefulSet for a specific shard's mongot deployment.
func CreateShardSearchStatefulSetFunc(mdbSearch *searchv1.MongoDBSearch, shardedSource ShardedSearchSourceDBResource, shardIdx int, searchImage string) statefulset.Modification {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be unified with the replicaset mongot

return xerrors.Errorf("error creating/updating shard search service %v: %w", svcName, err)
}

zap.S().Debugf("Updated shard search service %v: %s", svcName, op)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we seem to be using zap.S().Debugf and log.Debugf , is that intentional? Should we just use one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should use log passed on the arguments. In practice it's the same thing, but zap.S() is just returning a top level default logger

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed
75e00dc

Copy link
Contributor Author

evergreen retry

Copy link
Contributor Author

evergreen refresh

@lsierant lsierant mentioned this pull request Feb 27, 2026
3 tasks
@lsierant lsierant force-pushed the search/sharded-cluster branch from 6548799 to e528c67 Compare March 1, 2026 08:50
@lsierant lsierant force-pushed the search/sharded-cluster branch from 75e00dc to 4c82ef8 Compare March 2, 2026 13:06
lsierant added a commit that referenced this pull request Mar 3, 2026
<!-- start git-machete generated -->

# Based on PR #817

## Chain of upstream PRs as of 2026-03-03

* PR #806:
  `master` ← `search/base`

  * PR #816:
    `search/base` ← `search/sharded-cluster`

    * PR #817:
      `search/sharded-cluster` ← `search/multiple-mongot`

      * **PR #853 (THIS ONE)**:
`search/multiple-mongot` ← `search/lsierant/revert-rs-cluster-index`

<!-- end git-machete generated -->
@lsierant lsierant force-pushed the search/sharded-cluster branch from 4c82ef8 to f36f5bb Compare March 3, 2026 21:26
lsierant added a commit that referenced this pull request Mar 3, 2026
<!-- start git-machete generated -->

# Based on PR #817

## Chain of upstream PRs as of 2026-03-03

* PR #806:
  `master` ← `search/base`

  * PR #816:
    `search/base` ← `search/sharded-cluster`

    * PR #817:
      `search/sharded-cluster` ← `search/multiple-mongot`

      * **PR #853 (THIS ONE)**:
`search/multiple-mongot` ← `search/lsierant/revert-rs-cluster-index`

<!-- end git-machete generated -->
@lsierant lsierant requested a review from a team as a code owner March 6, 2026 13:30
@lsierant lsierant requested review from s-urbaniak and removed request for a team March 6, 2026 13:30
lsierant and others added 15 commits March 6, 2026 14:30
Add sharded cluster support to the MongoDBSearch operator:
- Per-shard mongot StatefulSet, Service, and ConfigMap resources
- ShardedEnterpriseSearchSource for operator-managed sharded clusters
- ShardedExternalSearchSource for external sharded MongoDB sources
- Per-shard and shared TLS certificate handling (CertsSecretPrefix)
- Sharded cluster controller applies per-shard mongod and mongos search
  config parameters pointing to internal shard-local mongot services
- ReplicaSet controller skips search config for sharded clusters
  (handled by sharded controller instead)
- ExternalShardedConfig/ExternalRouterConfig/ExternalShardConfig API types
- CRD updated with sharded external source fields and per-shard TLS prefix
Missing e2e test for sharded search with one mongot instance
# Summary

Addressing some of the review comments from the PR.

## Proof of Work

<!-- Enter your proof that it works here.-->

## Checklist

- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [ ] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details
…loyment` (#832)

# Summary

<!-- Enter your PR summary here. Try to emphasize on WHY this change is
needed, followed by what's being done in the PR. -->

## Proof of Work

<!-- Enter your proof that it works here.-->

## Checklist

- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [ ] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details
…#834)

## Summary
- Unify duplicate sharded vs non-sharded functions in the search
controller into single parameterized versions
- Extract shared config and service helpers (baseMongotConfig,
mongotServicePorts, buildSearchSetParameters)
- Fix missing owner references on non-sharded StatefulSet and Service
(was already set on sharded path)

## Proof of Work
Compiles, existing tests pass. Behavioral change is limited to adding
`SetOwnerReference` on non-sharded resources (aligning with the sharded
path).

## Checklist
- [x] Have you linked a jira ticket and/or is the ticket in the title?
- [x] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?

---------

Co-authored-by: Łukasz Sierant <lukasz.sierant@mongodb.com>
@lsierant lsierant force-pushed the search/sharded-cluster branch from f36f5bb to f36ca1e Compare March 6, 2026 13:30
lsierant added a commit that referenced this pull request Mar 6, 2026
<!-- start git-machete generated -->

# Based on PR #817

## Chain of upstream PRs as of 2026-03-03

* PR #806:
  `master` ← `search/base`

  * PR #816:
    `search/base` ← `search/sharded-cluster`

    * PR #817:
      `search/sharded-cluster` ← `search/multiple-mongot`

      * **PR #853 (THIS ONE)**:
`search/multiple-mongot` ← `search/lsierant/revert-rs-cluster-index`

<!-- end git-machete generated -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants