Rolling upgrade test (BATS, Docker)

[dsmiley]

I spent a bunch of time with a GitHub Copilot Agent to develop a new BATS test to test a rolling upgrade of SolrCloud.
It ignores the current project/build since it runs Docker containers of Solr.

I think this test should not be run by default; it's a slow test. Not sure yet on the best way to segment this test and maybe other slow ones. At least another directory.

It could be improved to build the local project in Docker and use that image as the upgrade destination target.

Separately, I want to use this as a basis to check for Overseer disablement settings, but that's not present here. Maybe I'll add it in this PR.

Disclaimer: Bash is not my comfort zone, but AI churned this out. IntelliJ Junie was also super helpful.

[janhoy]

Idea: Soon, solr will have Testcontainers as a dependency (#3670), and Testcontainers is ideal for orchestrating and managing containers like this. I suppose you can write a JUnit test that sets up a docker network, then adds 3 Solr containers in cloud mode, then write a loop where you swap one after the other with a newer version and inspect logs or what you need.

It makes sense for such a test to be disabled by default or included in nightly.

Here's Claude's draft stab at a rolling test: https://claude.ai/public/artifacts/c4d09d7b-df54-419a-b660-169e22191a26

[dsmiley]

Idea: Soon, solr will have Testcontainers as a dependency (#3670), and Testcontainers is ideal for orchestrating and managing containers like this.

Thank you for #3670 but you say having a dependency (on Testcontainers) as if we've achieved something as a project. It's a few lines of configuration to depend on it. An achievement would be https://issues.apache.org/jira/browse/SOLR-17653 albeit that particular issue wouldn't be helpful for the upgrade test.

I suppose you can write a JUnit test that sets up a docker network, then adds 3 Solr containers in cloud mode, then write a loop where you swap one after the other with a newer version and inspect logs or what you need.

It makes sense for such a test to be disabled by default or included in nightly.

Here's Claude's draft stab at a rolling test: https://claude.ai/public/artifacts/c4d09d7b-df54-419a-b660-169e22191a26

I am a huge fan of Testcontainers; I filed SOLR-17653 after all. I would have personally enjoyed writing/working with Java code to manipulate containers instead of Bash/BATS. I could have used my desire for SOLR-17653 as an excuse here to do it. I was awefully tempted. I contemplated doing that very thing and I had a 1:1 call with Eric Pugh about this rolling upgrade test to discuss how to go about it. I chose Bash/BATS in spite of disliking the framework. I am no good at this framework; GitHub Copilot did the critical 1st draft and most improvements afterwards. The rolling upgrade test scenario doesn't speak to the strengths of Testcontainers specifically (JUnit wrapper for Docker). It's to the strenghts of Docker, yes, put not specifically Testcontainers/JUnit. This is just orchestration here (Not SolrJ integration which definitely requires SOLR-17653), which is basically what our pile of Bash/BATS scripts do. We have a pile of similar Docker tests in Bash too, albeit not using BATS. Together (:solr:package:integrationTests & :solr:docker:testDocker) are high level integration tests, which is what the test here definitely is.

Put differently, if this specific test were to exist as a JUnit/Testcontainers based test, then it really calls into question why we have any Bash/BATS scripts at all -- why would/shouldn't they also be TestContainers based? "When in Rome, do as the romans do" -- I respected our status quo because it works, even though I'm not productive in it.

[epugh]

Did our Solr 10 base image change and remove curl?

We changed base image in #3782, and I can see that it does not have cURL. Perhaps we should edit the Dockerfile to add it?

i thought we changed out back to having curl in it?

[epugh]

Did our Solr 10 base image change and remove curl?

We changed base image in #3782, and I can see that it does not have cURL. Perhaps we should edit the Dockerfile to add it?

i thought we changed out back to having curl in it?

#3803 was where we restored curl.

[epugh]

I tried to use the new wait_for method but didn't have any luck. So instead, let's just have better handling for when timeout isn't available.

[epugh]

I wonder if this should be using a run and a assert_success type bats calls?

[epugh]

I've been doing some testing of Solr 9.7 to 9.10 wiht changing luceneMatchVersion and schema.xml versions, and using a variant of this test. I'm not sure when I have taken this one test too far.....

What would we want it to have to be considred worth merging? SSL setup? oauth based security? Other complex things?

[dsmiley]

What would we want it to have to be considred worth merging? SSL setup? oauth based security? Other complex things?

Gosh; nothing complex to get this useful test merged! I think the leading concern I have is that this test is slow and just different... I would like it to somehow get run separately from a normal integration test run. Like this test should perhaps be opt-in, and we do the opt-in on Jenkins CI but not elsewhere.

[epugh]

@dsmiley I think the one change I want to make is rename it test_rolling_upgrade.bats. The fact that it uses Docker isn't important, and we don't label other tests, like the extraction ones, with the word 'docker'.... WDYT?

I'd love to get this in to use in testing some other scenarios.

[dsmiley]

test_rolling_upgrade.bats is a much better name.

What keeps this heavy test from the normal rotation?

[dsmiley]

why change this test in this PR?

[epugh]

I was hoping to get both of them using better docker patterns, I'll back it out...

[epugh]

test_rolling_upgrade.bats is a much better name.

What keeps this heavy test from the normal rotation?

Will rename. As far as heavy test form normal rotation, it honestly isn't that much heavier than some other tests that run. I think in a seperate PR would should come up with criteria for what is considered heavy, and deal with it there...

[epugh]

@dsmiley how do you feel about me merging this? I'd like to build on it for another scenario...

[dsmiley]

Go for it Eric! Very happy to have team-work where I start something and others take it further.

[dsmiley]

this integration test has been failing every day since March 20th -- https://ci-builds.apache.org/job/Solr/job/Solr-TestIntegration-10.x/

I wish our CI failures would notify the hell out of us individually. Basically nobody monitors the build list unless it's like an act of charity (and I'm in the mood to be charitable at the moment)

[epugh]

Let la like "wait_for" command error? Though I thought that was a bash function we defined???

[epugh]

Okay, looks like we time out after 30 seconds, but that appears from the log like it's way not enough time due to slow machine downloading docker images... That is how I read it.. Before I bump it to 120 seconds or larger, would love it if you thought the same:

# Docker artifacts saved to: /home/jenkins/jenkins-agent/workspace/Solr/Solr-TestIntegration-10.x/solr/packaging/build/test-output/docker
not ok 79 Docker SolrCloud rolling upgrade # in 30596 ms
# (in test file test/test_rolling_upgrade.bats, line 118)
#   `wait_for 30 1 docker exec solr-node1 solr healthcheck -c test-collection' failed with status 127
# 9.10.0-SNAPSHOT-slim: Pulling from apache/solr-nightly
# af6eca94c810: Pulling fs layer
# cb0efb96dabd: Pulling fs layer
# 3e9d91201f40: Pulling fs layer
# 66b76b382631: Pulling fs layer
# 601f2c23751f: Pulling fs layer
# 0ed719aef393: Pulling fs layer
# 69a09c96e8bf: Pulling fs layer
# c937aee9a870: Pulling fs layer
# 9cdfe5320de8: Pulling fs layer
# 3aa9120e46f5: Pulling fs layer
# 0ed719aef393: Waiting
# 69a09c96e8bf: Waiting
# c937aee9a870: Waiting
# 601f2c23751f: Waiting
# 9cdfe5320de8: Waiting
# 4f4fb700ef54: Pulling fs layer
# 3aa9120e46f5: Waiting
# 4f4fb700ef54: Waiting
# cb0efb96dabd: Verifying Checksum
# cb0efb96dabd: Download complete
# 3e9d91201f40: Verifying Checksum
# 3e9d91201f40: Download complete
# 66b76b382631: Verifying Checksum
# 66b76b382631: Download complete
# af6eca94c810: Verifying Checksum
# af6eca94c810: Download complete
# 601f2c23751f: Verifying Checksum
# 601f2c23751f: Download complete
# 69a09c96e8bf: Download complete
# c937aee9a870: Verifying Checksum
# c937aee9a870: Download complete
# 3aa9120e46f5: Download complete
# 9cdfe5320de8: Download complete
# af6eca94c810: Pull complete
# 4f4fb700ef54: Verifying Checksum
# 4f4fb700ef54: Download complete
# 0ed719aef393: Verifying Checksum
# 0ed719aef393: Download complete
# cb0efb96dabd: Pull complete
# 3e9d91201f40: Pull complete
# 66b76b382631: Pull complete
# 601f2c23751f: Pull complete
# 0ed719aef393: Pull complete
# 69a09c96e8bf: Pull complete
# c937aee9a870: Pull complete
# 9cdfe5320de8: Pull complete
# 3aa9120e46f5: Pull complete
# 4f4fb700ef54: Pull complete
# Digest: sha256:15c8155ceffe1042f37c77bb537aa531804a3510702ff87f2af5fedf523f4203
# Status: Downloaded newer image for apache/solr-nightly:9.10.0-SNAPSHOT-sli

[epugh]

actually, that doens't make sense... where the wait_for on line 118 is way after we have pulled hte images, so it's not waiting for the docker image to download to run the command I don't think...

[dsmiley]

From the description of this PR:

I think this test should not be run by default; it's a slow test. Not sure yet on the best way to segment this test and maybe other slow ones. At least another directory.

I still think that. I doubt we'll ever write a more costly test.