Skip to content

Commit e14f155

Browse files
weiclaude
authored andcommitted
fix(mysql): orc switchover must fail closed when Orchestrator is unreachable
Stop merging orchestrator-client stderr into command substitutions: error text was parsed as a master name / instance list, making the two skip-guards exit 0 and a failed switchover report success. Check the client exit code explicitly and exit non-zero on query failure; count instances from non-empty lines only; include the real exit code in the failure message. Fixes #3077 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent b55707f commit e14f155

1 file changed

Lines changed: 16 additions & 7 deletions

File tree

addons/mysql/scripts/orc-switchover.sh

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,14 @@ if [[ "$KB_SWITCHOVER_ROLE" != "primary" ]]; then
2121
exit 0
2222
fi
2323

24-
# Skip if KB_SWITCHOVER_CURRENT_NAME is not the master
25-
master_from_orc=$(/kubeblocks/orchestrator-client -c which-cluster-master -i "${KB_SWITCHOVER_CURRENT_NAME}" 2>&1)
26-
if [ -z "$master_from_orc" ]; then
27-
mysql_error "Could not determine current master from Orchestrator"
24+
# Skip if KB_SWITCHOVER_CURRENT_NAME is not the master.
25+
# Keep stderr out of the command substitution: on an Orchestrator outage the
26+
# client's error text must not be mistaken for a master name (it previously
27+
# made this guard exit 0 and the switchover was silently skipped as success).
28+
master_from_orc=$(/kubeblocks/orchestrator-client -c which-cluster-master -i "${KB_SWITCHOVER_CURRENT_NAME}")
29+
rc=$?
30+
if [ $rc -ne 0 ] || [ -z "$master_from_orc" ]; then
31+
mysql_error "Could not determine current master from Orchestrator (rc=${rc})"
2832
fi
2933

3034
if [ "${KB_SWITCHOVER_CURRENT_NAME}" != "${master_from_orc%%:*}" ]; then
@@ -33,8 +37,13 @@ if [ "${KB_SWITCHOVER_CURRENT_NAME}" != "${master_from_orc%%:*}" ]; then
3337
fi
3438

3539
# Skip switch if there is only one instance
36-
instance_count=$(/kubeblocks/orchestrator-client -c which-cluster-instances -i "${KB_SWITCHOVER_CURRENT_NAME}" 2>&1 | wc -l)
37-
if [ "$instance_count" -eq 1 ]; then
40+
instances=$(/kubeblocks/orchestrator-client -c which-cluster-instances -i "${KB_SWITCHOVER_CURRENT_NAME}")
41+
rc=$?
42+
if [ $rc -ne 0 ]; then
43+
mysql_error "Could not list cluster instances from Orchestrator (rc=${rc})"
44+
fi
45+
instance_count=$(printf '%s\n' "$instances" | sed '/^$/d' | wc -l)
46+
if [ "$instance_count" -le 1 ]; then
3847
mysql_note "Only one instance in cluster, cannot switchover."
3948
exit 0
4049
fi
@@ -55,5 +64,5 @@ else
5564
fi
5665

5766
if [ $exit_code -ne 0 ]; then
58-
mysql_error "Switchover command failed with exit code: ${result}"
67+
mysql_error "Switchover command failed with exit code ${exit_code}: ${result}"
5968
fi

0 commit comments

Comments
 (0)