[Bug] v3.7.0-beta.0: Control-Plane Deadlock and Cluster Consensus Availability Loss During Single-Node Downgrade

[Champbreed]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

A operational logic deadlock occurs in v3.7.0-beta.0 during downgrade handling. When a single-node etcd instance triggers a minor-version rollback (downgrade enable 3.6.0), the system enters a deadlock state instead of executing a schema transition or safely rejecting the request. If this state coincides with high transaction volume or cluster membership adjustments, the backend storage and Raft consensus mechanics degrade, leading to permanent DeadlineExceeded errors across all KV and Maintenance API endpoints.

What did you expect to happen?

• Single-Node Optimization: The cluster should detect it is a single-node deployment (100% of the quorum) and execute the local schema downgrade without requiring multi-node consensus validation.

• Graceful Handling: Contradictory pipelines (e.g., downgrade enable vs downgrade cancel) should be serialized to avoid storage-layer context timeouts.

• Build Consistency: The module compilation graph should track v3.7.x upstream library sources uniformly, avoiding the mixed-state logs currently observed referencing v3.6.0-beta.0 internal module paths.

How can we reproduce it (as minimally and precisely as possible)?

How can we reproduce it?

1. Initialize a single-node instance:

/tmp/etcd-download-test/etcd --data-dir=/tmp/etcd-migration-test

2. Submit the downgrade request:

/tmp/etcd-download-test/etcdctl downgrade enable 3.6.0

3. Simulate topology adjustment during the loop:

/tmp/etcd-download-test/etcdctl member add s2 --peer-urls=http://localhost:2381

4. Stress the system to trigger the deadlock:

for i in {1..50}; do
  /tmp/etcd-download-test/etcdctl put /stress/key$i "value$i" &
  /tmp/etcd-download-test/etcdctl downgrade cancel &
done

Relevant log output

1. Version Monitor Loop

{"level":"warn","ts":"2026-05-20T02:12:32.772798+0100","caller":"version/monitor.go:212","msg":"remotes server has mismatching etcd version","remote-member-id":"8e9e05c52164694d","current-server-version":"3.7.0","target-version":"3.6.0"}
{"level":"warn","ts":"2026-05-20T02:15:35.180190+0100","caller":"etcdserver/v3_server.go:1260","msg":"reject downgrade request","error":"etcdserver: invalid downgrade target version"}

2. Raft Consensus Deadlock

{"level":"info","ts":"2026-05-20T02:31:25.635324+0100","logger":"raft","caller":"v3@v3.6.0-beta.0.0.20260116184858-6d944ca211ee/raft.go:930","msg":"8e9e05c52164694d became pre-candidate at term 2"}
{"level":"warn","ts":"2026-05-20T02:31:25.800941+0100","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"91fba30a589a5910","rtt":"0s","error":"dial tcp 127.0.0.1:2381: connect: connection refused"}

3. Client API Timeout

{"level":"warn","ts":"2026-05-20T03:01:42.635561+0100","logger":"etcd-client","caller":"v3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x3d1241a2000/localhost:2379","peer":"Peer{Addr: '127.0.0.1:2379', LocalAddr: '127.0.0.1:32916', AuthInfo: 'insecure'}","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL"}

Anything else we need to know?

This bug was reliably verified inside an isolated local testing sandbox environment. Combining a pending version rollback state machine transition with rapid-fire infrastructure lifecycle alterations completely breaks the Raft voting matrix, locking the single node into an irreversible candidate election loop.

Etcd version (please run commands below)

Details

$ etcd --version
etcd Version: 3.7.0-beta.0
Git SHA: 7ee95a6
Go Version: go1.26.3
Go OS/Arch: linux/amd64

$ etcdctl version
etcdctl version: 3.7.0-beta.0
API version: 3.7

Etcd configuration (command line flags or environment variables)

Details

Launched as a single-node instance via standard CLI invocation:

/tmp/etcd-download-test/etcd --data-dir=/tmp/etcd-migration-test

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details

Standard state table diagnostics cannot execute natively due to the operational deadlock. Client network contexts are instantly dropped via a gRPC stream termination event:

$ etcdctl member list -w table
Error: context deadline exceeded

$ etcdctl endpoint status -w table
Error: context deadline exceeded

Relevant log output

[serathius]

Thanks you for testing the release! Will take a look today

/cc @fuweid @ahrtr @Jefftree @siyuanfoundation

[serathius]

Simulate topology adjustment during the loop:

I don't think we ever considered membership change during upgrade. The basic downgrade for 1 node cluster should be already tested, but I don't know about downgrade with membership changes.

[fuweid]

I don't think it's issue. the deadlock is caused by non-ready vote member.

I am trying to reproduce this issue by e2e test - fuweid@66865db

{"level":"warn","ts":"2026-05-20T02:15:35.180190+0100","caller":"etcdserver/v3_server.go:1260","msg":"reject downgrade request","error":"etcdserver: invalid downgrade target version"}

You probably run the downgrade twice but you didn't replace that binary. It's not issue to me.

{"level":"warn","ts":"2026-05-20T03:01:42.635561+0100","logger":"etcd-client","caller":"v3/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x3d1241a2000/localhost:2379","peer":"Peer{Addr: '127.0.0.1:2379', LocalAddr: '127.0.0.1:32916', AuthInfo: 'insecure'}","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL"}

Timeout because there is no leader and cluster can't perform your request.

If you add active voting member, it won't be timeout.

it's same behaviour in v3.6.

[Champbreed]

@fuweid You are completely right about the underlying Raft math here. Since adding s2 instantly bumps the quorum requirement to a 2/2 majority while it is still offline, it makes total sense that the cluster loses its leader and throws the expected DeadlineExceeded errors. I see why this is standard behavior across both v3.6 and v3.7.

Just to clarify the reproduction steps regarding the invalid downgrade target version log: the script actually does not invoke downgrade enable twice. The initial enable 3.6.0 is sent exactly once, followed by a rapid concurrency loop executing downgrade cancel alongside the put keyspace stress. The warning error appears to be a side effect of the background state processing the cancel pipeline under heavy transaction load while membership mutations are pending, rather than an out-of-order duplicate command which i reproduce.

The broader design point I wanted to raise is how we handle these overlapping states at the API level. In production environments running automated GitOps loops or infrastructure orchestrators, these actions happen via automated scripts. If a controller automatically triggers a member add expansion at the exact moment a single-node system is evaluating a version downgrade or cancel routine, the API currently lets both operations run at the same time.

While the official documentation states that the best practice is to configure a single member at a time and verify it starts correctly before adding more new members, automated tools often break this sequence under concurrent lifecycle changes. Instead of letting a layout change accidentally tank a single-node footprint during automation stress, it might be worth adding a simple guardrail to the control-plane API. If the server detects an active upgrade or downgrade job processing in the background, it could defensively block membership modifications and return a clear error. This would gracefully prevent cross-subsystem race conditions and make automated lifecycle management a lot more resilient.

[fuweid]

Just to clarify the reproduction steps regarding the invalid downgrade target version log: the script actually does not invoke downgrade enable twice. The initial enable 3.6.0 is sent exactly once, followed by a rapid concurrency loop executing downgrade cancel alongside the put keyspace stress. The warning error appears to be a side effect of the background state processing the cancel pipeline under heavy transaction load while membership mutations are pending, rather than an out-of-order duplicate command which i reproduce.

Could you please prepare script for reproducing? I can't reproduce the issue about reject downgrade request without sending downgrade enable request. The reject downgrade request log is only available in enable path not in cancel .

The broader design point I wanted to raise is how we handle these overlapping states at the API level. In production environments running automated GitOps loops or infrastructure orchestrators, these actions happen via automated scripts. If a controller automatically triggers a member add expansion at the exact moment a single-node system is evaluating a version downgrade or cancel routine, the API currently lets both operations run at the same time.

As mentioned earlier, etcd does not block users from adding members during the downgrade process. This is a valid operation.

For example, consider a 3-member cluster being downgraded from v3.7.0 to v3.6.x. If two members have already been replaced with the v3.6.x binary, but the last member cannot be downgraded because of a host issue, the administrator may need to remove that member and add a new one. In that case, blocking all membership changes during downgrade would prevent a valid recovery path.

From what I can see, the issue here is caused by the script/operation sequence rather than the downgrade process itself. A newly added voting member is immediately counted in quorum, so the corresponding etcd process must be started after the member is added. Adding a voting member without bringing up the actual process will cause quorum loss, which is expected behavior.

While the official documentation states that the best practice is to configure a single member at a time and verify it starts correctly before adding more new members, automated tools often break this sequence under concurrent lifecycle changes.

Automated tools still need to respect the quorum requirements. Adding voting members without starting the corresponding etcd processes will make the cluster unable to meet quorum.

In this case, I think the script should be fixed first. After a voting member is added, the automation should immediately bring up the real etcd process for that member before continuing with more lifecycle operations.

etcd already has safeguards for this class of issue. With strict reconfiguration check enabled, etcd rejects adding more voting members when doing so would break quorum. The first voting member add for a single-member cluster is allowed because it is a valid expansion/recovery path, but the operator is responsible for starting that member.

And you can consider using learner mode. When learner is ready, you can promote it to be voting member. Before that, it won't break quorum

[serathius]

From what I can see, the issue here is caused by the script/operation sequence rather than the downgrade process itself.

If only we had example script/automation for downgrades.

[Champbreed]

@fuweid @serathius Thanks for the feedback. The scenario of replacing a failed host mid-downgrade makes perfect sense. I see how a hard API block would break critical recovery steps for an operator.

To clarify the sequence: the control plane requires registering a new node via member add before its physical process can safely join. However, the moment this configuration change executes, Raft instantly updates the cluster size and quorum requirements. Because the voting member is not yet online to vote, the subsequent keyspace write freezes up immediately due to the resulting quorum loss.

• Execution (Part 1: Adding the Offline Voting Member)

# Enable downgrade on single-node cluster
$ etcdctl downgrade enable 3.6.0
Downgrade enable success, cluster version 3.7

# Add voting member (process NOT started yet)
$ etcdctl member add s2 --peer-urls=http://localhost:2381
Member a018ac564b27f2cf added to cluster cdf818194e3a8c32

# Attempt write - cluster loses quorum immediately
$ etcdctl put /stress/reproCheck "confirmed"
Error: context deadline exceeded

• Server runtime log (Trapped in Pre-Vote Loop)

{"level":"info","ts":"2026-05-22T13:59:57.478529+0100","logger":"raft","caller":"v3@v3.6.0-beta.0.0.20260116184858-6d944ca211ee/raft.go:1064","msg":"8e9e05c52164694d [logterm: 2, index: 6] sent MsgPreVote request to 52453efc7b339bbe at term 2"}
{"level":"info","ts":"2026-05-22T13:59:57.478853+0100","logger":"raft","caller":"v3@v3.6.0-beta.0.0.20260116184858-6d944ca211ee/raft.go:1693","msg":"8e9e05c52164694d has received 1 MsgPreVoteResp votes and 0 vote rejections"}
{"level":"warn","ts":"2026-05-22T13:59:56.454471+0100","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","error":"dial tcp 127.0.0.1:2381: connect: connection refused"}

• Server runtime log (Part 2: Bringing the Voting Member Online to Recover Quorum)

Once the second voting member is physically started on port 2381, the 2/2 voting matrix constraint is satisfied. The pre-vote loop breaks, consensus forms, and the backend storage engines smoothly apply the downgrade transitions directly to the disk ledger:

{"level":"info","ts":"2026-05-22T14:02:04.124663+0100","caller":"apply/backend.go:278","msg":"Cluster version downgrade detected, forcing snapshot","prev-cluster-version":"3.7.0","new-cluster-version":"3.6.0"}
{"level":"info","ts":"2026-05-22T14:02:04.182453+0100","caller":"version/monitor.go:125","msg":"The server is ready to downgrade","target-version":"3.6.0","server-version":"3.7.0-beta.0"}

I think the initial friction simply came down to looking at two different lifecycle steps: the Control Plane API requires registering the configuration intent via member add first, while the Raft Consensus Protocol instantly demands starting the voting member to satisfy the new quorum math. Since etcd lets these operations overlap at the API level, having an official reference script would help tools and operators avoid this exact timing issue.

[fuweid]

I think the initial friction simply came down to looking at two different lifecycle steps: the Control Plane API requires registering the configuration intent via member add first, while the Raft Consensus Protocol instantly demands starting the voting member to satisfy the new quorum math. Since etcd lets these operations overlap at the API level, having an official reference script would help tools and operators avoid this exact timing issue.

You can try learner mode to avoid this issue - https://etcd.io/docs/v3.6/learning/design-learner/