ETCD learner members cannot be defragmented, causing persistent storage bloat.

[amolmishra23]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Observed Behavior:

• Learner member DB size remains at 30MB while voting members are successfully defragmented to ~7MB
• etcdctl defrag command silently skips learner endpoints
• Direct defragmentation of learner endpoint fails with: rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner
• Learner database size grows unbounded over time without maintenance capability

Error Message:

etcdserver: rpc not supported for learner
Failed to defragment etcd member[https://181140267029446:25687] (etcdserver: rpc not supported for learner)

What did you expect to happen?

Learner members should support defragmentation operations to maintain database size, either:

• etcdctl defrag should include learner endpoints by default, OR
• etcdctl defrag --endpoints=<learner> should work without errors, OR
• Provide a supported method to defragment learner members (e.g., --include-learners flag)

How can we reproduce it (as minimally and precisely as possible)?

Minimal Reproduction Steps:

# 1. Set up 3-node cluster + 1 learner member
# (cluster already running with learner)

# 2. Check initial status - note learner DB size
etcdctl endpoint status --write-out=table
# Result: Learner at 30MB, voting members at 8.5MB

# 3. Perform compaction 
etcdctl compact 2885950

# 4. Defragment cluster
etcdctl defrag  
# Result: Only voting members defragmented

# 5. Verify learner still has large DB
etcdctl endpoint status --write-out=table
# Result: Voting members ~7MB, learner still 30MB

# 6. Try to defrag learner directly
etcdctl --endpoints=https://181140267029446:25687 defrag
# Result: Fails with "rpc not supported for learner"

Cluster Topology:

• 3 voting members (fb2f8c3838629cdb, 2228c0c31b9ff622, 63d10718366c821d)
• 1 learner member (cf584302b1c47a59)

Anything else we need to know?

Impact:

• Production issue causing learner storage to grow unbounded
• No supported workaround for learner maintenance
• Affects cluster operations in learner-promotion scenarios

Observations:

• The CLI shows --includeLearner flag but it doesn't resolve the underlying RPC limitation
• This appears to be an intentional restriction but creates operational problems

Etcd version (please run commands below)

etcd version: 3.5.25
etcdctl version: 3.5.25

Etcd configuration (command line flags or environment variables)

Cluster Configuration:

• 4-member cluster (3 voting + 1 learner)
• HTTPS endpoints on port 25687
• Standard production setup

Relevant Settings:

--auto-compaction-retention (if applicable)
--quota-backend-bytes (if applicable)
--initial-cluster (4-member setup)

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Member List:

etcdctl member list -w table

Endpoint Status (Before Defrag):

+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://181140264581536:25687 | fb2f8c3838629cdb |  3.5.25 |  8.5 MB |     false |      false |         9 |    3276978 |            3276978 |        |
| https://181140266914826:25687 | 2228c0c31b9ff622 |  3.5.25 |  8.5 MB |     false |      false |         9 |    3276978 |            3276978 |        |
| https://181140266590820:25687 | 63d10718366c821d |  3.5.25 |  8.5 MB |      true |      false |         9 |    3276978 |            3276978 |        |
| https://181140267029446:25687 | cf584302b1c47a59 |  3.5.25 |   30 MB |     false |       true |         9 |    3276980 |            3276980 |        |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Endpoint Status (After Defrag - Issue Persists):

+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://181140264581536:25687 | fb2f8c3838629cdb |  3.5.25 |  7.0 MB |     false |      false |         9 |    3278633 |            3278633 |        |
| https://181140266914826:25687 | 2228c0c31b9ff622 |  3.5.25 |  7.0 MB |     false |      false |         9 |    3278633 |            3278633 |        |
| https://181140266590820:25687 | 63d10718366c821d |  3.5.25 |  7.0 MB |      true |      false |         9 |    3278633 |            3278633 |        |
| https://181140267029446:25687 | cf584302b1c47a59 |  3.5.25 |   30 MB |     false |       true |         9 |    3278633 |            3278633 |        |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

**Defragmentation Failure Logs:**

{"level":"warn","ts":"2026-05-11T08:07:05.721070-0700","logger":"etcd-client","caller":"v3@v3.5.25/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000f61e0/181140264581536:25687","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
...
{"level":"warn","ts":"2026-05-11T08:07:06.622187-0700","logger":"etcd-client","caller":"v3@v3.5.25/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000f61e0/181140264581536:25687","attempt":99,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
Failed to defragment etcd member[https://181140267029446:25687] (etcdserver: rpc not supported for learner)

[silentred]

• etcdctl defrag --endpoints=<learner> should work without errors,

This command should work, pls try it out.

[amolmishra23]

Hi @silentred, can you PTAL at the output below?

[admin@sac01-haswell17-bqkp91500051-node-3 ~]$ etcdctl endpoint status --write-out=table
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://181140264581536:25687 | fb2f8c3838629cdb |  3.5.25 |  6.6 MB |     false |      false |         9 |    3278045 |            3278045 |        |
| https://181140266914826:25687 | 2228c0c31b9ff622 |  3.5.25 |  6.6 MB |     false |      false |         9 |    3278045 |            3278045 |        |
| https://181140266590820:25687 | 63d10718366c821d |  3.5.25 |  6.6 MB |      true |      false |         9 |    3278045 |            3278045 |        |
| https://181140267029446:25687 | cf584302b1c47a59 |  3.5.25 |   30 MB |     false |       true |         9 |    3278045 |            3278045 |        |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+



[admin@sac01-haswell17-bqkp91500051-node-3 ~]$ etcdctl --endpoints=https://181140267029446:25687 defrag
Finished defragmenting etcd member[https://181140264581536:25687]
Finished defragmenting etcd member[https://181140266914826:25687]
Finished defragmenting etcd member[https://181140266590820:25687]
{"level":"warn","ts":"2026-05-11T08:07:05.721070-0700","logger":"etcd-client","caller":"v3@v3.5.25/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000f61e0/181140264581536:25687","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
{"level":"warn","ts":"2026-05-11T08:07:05.721689-0700","logger":"etcd-client","caller":"v3@v3.5.25/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000f61e0/181140264581536:25687","attempt":1,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
....
{"level":"warn","ts":"2026-05-11T08:07:06.595292-0700","logger":"etcd-client","caller":"v3@v3.5.25/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000f61e0/181140264581536:25687","attempt":98,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
{"level":"warn","ts":"2026-05-11T08:07:06.622187-0700","logger":"etcd-client","caller":"v3@v3.5.25/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000f61e0/181140264581536:25687","attempt":99,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
Failed to defragment etcd member[https://181140267029446:25687] (etcdserver: rpc not supported for learner)
Error: exit status 1
Usage:
  etcdctl <etcdctl-subcommand> [flags]

Flags:
  -h, --help             help for exec
      --includeLearner   Include learner endpoint for etcdctl utility

exit status 1


[admin@sac01-haswell17-bqkp91500051-node-3 ~]$ etcdctl endpoint status --write-out=table
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT            |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://181140264581536:25687 | fb2f8c3838629cdb |  3.5.25 |  7.0 MB |     false |      false |         9 |    3278633 |            3278633 |        |
| https://181140266914826:25687 | 2228c0c31b9ff622 |  3.5.25 |  7.0 MB |     false |      false |         9 |    3278633 |            3278633 |        |
| https://181140266590820:25687 | 63d10718366c821d |  3.5.25 |  7.0 MB |      true |      false |         9 |    3278633 |            3278633 |        |
| https://181140267029446:25687 | cf584302b1c47a59 |  3.5.25 |   30 MB |     false |       true |         9 |    3278633 |            3278633 |        |
+-------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+


[admin@sac01-haswell17-bqkp91500051-node-3 ~]$

[silentred]

Should we allow defrag request for a learner? @ahrtr
In my opinion, as long as a member uses boltdb, it requires access to defragment. So it is necessary for voter and learner members to do defragment. Witness member may not need it, because it only saves WALs.

etcd/server/etcdserver/api/v3rpc/util.go

Lines 141 to 150 in c0aa3b6

	func isRPCSupportedForLearner(req any) bool {
	switch r := req.(type) {
	case *pb.StatusRequest:
	return true
	case *pb.RangeRequest:
	return r.Serializable
	default:
	return false
	}
	}

[serathius]

Learner should not be a long term status, it should be promoted within finite time. Not sure there is a value of implementing defrag for it.

[silentred]

Maybe we could discuss it here #21756

[puneetdixit200]

I’d like to work on this. I can reproduce learner defrag behavior, add coverage for learner-member defragmentation, and patch the path that blocks cleanup or leaves storage bloat unresolved. Please assign this to me

[kunalworldwide]

The "learners should be short-lived" argument is correct in the happy path, but learners stay around longer than intended in practice — network partitions delaying promotion, operators being cautious before promoting, bootstrap scenarios where something goes wrong mid-process. In those cases the learner's DB grows from compaction (which frees logical space) but not defrag (which frees physical space).

The "remove and re-add" workaround is operationally risky — removing a learner loses the fast-sync candidate and means the new member replays the full log. On a cluster with a large dataset that's a meaningful time and I/O cost.

The rpc not supported for learner check in util.go is intentionally conservative, but the storage path for learners is identical to voters — there's nothing inherently unsafe about defragging one. A pragmatic middle ground would be allowing defrag on learner endpoints and documenting that learners are excluded from quorum, so defrag doesn't affect cluster consensus.