ZFS micro incompatibilities
I've been setting up ZFS replication between some servers, but between one pair, every sync was failed with:
Nov 30 11:07:11 sourcehost syncoid[890260]: Sending incremental tank/enc/var/spool@autosnap_2025-11-27_20:00:22_monthly ... autosnap_2025-11-30_11:00:39_hourly (~ 555 KB):
Nov 30 11:07:12 sourcehost syncoid[890982]: warning: cannot send 'tank/enc/var/spool@autosnap_2025-11-27_20:00:22_weekly': Invalid argument
Source machine (which was on the way to being upgraded to trixie but hadn't been rebooted yet):
root@sourcehost:~# uname -a
Linux sourcehost 6.1.0-37-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.140-1 (2025-05-22) x86_64 GNU/Linux
root@sourcehost:~# zfs version
zfs-2.3.2-2
zfs-kmod-2.3.1-1~bpo12+1
I had assumed it was some incompatibility with the destination machine and spent ages trying to track that
down until I read the error more carefully - cannot send, ie the problem is with the zfs
tool on
sourcehost, before it even talks to the destination.
Eventually I found a
reference to a bug in
openzfs 2.3.1 that broke the user-space API, and then 2.3.2
reverted the breakage, which broke compatibility with 2.3.1 itself. Going through PRs from
between 2.3.1 and 2.3.2, it's
this, and here's the explanation of the accidental ABI change change
A reboot in to the new kernel with the new, same-versioned zfs
userspace and kernel module fixed it.