systemd ip restrictions really do restrict

Now all my machine configs are in Ansible, I've been making things a bit more organised and safer, for instance by allowing SSH from a whitelisted host in case I break Tailscale. I saw systemd had support for restricting IP access to a service, so decided to give it ago. It's easy - just add in a dropin file, like /etc/systemd/system/ssh.service.d/override.conf:

[Service]
IPAddressDeny=any
# home
IPAddressAllow=1.2.3.4
# some other trusted server
IPAddressAllow=2001::1

Which does indeed work! Before this change is applied, I an login from any host. After, I can only log in via the whitelisted host. Trebles all around. Put it in an Ansible role for one host to let the change soak, and then off we go. I noticed a bit later, though, that that Ansible was no longer working on that host. SSHing (from one of the listed hosts, which works) works great, but then sudo apt update just spins, first failing to connect to some IPv4 addresses, then failing to connect to some IPv6 addresses.

As far as I can tell, it turns out that the above systemd option works by injecting BPF in to the kernel for this little cgroup, which does indeed block inbound SSH, but also blocks traffic for all things invoked under SSH, for instance Ansible, and all interactive users. On the web, I found lots of people suggesting using IPAddressDeny for SSH (one, two, three), but zero people pointing out that it's basically useless since it will make SSH shells useless for all users logging in to it.