I just want to mirror 10GB

2026-02-05 #debian #sysadmin #apt

I use a few third-party apt repositories on my servers, which is very convenient, but can also be slightly annoying:

kanidm uses GitHub Pages, which still isn't available via IPv6 in the year of our lord 2026
smallstep has some bizarre DNS load balancing config that is both IPv4-only and has long CNAME chains that make resolution unreliable for me (which is possibly my fault!)
incus releases very frequently and removes older packages, and I prefer to pin exact package versions for big complicated (but good! very good!) things like Incus which makes it regularly be uninstallable on a new host because the last release I pinned is gone from their repo

So, it seemed like time to set up a local mirror of this small amount of data that I would like to have reliably available.

debmirror was the obvious first choice for me, but no variation of --ignore="binary-all" that I could contrive would make it stop trying to fetch the non-existent binary-all arch:

Feb 05 10:50:07 myhost systemd[1]: Starting debmirror@kanidm.service - Mirror third-party apt repository: kanidm...
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: === Syncing mirror: kanidm ===
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Host: kanidm.github.io
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Root: /kanidm_ppa
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Dists: bookworm,trixie
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Sections: stable
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Archs: amd64,arm64
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Destination: /srv/apt-mirror/kanidm
Feb 05 10:50:07 myhost debmirror-sync.sh[462970]: Running: debmirror --method=https --host=kanidm.github.io --root=/kanidm_ppa --dist=bookworm,trixie --section=stable --arch=amd64,arm64 --keyring=/etc/apt/keyrings/thirdparty-mirror/kanidm_ppa.gpg --nosource --ignore-missing-release --ignore-small-errors --timeout=300 --rsync-extra=none --diff=none --progress /srv/apt-mirror/kanidm
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Warning: --rsync-extra is not configured to mirror the trace files.
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]:          This configuration is not recommended.
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Mirroring to /srv/apt-mirror/kanidm from https://anonymous@kanidm.github.io/kanidm_ppa/
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Arches: amd64,arm64
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Dists: bookworm,trixie
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Sections: stable
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Pdiff mode: none
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Will clean up after mirroring.
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Attempting to get lock ...
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: Getting meta files ...
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: [168B blob data]
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: ok
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: [180B blob data]
Feb 05 10:50:07 myhost debmirror-sync.sh[462972]: ok
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: [149B blob data]
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: ok
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: [196B blob data]
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: ok
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: [208B blob data]
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: ok
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: [140B blob data]
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: ok
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: Ignoring missing Release file for dists/bookworm/stable/binary-all/Packages.gz
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: [235B blob data]
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: Download of dists/bookworm/stable/binary-all/Packages.gz failed: 404 Not Found
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: Ignoring missing Release file for dists/trixie/stable/binary-all/Packages.gz
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: [231B blob data]
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: Download of dists/trixie/stable/binary-all/Packages.gz failed: 404 Not Found
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: Failed to download some Package, Sources or Release files!
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: WARNING: releasing 1 pending lock...
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]: Errors:
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]:  Ignoring missing Release file for dists/bookworm/stable/binary-all/Packages.gz
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]:  Download of dists/bookworm/stable/binary-all/Packages.gz failed: 404 Not Found
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]:  Ignoring missing Release file for dists/trixie/stable/binary-all/Packages.gz
Feb 05 10:50:08 myhost debmirror-sync.sh[462972]:  Download of dists/trixie/stable/binary-all/Packages.gz failed: 404 Not Found
Feb 05 10:50:08 myhost debmirror-sync.sh[462970]: === Sync failed with exit code 2 ===

(I think it is this bug).

aptly looked nice for try #2, and claims to support exactly this - mirroring third party repositories - but as far as I could tell it had no support for mirroring repository signatures, making clients in 2026 quite unhappy:

Error: The repository 'https://somehost.example.org/thirdparty/kanidm trixie Release' is not signed.
Notice: Updating from such a repository can't be done securely, and is therefore disabled by default.

I uh would really rather not have to set up my own secure remote package signing system for my very basic mirror.

Attempt #3 to just use wget to spider files doesn't work since some of the repositories are hosted on GitHub Pages, which doesn't do normal directory indexes.

Try #4 was apt-mirror2, which is blessedly itself packaged in Debian, and Just Works - it basically just takes some sources.list lines and some paths and does the thing:

# apt-mirror2 config for kanidm
# Generated by Ansible

set base_path         /srv/apt-mirror
set mirror_path       /srv/apt-mirror/public
set skel_path         /srv/apt-mirror/.skel
set var_path          /srv/apt-mirror/.var
set nthreads          4
set gpg_verify        on
set etc_trusted_parts /srv/apt-mirror/.gnupg

deb [arch=amd64,arm64 signed-by=/srv/apt-mirror/.gnupg/kanidm_ppa.gpg] https://kanidm.github.io/kanidm_ppa bookworm stable
deb [arch=amd64,arm64 signed-by=/srv/apt-mirror/.gnupg/kanidm_ppa.gpg] https://kanidm.github.io/kanidm_ppa trixie stable

It is quite surprising that this is much much more fiddly than running my own full-Debian mirror is - that was half an hour of fiddling with ftpsync and it was done.