ertius.org

git-lfs vs forgejo

#git-lfs #forgejo #sysadmin #nginx

My git-lfs trouble turned out to be troubles. Once I managed to back-fill the repository, the forgejo action runner became my new problem, just endlessly spewing HTTP 400 errors when trying to clone a repository with LFS enabled, via an nginx proxy.

It spewed 400s a lot, in amongst some 200s, which was quite confusing. The core bit:

> GET /user/repo.git/info/lfs/objects/9541ed0d06fca0fd1bb5df644fe2795ca3850aa671c335442dcf577df45aa679 HTTP/1.1
> Host: host
> Authorization: Basic * * * * *
> Authorization: Basic * * * * *
> User-Agent: git-lfs/3.7.1 (GitHub; linux amd64; go 1.24.4)

10:07:23.477845 trace git-lfs: HTTP: 400
< HTTP/1.1 400 Bad Request
< Connection: close
< Content-Length: 157
< Content-Type: text/html
< Date: Wed, 26 Nov 2025 10:07:23 GMT
< Server: nginx/1.22.1

That's some suspicious header-doubling there. Eventually some kagi-ing found this bug which seemed like a pretty good lead. I think I must have just done some more searches using those keywords until I found https://github.com/chrisliebaer/gitea-actions-fix, which amongst other things fixes up the headers. So:

      # Workaround LFS/Forgejo interaction issue:
      # https://codeberg.org/forgejo/forgejo/issues/7264
      - uses: https://github.com/chrisliebaer/gitea-actions-fix@v1
      - name: Checkout code
        uses: https://code.forgejo.org/actions/checkout@v4
        with:
          lfs: true

A mere four hours or so, it took me to find the problem and add this one line of code.

The issue seems to be:

  1. actions/checkout is a bit sloppy with how it auths to lfs - it just tells git to always add an Authorization: header

  2. Git LFS already sends it's own Authorization: header when requesting blob

  3. RFC7230 says:

    A sender MUST NOT generate multiple header fields with the same field name in a message unless either the entire field value for that header field is defined as a comma-separated list [i.e., #(values)] or the header field is a well-known exception (as noted below).

  4. when presented with two Authorization: headers, nginx says 400 RFC7230 says fuck off

One of the former Git LFS maintainers says:

the proper solution here is to use a credential helper

Which does seem fair enough in general, but isn't very helpful for this specific case of an action runner. But I guess it's just hard to solve in the general case, Git LFS uses two different network protocols and there isn't any particular reason for them to be unified with the how you cloned the git repository itself, and the runners need to authenticate autonomously using something single factor. A difficult situation.