cargo cannot read from local Git repositories with reftable

[bk2204]

Problem

Recent versions of Git have added a new backend for references called reftable. This backend does not store references as files, so it performs better and remains case-sensitive even on systems with case-insensitive file systems. (This is important if the remote has branches differing only in case and you're on a case-insensitive system.)

cargo, by default, uses libgit2, which does not support reftable. When cloning from a bare repository on the local system using reftable, cargo fails. (Cloning in this way could, for instance, be used to access a bare repository on a network file system mounted locally.)

Steps

With a relatively new Git (2.47 or newer, I think), run the following script:

#!/bin/sh -e

rm -fr testcase
mkdir testcase
cd testcase

BASEDIR="$(realpath "$PWD")"

cargo init --lib --name pkg1 pkg1-worktree
git init --bare --ref-format=reftable pkg1
(
    set -e
    cd pkg1-worktree
    git add .
    git commit -m +
    git tag -a -m v0.1.0 v0.1.0
    git push ../pkg1 v0.1.0
)

cargo init pkg2
(
    set -e
    cd pkg2
    echo "pkg1 = { git = \"file:///$BASEDIR/pkg1\", tag = \"v0.1.0\" }" >> Cargo.toml
    cargo build
)

Note that the output looks like this:

error: failed to get `pkg1` as a dependency of package `pkg2 v0.1.0 (/tmp/user/1000/testcase/pkg2)`

Caused by:
  failed to load source for dependency `pkg1`

Caused by:
  Unable to update file:///tmp/user/1000/testcase/pkg1?tag=v0.1.0

Caused by:
  failed to clone into: <redacted>

Caused by:
  unsupported extension name extensions.refstorage; class=Repository (6)

Note that this does work with CARGO_NET_GIT_FETCH_WITH_CLI=true:

    Updating git repository `file:///tmp/user/1000/testcase/pkg1`
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 7 (delta 0), reused 7 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (7/7), 909 bytes | 909.00 KiB/s, done.
From file:///tmp/user/1000/testcase/pkg1
 * [new tag]         v0.1.0     -> origin/tags/v0.1.0
     Locking 1 package to latest compatible version
   Compiling pkg1 v0.1.0 (file:///tmp/user/1000/testcase/pkg1?tag=v0.1.0#121aa4e0)
   Compiling pkg2 v0.1.0 (/tmp/user/1000/testcase/pkg2)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.26s

Possible Solution(s)

I'd recommend just using Git by default. This is an easy change to work around for right now, but as I've mentioned below, there are a bunch of other ugly edge cases where using reftable causes problems.

libgit2 is cool, but in general it sees far less investment than upstream Git, so most major forges are moving away from it. reftable will show up eventually, but it will be another case like #14942 where the support is a long time coming. (Full disclosure: I'm a regular contributor to Git and I have sent a few patches to libgit2.)

Notes

The reason this happens is that reftable is an extension to the repository. For v1 repositories (all those with extensions), a Git implementation must read the local config and immediately reject every operation on the repository if there's an extension it doesn't understand. While in this case that's obviously necessary because we can't read the refs, libgit2 is also unable to perform any operation at all on a repository with reftable, including opening the repository, so things like cargo package dirty worktree checking also fail if the local repository is using reftable.

Version

cargo 1.84.0 (66221abde 2024-11-19)
release: 1.84.0
commit-hash: 66221abdeca2002d318fde6efff516aab091df0e
commit-date: 2024-11-19
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Debian n/a (trixie) [64-bit]

[weihanglo]

For the reftable format support in libgit2, see libgit2/libgit2#5352.

@rustbot label +S-blocked-external -S-triage

[weihanglo]

I'd recommend just using Git by default. This is an easy change to work around for right now,

Changing from built-in Git support to depending on an external CLI will not be an easy change. Like, people already assume they don't need git in the environment to get Cargo work. Changing that may fail people's workflows and builds.

One possible solution is that we try abstracting every git operation over different Git backends (git CLI, libgit2, gitoxide). We have done that for git-fetch already, yet it comes with complexity and incompatibility, like you've said repo-cloned-by-git with certain configuration might not be recognized by libgit2.