Component manager checks out non-existing SHA (PACMAN-664)

[higaski]

The Component Manager version

IDF component manager v1.3.2

ESP-IDF Version

ESP-IDF v5.2-dev-1113-g28c643a56d

python Version

Python 3.11.3

Operating System

Arch

Browser (for https://components.espressif.com Issues)

No response

Description

The component manager seems to can get "stuck" on some specific SHA hash when trying to checkout a git branch. I've tried clearing the component managers cache but that didn't solve the issue. Neither did adding another commit to the dependency...

-- Building ESP-IDF components for target esp32s3
Processing 7 dependencies:
[1/7] bblanchon/arduinojson (6.21.2)
[2/7] dcc (f5114f028552f6128ac4a8baa86ce395bbbd8b93)
[3/7] espressif/esp_tinyusb (1.3.0)
[4/7] espressif/mdns (1.2.0)
[5/7] espressif/tinyusb (0.14.3)
[6/7] idf (5.2.0)
[7/7] mdu (1b601b8dd00ec7d8a409b41bd2b2ec35dbc3785e)  // <- This ain't an existing sha
CMake Error at /home/vinci/esp/esp-idf/tools/cmake/build.cmake:544 (message):
  WARNING: From https://github.com/ZIMO-Elektronik/MDU

   + 01815ef...a637011 packet     -> packet  (forced update)

  

  ERROR: Branch "1b601b8dd00ec7d8a409b41bd2b2ec35dbc3785e" doesn't exist in
  repo "https://github.com/ZIMO-Elektronik/MDU.git"

Call Stack (most recent call first):
  /home/vinci/esp/esp-idf/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:21 (project)

To Reproduce

On my system it's currently enough to add the following dependency:

mdu:
  version: packet
  git: https://github.com/ZIMO-Elektronik/MDU.git

Expected behaviour

Check out the latest commit when specifying a git branch.

Additional info

No response

I have checked existing issues and online Documentation

• I confirm I have checked existing issues and online Documentation.

[higaski]

The issue disappeared after some time... not sure why though. Could this be a GitHub API thing?

/edit
I haven't the slightest idea what determines which commit is considered the "last commit" of a branch here. At the time i am writing this the component manager is still checking out the 3rd most recent commit instead of the last one.

[kumekay]

Hello @higaski and thank you for the issue.
This error is shown when git ref is not found in the repo. Is the commit in the repo?

ZIMO-Elektronik/MDU@1b601b8

GitHub says This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

When I clone https://github.com/ZIMO-Elektronik/MDU.git - which is similar to what happens under the hood of the component manager, this commit is not in the cloned repo.
Did you push it to the repo itsel or to a fork? Did you push with force to the branch aftewards?

[higaski]

I have force pushed this commit once yes. So ZIMO-Elektronik/MDU@1b601b8 must refer to the old version which got overwritten. The question is though, why does the component manager keep referring to it? And why does clearing the cache not solve this issue?

/edit
2 days later (and on another machine) checking out and building the project which produced this issue now gives me the correct (and latest) commit: ZIMO-Elektronik/MDU@6a4d4d0

[kumekay]

It's intentional behavior. The goal of the component manager to make reproducible builds possible. For git sourced components it stores the commit id in dependencies lock, to be sure the same commit will be fetched next time.

So it will only try to update once dependencies.lock is removed or idf.py update-dependencies is executed.

We will document this behavior.

[higaski]

Oh ok thanks for clearing that up. Personally I find that a bit unintuitive, specially since this is not what CMake (or git actually...) does.

E.g. if I would like to add a dependency to an fmt branch called text I could use something along the lines of

FetchContent_Declare(
  fmt
  GIT_REPOSITORY https://github.com/fmtlib/fmt.git
  GIT_TAG text)

This would get me the latest commit from this branch, always. If I would be interested in checking out a specific commit I'd explicitly state that by using a tag, or, in case the maintainers of the dependency wouldn't care about versioning at all, a specific hash.

I understand that you want to give users the best possible and simplest experience, but all these "implicit" assumptions that the whole ESP-IDF framework now makes on how it should be used (e.g. how component dependencies are handled) feel like footguns.

[higaski]

So it will only try to update once dependencies.lock is removed or idf.py update-dependencies is executed.

For some reason removing dependencies.lock doesn't seem to work any longer... only idf.py update-dependencies reliably gets me the latest version.