Store full remote URL in state file and other updates

This version of mepo has some backend changes. The most important is that the full path of remote url is now stored in the state file. This allows for reducing the number of git calls needed.

There is also some refactoring of code as well as added newly added tests and updates to documentation.

---

From CHANGELOG.md

Fixed

• Fixed mepo completion

Added

• Added ability to print version info via mepo --version
• Added install instructions for spack and brew tap
• Added tests for Registry and MepoComponent classes

Changed

• Full path of remote url is now stored in the state file
• Some refactor of component.py
• Removed MepoState dependency in git.py

---

What's Changed

• Added ability to print version info by @pchakraborty in #309
• Add spack install instructions by @mathomp4 in #310
• Update README by @pchakraborty in #315
• Store complete remote url in state file by @pchakraborty in #316
• Merge main into develop by @pchakraborty in #317
• Merge develop into main by @pchakraborty in #318
• Do a merge commit from main into develop by @mathomp4 in #319
• Updated CHANGELOG.md by @pchakraborty in #320
• Merge develop into main by @pchakraborty in #321

Full Changelog: v2.0.0...v2.1.0

Initial release of mepo 2

This is a major release of mepo that includes a number of changes to the codebase and the way mepo is used.

The most significant change is the conversion of mepo to a Python project. This change was made to improve the maintainability of the codebase and to make it easier to add new features in the future. The conversion to a Python project also allows for better integration with other Python tools and libraries. See this commit message for more information.

Internal State Change

This major version also introduces internal changes to mepo that end users might encounter. The main change is moving from a pickle format to a json format for the internal state of mepo. This change was made to improve the readability and extensibility of the state file.

For example, if a user has an existing clone of a mepo repository that was created with mepo version 1, when they run with mepo version 2, they will see a message like:

> mepo status
Checking status...
Detected mepo1 style state
Run <mepo update-state> to permanently convert to mepo2 style
GEOSfvdycore           | (b) main
env                    | (t) v4.29.0 (DH)
cmake                  | (t) v3.45.1 (DH)
...

As the message says, the user can run mepo update-state to convert the state to mepo2 style (i.e., from pickle to json). Note that this is a one-way change!

macOS multiprocessing issue

If you have a mepo1 style state and you are on macOS, you may encounter an issue when running mepo status or mepo restore-state. For example:

> mepo status
Checking status...
Detected mepo1 style state
Run <mepo update-state> to permanently convert to mepo2 style
Process SpawnPoolWorker-2:
Traceback (most recent call last):
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "/Users/mathomp4/.homebrew/brew/Cellar/[email protected]/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
...

This seems to be a macOS issue with Python multiprocessing. The solution is to run mepo update-state to convert the state to mepo2 style. Then all works as expected:

> mepo update-state
Detected mepo1 style state
Run <mepo update-state> to permanently convert to mepo2 style

Converted mepo1 state to mepo2

> mepo status
Checking status...
GEOSfvdycore           | (b) main
env                    | (t) v4.29.0 (DH)
cmake                  | (t) v3.45.1 (DH)
ecbuild                | (t) geos/v1.3.0 (DH)
GMAO_Shared            | (t) v1.9.7 (DH)
GEOS_Util              | (t) v2.0.8 (DH)
...

Missing features

Note that at the moment the mepo completion scripts and mepo-cd scripts are not installed by default. This will be addressed in a future release.

---

From CHANGELOG.md

• Added pyproject.toml to aid with pip installation.

• Engineering
  -- Formatting with Black
  -- Linting with Pylint
  -- Dependency management and packaging with Rye

• Added tests to cover more mepo commands

• Add new command update-state to permanently convert mepo1 style state to mepo2

Changed

• Converted mepo to a Python project via the following renaming
  -- Added src/mepo/__init__.py
  -- Renamed mepo.d -> src/mepo
  -- Renamed mepo.d/utest -> tests
  -- Renamed doc --> docs
  -- A mepo config file is now called a mepo registry
  -- More code reorganization

• Helper script mepo, used for development, moved to the bin directory.

• Added README for docs/make_md_docs.py script

• State: pickle format (mepo1 style) to json format (mepo2 style)
  -- If mepo1 style state is detected, print warning and suggest running mepo update-state

---

What's Changed

• Make mepo a pip-installable Python project by @pchakraborty in #269
• Fix small typo by @mathomp4 in #271
• Patch for reading mepo1 state by @pchakraborty in #279
• Python package with expanded testing by @pchakraborty in #278
• Make doc generation work again by @pchakraborty in #286
• Switch from Poetry to Rye for dependency management and packaging by @pchakraborty in #290
• Some engineering by @pchakraborty in #292
• Fixed a bug in mepo state creation during 'mepo update-state' by @pchakraborty in #295
• Make mdutils a dev-dependency instead of a dependency by @pchakraborty in #294
• A bug fix and '--serial' option for status and restore-state commands by @pchakraborty in #296
• Version 2.0.0rc4 by @pchakraborty in #297
• Add docs README, update main README by @mathomp4 in #272
• Ready to release 2.0.0 by @pchakraborty in #298
• Merge develop into main, for release (2.0.0) by @pchakraborty in #300
• Add brew tap installation instructions by @mathomp4 in #302

Full Changelog: v1.52.0...v2.0.0

Support for partial clones

This release adds support for partial clones with mepo clone. A new option --partial has three settings:

• off: performs a "normal" git clone
• blobless: performs a blobless clone via git clone --filter=blob:none
• treeless: performs a treeless clone via git clone --filter=tree:0

The default is tecnically none of these, but that is equivalent to off. The off option is mainly added here to allow overriding of any .mepoconfig setting (see below).

The motivation for this is that it has been noticed that clones of GEOS (mainly MAPL) are often extremely slow. For example:

❯ time git clone [email protected]:GEOS-ESM/MAPL.git MAPL-normal
Cloning into 'MAPL-normal'...
remote: Enumerating objects: 704212, done.
remote: Counting objects: 100% (271489/271489), done.
remote: Compressing objects: 100% (6882/6882), done.
remote: Total 704212 (delta 269709), reused 266119 (delta 264584), pack-reused 432723
Receiving objects: 100% (704212/704212), 997.08 MiB | 2.59 MiB/s, done.
Resolving deltas: 100% (694344/694344), done.
noglob git clone [email protected]:GEOS-ESM/MAPL.git MAPL-normal  95.46s user 13.88s system 24% cpu 7:17.37 total

This took over 7 minutes to clone!

However, git supports partial clones as detailed in this GitHub Blog post. Now, of the two, blobless clones are fairly safe and give you faster initial clone speed at the cost of slower operations after that. As a test:

❯ time git clone --filter=blob:none [email protected]:GEOS-ESM/MAPL.git MAPL-blobless-from-git
Cloning into 'MAPL-blobless-from-git'...
remote: Enumerating objects: 21299, done.
remote: Counting objects: 100% (3171/3171), done.
remote: Compressing objects: 100% (1324/1324), done.
remote: Total 21299 (delta 1972), reused 2989 (delta 1829), pack-reused 18128
Receiving objects: 100% (21299/21299), 20.80 MiB | 2.99 MiB/s, done.
Resolving deltas: 100% (13343/13343), done.
remote: Enumerating objects: 942, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (500/500), done.
remote: Total 942 (delta 112), reused 126 (delta 50), pack-reused 390
Receiving objects: 100% (942/942), 1.85 MiB | 136.00 KiB/s, done.
Resolving deltas: 100% (249/249), done.
Updating files: 100% (1064/1064), done.
noglob git clone --filter=blob:none [email protected]:GEOS-ESM/MAPL.git   1.38s user 0.54s system 6% cpu 27.609 total

28 seconds!

Treeless clones are usually faster than blobless as you aren't just filtering out blobs but whole trees. But per the blog:

We strongly recommend that developers do not use treeless clones for their daily work. Treeless clones are really only helpful for automated builds when you want to quickly clone, compile a project, then throw away the repository. In environments like GitHub Actions using public runners, you want to minimize your clone time so you can spend your machine time actually building your software! Treeless clones might be an excellent option for those environments.

As there are possible scenarios with CI that this could be useful for, the option is added. As for speed:

❯ time git clone --filter=tree:0 [email protected]:GEOS-ESM/MAPL.git MAPL-treeless
Cloning into 'MAPL-treeless'...
remote: Enumerating objects: 6875, done.
remote: Counting objects: 100% (843/843), done.
remote: Compressing objects: 100% (730/730), done.
remote: Total 6875 (delta 124), reused 805 (delta 113), pack-reused 6032
Receiving objects: 100% (6875/6875), 2.24 MiB | 277.00 KiB/s, done.
Resolving deltas: 100% (757/757), done.
remote: Enumerating objects: 106, done.
remote: Counting objects: 100% (70/70), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 106 (delta 1), reused 19 (delta 0), pack-reused 36
Receiving objects: 100% (106/106), 37.34 KiB | 538.00 KiB/s, done.
Resolving deltas: 100% (3/3), done.
remote: Enumerating objects: 942, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (500/500), done.
remote: Total 942 (delta 112), reused 126 (delta 50), pack-reused 390
Receiving objects: 100% (942/942), 1.85 MiB | 2.09 MiB/s, done.
Resolving deltas: 100% (249/249), done.
Updating files: 100% (1064/1064), done.
noglob git clone --filter=tree:0 [email protected]:GEOS-ESM/MAPL.git   0.45s user 0.25s system 4% cpu 15.950 total

16 seconds!

Along with this option, we also add a new .mepoconfig setting where one can add:

[clone]
partial = blobless

and blobless clones will be the default.

---

From CHANGELOG.md

Added

• Added new --partial option to mepo clone with two settings: off, blobless, and treeless. If you set, --partial=blobless then
  the clone will not download blobs by using --filter=blob:none. If you set --partial=treeless then the clone will not download
  trees by using --filter=tree:0. The blobless option is useful for large repos that have a lot of binary files that you don't
  need. The treeless option is even more aggressive and SHOULD NOT be used unless you know what you are doing. The
  --partial=off option allows a user to override the default behavior of --partial in .mepoconfig and turn it off for a
  run of mepo clone.
• Add a new section for .mepoconfig to allow users to set --partial as a default for mepo clone.

---

What's Changed

• GitFlow: Merge main into develop for hotfix by @mathomp4 in #262
• Add support for partial clones by @mathomp4 in #264
• Protect submodule repos from treeless, add off option by @mathomp4 in #266
• GitFlow: Merge develop into main by @mathomp4 in #265

Full Changelog: v1.51.1...v1.52.0

Fix for ignore_submodules in older mepo clones

This patch release of mepo fixes an issue running mepo commands on older mepo clones made before v1.51.0 when ignore_submodules was added. Mea culpa.

---

From CHANGELOG.md

Fixed

• Fixes to allow mepo to work on older mepo clones that don't have ignore_submodules in their state

---

What's Changed

• Fixes for older mepo clones in re ignore_submodules by @mathomp4 in #261

Full Changelog: v1.51.0...v1.51.1

Add ability to ignore submodules in diff and status

TL;DR This release adds a new field to components.yaml, ignore_submodules which when true will ignore submodules in the mepo status and mepo diff calls.

---

A mepo oddity occurs in GEOSgcm because of how CICE and icepack are handled.

Technically, icepack is a submodule of CICE in the repo (see https://github.com/GEOS-ESM/CICE) and git sees it as that. However, currently GEOS handles icepack as a separate repo (most likely for ease of development) but git doesn't really know this. So when you go to CICE in GEOS in a fresh clone you see:

$ git status
HEAD detached at geos/v0.0.2
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   icepack (new commits)

Submodules changed but not updated:

* icepack 81a7628...8d80ff5 (1):
  > Merge pull request #1 from GEOS-ESM/bugfix/zhaobin74/recompute-enthalpy

no changes added to commit (use "git add" and/or "git commit -a")

and since mepo status just sort of "echoes" status output with mepo status we see:

cice6                  | (t) geos/v0.0.2 (DH)
   | icepack: modified, not staged

So git says "something changed" but we want to ignore that in this case.

Thus, this PR adds a new field to components.yaml, ignore_submodules:

cice6:
  local: ./src/Components/@GEOSgcm_GridComp/GEOSogcm_GridComp/GEOSseaice_GridComp/CICE_GEOSPlug/@cice6
  remote: ../CICE.git
  tag: geos/v0.0.2
  develop: geos/develop
  ignore_submodules: true

If this is true, then we add --ignore-submodules=all to underlying git calls when we run mepo diff and mepo status

---

From CHANGELOG.md

Added

• Added new ignore_submodules field in components.yaml to allow ignoring submodules in a repo. Currently used for status and
  diff commands.

---

What's Changed

• Add ability to ignore submodules by @mathomp4 in #258
• GitFlow: Merge Develop into main by @mathomp4 in #259

Full Changelog: v1.50.0...v1.51.0

Add hashes option to status command

This release of mepo adds a new --hashes option for mepo status that displays the hashes of each repo:

❯ mepo status
Checking status...
MAPL       | (b) main
ESMA_env   | (t) v4.19.0 (DH)
ESMA_cmake | (t) v3.31.0 (DH)
ecbuild    | (t) geos/v1.3.0 (DH)
❯ mepo status --hashes
Checking status...
MAPL       | (b) main (1afeb43d6af485c6fbedd7ce72ccc1eec20a5c5a)
ESMA_env   | (t) v4.19.0 (DH) (7a8e3b158967bf3144d396d836489f46888b7f56)
ESMA_cmake | (t) v3.31.0 (DH) (7dc4c819dc5169e6b887374aa0fa0a8f71846832)
ecbuild    | (t) geos/v1.3.0 (DH) (4a8c91a579e112a6e2a5224068e95c3a8935204b)

---

From CHANGELOG.md

Added

• Command status has now a --hashes option that list current HEAD hash for each component.

---

What's Changed

• Update zenodo.json by @mathomp4 in #252
• Add --hashes reading HEAD to get the current hash by @FlorianDeconinck in #253
• Update changelog for 1.50 release by @mathomp4 in #254
• GitFlow: Merge Develop into main for release by @mathomp4 in #255

New Contributors

• @FlorianDeconinck made their first contribution in #253

Full Changelog: v1.49.0...v1.50.0

Better mepo compare display

Previous to this release, if an original branch name is very long, this can cause odd output of mepo compare on narrow (relative to the branch name) terminals even if the repo wasn't changed. The compare command would use the maximum length of all repos, and not just those that changed.

With this fix, mepo compare will use only the repos that change to figure out how wide to make the output rather than all the branches on all the repos.

Of course, if you do mepo compare --all, it will base the output on all the repos as usual.

---

What's Changed

• Compare headers based on changed branch length by @mathomp4 in #249
• GitFlow: Merge Develop into Main for release by @mathomp4 in #250

Full Changelog: v1.48.0...v1.49.0

Add mepo reset command

This release adds a new mepo reset command:

❯ mepo reset -h
usage: mepo reset [-h] [-f] [--reclone] [-n]

Reset the current mepo clone to the original state.This will delete all subrepos and does not check for uncommitted changes! Must be run in the root of
the mepo clone.

options:
  -h, --help     show this help message and exit
  -f, --force    Force action.
  --reclone      Reclone repos after reset.
  -n, --dry-run  Dry-run only

By default, it will not re-clone a repo and will require a user to answer another question to run (without force flag):

❯ mepo clone
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
❯ mepo reset
Are you sure you want to reset the project? If so, type 'yes' and press enter. yes
Removing ESMA_cmake/ecbuild...done.
Removing ESMA_cmake...done.
Removing ESMA_env...done.
Removing .mepo...done.
❯ ls -1 ESMA_env | head -1
ls: ESMA_env: No such file or directory

You can do a reclone with --reclone:

❯ mepo clone
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
❯ mepo reset --reclone -f
Removing ESMA_cmake/ecbuild...done.
Removing ESMA_cmake...done.
Removing ESMA_env...done.
Removing .mepo...done.
Re-cloning all subrepos
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
Recloning done.
❯ ls -1 ESMA_env | head -1
BASEDIR.rc.in

Note we also require the user to be in the root of a fixture:

❯ mepo clone
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
❯ cd ESMA_cmake/ecbuild
❯ mepo reset
Error! As a safety precaution, you must be in the root directory of the project to reset

---

What's Changed

• Gitflow. Main to develop by @mathomp4 in #243
• Add new reset command by @mathomp4 in #245
• Update GitHub Actions by @mathomp4 in #247
• GitFlow: Merge Develop into Main For Release by @mathomp4 in #246

Full Changelog: v1.47.0...v1.48.0

Add mepo tag push --delete

This release of mepo adds a capability to delete tags on the remote with:

mepo tag push --delete tagname repo1 repo2

---

From CHANGELOG.md

Added

• Added ability to do mepo tag push --delete so you can delete a tag on the remote

---

What's Changed

• Add tag push delete by @mathomp4 in #240
• Merge Develop into Main for release by @mathomp4 in #241

Full Changelog: v1.46.0...v1.47.0

Add changed-files command

This release adds a new changed-files command to mepo. Essentially it will list all files that are different compared to the original tags/branches in the components.yaml (technically the mepo state, so if mepo save was run, the new components.yaml).

For example, if there are no changes:

❯ mepo compare
No repositories have changed
❯ mepo changed-files

Now we checkout a branch:

❯ mepo develop MAPL
Checking out development branch develop in MAPL
❯ mepo changed-files
./src/Shared/@MAPL/CHANGELOG.md
./src/Shared/@MAPL/generic/MAPL_Generic.F90

This shows that compared to where we started, we have two new files. Now let us alter a file and add a file:

❯ touch foo
❯ echo "bobob" >> src/Components/@GEOSgcm_GridComp/README.md
❯ mepo changed-files
./foo
./src/Shared/@MAPL/CHANGELOG.md
./src/Shared/@MAPL/generic/MAPL_Generic.F90
./src/Components/@GEOSgcm_GridComp/README.md

Finally, it has a --full-path option for a full path.

❯ mepo changed-files --full-path
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/foo
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/src/Shared/@MAPL/CHANGELOG.md
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/src/Shared/@MAPL/generic/MAPL_Generic.F90
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/src/Components/@GEOSgcm_GridComp/README.md

---

From CHANGELOG.md

Added

• Add new changed-files command to list all changed files vs original state

---

What's Changed

• Add ability to display changed files by @mathomp4 in #236
• GitFlow: Merge Develop into Main for release by @mathomp4 in #237

Full Changelog: v1.45.0...v1.46.0