Use tokei on a git commit without checkout

[infogulch]

Hypothetically, tokei could traverse git objects directly as a virtual file system (VFS) as of a particular ref. This would mean you could run tokei on historical revisions without changing any state on the filesystem. This is helpful in very large repos where just checkout itself can take a long time, or the filesystem is immutable, or in bare git repos that don't create any physical files at all. This could even have performance benefits because many small files are often combined into packfiles and compressed meaning fewer disk accesses.

I'm trying to script tokei to create a summarization for every week of the history of a repo, and I'm finding that it's more trouble and time just managing git checkout state (waiting for checkout, restoring the original branch, trapping on early exit, etc) than using tokei itself.

Thoughts?

[XAMPPRocky]

Thank you for your idea! Overall I would open to adding something like this, one of the features cloc has over tokei is that it can operate on a tarball or zip file. However I think I'd need to know more about the technical details of implementing such a change. Right now, I don't think I would want to have a bespoke virtual file system handling entirely inside tokei, because we already use a third party library for filesystem traversal (which I would like to keep using), it seems a bit out of scope of tokei, and it seems like something that would be useful for other projects.

So if there's a way we can have this feature such that it integrates with ignore, and/or can be mostly achieved with a third-party library I would definitely be open to including it.

[infogulch]

Thanks for your attention! I think it's reasonable to prefer this be handled in a third party library.

The ripgrep repo contains the ignore crate which is used by tokei to perform filesystem traversal. Searching archives was added to ripgrep in #539 (search CHANGELOG.md for --search-zip), but it looks like that is done outside of ignore and is implemented using the generic io::Read trait. I assume means that it doesn't actually "walk" the files in the archive individually, but instead it searches the contents of all files in the archive. I think this strategy is reasonable for ripgrep, but is not suitable for tokei.

It looks like the ideal scenario is a new crate that integrates with ignore and exposes these filesystem-like archival formats (tar, gzip, zip, git, etc) as another recursive directory that can be descended into and used in place of the usual filesystem api. Probably with options to control which formats to enable and other per-format options like which git tree to traverse. Seems like an interesting, useful, and daunting project.

[Libbum]

I've needed to build something like this myself. Here's a start towards a solution using git2-rs:

use anyhow::{anyhow, Result};
use git2::Repository;
use std::ffi::OsStr;
use std::path::Path;
use tokei::{Config, LanguageType, CodeStats};


// TODO: This doesn't allow us to track types with shebangs. We need to read the blob for that.
// See the `blob.content()` piece below on the how - just something I didn't need right now.
fn file_language(path: &Path) -> Result<LanguageType> {
   let extension = path
       .extension()
       .and_then(OsStr::to_str)
       .unwrap_or_else(|| path.to_str().unwrap()); //This will allow for types like `Makefile`
   LanguageType::from_file_extension(&extension.to_lowercase())
       .ok_or_else(|| anyhow!("Could not map extension to language"))
}

fn code_stats_history_for_file(file_path: &Path, repo: &Repository) -> Result<Vec<CodeStats>> {
   let config = Config::default();
   let mut stats = Vec::new();
   let language = file_language(file_path)?;
   let mut revwalk = repo.revwalk()?;
   revwalk.set_sorting(git2::Sort::REVERSE)?;
   revwalk.push_head()?;
   for commit_oid in revwalk {
       let commit_oid = commit_oid?;
       let commit = repo.find_commit(commit_oid)?;

       //Skip initial commit since there's no diff
       if commit.parent_count() == 1 {
           let previous_commit = commit.parent(0)?;
           let tree = commit.tree()?;
           let previous_tree = previous_commit.tree()?;
           let diff = repo.diff_tree_to_tree(Some(&previous_tree), Some(&tree), None)?;
           let delta = diff.deltas().find_map(|d| {
               if d.new_file().path().unwrap_or_else(|| Path::new(".")) == file_path {
                   d.new_file().path()
               } else {
                   None
               }
           });
           if let Some(path) = delta {
               let file = tree.get_path(path)?;
               let obj = file.to_object(repo)?;
               if let Some(blob) = obj.as_blob() {
                   let code_stats =
                       LanguageType::parse_from_slice(language, blob.content(), config);
                   stats.push(code_stats);
               } else {
                   return Err(anyhow!("Could not load blob from delta file path"));
               }
           }
       }
   }
   Ok(stats)
}

With inputs like:

let repo = Repository::open("repository/path")?;
let file_path = Path::new("file/path/in/repo.jl");