all-repos

Clone all your repositories and apply sweeping changes.

Installation

pip install all-repos

CLI

All command line interfaces provided by all-repos provide the following options:

• -h / --help: show usage information
• -c CONFIG_FILENAME / --config-filename CONFIG_FILENAME: use a non-default config file (default all-repos.json).
• --color {auto,always,never}: use color in output (default auto).

all-repos-complete [options]

Add git clone tab completion for all-repos repositories.

Requires jq to function.

Add to .bash_profile:

eval "$(all-repos-complete -C ~/.../all-repos.json --bash)"

all-repos-clone [options]

Clone all the repositories into the output_dir.

Options:

• -j JOBS / --jobs JOBS: how many concurrent jobs will be used to complete the operation. Specify 0 or -1 to match the number of cpus. (default 8).

Sample invocations:

• all-repos-clone: clone the repositories specified in all-repos.json
• all-repos-clone -C all-repos2.json: clone using a non-default config filename.

all-repos-find-files [options] PATTERN

Similar to a distributed git ls-files | grep -P PATTERN.

Arguments:

• PATTERN: the python regex to match.

Options:

• --repos-with-matches: only print repositories with matches.

Sample invocations:

• all-repos-find-files setup.py: find all setup.py files.
• all-repos-find-files --repos setup.py: find all repositories containing a setup.py.

all-repos-grep [options] [GIT_GREP_OPTIONS]

Similar to a distributed git grep ....

Options:

• --repos-with-matches: only print repositories with matches.
• GIT_GREP_OPTIONS: additional arguments will be passed on to git grep. see git grep --help for available options.

Sample invocations:

• all-repos-grep pre-commit -- 'requirements*.txt': find all repositories which have pre-commit listed in a requirements file.
• all-repos-grep -L six -- setup.py: find setup.py files which do not contain six.

all-repos-list-repos [options]

List all cloned repository names.

all-repos-sed [options] EXPRESSION FILENAMES

Similar to a distributed git ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION.

note: this assumes GNU sed, if you're on osx use brew install gnu-sed --with-default-names.

Arguments:

• EXPRESSION: sed program. For example: s/hi/hello/g.
• FILENAMES: filenames glob (passed to git ls-files).

Options:

• autofix options: all-repos-sed is an autofixer and supports all of the autofixer options.
• -r / --regexp-extended: use extended regular expressions in the script. See man sed for further details.
• --branch-name BRANCH_NAME override the autofixer branch name (default all-repos-sed).
• --commit-msg override the autofixer commit message. (default git ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION).

Sample invocations:

• all-repos-sed 's/foo/bar' -- '*': replace foo with bar in all files.

Configuring

A configuration file looks roughly like this:

{
    "output_dir": "output",
    "source": "all_repos.source.github",
    "source_settings":  {
        "api_key": "...",
        "username": "asottile"
    },
    "push": "all_repos.push.github_pull_request",
    "push_settings": {
        "api_key": "...",
        "username": "asottile"
    }
}

• output_dir: where repositories will be cloned to when all-repos-clone is run.
• source: the module import path to a source, see below for builtin source modules as well as directions for writing your own.
• source_settings: the source-type-specific settings, the source module's documentation will explain the various possible values.
• push: the module import path to a push, see below for builtin push modules as well as directions for writing your own.
• push_settings: the push-type-specific settings, the push module's documentation will explain the various possible values.
• include (default ""): python regex for selecting repositories. Only repository names which match this regex will be included.
• exclude (default "^$"): python regex for excluding repositories. Repository names which match this regex will be excluded.

Source modules

all_repos.source.json_file

Clones all repositories listed in a file. The file must be formatted as follows:

{
    "example/repo1": "https://git.example.com/example/repo1",
    "repo2": "https://git.example.com/repo2"
}

Required source_settings

• filename: file containing repositories one-per-line.

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {repo_key1}/
+--- {repo_key2}/
+--- {repo_key3}/

all_repos.source.github

Clones all repositories available to a user on github.

Required source_settings

• api_key: the api key which the user will log in as.
  • Use the settings tab to create a personal access token.
  • The minimum scope required to function is public_repo, though you'll need repo to access private repositories.
• username: the github username you will log in as.

Optional source_settings

• collaborator (default false): whether to include repositories which are not owned but can be contributed to as a collaborator.
• forks (default false): whether to include repositories which are forks.
• private (default false): whether to include private repositories.

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
    +--- {repo1}/
    +--- {repo2}/
+--- {username2}/
    +--- {repo3}/

all_repos.source.gitolite

Clones all repositories available to a user on a gitolite host.

Required source_settings

• username: the user to SSH to the server as (usually git)
• hostname: the hostname of your gitolite server (e.g. git.mycompany.com)

The gitolite API is served over SSH. It is assumed that when all-repos-clone is called, it's possible to make SSH connections with the username and hostname configured here in order to query that API.

Optional source_settings

• mirror_path (default None): an optional mirror to clone repositories from. This is a Python format string, and can use the variable repo_name.

  This can be anything git understands, such as another remote server (e.g. gitmirror.mycompany.com:{repo_name}) or a local path (e.g. /gitolite/git/{repo_name}.git).

Directory location

output/
+--- repos.json
+--- repos_filtered.json
+--- {repo_name1}.git/
+--- {repo_name2}.git/
+--- {repo_name3}.git/

Writing your own source

First create a module. This module must have the following api:

A Settings class

This class will receive keyword arguments for all values in the source_settings dictionary.

An easy way to implement the Settings class is by using a namedtuple:

Settings = collections.namedtuple('Settings', ('required_thing', 'optional'))
Settings.__new__.__defaults__ = ('optional default value',)

In this example, the required_thing setting is a required setting whereas optional may be omitted (and will get a default value of 'optional default value').

def list_repos(settings: Settings) -> Dict[str, str]: callable

This callable will be passed an instance of your Settings class. It must return a mapping from {repo_name: repository_url}. The repo_name is the directory name inside the output_dir.

Push modules

all_repos.push.merge_to_master

Merges the branch directly to master and pushes. The commands it runs look roughly like this:

git checkout master
git pull
git merge --no-ff $BRANCH
git push origin HEAD

push_settings

There are no configurable settings for merge_to_master.

all_repos.push.github_pull_request

Pushes the branch to origin and then creates a github pull request for the branch.

Required push_settings

• api_key: the api key which the user will log in as.
  • Use the settings tab to create a personal access token.
  • The minimum scope required to function is public_repo, though you'll need repo to access private repositories.
• username: the github username you will log in as.

Writing your own push module

First create a module. This module must have the following api:

A Settings class

This class will receive keyword arguments for all values in the push_settings dictionary.

def push(settings: Settings, branch_name: str) -> None:

This callable will be passed an instance of your Settings class. It should deploy the branch. The function will be called with the root of the repository as the cwd.

Writing an autofixer

An autofixer applies a change over all repositories.

all-repos provides several api functions to write your autofixers with:

all_repos.autofix_lib.add_fixer_args

def add_fixer_args(parser):

Adds the autofixer cli options.

Options:

• --dry-run: show what would happen but do not push.
• -i / --interactive: interactively approve / deny fixes.
• -j JOBS / --jobs JOBS: how many concurrent jobs will be used to complete the operation. Specify 0 or -1 to match the number of cpus. (default 1).
• --limit LIMIT: maximum number of repos to process (default: unlimited).
• --author AUTHOR: override commit author. This is passed directly to git commit. An example: --author='Herp Derp <[email protected]>'.
• --repos [REPOS [REPOS ...]]: run against specific repositories instead. This is especially useful with xargs autofixer ... --repos. This can be used to specify repositories which are not managed by all-repos`.

all_repos.autofix_lib.from_cli

def from_cli(args, *, find_repos, msg, branch_name):

Parse cli arguments and produce autofix_lib primitives. Returns (repos, config, commit, autofix_settings). This is handled separately from fix to allow for fixers to adjust arguments.

• find_repos: callback taking Config as a positional argument.
• msg: commit message.
• branch_name: identifier used to construct the branch name.

all_repos.autofix_lib.fix

def fix(
        repos, *,
        apply_fix,
        check_fix=_noop_check_fix,
        config: Config,
        commit: Commit,
        autofix_settings: AutofixSettings,
):

Apply the fix.

• apply_fix: callback which will be called once per repository. The cwd when the function is called will be the root of the repository.

all_repos.autofix_lib.run

def run(*cmd, **kwargs):

Wrapper around subprocess.run which prints the command it will run. Unlike subprocess.run, this defaults check=True unless explicitly disabled.

Example autofixer

The trivial autofixer is as follows:

import argparse

from all_repos import autofix_lib

def find_repos(config):
    return []

def apply_fix():
    pass

def main(argv=None):
    parser = argparse.ArgumentParser()
    autofix_lib.add_fixer_args(parser)
    args = parser.parse_args(argv)

    repos, config, commit, autofix_settings = autofix_lib.from_cli(
        args, find_repos=find_repos, msg='msg', branch_name='branch-name',
    )
    autofix_lib.fix(
        repos, apply_fix=apply_fix, config=config, commit=commit,
        autofix_settings=autofix_settings,
    )

if __name__ == '__main__':
    exit(main())

You can find some more involved examples in all_repos/autofix:

• all_repos.autofix.pre_commit_autoupdate: runs pre-commit autoupdate.
• all_repos.autofix.pre_commit_cache_dir: updates the cache directory for travis-ci / appveyor for pre-commit 1.x.
• all_repos.autofix.pre_commit_migrate_config: runs pre-commit migrate-config.