Python Development Environment 2022 Edition

Just a collection of tips and tools to improve your python development experience inspired by this blogpost.

Since the escape key is back, it's now possible to use macbooks as development machines again. So I got one and had to redo my python development setup. After some time I thought that I should write down the required steps, because otherwise I would have to google this stuff all over again next time. After some more time I thought that it might be a good idea to publish those steps. Because I clearly have no clue what I'm doing and just asking for help wouldn't be as effective as posting the wrong answer and waiting for a poor soul to take the bait. So here we are :).

macOS

The macOS specific stuff has a page on it's own.

Shell

Package Manager

Using a package manager for your shell makes things a lot easier. You can get oh-my-fish by typing:

$ curl -L https://get.oh-my.fish | fish

Update your packages and install one:

$ omf update
$ omf install bobthefish

Use a theme from this list of available themes.

$ omf theme bobthefish

If you see strange character appearing in your prompt, you probably have to installed a powerline patched font. You can install one line this:

$ omf install fonts
$ fonts install --powerline Inconsolata

Maybe you have to restart iTerm and change your font setting for your iTerm profile, too.

Config

To start the web based fish configuration call:

$ fish_config

Python

Update Python

Make sure pyenv is up to date:

$ cd ~/.pyenv
$ git pull

Uninstall virtualfish

$ vf uninstall
$ pipx uninstall virtualfish

Uninstall pipx:

$ PIP_REQUIRE_VIRTUALENV="false" python -m pip uninstall pipx

Install new python version:

$ pyenv install new.version.number

Use new python version:

$ pyenv global new.version.number

Uninstall the old python version since virtualfish has the tendency to create virtualenvs with an old python version by default even if it's running a newer version:

$ pyenv uninstall old.version.number

Install pipx

PIP_REQUIRE_VIRTUALENV="false" python -m pip install --user pipx

Upgrade pipx

$ pipx upgrade-all

Maybe reinstall all if some interpreters are missing.

$ pipx reinstall-all

Install virtualfish

$ pipx install virtualfish
$ vf install compat_aliases projects environment auto_activation

Outdated Stuff

Installing python on macOS (or any other OS I know of) is much more difficult than you would expect. After using a mixture of system provided python, python installed by homebrew and python installed by conda I finally settled using pyenv. With pyenv it's easy to install a lot of different python versions. After that you are able to define which python version should be used per directory. Even things like pypy or miniconda are possible. Together with pyenv-virtualenv I use it for my data science conda environments and my web development virtualenvs. Environments are automatically switched by entering the respective project directory:

When installing miniconda3-latest, make sure PYTHON_CONFIGURE_OPTS="--enable-framework" is not set (you might need this for some vim plugins etc.) otherwise miniconda installation will fail.

$ brew install xz pyenv pyenv-virtualenv

There should be something like this in .config/fish/config.fish:

    # pyenv / pyenv-virtualenv
    set -gx PYENV_ROOT $HOME/.pyenv
    set -gx PATH $PYENV_ROOT/bin $PATH
    status --is-interactive; and source (pyenv init -|psub)
    status --is-interactive; and source (pyenv virtualenv-init -|psub)

Use a recent python version as system python:

$ pyenv global 3.10.5

Caveats

• pyenv shims are masking system commands (ffmpeg/ffprobe for me) - I found pyenv-which-ext helpful

Using poetry for packages and dependency management

Install poetry:

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3

If you used pyenv to create your virtualenv, you have to activate it. Otherwise poetry will create an own virtualenv on poetry install.

$ pyenv activate your_env

Getting A Django Project Up And Running

Database

At first you also have to install postgres via homebrew:

$ brew install postgresql
$ brew services start postgresql   # to have launchd start postgres

To simplify the creation of databases for my projects I created a little helper fish function and put it into ~/.config/fish/functions:

function init_django_db
  set user $argv[1]
  set dbname $argv[2]
  dropdb $dbname
  dropuser $user
  createdb $dbname
  createuser $user
  psql -d $dbname -c "GRANT ALL PRIVILEGES ON DATABASE $dbname to $user;"
end

Recreating a database and user with access to this database is just a call to this function:

$ init_django_db username dbname

Environment Variables

There are usually a lot of environment variables that need to be set for each project. I usually keep them in an .env file in the root project directory. Now I only have to make sure those variables get set when I enter the projects directory and this is done by the following function in ~/.config/fish/config.fish:

# on pwd change look for env file and load it
function load_env --on-variable PWD
  if test -e .env
    export (grep -Ev '^\s*$|^\s*\#' .env)
  end
end

Yeah, I know about autoenv and direnv, but I'm also using python-dotenv and need to resolve some compatibility issues first.

Virtualenv

You should now be able to just clone one of your old django projects from github for example, create a new virtualenv and install the requirements:

$ pyenv virtualenv 3.8.1 venv_name
$ cd projects/project_name
$ pyenv local venv_name    # store the name of the venv you like to use for this dir in .python-version
$ pyenv virtualenvs        # check the right virtualenv is activated on entering project directory
$ pip install -r requirements/local.txt

Create the database and run the development server:

$ ./manage.py migrate
$ ./manage.py runserver 0.0.0.0:8000

Getting A Data Science Project Up And Running

To activate a conda environment automatically on entering a directory containing an environment.yml file, I use a small function in .config/fish/config.fish:

# on pwd change look for conda environment.yml and activate conda env
function activate_conda_env --on-variable PWD
  if test -e environment.yml
    pyenv activate (pyenv virtualenvs | grep miniconda | cut -d " " -f 3)
    conda activate (grep name environment.yml | cut -d " " -f 2)
  end
end

Setting up the virtual environment:

$ pyenv activate miniconda3-latest
$ conda env create        # after creating an environment.yml
$ conda deactivate
$ cd
$ cd projects/project_dir
$ conda env list             # right environment should be activated

Using fzf

fzf is a really great tool, and the most enjoyed recent addition to my toolbelt.

Install it with:

$ brew install fzf
/opt/fzf/install  # install useful key bindings and fuzzy completion

You can put some helpful environment variables in ~/.config/fish/config.fish:

# fzf
set -gx FZF_DEFAULT_COMMAND "fd --type f"
set -gx FZF_DEFAULT_OPTS "-layout=reverse --inline-info"

Here are some examples on how to use it:

$ vim (fzf)  # search for file and open in vim
$ history | fzf +s --tac  # search in history in reverse order and dont sort result
$ fzf --preview 'bat --style=numbers --color=always {} | head -500'  # use fzf with bat preview

macOS quirks

On OS X, Alt-c (Option-c) types ç by default. In iTerm2, you can send the right escape sequence with Esc-c. If you configure the option key to act as +Esc (iTerm2 Preferences > Profiles > Default > Keys > Left option (⌥) acts as: > +Esc), then alt-c will work for fzf as documented

Fixing Vim

Use homebrew to install vim:

$ brew install vim

Install Vundle

$ git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim

List of vim plugins I like to use:

• fugitive.vim interacting with git from vim
• jedi-vim autocompletion and code navigation
• Command-T fuzzy search/open files from vim
• supertab configure behaviour
• YouCompleteMe code completion engine
• psf/black vim plugin for python formatting :BlackUpgrade to update

Awesome list of vim plugins.

Useful Command Line Tools

• bat
• tree
• ripgrep
• fd
• fzf
• git-flow brew install git-flow
• wget
• tldr
• asciinema
• multitail
• ntop (brew install ntopng)
• slurm brew install slurm
• gitversion
• gource
• pandoc
• tmux
• neofetch