New tutorials: make a package

[ddbeck]

(For more information about this issue, please see #194).

We need a short section to follow the section about making a distribution package (see #202) (i.e., as its own file) that will guide the reader through making a bare module or (import) package ready for installing with pip (and, in the following section, publishing on PyPI). The section should satisfy these goals:

• show how to make a setup.py file that uses setup()
• briefly acquaint the reader with friendly practices such as including a README file, a license, and using a PEP 440-compliant versioning scheme (link generously to resources that cover these things in detail)
• show how to install the package in editable/develop mode
• show how to create an sdist and wheel
• show how to test that the package is, in fact, installable
• show the work in the context of a virtual environment (as this will have been covered in the previous section)

Please note that this section should not assume the reader already has modules to package (so provide some short sample code where applicable). Also, don't cover publishing the package to PyPI or elsewhere—that's going to be covered in the next tutorial section.

When opening pull requests for this issue, please submit to the develop branch. If you have questions that concern contributing more generally, please use issue #194. Otherwise, feel free to comment with questions or feedback. Thanks!

[ddbeck]

While writing the description for #204, I realized this: the reader should decide what this package is named. If they use it in the next section, it probably needs to not have the same name as everyone else's test package or else confusion will likely follow.

[moigagoo]

@ddbeck Hi! I'd like to tackle this one.

The plan is:

1. Module + setup.py
2. Module + setup.py + README
3. Package + setup.py + README
4. Package + resource files + setup.py + README
5. Package + subpackage + resource files + setup.py + README

show how to install the package in editable/develop mode
I'd rather not tell about it because this functionality is somewhat broken. For reasons unknown, the dev mode doesn't use symlinks but a custom substitute for them. This introduces issues when you install packages that are subpackages for other packages. I know because I tried it multiple times and failed :-)

[ddbeck]

@moigagoo That looks like a great start to me. I can't wait to see a first draft!

About the editable thing, this was my thinking: it's one of those things that comes up again and again in various Python docs, blog posts, etc. I'd like the reader to be able to recognize and understand its use, even if it has limitations (just because it is a common practice). It just seemed like a natural fit here, since we'll have the reader work with package source anyway. But if you have an idea about how else we might cover it, I'd be interested in hearing about it!

[moigagoo]

@ddbeck I think it's more natural for a beginner to install packages locally into a virtualenv with pip install -U /path/to/my/package. It's a single command and it's guaranteed to work 100% of times; if it doesn't, there's something wrong with the package, and you should fix it.

If you are in a real hurry and you really hate to reinstall packages after modifying the source code, use -e, but prepare to face the caveats. This is how I'd cover it: Use it if you are absolutely sure it's the thing you need. In other words, I see it as an advanced tool, not a beginner's one.

[ddbeck]

@moigagoo OK, you've convinced me. Don't mention the editable stuff here at all and I'll think up some other way to deal with this particular tacit knowledge problem. Thanks!

[ncoghlan]

I can provide context on the "Why does -e use a .pth file rather than symlinks?" question:

• firstly, because Windows in 2004 (when the precursor to pip -e, setuptools develop mode, was first defined) didn't support symlinks at all, and Windows symlinks in 2015 are still considered an administrator-only feature
• secondly, because symlinking a directory as a subdirectory of an existing sys.path entry isn't the same as adding that directory to sys.path with a generated metadata directory (the former still lets you import the directory, but things like setuptools entry points and pip list won't work due to the missing metadata)

However, I also agree with @moigagoo - even using editable mode correctly to speed your edit/run cycles back up to where they are if you're not worried about package based distribution can still get you into all sorts of interesting potential problems with sys.path manipulation and module reloading, so there's value in avoiding it until you've developed some solid intuitions about how the import system works in the presence of code that is changing on disk.

When editable mode does get introduced, it would likely make the most sense to do it in the context of an auto-reloading local web server like the development servers in Flask and Django (as that avoids the need to introduce manual module reloading at the interactive prompt).

[moigagoo]

@ncoghlan Thanks a lot for the clarification! It all makes sense now.

I'm currently half time on Windows, half on Mac, and I use symlinks to connect subpackages during development everywhere, works like charm. It is indeed silly of Microsoft to make symlinks an admin-only feature.

[e12e]

Hi,

I realize this is a recent issue - but is there any progress?

I just started porting/hacking/forking a little cli-tool (basically moving it to python3, use docopt, and wrap it with a setup.py so it's easy to pip install in a venv). And on reading http://python-packaging-user-guide.readthedocs.org/en/latest/distributing/ in particular: http://python-packaging-user-guide.readthedocs.org/en/latest/single_source_version/#single-sourcing-the-version I found that it might be nice to have the version in a VERSION file. And all that's needed then is to make sure that the VERSION-file is installed along with the package...[1]

So far so good, except there's a lot of contradictory information on how to actually do that. Basically I'd like to be able to (tell people to):

pyvenv venv
./venv/bin/pip install -U pip setuptools
./venv/bin/pip install git+https://github.com/user/package
./venv/bin/package #yay!

But then I don't (currently) get my VERSION file (or other resources, obviously). I managed to get it to work with /venv/bin/python setup.py install from a local checkout -- but only using data_files.

FWIW the incomplete code/refactoring now lives here: https://github.com/e12e/b2/ I'm not opposed to eventually uploading that to the cheese shop (although in this particular case, it would probably make sense to coordinate with upstream, this fork is more of a personal project).

What I could really need now, is a simple "This is how you make a python package with resource files that can easily be installed by pip in a virtualenv, and is expected to continue to work for python3. Bonus points if there's an appendix for code that runs on python2 and/or pypy as well.

It might seem daunting, but I think the howistart-article for Ruby might be a great source of inspiration: http://howistart.org/posts/ruby/1

[1] I'm fully aware that using a single VERSION file might not be the best idea, but then I suppose someone(tm) should file a bug/update the official docs...

[moigagoo]

@e12e The rewriting progress is still too small to share, I'm afraid. However, I won't cover this topic in the tutorial anyway—it's way out of the beginner's scope.

Still, here's how I support multiple Python versions while keeping the version in one place. Maybe you'll find my approach useful.

I'll illustrate the approach with my project Cliar: https://bitbucket.org/moigagoo/cliar/src/

Separate Source for Different Pythons

I keep sources for different versions of Python in separate places: cliar/py2 and cliar/py3.

Instead of a single source file with lots of if version_info.major == 3: ... elif version_info.major == 2: ... I have two clean source files. This helps me keep the source tidy and allows contributors to modify the code without worrying about backward compatibility.

On the downside, I have to implement the same feature twice. In the simplest case, I just copy-paste the code from one version to another. This is not as bad as it sounds: I'd have to check and possibly rewrite the implementation anyway. With this workflow, I'm forced to make sure I correctly implemented the feature for both versions.

Switch Between the Sources Globally

The sources for each Python version are subpackages in my package. I include both in the distribution with find_packages(): https://bitbucket.org/moigagoo/cliar/src/a739240126ff2b750d66bb5b9617655e06da2975/setup.py?at=default&fileviewer=file-view-default#setup.py-20

When the package is used, I check the Python version and import the proper source: https://bitbucket.org/moigagoo/cliar/src/a739240126ff2b750d66bb5b9617655e06da2975/cliar/__init__.py?at=default&fileviewer=file-view-default

---

With this approach, when I release a new version, I update the version number only in setup.py and only once. There a single common changelog, too.

Another advantage is that if I have to support more Python versions, say, 3.2, I'll just duplicate the code into py32 and slightly update the version check.

The next step would be moving the meta data into cliar/init.py and importing it in setup.py. I do a similar thing in this package: https://bitbucket.org/moigagoo/sloth-ci/src/42406ee83e0a1fb3a49cfa3c0e732a271329ae6e/setup.py?at=default&fileviewer=file-view-default#setup.py-14

[e12e]

@moigagoo Thank you for your reply! I might have been unclear, I'm not (at this time) interested in supporting more than one version of python. I'm just trying to make sure my VERSION-file is installed when I pip-install my package.

As it is, ./venv/bin/python setup.py install picks it up, while ./venv/bin/pip install git+https://github.com/user/package does not.
I'm just trying to find a supported way to include data-files in my package.

[moigagoo]

@e12e Sorry, I misunderstood you.

The supported way to include data files in your package is with package_data:

package_data={
    'sample': ['package_data.dat'],
},

For example, here's how I ship robots.txt with one of my packages: https://bitbucket.org/moigagoo/sloth-ci-extensions/src/d725682b4de28f1950d6b93220d6b5baa572cecb/robots_txt/setup.py?at=default&fileviewer=file-view-default#setup.py-19

[theacodes]

@ddbeck I'm going to start on this next week. See my comment here

[lgh2]

I think it would also make sense to discuss in the packaging tutorial sample project how to structure the __init__.py file, and what can go in it.

I recently tried making my own package from the official tutorial and ran into problems getting the __init__.py file to behave the way I wanted it to.

For instance, the sample project seems to set up its imports in the __init__.py in such a way that it is necessary to do the following to get to a function in a file in the package:

from package_folder import package_file
from package_file import function

However, in my experience, not many Python packages are set up this way, and it would be helpful to give people the option of setting up their imports so that they can behave in the following manner:

from package_folder import function

For instance, as in:

from requests import get

In addition, the documentation doesn't mention other things that might go in the __init__.py, such as __all__, which controls what the user gets when they run

from package_folder import *

I think some more guidance about what options are available for structuring the imports in __init__.py would be helpful, along with a brief discussion about why the package creator might want to choose one way of doing things over another.

If this isn't the appropriate place for this, please feel free to move this to a new issue.

[theacodes]

That's good feedback, @lgh2. I think at minimum we should link off to the Python doc's description of packages and imports.

[lgh2]

I agree that adding a link to the Python docs about how packages and imports work would be helpful, and it would make sense to link to it and not try to re-create it ourselves.

However, I briefly looked in the Python documentation for distutils and they don't seem to provide the kind of information I was looking for about __init__.py files. I think something like what I was looking for were examples of what the files looked like with examples showing how their imports worked. This might fit better in the sample project, though.

[theacodes]

I meant more of this: https://docs.python.org/3/tutorial/modules.html

But definitely useful for us to provide guidance on using __all__ and import aliases.

[ncoghlan]

I think having an explicit guide in PyPUG for this would make sense (we have one for namespace packages, after all - we just took the design of self-contained packages for granted, even though it's a surprisingly complex topic, especially if you want to keep the internal package structure from leaking into things like pickle files). I don't believe it's tutorial level material though - at the tutorial level, I'm still strongly inclined to favour "Leave your init files empty or omit them entirely". As an example of how symbol re-export can cause problems, reloading submodules of a parent package that exports submodule attributes as package level attributes is one of the easiest ways to end up with two different copies of a module attribute. That means you have to reload the parent package as well to ensure both name bindings get updated.

[lgh2]

Based on what you said in your second paragraph, @ncoghlan, and how complex the topic seems to be, I agree that it would make sense to split out the discussion of the __init__ files into a separate guide.