tox does not support tags out of the box

[themr0c]

Issue Type

• Bug report

Molecule and Ansible details

master branch

Detail any linters or test runners used:

Desired Behavior

Better documentation about usage of tox tags.

Actual Behaviour

In https://molecule.readthedocs.io/en/latest/testing.html, documentation is stating that you can test with tox tags

pip install tox
tox -t unit

But tox (3.7.0) does not support tags out of the box.

tox -t unit
usage: tox [--version] [-h] [--help-ini] [-v] [-q] [--showconfig] [-l] [-a] [-c CONFIGFILE] [-e envlist] [--notest] [--sdistonly] [-p VAL] [-o] [--parallel--safe-build] [--installpkg PATH]
           [--develop] [-i URL] [--pre] [-r] [--result-json PATH] [--hashseed SEED] [--force-dep REQ] [--sitepackages] [--alwayscopy] [-s [val]] [--workdir PATH]
           [args [args ...]]
tox: error: unrecognized arguments: -t

Is it a feature removed from tox or a dependency we are missing in the documentation ?

[themr0c]

We have better results when installing tox-tags (which is in test-requirements.txt)

either:

pip install -r test-requirements.txt

or:

pip install tox https://github.com/AndreLouisCaron/tox-tags/archive/master.zip#egg=tox-tags

I see later in the file this line, where the usage of the tag has been replaced by a search which is probably obscure for the profane.

tox -e $(tox -l | grep lint | paste -d, -s -)

I believe the most sane solution is following:

1. Replace

pip install pip

by

pip install -r test-requirements.txt

2. Welcome back usage of tags in the documentation, for clarity purpose:

tox -t lint

What is your opinion ? @lwm ? @gundalow ? @webknjaz ?

[themr0c]

Hum, i have not better result: tox -t unit is running only on one version of python apparently. So maybe the best option is to remove tags completely in fact ?

tox -t unit is fine

[decentral1se]

RE: tags, we saw the idea of removing it in #1732 (comment), which I think sounds fine if we can make the setup simpler! If you want to use this issue as a diving board for that, then please do! I have a documentation PR for testing come one day TM.

[themr0c]

tox -e unit and tox -t unit are doing really different things.

IMHO, to run tests on a workstation, the tagged version is more featurefull, from an end user (tester) perspective. I would not run after removing the tags feature.

tox -e unit

=> runs unit tests on one virtualenv, using same python version as the python you are running from (but it does not appear clearly)

  unit: commands succeeded

tox -t unit

=> runs unit tests on all environments tagged as unit:

  py27-ansible25-unit: commands succeeded
  py27-ansible26-unit: commands succeeded
  py27-ansible27-unit: commands succeeded
SKIPPED:  py35-ansible25-unit: InterpreterNotFound: python3.5
SKIPPED:  py35-ansible26-unit: InterpreterNotFound: python3.5
SKIPPED:  py35-ansible27-unit: InterpreterNotFound: python3.5
ERROR:   py36-ansible25-unit: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python3.6 py36-ansible25-unit (see /home/.ecryptfs/src/externals/themr0c/molecule/.tox/py36-ansible25-unit/log/py36-ansible25-unit-0.log)', 1)
ERROR:   py36-ansible26-unit: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python3.6 py36-ansible26-unit (see /home/.ecryptfs/src/externals/themr0c/molecule/.tox/py36-ansible26-unit/log/py36-ansible26-unit-0.log)', 1)
ERROR:   py36-ansible27-unit: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python3.6 py36-ansible27-unit (see /home/.ecryptfs/src/externals/themr0c/molecule/.tox/py36-ansible27-unit/log/py36-ansible27-unit-0.log)', 1)
  py37-ansible25-unit: commands succeeded
  py37-ansible26-unit: commands succeeded
  py37-ansible27-unit: commands succeeded

To achieve same behavior without tags, we would use this insane command, which is really not user friendly.

tox -e $(tox -l | grep unit | paste -d, -s -)

Maybe we should discuss that matter further in the ticket and in next irc meeting ?

[webknjaz]

I didn't know that somebody uses those... OTOH I wonder if there's any other ways which would be easier maintainable.

[themr0c]

@webknjaz i have the same feeling about maintainability.

I could not find in tox documentation some way to aggregate the run of different environments in one command. This tag plugin adresses precisely that: have a simple command to put in the documentation, and to use to test before pushing.

On my laptop I have some issue, though, because py35 and py36 are respectively skipped and failed :/.

[webknjaz]

@themr0c maybe there's not enough copy-paste in tox.ini and some tags are missing?