ENH: Cleanup with asv_runner

[HaoZeke]

Proof of concept implementation of #1286. What's left is:

• [ ] Add a configuration option to select additional benchmarks mem
• Add pympler in a backwards compatible manner

Adding additional benchmarks should be handled elsewhere, maybe in #1283, since for BC reasons, mem and pympler shouldn't have to be selected by users.

Closes #1286.

EDIT: Updated the name of the core repository

[HaoZeke]

At the moment, if pympler is present; e.g. via:

    "matrix": {
        "req": {
            "pympler": []
        },
    },

then Mem benchmarks will be run, otherwise they're skipped silently.

[HaoZeke]

At the moment, if pympler is present; e.g. via:

    "matrix": {
        "req": {
            "pympler": []
        },
    },

then Mem benchmarks will be run, otherwise they're skipped silently.

This is a backwards incompatible change. Since asv hardcodes the Mem benchmarks, downstream projects expect it to continue working, including the names of the benchmark results files and the requirements matrix. I guess the most compatible way is to ensure pympler is part of _base_requirements whenever possible (i.e., not on pypy).

[HaoZeke]

@mattip this is feature complete, and ready for review, I'm rather baffled by the Appveyor failure, hoping I can deprecate appveyor altogether with #1119.

[mattip]

How is the asvcore package on PyPI created? Is it from this repo?

[HaoZeke]

Currently it's over here:

https://github.com/HaoZeke/asvcore

If the PR and implementation looks good the next step would be to migrate it to the airspeed-velocity organization.

[mattip]

Is there a design document somewhere with how these pieces fit together: what belongs in asvcore and what remains in asv? I am not sure I understand the division.

[HaoZeke]

The design and separation is described a bit in #1286

[mattip]

Where is the pyproject.toml for asvcore? It seems the PyPI entry is missing metadata like the repo, where to file issues, ...

[HaoZeke]

Where is the pyproject.toml for asvcore?

For now:
https://github.com/HaoZeke/asv_runner

It seems the PyPI entry is missing metadata like the repo, where to file issues, ...

Yup I thought it would be better to fill those out once it's migrated to the airspeed-velocity organization instead of where it is now (a repo under my github handle).

[HaoZeke]

@mattip and I discussed that perhaps later it might make sense to have a mechanism for discovering all benchmarks, and then skipping or mark as fail or something where it isn't feasible to run.

Currently this design is because the regex is part of the classes and in the new approach the benchmark types are dynamically populated.

[HaoZeke]

The appveyor failure might stem from an incorrect (hardcoded) file path separator as suggested by @mattip

[HaoZeke]

Incredibly the appveyor failure seems to stem from windows blacklisting aux.* as a filename.. will update asv_runner.

[HaoZeke]

@mattip can we go ahead and merge this since the test failures have been fixed? The rest of the issues (documentation, repo location) can be worked on after that.

[mattip]

This is a big change. Does it have some echo in the documentation? Maybe here?

[HaoZeke]

@mattip I've added some documentation, and also started working on API documentation for asv_runner.

Let me know if it looks alright / what specific questions are not addressed :)