Refactor _get_html_page() to use exceptions for flow control #5838

uranusjr · 2018-10-01T08:48:28Z

#5800 part 1b-2 (part of #5822). Flow control of HTTP requests is now done with exceptions. I’m going to keep pumping out patches for a while, so feel free to call cut at any point to make a release.

Also switch to use exceptions as flow control for VCS and Content-Type detection. Hopefully this makes the flow much clearer.

cjerdonek

Some initial comments.

cjerdonek · 2018-10-01T09:12:41Z

src/pip/_internal/index.py

+    # Check for VCS schemes that do not support lookup as web pages.
+    vcs_scheme = _match_vcs_scheme(url)
+    if vcs_scheme:
+        raise _SchemeIsVCS(vcs_scheme)


Hmm, using exceptions for flow control here seems to me to be adding extra complexity for not much gain. For example, wouldn't it be more straightforward simply to call a _detect_vcs_scheme() function inside _get_html_page() prior to making the request, and then returning None as needed? Then you don't need to introduce the one-off _SchemeIsVCS as an intermediary. Normally, using exceptions is a last resort thing IMO. I imagine at least some of the other ones can be handled similarly but thought I'd spell out only one.

Hmm, I guess it would work. I went for this still because I wanted to distinguish between the two different early returns, so _get_html_page() get log them appropriately, but you’re right, the VCS check can be done in _get_html_page() instead.

I feel the content type check might be a bit more tricky to handle. The only alternative I can think of is to have two duplicate logging calls.

Yes, I think it will become clearer if you start with some of the easier stuff, and then we can see if the duplication is needed or not. I was also wondering if you might actually want two different log messages anyways, since the two code paths are different.

cjerdonek · 2018-10-01T09:20:59Z

src/pip/_internal/index.py

+    if vcs_scheme:
+        raise _SchemeIsVCS(vcs_scheme)
+
+    for bad_ext in ARCHIVE_EXTENSIONS:


How about breaking out a helper function that simply returns whether the file name ends in a bad extension? Then you can have a simple if in your function instead of a for loop, which should make the calling function a lot flatter and more readable.

cjerdonek · 2018-10-01T09:21:58Z

src/pip/_internal/index.py

+    # url we cannot know ahead of time for sure if something is HTML
+    # or not. However we can check after we've downloaded it.
+    content_type = resp.headers.get('Content-Type', 'unknown')
+    if not content_type.lower().startswith("text/html"):


I see an opportunity for DRY'ing up since content_type.lower().startswith('text/html') occurs twice in this function.

Hmm, I’ll need to think about how this can be done better (along with the bad extensions part).

One way of approaching this might be to start with the more obvious, easier things, and then what you're left with will be simpler to look at and discuss. Let me know if you want to discuss in advance of preparing anything.

For the extensions thing, to start out with, I was suggesting a simple helper function that doesn't involve the network (just returns the bad_ext that matches, if there is one).

cjerdonek · 2018-10-01T11:24:08Z

src/pip/_internal/index.py

+    pass
+
+
+def _raise_for_non_html(headers):


I think this would read more clearly and be easier to understand if phrased positively, like _ensure_html_header(), and similarly for the function below.

cjerdonek · 2018-10-01T11:38:13Z

src/pip/_internal/index.py

+        if not url.endswith('/'):
+            url += '/'
+        url = urllib_parse.urljoin(url, 'index.html')
+        logger.debug(' file: URL is directory, getting %s', url)


Btw, is the above block re-doing the isdir() check from before? If so, can the result of that check be stored along with the URL when the check is done the first time, or alternatively can index.html be appended when it's going through the first time? That way you wouldn't need to go back and forth with url2pathname(), etc.

This applies when expand_dir is false, so is basically mutually exclusive with os.lsdir(). It is super confusing, and I want to sort it out after I’m done with HTMLPage. One at a time…

I agree this can / should be done later. I was just mentioning it because I noticed it. What I was referring to is that isdir() is called in the code here before expand_dir is checked. So I was saying that you could do something like pass around with the URL the information of whether or not it's a directory. That way you wouldn't have to convert the URL to a path again inside this function -- you just do it once. What I don't know though is whether every URL goes through the code path I linked to in my earlier sentence.

uranusjr · 2018-10-01T14:30:46Z

I’ve reworked what each function contains a bit, but I sort of come to the conclusion that _NotHTML is needed. The problem is that I want to (like existing code does) print link in the log messages, but don’t want to pass it into the inner functions. An exception is best handling this kind of jumps IMO.

cjerdonek

This is becoming simpler to understand now. Thanks. So another round of comments based on the most recent changes.

src/pip/_internal/index.py

cjerdonek · 2018-10-02T00:29:48Z

src/pip/_internal/index.py

+
+
+def _ensure_html_response(url, session):
+    """Send a HEAD request to the URL, and ensure the Content-Type is HTML.


Ditto documenting in the docstring that this can raise.

cjerdonek · 2018-10-02T00:32:37Z

src/pip/_internal/index.py

+
+    # Tack index.html onto file:// URLs that point to directories
+    scheme, _, path, _, _, _ = urllib_parse.urlparse(url)
+    if (scheme == 'file' and os.path.isdir(urllib_request.url2pathname(path))):


Can this check and URL conversion be moved to before the try-except and after the VCS check? That way you'd be passing to _get_html_response() the URL that you're actually requesting. It would also simplify the logic because this logic isn't needed to be wrapped in the try-except, so it would make everything easier to understand.

I think there is an edge case if a directory exists at file:///var/files/pip/pip-18.0.tar.gz. The current implementation would reject it (via the archive extension check before), but if this is moved outside of try-except, the index.html file inside it would be requested instead. This naming scheme doesn’t really make sense, so maybe we can rethink whether to treat it as a bug and “fix” it with this refactor.

Good eye! So are you suggesting moving it out of the try-except now, or postponing until a subsequent PR? Since it would both simplify the code a bit as well as fix an edge case (that in addition is probably extremely rare), it doesn't seem like there's any reason not to do it now.

I think we can do it now, and maybe add a fragment to note the behavioural change for the off chance someone is actually depending on it.

cjerdonek · 2018-10-02T00:37:12Z

src/pip/_internal/index.py

+        if not url.endswith('/'):
+            url += '/'
+        url = urllib_parse.urljoin(url, 'index.html')
+        logger.debug(' file: URL is directory, getting %s', url)


I agree this can / should be done later. I was just mentioning it because I noticed it. What I was referring to is that isdir() is called in the code here before expand_dir is checked. So I was saying that you could do something like pass around with the URL the information of whether or not it's a directory. That way you wouldn't have to convert the URL to a path again inside this function -- you just do it once. What I don't know though is whether every URL goes through the code path I linked to in my earlier sentence.

cjerdonek · 2018-10-02T00:39:38Z

src/pip/_internal/index.py

-            },
+        # If the URL looks suspiciously like an archive, send a HEAD first to
+        # check the Content-Type is HTML, to avoid downloading a large file.
+        if _is_filename_like_archive(link.filename):


It seems like these two lines are better inside _get_html_response() since it's essentially an optimization of the _get_html_response() function. It would also make the inside of the try-except just one line / function call, which is more attractive.

I was hesitant to do so since it would require passing in link.filename into _get_html_response(), which does not feel right to me. I’ll push another version of it (by creating an extra Link instance).

src/pip/_internal/index.py

cjerdonek

Some more comments.

cjerdonek · 2018-10-02T10:49:19Z

src/pip/_internal/index.py

+    """Send a HEAD request to the URL, and ensure the response contains HTML.
+
+    Raises `_NotHTML` is the content type is not text/html.
+    """
    scheme, netloc, path, query, fragment = urllib_parse.urlsplit(url)
    if scheme not in {'http', 'https'}:
        # FIXME: some warning or something?


[GitHub isn't letting put the comment two lines down.] With the way you've refactored, should the return '' be replaced by raising _NotHTML? You might want to double-check that. With this new code, an alternative scheme would bypass the content type check.

Also, typo :p

Great catch about the logic error, thanks so much. I added an exception type to reflect this error.

cjerdonek · 2018-10-02T10:50:28Z

src/pip/_internal/index.py

+    1. If the URL looks suspiciously like an archive, send a HEAD first to
+       check the Content-Type is HTML, to avoid downloading a large file.
+    2. If URL scheme is file: and points to a directory, make it point to
+       index.html instead.


The docstring needs to be updated now. Also, (1) can raise _NotHTML, too.

cjerdonek · 2018-10-02T10:52:47Z

src/pip/_internal/index.py

+
+
+class _NotHTML(Exception):
+    def __init__(self, content_type, response):


I worry a little about attaching complex objects to exceptions. Like, could it perhaps prevent the response objects from being garbage-collected since you are later passing the exception to the logging machinery. Normally, it's best to use simple types like strings, etc, IMO. It also ensures the exceptions are serializable (e.g. for logging).

I tend to do this to avoid assumption on what is needed at the catch side, but I guess this doesn’t really matter in this particualr scenario, since _NotHTML would always be internal anyway.

cjerdonek · 2018-10-02T10:56:04Z

src/pip/_internal/index.py

+def _ensure_html_response(url, session):
+    """Send a HEAD request to the URL, and ensure the response contains HTML.
+
+    Raises `_NotHTML` is the content type is not text/html.


src/pip/_internal/index.py

uranusjr · 2018-10-03T18:00:10Z

Added some tests for early return cases. I plan to add more, but am done for the day.

cjerdonek

Great job with the tests!

cjerdonek · 2018-10-03T19:19:32Z

tests/unit/test_index.py

+        ("git+https://github.com/pypa/pip.git", "git"),
+    ],
+)
+def test_get_html_page_invalid_scheme(caplog, url, vcs_scheme):


I think it would help a lot if the ordering of the tests inside the test module can match the ordering of the functions as much as possible. It makes it easier to locate tests and know whether they already exist, etc.

Yeah that would make sense. I was walking through the function and write tests for each point an edge case would appear, hence the current order.

Also (not related to this particular comment) I’m moving the tests to its own file because test_index.py is getting very long after adding these (and there are more to be added).

cjerdonek · 2018-10-03T19:21:03Z

tests/unit/test_index.py

+    """`_get_html_page()` Should append "index.html" to a directory URL.
+    """
+    dirpath = tmpdir.mkdir("something")
+    dir_url = "file:{}".format(urllib_request.pathname2url(dirpath))


There was a test failure around this because of the number of leading slashes. You might want to use urllib_parse / look at how the real code does it, which would also help with cross-platform file paths. Alternatively, you could "spot check" the URL passed, like making sure it begins with "file:" and ends with "/index.html".

Also, how about parametrizing and including a path that's both a directory and looks like an archive? That would cover one of the "fixes" in this PR. Even if that will change later, it would be good to be able to know from the test that something is changing.

PS - re: test failures, do you know how to run tests for a single file or single test function using tox? That makes it a lot easier to run tests locally.

I was a little hesitant to do this since I kind of feel it makes the test less useful if it just matches the implementation. I wonder if I can tweak the temp dir creation to avoid using formatting at all… (p.s. this passed on Windows so I didn’t notice.)

Re: tox, you can pass additional arguments directly to tox, they will be bridged into the command, so something like

tox -e py36 -- tests/unit/test_index.py::test_egg_info_matches

Re: tox, you can pass additional arguments directly to tox

Yes, exactly. I was going to tell you for your benefit in case you didn't already know..

cjerdonek · 2018-10-03T19:27:02Z

tests/unit/test_index.py

+)
+def test_get_html_response_archive_to_http_scheme(url, content_type):
+    """
+    `_get_html_response()` should send a HEAD request on archive-like URL if


"on an archive-like"

cjerdonek · 2018-10-03T19:27:27Z

tests/unit/test_index.py

+)
+def test_get_html_response_archive_to_naive_scheme(url):
+    """
+    `_get_html_response()` should error on archive-like URL if the scheme


"on an archive-like"

cjerdonek · 2018-10-03T19:27:49Z

tests/unit/test_index.py

+
+
+def test_get_html_page_directory_append_index(tmpdir):
+    """`_get_html_page()` Should append "index.html" to a directory URL.


lower-case "should"

cjerdonek · 2018-10-03T19:28:16Z

tests/unit/test_index.py

+    ],
+)
+def test_get_html_page_invalid_scheme(caplog, url, vcs_scheme):
+    """`_get_html_page()` Should error an invalid scheme is given.


"should error if an" (also lower-case should)

uranusjr · 2018-10-04T09:10:08Z

I feel this should cover all code paths :D

cjerdonek

Looks great! Thanks for all your work on this. Just a few minor wording things.

Also, would you mind squashing any of the inessential / throwaway commits like typo commits, etc. with e.g. their previous commit using git rebase -i HEAD~N? (It's okay to have multiple commits in the PR. It would just be nice if each one is more meaningful.) Then I can merge. Thanks again.

cjerdonek · 2018-10-10T08:46:27Z

src/pip/_internal/index.py

+
+
+def _is_url_like_archive(url):
+    """Check whether the URL looks like an archive.


"Return whether ..."

cjerdonek · 2018-10-10T08:48:34Z

src/pip/_internal/index.py

+       `_NotHTML` if it is not HTML.
+    2. Actually perform the request. Raise HTTP exceptions on network failures.
+    3. Check whether Content-Type header to make sure the thing we got is HTML,
+       and raise `_NotHTML` if it's not.


"Check the Content-Type header to make sure we got HTML, and raise _NotHTML if we didn't."

cjerdonek · 2018-10-10T08:49:53Z

tests/unit/test_index_html_page.py

+def test_get_html_response_archive_to_http_scheme_is_html(url):
+    """
+    `_get_html_response()` should work with archive-like URLs if the HEAD
+    request is responded by text/html.


"request responded with text/html."

Should this be “… request is responded with text/html” instead?

Sorry, yes, for that form I would say, "request is responded to with text/html."

But probably even simpler is "if the response to the HEAD request is text/html" or perhaps even "if the HEAD request response is text/html."

This also "fixes" an edge case where a directory is dropped if it is named like an archive.

The original implementation erros out when this happens, and the new one follows. The debug logging message is however changed to be clearer about what happened. Also fixed some minor typos.

cjerdonek · 2018-10-11T19:27:42Z

Thanks again!

1723: Scheduled weekly dependency update for week 05 r=mythmon a=pyup-bot ### Update [atomicwrites](https://pypi.org/project/atomicwrites) from **1.2.1** to **1.3.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/atomicwrites - Repo: https://github.com/untitaker/python-atomicwrites </details> ### Update [botocore](https://pypi.org/project/botocore) from **1.12.82** to **1.12.86**. <details> <summary>Changelog</summary> ### 1.12.86 ``` ======= * api-change:``devicefarm``: Update devicefarm client to latest version * api-change:``codecommit``: Update codecommit client to latest version * api-change:``medialive``: Update medialive client to latest version * api-change:``mediaconnect``: Update mediaconnect client to latest version ``` ### 1.12.85 ``` ======= * api-change:``logs``: Update logs client to latest version * api-change:``elbv2``: Update elbv2 client to latest version * api-change:``rds``: Update rds client to latest version * api-change:``codebuild``: Update codebuild client to latest version * api-change:``sms-voice``: Update sms-voice client to latest version * api-change:``ecr``: Update ecr client to latest version ``` ### 1.12.84 ``` ======= * api-change:``worklink``: Update worklink client to latest version * api-change:``apigatewaymanagementapi``: Update apigatewaymanagementapi client to latest version * api-change:``acm-pca``: Update acm-pca client to latest version ``` ### 1.12.83 ``` ======= * api-change:``appstream``: Update appstream client to latest version * api-change:``discovery``: Update discovery client to latest version * api-change:``dms``: Update dms client to latest version * api-change:``fms``: Update fms client to latest version * api-change:``ssm``: Update ssm client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/botocore - Changelog: https://pyup.io/changelogs/botocore/ - Repo: https://github.com/boto/botocore </details> ### Update [Faker](https://pypi.org/project/Faker) from **1.0.1** to **1.0.2**. <details> <summary>Changelog</summary> ### 1.0.2 ``` -------------------------------------------------------------------------------------- * Fix state abbreviations for ``id_ID`` to be 2-letters. Thanks dt-ap. * Fix format for ``city_with_postcode`` on ``de_DE`` locale. Thanks TZanke. * Update ``person`` providers for ``zh_CN``. Thanks TimeFinger. * Implement ``zipcode_in_state`` and aliases in ``en_US`` locale for generating a zipcode for a specified state. Thanks mattyg. * Group first names by gender on ``zh_CN`` provider. Thanks TimeFinger. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/faker - Changelog: https://pyup.io/changelogs/faker/ - Repo: https://github.com/joke2k/faker </details> ### Update [pycodestyle](https://pypi.org/project/pycodestyle) from **2.4.0** to **2.5.0**. <details> <summary>Changelog</summary> ### 2.5.0 ``` ------------------ New checks: * E117: Over-indented code blocks * W505: Maximum doc-string length only when configured with --max-doc-length Changes: * Remove support for EOL Python 2.6 and 3.3. PR 720. * Add E117 error for over-indented code blocks. * Allow W605 to be silenced by ` noqa` and fix the position reported by W605 * Allow users to omit blank lines around one-liner definitions of classes and functions * Include the function return annotation (``->``) as requiring surrounding whitespace only on Python 3 * Verify that only names can follow ``await``. Previously we allowed numbers and strings. * Add support for Python 3.7 * Fix detection of annotated argument defaults for E252 * Cprrect the position reported by W504 ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pycodestyle - Changelog: https://pyup.io/changelogs/pycodestyle/ - Docs: https://pycodestyle.readthedocs.io/ </details> ### Update [pyflakes](https://pypi.org/project/pyflakes) from **2.0.0** to **2.1.0**. <details> <summary>Changelog</summary> ### 2.1.0 ``` - Allow intentional assignment to variables named ``_`` - Recognize ``__module__`` as a valid name in class scope - ``pyflakes.checker.Checker`` supports checking of partial ``ast`` trees - Detect assign-before-use for local variables which shadow builtin names - Detect invalid ``print`` syntax using ``>>`` operator - Treat ``async for`` the same as a ``for`` loop for introducing variables - Add detection for list concatenation in ``__all__`` - Exempt ``typing.overload`` from duplicate function declaration - Importing a submodule of an ``as``-aliased ``import``-import is marked as used - Report undefined names from ``__all__`` as possibly coming from a ``*`` import - Add support for changes in Python 3.8-dev - Add support for PEP 563 (``from __future__ import annotations``) - Include Python version and platform information in ``pyflakes --version`` - Recognize ``__annotations__`` as a valid magic global in Python 3.6+ - Mark names used in PEP 484 `` type: ...`` comments as used - Add check for use of ``is`` operator with ``str``, ``bytes``, and ``int`` literals ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pyflakes - Changelog: https://pyup.io/changelogs/pyflakes/ - Repo: https://github.com/PyCQA/pyflakes </details> ### Update [setuptools](https://pypi.org/project/setuptools) from **40.6.3** to **40.7.3**. <details> <summary>Changelog</summary> ### 40.7.3 ``` ------- * 1670: In package_index, revert to using a copy of splituser from Python 3.8. Attempts to use ``urllib.parse.urlparse`` led to problems as reported in 1663 and 1668. This change serves as an alternative to 1499 and fixes 1668. ``` ### 40.7.2 ``` ------- * 1666: Restore port in URL handling in package_index. ``` ### 40.7.1 ``` ------- * 1660: On Python 2, when reading config files, downcast options from text to bytes to satisfy distutils expectations. ``` ### 40.7.0 ``` ------- * 1551: File inputs for the `license` field in `setup.cfg` files now explicitly raise an error. * 1180: Add support for non-ASCII in setup.cfg (1062). Add support for native strings on some parameters (1136). * 1499: ``setuptools.package_index`` no longer relies on the deprecated ``urllib.parse.splituser`` per Python 27485. * 1544: Added tests for PackageIndex.download (for git URLs). * 1625: In PEP 517 build_meta builder, ensure that sdists are built as gztar per the spec. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/setuptools - Changelog: https://pyup.io/changelogs/setuptools/ - Repo: https://github.com/pypa/setuptools </details> ### Update [cachetools](https://pypi.org/project/cachetools) from **3.0.0** to **3.1.0**. <details> <summary>Changelog</summary> ### 3.1.0 ``` ------------------- - Fix Python 3.8 compatibility issue. - Use ``time.monotonic`` as default timer if available. - Improve documentation regarding thread safety. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/cachetools - Changelog: https://pyup.io/changelogs/cachetools/ - Repo: https://github.com/tkem/cachetools </details> ### Update [boto3](https://pypi.org/project/boto3) from **1.9.82** to **1.9.86**. <details> <summary>Changelog</summary> ### 1.9.86 ``` ====== * api-change:``devicefarm``: [``botocore``] Update devicefarm client to latest version * api-change:``codecommit``: [``botocore``] Update codecommit client to latest version * api-change:``medialive``: [``botocore``] Update medialive client to latest version * api-change:``mediaconnect``: [``botocore``] Update mediaconnect client to latest version ``` ### 1.9.85 ``` ====== * api-change:``logs``: [``botocore``] Update logs client to latest version * api-change:``elbv2``: [``botocore``] Update elbv2 client to latest version * api-change:``rds``: [``botocore``] Update rds client to latest version * api-change:``codebuild``: [``botocore``] Update codebuild client to latest version * api-change:``sms-voice``: [``botocore``] Update sms-voice client to latest version * api-change:``ecr``: [``botocore``] Update ecr client to latest version ``` ### 1.9.84 ``` ====== * api-change:``worklink``: [``botocore``] Update worklink client to latest version * api-change:``apigatewaymanagementapi``: [``botocore``] Update apigatewaymanagementapi client to latest version * api-change:``acm-pca``: [``botocore``] Update acm-pca client to latest version ``` ### 1.9.83 ``` ====== * api-change:``appstream``: [``botocore``] Update appstream client to latest version * api-change:``discovery``: [``botocore``] Update discovery client to latest version * api-change:``dms``: [``botocore``] Update dms client to latest version * api-change:``fms``: [``botocore``] Update fms client to latest version * api-change:``ssm``: [``botocore``] Update ssm client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/boto3 - Changelog: https://pyup.io/changelogs/boto3/ - Repo: https://github.com/boto/boto3 </details> ### Update [flake8](https://pypi.org/project/flake8) from **3.6.0** to **3.7.5**. <details> <summary>Changelog</summary> ### 3.7.5 ``` ------------------- You can view the `3.7.5 milestone`_ on GitLab for more details. Bugs Fixed ~~~~~~~~~~ - Fix reporting of pyflakes "referenced before assignment" error (See also `GitLab!301`_, `GitLab503`_) .. all links .. _3.7.5 milestone: https://gitlab.com/pycqa/flake8/milestones/28 .. issue links .. _GitLab503: https://gitlab.com/pycqa/flake8/issues/503 .. merge request links .. _GitLab!301: https://gitlab.com/pycqa/flake8/merge_requests/301 ``` ### 3.7.4 ``` ------------------- You can view the `3.7.4 milestone`_ on GitLab for more details. Bugs Fixed ~~~~~~~~~~ - Fix performance regression with lots of ``per-file-ignores`` and errors (See also `GitLab!299`_, `GitLab501`_) .. all links .. _3.7.4 milestone: https://gitlab.com/pycqa/flake8/milestones/27 .. issue links .. _GitLab501: https://gitlab.com/pycqa/flake8/issues/501 .. merge request links .. _GitLab!299: https://gitlab.com/pycqa/flake8/merge_requests/299 ``` ### 3.7.3 ``` ------------------- You can view the `3.7.3 milestone`_ on GitLab for more details. Bugs Fixed ~~~~~~~~~~ - Fix imports of ``typing`` in python 3.5.0 / 3.5.1 (See also `GitLab!294`_, `GitLab498`_) - Fix ``flake8 --statistics`` (See also `GitLab!295`_, `GitLab499`_) - Gracefully ignore ``flake8-per-file-ignores`` plugin if installed (See also `GitLab!297`_, `GitLab495`_) - Improve error message for malformed ``per-file-ignores`` (See also `GitLab!298`_, `GitLab489`_) .. all links .. _3.7.3 milestone: https://gitlab.com/pycqa/flake8/milestones/26 .. issue links .. _GitLab489: https://gitlab.com/pycqa/flake8/issues/489 .. _GitLab495: https://gitlab.com/pycqa/flake8/issues/495 .. _GitLab498: https://gitlab.com/pycqa/flake8/issues/498 .. _GitLab499: https://gitlab.com/pycqa/flake8/issues/499 .. merge request links .. _GitLab!294: https://gitlab.com/pycqa/flake8/merge_requests/294 .. _GitLab!295: https://gitlab.com/pycqa/flake8/merge_requests/295 .. _GitLab!297: https://gitlab.com/pycqa/flake8/merge_requests/297 .. _GitLab!298: https://gitlab.com/pycqa/flake8/merge_requests/298 ``` ### 3.7.2 ``` ------------------- You can view the `3.7.2 milestone`_ on GitLab for more details. Bugs Fixed ~~~~~~~~~~ - Fix broken ``flake8 --diff`` (regressed in 3.7.0) (See also `GitLab!292`_, `GitLab490`_) - Fix typo in plugin exception reporting (See also `GitLab!275`_, `GitLab491`_) - Fix ``AttributeError`` while attempting to use the legacy api (regressed in 3.7.0) (See also `GitLab!293`_, `GitLab497`_) .. all links .. _3.7.2 milestone: https://gitlab.com/pycqa/flake8/milestones/25 .. issue links .. _GitLab490: https://gitlab.com/pycqa/flake8/issues/490 .. _GitLab491: https://gitlab.com/pycqa/flake8/issues/491 .. _GitLab497: https://gitlab.com/pycqa/flake8/issues/497 .. merge request links .. _GitLab!292: https://gitlab.com/pycqa/flake8/merge_requests/292 .. _GitLab!275: https://gitlab.com/pycqa/flake8/merge_requests/275 .. _GitLab!293: https://gitlab.com/pycqa/flake8/merge_requests/293 ``` ### 3.7.1 ``` ------------------- You can view the `3.7.1 milestone`_ on GitLab for more details. Bugs Fixed ~~~~~~~~~~ - Fix capitalized filenames in ``per-file-ignores`` setting (See also `GitLab!290`_, `GitLab488`_) .. all links .. _3.7.1 milestone: https://gitlab.com/pycqa/flake8/milestones/24 .. issue links .. _GitLab488: https://gitlab.com/pycqa/flake8/issues/488 .. merge request links .. _GitLab!290: https://gitlab.com/pycqa/flake8/merge_requests/290 ``` ### 3.7.0 ``` ------------------- You can view the `3.7.0 milestone`_ on GitLab for more details. New Dependency Information ~~~~~~~~~~~~~~~~~~~~~~~~~~ - Add dependency on ``entrypoints`` >= 0.3, < 0.4 (See also `GitLab!264`_, `GitLab!288`_) - Pyflakes has been updated to >= 2.1.0, < 2.2.0 (See also `GitLab!283`_, `GitLab!285`_) - pycodestyle has been updated to >= 2.5.0, < 2.6.0 (See also `GitLab!287`_) Features ~~~~~~~~ - Add support for ``per-file-ignores`` (See also `GitLab!259`_, `GitLab156`_, `GitLab!281`_, `GitLab471`_) - Enable use of ``float`` and ``complex`` option types (See also `GitLab!261`_, `GitLab452`_) - Improve startup performance by switching from ``pkg_resources`` to ``entrypoints`` (See also `GitLab!264`_) - Add metadata for use through the `pre-commit`_ git hooks framework (See also `GitLab!268`_, `GitLab!284`_) - Allow physical line checks to return more than one result (See also `GitLab!269`_) - Allow `` noqa:X123`` comments without space between the colon and codes list (See also `GitLab!273`_, `GitLab470`_) - Remove broken and unused ``flake8.listen`` plugin type (See also `GitLab!274`_, `GitLab480`_) .. all links .. _3.7.0 milestone: https://gitlab.com/pycqa/flake8/milestones/23 .. _pre-commit: https://pre-commit.com/ .. issue links .. _GitLab156: https://gitlab.com/pycqa/flake8/issues/156 .. _GitLab452: https://gitlab.com/pycqa/flake8/issues/452 .. _GitLab470: https://gitlab.com/pycqa/flake8/issues/470 .. _GitLab471: https://gitlab.com/pycqa/flake8/issues/471 .. _GitLab480: https://gitlab.com/pycqa/flake8/issues/480 .. merge request links .. _GitLab!259: https://gitlab.com/pycqa/flake8/merge_requests/259 .. _GitLab!261: https://gitlab.com/pycqa/flake8/merge_requests/261 .. _GitLab!264: https://gitlab.com/pycqa/flake8/merge_requests/264 .. _GitLab!268: https://gitlab.com/pycqa/flake8/merge_requests/268 .. _GitLab!269: https://gitlab.com/pycqa/flake8/merge_requests/269 .. _GitLab!273: https://gitlab.com/pycqa/flake8/merge_requests/273 .. _GitLab!274: https://gitlab.com/pycqa/flake8/merge_requests/274 .. _GitLab!281: https://gitlab.com/pycqa/flake8/merge_requests/281 .. _GitLab!283: https://gitlab.com/pycqa/flake8/merge_requests/283 .. _GitLab!284: https://gitlab.com/pycqa/flake8/merge_requests/284 .. _GitLab!285: https://gitlab.com/pycqa/flake8/merge_requests/285 .. _GitLab!287: https://gitlab.com/pycqa/flake8/merge_requests/287 .. _GitLab!288: https://gitlab.com/pycqa/flake8/merge_requests/288 ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/flake8 - Changelog: https://pyup.io/changelogs/flake8/ - Repo: https://gitlab.com/pycqa/flake8 </details> ### Update [newrelic](https://pypi.org/project/newrelic) from **4.10.0.112** to **4.12.0.113**. <details> <summary>Changelog</summary> ### 4.12.0.113 ``` This release of the Python agent extends support of Amazon's boto3 library and includes bug fixes. The agent can be installed using easy_install/pip/distribute via the Python Package Index or can be downloaded directly from the New Relic download site. Features AWS operation and request ID will now be reported in transaction traces and spans when using boto3 and botocore. The agent will now report aws.requestId and aws.operation for all calls to AWS made using botocore and boto3. DynamoDB calls are now reported under the Databases tab. The agent will now record DynamoDB query performance in the Databases tab in APM in addition to table name for the following calls: put_item get_item update_item delete_item create_table delete_table query scan Certain SQS calls will now report additional data for spans and transaction traces. The agent will now record the queue name in spans and transaction traces for the following SQS calls: send_message send_message_batch receive_message SNS publish will now report additional data for spans and transaction traces. The SNS topic, target, or the string literal PhoneNumber will be reported to New Relic inside of spans and transaction traces. The full URL path will now be recorded on span events and transaction traces when using boto3 or botocore. The agent will now record the full URL path for API calls made to AWS through the boto3 / botocore libraries. The path will be available through span events and transaction traces. Bug Fixes Using newrelic-admin to start a GunicornWebWorker with an application factory resulted in an application crash. The agent would fail to start if using the newrelic-admin command to start an aiohttp application factory with GunicornWebWorker. This issue has now been fixed. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/newrelic - Changelog: https://pyup.io/changelogs/newrelic/ - Homepage: http://newrelic.com/docs/python/new-relic-for-python </details> ### Update [psycopg2](https://pypi.org/project/psycopg2) from **2.7.6.1** to **2.7.7**. <details> <summary>Changelog</summary> ### 2.7.7 ``` ^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Cleanup of the cursor results assignment code, which might have solved double free and inconsistencies in concurrent usage (:tickets:`346, 384`). - Wheel package compiled against OpenSSL 1.0.2q. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/psycopg2 - Changelog: https://pyup.io/changelogs/psycopg2/ - Homepage: http://initd.org/psycopg/ </details> ### Update [pyasn1-modules](https://pypi.org/project/pyasn1-modules) from **0.2.3** to **0.2.4**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pyasn1-modules - Changelog: https://pyup.io/changelogs/pyasn1-modules/ - Repo: https://github.com/etingof/pyasn1-modules </details> ### Update [pytest-django](https://pypi.org/project/pytest-django) from **3.4.5** to **3.4.7**. <details> <summary>Changelog</summary> ### 3.4.7 ``` ------------------ Bugfixes ^^^^^^^^ * Fix disabling/handling of unittest methods with pytest 4.2+ (700) ``` ### 3.4.6 ``` ------------------ Bugfixes ^^^^^^^^ * django_find_project: add cwd as fallback always (690) Misc ^^^^ * Enable tests for Django 2.2 and add classifier (693) * Disallow pytest 4.2.0 in ``install_requires`` (697) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest-django - Changelog: https://pyup.io/changelogs/pytest-django/ - Docs: https://pytest-django.readthedocs.io/ </details> ### Update [pytest](https://pypi.org/project/pytest) from **4.1.1** to **4.2.0**. <details> <summary>Changelog</summary> ### 4.2.0 ``` ========================= Features -------- - `3094 <https://github.com/pytest-dev/pytest/issues/3094>`_: `Class xunit-style <https://docs.pytest.org/en/latest/xunit_setup.html>`__ functions and methods now obey the scope of *autouse* fixtures. This fixes a number of surprising issues like ``setup_method`` being called before session-scoped autouse fixtures (see `517 <https://github.com/pytest-dev/pytest/issues/517>`__ for an example). - `4627 <https://github.com/pytest-dev/pytest/issues/4627>`_: Display a message at the end of the test session when running under Python 2.7 and 3.4 that pytest 5.0 will no longer support those Python versions. - `4660 <https://github.com/pytest-dev/pytest/issues/4660>`_: The number of *selected* tests now are also displayed when the ``-k`` or ``-m`` flags are used. - `4688 <https://github.com/pytest-dev/pytest/issues/4688>`_: ``pytest_report_teststatus`` hook now can also receive a ``config`` parameter. - `4691 <https://github.com/pytest-dev/pytest/issues/4691>`_: ``pytest_terminal_summary`` hook now can also receive a ``config`` parameter. Bug Fixes --------- - `3547 <https://github.com/pytest-dev/pytest/issues/3547>`_: ``--junitxml`` can emit XML compatible with Jenkins xUnit. ``junit_family`` INI option accepts ``legacy|xunit1``, which produces old style output, and ``xunit2`` that conforms more strictly to https://github.com/jenkinsci/xunit-plugin/blob/xunit-2.3.2/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd - `4280 <https://github.com/pytest-dev/pytest/issues/4280>`_: Improve quitting from pdb, especially with ``--trace``. Using ``q[quit]`` after ``pdb.set_trace()`` will quit pytest also. - `4402 <https://github.com/pytest-dev/pytest/issues/4402>`_: Warning summary now groups warnings by message instead of by test id. This makes the output more compact and better conveys the general idea of how much code is actually generating warnings, instead of how many tests call that code. - `4536 <https://github.com/pytest-dev/pytest/issues/4536>`_: ``monkeypatch.delattr`` handles class descriptors like ``staticmethod``/``classmethod``. - `4649 <https://github.com/pytest-dev/pytest/issues/4649>`_: Restore marks being considered keywords for keyword expressions. - `4653 <https://github.com/pytest-dev/pytest/issues/4653>`_: ``tmp_path`` fixture and other related ones provides resolved path (a.k.a real path) - `4667 <https://github.com/pytest-dev/pytest/issues/4667>`_: ``pytest_terminal_summary`` uses result from ``pytest_report_teststatus`` hook, rather than hardcoded strings. - `4669 <https://github.com/pytest-dev/pytest/issues/4669>`_: Correctly handle ``unittest.SkipTest`` exception containing non-ascii characters on Python 2. - `4680 <https://github.com/pytest-dev/pytest/issues/4680>`_: Ensure the ``tmpdir`` and the ``tmp_path`` fixtures are the same folder. - `4681 <https://github.com/pytest-dev/pytest/issues/4681>`_: Ensure ``tmp_path`` is always a real path. Trivial/Internal Changes ------------------------ - `4643 <https://github.com/pytest-dev/pytest/issues/4643>`_: Use ``a.item()`` instead of the deprecated ``np.asscalar(a)`` in ``pytest.approx``. ``np.asscalar`` has been `deprecated <https://github.com/numpy/numpy/blob/master/doc/release/1.16.0-notes.rstnew-deprecations>`__ in ``numpy 1.16.``. - `4657 <https://github.com/pytest-dev/pytest/issues/4657>`_: Copy saferepr from pylib ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Homepage: https://docs.pytest.org/en/latest/ </details> ### Update [pytest-mock](https://pypi.org/project/pytest-mock) from **1.10.0** to **1.10.1**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest-mock - Changelog: https://pyup.io/changelogs/pytest-mock/ - Repo: https://github.com/pytest-dev/pytest-mock/ </details> ### Update [Sphinx](https://pypi.org/project/Sphinx) from **1.8.3** to **1.8.4**. <details> <summary>Changelog</summary> ### 1.8.4 ``` ===================================== Bugs fixed ---------- * 3707: latex: no bold checkmark (✔) available. * 5605: with the documentation language set to Chinese, English words could not be searched. * 5889: LaTeX: user ``numfig_format`` is stripped of spaces and may cause build failure * C++, fix hyperlinks for declarations involving east cv-qualifiers. * 5755: C++, fix duplicate declaration error on function templates with constraints in the return type. * C++, parse unary right fold expressions and binary fold expressions. * pycode could not handle egg files on windows * 5928: KeyError: 'DOCUTILSCONFIG' when running build * 5936: LaTeX: PDF build broken by inclusion of image taller than page height in an admonition * 5231: "make html" does not read and build "po" files in "locale" dir * 5954: ``:scale:`` image option may break PDF build if image in an admonition * 5966: mathjax has not been loaded on incremental build * 5960: LaTeX: modified PDF layout since September 2018 TeXLive update of :file:`parskip.sty` * 5948: LaTeX: duplicated labels are generated for sections * 5958: versionadded directive causes crash with Python 3.5.0 * 5995: autodoc: autodoc_mock_imports conflict with metaclass on Python 3.7 * 5871: texinfo: a section title ``.`` is not allowed ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/sphinx - Changelog: https://pyup.io/changelogs/sphinx/ - Homepage: http://sphinx-doc.org/ </details> ### Update [pip](https://pypi.org/project/pip) from **18.1** to **19.0.1**. <details> <summary>Changelog</summary> ### 19.0 ``` ================= Deprecations and Removals ------------------------- - Deprecate support for Python 3.4 (`6106 <https://github.com/pypa/pip/issues/6106>`_) - Start printing a warning for Python 2.7 to warn of impending Python 2.7 End-of-life and prompt users to start migrating to Python 3. (`6148 <https://github.com/pypa/pip/issues/6148>`_) - Remove the deprecated ``--process-dependency-links`` option. (`6060 <https://github.com/pypa/pip/issues/6060>`_) - Remove the deprecated SVN editable detection based on dependency links during freeze. (`5866 <https://github.com/pypa/pip/issues/5866>`_) Features -------- - Implement PEP 517 (allow projects to specify a build backend via pyproject.toml). (`5743 <https://github.com/pypa/pip/issues/5743>`_) - Implement manylinux2010 platform tag support. manylinux2010 is the successor to manylinux1. It allows carefully compiled binary wheels to be installed on compatible Linux platforms. (`5008 <https://github.com/pypa/pip/issues/5008>`_) - Improve build isolation: handle ``.pth`` files, so namespace packages are correctly supported under Python 3.2 and earlier. (`5656 <https://github.com/pypa/pip/issues/5656>`_) - Include the package name in a freeze warning if the package is not installed. (`5943 <https://github.com/pypa/pip/issues/5943>`_) - Warn when dropping an ``--[extra-]index-url`` value that points to an existing local directory. (`5827 <https://github.com/pypa/pip/issues/5827>`_) - Prefix pip's ``--log`` file lines with their timestamp. (`6141 <https://github.com/pypa/pip/issues/6141>`_) Bug Fixes --------- - Avoid creating excessively long temporary paths when uninstalling packages. (`3055 <https://github.com/pypa/pip/issues/3055>`_) - Redact the password from the URL in various log messages. (`4746 <https://github.com/pypa/pip/issues/4746>`_, `6124 <https://github.com/pypa/pip/issues/6124>`_) - Avoid creating excessively long temporary paths when uninstalling packages. (`3055 <https://github.com/pypa/pip/issues/3055>`_) - Avoid printing a stack trace when given an invalid requirement. (`5147 <https://github.com/pypa/pip/issues/5147>`_) - Present 401 warning if username/password do not work for URL (`4833 <https://github.com/pypa/pip/issues/4833>`_) - Handle ``requests.exceptions.RetryError`` raised in ``PackageFinder`` that was causing pip to fail silently when some indexes were unreachable. (`5270 <https://github.com/pypa/pip/issues/5270>`_, `5483 <https://github.com/pypa/pip/issues/5483>`_) - Handle a broken stdout pipe more gracefully (e.g. when running ``pip list | head``). (`4170 <https://github.com/pypa/pip/issues/4170>`_) - Fix crash from setting ``PIP_NO_CACHE_DIR=yes``. (`5385 <https://github.com/pypa/pip/issues/5385>`_) - Fix crash from unparseable requirements when checking installed packages. (`5839 <https://github.com/pypa/pip/issues/5839>`_) - Fix content type detection if a directory named like an archive is used as a package source. (`5838 <https://github.com/pypa/pip/issues/5838>`_) - Fix listing of outdated packages that are not dependencies of installed packages in ``pip list --outdated --not-required`` (`5737 <https://github.com/pypa/pip/issues/5737>`_) - Fix sorting ``TypeError`` in ``move_wheel_files()`` when installing some packages. (`5868 <https://github.com/pypa/pip/issues/5868>`_) - Fix support for invoking pip using ``python src/pip ...``. (`5841 <https://github.com/pypa/pip/issues/5841>`_) - Greatly reduce memory usage when installing wheels containing large files. (`5848 <https://github.com/pypa/pip/issues/5848>`_) - Editable non-VCS installs now freeze as editable. (`5031 <https://github.com/pypa/pip/issues/5031>`_) - Editable Git installs without a remote now freeze as editable. (`4759 <https://github.com/pypa/pip/issues/4759>`_) - Canonicalize sdist file names so they can be matched to a canonicalized package name passed to ``pip install``. (`5870 <https://github.com/pypa/pip/issues/5870>`_) - Properly decode special characters in SVN URL credentials. (`5968 <https://github.com/pypa/pip/issues/5968>`_) - Make ``PIP_NO_CACHE_DIR`` disable the cache also for truthy values like ``"true"``, ``"yes"``, ``"1"``, etc. (`5735 <https://github.com/pypa/pip/issues/5735>`_) Vendored Libraries ------------------ - Include license text of vendored 3rd party libraries. (`5213 <https://github.com/pypa/pip/issues/5213>`_) - Update certifi to 2018.11.29 - Update colorama to 0.4.1 - Update distlib to 0.2.8 - Update idna to 2.8 - Update packaging to 19.0 - Update pep517 to 0.5.0 - Update pkg_resources to 40.6.3 (via setuptools) - Update pyparsing to 2.3.1 - Update pytoml to 0.1.20 - Update requests to 2.21.0 - Update six to 1.12.0 - Update urllib3 to 1.24.1 Improved Documentation ---------------------- - Include the Vendoring Policy in the documentation. (`5958 <https://github.com/pypa/pip/issues/5958>`_) - Add instructions for running pip from source to Development documentation. (`5949 <https://github.com/pypa/pip/issues/5949>`_) - Remove references to removed ``egg=<name>-<version>`` functionality (`5888 <https://github.com/pypa/pip/issues/5888>`_) - Fix omission of command name in HTML usage documentation (`5984 <https://github.com/pypa/pip/issues/5984>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <[email protected]> Co-authored-by: Peter Bengtsson <[email protected]>

### Update [pip](https://pypi.org/project/pip) from **18.1** to **19.0.1**. <details> <summary>Changelog</summary> ### 19.0 ``` ================= Deprecations and Removals ------------------------- - Deprecate support for Python 3.4 (`6106 <https://github.com/pypa/pip/issues/6106>`_) - Start printing a warning for Python 2.7 to warn of impending Python 2.7 End-of-life and prompt users to start migrating to Python 3. (`6148 <https://github.com/pypa/pip/issues/6148>`_) - Remove the deprecated ``--process-dependency-links`` option. (`6060 <https://github.com/pypa/pip/issues/6060>`_) - Remove the deprecated SVN editable detection based on dependency links during freeze. (`5866 <https://github.com/pypa/pip/issues/5866>`_) Features -------- - Implement PEP 517 (allow projects to specify a build backend via pyproject.toml). (`5743 <https://github.com/pypa/pip/issues/5743>`_) - Implement manylinux2010 platform tag support. manylinux2010 is the successor to manylinux1. It allows carefully compiled binary wheels to be installed on compatible Linux platforms. (`5008 <https://github.com/pypa/pip/issues/5008>`_) - Improve build isolation: handle ``.pth`` files, so namespace packages are correctly supported under Python 3.2 and earlier. (`5656 <https://github.com/pypa/pip/issues/5656>`_) - Include the package name in a freeze warning if the package is not installed. (`5943 <https://github.com/pypa/pip/issues/5943>`_) - Warn when dropping an ``--[extra-]index-url`` value that points to an existing local directory. (`5827 <https://github.com/pypa/pip/issues/5827>`_) - Prefix pip's ``--log`` file lines with their timestamp. (`6141 <https://github.com/pypa/pip/issues/6141>`_) Bug Fixes --------- - Avoid creating excessively long temporary paths when uninstalling packages. (`3055 <https://github.com/pypa/pip/issues/3055>`_) - Redact the password from the URL in various log messages. (`4746 <https://github.com/pypa/pip/issues/4746>`_, `6124 <https://github.com/pypa/pip/issues/6124>`_) - Avoid creating excessively long temporary paths when uninstalling packages. (`3055 <https://github.com/pypa/pip/issues/3055>`_) - Avoid printing a stack trace when given an invalid requirement. (`5147 <https://github.com/pypa/pip/issues/5147>`_) - Present 401 warning if username/password do not work for URL (`4833 <https://github.com/pypa/pip/issues/4833>`_) - Handle ``requests.exceptions.RetryError`` raised in ``PackageFinder`` that was causing pip to fail silently when some indexes were unreachable. (`5270 <https://github.com/pypa/pip/issues/5270>`_, `5483 <https://github.com/pypa/pip/issues/5483>`_) - Handle a broken stdout pipe more gracefully (e.g. when running ``pip list | head``). (`4170 <https://github.com/pypa/pip/issues/4170>`_) - Fix crash from setting ``PIP_NO_CACHE_DIR=yes``. (`5385 <https://github.com/pypa/pip/issues/5385>`_) - Fix crash from unparseable requirements when checking installed packages. (`5839 <https://github.com/pypa/pip/issues/5839>`_) - Fix content type detection if a directory named like an archive is used as a package source. (`5838 <https://github.com/pypa/pip/issues/5838>`_) - Fix listing of outdated packages that are not dependencies of installed packages in ``pip list --outdated --not-required`` (`5737 <https://github.com/pypa/pip/issues/5737>`_) - Fix sorting ``TypeError`` in ``move_wheel_files()`` when installing some packages. (`5868 <https://github.com/pypa/pip/issues/5868>`_) - Fix support for invoking pip using ``python src/pip ...``. (`5841 <https://github.com/pypa/pip/issues/5841>`_) - Greatly reduce memory usage when installing wheels containing large files. (`5848 <https://github.com/pypa/pip/issues/5848>`_) - Editable non-VCS installs now freeze as editable. (`5031 <https://github.com/pypa/pip/issues/5031>`_) - Editable Git installs without a remote now freeze as editable. (`4759 <https://github.com/pypa/pip/issues/4759>`_) - Canonicalize sdist file names so they can be matched to a canonicalized package name passed to ``pip install``. (`5870 <https://github.com/pypa/pip/issues/5870>`_) - Properly decode special characters in SVN URL credentials. (`5968 <https://github.com/pypa/pip/issues/5968>`_) - Make ``PIP_NO_CACHE_DIR`` disable the cache also for truthy values like ``"true"``, ``"yes"``, ``"1"``, etc. (`5735 <https://github.com/pypa/pip/issues/5735>`_) Vendored Libraries ------------------ - Include license text of vendored 3rd party libraries. (`5213 <https://github.com/pypa/pip/issues/5213>`_) - Update certifi to 2018.11.29 - Update colorama to 0.4.1 - Update distlib to 0.2.8 - Update idna to 2.8 - Update packaging to 19.0 - Update pep517 to 0.5.0 - Update pkg_resources to 40.6.3 (via setuptools) - Update pyparsing to 2.3.1 - Update pytoml to 0.1.20 - Update requests to 2.21.0 - Update six to 1.12.0 - Update urllib3 to 1.24.1 Improved Documentation ---------------------- - Include the Vendoring Policy in the documentation. (`5958 <https://github.com/pypa/pip/issues/5958>`_) - Add instructions for running pip from source to Development documentation. (`5949 <https://github.com/pypa/pip/issues/5949>`_) - Remove references to removed ``egg=<name>-<version>`` functionality (`5888 <https://github.com/pypa/pip/issues/5888>`_) - Fix omission of command name in HTML usage documentation (`5984 <https://github.com/pypa/pip/issues/5984>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> ### Update [tox](https://pypi.org/project/tox) from **3.6.1** to **3.7.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/tox - Docs: https://tox.readthedocs.org/ </details> ### Update [PyYAML](https://pypi.org/project/PyYAML) from **4.2b1** to **4.2b4**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pyyaml - Homepage: http://pyyaml.org/wiki/PyYAML </details>

uranusjr added 4 commits October 1, 2018 15:54

Extract VCS scheme check into helper function

7a17a1f

Extract request part of get_page into helper

72779c6

Also switch to use exceptions as flow control for VCS and Content-Type detection. Hopefully this makes the flow much clearer.

Inline PackageFinder._get_page()

bb6dae6

Don't need to unpack prematurely

025dc45

cjerdonek reviewed Oct 1, 2018

View reviewed changes

uranusjr changed the title ~~Refactor _get_html_page() to use exceptions for flow control~~ (WIP) Refactor _get_html_page() to use exceptions for flow control Oct 1, 2018

cjerdonek reviewed Oct 1, 2018

View reviewed changes

uranusjr force-pushed the htmlpage-extract-breakdown-get-page branch 2 times, most recently from 2d872b1 to 8a77bfa Compare October 1, 2018 14:23

uranusjr changed the title ~~(WIP) Refactor _get_html_page() to use exceptions for flow control~~ Refactor _get_html_page() to use exceptions for flow control Oct 1, 2018

cjerdonek added type: refactor Refactoring code skip news Does not need a NEWS file entry (eg: trivial changes) C: finder PackageFinder and index related code labels Oct 1, 2018

cjerdonek added this to the Internal Cleansing milestone Oct 1, 2018

cjerdonek reviewed Oct 2, 2018

View reviewed changes

uranusjr force-pushed the htmlpage-extract-breakdown-get-page branch from e967d84 to e5504af Compare October 2, 2018 14:24

uranusjr mentioned this pull request Oct 3, 2018

Rewrite PackageFinder’s _sort_locations() and related parsing logic #5846

Closed

cjerdonek reviewed Oct 3, 2018

View reviewed changes

src/pip/_internal/index.py Show resolved Hide resolved

src/pip/_internal/index.py Show resolved Hide resolved

cjerdonek reviewed Oct 3, 2018

View reviewed changes

cjerdonek approved these changes Oct 10, 2018

View reviewed changes

uranusjr added 5 commits October 11, 2018 17:44

Move VCS scheme check back out

44713ca

Extract content type detection logic

bad04d8

Group HEAD optimization logic into HTML getter, add docs

231e616

Move directory check out of _get_html_response()

d4da76c

This also "fixes" an edge case where a directory is dropped if it is named like an archive.

More detailed information on Content-Type error, fix typos

a9ebb79

uranusjr added 4 commits October 11, 2018 17:45

Only store what we need in _NotHTML

68cf77a

Fix behavior if archive-like URL is not HTTP(S)

65260d0

The original implementation erros out when this happens, and the new one follows. The debug logging message is however changed to be clearer about what happened. Also fixed some minor typos.

Some tests for error and input correction

5441fc7

Move HTML page tests to their own file, fix typos

200f126

uranusjr force-pushed the htmlpage-extract-breakdown-get-page branch from caa0351 to e1556ae Compare October 11, 2018 09:46

More tests on successful scenarios, fix docstrings

fc53f71

uranusjr force-pushed the htmlpage-extract-breakdown-get-page branch from e1556ae to fc53f71 Compare October 11, 2018 09:47

cjerdonek merged commit 8dbbe16 into pypa:master Oct 11, 2018

uranusjr deleted the htmlpage-extract-breakdown-get-page branch November 1, 2018 08:33

pradyunsg added the pradyunsg-moving-issues label May 25, 2019

pradyunsg removed pradyunsg-moving-issues labels May 25, 2019

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 24, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 24, 2019



		def _ensure_html_response(url, session):
		"""Send a HEAD request to the URL, and ensure the Content-Type is HTML.



		class _NotHTML(Exception):
		def __init__(self, content_type, response):



		def test_get_html_page_directory_append_index(tmpdir):
		"""`_get_html_page()` Should append "index.html" to a directory URL.



		def _is_url_like_archive(url):
		"""Check whether the URL looks like an archive.

Refactor _get_html_page() to use exceptions for flow control #5838

Refactor _get_html_page() to use exceptions for flow control #5838

Conversation

uranusjr commented Oct 1, 2018

cjerdonek left a comment

Choose a reason for hiding this comment

cjerdonek Oct 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr Oct 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr commented Oct 1, 2018

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr Oct 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr commented Oct 3, 2018

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek Oct 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr commented Oct 4, 2018

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek Oct 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjerdonek Oct 11, 2018 • edited Loading

Choose a reason for hiding this comment

cjerdonek commented Oct 11, 2018

cjerdonek Oct 1, 2018 •

edited

Loading

uranusjr Oct 1, 2018 •

edited

Loading

uranusjr Oct 2, 2018 •

edited

Loading

cjerdonek Oct 3, 2018 •

edited

Loading

cjerdonek Oct 10, 2018 •

edited

Loading

cjerdonek Oct 11, 2018 •

edited

Loading