Add cursor to updates RSS feed.

[tiegz]

Greetings! I have two changes to propose for the rss/updates.xml endpoint:

• adds an after cursor parameter: e.g. https://pypi.org/rss/updates.xml?after=1542745017. This would only return releases after the timestamp, in oldest->newest order, which should be safe since there's an index on Release#created.
• increases the page size from 40 to 100 releases per page. (I couldn't figure out why 40 was the original choice when this endpoint was introduced in Add RSS feeds #990)

This should make the feed more useful to regular followers of it, allow them to hit the endpoint less frequently, and allow a way to catch up using the cursor. The legacy changelog XMLRPC call also has a since parameter for pagination, so it seemed useful to allow similar pagination in the RSS feed.

[di]

Thanks for the PR!

I don't see any reason not to increase the page size to 100 releases. I'm not sure there is a historical reason for the current limit of 40.

With regards to the after parameter, we need to consider how this will affect the size of PyPI's cache, and the likelihood that a given request will hit the cache instead of passing through to PyPI's backends.

Right now, we have a single cache entry for this page, which gets updated every time a new release is made, which happens every few seconds or so. Most requests hit the cache.

If we add the after parameter, we go from 1 cache entry to a potential maximum of 2³¹-1 cache entries. In addition, the increase in the number of cache entries reduces the likelihood that two or more given requests will have the same timestamp, and thus hit the cache instead of our backends.

This is part of the consideration for #284 with regards to the "changelog" XML-RPC endpoint as it has the same issue, and attempting to reproduce that as a JSON API has the same problem with the cache here.

[tiegz]

@di Thanks, just broke out the page size change to #9177 to keep it separate.

That caching problem is indeed tricky. The only fix we can think of so far is:

• after date param (e.g. after=2021-03-08): could either do a created desc or created asc ordering
• page param (e.g. page=3): used in conjunction with after, which lets you page through 100-at-a-time for that day.

That'd get us down to a worst case of 21,474,836 cache entries.

[di]

@tiegz One thing that's mentioned in #284 that would work for a JSON API but not for RSS would be including next/previous links in the response. This would let someone consuming the changelog/updates feed start at any point and consume up to the latest change. There would only be N cache entries (where N is the number of changes/releases/etc) and the cache would only ever be invalidated once: when there is a new "head".

I think given that, we probably rather put effort into bringing this new API into existence rather than trying to make the RSS feed more useful.

[tiegz]

@di Makes sense, that feature is definitely better suited for that JSON API. I'll close this and subscribe to that thread to follow future discussion.