Assets management refactored

[gorhill]

This fixes the following issues: #2281, #2037, #1961, #1749, #1429, #1293, #1275, #344.

The purpose of the refactoring was to address many issues arising from the old assets management code:

• uBO can now change the underlying URL of a stock filter list seamlessly, without having to change the subscription itself (what I referred to as "technical debt" in this pull request).
• Stock filter lists (and other assets such as the Public Suffix List) can now be given more than one URL as to where the resource must be fetched. This hardens uBO if ever a specific remote server stops responding -- uBO will fall back on alternative URL(s) if any.
  • A nice side effect is to also solve EasyList & EasyPrivacy - can't update (from "standard" check-boxes) on Windows XP #2037 -- since now alternative URLs can be provided. This is also useful to simplify the handling of Workarounds in build script for Edge and local files without extensions nicole-ashley/uBlock-Edge#53.
• uBO will now respect the Expires: directive in Adblock Plus-compatible filter lists, if present. However the minimal interval is set to 2 days.
• uBO no longer uses checksums.txt resource hosted on GitHub to find out whether some specific assets have changed: the update logic is now completely time-based.
  • checksums.txt will be deprecated and will no longer be updated. Eventually in some future it will be removed from the repository.
• You can view the list of all stock assets in uBO in assets/assets.json, which is itself an update-able asset from uBO's point of view.
  • The setting assetsBootstrapLocation can be used by derivative works of uBO to provide a different set of stock assets, including which filter lists should be enabled by default at install time. There is also a corresponding setting assetsBootstrapLocation which can be used by admins to tell uBO what are the stock assets and which filter lists are enabled by default at install time.
  • assets/ublock/filter-lists.json will be deprecated and will no longer be updated. Eventually in some future it will be removed from the repository.
• The transition to the new asset management code is seamless from the user point of view, i.e. uBO 1.10.5b10 is backward compatible.
  • 1.11.0 (release version of current beta) will also be forward compatible for a limited time ("forward compatible" means one can load newer uBO data in older versions of uBO -- mostly useful to beta testers). This forward compatibility will be removed once 1.11+ is widespread (being forward compatible requires uBO to duplicate data structures in its storage).
• The auto-updater has also been refactored (simplified), and as a result the manual update has changed: there is no more modal widget when manually launching an update operation from the dashboard. Manually launching an update operation merely launches the execution of the exact same code used by auto-updater except with tighter timings.
• Various notes:
  • The "3rd-party filters" pane in the dashboard now renders incrementally (whereas before the lists of filter lists was re-generated from scratch in the DOM).
  • Various "hidden" settings have been added to enable the customization of the assets updater timings (see "Advanced settings").

[ghost]

Sorry for asking this here but:

However the minimal interval is set to 2 days.

Any reason for it being that long? A shorter update interval of maybe a few hours could be useful for lists such as Yhonay/antipopads which have a high update frequency.

Thanks for this huge update by the way.

[gorhill]

I just mind the hit on the remote servers.

[ghost]

That makes sense. I just thought whoever uploads the list can decide how much load they want to have on their server by updating the Expires: directive. They're still under control even if uBO accepted shorter intervals, right?

[okiehsch]

the update logic is now completely time-based

Does this also apply to ublock-resources, or can I still force an update of ublock-resources
by updating another ublock-list?

[gorhill]

can I still force an update of ublock-resources by updating another ublock-list?

No resources.txt won't be marked as out-of-date if purging one filter list. I hadn't thought of this case.

What I could do is to automatically mark resources.txt as out of date when purging "uBlock filters", since this is the filter list which relies the most on resources.txt.