Switch to ServiceWorker mode by default

[mossroy]

The ServiceWorker mode is faster and cleaner.
At startup, we should first try to enable it by default (except if the user has chosen jQuery mode before).
If it fails, we fallback to jQuery mode.

[sharun-s]

It would be great if you could add some more detail on the difference between these two modes? I think it will help future contributors to get a better idea of the code and why they exist. Right now given the two different formats zim/evo archive, two different content injection modes (probably a third coming with webrequest), the weird zim format(whose advantages aren't clear), references to firefox os/geo location etc the barrier to entry for a dev to figure out what is going on is a bit high.

I think this is a fantastic project with lot of potential for all kinds of offline use-cases and I feel you will get more devs involved and exploring those use cases, if the core pieces of this project are well documented. Thanks for all the work you guys do!

[mossroy]

Thanks for your feedback @sharun-s
I think what you mentioned is not only related to jQuery & ServiceWorker modes, so I created #218 to discuss all this. I'll answer in this new issue

[Jaifroid]

In #383 @mossroy wrote:

Regarding the ServiceWorker mode in general, there are many things that should be easier and faster to achieve with it, using the browser capabilities instead of reimplementing things ourself. That's why I'm willing to focus on it for the next hackathon, in order to (hopefully) make it the default mode (when available on the device/browser). It does not mean we would drop jQuery mode support, because it's still necessary in many contexts.

I agree that the Service Worker approach is cleaner, and the logical way to do things. Effectively, we've written an emulation of ServiceWorker in jQuery, which is quite high maintenance. However, I perceive a number of issues with Service Worker still:

• Firefox (even Quantum) still has not (seemingly) enable Service Worker mode as standard. Running Firefox from localhost, on Windows, I am unable to use Service Worker. Perhaps https: is required? Edge and Chromium now work out of the box, running from localhost in http: ;

• No browser allows Service Workers under the file:// protocol. This does not affect browser extensions, of course. But I've always felt that a major advantage of Kiwix JS is that you can simply run it by clicking on index.html. In fact, I've always thought of this as the major flexibility and use-case for the app. The simplicity of being able to distribute an offline ZIM reader, with a ZIM, on, say, a thumb drive, is a major advantage of Kiwix JS. We cannot expect users to have servers installed on their machines. I realize this advantage fails with Chrome.

• A minor consideration: while Service Workers now work without flags in Edge, they have not yet been ported to UWP. However, this will come soon for Windows 10. It won't, unfortunately, be backported to Windows Mobile. Any future mobile devices (including tablets and small PDA-style devices) from Microsoft and other manufacturers will run Windows on ARM (which is full Windows 10). It will have Service Worker. But current mobile devices will not get the backport or be upgradeable without hacking, sadly.

[mossroy]

Firefox has implemented Service Workers, and it works for me when running from localhost (without https). They are however disabled in ESR versions (so far) : Firefox 52 ESR and Firefox 60 ESR. Please check you're not using an ESR version.
Please check also in about:config that the dom.serviceWorkers.enabled property is set to true.

https should be required for Service Workers only when the host is distant (not from localhost)

But, for now, Firefox disables Service Workers inside a browser extension. I've asked to enable them in https://bugzilla.mozilla.org/show_bug.cgi?id=1344561 : they first answered no, but finally answered yes (but they currently have no timeline for doing it)

Service Workers are disabled under the file:// protocol. It's true that it's at least annoying for development. But I'm not sure double-clicking on index.html is the major use case. IMHO, major use-cases are currently running as an app on mobile and other non-desktop platforms (Windows, FFOS, Ubuntu Touch, maybe other ones), and running as a browser extension. I'm not sure for UWP and Ubuntu touch apps, but the other contexts do not use file:// protocol. In any case, it's something we might ask to the browsers vendors, because I don't see the security threat (provided that the Service Worker only traps requests from its own content to its own content)

It's true that Service Workers are not usable in all contexts. That's why we need to keep the jQuery mode as a fallback.
But I feel the need to put more focus on this ServiceWorker mode now, as it is IMHO the way to go for the future.
I'll try to fix the stability issues of our implementation (#145), so that to be able to make it the default mode (when available)

[Jaifroid]

Apologies, yes, Service Worker is now working for me on Firefox Quantum in localhost. what happens is that sometimes I've been using the ESR version of Firefox, and then when I open the Quantum profile it nevertheless loads up ESR, probably because ESR hasn't fully exited, and that confused me.

Your strategy seems good to me. I think so long as a version is still available that works without SW in the medium term, it'll be fine to initiate a pivot towards SW. UWP works in app context, not in file: context, so Service Workers will work if targeting the latest SDK (but I can't do that because of mobile).

[mossroy]

Maybe it is time to implement this, now that #556 has been merged?
The only drawback I can see is #404 but maybe it's not a blocker.

[Jaifroid]

I would be happy to see this implemented, assuming the device would silently fall back to jQuery mode if SW is not implemented on first startup. Thereafter, if a user attempts to switch to SW mode on a non-supported device, they would receive an explicit error message explaining why not. Regarding #404, I can fix it for you with one line of code, but you won't like the solution (!). I don't think #404 is a blocker, but it is poor UX. However, it should be discussed further there.

Are you happy to implement this issue? (I'm travelling and away from my dev machine for the next week.)

[mossroy]

I agree. The idea is to use SW mode (by default) on platforms that support it, and automatically fallback to jQuery mode (by default) where it's not.
And we should still let the user manually choose the mode (kept in a cookie).
I propose to use another cookie name, so that, when upgrading kiwix-js (only once), we ignore the previous choice of the user (from previous versions where the SW was not great), and switch to the default one : SW if available, jQuery if it's not.

Yes, I can implement it (it is already assigned to me). But I probably won't be able to do that in the near future. I have many things to do (aside from kiwix-js) that will let me very little availability to actually develop, until the end of January 2020.
If you want to implement it by yourself, please do, but don't feel forced to.

[Jaifroid]

Steps remaining to complete this issue (add to this as necessary):

• Add a step in our CI to check the list of pre-cached files for the PWA-ServiceWorker #772
• In ServiceWorker mode, the external links are opened inside the iframe #404
• Fix scrapers to remove inline javascript in ZIM files #865
• Seeking in videos does not always work in SW mode in Chromium extension #531

[kelson42]

#732 is in milestone 3.6 which makes it impossible to implement this ticket in 3.5 milestone.

[Jaifroid]

@kelson42 @mossroy I just saw the discussion of this on Slack. I don't think #732 is a blocker at all. It was more a desirable thing at some point. It can be removed from this list.

[mossroy]

I've updated the list of prerequisites to remove #732 but add #531

[mossroy]

I've started to work a bit on that. I've created a new branch "Switch-to-SW-mode-by-default", but it's still a work-in-progress

[Jaifroid]

@mossroy Can I (or you if you wish) rebase that branch "Switch-to-SW-mode-by-default" on master and then open a PR? I think it would be good to list the steps that need to be completed. In principle, I don't think it's complicated, as the PWA version (moz-extension.kiwix.org) already starts in SW mode by default, but maybe there are some complications we need to take into account, or fallbacks/tests we need to put in place.

[Jaifroid]

PS I only ask because you started that work.

[mossroy]

Sure, you can.
It was still a work-in-progress.

It should not be very complicated, but is not really straight-forward either.
There are indeed many different uses-cases to handle (and test). Here is what I thought about (open to discuss), and started to implement:

• new user/installation with SW supported: use SW (but warn about the switch to PWA for Firefox)
• new user/installation but SW unsupported: fallback to jQuery mode with a one-time warning
• existing installation with SW mode selected: keep SW
• existing installation with jQuery mode and SW supported: we should switch to SW mode with a one-time information (but warn about the switch to PWA for Firefox)
• existing installation with jQuery mode but SW unsupported: we should stay in jQuery mode with a one-time warning

The case of old Firefox browsers with no SW support should be tested, just to check the UX is not too awful, without spending too much time on them (remember what we wrote in 53ba099#diff-484c9e8aa09e09b8eb1fafaf55e8eadcd0eaab230e38d02bb9440841bfa090a4R786)

[mossroy]

Forgot to mention that all these use-cases are only relevant if we want to keep the very broad compatibility we kept since today