Deprecate/feature-freeze the jQuery mode

[mossroy]

We currently have 2 modes : jQuery and ServiceWorker
The ServiceWorker mode natively supports all kind of ZIM files and has more features than the historical jQuery mode. The corresponding code is also simpler, as the jQuery mode needs to manually code many things that should be done by the browser itself.

We have to maintain the jQuery mode for the platforms that do not support ServiceWorkers : old mobile platforms (Firefox OS, Windows Mobile, not sure about Ubuntu Touch), old browsers (see https://caniuse.com/serviceworkers) AND unfortunately Firefox extensions.

My opinion is that we should do a "feature-freeze" on the jQuery mode : continue fixing major bugs but not adding improvements (except if it's trivial).
ServiceWorker mode should become the default mode (#196) and we should focus our energy on it.

Old platforms and old browsers are less and less used : it's not a shame if we don't improve kiwix-js for them. And it would save time to improve it on current and not-so-old platforms.
To me, the main issue is for Firefox extensions. Firefox support ServiceWorkers for years, but not in extensions : we have opened https://bugzilla.mozilla.org/show_bug.cgi?id=1344561 4 years ago but it did not happen yet, and it does not seem to be in their priorities. A a Firefox user and supporter, I'm very sad of this situation, but we can't carry on trying to compensate this missing support in Firefox.

Maybe we could add a kind of banner on platforms that do not support the ServiceWorker mode, saying that kiwix-js is running in a degraded/deprecated mode. If it's a Firefox-family browser, we might also point to the bugzilla issue (and invite people to vote for it). We already display such warning if the user tries to switch to SW mode : we might make that always visible, to avoid user frustrations and bug reports.

[Jaifroid]

I agree SW mode is the way forward. But how do we position Kiwix JS? By default it has come to be the version that serves older and/or minority operating systems and browsers, which is not necessarily a niche that anyone would actively seek, but nevertheless does seem to have a following.

I was looking at statistics for OS usage, which may not be terribly reliable, but as of April 2021, the number of Windows XP and Vista users together (1.3% of worldwide Desktop market share) are not far behind the number of Linux users worldwide (2.2%) - source: amalgamated from this and this, (taking (XP + Vista) / total Windows share). Of course precious few of those will be looking to run Kiwix JS...,

But if we then add all the Firefox Extension users, including one rather famous one (OK, Ice Cat), there is an argument for not abandoning jQuery mode...

I'm not citing this to contradict you, @mossroy, just food for thought about who exactly Kiwix JS aims to reach. I'm not even convinced myself. (It was only recently I learnt how to enable Service Worker mode in the UWP app (main Windows 10, not mobile) with a bit of a backdoor trick.) I do understand that "deprecate" doesn't mean "abandon".

I would be in favour of switching on SW mode by default if it is detected that the system supports it.

[Jaifroid]

I wonder if there is room to be more ambitious than deprecation. I believe, as noted in #725, that it would be possible to unify the API for Service Worker and "jQuery" modes, by channeling all data over the message channel.

This would work by loading a script with article.html that would request and receive data from app.js over the same message channel that the Service Worker currently uses (or a separate message channel leveraging the same code on the app.js side). This would then work in the iframe, in a window, in a tab, or any other container that can communicate via a message channel. It would incidentally eliminate CORS issues. But the big advantage would be that once written, we would eliminate code duplication in app.js, meaning that we could focus on UI and feature improvement without having to implement each feature twice.

Channel messaging is supported in all browsers from IE10 up. See https://developer.mozilla.org/en-US/docs/Web/API/MessageChannel.

I should stress this is just trying to think out of the box. I haven't done any prototyping whatsoever. I cannot think of major obstacles that would prevent this from working, and I think we have a thorough understanding of the APIs involved, but there would no doubt be technical challenges.

[mossroy]

I agree with you that it somehow raises the question of the position of kiwix-js. The point of view of @kelson42 and maybe @Popolechien would be interesting here.

I don't consider Kiwix as a classical app that can afford to easily drop support of old devices, just because their market share is too low. I think we need to do our best to keep support for old devices, because their users might be the ones that need it most.
The fact that kiwix-js runs in the ubiquitous HTML/javascript language makes it probably the kiwix client that supports the rarest devices (I mean devices that have a very low market share).
As you said, deprecating old platforms does not mean dropping their support.
So far, we did not drop support for any device on kiwix-js. And I'm not asking to do it here. It will certainly happen at some point (Internet Explorer is sometimes painful to keep supporting, in particular) but would have to be discussed to check it does not leave too much people with no other solution.

Devices that don't support ServiceWorkers would still be able to use future versions of kiwix-js, and would still benefit from updated ZIM support and general enhancements of the app. But we would not take time to implement jQuery-specific improvements any more. That's the proposal

[mossroy]

Regarding the use of a message Channel in jQuery mode (of #725), it might be interesting to move the jQuery code appart for maintainability (see my comment there).
But, as far as I understood, it would not solve the issues and limitations we have with jQuery mode, and I don't see how if would remove code duplication. We would still have to maintain/tweak the DOM parsing, programmatically do what a browser natively does in SW mode, etc.
To me, this issue and #725 are two distinct topics

[mossroy]

Regarding Firefox extensions, it's the main pain point in my opinion.
Deprecating the jQuery mode means deprecating kiwix-js as a Firefox extension, unless Mozilla finally allows SW in extensions.
We could see it as a way to pressure Mozilla on this missing support, but I don't think we have much weight...

OTOH, we currently don't really distinguish both modes. Except for unsupported ZIM files (with "active content"), nothing warns the user that he/she's using a degraded mode.
That's a reason why we get bug reports on unsupported features in jQuery mode, and people expect them to be "fixed" (which is for us implementing new features with significant additional code)
If people are aware they use a degraded mode, we could instead invite them to switch to another solution (if they can, and if they really need the missing features) : Chrome/Chromium extension, kiwix-desktop, or maybe some Electron/NWJS versions if they now support ServiceWorker mode?

[Jaifroid]

I would hesitate to bug Firefox Extension users with banners. Firefox users tend to be loyal and have strongly held reasons why they won't use Google or Microsoft browsers 😉 (which I do respect and understand), even if underlying Chromium is open-source. Maybe a one-time information panel would be fine (and incorporate the same info into the About page). so long as it is sensitively worded.

Just for information, both Electron and NWJS support Service Workers running in locally packaged code (i.e. fully offline). I've actually released packaged versions of these using the kiwix-js-windows codebase. You could try out the plain Electron/NWJS versions if you want to road-test. There are also WikiMed versions. On user request (kiwix/kiwix-js-pwa#146), I built installable packages for both Windows and Linux (the Linux one is called "Kiwix JS PWA", because it obviously can't have "Windows" in its name). I'll add some notes to #549 to explain my further experience comparing NWJS and Electron.

There is another possible solution for Firefox users: a Progressive Web App (PWA) - possibly inside a Firefox extension. I'll update my notes on this in #388.

[Jaifroid]

@mossroy I think I have a workaround for Firefox Extension users that allows them to access the Service Worker mode. I've done a bit of experimenting today which confirms that:

• Service Worker mode works fine in a Firefox Extension if the extension opens a tab that points to a remote PWA (served over https);
• If the PWA is coded correctly (to cache its own code), the extension works entirely offline (with Service Worker of course).

This means that I believe it would be feasible to adopt the same solution I've adopted with the UWP app, viz.:

• In jQuery mode the extension/app runs from local code (in this case from a moz-extension: URL);
• If the user turns on Service Worker, a banner can be shown which warns the user that one-time access to the secure server will be needed to enable SW mode, so that the app can cache its own code;
• If the user agrees, the app will access the remote server;
• On startup, if SW mode is still on, the app will bootstrap itself from local code to its cached (remote) code, effectively with a redirect or simply by opening the remote tab if it detects SW mode;
• If the user wishes to switch back to jQuery mode, the user will be alerted that the app will switch back to local code, and will be asked to give permission to proceed.

Below are some screenshots of how this works in the UWP app. I think it is a feasible model for Firefox Extensions. It would require #388 to be completed first. It is a workaround rathet than a solution, but one that is perfectly viable.

SWITCHING FROM JQUERY TO SW MODE:

SWITCHING BACK FROM SW MODE TO JQUERY MODE:

To be clear, apart from some controlling code (in backgroundscript.js) that detects the mode of operation and opens the appropriate tab, what this workaround would do is to act as a glorified bookmark to the PWA version. The clever work is done by the Service Worker and the Cache API, not by the extension itself. But for the user, this makes little difference other than that they will notice an https:// address in their browser instead of a moz-extension: address. Above all, this workaround is confirmed to work fully offline once the app has cached its code.

[mossroy]

Thanks @Jaifroid, it's very interesting.
I don't doubt that ServiceWorkers are working in a PWA bundled as a Firefox extension (because files are read from https: scheme, not from moz-extension:) and it might be a kind of workaround.

But the main issue IMHO is that a PWA is not fully offline : the user needs an Internet access on first use (and maybe afterwards if the browser cache happens to be cleared). It will be impossible for some users, but might be acceptable for some other ones.
Is it worth it? I'm asking @kelson42 and @Popolechien

If we go in that direction, we should maybe name and package the PWA differently, so that there is no confusion with the other modes/clients.
The version number of the extension should be distinct for example, because the version of kiwix-js fetched from pwa.kiwix.org would not be linked any more with the version of the extension

[Popolechien]

But the main issue IMHO is that a PWA is not fully offline : the user needs an Internet access on first use (and maybe afterwards if the browser cache happens to be cleared). It will be impossible for some users, but might be acceptable for some other ones. Is it worth it?

I have no visibility on use cases here. I'd say users in this case are probably somewhat online with a poor connection, but that's just a hunch.

Generally speaking I would advise against broadening your portfolio, because at the end of the day it also extends the amount of maintenance work required. Having one product that really nails it (starting with UX) might encourage others to develop variants for those use cases you purposedly left aside, but it really is just my 2c.

[Jaifroid]

@Popolechien This proposal is not really about broadening the portfolio. It's about adding a missing technology back into the Firefox extension, one which Mozilla has seemingly decided against developing (see https://bugzilla.mozilla.org/show_bug.cgi?id=1344561). This is a proposed workaround that provides the missing technology by leveraging another one (Progressive Web Apps) inside the first.

To address @mossroy's points:

• Regarding offline access, the PWA (whether in an Extension or not), works fully offline (after installation) if the machine is offline. If the machine is online, then as per the the Service Worker specification, the worker will check occasionally for any byte changes in its own code, and will update itself if code has changed. The PWA uses an "offline-first" or "cache-first" strategy (see https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Offline_Service_workers).
• What I suggest above provides a lot of control to the user. They can use the app fully offline in jQuery mode, or they can put their whole machine into offline mode if they want/need to use the Service Worker mode fully offline too. When SW mode is requested by the user, the code a) requests permission from the user to access the server; b) explicitly checks that it has access to the site before it attempts to switch to and cache the remote code. This ought to satisfy even the most security-conscious of Firefox users...
• We control what is published on pwa.kiwix.org/kiwix-js .Therefore we can ensure the extension's code and the PWA are always at the same version number.

Is it worth it? It depends how much you value the Firefox users! The others don't need this, unless you like the idea of targeting a simple, installable lightweight, universal PWA! I decided it was worth it to bring Service Worker mode to UWP users, but that is by far the largest target platform for Kiwix JS Windows (18,736 acquisitions).

[mossroy]

Thanks a lot for these explanations.

I thought it was not possible to use the PWA for someone that has no Internet access at all, but maybe I was wrong. Here is the scenario : a user has a computer with Firefox but no Internet, and gets a USB stick with the .xpi extension file and some ZIM files. He/she can install the extension and use it to read the ZIM files : it works with our current versions of kiwix-js. You mean it would also work with a PWA version of the extension (in SW mode, of course)? It would use the bundled HTML/javascript files and serve them through a https: scheme?
This scenario is what I called "fully offline" above. It was not a concern about security/privacy

[Jaifroid]

The user with no Internet access at all is catered for by the jQuery mode, with the .xpi extension on USB stick. If they want to use Service Worker (in the scenario I am proposing), then with the same .xpi extension, they will need to access pwa.kiwix.org at least once to cache the code (as my warnings to the user in the screenshots above make clear). The user can always cancel. If they do access the server once, then the machine can be used offline in this mode thereafter (so long as the user doesn't deliberately clear out the Service Worker, of course). AFAIK, we can't fake or simulate https:// , and CORS issues would probably prevent re-use of the moz-extension: code within the https: code.

What you propose is a pretty extreme scenario @mossroy , but it is nevertheless catered for via jQuery mode. Most users get Kiwix from an online Store or via download, and they also need to download a ZIM file as well (we stress this in our instructions to users). Enabling SW mode in this scenario is no different from any other download, except that it happens transparently. And if the server cannot be reached, the user can stay in jQuery mode, and still have good-enough (for now) functionality.

I'm not pushing this, just offering it if it is of interest. It is only a workaround, not perfect. If it is preferable to leave Firefox Extension users behind by freezing and deprecating jQuery mode, and encourage them to switch to another browser with a warning banner, it's no problem for me! There may well be other, maybe even better, solutions than this one.

[mossroy]

To sum up the discussion we had with @Jaifroid and @kelson42 on Tuesday :

• we all agree to implement Switch to ServiceWorker mode by default #196
• we also agree to freeze the jQuery mode (i.e. do not spend more time on jQuery-specific improvements)
• in the case of Firefox extensions, we might implement the PWA workaround suggested by @Jaifroid : I'd suggest to create a separate issue to discuss it. It looks promising (even if it's not a perfect replacement, as explained above). It looks like a third mode, besides jQuery and SW modes (that might be called "ServiceWorker over PWA"?). We'll have to be careful about how upgrades are handled, so that there can be no confusion on which version is running

The detail on how to handle these changes still has to be discussed : how do we handle the transition, how do we warn the user if stuck in jQuery mode etc

I'm closing this issue : feel free to reopen if needed

[Jaifroid]

Agreed!