Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make selfoss an offline web application #1014

Merged
merged 1 commit into from
Feb 8, 2019
Merged

Conversation

niol
Copy link
Collaborator

@niol niol commented Jan 11, 2018

Here are a couple of patches that make selfoss an offline web application. I've been using this for some months now and fixed the last issues I could encounter a few weeks ago. This has been working quite well for me in planes or subways with no network access and the app seamlessly syncs back to the server when available.
This pull request contains a lot of code but it is mostly different code paths from the current online behavior, that's why regression are unlikely. I'd be happy to fix any issue reported with this.

controllers/Items.php Outdated Show resolved Hide resolved
public/css/style.css Outdated Show resolved Hide resolved
public/js/selfoss-base.js Outdated Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
public/selfoss.webapp Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
controllers/Index.php Outdated Show resolved Hide resolved
@niol niol force-pushed the ppr/offline branch 2 times, most recently from dde1ad2 to 417bf48 Compare August 21, 2018 13:56
.eslintrc.json Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
public/selfoss.webapp Outdated Show resolved Hide resolved
helpers/View.php Outdated Show resolved Hide resolved
helpers/View.php Outdated Show resolved Hide resolved
public/js/selfoss-base.js Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
public/js/selfoss-db.js Show resolved Hide resolved
if (updatedStatuses) {
// Ensure the status queue is not cleared and gets sync'ed at
// next sync.
var d = $.Deferred();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could use real Promise, if the signature claims it is returned. It is widely supported https://caniuse.com/#feat=promises and if we really need to support IE 11, we can use node_modules/es6-promise/dist/es6-promise.auto.js from this polyfill.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A while ago we wanted to support IE11...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not really know what we want to support. We would probably have to ask Tobias. But I like transpiling releases as described above more and more.

url: 'items/sync',
type: 'GET',
type: updatedStatuses ? 'POST' : 'GET',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not always use POST?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POST changes things whereas GET does not change the database. This is a HTTP convention.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I view POST /items/sync as an endpoint that changes x things, where x can be zero as well. I guess, in true REST, the endpoint would be split into two, one for each synchronization direction.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose this because in one request, changes are committed to the server and new items are retrieved. On a slow link, this may count.

@niol
Copy link
Collaborator Author

niol commented Oct 8, 2018

Thanks a lot for the thorough review.

@niol
Copy link
Collaborator Author

niol commented Oct 8, 2018

Sorry for the missing hunks, I've been in merge hell regarding this for so long that I do not even know what should be part of it anymore.

@jtojnar jtojnar added this to the 2.19 milestone Oct 8, 2018
public/js/selfoss-db.js Outdated Show resolved Hide resolved
daos/mysql/Items.php Outdated Show resolved Hide resolved
public/selfoss.webapp Outdated Show resolved Hide resolved
helpers/View.php Outdated Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
public/js/selfoss-ui.js Outdated Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
templates/home.phtml Outdated Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
@niol
Copy link
Collaborator Author

niol commented Nov 6, 2018

Is there any more work required here?

Copy link
Member

@jtojnar jtojnar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to make sure the new API endpoints are consistent. That means the parameter names will need to be to the point and have clear types. Everything else can be done iteratively after this is merged.

public/js/selfoss-db.js Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
helpers/View.php Outdated Show resolved Hide resolved
helpers/View.php Outdated
'public/css/fonts.css'
],
self::ls('public/images/*'),
self::ls('public/fonts/*.woff')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing fonts in #1072

.htaccess Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
public/js/selfoss-db.js Outdated Show resolved Hide resolved
@@ -190,26 +196,88 @@ public function sync() {
'lastUpdate' => $last_update->format(\DateTime::ATOM),
];

$sinceId = 0;
$wantNewItems = array_key_exists('itemsSinceId', $params)
&& $params['itemsSinceId'] != 'false';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not this be changed now?

helpers/View.php Outdated Show resolved Hide resolved
@jtojnar
Copy link
Member

jtojnar commented Dec 9, 2018

  1. Please add the following message to the first place in “New Features” section of 2.19 changes:
- Support for **using selfoss offline** was added. Note that this is only available in [secure contexts](https://developer.mozilla.org/en-US/docs/Web/Security/Secure_Contexts), that is, over HTTPS. ([#1014](https://github.com/SSilence/selfoss/issues/1014))

  1. In Firefox, logging in with “offline storage” enabled in Private Browsing mode will produce the following error in console:
Unhandled rejection: OpenFailedError: InvalidStateError A mutation operation was attempted on a database that did not allow mutations.
create@http://localhost/selfoss/all.js?v=1544372229:17921:31
enterTransactionScope/<@http://localhost/selfoss/all.js?v=1544372229:17253:21
nativeAwaitCompatibleWrap/<@http://localhost/selfoss/all.js?v=1544372229:16387:20
callListener@http://localhost/selfoss/all.js?v=1544372229:16070:19
endMicroTickScope@http://localhost/selfoss/all.js?v=1544372229:16157:17
wrap/<@http://localhost/selfoss/all.js?v=1544372229:16224:17

From previous: 
then@http://localhost/selfoss/all.js?v=1544372229:15800:22
enterTransactionScope@http://localhost/selfoss/all.js?v=1544372229:17237:38
nativeAwaitCompatibleWrap/<@http://localhost/selfoss/all.js?v=1544372229:16387:20
callListener@http://localhost/selfoss/all.js?v=1544372229:16070:19
endMicroTickScope@http://localhost/selfoss/all.js?v=1544372229:16157:17
wrap/<@http://localhost/selfoss/all.js?v=1544372229:16224:17

From previous: 
resolve@http://localhost/selfoss/all.js?v=1544372229:15915:18
enterTransactionScope@http://localhost/selfoss/all.js?v=1544372229:17237:20
nativeAwaitCompatibleWrap/<@http://localhost/selfoss/all.js?v=1544372229:16387:20
callListener@http://localhost/selfoss/all.js?v=1544372229:16070:19
endMicroTickScope@http://localhost/selfoss/all.js?v=1544372229:16157:17
wrap/<@http://localhost/selfoss/all.js?v=1544372229:16224:17

From previous: 
Dexie/this._whenReady@http://localhost/selfoss/all.js?v=1544372229:16906:56
Dexie/this._transaction@http://localhost/selfoss/all.js?v=1544372229:17235:17
Dexie/this.transaction@http://localhost/selfoss/all.js?v=1544372229:17150:16
_tr@http://localhost/selfoss/all.js?v=1544372229:20533:16
reloadList@http://localhost/selfoss/all.js?v=1544372229:20737:16
reload@http://localhost/selfoss/all.js?v=1544372229:21155:13
reloadList@http://localhost/selfoss/all.js?v=1544372229:21164:13
processHash@http://localhost/selfoss/all.js?v=1544372229:21830:13
setHash@http://localhost/selfoss/all.js?v=1544372229:21894:9
initHash@http://localhost/selfoss/all.js?v=1544372229:21697:9
success@http://localhost/selfoss/all.js?v=1544372229:19708:21
fire@http://localhost/selfoss/all.js?v=1544372229:3188:11
fireWith@http://localhost/selfoss/all.js?v=1544372229:3318:7
done@http://localhost/selfoss/all.js?v=1544372229:8758:5
callback/<@http://localhost/selfoss/all.js?v=1544372229:9124:9
all.js:16428:17
Source map error: request failed with status 404
Resource URL: http://localhost/selfoss/all.js?v=1544372229
Source Map URL: dexie.js.map

  1. In Chromium, in Incognito mode, when I log in with “ofline storage” enabled and then try to mark an item as unread, I get the following error:
Unhandled rejection: TypeError: Cannot read property 'value' of undefined
    at http://localhost/selfoss/all.js?v=1544372229:20983:49
    at http://localhost/selfoss/all.js?v=1544372229:16387:23
    at callListener (http://localhost/selfoss/all.js?v=1544372229:16070:19)
    at endMicroTickScope (http://localhost/selfoss/all.js?v=1544372229:16157:25)
    at IDBRequest.<anonymous> (http://localhost/selfoss/all.js?v=1544372229:16224:17)
From previous: 
    at Promise.then (http://localhost/selfoss/all.js?v=1544372229:15800:22)
    at Table.get (http://localhost/selfoss/all.js?v=1544372229:17397:16)
    at Transaction.<anonymous> (http://localhost/selfoss/all.js?v=1544372229:20980:54)
    at http://localhost/selfoss/all.js?v=1544372229:17262:45
    at http://localhost/selfoss/all.js?v=1544372229:15957:17
    at usePSD (http://localhost/selfoss/all.js?v=1544372229:16368:16)
    at newScope (http://localhost/selfoss/all.js?v=1544372229:16265:14)
    at http://localhost/selfoss/all.js?v=1544372229:15944:20
From previous: 
    at Transaction._promise (http://localhost/selfoss/all.js?v=1544372229:17971:25)
    at Table.getTransaction (http://localhost/selfoss/all.js?v=1544372229:17371:27)
    at Table.getIDBObjectStore (http://localhost/selfoss/all.js?v=1544372229:17382:25)
    at Table.get (http://localhost/selfoss/all.js?v=1544372229:17391:25)
    at Transaction.<anonymous> (http://localhost/selfoss/all.js?v=1544372229:20980:54)
    at http://localhost/selfoss/all.js?v=1544372229:17262:45
    at http://localhost/selfoss/all.js?v=1544372229:15957:17
    at usePSD (http://localhost/selfoss/all.js?v=1544372229:16368:16)

Curiously once I refresh the page, it will no longer complain.


  1. After I first time log in with “offline storage” enabled (e.g. in Chromium anonymous mode), it will claim “no items available”:

image

If fixes after reloading.


  1. If I enable offline mode in the Chromium dev tools and reload the page, selfoss will crash with the following error:
The FetchEvent for "http://localhost/selfoss/items/sync?since=2018-12-09T15%3A53%3A02.000Z&tags…emsSinceId=562&itemsNotBefore=2018-12-08T16%3A58%3A18.754Z&itemsHowMany=50" resulted in a network error response: the promise was rejected.
selfoss-sw-offline.js:1 Uncaught (in promise) TypeError: Failed to fetch
all.js?v=1544372229:9176 GET http://localhost/selfoss/items/sync?since=2018-12-09T15%3A53%3A02.000Z&tags…emsSinceId=562&itemsNotBefore=2018-12-08T16%3A58%3A18.754Z&itemsHowMany=50 net::ERR_FAILED

  1. Switching to offline mode will hide tags and sources.

@niol
Copy link
Collaborator Author

niol commented Dec 10, 2018

Point 2: disabled offline because of https://bugzilla.mozilla.org/show_bug.cgi?id=781982
Points 3-4-5: no chrome/chromium available now, will get back to you.
Point 6: source and tags are disabled because not supported in offline mode. #1064 will have to be merged before I add support for source and tags offline.

@niol
Copy link
Collaborator Author

niol commented Dec 11, 2018

I had no luck reproducing points 3 and 4. Will try again another time.

@jtojnar
Copy link
Member

jtojnar commented Dec 11, 2018

I have the following config:

[globals]
username=admin
password=15fa12535a14fdcf8d6587239d7d01b26192ad987d449e37c435e7fe375020d687ceb66f4f836733299b42ac41ed4b860849074d0ec954957f00b50791ce7d30
salt=lkjl1289
db_port=
db_prefix=
public=1
logger_level=sDEBUG
auto_mark_as_read=1
; logger_destination=error_log
language=cs
  1. Here is the video: https://gfycat.com/UncommonDefiniteBlowfish

    The error message bar now appears and the console error is different:

    all.js?v=1544565637:16428 Unhandled rejection: TypeError: Cannot read property 'entries' of null
        at refreshStats (http://localhost/selfoss/all.js?v=1544565637:20851:62)
        at http://localhost/selfoss/all.js?v=1544565637:16387:23
        at callListener (http://localhost/selfoss/all.js?v=1544565637:16070:19)
        at endMicroTickScope (http://localhost/selfoss/all.js?v=1544565637:16157:25)
        at IDBRequest.<anonymous> (http://localhost/selfoss/all.js?v=1544565637:16224:17)
    From previous: 
        at Promise.then (http://localhost/selfoss/all.js?v=1544565637:15800:22)
        at Object.storeEntryStatuses (http://localhost/selfoss/all.js?v=1544565637:20989:16)
        at Object.entriesMark (http://localhost/selfoss/all.js?v=1544565637:20998:34)
        at Object.entryMark (http://localhost/selfoss/all.js?v=1544565637:21003:34)
        at HTMLButtonElement.<anonymous> (http://localhost/selfoss/all.js?v=1544565637:22651:35)
        at HTMLButtonElement.dispatch (http://localhost/selfoss/all.js?v=1544565637:4738:27)
        at HTMLButtonElement.elemData.handle (http://localhost/selfoss/all.js?v=1544565637:4550:28)
        at Object.trigger (http://localhost/selfoss/all.js?v=1544565637:7808:12)
    From previous: 
        at Promise.then (http://localhost/selfoss/all.js?v=1544565637:15800:22)
        at Promise.catch (http://localhost/selfoss/all.js?v=1544565637:15830:25)
        at Object._tr (http://localhost/selfoss/all.js?v=1544565637:20535:19)
        at Object.storeEntryStatuses (http://localhost/selfoss/all.js?v=1544565637:20934:34)
        at Object.entriesMark (http://localhost/selfoss/all.js?v=1544565637:20998:34)
        at Object.entryMark (http://localhost/selfoss/all.js?v=1544565637:21003:34)
        at HTMLButtonElement.<anonymous> (http://localhost/selfoss/all.js?v=1544565637:22651:35)
        at HTMLButtonElement.dispatch (http://localhost/selfoss/all.js?v=1544565637:4738:27)
    From previous: TypeError: Cannot read property 'value' of undefined
        at http://localhost/selfoss/all.js?v=1544565637:20983:49
        at http://localhost/selfoss/all.js?v=1544565637:16387:23
        at callListener (http://localhost/selfoss/all.js?v=1544565637:16070:19)
        at endMicroTickScope (http://localhost/selfoss/all.js?v=1544565637:16157:25)
        at IDBRequest.<anonymous> (http://localhost/selfoss/all.js?v=1544565637:16224:17)
    From previous: 
        at Promise.then (http://localhost/selfoss/all.js?v=1544565637:15800:22)
        at enterTransactionScope (http://localhost/selfoss/all.js?v=1544565637:17237:38)
        at Dexie._whenReady (http://localhost/selfoss/all.js?v=1544565637:16906:49)
        at Dexie._transaction (http://localhost/selfoss/all.js?v=1544565637:17235:20)
        at Dexie.transaction (http://localhost/selfoss/all.js?v=1544565637:17150:34)
        at Object._tr (http://localhost/selfoss/all.js?v=1544565637:20534:14)
        at Object.storeEntryStatuses (http://localhost/selfoss/all.js?v=1544565637:20934:34)
        at Object.entriesMark (http://localhost/selfoss/all.js?v=1544565637:20998:34)
    From previous: 
        at Function.resolve (http://localhost/selfoss/all.js?v=1544565637:15915:18)
        at enterTransactionScope (http://localhost/selfoss/all.js?v=1544565637:17237:28)
        at Dexie._whenReady (http://localhost/selfoss/all.js?v=1544565637:16906:49)
        at Dexie._transaction (http://localhost/selfoss/all.js?v=1544565637:17235:20)
        at Dexie.transaction (http://localhost/selfoss/all.js?v=1544565637:17150:34)
        at Object._tr (http://localhost/selfoss/all.js?v=1544565637:20534:14)
        at Object.storeEntryStatuses (http://localhost/selfoss/all.js?v=1544565637:20934:34)
        at Object.entriesMark (http://localhost/selfoss/all.js?v=1544565637:20998:34)
    

  1. And here is a recording for this issue: https://gfycat.com/UnsteadyGroundedHyrax

@niol
Copy link
Collaborator Author

niol commented Dec 29, 2018

4 should be ok, still cannot reproduce 3.

@jtojnar
Copy link
Member

jtojnar commented Dec 29, 2018

I can still reproduce both issues on e801e85. Here are the screenshots of the debugger for third issue (both the console error and the error bar message):

screenshot from 2018-12-29 21-19-30
screenshot from 2018-12-29 21-17-37

Also, the first time I open the app in the anonymous mode of Chrome, it shows “selfoss has been updated, please reload. Reload” but the reload button only dismisses the message bar, no reload is performed. I am not even sure why it is displayed, since I would assume there is no previous state in the service worker.

@jtojnar
Copy link
Member

jtojnar commented Jan 26, 2019

  1. Table.get returns undefined when the item could not be found. That means The only two places wherestats.put` is called is

    1. https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/public/js/selfoss-db.js#L771 where the issue occurs
    2. in a function https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/public/js/selfoss-db.js#L498-L510
      called in the dbOnline.sync function https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/public/js/selfoss-db.js#L171-L173
      when sync property is in the object returned by the request to /items/sync

And I can reproduce both issues in Firefox too, it is just that deleting localStorage, indexedDB, and unregistering the worker in about:debugging#workers is much less convenient than running in fresh anonymous session like Chromium allows.

@jtojnar
Copy link
Member

jtojnar commented Jan 26, 2019

If I wrap the then handler for the transaction in storeEntryStatuses into a named closure, I can see that it is here what calls refreshStats causing the secondary exception from operating on already nulled storage:

screenshot from 2019-01-26 21-10-20

I do not understand why does it call then handler, even when the error is already handled in catch inside _tr definition.


On a side note, perhaps a method like the following one would be useful for developments:

selfoss.nukeLocalData = function() {
    selfoss.db.clear(); // will not work after a failure, since storage is nulled
    window.localStorage.clear();
    navigator.serviceWorker.getRegistrations().then(function(registrations) {
        registrations.forEach(function(reg) {
            reg.unregister();
        });
    });
    selfoss.logout();
};

@jtojnar
Copy link
Member

jtojnar commented Jan 26, 2019

As for number four, I think this is caused by reloadList not being called https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/public/js/selfoss-db.js#L190-L193 again occurring when stats is missing from response.

@jtojnar
Copy link
Member

jtojnar commented Feb 5, 2019

This fixed the second part of 3 but the initial issue of stats table being empty persists.

Perhaps we should just use stat.value || 0 in:

https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/public/js/selfoss-db.js#L773

As for 4, what do you think about removing 'stats' in data && data.stats.unread > 0 from the condition:

https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/public/js/selfoss-db.js#L190-L193

@niol
Copy link
Collaborator Author

niol commented Feb 5, 2019

Can I have the actual stack for both errors, as a summary of what is left to be fixed ?
stats is never missing even on my first query. Maybe I need to try with an empty database.

Maybe your workarounds would work, but I fail to understand why the failure occurs in the first place. And has I cannot reproduce this behaviour, this is quite difficult to fix for me.

@jtojnar
Copy link
Member

jtojnar commented Feb 5, 2019

There are no more exceptions, the population fixed it. (Though now I am going to negative numbers when I read articles.)

I do not have an empty database.

--- a/common.php
+++ b/common.php
@@ -13,7 +13,7 @@ if ($autoloader === false) {
 
 $f3 = $f3 = Base::instance();
 
-$f3->set('DEBUG', 0);
+$f3->set('DEBUG', 1);
 $f3->set('version', '2.19-SNAPSHOT');
 $f3->set('AUTOLOAD', false);
 $f3->set('cache', __DIR__ . '/data/cache');
--- a/controllers/Items.php
+++ b/controllers/Items.php
@@ -191,6 +191,8 @@ class Items extends BaseController {
         $itemsDao = new \daos\Items();
         $last_update = new \DateTime($itemsDao->lastUpdate());
 
+        \F3::get('logger')->info('Sync since=' . print_r($since, true) . '; last_update=' . print_r($last_update, true));
+
         $sync = [
             'lastUpdate' => $last_update->format(\DateTime::ATOM),
         ];

I am just getting

[2019-02-05 17:17:29] selfoss.INFO: Sync since=DateTime Object
(
    [date] => 2019-02-05 15:01:32.000000
    [timezone_type] => 2
    [timezone] => Z
)
; last_update=DateTime Object
(
    [date] => 2019-02-05 16:01:32.000000
    [timezone_type] => 3
    [timezone] => Europe/Berlin
)

which would invalidate the condition

https://github.com/SSilence/selfoss/blob/e801e85919cf4ee35994cbfede45b0a5c9fc5d85/controllers/Items.php#L242-L243

@niol
Copy link
Collaborator Author

niol commented Feb 6, 2019

Nailed it: when there are many items to load, the items are loaded online for the user not to wait the complete population of the offline db. However, in that particular case of storage being null in private mode, reloadList sets the lastUpdate as received and this one gets sent in the sync request instead of Date(0), and stats are not sent.
I'll think about a fix for this which I'll send probably today.

@jtojnar
Copy link
Member

jtojnar commented Feb 6, 2019

Thanks, this has done it. I think it is ready to merge after a rebase/squash.

@niol
Copy link
Collaborator Author

niol commented Feb 6, 2019

Do not hesitate to let me know if I should do the rebase/squash.

@jtojnar
Copy link
Member

jtojnar commented Feb 6, 2019

It is always better if the author does the rebase, since then it will be signed by their GPG key.

This change implements client side storing of items in a IndexedDB
database. It also makes the changes necessary to use it instead
of loading everything from the server each time.

This makes it possible to use selfoss offline for items that have
been loaded when last online.
@jtojnar jtojnar merged commit 4e44392 into fossar:master Feb 8, 2019
@jtojnar
Copy link
Member

jtojnar commented Feb 8, 2019

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants