GNIP-84: Upload Page Enhancements

[afabiani]

GNIP-84: Upload Page Enhancements

Overview

The way uploads are currently managed by GeoNode through the “Upload Layers” form is weak and inconsistent most of the time. There are plenty of issues mostly due to a wrong architectural choice, not allowing to correctly manage the “Upload Sessions”.

The proposal is to improve a bit this mechanism in order to make this more robust.

Proposed By

@afabiani [email protected]
@giohappy [email protected]

Assigned to Release

This proposal is for GeoNode 3.2.0.

State

• Under Discussion
• In Progress
• Completed
• Rejected
• Deferred

Motivation

The way uploads are currently managed by GeoNode through the “Upload Layers” form is weak and inconsistent most of the time. There are plenty of issues mostly due to a wrong architectural choice, not allowing to correctly manage the “Upload Sessions”.

Proposal

Upload Sessions are not treated as stateful

While uploading a dataset by using the Upload form, an object will be created into the DB in order to keep the state of the session, along with the uploaded files and user choices, among the upload workflow.

Nevertheless, the UploadSession is not treated as stateful by the Upload form. This causes a set of severe side effects:

1. Whenever something happens during the upload, there won’t be any way to recover the Upload Session or clean up the inconsistent states and partially uploaded files.

2. It won’t be possible to finalize the upload of more than one dataset requiring additional configuration steps, e.g.:

   
   In this example here, it won’t be possible to finalize the upload of all temporal layers, but only one.

   All the files that have been previously uploaded will be lost.

   If the page expires or gets refreshed for some reason, there won’t be any way to recover the uploads.

3. Partially configured datasets appear among the layers’ list, giving the impression to the user that something went wrong.

Technical Proposal

The proposal here would be to make the Upload page stateful and aware of the ongoing Upload sessions, similar to what we already do with the Remote Services.

When harvesting a Remote layer, we take benefits from the asynchronous tasks by following their progress and having the possibility of issue again a failed harvesting job (see the figure below)

The idea here would be to improve the Upload Form in order to be able to read the list of the currently active Upload Session, follow their progress and allow the user to either finalize the workflow or cancel it and consequently make GeoNode able to clean up the pending, unfinished resources.

Moreover the proposal would also like to avoid publishing resources before the whole upload process has finished and notify the user about some pending upload in progress.

By making the Upload form stateful, it would be possible also to allow a superuser to control the pending uploads from other users and take action before the upload actually finishes or recover from broken ones.

Mockups of the Upload Form Improvements

Notify the user about pending uploads

An immediate visual marker will make the user aware of still ongoing uploads.

Stateful Upload Sessions

Notes: We must define a criteria to determine which completed upload should be visible in the list. They cannot disappear immediately, otherwise the user won't understand, but we cannot keep them forever of course. Using a “notification” approach (keeping track of dismiss requests from the user, or the number of times the notification has been shown) would require more work which isn’t worth it now; 1h from upload completion time could be a good good compromise. However if the user uploads something big, goes away and get back to that page after more then 1 hour, he won’t see the completed layer notification...

Backwards Compatibility

Compatible with 3.2.0 and 3.1.2.

Future evolution

Complete refactor of the GeoNode Upload Workflow.

Feedback

Update this section with relevant feedbacks, if any.

Voting

Project Steering Committee:

• Alessio Fabiani: 👍
• Francesco Bartoli: 👍
• Giovanni Allegri: 👍
• Simone Dalmasso:
• Toni Schoenbuchner: 👍
• Florian Hoedt: 👍

Links

Remove unused links below.

• Email Discussion
• Pull Request
• Mail Discussion
• Linked Issue

[t-book]

Thanks Alessio. my +1

[gannebamm]

+1 with a minor question:

Using a “notification” approach ([..]) would require more work which isn’t worth it now [..]

The current system will notify other group members that a new dataset was uploaded via eg email and will list it in the social-activities stream. The new system will not use those?

[francbartoli]

+1 @afabiani.

Just want to check if you have a timeline for this and the overall 3.2. That would help me with a project.

Thanks!

[afabiani]

@francbartoli as soon as this GNIP is finished, we would like to finalize the release. Hopefully we could be able to finalize this one this week.