Design & Plan to add Local Filesystem cloning

[mpacer]

Proposal - Local Filesystem Cloning

This proposes adding the ability to clone from files beneath a privileged directory on your filesystem to your local directory.

Current Limitation

The Jupyter ContentManager API is not normally allowed to access anything outside of its base directory. This prevents users from accessing files above the base directory.

Use Case

The use case arises in cases where you want to provide access to content that is not normally available to the Jupyter Contents API.

For example, suppose that you were running JupyterHub for a number of users and had a filesystem of content that all users should have access to in a readonly way.

Workflow option - Rejected

You could display the contents with commuter, but users would then need to download and upload the content to their local server to be able to get an executable copy of the content. You could mount the file system beneath their Jupyter directory which would let you copy the contents. But that could lead to confusion and clutter in the user's filesystem.

Proposed Workflow

File system cloning would allow you to mount the volume at a in a standard location not beneath the users' base dir, and provide a way for users to directly clone that content as a copy into their local directory. Commuter could then show a clone button that would trigger this API, massively simplifying the workflow needed to go from a readonly to an executable copy of a notebook.

Key Benefit: This allows a similar workflow to be shared by both content hosted on s3 (what bookstore currently supports) as well as content on a filesystem.

^ Edited: The above description was added based on a super helpful suggestion from @willingc

A proposed model for filesystem cloning:

Design

• filesystem cloning is disabled by default (any default would potentially pose a security issue)
• filesystem cloning will only work for files that are children of a specific base directory (fs_cloning_basedir)
• cloning source will be defined using a relative path from the fs_cloning_basedir
• we will use the notebook contents manager to handle security policies around the cloning target path
• separate validation checks for s3 cloning and fs cloning
• create new /api/fs-clone and fs-clone endpoints with relpath query parameter and model key
• modify clone.html template to be used by both sets of handlers

Plan

• Add traitlet for setting filesystem cloning base directory Fs cloning traitlet #155
  • Add validation check for filesystem cloning; change existing cloning validation to be s3 specific Fs cloning traitlet #155
  • Add tests for validation checks Fs cloning traitlet #155
  • Update Configuration docs Fs cloning traitlet #155
  • Expand validation checks to log when server is misconfigured End to End testing Cloning Landing page #159 Add functionality specific logging to validate_bookstore #160
• Create new BookstoreFSCloneHandler with relpath query params Fs cloning handler #158
  • Update clone.html template to be more generic Update clone handler template #156
  • Update REST API docs Add REST docs for file system cloning endpoint #161
  • Update API docs Fs cloning handler #158
  • Create new handler unittests Fs cloning handler #158
  • add /bookstore/fs-clone endpoint End to End testing Cloning Landing page #159
• Create new BookstoreFSCloneAPIHandler with relpath and target_path model params Fs cloning api handler #163
  • Update REST API docs Fs cloning api route #164
  • Update API docs Fs cloning api handler #163
  • add /api/bookstore/fs-clone route & update handler tests Fs cloning api route #164
  • Create new handler unittests Fs cloning api handler #163
    • Test for invalid notebook contents Fs cloning api handler #163
    • Test for trying to target outside of basedir Fs cloning api handler #163
    • Test for trying to get source notebook outside of that specified by fs_cloning_basedir Fs cloning api handler #163
    • Create test_files dir with an example EmptyNotebook.ipynb Fs cloning api handler #163
    • Create successful clone test for EmptyNotebookFs cloning api handler #163
• Add utility method for handling common relpath validation patterns Validate relpath util #157
  • Test for
    • escaping out of base directory Validate relpath util #157
    • submitting absolute path Add functionality specific logging to validate_bookstore #160
• Improve S3 cloning integration testing Add s3clone integration test #162
  • Add S3 clone test for the post endpoint Add s3clone integration test #162
  • Add cleanup methods for the notebook files created. Add s3clone integration test #162
• File system cloning integration testing Add fsclone integration test & format improvements #165
  • Add in tests for fs cloning to integration.js Add fsclone integration test & format improvements #165
  • Create target example ipynb file in /bookstore/tests/test_files directory Add fsclone integration test & format improvements #165
  • Update server configuration to target /bookstore/tests directory Add fsclone integration test & format improvements #165

@chyzzqo2, does this sound like a good plan?

Replaces #110.

[willingc]

@mpacer Thanks for updating the design and plan. I've added a few headings for readability for devs in the future. 😄

[mpacer]

@willingc Thank you so much for adding the structure — it really does make this much easier to read!

[mpacer]

So happy we got through all this so quickly thanks @willingc and @chyzzqo2 for helping make that happen!!