Parachute: a zero knowledge backup system

Use Case

Sink a chute blob

synk ~/Movies/

Installing Chute

Container (Docker, Podman)

Build in Container

podman build -t YOU/chute . 

Run in Container

podman run -it YOU/chute

Raspberry PI (arm32)

To build an image on arm32 with Podman

bash make-arm32-dockerfile.bash 
podman build --file Dockerfile-arm32  . 

Local User Installation

How to build a version for local use, potentially from source.

Unix (macOS)

Install the appropiate package forms (Debian or MacPorts).

See <file:Dockerfile::apt-get install -y screen wget rsync binutils gcc \>

Basic installation of parachute in one command

bash -x ./setup/install-parachute.bash

with the appropiate permissions should enable all the following subclauses to be met automatically.

Windows

Unsupported. Try bash via WSL .

Installing Quicklisp

bash -x setup/install-quicklisp.bash
ansi_cl=abcl
"${ansi_cl}" --batch setup/install-quicklisp.lisp 

If that fails, use the downloaded quicklisp.lisp artifact

(load "~/quicklisp.lisp")
(quicklisp-quickstart:install)
(ql:add-to-init-file)

(ABCL) Install the Quicklisp packages needed by chute

(asdf:make :quicklisp-abcl)
(asdf:make :chute)

Have ASDF build chute

Needed once locally, or after one has updated the sources.

(time (asdf:make :chute))

Test

Have ASDF test chute

(asdf:test-system :chute)

Implementation PRE-ALPHA Quite Unstable

chute is purely a peer: it mediates select parts of your local filesystem with a future version on their worldline. All nodes boot can potentially boot either.

A local management REST API is always available bound to tcp4:2002

URI

Every URI has a local and remote morphism.

URI addressing for previous link

Bootstrap

Boot node for previous references to artifact.

(bootstrap) local

A Common Lisp implementation.

(primary) ABCL

RDF model optionally loaded under ABCL

SBCL

CCL

Most efficient runtime on arm32 platform http://trac.clozure.com/ccl/wiki/PlatformNotes

(bootstrap) remote

Orchestrated services

REST

URI

API

-->  POST /chute/blob/
(index.json)                
<--  201 Resource Created or [45]xx Error
("/new/uri/to/use")
-->  GET /chute/blob/<URI_PATH>/index.json              
nil
<--  200 Original or 304 Not Modified or [45]xx Error
(index.json) or nil
-->  PUT /chute/blob/<URI_PATH>/0    
(application/octet-bytes)
<--  201 Ok
(json) "true" or "false" 
-->  GET /chute/blob/<URI_PATH>/0
<--  200 
(json) SHA256 Hash

Basic Primitive: secure transfer of opaque binary data

PUT with ‘Byte-range’ headers n bytes in m windows. Use timing of previous sessions to dynamically expand/contract number of bytes in each PUT request.

Each chunk of a blob is adressed as a URI of the form:

…/<domain>/<node>/<mount-path>/<timestamp>/<n-byte-chunks>/<mth-chunk>

MIME type for PUT content is “application/octet-stream”.

What is to be DONE

Replace AES-CTR with CHACHA (xCHACHA ?? )

Enable seekable blobs.

Initial implementation at without byte ranges

Implementation use HTTP ‘Byte-range’ header to files attached/detatched via mmap().

Tests for use cases

Metadata transcriptions

Use basic DEFCLASS form, but allow metadata to be as freeform as possible.

Start with CL:PATHNAME for local file objects.

Possibly use mtree for hashing where available as an optimization.

Verify basic transfer

Basic transfer of backups needs to be implemented completely and tested:

REST Transfer Implementation

Initial mplementation completed. Mocks in place for many other systems.

Tests of transfer integrity

CHUTE.TEST::TRANSFER.BLOB.1

Transform a given file into a blob

CHUTE.TEST::TRANSFER.BLOB.2

Use results of BTRFS/SEND into a blob

Read key from key device

Propsoal: extract appropiate number of bytes from SHA256 of the LuKS key

Initialize nonce from random data

Done. But needs to be verified via a test.

Read domain from /opt/cloudfleet/data/config/blimp-vars.sh

Best implementation: exec a bash process then read its environment table

Authentication

Reads don’t matter without the local worldline’s key.

Writes on behalf of need authentication. Chaumian escrow at genesis?

Authentication to be handled via HTTP Headers submitted to
```spire.marina.io``` API.

Future interface for subaddressing components of a blob

For resumable transfers

PUT /<URI>/0/<chunk-bytes>/<nth-chunk>
->>   201 on success or [345]00 
  (json) "true" or "false"

GET /<URI>/0/<chunk-bytes>/<nth-chunk>/hash/sha256  
->>   20x [345]xx (does 314 make sense?)
  (json) SHA256 Hash

BTRFS volume must have “.snapshot/” directory

• CLOSING NOTE [2015-12-14 Mon 13:55]
  Should have been completed in the setup procedure. File issue if found to be otherwise.

The CHUTE:SNAPSHOT command will create snapshots under a the specified volume (by default ”opt/cloudfleet/data”) in a sub-directory named “.snapshot”, i.e. for the default in ”opt/cloudfleet/data.snapshot/<TIMESTAMP>”.

Therefore, the ”opt/cloudfleet/data.snapshot/” must exist and be owned by root. Currently this is done once per node as part of installation when “setup/add-subvolume.bash” is executed. It should be made part of the setup done by setting up the encrypted volumes.

make a hard link to btrfs, setuid

• CLOSING NOTE [2015-10-27 Tue 12:44]
  Done as setup/make-suid-btfs.bash

BTRFS/SEND

• CLOSING NOTE [2024-09-11 Wed 17:38]
  Deprecated. Move purely to working with copies of local files.

stdout/stderr mixing

• CLOSING NOTE [2015-11-14 Sat 10:16]
  Fixed by specifying separate error/output arguments to CCL:RUN-PROGAM

SBCL/CCL returning different starts of output stream. CCL contains “At subvolume” which means it is mixing stdout/stderr?

Use octet streams

• CLOSING NOTE [2015-11-19 Thu 10:47]
  Resolved by use of octets streams directly on MAKE-BLOB

We should pass the streams we wish to read from, rather than having RUN-PROGAM construct them for us.

Unfortunately, this doesn’t seem to work:

(ccl:run-program
   "/bin/ls" nil
   :wait nil
   :output (ironclad:make-octet-input-stream
           (make-array 16 :element-type '(unsigned-byte 8)))
   :element-type '(unsigned-byte 8))

complaining about

There is no applicable method for the generic function:
  #<STANDARD-GENERIC-FUNCTION STREAM-WRITE-VECTOR #x30200006518F>
when called with arguments:
  (#<IRONCLAD::OCTET-INPUT-STREAM #x3020023CA75D> "chute
LICENSE
parachute.org
setup
...

Fortunately, CCL:RUN-PROGRAM takes an :ELEMENT-TYPE keyword which allows us to get octet streams out of the subprocesses.

parse id for PUT uri from POST

• CLOSING NOTE [2015-12-14 Mon 14:09]
  Finished with proper implementation of POST followed by subsequent PUTs to the unspecified sub-uri namespace.

MAKE-NEW-DIRECTORY

• CLOSING NOTE [2024-09-11 Wed 17:39]
  Renamed as CHUTE/FS:MAKE-DIRECTORY.
  • CLOSING NOTE [2015-12-13 Sun 10:40]
    Use CL-FAD routines instead of ASDF.

Fails until run from the REPL. Unsure what this entails.

Generalize filesytem snapshoting abstraction

• CLOSING NOTE [2024-09-11 Wed 16:44]
  Not needed for stage 1.

Implement ZFS snapshotting.

Document configuration of a local ANSI CL implementation and attendent Quicklisp installation

Notes

Odds and Ends

Place chute.asd in ASDF3 system registry

bash -x ./chute/configure-asdf.bash

• CLOSING NOTE [2024-09-11 Wed 16:43]

Osicat failing to grovel

https://github.com/osicat/osicat/pull/11/files

20241109b Colophon

Mark     <[email protected]>
Created: 01-OCT-2015
Revised: <2024-09-12 Thu 10:01>