Keystore/encryption/ipfs usecase for a file-sharing application

[MichaelMure]

As asked by @whyrusleeping, here is my usecase for ipfs and in particular for the keystore/encryption feature.

I'm building a file sharing application presented as a social network. The main target is friend to friend sharing (think holiday pictures) with a control on who has access to the data by using encryption. The underlying goal is to reduce the need to use solution like Facebook, Dropbox, Drive... for data exchange that should be kept private and to create a better solution.

The project is here (no readme, sorry, I should do that). The tech stack is electron, react, redux, ipfs-connector --> js-ipfs-api --> go-ipfs. I'm working on it full-time and hope to have a first (crippled) version in 1-2 months (there is no chance I'll regret having said that in the future, right ? ;) )

What is described below is a general plan and not fully figured out.

Identity

I don't use a central authority like an identity server or a blockchain. As the target is a friend-to-friend network, I chose to have identity simply being a crypto keypair (ipfs key gen with a passphrase) and to build a trust network contact-list for each user. Login in the application will be selecting a profile and entering the passphrase to unlock the private key. Each user publish a profile in his IPNS namespace (ipfs name publish --key mykey), including his public key. I guess it's a good idea to also sign this data (ipfs crypt sign).

Contact management

Each user will have a contact list with the keys stored in the keystore. Contact can be added 'manually' by entering the public key hash of the contact. The profile is then resolved on IPNS (ipfs name resolve contact-hash), fetched, verified (ipfs crypt verify) and imported in the application. I need a way to import the public key in the keystore (MISSING). As both user are not necessarily online at the same time ipfs key send will not do as it is a push. I need a pull version from the profile data.

Users share their contact list with their contact (with some privacy settings). That mean that once the first contact is added, the application can build a suggestion list for the user to easily add new contacts and build his social graph. The same way as before, the profile is retrieved from ipfs and stored locally.

Sharing and encryption

There is 3 kind of sharing. The first second version (I guess the first version will have no encryption at all) will only provide only the targeted sharing.

When a user share something, the data is added to the IPFS repo (using the no-copy filestore ❤️). Metadata are stored as well and a notification is sent using pubsub (ipfs pubsub pub contact-hash share-hash).

Targeted sharing (main priority)

Users can share with a set of specific contact. In this case the data should be encrypted with a generated key that is then encrypted for each recipient public key and added as metadata of the share. As far as I can tell this is not possible with the current encryption spec (MISSING)

The root is also signed by the author (ipfs crypt sign/verify).

Public sharing

Users can share for anyone interested and follow public identity. In this case the data don't really need to be encrypted (?), but the root should be signed (ipfs crypt sign/verify) with the author's key.

Private sharing

User can duplicate their identity on several node and share files to themselves to have kind of private repository of data or synchronization tool. Not really a priority but could be nice. In this case, the data is encrypted using the own author keys and signed.

Relax the online constraint

To relax the constraint of having both user online at the same time for them to communicate, contacts could help each others (they are friends, right ?). That include:

• keeping online contact's profiles (ipfs pin) and profiles ipns record (just a long ipns ttl set by the author)
• keeping online share notification
• keeping online share data using an allocated disk space

Another way would be to use a commercial pinner.

Tracking the download progress

While I'm at it, I need a way to track the progress of a download (MISSING). That means:

• completion event (file added/removed)
• progress event (for the root hash or each file in the tree)
• speed event (how much bandwidth contribute to the progress)

This can partially done by retrieving the file graph using ipfs ls and ipfs refs --recursive and then polling ipfs refs local but it's very inefficient and could easily destroy the performance with big repository.

[whyrusleeping]

I'll need some time to dissect this more thoroughly, but on a first readthrough sounds very interesting.
With regards to relaxing the online constraint, the open bazaar guys have been doing a bit of similar work (cc @cpacia)

As for tracking download progress, we recently added a --progress option to ipfs pin add that might help towards this end.

I'll try and get the keystore encryption apis worked on soon, but no promises on a timeline :)

[whyrusleeping]

The first steps towards signed and encrypted ipfs objects will be deciding on the format. Now that we have ipld we can do better than the weird protobuf things i sketched in the original keystore spec

[cpacia]

What's the plan public sharing? Pubsub with the 'followers' as subscribers?

We'd like to have similar functionality but I'm not sure how well it will work if someone follows a ton of people.

There's also an issue if they come back online from an absence then we want to load the objects that were shared while they were away (easy to do if you only follow a couple nodes but won't work well with larger numbers).

I suppose we could just not load them and only display the last seen messages but there's still the issue of subscribing to many.

[MichaelMure]

Well, that's the more "long term and not really figured out" part.

One of the solution may be to use https://github.com/orbitdb/orbit-db, especially if orbitdb/orbitdb#57 is solved ( @whyrusleeping this needs the keystore/crypto too ;-) ).

This way, each user could have a namespaced space in this database to publish sharing metadata. Data would have a longer lifetime and the would be replicated. That could solve both of your problems.

[whyrusleeping]

Also cc @ianopolous, he's doing similar things with peergos

[ianopolous]

Yep you can check out what we're doing here:
https://github.com/Peergos/Peergos
or
https://peergos.gitlab.io/
and try it out here:
https://demo.peergos.net/

In summary, we are a decentralised file store (built on IPFS) with private sharing between users and we try to hide not only the file data and metadata, but also the social graph (You don't even see your friends' friends by default). Access is capability (CAP) based and can be read only or writable. We achieve, public sharing by just sharing a file/directory's CAP publicly, e.g. in a link:
IPFS Matrix

We don't need a key store, because the root keys are derived from your username and password (and never writtent o disk), which gets you decentralized log-in for free. The remaining keys (every file/dir has several keys associated with it) are stored in a cryptree data structure. All the encrypted file content, including the cryptree structure is stored in a merkle-btree in IPFS.

We can share files 1-to-many in a way that the network can't deduce the social graph (in the data structures at rest, or with live traffic analysis once we use Tor). We are also are careful about quantum computers and current all files are resistant to quantum computer attacks, excluding ones that have been shared (and we're working on that).

We get tracking of file download progress for free because we have to download the fragments ourselves to decrypt them locally (each 5MiB chunk is encrypted independently, and each of those is split into 128 KiB fragments).

Incidentally, @cpacia we've been investigating getting IPFS working over Tor and tried to follow your efforts. Do you have any detailed write-ups of the process?

[cpacia]

@ianopolous yeah it uses OpenBazaar/go-onion-transport and if you look at openbazaard.go in the openbazaar-go repo you should be able to get an idea how to wire it in.

If not let me know.

[autonome]

We're triaging old go-ipfs issues - looks like this has a bunch use-cases that likely live up in the application layer above the protocol layer, but depend on content encryption, which may yet live in the protocol. Closing for now - please reopen if this is a core protocol issue for you.

ipfs/notes#270