feat: pin API (#1045)

* revert: default assets are not added when running on a browser

refactor: change pin.flush logging message

* feat(test): add tests for failure cases of normalizeHashes

fix: don't need to cast the object.get result with toJSON

revert: use interface-datastore.Key for datastore pin storage
The proper change would be that datastore-level automatically casts operations into Keys

fix: do not invoke callback within a try/catch

feat(test): make cli pin tests more robust
By using files that aren't added on IPFS initialization. Still needs work on files.rm (direct) and ipfs ls (indirect).

fix: remove commented code, traced test failures to pin-set
Got to go for the night, though, so will checkpoint here and address tomorrow.

feat: parseIpfsPath now throws errors for consistency

feat: resolveIpfsPaths error message lists the relative path that failed

feat: use follow.bind instead of mutating the links
Also decided not show relative paths. Less human friendly but probably cleaner otherwise.

refactor: resolveIpfsPaths -> resolvePaths

feat: promisify resolvePaths

test: change parseIpfsPath failure tests to use try/catch

docs: edit resolvePath doc

revert: accidentally deleted commands/pin.js

* feat: jsipfs pin improvements (#1249)

* initial sweep through to understand how pin works. Did make some changes but mostly minor.

* refactor pb schema to it's own file

* fix: don't pin files during files.add if opts.pin === false

* feat: add some http qs parsing, http route/resources cleanup, cleanup core/utils.parseIpfsPath

* feat: expand pin tests. \nFirst draft. still needs some further work.

* feat: Add logging for entry/exit of pins: add/rm/flush/load. Clean some documentation.

* feat: add --pin to files.add, fix: improper pin option parsing in core.

* feat: Use ipfs.files.add to add init-docs instead of directly using the unix-fs importer.

* feat(tests): Add tests for cli --pin option. I know this should be more of an integration test. Should be written in /core. Maybe talk with Victor about testing different layers

* feat: use isIPFS to valiate a multihash.

* fix: add some changes missed during rebase, syntax fixes, etc

I think my original rebase for this branch 2 weeks ago might have
changed history for the intervening commits, indirectly causing some of
these missed changes. or I just rebase onto the wrong oldparent.

fix: some onlyHash and pin tests broke after merging
onlyHash and pin interact: shouldn't pin when --only-hash.

fix: trim output for 'pin ls when no hash is passed'

test: indirect pins supersede direct pins: turns out we had a bug

feat: add expectTimeout test utility

feat: promisify some additional pin utils

* test: initial work testing the core/pin.js implementation
I think I'll end up moving most tests here.

test: add tests for pin.ls and pin.rm
Based tests on other pin fixtures, need to migrate the isPinned* tests to them as well.

fix: direct pins are now deleted by a default pin.rm(hash)

test: prepare for pin.add tests
'indirect supersedes direct' test exposes a bug in pin.ls

feat: switch away from multihashes for isPinned* tests

test: impl pin.add tests

fix: add fixture files only once

test: add test for a potential bug, clean isPinned* tests

refactor: remove a test that's no longer needed

fix: pin.ls, indirect pins should supersede direct pins

test: naive pin.load, pin.flush tests

feat: remove most pin cli tests as functionality is tested in pin core tests

refactor: rename solarSystem

* refactor: move pin http-api tests to http-api/inject

fix: attempt to find a way to use http-api/inject test structure for pin tests

test: fix pin.rm http-api tests

test: fix pin.add http-api tests

docs: docs and cleanup of http-api pin tests

refactor: renaming

fix: lint errors

fix: resolvePaths tests are failing on CI, it might be long ops, testing a timeout bump

fix: add files explicitly before testing resolvePaths

fix: remove mocha.only from resolvePaths. let's hope tests pass, they are passing CI now

fix: rename test/core/utils.spec.js -> utils.js so it's not run during browser tests

* test: first draft of pin-set tests
Need to leave computer, this is a checkpoint.

test: add sanity test for walkItems and hasChild, clean others
These tests are more descriptive than really pushing the impl. I'd love others' thoughts on what else should be hit and how. I also need to compare go's pinset impl against ours

fix: stop daemons

feat: documentation and multihash buffer handling for dag.get

fix: lint

* feat: simplify root dagnode generation for storeItems base case

* feat: rename vars, fix _depth default value, add docs

fix: pinset.hasChild buffer check

feat: hardcode expected length for flush/load tests

* feat: parallelize pin.isPinnedWithType

* refactor: refactor pinset.storeItems

* fix: re-add pin interface tests
I must have missed a commit during a rebase.

* fix: lint

* feat: docs, rename resolvePaths, pin.getIndirectKeys now uses eachLimit

* chore: rebase a month of changes, resolve minor issues from that

fix: yarg arugment naming for pin cli commands

fix: convert file multihashes to a b58 string

fix: another way of checking for CID-ness

fix: lint

fix: toB58String handles non-buffers

fix: key-exchange core tests now shutdown daemon.

* chore: update big.js version

* revert: do not pin content added with a non-default hash algorithm

* revert: internalKey recording

* refactor: use lodash.flattenDeep

refactor: pinset.hasChild -> pinset.hasDescendent

fix: invoke someCb if we've seen the hash before

refactor: async patterns in dag._getRecursive

refactor: pinset.hasDescendant

refactor: pinset.storeItems async patterns

refactor: pinset.loadSet and pin.walkItem async patterns

docs: add link to go-ipfs' fanout bin implementation

refactor: async patterns of pin.load/flush

refactor: lint

refactor: privatize internal pin key storage

refactor: change encapsulation of ipfs.pin, fix resulting issues

fix: lint

fix: 'files add --pin=false' test was giving a false positive

refactor: use is-ipfs to check CID-ability of a string

refactor: remove last instance of 'once' in the pin code

* refactor: do not expose pinTypes

They're simple enough, documented elsewhere, and not used by any exposed functionality.

* fix: do not destructure node callback results

package.json

@@ -88,7 +88,7 @@
   },
   "dependencies": {
     "async": "^2.6.0",
-    "big.js": "^5.0.3",
+    "big.js": "^5.1.2",
     "binary-querystring": "~0.1.2",
     "bl": "^1.2.2",
     "boom": "^7.2.0",
@@ -98,13 +98,15 @@
     "debug": "^3.1.0",
     "file-type": "^7.7.1",
     "filesize": "^3.6.1",
+    "fnv1a": "^1.0.1",
     "fsm-event": "^2.1.0",
     "get-folder-size": "^1.0.1",
     "glob": "^7.1.2",
     "hapi": "^16.6.2",
     "hapi-set-header": "^1.0.2",
     "hoek": "^5.0.3",
     "human-to-milliseconds": "^1.0.0",
+    "interface-datastore": "^0.4.1",
     "ipfs-api": "^22.0.0",
     "ipfs-bitswap": "~0.20.0",
     "ipfs-block": "~0.7.1",
@@ -137,6 +139,7 @@
     "libp2p-websocket-star": "~0.8.0",
     "libp2p-websockets": "~0.12.0",
     "lodash.flatmap": "^4.5.0",
+    "lodash.flattendeep": "^4.4.0",
     "lodash.get": "^4.4.2",
     "lodash.set": "^4.3.2",
     "lodash.sortby": "^4.7.0",
@@ -218,7 +221,7 @@
     "Jade Meskill <[email protected]>",
     "Johannes Wikner <[email protected]>",
     "Jon Schlinkert <[email protected]>",
-    "Jonathan <[email protected]>",
+    "Jonathan Krone <[email protected]>",
     "João Antunes <[email protected]>",
     "João Santos <[email protected]>",
     "Kevin Wang <[email protected]>",

src/cli/commands/files/add.js

@@ -173,6 +173,11 @@ module.exports = {
       type: 'boolean',
       default: false,
       describe: 'Write no output'
+    },
+    pin: {
+      type: 'boolean',
+      default: true,
+      describe: 'Pin this object when adding'
     }
   },
 
@@ -188,7 +193,8 @@ module.exports = {
       rawLeaves: argv.rawLeaves,
       onlyHash: argv.onlyHash,
       hashAlg: argv.hash,
-      wrapWithDirectory: argv.wrapWithDirectory
+      wrapWithDirectory: argv.wrapWithDirectory,
+      pin: argv.pin
     }
 
     // Temporary restriction on raw-leaves:

src/cli/commands/pin.js

@@ -0,0 +1,15 @@
+'use strict'
+
+module.exports = {
+  command: 'pin',
+
+  description: 'Pin and unpin objects to local storage.',
+
+  builder (yargs) {
+    return yargs
+      .commandDir('pin')
+  },
+
+  handler (argv) {
+  }
+}

src/cli/commands/pin/add.js

@@ -0,0 +1,29 @@
+'use strict'
+
+const print = require('../../utils').print
+
+module.exports = {
+  command: 'add <ipfsPath...>',
+
+  describe: 'Pins object to local storage.',
+
+  builder: {
+    recursive: {
+      type: 'boolean',
+      alias: 'r',
+      default: true,
+      describe: 'Recursively pin the object linked to by the specified object(s).'
+    }
+  },
+
+  handler (argv) {
+    const recursive = argv.recursive
+    const type = recursive ? 'recursive' : 'direct'
+    argv.ipfs.pin.add(argv.ipfsPath, { recursive: recursive }, (err, results) => {
+      if (err) { throw err }
+      results.forEach((res) => {
+        print(`pinned ${res.hash} ${type}ly`)
+      })
+    })
+  }
+}

src/cli/commands/pin/ls.js

@@ -0,0 +1,43 @@
+'use strict'
+
+const print = require('../../utils').print
+
+module.exports = {
+  // bracket syntax with '...' tells yargs to optionally accept a list
+  command: 'ls [ipfsPath...]',
+
+  describe: 'List objects pinned to local storage.',
+
+  builder: {
+    type: {
+      type: 'string',
+      alias: 't',
+      default: 'all',
+      choices: ['direct', 'indirect', 'recursive', 'all'],
+      describe: 'The type of pinned keys to list.'
+    },
+    quiet: {
+      type: 'boolean',
+      alias: 'q',
+      default: false,
+      describe: 'Write just hashes of objects.'
+    }
+  },
+
+  handler: (argv) => {
+    const paths = argv.ipfsPath
+    const type = argv.type
+    const quiet = argv.quiet
+
+    argv.ipfs.pin.ls(paths, { type }, (err, results) => {
+      if (err) { throw err }
+      results.forEach((res) => {
+        let line = res.hash
+        if (!quiet) {
+          line += ` ${res.type}`
+        }
+        print(line)
+      })
+    })
+  }
+}

src/cli/commands/pin/rm.js

@@ -0,0 +1,28 @@
+'use strict'
+
+const print = require('../../utils').print
+
+module.exports = {
+  command: 'rm <ipfsPath...>',
+
+  describe: 'Removes the pinned object from local storage.',
+
+  builder: {
+    recursive: {
+      type: 'boolean',
+      alias: 'r',
+      default: true,
+      describe: 'Recursively unpin the objects linked to by the specified object(s).'
+    }
+  },
+
+  handler: (argv) => {
+    const recursive = argv.recursive
+    argv.ipfs.pin.rm(argv.ipfsPath, { recursive: recursive }, (err, results) => {
+      if (err) { throw err }
+      results.forEach((res) => {
+        print(`unpinned ${res.hash}`)
+      })
+    })
+  }
+}

src/core/boot.js

@@ -29,6 +29,7 @@ module.exports = (self) => {
 
     series([
       (cb) => self._repo.open(cb),
+      (cb) => self.pin._load(cb),
       (cb) => self.preStart(cb),
       (cb) => {
         self.log('initialized')

src/core/components/dag.js

@@ -3,6 +3,8 @@
 const promisify = require('promisify-es6')
 const CID = require('cids')
 const pull = require('pull-stream')
+const mapAsync = require('async/map')
+const flattenDeep = require('lodash.flattendeep')
 
 module.exports = function dag (self) {
   return {
@@ -33,6 +35,12 @@ module.exports = function dag (self) {
         } else {
           path = '/'
         }
+      } else if (Buffer.isBuffer(cid)) {
+        try {
+          cid = new CID(cid)
+        } catch (err) {
+          return callback(err)
+        }
       }
 
       self._ipld.get(cid, path, options, callback)
@@ -73,6 +81,23 @@ module.exports = function dag (self) {
         self._ipld.treeStream(cid, path, options),
         pull.collect(callback)
       )
+    }),
+
+    // TODO - use IPLD selectors once they are implemented
+    _getRecursive: promisify((multihash, callback) => {
+      // gets flat array of all DAGNodes in tree given by multihash
+
+      self.dag.get(new CID(multihash), (err, res) => {
+        if (err) { return callback(err) }
+
+        mapAsync(res.value.links, (link, cb) => {
+          self.dag._getRecursive(link.multihash, cb)
+        }, (err, nodes) => {
+          // console.log('nodes:', nodes)
+          if (err) return callback(err)
+          callback(null, flattenDeep([res.value, nodes]))
+        })
+      })
     })
   }
 }

src/core/components/files.js

@@ -32,7 +32,9 @@ function prepareFile (self, opts, file, callback) {
   }
 
   waterfall([
-    (cb) => opts.onlyHash ? cb(null, file) : self.object.get(file.multihash, opts, cb),
+    (cb) => opts.onlyHash
+      ? cb(null, file)
+      : self.object.get(file.multihash, opts, cb),
     (node, cb) => {
       const b58Hash = cid.toBaseEncodedString()
 
@@ -87,6 +89,19 @@ function normalizeContent (opts, content) {
   })
 }
 
+function pinFile (self, opts, file, cb) {
+  // Pin a file if it is the root dir of a recursive add or the single file
+  // of a direct add.
+  const pin = 'pin' in opts ? opts.pin : true
+  const isRootDir = !file.path.includes('/')
+  const shouldPin = pin && isRootDir && !opts.onlyHash && !opts.hashAlg
+  if (shouldPin) {
+    return self.pin.add(file.hash, err => cb(err, file))
+  } else {
+    cb(null, file)
+  }
+}
+
 class AddHelper extends Duplex {
   constructor (pullStream, push, options) {
     super(Object.assign({ objectMode: true }, options))
@@ -130,7 +145,8 @@ module.exports = function files (self) {
     }
 
     let total = 0
-    let prog = opts.progress || (() => {})
+
+    const prog = opts.progress || noop
     const progress = (bytes) => {
       total += bytes
       prog(total)
@@ -141,7 +157,8 @@ module.exports = function files (self) {
       pull.map(normalizeContent.bind(null, opts)),
       pull.flatten(),
       importer(self._ipld, opts),
-      pull.asyncMap(prepareFile.bind(null, self, opts))
+      pull.asyncMap(prepareFile.bind(null, self, opts)),
+      pull.asyncMap(pinFile.bind(null, self, opts))
     )
   }
 

src/core/components/index.js

@@ -18,6 +18,7 @@ exports.swarm = require('./swarm')
 exports.ping = require('./ping')
 exports.pingPullStream = require('./ping-pull-stream')
 exports.pingReadableStream = require('./ping-readable-stream')
+exports.pin = require('./pin')
 exports.files = require('./files')
 exports.bitswap = require('./bitswap')
 exports.pubsub = require('./pubsub')

src/core/components/init-assets.js

@@ -1,9 +1,7 @@
 'use strict'
 
 const path = require('path')
-const fs = require('fs')
 const glob = require('glob')
-const importer = require('ipfs-unixfs-engine').importer
 const pull = require('pull-stream')
 const file = require('pull-file')
 const CID = require('cids')
@@ -15,23 +13,20 @@ module.exports = function addDefaultAssets (self, log, callback) {
 
   pull(
     pull.values([initDocsPath]),
-    pull.asyncMap((val, cb) => glob(path.join(val, '/**/*'), cb)),
+    pull.asyncMap((val, cb) =>
+      glob(path.join(val, '/**/*'), { nodir: true }, cb)
+    ),
     pull.flatten(),
-    pull.map((element) => {
+    pull.map(element => {
       const addPath = element.substring(index + 1)
-
-      if (fs.statSync(element).isDirectory()) { return }
-
       return { path: addPath, content: file(element) }
     }),
-    // Filter out directories, which are undefined from above
-    pull.filter(Boolean),
-    importer(self._ipld),
-    pull.through((el) => {
-      if (el.path === 'init-docs') {
-        const cid = new CID(el.multihash)
+    self.files.addPullStream(),
+    pull.through(file => {
+      if (file.path === 'init-docs') {
+        const cid = new CID(file.hash)
         log('to get started, enter:\n')
-        log(`\t jsipfs files cat /ipfs/${cid.toBaseEncodedString()}/readme\n`)
+        log(`\tjsipfs files cat /ipfs/${cid.toBaseEncodedString()}/readme\n`)
       }
     }),
     pull.collect((err) => {

src/core/components/init.js

@@ -85,16 +85,18 @@ module.exports = function init (self) {
           return cb(null, true)
         }
 
-        self.log('adding assets')
         const tasks = [
           // add empty unixfs dir object (go-ipfs assumes this exists)
           (cb) => self.object.new('unixfs-dir', cb)
         ]
 
         if (typeof addDefaultAssets === 'function') {
+          // addDefaultAssets is undefined on browsers.
+          // See package.json browser config
           tasks.push((cb) => addDefaultAssets(self, opts.log, cb))
         }
 
+        self.log('adding assets')
         parallel(tasks, (err) => {
           if (err) {
             cb(err)