Background: Re-architect IPFS & libp2p Documentation

[Mr0grog]

Let’s develop a roadmap towards a new, improved docs site in this repo. Here’s my take to get us started:

---

Overall, I think there are a few major problems users and contributors are running into that the new docs need to solve. First here’s a breakdown of the typical introductor experience:

1. For most all people, their journey with IPFS starts with blog post, Hacker News, or Reddit.
2. The only official resource we have that comprehensively describes IPFS is the white paper and that’s where most people wind up next when they decide to try and actually learn about or use IPFS.
   • Many folks skip or quickly skim this — its academic style can be intimidating and it assumes familiarity with a number of computer science or math concepts like graphs (as models of objects and relationships, rather than visualizations of 2/3/4-dimensional data)
   • More enterprising folks or those with a more expertise read it, but it’s very broad and very old. It doesn’t connect especially well with the concrete reality of IPFS today.
   • In both cases, people wind up making a number of assumptions that are inaccurate in different ways.
3. Go through the getting started tutorial
   • It doesn’t attempt to cover what is conceptually going on much at all, so any poor assumptions people have made from step 2 generally stay in place.
   • It is focused on the CLI, which is off-putting to many folks with less expertise and leaves a lot of developers who are immediately wondering how to interface with it programmatically in the cold.
   • It doesn’t leave you with any suggestions for next steps or what to do.
4. ???? At this point, we see a wide variety of things:
   • Giving up because it’s not clear
   • Try storing or sharing some files with a friend, which might work, but often leads to oddball questions about the basics on discuss.ipfs.io, IRC, or GitHub, because of conceptual assumptions they’ve made in steps 2 and 3. Some examples are questions about “uploading” (people with less expert programming background) or questions about why an IPFS hash doesn’t match a simple SHA-256 (or even a multihash) of the file they added to IPFS (more expert).
   • Spend a lot of time digging through GitHub trying to figure out how best to program against it as a library (especially if their primary language is not JS or Go). There is little about using IPFS programmatically on the docs site. We have a lot of docs that people never find because:
     • They’re buried in a source code repo.
     • They’re not even in the repo they’d expect. The docs in ipfs/interface-ipfs-core are dramatically better than the Rest API Docs, for example. One person said knowing those were there would have made things way easier, but they’d never seen them because they are only well linked from js-ipfs and this person wasn’t writing JavaScript (or Go).
     • There might be useful generated API docs, but we often don’t link them. (e.g. go-ipfs)

See lots more of this stuff over at #52 or https://gateway.ipfs.io/ipns/ipfs-docs-research.robbrackett.com/html/General-Notes.html .

What are the big learning/documentation problems here?

• No clear introduction to the overall idea of exactly how IPFS works and what it’s doing
• Lots of new concepts (whether you are knowledgeable about things like graphs or not) that are just very different from the web technologies people know today.
• Docs are inconsistently located and spread across a number of repos people have to hunt through.
• Clear, standard API docs are not always available.
• Hunting through GitHub is hard. (Which repos have docs? Where in the repo are they? Which projects are important and how do they relate to the others? Which repos and docs are up-to-date?)

What do we need to do?

High priority:

1. Write a new introductory guide that focuses on what IPFS is doing conceptually, explaining how addresses and lookups and the DAG work. It should use the CLI as an concrete example of these things rather than the other way around, as the current getting started guide does. (Write new “Introduction to IPFS” guide #60)

2. Build a concepts dictionary that provides a brief and clear overview of major concepts in IPFS. ([NEW CONTENT] [BOUNTY] Glossary #56)

   • Some examples:
     • https://qri.io/docs/datasets/
     • Items under the “concepts” heading in Stellar’s docs, e.g: https://www.stellar.org/developers/guides/concepts/accounts.html (disclosure: I worked on this)
     • Slack sort of does this with the “app features” category in the docs sidebar: https://api.slack.com (they’re somewhere between conceptual and tutorial, and may or may not be more detailed than we want to need to be)

3. Link to (not necessarily embed) our best existing reference docs and major repos (Gather & organize links to existing docs and examples #59)

4. link to (not necessarily embed) our best existing example projects (e.g. https://github.com/ipfs/js-ipfs/tree/master/examples) (Gather & organize links to existing docs and examples #59)

5. Clean up the problems in GitHub organization that make things hard to find: GitHub Hygiene: Develop a standard way to indicate a repo/project is deprecated #54, GitHub Hygiene: Make sure every repo has a description and readme #55, GitHub Hygiene: Develop a standard way to indicate non-code repos #57, and some parts of Provide clear, reliable, up-to-date roadmaps and information about statuses of features #53)

6. Design a site layout that can accommodate the above and launch it :) (Refactor/expand docs on ipfs.io  #36?)

Note that 3 & 4 are about linking and not about embedding docs. I think we should focus on getting links to what we’ve already got (we have a lot of stuff) first. Simply having the docs site serve as an index to all the things that would be useful but people aren’t finding is incredibly helpful and relatively quick to do.

This is way more minimal than the expansive set of content currently described in #36, but that was half a year ago, so I it might make sense to focus on something smaller for now.

Later:

(To be clear, I think things below like improving existing API docs is much less important than #53 — giving people a clear picture of the status and direction of various APIs and concepts. Developers are getting by, though with plenty of trouble, with what they’ve got. But nothing solves the “where is this going and how should I think about using/not using it?” question for them. Addressing that in some form is critical.)

Not necessarily in any order:

• Add some brief narrative/introductory guides on using go-ipfs, js-ipfs, *-ipfs-api as libraries.

• Document how all the various sub-libraries relate to each other and to the top-level libraries (i.e. go-ipfs and js-ipfs). (Covering these relationships is one of the major missing pieces in a lot of the READMEs.)

• Improve the REST API docs with more detailed information about arguments (many are missing) and example input/output. More than one person was blown away by how useful the docs in interface-ipfs-core were compared to the REST API or Commands docs, which they had been depending on.

• Start to generate/embed API docs from all the code projects.

• Create guides/tutorials for common use cases

  • Archiving big datasets
  • Hosting your web site
  • Dynamic data & (web) apps
  • Securing content
  • Storing/referencing data in non-file data structures (e.g. IPLD)

[pmthomps]

Reading through Rob's comprehensive research on the current docs situation and the roadmap to address, there's a lot to tackle. In the prioritization of it, if we were to group doc consumers by their goals, what would that look like?

Still being new to IPFS myself but having been reading through the posts on discuss.ipfs.io to tag them, here's a strawman to start a conversation:

Group*	Description / Current level of understanding	Goals	Priorities (in document roadmap terms)
Absolute newbie	Saw IPFS referenced in passing and landed on IPFS homepage to learn more	* Understand what IPFS is from high-level * Find a use case relevant to their world (and hopefully solves a problem they have) * Get started with a relevant use case * Understand the future of IPFS (E.g., is this going to be around? What projects are using it? What does the future hold?) * Get specific questions answered	* Introductory Guide * Use Cases + tutorials for each use case * List of active projects that use IPFS (e.g., d.tube) * Link to discuss / irc
Hacker	Has a base level understanding of IPFS capabilities and has an idea for implementation with IPFS	* Determine whether to pursue IPFS in a proof-of-concept * Understand IPFS concepts Find examples / documentation to help with an implementation * Understand known limitations in current version * Get help troubleshooting an issue	* Concepts Dictionary Common use cases with sample code * Feature Status chart + Known  Issues * What are the supported features/concepts/approaches, known issues, upcoming features (w- approx timeline), and no plan to support list * Example projects
Developer	Currently developing a solution with IPFS and needs reference/guidance	* Get implementation/API questions answered * Get help troubleshooting blocking issues	* Links to major repos and best existing reference docs * Guides on using go-ipfs, js-ipfs, *-ipfs-api as libraries (covering how they interrelate) * Improvement to REST API Docs * API docs (gen from code projects) * Link to discuss / irc

From an IPFS project perspective, which group / goals are the most important to address first?

[Mr0grog]

Oh, thanks for also classifying these in an expertise/familiarity dimension! (As opposed to the two angles on use cases here)

which group / goals are the most important to address first?

I hadn’t really thought to classify this way mainly because the feedback I’ve gotten is that they are all similarly important (maaaaaaaybe your “developer” category a little less, because they have figured out a system for digging through source code and issues that sorta-kinda-at-least-a-little works well for them).

A few things I think are particularly relevant to this categorization:

• I feel like “hacker” and “developer” here are the same person, just at different points in time/levels of experience with IPFS (an extremely senior developer would still be in the “hacker” category here if they are just trying to evaluate IPFS). I’m not sure on better names to clarify that, but also not sure how important that is.

• “Feature Status chart + Known Issues” is actually also a critical issue for developers who are very familiar and actively building things on top of IPFS! It’s not just the “hackers” who just have a base-level familiarity, but I think you’re right they need this, too—they just don’t actually know it.

  (The general story I heard repeated from almost everyone is that things are presented or couched in terms where they look pretty good; only after actually starting to build something and getting pretty far in do they realize things are less stable than they appeared.)

• For “hackers,” they also need to know what tools are available/what tools they should be using — e.g. the confusion between the REST API and the “gateway API,” how js-ipfs-api relates to js-ipfs and when/why to use which, what <language>-ipfs-api libraries are available (and how usable they are). Basically “what’s the right way to program against IPFS?” is a question with a surprisingly unclear/hard to find answer.

[fsdiogo]

I feel like “hacker” and “developer” here are the same person, just at different points in time/levels of experience with IPFS (an extremely senior developer would still be in the “hacker” category here if they are just trying to evaluate IPFS)

Couldn't agree more. Everyone that tinkers with the IPFS code starts by being a hacker and then gets to the developer group after a while, but I think their priorities are very much the same.

[jessicaschilling]

Note: Many (hopefully most) of the important insights surfaced in this issue are being addressed in a variety of different venues/issues as part of our overall Q3 2019 docs effort, as well as work that follows up from that beyond Q3 '19. Leaving this issue open, but primarily as context and background as we continue to execute upon these key findings. 😄

[jessicaschilling]

The core concepts in this issue have been incorporated into a number of Q3 and Q4 '19 issues/epics, so closing this one.