Improve overall state of js-ipfs tests

[hugomrdias]

Goals:

1. Improve developer productivity
2. Reduce the complexity of adding new tests for core and community devs
3. Increase overall coverage and quality of the tests

Tasks:

• P0 ipfsd-ctl needs to be cleaned up Fix non disposable daemon init/start and attach to running daemons js-ipfsd-ctl#308
• P0 ipfsd-ctl Complete Rewrite more info here feat: new ctl js-ipfsd-ctl#395 (comment)
  • feat: add ability to use existing config during init kubo#6489
  • feat: allow daemon to init and start in a single cmd #2428
  • fix: pass config to init js-ipfsd-ctl#303
  • feat: init start with one cmd js-ipfsd-ctl#369
  • feat: new ctl js-ipfsd-ctl#395 (complete rewrite)
  • ipfsd-ctl 1.0.0 released
    • Highlights:
      • All issues and prs closed
      • Much cleaner codebase to work on
• P0 simpler, faster and async/await for setup for tests using the new ctl
  • feat: add support new ctl ipfs-inactive/interface-js-ipfs-core#541
  • fix: use new ipfsd-ctl setup ipfs-inactive/js-ipfs-http-client#1127
  • feat: support new ctl api #2529
    • Many tests were skipped in the browser and now almost all are running and passing
  • interop
  • ipfs-desktop
• P0 All interface-core tests in js-ipfs run and pass in the browser
• P0 improve errors handling [WIP] feat: add errors with codes and utils #1746
  • refactor: error objects and handling js-ipfs-repo#203 (comment)
• P0 js-ipfs tests need some love
  • CLI tests needs to test cli logic and not core logic
    • cli needs to allow dependency injection to reduce mocks and spy (plus weird hacks to the node js require system)
    • internal yargs handling of promises needs to be rewritten
    • improve CLI reliability to avoid zombie processes fix: move mfs cmds and safer exit #1981
• P1 Deduplication of tests from core tests and interface-core tests
  • Core API tests are all in the interface-core repo dedupe tests ipfs-inactive/interface-js-ipfs-core#573
  • js-ipfs repo only has tests for non Core API logic dedupe tests #2669
  • js-ipfs-http-client repo only has tests for non Core API logic dedupe tests ipfs-inactive/js-ipfs-http-client#1200
• More generic PRs
  • Reduce bundle size in bitswap fix: reduce size js-ipfs-bitswap#203
  • Release ipfs-utils 0.5.0 and backport unixfs 1.5 changes to 0.4.1

Issues discovered

• interface-core pubsub subscribe fails in firefox interface-core pubsub subscribe fails in firefox #2662
  • fix: fix pubsub subscribe with multiple node in firefox ipfs-inactive/interface-js-ipfs-core#571
• Make sure we have interop with differents types of secondary nodes https://github.com/ipfs/interface-js-ipfs-core/issues/570
• Support relay and hop as "config" options like Go Support relay and hop as "config" options like Go #2667

---

future work

• P1 Dependencies between tests needs to be removed
  • js-ipfs cli
• P1 Environmental skip of tests need to be self applied by each tests (relates to ipfs-utils)
  • ipfsd-ctl
  • interface-core
  • js-ipfs
  • http-client
• P2 aegir test -t node needs to be reworked
• P3 HTTP API tests need to have coverage instrumentation (relates to ipfsd-ctl)
• P4 Browser tests need to build coverage output
• P4 Test isolation needs to be analysed (daemon spawns between tests)
• P4 https://github.com/ipfs/interop
• P4 Travis CI and Windows is still a problem at least for js-ipfs repo

[daviddias]

@hugomrdias as discussed during the OKRs review. It would be great if endeavors like this and others to improve developer productivity such as #1670 had some metrics that capture the improvement. Things suggested to measure:

• Bundle Size reduction overtime
• Time to PR to get merged/closed
• Time to close an Issue
• Time that tests take to run
• Number of new contributions from outside core team

[autonome]

Does the new devstats stuff get us this information across groups of repos?

…

On Tue, Oct 1, 2019 at 7:14 AM David Dias ***@***.***> wrote: @hugomrdias <https://github.com/hugomrdias> as discussed during the OKRs review. It would be great if endeavors like this and others to improve developer productivity such as #1670 <#1670> had some metrics that capture the improvement. Things suggested to measure: - Bundle Size reduction overtime - Time to PR to get merged/closed - Time to close an Issue - Time that tests take to run - Number of new contributions from outside core team — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2276?email_source=notifications&email_token=AAAMHN4QTHS6UZ3CKNMHMX3QMMWIFA5CNFSM4IEZPEDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAA42AQ#issuecomment-536988930>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAMHN2YXYHO55U2AF3YH5DQMMWIFANCNFSM4IEZPEDA> .

[momack2]

Given the need to start executing on 2020 work next quarter, please plan to finish this testing spike in Q4. That means prioritize your time to ensure we hit the highest priority testing improvements and think about ways to leverage the remaining time on this thread. Something to think about is our iteration velocity here - how do we shorten the cycle of making and shipping testing improvements so we can externalize the value of the work being done quickly?

[hugomrdias]

Going forward after shipping the new ctl PRs, small units of improvement should be much easier to make.