Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[key packages] Pilot Packages #142

Open
Eomm opened this issue Jan 29, 2019 · 35 comments
Open

[key packages] Pilot Packages #142

Eomm opened this issue Jan 29, 2019 · 35 comments
Labels
pilot Pilot packages

Comments

@Eomm
Copy link
Member

Eomm commented Jan 29, 2019

Split 1/3 of #105

We can use to start collaborating with the authors to learn and iterate with.

Edit more info:
to reach the WG goals we need to start to help some pilot packages.
These modules will be helped by this WG adding support tools, documentation, fixes and all things to archive our goals and carry benefit to the package itself.
This WG will improve his guidelines, will learn different requirements and will be able to find patterns and solutions, test all the process and tools that are gonna to be built in order to evangelize the community and let the maintenance of an OSS package will be easier for maintainers.
What this WG ask to the pilot packages is:


Express and all its direct and transient dependencies like pillarjs and jshttp are the most mentioned package to start with.
@wesleytodd has discussed a bit of this possibility on the last Express TC meeting so I think the Express team would be on board to participating.

What to discuss here

  • We need to agree or not to start with Express as a pilot package. If "no" what else?
  • To define if we can and should add more "friendly" pilot packages

After that we could start to get "Directions of Help": the way package maintainers wish to receive help and define the first tasks.

Ex:
Express needs help with repo cleanup, automation, documentation and, user support.

@mhdawson
Copy link
Member

Starting with Express sounds good to me, given that they are a key part of the ecosystem and it sounds like they are open to working with us.

@wesleytodd
Copy link
Member

While clearly I am a fan of working with Express, I think we need at least a few others to make sure we have a variety of input. Anyone have other suggestions?

@mcollina
Copy link
Member

mcollina commented Feb 1, 2019

I think https://github.com/mqttjs/MQTT.js and dependencies are another good place to start.

@mhdawson
Copy link
Member

mhdawson commented Feb 1, 2019

I agree we need to have several and MQTT sounds like a good one to add to the list.

@Eomm
Copy link
Member Author

Eomm commented Feb 1, 2019

I don't know other friendly packages that could collaborate with us in this pilot phase.

Appling manually the basic criteria we talk about, I think that request could be a candidate: it is widely used and have a lot of issue and PR: it seems they need help to break all that work!

I looked here, there is a nice bubble schema that show the libs per usage, I took the first one with many issue&PR opened. Nothing personal 😁

@wesleytodd
Copy link
Member

wesleytodd commented Feb 1, 2019

I agree request seems like a good target. @reconbot & @mikeal Does this program seem like something you would be willing to participate in? I know you two are involved in request, but if there is anyone else we should ping please feel free!

@mcollina Are you the primary maintainer for MQTT? If not, are there others we should reach out to to see if they are willing to participate?

@nschonni
Copy link
Member

nschonni commented Feb 1, 2019

Maybe some of the plumbing packages that are depended on for the native packages like node-pre-gyp
I'm hoping to move node-sass to that in the next major version, which would bump it up close to request numbers (which is also a dependency for both)

@mcollina
Copy link
Member

mcollina commented Feb 1, 2019

@wesleytodd I'm the only active one with publish access. I can add more if it is needed, but I'm probably the only one that knows how all the pieces fits together.

@reconbot
Copy link

reconbot commented Feb 2, 2019

I've been doing some bug triage for request, and have landed some patches but haven't yet done a release. The future of that package is a discussion. There is a desire to leave it as is and encourage the use of more modern packages even, however keeping it supported is obviously desirable. I've read the linked issue and the blog post, but I'm not sure what the work would be to be part of this project.

@wesleytodd
Copy link
Member

I think that is a great stage for this WG to look at. I think we should be taking into consideration the entire life cycle of maintenance, including EOL like you say request is (although I feel like this might come as a shock so some users). That is to say, I don't believe this should disqualify it from consideration here.

@wesleytodd
Copy link
Member

Also, @reconbot, maybe you will have some input on #139. Does that proposal seem like something which would help signal to users the state the maintainers think the project is in?

@Eomm Eomm added the package-maintenance-agenda Agenda items for package-maintenance team label Feb 3, 2019
@chrkaatz
Copy link
Contributor

chrkaatz commented Feb 3, 2019

One other candidate might be nedb as the creator of it asked for help himself but I know that the package is not used very much anymore these days.

@mikeal
Copy link

mikeal commented Feb 4, 2019

It’s not clear to me from the thread here and the links exactly what this pilot program means and what the goals are.

When engaging projects it would be great to have a concise write up of:

  • The problems this program is designed to address.
  • What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.
  • What is being offered by this group to these projects.

@wesleytodd
Copy link
Member

wesleytodd commented Feb 4, 2019

Great points. @Eomm maybe we could edit the OP and add some of this?

The problems this program is designed to address.

The goals of this WG can be found here.

What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.

A pilot project would be the first group we try to help. The requirement would be a willingness to try some ideas and provide feedback on how they work.

What is being offered by this group to these projects.

This is in the process of being defined. And the hope is that the pilot projects help us refine and decide on what specific things would be best for us to provide.

@Eomm
Copy link
Member Author

Eomm commented Feb 4, 2019

@wesleytodd edited 👍
I hope to have clarified the benefit and the effort in play.

@Eomm
Copy link
Member Author

Eomm commented Feb 10, 2019

Hi, I would like to recap the topic to speed up the meeting tomorrow:

Candidate Approved by candidate Main Contact
express yes @wesleytodd
mqtt yes @mcollina
request need information @mikeal
plumbing packages that are depended on for the native packages - -
node-sass - @nschonni
nedb yes @chrkaatz

I hope I have correctly summarized 😁

@Eomm
Copy link
Member Author

Eomm commented Feb 11, 2019

It’s not clear to me from the thread here and the links exactly what this pilot program means and what the goals are.

When engaging projects it would be great to have a concise write up of:

  • The problems this program is designed to address.
  • What is being asked for from “pilot” projects and what may be expected from projects in the future beyond the pilot.
  • What is being offered by this group to these projects.

Hi @mikeal, to reach the WG goals we need to start to help some pilot packages.
These modules will be helped by this WG adding support tools, documentation, fixes and all things to archive our goals and carry benefit to the package itself.
This WG will improve his guidelines, will learn different requirements and will be able to find patterns and solutions, test all the process and tools that are gonna to be built in order to evangelize the community and let the maintenance of an OSS package will be easier for maintainers.
What this WG ask to the pilot packages is:

I hope to be exhaustive, let us know what do you think about it
Thank you

@JamesMGreene
Copy link
Contributor

I would also be interested in being a maintainer of NeDB. I try to respond to most of the newly opened issues in the existing repository, and have also forked the project as NestDB to implement some requested new features, bug fixes, and performance improvements that its sole maintainer was not keen on merging or able to give time to (a problem with maintainership that so many of us know all too well).

I've done my best to track the relationships of my changes to the original NeDB issues that most of them originated from in the hopes that, someday, they might make it back into the NeDB core codebase.

There have also been other forks of NeDB, as nedb2, nedb3, nedbhq/nedb-core, and even the now unrecognizable TeDB (TypeScript-based), so the popularity of NeDB -- and its consumers seeking for actual maintenance -- definitely has no doubt in my mind.

@mhdawson
Copy link
Member

@wesleytodd We touched on this briefly and we wondered if we are ready to take the next steps with either express or mqtt as we continue to build out the pilot list?

@mhdawson mhdawson removed the package-maintenance-agenda Agenda items for package-maintenance team label Feb 25, 2019
@wesleytodd
Copy link
Member

Hey, yes any time. Do we have a concrete list of the first steps? I seem to remember an issue regarding this, but I have so many unread messages at the moment (👶 time). The start was a survey right? If so I can get that posted up in the express repo so we can fill it in.

@wesleytodd
Copy link
Member

Ok, I posted this on the express discussion repo. I will hopefully get other contributor feedback and start working on those answers soon.

expressjs/discussions#77

@Eomm
Copy link
Member Author

Eomm commented Mar 4, 2019

I've done my best to track the relationships of my changes to the original NeDB issues that most of them originated from in the hopes that, someday, they might make it back into the NeDB core codebase.

@JamesMGreene could you ping the maintainer of NeDB to join the discussion?
Because I think we can't help that package if the maintainer can't collaborate for whatever reason.

@Eomm
Copy link
Member Author

Eomm commented Apr 10, 2019

In last meeting we talked to find more packages

We could start digging from this list:

The top 20, to save you a click:
debug
kind-of
supports-color
readable-stream
source-map
yargs
camelcase
minimist
strip-ansi
chalk
commander
@types/node
punycode
ansi-regex
ms
glob
define-property
semver
async
string_decoder

https://twitter.com/seldo/status/1115725616433602560?s=19

@mcollina
Copy link
Member

https://docs.google.com/spreadsheets/d/1lZDNYsLntwD2q9XaTw-XLuG0VpHH6cOV-Uoa7Y1aTSM/edit#gid=1745448509 contains a list of the top 1000 most downloaded modules.
It would be good to do some sort of data analysis of the dependants and dependencies of these, as I guess there are a lot of overlaps and macro-trends.

@dominykas
Copy link
Member

dominykas commented Apr 13, 2019

Here's some percentiles on last update age of top 1000 packages:

percentile last commit last publish last update to .travis.yml
5% 1 days old 7 days old 10 days old
10% 4 days old 11 days old 34 days old
15% 5 days old 18 days old 56 days old
20% 5 days old 37 days old 92 days old
25% 10 days old 58 days old 126 days old
30% 19 days old 88 days old 129 days old
35% 33 days old 131 days old 129 days old
40% 57 days old 157 days old 160 days old
45% 81 days old 200 days old 196 days old
50% 119 days old 246 days old 259 days old
55% 150 days old 300 days old 324 days old
60% 194 days old 369 days old 387 days old
65% 256 days old 438 days old 501 days old
70% 328 days old 559 days old 639 days old
75% 418 days old 669 days old 723 days old
80% 563 days old 777 days old 891 days old
85% 706 days old 944 days old 1036 days old
90% 887 days old 1098 days old 1214 days old
95% 1194 days old 1372 days old 1501 days old
100% 2201 days old 2792 days old 2620 days old

This means that roughly half of them have not been published since last node LTS... Now, that in itself does not mean they weren't tested in that LTS (even it travis.yml wasn't update, although that's a good indicator), but it gives perspective on how much of just the top 1000 is either "abandoned" or "done".

What kind of dependents / dependencies data did you want to see @mcollina?

@dominykas
Copy link
Member

dominykas commented Apr 13, 2019

And here's some stats from versions extracted from .travis.yml (didn't bother cleaning those up, just yet):

node version count in travis.yml
version count
1 3
2 3
3 3
4 269
5 128
6 507
7 112
8 459
9 85
10 381
11 53
0.1 6
0.10 257
0.10.12 1
0.11 29
0.12 244
0.12.0 1
0.4 1
0.6 31
0.8 98
0.8.6 1
0.9 2
1.7.1 1
1.8 37
10.0 2
10.11 2
10.12 5
10.13 2
10.14 3
10.15 25
10.2 1
10.4 1
10.6 1
10.7 2
10.8 1
11.0 2
11.10 8
11.12 1
11.13 1
11.14 1
11.2 1
11.4 2
11.6 5
11.7 3
11.8 1
11.9 2
2.5 37
3.3 37
4.0 6
4.0.0 3
4.1 4
4.2 4
4.3 1
4.4 3
4.4.0 1
4.5 2
4.7 1
4.8 7
4.9 48
5.0 2
5.0.0 1
5.1 1
5.10 3
5.11 1
5.12 55
5.2 1
5.3 1
5.4 1
5.5 1
5.6 2
5.7 2
5.8 1
5.9 1
6.0 4
6.0.0 1
6.1 1
6.10 1
6.11 1
6.12 2
6.13 2
6.14 18
6.15 4
6.16 21
6.17 5
6.2 1
6.3 1
6.4 1
6.5 2
6.6 1
6.9 1
7.10 53
7.7 1
8.0 1
8.11 9
8.12 8
8.13 2
8.14 3
8.15 26
8.6 1
8.9 5
9.11 46
9.3 1
9.5 2
iojs 50
iojs-1 1
iojs-2 1
iojs-3 1
iojs-v1.0 1
iojs-v1.1 1
iojs-v1.2 1
iojs-v1.3 1
iojs-v1.4 1
iojs-v1.5 1
iojs-v1.6 1
iojs-v1.7 1
iojs-v1.8 17
iojs-v2.0 1
iojs-v2.1 1
iojs-v2.2 1
iojs-v2.3 1
iojs-v2.4 1
iojs-v2.5 17
iojs-v3.0 1
iojs-v3.1 1
iojs-v3.2 1
iojs-v3.3 17
lts/* 18
node 196
stable 52
v0.12 1
v10.* 1
v4 3
v5 2
v6 2
v6.* 1
v8.* 1

@mcollina
Copy link
Member

Scavenging that data is very interesting.
I’d like to see if there are some dependency relationship. For example, we know that express uses debug. This info is helpful because we can take some macro-family.

Another data that would be useful are the number of issues and prs open on those repo to gauge activity.

@ljharb
Copy link
Member

ljharb commented Apr 14, 2019

tbh what I’d really like to see is top packages on npm with download counts of dependents and devDependents subtracted out

@dominykas
Copy link
Member

dominykas commented Apr 20, 2019

This info is helpful because we can take some macro-family.

Sorry, not sure I'm following. You'd like to have the top1k somehow clustered into a group of deps that belong to a single dependent or smth? Or to put it differently - try to find a group that would likely get downloaded together? I can try to look into the dependency links in the top1k, but we don't have an easy, publically available method to go beyond that...

Another data that would be useful are the number of issues and prs open on those repo to gauge activity.

So getting these numbers is easy, but gaining insight is hard... Do you have any specific ideas or questions in mind?

A number of open issues on its own does not say anything (some maintainers close everything, others let issues linger, esp. support ones), the issue/pr rate over time is also meaningless - low activity could just be an indicator of done-ness and stability and I'm not sure how that is helpful?


What problems are we looking to solve with this? What problems can be discovered? I'm usually a fan of looking for data that can help make decisions, e.g. the numbers I posted can be used to determine which packages need some touching up to at least have CI run with newer node versions. Are there any other specific things we could be looking for?

@Eomm
Copy link
Member Author

Eomm commented Apr 23, 2019

Are there any other specific things we could be looking for?

I think that this question is more related to this issue #143 (comment) and here (pilot packages) I think we are looking for collaboration mainly from the maintainers of the modules and download numbers.

Adding issues/PR numbers will be useful for sure, but we could proceed step by step building the tool that we will use to list the modules to find them out so.. who are those packages in the >=50% from last travis update?

@dominykas
Copy link
Member

who are those packages in the >=50% from last travis update?

I'll open a separate issue with the list shortly and we can take it from there.

@Eomm
Copy link
Member Author

Eomm commented Aug 31, 2019

I would close the issue since the pilot packages have been selected

I don't know if we could ask to join in the #221 #239

@wesleytodd
Copy link
Member

Do we feel that the pilot packages issues have been addressed? I know we have a lot of things in the works, but it seems to me that unless we think we are ready to move onto the next phase we should keep this open.

IMO, the goal of the pilots are to get some actual working tests for the ideas we have. From express, we are only just now starting to implement the stuff we have talked about, and have not really even gotten to give feedback on their success/failure. I dont know how the MQTT progress is on this, but I haven't seen anything concrete here yet.

Thoughts?

@Eomm
Copy link
Member Author

Eomm commented Sep 2, 2019

It is true, I thought the target for this issue was to find the pilot packages.
We can keep it open to collect the results also for sure 👍

@mhdawson
Copy link
Member

mhdawson commented Sep 5, 2019

+1 to keeping it open. We are starting to make some progress but I'd agree there is more work before we can consider this part complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pilot Pilot packages
Projects
None yet
Development

No branches or pull requests