Clarify rules regarding the review committee

[ShriyaPalsamudram]

The current rules are not clear about review chair selection process and the review committee composition. So this PR aims to clarify those reviews.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[TheKanter]

Hi Shriya - I think it's good to clarify the rules here, but a few comments:

1. We need to change the review committee to reference the head of MLPerf and probably rethink composition a bit
2. I'd like to work with other impacted WGs to discuss (and possibly figure out a good system that gives folks a voice, but doesn't prevent changes)
3. I haven't read it in any further detail, so more to come :)

[ShriyaPalsamudram]

Hey David, Thanks for the feedback, I agree that we need to think about this a bit more. But there’s a lot of value in writing down rules about how the review committee is setup, and who it comprises of so it helps all WGs in the future. 1. I can update the PR to add in “head of MLPerf” in a structured way. 2. We spoke about it a bit in the Training WG meeting today, and got some good feedback. Please do share with other WGs and we can discuss modifications/suggestions on the PR. 😊 3. Looking forward to (3)

[nv-ananjappa]

@mrmhodak @mrasquinha-g Can we review this rules change for v4.1 submission in next Tue WGM?

[arjunsuresh]

Can we add MLCommons staff to the review meeting? We do have a requirement to attend the review meeting for reproducibility purposes.

[psyhtest]

The relevant working group chair or power working group chair may be a valid candidate as well if approved by unanimous consent.

Unanimous consent for approval sounds too strict. A simple majority should suffice, IMHO.

[mrasquinha-g]

Can we please add this to the next benchmark chairs working group meeting agenda for discussion.

[swasson488]

I have added this item to the agenda document.

…

On Tue, Jul 16, 2024 at 12:00 PM Mitchelle Rasquinha < ***@***.***> wrote: Can we please add this to the next benchmark chairs working group meeting agenda for discussion. — Reply to this email directly, view it on GitHub <#180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BFC3WA4ENXBW5ZPS3QD4MSDZMVGUPAVCNFSM6AAAAABKXCYNT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGQYDKNRTHA> . You are receiving this because your review was requested.Message ID: ***@***.***>

[rnchukka]

@ShriyaPalsamudram Do you have any context to the rules change proposal? In specific to the part where wg or power wg chairs participation/omission was mentioned. Since Monthly WG chair meeting is planning to discuss this item, should we wait for the opinion from that meeting before the changes are codified. To limit the results exposure for v4.1 Inference submission, Results committee can decide on some solution to let WG chairs participation on a need to know basis or let one WG co-chair join the meetings.

[ShriyaPalsamudram]

Given that discussion on this is pending, I marked it as a draft. Let's talk about it further in the Chairs WG meeting.

For now, we can keep everything as is, especially for v4.1 Inference submission.

[arjunsuresh]

Hi @ShriyaPalsamudram , please consider the below requests too

1. Should we limit the number of representatives from an organization attending the review meetings? Also, if some company is outsourcing their submission work to a new company (say to a non-MLC member company) - can those representatives attend the review meeting on behalf of the submitter company? My proposal would be upto 2 representatives from each submitter - they can be an employee or a representative.
2. We need to clarify who among the MLC staff can attend the review meetings - even the current rule doesn't clarify this but MLC staffs are managing the submission repositories. May be "upto 2 MLC staffs as agreed by the respective WG".

[TheKanter]

From an MLC standpoint, we see some challenges with Line 42 and wish to ensure WG chairs are included in the review committee. However, we understand the concerns that this potentially enables non-submitters into the confidentiality of the results review process. We share the goal of encouraging submission and want to avoid mechanisms to arbitrage access to the results. Instead suggest that we make it an explicit requirement that any all review committee members respect confidentiality and keep all results confidential even from their employer.

Further comments:

1. MLC needs WG chairs to see results to help analyze for marketing and drive better results briefing, press coverage, commentary, etc. MLC staff alone may lack the detailed knowledge and background to fully understand on their own. This is critical to driving benefits for all submitters.
2. From a marketing standpoint, we often quote WG chairs which is difficult unless they can see the results.
3. WG chair is an elected position and therefore a position of trust. Allowing any submitter to unilaterally over-rule that election seems bad form and contradictory.
4. MLC has a culture of trust and collaboration that has worked in the past and helps us to move quickly which is necessary for working with ML generally. WG chair is a significant investment of time and we should honor that investment and presume good faith - especially as they empowered to act on behalf of MLC. Culturally, we should only depart from this norm where abuse is an issue.
5. Observing and participating in the review process often directly informs subsequent rules changes and is valuable for this reason.
6. MLC cannot always find independent review chairs which makes WG chairs more helpful