Calculating mod multipliers based on community survey (n = 68)

[MaklovitzLazer]

Abstract / tldr:

Methodology and results:

Based on the raw data from "What % accuracy should X scores tie NM SS scores?" survey assessed in osu! community (https://docs.google.com/forms/d/1ccBSyGq9tN_phmJIFetGOxdfBlHtB2RHNtyMLFJuR1A) I calculated mod multipliers for HD, HR, DT and FL in a way that they match the mean expectation of 68 voters. I got surveys' raw data from Elijah (sevenend7 on discord).

1. I calculated mean results of What accuracy should X scores tie NM SS scores?
   
2. Fitted bell curves (assumed normal distribution) to that
   
3. Used lazer score formula from ppy github

   osu/osu.Game/Rulesets/Scoring/ScoreProcessor.cs

   Line 380 in ef2e230

   protected virtual double ComputeTotalScore(double comboProgress, double accuracyProgress, double bonusPortion)

   
   
   to evaluate mod multipliers, so that for the cutoff accuracy (from the survey) of the FC score with given mod gives as much score as NM SS, which is 1 000 000 (assumed bonus portion is 0 & assumed FC with no sliderends dropped). The resulted formula is
   $m = \frac{2}{a + a^5}$
   a - mean accuracy from the survey
   m - corresponding mod multiplier
4. Calculated mod multipliers for HD, HR, DT and FL and for the most popular mod combinations.
   
   Here the exactly fitted values, if someone wants to use them for their own research:
   HD | 1.0815889623755
   HR | 1.14705534707846
   FL | 1.22892198047254
   DT | 1.26891860957178
5. Used score formula to calculate FC score in function of accuracy for the most popular mod multipliers
   

Conclusions:

Due to the new accuracy scaling
stable score ~ acc (if FC)
lazer score ~ $\frac{acc^5 + acc}{2}$ (if FC)
people find ALL mod multipliers underrated, which causes weird looking leaderboards (for example HDDT SS worth as much score as 98.0% HDHRDT FC) and a lot of frustration in leaderboard playing community. My take on this topic can be found here:
https://twitter.com/Maklovitz_osu/status/1752436258465259745
This analysis showed that we can easily evaluate the best mod multipliers, based on what players think about comparing an FC with a mod to a nomod SS. The upgraded version of this experiment would require asking thousands of active osu! community members (maybe even vote weight based on voter's pp rank & ranked score/# of lb scores), which would be hard without an official survey from peppy. High diversity of answers and independence of the voters is the key factor to benefit from the intelligence of the crowd.

Thanks to Elijah (SevenEnd7) for inspring me to do this analysis. His tweet that started all this:
https://twitter.com/SevenEnd7/status/1753668111998546200

WitherFlower's spreadsheet that works alomst the same as my calculations:
You can make a copy to play with different values: https://docs.google.com/spreadsheets/d/1iv7ptvppa9n-cBWrUlSDIlKZgjM0kf9LYMAMPpnSfow/edit?usp=sharing

More discussion about this topic can be found under this tweet:
https://twitter.com/Maklovitz_osu/status/1753743401990697415
Note that the results included in my tweet are slightly incorrect, which I explain in the comment:
https://twitter.com/Maklovitz_osu/status/1753755502880715089

[MaklovitzLazer]

Additional thoughts: In the survey we shouldn't we ask users
What would be the perfect mod multipliers
but
With what accuracy should scores with mod X tie NM SS scores,
because lazer score formula is not as intuitive as the stable one. If a mod multiplier is 1.12x then on stable you can estimate that with this mod you need FC with x/1.12 accuracy to match the NM FC with x accuracy. With lazer score this thinking pattern is just wrong and the best* mod multipliers will always look too big for most of the users, who still think about them in a stable way.

*the best = maximizing the user's experience that scores on the leaderboards are ordered correctly

[ominoussage]

Great work but I'm mostly concerned about this survey that took place since I had NO IDEA this even occurred. Also, who were the people that participated in this survey? I think it's very important to know what players participated and their respective rankings since we don't want some random low ranked 6-digit with 100 hours of playtime to participate in this.

Lastly, ONLY 68??? That isn't enough to capture the entirety of the osu community's opinion based on this topic. I also care about the mod multipliers myself and the fact that only a handful of people have been lucky to give their own thoughts about this is sad to see.

I hope another survey will be published and make sure that the survey will be published to a place that can gather attention and a lot of eyes can see, like r/osugame or post it as an official news article on the osu! website.

[Artcens]

@bdach suggestion for a poll feature in the client and roll the poll for spesific range of ranks. i think its such a cool feature to have.

[bdach]

which client? lazer? then stable people will say they werent asked. stable is feature locked so the best that can be done is probably like a banner announcement. and even then someone will say they didnt launch client for the N days that the banner was shown and therefore will complain of not being asked

[Artcens]

@bdach yep, lazer. i don't really care about stable client(since its feature locked) but yeah maybe banner for the poll in stable. but Lazer is the future, implement this stuff will make future discussion and decision much easier. and in lazer we got at least a thousand people playing daily now.

[Artcens]

@bdach about people complaining, lets the poll run for a week or even a month(since you say it need a few weeks to stabilize). a day poll seems too rush. because for example this problem with scoring, scoring stuff is a long term(or even permanent) so take time with it is a good thing.

if they still not aware, then don't care about it.

[Zyfarok]

There's always this guy that complains, but if the poll is put on the front-page or through an in-game banner then these kinds of people can hardly justify themselves. Even a lazer-only banner would be enough IMO.

[Purplegaze]

I feel like interactions between common mod combinations should definitely be accounted for too when surveying a question like "what accuracy should be required to snipe an SS".

In stable, ~94.6% HDDT is needed to snipe an HDHR SS. Do people really agree with raising this to ~96.6% in lazer? (as well as the comparatively high value of 99.3% for DT vs HDHR) What about other contentious mod combinations, such as FL being worth less than HDHR in this proposal?

If further surveying is done, this is definitely something to consider.

[ominoussage]

I personally think even a 95% or 96% DT-only FC should have more score than a HDHR SS. Hidden, for most maps, isn't that much difficult than NoMod so not requiring Hidden for DT to beat HDHR make sense since DT alone is just far more difficult to play than HD, HR or HDHR.
I played all mod combinations (HD, HR, HDHR, HDDT, HDDTHR and FL) and I really think DT should have more score.

For FL, I think it should be less then DT but more than HDHR. I can see it being far more difficult than HDHR on long maps.

[Livium129]

Really like this change overall, but had a little bit of an issue with the HDHR-FL weighting. Generally, FL and DTFL scores are much rarer than HDHR and 3mod scores, and giving a larger boost to HDHR disincentivizes playing an already rare mod combo for leaderboards outside of Easy and Normal difficulties. This problem exists on scorev1 as well, and it's also annoying there. So, I reweighted HD and HR to be slightly below FL when combined, giving FL players an ~0.5% boost over HDHR players. My new acc chart, and new multipliers, attached below.

This change also had the side effect of giving HDHRFL a 1% disadvantage compared to DTFL and 3mod. HDHRFL's equal weighting to 3mod and DTFL has been a constant annoyance for both DT playerbases, since the lack of rate increase makes it much easier than the other two mod combos in basically all circumstances.

These are the new multipliers that I used for HD and HR. The HR boost is exactly twice HD's, which is just an arbitrary choice on my part. Feel free to change around if necessary.

[OwenCMYK]

I think it would be best to instead gather this data based on real-world data of how many players pass/FC a beatmap depending on different mods. And even then, I personally think mods should be slighty underscored on average compared to their difficulty. Because the difficulty that each mod adds depends heavily on the way the map is designed and the skillset a player has. And I think it's better for mods to be "not worth the effort" than it is for unmodded plays to be "not worth the effort". Because not matter what you make the multiplier, as long as it's above 1, HR will always be worth it to somebody. Even more so with HD. So I feel it's better to ere on the side of caution.