You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The addition of BERTScore to the QAAccuracy in version 1.2.0 has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution here) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.
Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in QAAccuracyConfig to disable it?
The text was updated successfully, but these errors were encountered:
The addition of BERTScore to the QAAccuracy in version 1.2.0 has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution here) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.
Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in
QAAccuracyConfigto disable it?The text was updated successfully, but these errors were encountered: