[REVIEW]: SSMSE: An R package for Management Strategy Evaluation with Stock Synthesis Operating Models

[editorialbot]

Submitting author: @k-doering-NOAA (Kathryn Doering)
Repository: https://github.com/nmfs-fish-tools/ssmse
Branch with paper.md (empty if default branch):
Version: v0.2.8
Editor: @sbenthall
Reviewers: @quang-huynh, @iagomosqueira
Archive: 10.5281/zenodo.10014307

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/9af77892379058580aced7199f3dc6dd"><img src="https://joss.theoj.org/papers/9af77892379058580aced7199f3dc6dd/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/9af77892379058580aced7199f3dc6dd/status.svg)](https://joss.theoj.org/papers/9af77892379058580aced7199f3dc6dd)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@seananderson & @quang-huynh, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @sbenthall know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @seananderson

📝 Checklist for @quang-huynh

📝 Checklist for @iagomosqueira

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.12 s (860.1 files/s, 193990.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               58            566           2362           8736
Scheme                          19             18              0           7830
Markdown                         5            176              0           1004
Rmd                              8            336            612            539
CSS                              1             66              1            275
TeX                              1            271              0            275
YAML                            11             17             19            205
HTML                             1             58              8             83
-------------------------------------------------------------------------------
SUM:                           104           1508           3002          18947
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 3415

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1371/journal.pone.0092725 is OK
- 10.1577/1548-8446(2006)31[590:TCFIE]2.0.CO;2 is OK
- 10.1126/science.1149016 is OK
- 10.1002/fsh.10131 is OK
- 10.1016/j.fishres.2020.105725 is OK
- 10.7755/TMSPO.183 is OK
- 10.1016/j.fishres.2020.105854 is OK
- 10.1139/f03-109 is OK
- 10.1016/j.fishres.2018.12.014 is OK
- 10.1016/j.fishres.2012.10.012 is OK
- 10.2760/18924 is OK
- 10.1016/j.fishres.2022.106229 is OK
- 10.1111/faf.12104 is OK
- 10.1016/j.fishres.2021.106180 is OK
- 10.1006/jmsc.2000.0737 is OK
- 10.1139/cjfas-2020-0257 is OK
- 10.1111/j.1467-2979.2011.00417.x is OK
- 10.1111/faf.12480 is OK
- 10.1016/j.fishres.2021.105924 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[sbenthall]

👋🏼 @k-doering-NOAA @seananderson @quang-huynh this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#4937 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@sbenthall) if you have any questions/concerns.

[seananderson]

Review checklist for @seananderson

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/nmfs-fish-tools/ssmse?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@k-doering-NOAA) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[quang-huynh]

Review checklist for @quang-huynh

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/nmfs-fish-tools/ssmse?
• License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@k-doering-NOAA) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[quang-huynh]

@sbenthall I have published with John Walter, Nancie Cummings, and Cassidy Peterson in the past 4-5 years but the papers are not related to any work in this submission. Can I continue?

[seananderson]

@sbenthall Regarding potential conflicts of interest: I have published in the last ~3 years with K. Marshal but on work unrelated to this. I've published with several of the co-authors at ~6 years ago on work related to this paper (ss3sim), but that should be outside the 4-year window. I also have some level of ongoing collaboration with co-authors K. Johnson, I. Taylor, and C. Wetzel but that is unrelated to this paper or topic and would not impact my impartial scientific judgment or evaluation.

[sbenthall]

@quang-huynh and @seananderson -- Thank you so much for reporting these COIs.

@Kevin-Mattheus-Moerman Given the difficulty in finding qualified reviewers in this area, I would like to request that these COI's be waived. I believe this is your decision to make as track editor? Please let us know.

(See this Slack thread for discussion with @danielskatz .)

[sbenthall]

Hello. @Kevin-Mattheus-Moerman has weighed in on this.

@quang-huynh We can waive your COI in this case. Please proceed with your review!

@seananderson Unfortunately, we cannot waive your COI in this case. Thanks so much for reporting it. We will look for a different reviewer to replace you. If you could recommend anybody, that would be very helpful.

[seananderson]

OK. Other recommendations that may be less likely to have conflicts of interest:

• Laurence Kell or any co-authors of FLR: https://doi.org/10.1093/icesjms/fsm012 (should be fewer conflicts of interest given the European origin)
• Cathy Dichmont or Natalie Dowling from Australia

[sbenthall]

Dear @ejardim,

I am reaching out as an Editor of the Journal of Open Source Software, an on-line journal of scientific research software.

The submission tracked in this issue is from a team led by Kathryn Doering (@k-doering-NOAA) and is an R package about fishery stock assessment. We have been struggling to find qualified reviewers as this is a specialized area. You were suggested as a reviewer by Sean Anderson, who was not able to review the submission because of a conflict of interest.

On behalf of the journal, I would like to ask you if you'd be willing to review this submission. The process takes place via GitHub and I'll be happy to walk you through it.

Best regards,

Sebastian Benthall @sbenthall

[quang-huynh]

Here is my written review:

The authors present SSMSE, a R package that facilitates use of SS3, an executable software package, for management strategy evaluation, a simulation exercise which necessitates iterative application of SS3. I am familar with SS3 and a somewhat frequent user of the software.

Manuscript comments

Line 20: Recommend "..that can dynamically..." It is also possible to compare static harvest strategies (fixed catches) in MSE.
Line 22: ..may specify how a stock assessment model is configured..
Line 24: I recommend "hypothesized" over "true". If we knew the true dynamics, we wouldn't need to generate multiple OMs and evaluate misspecification.

Figure 3. Expand 'User inputs' and 'Models' bubbles to include setup of management procedures and operating models with create_om_list and sample_struct.
Figure 4. Specify that this figure describes the case study in the caption. Also state the assumption of constant M in all EMs.

The paper lists five types of uncertainty that should be considered in MSE. The example demonstrates 1 and 2. Implicitly, one conditions a different SS3 model in advance to address model uncertainty (point 3). I couldn't figure out how to specify catch bias and implementation error in the operating model? (Bullets 4 and 5). I think I'd have to set up a model with these adjustments in the control or forecast file? I'm not sure. Example code in the manual would be useful to cover these last two bullets.

Comments related to checklist

Reproducibility
I ran the script in: https://nmfs-fish-tools.github.io/SSMSE/manual/M-case-study-ex.html which I believe replicates the cod case study. No problems or convergence issues detected although I only ran 5 iterations instead of 100 due to runtime. The ggplot code to generate Figure 5 should be added on that web page.

Functionality documentation
Projected dynamics in the operating model can be tinkered with a list, with a template provided by SSMSE::create_future_om_list. However, there's no documentation in the help file on what information is needed and what options are possible. What are the required values for 'pars', 'scen', 'pattern', and 'input' entries in the list? I can figure it out with pattern recognition from the examples but that limits me to the case study.

Intimate familiarity with the SS3 model is a prerequisite to use SSMSE. To adjust natural mortality, I needed to know that the corresponding internal parameter name in this cod model is "NatM_p_1_Fem_GP_1". State where to find this in the r4ss output.

create_sample_struct seems to create a list that specifices the future data observations that will be simulated for the EM. State the options (catch, index, ages, etc) in the help file.

Examples can be documented in the help files with URL links to the online manual.

Additional nice things to add

A simple wrapper function to generate the file structure needed for the analysis (model_runs, figures, input_models) would be helpful.

Due to long runtime, I recommend progress bars for run_SSMSE to inform users of elapsed and projected time. See the pbapply R package.

Are catch limits the only type of management advice that can be tested with SSMSE? Size limits are used to manage U.S. fisheries but I'm not sure if it can be implemented here?

[k-doering-NOAA]

Thank you @quang-huynh for the thorough review! we plan to incorporate the suggestions once we have both reviews. @sbenthall I think we are still waiting to identify an additional reviewer? Please let me know if providing more suggested reviewers would be helpful. Thanks!

[sbenthall]

@k-doering-NOAA Yes, please, suggesting more reviewers would be very helpful. I have reached out to several people based on your earlier recommendations and the JOSS staff, but I'm afraid the fish haven't been biting ;-)

[k-doering-NOAA]

@sbenthall here are 2 suggestions for reviewers:

• Iago Mosqueira (Github Handle: iagomosqueira)
• Emil Aalto (https://deleolab.stanford.edu/people/emil-aalto)

[kellijohnson-NOAA]

@sbenthall here are a few more reviewer suggestions, I was on a review paper with all of them that was published in 2021 so I am unsure if that invokes a conflict of interest

• Nis Sand Jacobsen (GitHub handle: nissandjac)
• Carryn L. de Moor (http://www.maram.uct.ac.za/maram/people/demoor)
• Jin Gao (GitHub handle: jin-gao)

[sbenthall]

Hello @iagomosqueira ; would you be able to review this submission to the Journal of Open Source Software?

[iagomosqueira]

I would be willing to do so, but it will be more than impossible before the second half of May. That might be too late?

[sbenthall]

Thanks @iagomosqueira . I'll keep you in mind if we haven't found an alternate reviewer by May. But I'll keep looking for now...

[kthyng]

@sbenthall Looks like it has been hard to find reviewers for this submission! Any ideas at the moment?

[editorialbot]

👋 @openjournals/ese-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#4701, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[k-doering-NOAA]

@sbenthall , anything needed from the authors on this?

[sbenthall]

@k-doering-NOAA Not that I know of. The next step is that an Editor-in-Chief is supposed to finalize the acceptance. That might be @kthyng

[kthyng]

Yep ok my steps are

• Check that version was updated
• Check that software archive exists, has been input to JOSS, and title and author list look good
• Check paper

[kthyng]

@k-doering-NOAA We usually ask that authors change the metadata in the zenodo archive to match the JOSS title and author list — would you mind doing this?

[kthyng]

@k-doering-NOAA
Please check the capitalization in your references. You can preserve capitalization by placing {} around characters/words in your .bib file.

[k-doering-NOAA]

@kthyng I just changed the Zenodo metadata title to match the JOSS title. Author list already matches.

For the references, do I need to edit the .bib file and resubmit to Zenodo again?

[kthyng]

Yes edit the .bib and let's regenerate the pdf to check everything. No need to resubmit to Zenodo since it is just the paper text changing.

[k-doering-NOAA]

@kthyng, thank you, I made changes to the bib and pushed them to the main branch (https://github.com/nmfs-fish-tools/SSMSE). I used the JOSS paper draft github action to generate an informal proof and I think the capitalization issue is fixed. Thanks for the tip about {}!

[k-doering-NOAA]

@kthyng let me know if anything else needs to be fixed.

[kthyng]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[kthyng]

@k-doering-NOAA Just one more — the "K" in "karenia brevis" should be capitalized.

[k-doering-NOAA]

ok, just pushed a commit to fix the K, @kthyng

[kthyng]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[kthyng]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Doering
  given-names: Kathryn L.
  orcid: "https://orcid.org/0000-0002-0396-7044"
- family-names: Vaughan
  given-names: Nathan R.
  orcid: "https://orcid.org/0009-0000-3054-6950"
- family-names: Walter
  given-names: John F.
- family-names: Methot
  given-names: Richard D.
- family-names: Sagarese
  given-names: Skyler R.
  orcid: "https://orcid.org/0000-0002-6631-0103"
- family-names: Smith
  given-names: Matthew
- family-names: Farmer
  given-names: Nicholas A.
- family-names: Calay
  given-names: Shannon
- family-names: Cummings
  given-names: Nancie J.
- family-names: Johnson
  given-names: Kelli F.
  orcid: "https://orcid.org/0000-0002-5149-451X"
- family-names: Marshall
  given-names: Kristin
- family-names: Peterson
  given-names: Cassidy D.
  orcid: "https://orcid.org/0000-0002-0836-3039"
- family-names: Taylor
  given-names: Ian G.
  orcid: "https://orcid.org/0000-0002-4232-5669"
- family-names: Wetzel
  given-names: Chantel R.
  orcid: "https://orcid.org/0000-0002-7573-8240"
contact:
- family-names: Doering
  given-names: Kathryn L.
  orcid: "https://orcid.org/0000-0002-0396-7044"
doi: 10.5281/zenodo.10014307
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Doering
    given-names: Kathryn L.
    orcid: "https://orcid.org/0000-0002-0396-7044"
  - family-names: Vaughan
    given-names: Nathan R.
    orcid: "https://orcid.org/0009-0000-3054-6950"
  - family-names: Walter
    given-names: John F.
  - family-names: Methot
    given-names: Richard D.
  - family-names: Sagarese
    given-names: Skyler R.
    orcid: "https://orcid.org/0000-0002-6631-0103"
  - family-names: Smith
    given-names: Matthew
  - family-names: Farmer
    given-names: Nicholas A.
  - family-names: Calay
    given-names: Shannon
  - family-names: Cummings
    given-names: Nancie J.
  - family-names: Johnson
    given-names: Kelli F.
    orcid: "https://orcid.org/0000-0002-5149-451X"
  - family-names: Marshall
    given-names: Kristin
  - family-names: Peterson
    given-names: Cassidy D.
    orcid: "https://orcid.org/0000-0002-0836-3039"
  - family-names: Taylor
    given-names: Ian G.
    orcid: "https://orcid.org/0000-0002-4232-5669"
  - family-names: Wetzel
    given-names: Chantel R.
    orcid: "https://orcid.org/0000-0002-7573-8240"
  date-published: 2023-10-30
  doi: 10.21105/joss.04937
  issn: 2475-9066
  issue: 90
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 4937
  title: "SSMSE: An R package for Management Strategy Evaluation with
    Stock Synthesis Operating Models"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.04937"
  volume: 8
title: "SSMSE: An R package for Management Strategy Evaluation with
  Stock Synthesis Operating Models"

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

[editorialbot]

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.04937 joss-papers#4739
1. Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.04937
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[kthyng]

Congrats on your new publication @k-doering-NOAA!! Many thanks to editor @sbenthall and reviewers @quang-huynh, and @iagomosqueira for your time, hard work, and expertise!!

[editorialbot]

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.04937/status.svg)](https://doi.org/10.21105/joss.04937)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.04937">
  <img src="https://joss.theoj.org/papers/10.21105/joss.04937/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.04937/status.svg
   :target: https://doi.org/10.21105/joss.04937

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

• Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
• Making a small donation to support our running costs here: https://numfocus.org/donate-to-joss

[kthyng]

@k-doering-NOAA Would love to have you review sometime in the future! If you're interested, sign up here: https://reviewers.joss.theoj.org/