Merge pull request #41 from bbtufty/romparser-patches

RAHashes with patch files

CHANGES.rst

@@ -14,6 +14,7 @@ Fixes
 DATParser
 ~~~~~~~~~
 
+- Ensure we pick up the right dat file if names are similar
 - Return dat dict directly from ``run``
 
 DupeParser
@@ -25,15 +26,23 @@ ROMChooser
 ~~~~~~~~~~
 
 - Added in scoring if ROM has associated RA achievements
+- Overhauled the ROMChoosing. Is now clearer with filters and then scores
+- Fixed issue with ordering versions for scoring
 
 ROMDownloader
 ~~~~~~~~~~~~~
 
 - Fixed crash if file does not exist on remote
 
+ROMMover
+~~~~~~~~
+
+- Include patch info in cache file
+
 ROMParser
 ~~~~~~~~~
 
+- Add parsing for RetroAchievement-supported ROMs
 - Parse checksums out of dat files
 - Can take dat and dupe dicts directly, to avoid file I/O
 

docs/1g1r.rst

@@ -3,49 +3,42 @@
 ####
 
 ROMSearch operates on a "one game, one ROM" (1G1R) philosophy. This means that for each game, it will find the
-ROM file that it believes to be the best. The approach ROMSearch takes depends mostly on regions, languages,
-versioning, and optional name priority, with regions being the ultimate discriminator.
+ROM file that it believes to be the best. The approach ROMSearch takes is to (1) filter out ROMs that the user does
+not want, and then (2) score the remaining ROMs to find an overall "best" ROM.
 
-Languages
----------
-
-Many ROMs are tagged with languages (En, Es, etc), that will be parsed out during the run. If a ROM does not contain
-the language the user has specified (most likely En), then this will be removed from the choice.
-
-A lot of ROMs do not have any language tags (particularly US ones). For these, we currently do not cut them out,
-but this may change in the future to associate a region with an implicit language.
-
-Versions
---------
-
-There may be different versions of ROMs (e.g. Rev 1, v2.0, etc). For these, we will take the latest one per unique
-region combination as the latest and greatest version.
+Filters
+-------
 
-Priority
---------
+Firstly, ROMs are filtered.
 
-Some files are essentially stripped down versions of games and so should be de-prioritised versus others. To get
-a priority, we use the ``retool`` clonelists.
+* This starts with dat categories the user may not want (by default, this is everything not a game,
+  so demos, beta, etc.)
+* We then filter out ROMs that don't have languages (or implied languages) that are in the user preferences
+* ROMs are filtered if they don't have regions in the user preferences
 
-Regions
+Scoring
 -------
 
-After the various cuts, there will still be a number ROMs that pass all checks. We therefore do a filter by
-region (e.g. USA, Europe) to get to a final ROM. ROMSearch will generally choose the ROM with the highest
-region preference for each game, so order is important here!
+After the initial round of filtering, we then score the ROMs. Some have an ordered priority, which means that
+the order the user has selected matters (e.g. USA, World, Europe for regions).
 
-Achievements
-------------
+Scores are boosted in this order of priority:
 
-ROMSearch will prioritise ROMs that are RetroAchievement enabled over others. This may mean you don't necessarily
-get the most up-to-date version or your top region.
+* Has RetroAchievements
+* Regions (ordered priority)
+* Languages (ordered priority)
+* Budget editions
+* Versions/revisions
+* Improved versions (e.g. EDC)
 
-Others
-------
+There are also some demotions that go on. The priority is (from most to least demoted):
 
-There are some other cuts that go into deciding the best ROM. These include:
+* Priority from ``retool`` clonelists
+* Modern versions
+* Alternate versions
+* Demoted versions (e.g. arcade versions)
 
-* Improved version tags (e.g. "EDC" for PS1 games)
-* Removing of demos/preproduction ROMs
+End result
+----------
 
 The final result is that you should get the single best ROM for your preferences. Hooray!

docs/configs/config.rst

@@ -76,8 +76,6 @@ Syntax: ::
       dry_run: false                    # OPTIONAL. Set to true to not make any changes to filesystem. Defaults to false
       use_best_version: true            # OPTIONAL. Whether to choose only what ROMChooser decides is the best version.
                                         #           Defaults to true
-      allow_multiple_regions: false     # OPTIONAL. If true, will allow files from multiple regions, else will choose the
-                                        #           highest region in the list. Defaults to false
       filter_regions: true              # OPTIONAL. Whether to filter by region or not. Defaults to true
       filter_languages: true            # OPTIONAL. Whether to filter by language or not. Defaults to true
       bool_filters: "all_but_games"     # OPTIONAL. Can filter out non-games by various dat categories. If you want to

docs/modules/romparser.rst

@@ -8,6 +8,11 @@ languages, and various flags for things like Demos or BIOS that then can be used
 If explicit languages are not found, then ROMParser will generally assign a language based on the region. This is not
 true for regions like Scandinavia, where it could be any of multiple languages.
 
+ROMParser can also determine whether ROMs are compatible with RetroAchievements. It does this in two passes - first
+by matching to the hash (if RA uses MD5 hashes) or the name. Secondly, for ROMs that require patches it will search
+by those by name, matching to things like region/version. In the second case, if a patch is needed it will pull that
+out. This does rely on the naming on RA being pretty accurate, which is mostly the case but some may be missed.
+
 For more details on the ROMParser arguments, see the :doc:`config file documentation <../configs/config>`.
 
 API

romsearch/configs/dats/no-intro.yml

@@ -1,11 +1,11 @@
 Nintendo - Game Boy:
-  file_mapping: "Nintendo - Game Boy"
+  file_mapping: "Nintendo - Game Boy (*)"
 
 Nintendo - Game Boy Color:
-  file_mapping: "Nintendo - Game Boy Color"
+  file_mapping: "Nintendo - Game Boy Color (*)"
 
 Nintendo - Nintendo Entertainment System:
-  file_mapping: "Nintendo - Nintendo Entertainment System (Headered)"
+  file_mapping: "Nintendo - Nintendo Entertainment System (Headered) (*)"
 
 Nintendo - Super Nintendo Entertainment System:
-  file_mapping: "Nintendo - Super Nintendo Entertainment System"
+  file_mapping: "Nintendo - Super Nintendo Entertainment System (*)"

romsearch/configs/dats/redump.yml

@@ -2,16 +2,16 @@ url: "http://redump.org/datfile"
 
 Nintendo - GameCube:
   web_mapping: "gc"
-  file_mapping: "Nintendo - GameCube - Datfile"
+  file_mapping: "Nintendo - GameCube - Datfile (*)"
 
 Sony - PlayStation:
   web_mapping: "psx"
-  file_mapping: "Sony - PlayStation - Datfile"
+  file_mapping: "Sony - PlayStation - Datfile (*)"
 
 Sony - PlayStation 2:
   web_mapping: "ps2"
-  file_mapping: "Sony - PlayStation 2 - Datfile"
+  file_mapping: "Sony - PlayStation 2 - Datfile (*)"
 
 Sony - PlayStation Portable:
   web_mapping: "psp"
-  file_mapping: "Sony - PlayStation Portable - Datfile "
+  file_mapping: "Sony - PlayStation Portable - Datfile (*)"

romsearch/configs/defaults.yml

@@ -208,3 +208,28 @@ dat_checksums:
   - "md5"
   - "sha1"
   - "sha256"
+
+ra_file_exts:
+  - ".bin"
+  - ".bs"
+  - ".iso"
+  - ".fds"
+  - ".gb"
+  - ".gbc"
+  - ".nds"
+  - ".nes"
+  - ".sfc"
+  - ".sgb"
+  - ".unf"
+  - ".zip"
+
+ra_labels:
+  - "redump"
+  - "nointro"
+
+ra_patch_checks:
+  - "regions"
+  - "version_no"
+  - "revision"
+  - "multi_disc"
+  - "demo"

romsearch/configs/sample_config.yml

@@ -53,7 +53,6 @@ romparser:
 romchooser:
   dry_run: false
   use_best_version: true
-  allow_multiple_regions: false
   filter_regions: true
   filter_languages: true
   bool_filters: "all_but_games"

romsearch/dev/__init__.py

@@ -1,6 +1,8 @@
-from .parsing_tools import check_regex_parsing, parse_games_from_dat
+from .gamefinder_dev import check_regex_parsing, parse_games_from_dat
+from .rahasher_dev import check_rahasher_names
 
 __all__ = [
     "check_regex_parsing",
+    "check_rahasher_names",
     "parse_games_from_dat",
 ]

romsearch/dev/rahasher_dev.py

@@ -0,0 +1,37 @@
+import os
+
+from ..util import load_yml, load_json
+
+
+def check_rahasher_names(config_file, platform):
+    """Using a config file and platform, get all applicable RA names with hashes
+
+    Args:
+        config_file (string): Path to the config file
+        platform (string): Platform name
+    """
+
+    config = load_yml(config_file)
+
+    ra_hash_dir = config.get("dirs", {}).get("ra_hash_dir", None)
+
+    if ra_hash_dir is None:
+        raise ValueError("No ra_hash_dir found in config file")
+
+    ra_hash_filename = os.path.join(ra_hash_dir, f"{platform}.json")
+
+    if not os.path.exists(ra_hash_filename):
+        raise ValueError("No RA hash file found")
+
+    ra_hash_dict = load_json(ra_hash_filename)
+
+    ra_hashes = {}
+    for r in ra_hash_dict:
+        for h in ra_hash_dict[r]["Hashes"]:
+            ra_hashes[h["MD5"]] = {
+                "name": h["Name"],
+                "hash": h["MD5"],
+                "patch_name": h["PatchUrl"],
+            }
+
+    return ra_hashes

romsearch/gui/gui_config.py

@@ -216,7 +216,6 @@ def __init__(
 
         self.all_romchooser_options = {
             "use_best_version": self.ui.checkBoxConfigRomChooserUseBestVersion,
-            "allow_multiple_regions": self.ui.checkBoxConfigRomChooserAllowMultipleRegions,
             "filter_regions": self.ui.checkBoxConfigRomChooserFilterRegions,
             "filter_languages": self.ui.checkBoxConfigRomChooserFilterLanguages,
             "dry_run": self.ui.checkBoxConfigRomChooserDryRun,

romsearch/gui/layout_romsearch.py

@@ -1003,19 +1003,6 @@ def setupUi(self, RomSearch):
 
         self.verticalLayoutConfigRomChooserSettings.addWidget(self.checkBoxConfigRomChooserUseBestVersion)
 
-        self.lineConfigRomChooserUseBestVersionDividerBottom = QFrame(self.tabConfigRomChooser)
-        self.lineConfigRomChooserUseBestVersionDividerBottom.setObjectName(u"lineConfigRomChooserUseBestVersionDividerBottom")
-        self.lineConfigRomChooserUseBestVersionDividerBottom.setFrameShadow(QFrame.Shadow.Plain)
-        self.lineConfigRomChooserUseBestVersionDividerBottom.setFrameShape(QFrame.Shape.HLine)
-
-        self.verticalLayoutConfigRomChooserSettings.addWidget(self.lineConfigRomChooserUseBestVersionDividerBottom)
-
-        self.checkBoxConfigRomChooserAllowMultipleRegions = QCheckBox(self.tabConfigRomChooser)
-        self.checkBoxConfigRomChooserAllowMultipleRegions.setObjectName(u"checkBoxConfigRomChooserAllowMultipleRegions")
-        self.checkBoxConfigRomChooserAllowMultipleRegions.setChecked(False)
-
-        self.verticalLayoutConfigRomChooserSettings.addWidget(self.checkBoxConfigRomChooserAllowMultipleRegions)
-
         self.lineConfigRomChooserFilterRegionsDividerTop = QFrame(self.tabConfigRomChooser)
         self.lineConfigRomChooserFilterRegionsDividerTop.setObjectName(u"lineConfigRomChooserFilterRegionsDividerTop")
         self.lineConfigRomChooserFilterRegionsDividerTop.setFrameShadow(QFrame.Shadow.Plain)
@@ -1674,10 +1661,6 @@ def retranslateUi(self, RomSearch):
         self.checkBoxConfigRomChooserUseBestVersion.setStatusTip(QCoreApplication.translate("RomSearch", u"Whether to only use the best ROM version per-region. Default checked", None))
 #endif // QT_CONFIG(statustip)
         self.checkBoxConfigRomChooserUseBestVersion.setText(QCoreApplication.translate("RomSearch", u"Use best version?", None))
-#if QT_CONFIG(statustip)
-        self.checkBoxConfigRomChooserAllowMultipleRegions.setStatusTip(QCoreApplication.translate("RomSearch", u"Whether to allow multiple regions, or to just use the most preferred region. Default unchecked", None))
-#endif // QT_CONFIG(statustip)
-        self.checkBoxConfigRomChooserAllowMultipleRegions.setText(QCoreApplication.translate("RomSearch", u"Allow multiple regions?", None))
 #if QT_CONFIG(statustip)
         self.checkBoxConfigRomChooserFilterRegions.setStatusTip(QCoreApplication.translate("RomSearch", u"Whether to remove ROMs that don't fall into the selected region choices. Default checked", None))
 #endif // QT_CONFIG(statustip)

romsearch/gui/layout_romsearch.ui

@@ -1755,29 +1755,6 @@
                   </property>
                  </widget>
                 </item>
-                <item>
-                 <widget class="Line" name="lineConfigRomChooserUseBestVersionDividerBottom">
-                  <property name="frameShadow">
-                   <enum>QFrame::Shadow::Plain</enum>
-                  </property>
-                  <property name="orientation">
-                   <enum>Qt::Orientation::Horizontal</enum>
-                  </property>
-                 </widget>
-                </item>
-                <item>
-                 <widget class="QCheckBox" name="checkBoxConfigRomChooserAllowMultipleRegions">
-                  <property name="statusTip">
-                   <string>Whether to allow multiple regions, or to just use the most preferred region. Default unchecked</string>
-                  </property>
-                  <property name="text">
-                   <string>Allow multiple regions?</string>
-                  </property>
-                  <property name="checked">
-                   <bool>false</bool>
-                  </property>
-                 </widget>
-                </item>
                 <item>
                  <widget class="Line" name="lineConfigRomChooserFilterRegionsDividerTop">
                   <property name="frameShadow">