Merge pull request #11 from bbtufty/romsearch_method

ROMSearch method

CHANGES.rst

@@ -1,9 +1,27 @@
 0.0.5 (Unreleased)
 ==================
 
+Features
+--------
+
+- ROMSearch now has two modes: the first is `filter_then_download` (default), which will use the dat file to filter,
+  then only download relevant files. The second is `download_then_filter`, which will download everything and then
+  filter. For data hoarders!
+
 Fixes
 -----
 
+GameFinder
+~~~~~~~~~~
+
+- Ensure includes/excludes works the same as it does for ROMDownloader
+- Includes/excludes will now search dupes as well, for consistency
+
+ROMDownloader
+~~~~~~~~~~~~~
+
+- Ensure output directory exists before downloading files
+
 General
 ~~~~~~~
 

docs/configs/config.rst

@@ -4,6 +4,9 @@ config
 
 The ``config.yml`` file defines how ROMSearch will do the run. As such, it has quite a number of options.
 
+As a note on includes, this will match something from the start of the string. So "Game Title VII" would include
+"Game Title VII", "Game Title VIII", but not "Game Title Anthology - Game Title VII", for example.
+
 Syntax: ::
 
     dirs:
@@ -31,6 +34,8 @@ Syntax: ::
         - [game]
 
     romsearch:                          # ROMSearch specific options
+      method: 'filter_then_download'    # OPTIONAL. Method to use, option are 'filter_then_download', or
+                                        #           'download_then_filter'. Defaults to 'filter_then_download'
       run_romdownloader: true           # OPTIONAL. Whether to run ROMDownloader. Defaults to true
       run_datparser: true               # OPTIONAL. Whether to run DATParser. Defaults to true
       run_dupeparser: true              # OPTIONAL. Whether to run DupeParsed. Defaults to true
@@ -58,15 +63,15 @@ Syntax: ::
     romchooser:                         # ROMChooser specific options
       dry_run: false                    # OPTIONAL. Set to true to not make any changes to filesystem. Defaults to false
       use_best_version: true            # OPTIONAL. Whether to choose only what ROMChooser decides is the best version.
-                                                    Defaults to true
+                                        #           Defaults to true
       allow_multiple_regions: false     # OPTIONAL. If true, will allow files from multiple regions, else will choose the
-                                                    highest region in the list. Defaults to false
+                                        #           highest region in the list. Defaults to false
       filter_regions: true              # OPTIONAL. Whether to filter by region or not. Defaults to true
       filter_languages: true            # OPTIONAL. Whether to filter by language or not. Defaults to true
       bool_filters: "all_but_games"     # OPTIONAL. Can filter out non-games by various dat categories. If you want to
-                                                    include e.g. just games and applications, set to
-                                                    ['games', 'applications']. Defaults to 'all_but_games', which will
-                                                    remove everything except games
+                                        #           include e.g. just games and applications, set to
+                                        #           ['games', 'applications']. Defaults to 'all_but_games', which will
+                                        #           remove everything except games
 
     discord:                            # OPTIONAL. If defined, supply a webhook URL so that ROMSearch can post Discord
       webhook_url: [webhook_url]        #           notifications

docs/intro.rst

@@ -27,6 +27,11 @@ ROMSearch offers the ability to:
 To get started, see the :doc:`installation <installation>` and :doc:`configuration <configuration>` pages. For the
 philosophy behind how ROMSearch chooses a ROM, see :doc:`1G1R <1g1r>`.
 
+ROMSearch offers two modes: the default is "filter, then download" which will use the .dat file to find the best ROMs
+and only download those. For data hoarders, we also offer a "download, then filter" option, which will download
+everything and then filter from the downloaded files. For more details, see the
+:doc:`ROMSearch module docs <modules/romsearch>`.
+
 Currently, ROMSearch is in early development, and so many features may be added over time. At the moment, ROMSearch
 has the capability for:
 

docs/known_issues.rst

@@ -2,6 +2,10 @@
 Known Issues
 ############
 
+* In GameFinder, includes/excludes can cause some unexpected behavior since it will also search through duplicate files.
+  For example, having an include of "Crash Bandicoot" for the PS1 will also grab "Crash Bash" and
+  "CTR - Crash Team Racing" since at least one of their duplicates starts with "Crash Bandicoot".
+
 * Currently, the code is not aware of ``retool``'s supersets or compilations array.
 
 * Occasionally, multiple ROMs will be found with the same priority.

docs/modules/romsearch.rst

@@ -5,6 +5,10 @@ ROMSearch
 This is the main part that controls the various other modules. It essentially calls everything (given user preferences),
 and so while it doesn't really do all that much on its own, is the interface to everything else.
 
+ROMSearch has 2 modes, the default will parse from the .dat file then download relevant files, to minimize disc space
+used (`filter_then_download`). For completionists/data hoarders, there's also a `download_then_filter` option, which
+will download and then filter from the downloaded files.
+
 For more details on the ROMSearch arguments, see the :doc:`config file documentation <../configs/config>`.
 
 API

romsearch/configs/sample_config.yml

@@ -21,6 +21,7 @@ include_games:
     - "Chrono Trigger"
 
 romsearch:
+  method: "filter_then_download"
   run_romdownloader: true
   run_datparser: true
   run_dupeparser: true

romsearch/modules/gamefinder.py

@@ -133,24 +133,6 @@ def get_game_dict(self,
                               regex_config=self.regex_config,
                               )
 
-        # Remove any excluded files
-        if self.exclude_games is not None:
-            games_to_remove = self.get_game_matches(games,
-                                                    self.exclude_games,
-                                                    )
-
-            if games_to_remove is not None:
-                for i in sorted(games_to_remove, reverse=True):
-                    games.pop(i)
-
-        # Include only included files
-        if self.include_games is not None:
-            games_to_include = self.get_game_matches(games,
-                                                     self.include_games,
-                                                     )
-            if games_to_include is not None:
-                games = np.asarray(games)[games_to_include]
-
         # We need to trim down dupes here. Otherwise, the
         #  dict is just the list we already have
         game_dict = None
@@ -166,10 +148,41 @@ def get_game_dict(self,
                 game_dict[game] = {"priority": 1,
                                    }
 
+        # Remove any excluded files
+        if self.exclude_games is not None:
+            games_to_remove = self.get_game_matches(game_dict,
+                                                    self.exclude_games,
+                                                    )
+            if games_to_remove is not None:
+                for g in games_to_remove:
+                    del game_dict[g]
+
+        # Include only included files
+        if self.include_games is not None:
+
+            games_to_include = self.get_game_matches(game_dict,
+                                                     self.include_games,
+                                                     )
+            if games_to_include is not None:
+
+                filtered_game_dict = {}
+                for g in games_to_include:
+                    filtered_game_dict[g] = game_dict[g]
+
+                game_dict = copy.deepcopy(filtered_game_dict)
+
         return game_dict
 
-    def get_game_matches(self, files, games_to_match):
-        """Get files that match an input list (games_to_match)"""
+    def get_game_matches(self,
+                         game_dict,
+                         games_to_match,
+                         ):
+        """Get files that match an input dictionary (so as to properly handle dupes
+
+        Args:
+            - game_dict (dict): Dictionary of games to match against
+            - games_to_match (list): List of values to match against
+        """
         games_matched = []
 
         if isinstance(games_to_match, dict):
@@ -182,22 +195,41 @@ def get_game_matches(self, files, games_to_match):
 
         games_matched.extend(games_to_match)
 
-        idx = []
-        for i, f in enumerate(files):
+        game_dict_keys = []
+        for g in game_dict:
             found_f = False
-            # Search within each item since the matches might not be exact
+
             for game_matched in games_matched:
 
                 if found_f:
                     continue
 
-                re_find = re.findall(f"{game_matched}*", f)
+                # Look in the group name
+                re_find = re.findall(f"^({re.escape(game_matched)}).*", g)
 
                 if len(re_find) > 0:
-                    idx.append(i)
+                    game_dict_keys.append(g)
                     found_f = True
 
-        return idx
+                # If not found, look in the dupe names
+                if not found_f:
+                    for g_d in game_dict[g]:
+
+                        if found_f:
+                            continue
+
+                        re_find = re.findall(f"^({re.escape(game_matched)}).*", g_d)
+
+                        if len(re_find) > 0:
+                            game_dict_keys.append(g)
+                            found_f = True
+
+        game_dict_keys = np.unique(game_dict_keys)
+
+        if len(game_dict_keys) == 0:
+            game_dict_keys = None
+
+        return game_dict_keys
 
     def get_filter_dupes(self, games):
         """Parse down a list of files based on an input dupe list"""

romsearch/modules/romdownloader.py

@@ -13,8 +13,8 @@
 
 
 def add_rclone_filter(pattern=None,
-                      # bracketed_pattern=None,
                       filter_type="include",
+                      include_wildcard=True,
                       ):
     if filter_type == "include":
         filter_str = "+"
@@ -26,14 +26,11 @@ def add_rclone_filter(pattern=None,
     # rclone wants double curly braces which we need to escape in python strings (yum)
     filter_pattern = ""
 
-    # # Add in non-bracketed stuff (i.e. game names) at the start
-    # if non_bracketed_pattern is not None:
-    #     filter_pattern += f"{{{{{non_bracketed_pattern}}}}}"
-    #
-    # filter_pattern += "*"
-
     if pattern is not None:
-        filter_pattern += f"{{{{{pattern}}}}}*"
+        filter_pattern += f"{{{{{pattern}}}}}"
+
+    if include_wildcard:
+        filter_pattern += "*"
 
     cmd = f' --filter "{filter_str} {filter_pattern}"'
 
@@ -63,6 +60,9 @@ def __init__(self,
                  config=None,
                  platform_config=None,
                  logger=None,
+                 override_includes=None,
+                 override_excludes=None,
+                 include_filter_wildcard=True,
                  ):
         """Downloader tool via rclone
 
@@ -74,6 +74,12 @@ def __init__(self,
             config (dict, optional): Configuration dictionary. Defaults to None
             platform_config (dict, optional): Platform configuration dictionary. Defaults to None
             logger (logging.Logger, optional): Logger instance. Defaults to None
+            override_includes (list, optional): If set, will override the config includes with custom
+                ones. Defaults to None.
+            override_excludes (list, optional): If set, will override the config excludes with custom
+                ones. Defaults to None.
+            include_filter_wildcard (bool, optional): If set, will include wildcards in rclone filters. Defaults to
+                True.
         """
 
         if platform is None:
@@ -107,13 +113,21 @@ def __init__(self,
             include_games = include_games.get(platform, None)
         else:
             include_games = copy.deepcopy(include_games)
+
+        if override_includes is not None:
+            include_games = copy.deepcopy(override_includes)
+
         self.include_games = include_games
 
         exclude_games = self.config.get("exclude_games", None)
         if isinstance(exclude_games, dict):
             exclude_games = exclude_games.get(platform, None)
         else:
             exclude_games = copy.deepcopy(exclude_games)
+
+        if override_excludes is not None:
+            exclude_games = copy.deepcopy(override_excludes)
+
         self.exclude_games = exclude_games
 
         remote_name = self.config.get("romdownloader", {}).get("remote_name", None)
@@ -122,8 +136,15 @@ def __init__(self,
         self.remote_name = remote_name
 
         sync_all = self.config.get("romdownloader", {}).get("sync_all", True)
+
+        # If we have includes or excludes, force sync all False
+        if self.include_games is not None or self.exclude_games is not None:
+            sync_all = False
+
         self.sync_all = sync_all
 
+        self.include_filter_wildcard = include_filter_wildcard
+
         # Read in the specific platform configuration
         mod_dir = os.path.dirname(romsearch.__file__)
 
@@ -213,6 +234,7 @@ def rclone_sync(self,
             if pattern:
                 cmd += add_rclone_filter(pattern=pattern,
                                          filter_type="exclude",
+                                         include_wildcard=self.include_wildcard,
                                          )
 
             # Now onto positive filters
@@ -230,6 +252,7 @@ def rclone_sync(self,
             if pattern:
                 cmd += add_rclone_filter(pattern=pattern,
                                          filter_type="include",
+                                         include_wildcard=self.include_filter_wildcard,
                                          )
 
                 cmd += ' --filter "- *"'
@@ -238,7 +261,10 @@ def rclone_sync(self,
             self.logger.info(f"Dry run, would rclone_sync with:")
             self.logger.info(cmd)
         else:
-            # os.system(cmd)
+
+            if not os.path.exists(self.out_dir):
+                os.makedirs(self.out_dir)
+
             # Execute the command and capture the output
             with subprocess.Popen(cmd, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as process:
                 for line in process.stdout:
@@ -267,6 +293,8 @@ def post_to_discord(self,
 
         if len(items_added) > 0:
 
+            items_added.sort()
+
             for items_split in split(items_added, chunk_size=max_per_message):
 
                 fields = []
@@ -284,6 +312,8 @@ def post_to_discord(self,
 
         if len(items_deleted) > 0:
 
+            items_deleted.sort()
+
             for items_split in split(items_deleted, chunk_size=max_per_message):
 
                 fields = []

romsearch/modules/romparser.py

@@ -1,12 +1,10 @@
 import copy
 import os
 import re
-import time
-from datetime import datetime
 
 import romsearch
 from ..util import (setup_logger,
-
+                    get_file_time,
                     load_yml,
                     load_json,
                     get_game_name,
@@ -70,21 +68,6 @@ def get_pattern_val(regex,
     return pattern_val
 
 
-def get_file_time(f,
-                  datetime_format,
-                  ):
-    """Get created file time from the file itself"""
-
-    if os.path.exists(f):
-        ti_m = os.path.getmtime(f)
-        date_ti_m = datetime.strptime(time.ctime(ti_m), "%a %b %d %H:%M:%S %Y")
-    else:
-        date_ti_m = datetime(year=1900, month=1, day=1, hour=0, minute=0, second=0)
-    date_ti_m_str = date_ti_m.strftime(format=datetime_format)
-
-    return date_ti_m_str
-
-
 class ROMParser:
 
     def __init__(self,