Merge pull request #51 from lemoney/feature/documentation-drive-1

Adding a User Guide

.github/PULL_REQUEST_TEMPLATE.md

@@ -15,13 +15,6 @@ Use this space to explain what you expect to happen when if this PR
 is merged.
 
 ### Checklist
- - [ ] Local test from submitter [link to any relevant results]()
+ - [ ] Tests
+ - [ ] Documentation
  - [ ] Maintainer Code Review
- - [ ] Added To Wiki
-
-
-### Optional Additional Checklist
-###### This is for external dependencies you have also need before your PR is merged
-
- - [ ] If you have no external things just delete this checkbox
-

bin/grease.ps1

@@ -6,4 +6,4 @@ foreach($arg in $args){
 
 $loc = split-path $SCRIPT:MyInvocation.MyCommand.Path -parent
 
-Invoke-Expression "python $loc\grease $argsString"
+python "$loc\grease" $argsString

docs/GREASE_Guide.rst

@@ -8,6 +8,7 @@ below for more information on how to get started.
    :caption: Contents:
 
    GreaseGuide/what_is_grease
+   GreaseGuide/user_guide
    GreaseGuide/Installation
    GreaseGuide/data_model
 

docs/GreaseGuide/Installation.rst

@@ -1,20 +1,23 @@
 .. _installing-grease:
 
-GREASE Installation
-*********************
+GREASE Administration:  Up & Running
+***************************************
+
+Initial Installation
+=======================
 
 Via PyPi: The Traditional Way
-===================================
+---------------------------------
 
 GREASE is built like any other traditional python software, as a package. This means many things but luckily for
 the systems administrator it means we traverse the typical PyPi pipeline allowing for you to specify your version
 and allow for updates to be pretty easy. Here is the installation steps:
 
 #. Run `pip install tgt_grease`
-#. Setup your configuration file
+#. Setup your configuration file following
 
 From Source on GitHub: Because you're cool like that
-=========================================================
+-------------------------------------------------------
 
 GREASE is developer friendly! We're always looking for new ways to bring joy to our guests as well as our developers. To
 install GREASE via source follow these steps:
@@ -26,3 +29,129 @@ install GREASE via source follow these steps:
    - **NOTE:** We recommend for all use cases to use a virtual environment
 
 #. Setup your configuration file
+
+Understanding & Configuring Your System
+========================================
+
+There are multiple definitions for configuration in GREASE. The primary ones are:
+
+#. Node Configuration: This refers to the local server's configuration for things like MongoDB credentials & resource limits
+#. Cluster Configuration: This refers to the configuration of Job Servers inside of the cluster
+#. Prototype Configuration: These are the configurations for prototypes, things such as sourcing & detection
+
+Node Configuration
+----------------------
+
+.. _nodeconfig:
+
+Node configuration is the local file the running instance uses to execute GREASE operations. It is stored in a file
+called :code:`grease.conf.json`. This is stored in the GREASE directory. On Unix-like operating systems is found at
+:code:`/opt/grease/` and for Windows :code:`C:\\grease\\`. You can override this behavior using the environment variable
+:code:`GREASE_DIR`.
+
+The default configuration looks like this::
+
+    {
+        'Connectivity': {
+            'MongoDB': {
+                'host': 'localhost',
+                'port': 27017
+            }
+        },
+        'Logging': {
+            'mode': 'filesystem',
+            'verbose': False,
+            'trace': False,
+            'foreground': False,
+            'file': Configuration.greaseDir + 'log' + os.sep + 'grease.log'
+        },
+        'Notifications': {
+            'HipChat': {
+                'enabled': False,
+                'token': None,
+                'room': None
+            }
+        },
+        'Configuration': {
+            'dir': Configuration.greaseDir + 'etc' + os.sep
+        },
+        'Sourcing': {
+            'dir': Configuration.greaseDir + 'etc' + os.sep,
+            'source': None,
+            'config': None,
+            'mock': False
+        },
+        'Import': {
+            'searchPath': [
+                'tgt_grease.router.Commands',
+                'tgt_grease.enterprise.Prototype',
+                'tgt_grease.management.Commands',
+                'tgt_grease.enterprise.Sources',
+                'tgt_grease.enterprise.Detectors',
+                'tgt_grease.core',
+                'tgt_grease'
+            ]
+        },
+        "NodeInformation": {
+            "ResourceMax": 95,
+            "DeduplicationThreads": 150
+        },
+        "Additional": {}
+    }
+
+Lets go through each key and the properties below, what they control and some values you may want to use.
+
+* Connectivity: This is the store for details around connectivity
+    * MongoDB: This key is the key used to find connection details about the central database
+
+        =========== =============   ============
+        Key         value type      default
+        =========== =============   ============
+        host        str             localhost
+        port        int             27017
+        username    str
+        password    str
+        db          str
+        =========== =============   ============
+* Logging: Logging configuration information
+    * mode: only supports filesystem logging currently to the log file
+    * verbose: Can either be True or False. Setting it to True would print any message where the verbose flag was passed. Note, the only internal system of GREASE that utilizes verbose is deduplication. The rest is in tracing
+    * trace: Can either be True or False. This enables tracing from within GREASE. This will show a "stream of consciousness" in the log files.
+    * foreground: Can either be True or False. True would print log messages to stdout as well as a log file
+    * file: Log file to write messages to
+* Notifications: Stores information about notification channels. All channels will need at least one key, "enabled" with a boolean True/False value to enable or disable the channel. All other keys are dependent on the notification channel
+* Configuration: This section contains information about this node's prototype configurations
+    * dir: A directory string on where to load configurations from
+* Sourcing: This section contains information about this node's sourcing prototype configuration
+    * dir: A directory string on where to load configurations from
+    * source: A string defaulted to null that if provided sourcing will focus only on prototype configurations from that source to get source data from
+    * config: A string defaulted to null that if provided sourcing will focus only that prototype configuration
+    * mock: A boolean value which when enabled will attempt to source mocking data dependent from the prototype configurations
+* Import: This section holds information about the import system
+    * searchPath: A list of strings of packages to attempt loading commands from
+* NodeInformation: This section controls how GREASE performs on the Node
+    * ResourceMax: Integer that GREASE uses to ensure that new jobs or processes are not spun up if *memory or CPU* utilization exceed this limit
+    * DeduplicationThreads: This integer is how many threads to keep open at one time during deduplication. On even the largest source data sets the normal open threads is 30 but this provides a safe limit at 150 by default
+* Additional: Unused currently but can be used for additional user provided configuration
+
+Cluster Configuration
+-----------------------
+
+Cluster configuration is stored in the MongoDB collection JobServer. Check the :ref:`datamodel` for more information
+about what is stored here.
+
+Prototype Configuration
+------------------------
+
+Prototype configuration is stored in the MongoDB collection Configuration, in the filesystem or located in the package.
+Check the :ref:`datamodel` for more information about what is stored here and the schema.
+
+Installing the Daemon
+=======================
+
+Installing the GREASE daemon is super easy. **Be sure you are logged in or running a console with administrative privileges**. Now
+install the daemon *on all supported platforms* by running :code:`grease daemon install`. On Unixlike systems you should now
+have a Systemd Service installed with the service file being stored at :code:`/etc/systemd/system/grease.service` and for
+Launchd at :code:`/Library/LaunchDaemons/net.grease.daemon.plist`. For windows you will have a new service installed.
+
+You can now control the operations of your cluster!

docs/GreaseGuide/data_model.rst

@@ -1,3 +1,5 @@
+.. _datamodel:
+
 The GREASE Data Models
 ***************************