diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 72c8e3ff..53f5d9db 100755 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -15,13 +15,6 @@ Use this space to explain what you expect to happen when if this PR is merged. ### Checklist - - [ ] Local test from submitter [link to any relevant results]() + - [ ] Tests + - [ ] Documentation - [ ] Maintainer Code Review - - [ ] Added To Wiki - - -### Optional Additional Checklist -###### This is for external dependencies you have also need before your PR is merged - - - [ ] If you have no external things just delete this checkbox - \ No newline at end of file diff --git a/bin/grease.ps1 b/bin/grease.ps1 index 45974a2f..68b28eca 100755 --- a/bin/grease.ps1 +++ b/bin/grease.ps1 @@ -6,4 +6,4 @@ foreach($arg in $args){ $loc = split-path $SCRIPT:MyInvocation.MyCommand.Path -parent -Invoke-Expression "python $loc\grease $argsString" \ No newline at end of file +python "$loc\grease" $argsString \ No newline at end of file diff --git a/docs/GREASE_Guide.rst b/docs/GREASE_Guide.rst index 223c61a5..570a96be 100644 --- a/docs/GREASE_Guide.rst +++ b/docs/GREASE_Guide.rst @@ -8,6 +8,7 @@ below for more information on how to get started. :caption: Contents: GreaseGuide/what_is_grease + GreaseGuide/user_guide GreaseGuide/Installation GreaseGuide/data_model diff --git a/docs/GreaseGuide/Installation.rst b/docs/GreaseGuide/Installation.rst index a818f992..cc7d8c97 100644 --- a/docs/GreaseGuide/Installation.rst +++ b/docs/GreaseGuide/Installation.rst @@ -1,20 +1,23 @@ .. _installing-grease: -GREASE Installation -********************* +GREASE Administration: Up & Running +*************************************** + +Initial Installation +======================= Via PyPi: The Traditional Way -=================================== +--------------------------------- GREASE is built like any other traditional python software, as a package. This means many things but luckily for the systems administrator it means we traverse the typical PyPi pipeline allowing for you to specify your version and allow for updates to be pretty easy. Here is the installation steps: #. Run `pip install tgt_grease` -#. Setup your configuration file +#. Setup your configuration file following From Source on GitHub: Because you're cool like that -========================================================= +------------------------------------------------------- GREASE is developer friendly! We're always looking for new ways to bring joy to our guests as well as our developers. To install GREASE via source follow these steps: @@ -26,3 +29,129 @@ install GREASE via source follow these steps: - **NOTE:** We recommend for all use cases to use a virtual environment #. Setup your configuration file + +Understanding & Configuring Your System +======================================== + +There are multiple definitions for configuration in GREASE. The primary ones are: + +#. Node Configuration: This refers to the local server's configuration for things like MongoDB credentials & resource limits +#. Cluster Configuration: This refers to the configuration of Job Servers inside of the cluster +#. Prototype Configuration: These are the configurations for prototypes, things such as sourcing & detection + +Node Configuration +---------------------- + +.. _nodeconfig: + +Node configuration is the local file the running instance uses to execute GREASE operations. It is stored in a file +called :code:`grease.conf.json`. This is stored in the GREASE directory. On Unix-like operating systems is found at +:code:`/opt/grease/` and for Windows :code:`C:\\grease\\`. You can override this behavior using the environment variable +:code:`GREASE_DIR`. + +The default configuration looks like this:: + + { + 'Connectivity': { + 'MongoDB': { + 'host': 'localhost', + 'port': 27017 + } + }, + 'Logging': { + 'mode': 'filesystem', + 'verbose': False, + 'trace': False, + 'foreground': False, + 'file': Configuration.greaseDir + 'log' + os.sep + 'grease.log' + }, + 'Notifications': { + 'HipChat': { + 'enabled': False, + 'token': None, + 'room': None + } + }, + 'Configuration': { + 'dir': Configuration.greaseDir + 'etc' + os.sep + }, + 'Sourcing': { + 'dir': Configuration.greaseDir + 'etc' + os.sep, + 'source': None, + 'config': None, + 'mock': False + }, + 'Import': { + 'searchPath': [ + 'tgt_grease.router.Commands', + 'tgt_grease.enterprise.Prototype', + 'tgt_grease.management.Commands', + 'tgt_grease.enterprise.Sources', + 'tgt_grease.enterprise.Detectors', + 'tgt_grease.core', + 'tgt_grease' + ] + }, + "NodeInformation": { + "ResourceMax": 95, + "DeduplicationThreads": 150 + }, + "Additional": {} + } + +Lets go through each key and the properties below, what they control and some values you may want to use. + +* Connectivity: This is the store for details around connectivity + * MongoDB: This key is the key used to find connection details about the central database + + =========== ============= ============ + Key value type default + =========== ============= ============ + host str localhost + port int 27017 + username str + password str + db str + =========== ============= ============ +* Logging: Logging configuration information + * mode: only supports filesystem logging currently to the log file + * verbose: Can either be True or False. Setting it to True would print any message where the verbose flag was passed. Note, the only internal system of GREASE that utilizes verbose is deduplication. The rest is in tracing + * trace: Can either be True or False. This enables tracing from within GREASE. This will show a "stream of consciousness" in the log files. + * foreground: Can either be True or False. True would print log messages to stdout as well as a log file + * file: Log file to write messages to +* Notifications: Stores information about notification channels. All channels will need at least one key, "enabled" with a boolean True/False value to enable or disable the channel. All other keys are dependent on the notification channel +* Configuration: This section contains information about this node's prototype configurations + * dir: A directory string on where to load configurations from +* Sourcing: This section contains information about this node's sourcing prototype configuration + * dir: A directory string on where to load configurations from + * source: A string defaulted to null that if provided sourcing will focus only on prototype configurations from that source to get source data from + * config: A string defaulted to null that if provided sourcing will focus only that prototype configuration + * mock: A boolean value which when enabled will attempt to source mocking data dependent from the prototype configurations +* Import: This section holds information about the import system + * searchPath: A list of strings of packages to attempt loading commands from +* NodeInformation: This section controls how GREASE performs on the Node + * ResourceMax: Integer that GREASE uses to ensure that new jobs or processes are not spun up if *memory or CPU* utilization exceed this limit + * DeduplicationThreads: This integer is how many threads to keep open at one time during deduplication. On even the largest source data sets the normal open threads is 30 but this provides a safe limit at 150 by default +* Additional: Unused currently but can be used for additional user provided configuration + +Cluster Configuration +----------------------- + +Cluster configuration is stored in the MongoDB collection JobServer. Check the :ref:`datamodel` for more information +about what is stored here. + +Prototype Configuration +------------------------ + +Prototype configuration is stored in the MongoDB collection Configuration, in the filesystem or located in the package. +Check the :ref:`datamodel` for more information about what is stored here and the schema. + +Installing the Daemon +======================= + +Installing the GREASE daemon is super easy. **Be sure you are logged in or running a console with administrative privileges**. Now +install the daemon *on all supported platforms* by running :code:`grease daemon install`. On Unixlike systems you should now +have a Systemd Service installed with the service file being stored at :code:`/etc/systemd/system/grease.service` and for +Launchd at :code:`/Library/LaunchDaemons/net.grease.daemon.plist`. For windows you will have a new service installed. + +You can now control the operations of your cluster! diff --git a/docs/GreaseGuide/data_model.rst b/docs/GreaseGuide/data_model.rst index 9b8f4eb0..3d46fc34 100644 --- a/docs/GreaseGuide/data_model.rst +++ b/docs/GreaseGuide/data_model.rst @@ -1,3 +1,5 @@ +.. _datamodel: + The GREASE Data Models *************************** diff --git a/docs/GreaseGuide/user_guide.rst b/docs/GreaseGuide/user_guide.rst new file mode 100644 index 00000000..7cd00534 --- /dev/null +++ b/docs/GreaseGuide/user_guide.rst @@ -0,0 +1,220 @@ + +.. _userguide: + +The GREASE User Guide +*********************** + + + +.. _ugconfig: + +1. Writing New Configurations +=============================== + +Typically a user will write prototype configurations, so we will focus on those here. To learn more about all the +different types of configurations see the :ref:`datamodel` page for more information. + +A Prototype Config tells GREASE where to look, what to watch for , where to run, and what to pass to a command. The +schema for a configuration can be found here :ref:`datamodel`. Let's say you have GREASE installed and you want to +monitor :code:`localhost` to make sure a webserver is still running via a synthetic transaction using a GET request. +That configuration would look something like this:: + + { + "name": "webserver_check_alive", + "job": "reboot_local_webserver", + "source": "url_source", + "url": ["http://localhost:3000/users/7"], + 'minute': 30, + "logic": { + "Range": [ + { + "field": "status_code", + "min": 200, + "max": 200, + "variable": true, + "variable_name": "status_code" + } + ] + } + } + +This configuration will tell GREASE to perform a HTTP GET on the address :code:`http://localhost:3000/users/7` once +every hour and ensure that a status code of 200 is returned. If it is not then the job :code:`reboot_local_webserver` +will be scheduled to run. It will be passed the variable :code:`status_code` in its' context. + +Now that we have a config written lets focus on the command :code:`reboot_local_webserver` in the next section. + +2. Writing Commands +====================== + +Commands are Python classes that are executed when configurations determine to do so. Building from :ref:`ugconfig` lets +now write the command :code:`reboot_local_webserver`. + +Commands extend a base class :code:`tgt_grease.core.Types.Command` found here: :ref:`commandtype`. Here is a basic command +for explaining a command:: + + from tgt_grease.core.Types import Command + import subprocess + + # Class name doesn't have to match command. But your Plugin Package must export the matching name (use alias') + class RestartLocalWebServer(Command): + + def __init__(self): + super(RestartLocalWebServer, self).__init__() + + def execute(self, context): + # Send a notification + self.ioc.getNotification().SendMessage("ERROR: Local Web Server returned back health check::status code [{0}]".format(context.get("status_code"))) + # Attempt the restart + return self.restart_nginx() + + def restart_nginx(self): + if subprocess.call(["systemctl", "restart", "nginx"]) == 0: + return True + else: + return False + +This command attempts to restart Nginx. If successful it will return true telling the engine the recovery has been +performed. If it does not get a good exit code it returns false letting the engine know it needs to attempt the recovery +again. **NOTE**: GREASE will only attempt a recovery 6 times before determining the command cannot succeed and stops attempting +execution of it. + +Since this is just traditional Python code you can do anything here! Call PowerShell scripts, interact with executables, +call some Java Jar you need, the possibilities are endless. All the GREASE Prototypes extend this base class and run all +of what we call GREASE. + +Variable Storage +------------------- + +Lets say we want to expand this example. We now want to execute only on the fifth 404. Rather than making the configuration +language very complicated we chose to put application logic in the applications, or as we call them, commands. This is where +variable storage comes into play. Since commands are typically short lived executions we offer the :code:`variable_storage` +property to your commands which is a provisioned collection just for your command. Lets refactor our command to use this +feature:: + + from tgt_grease.core.Types import Command + import subprocess + import datetime + + # Class name doesn't have to match command. But your Plugin Package must export the matching name (use alias') + class RestartLocalWebServer(Command): + + def __init__(self): + super(RestartLocalWebServer, self).__init__() + + def execute(self, context): + # Store the new bad status code + self.variable_storage.insert_one( + { + 'createTime': (datetime.datetime.utcnow() + datetime.timedelta(hours=6)), + 'docType': 'statusCode', + 'status_code': context.get('status_code') + } + ) + # Ensure a TTL for these documents + self.variable_storage.create_index([('createTime', 1), ('expireAfterSeconds', 1)]) + # Count the number of bad status' + if self.variable_storage.find({'docType': 'status_code'}).count() > 5: + # Send a notification + self.ioc.getNotification().SendMessage("ERROR: Local Web Server returned back health check::status code [{0}]".format(context.get("status_code"))) + # Attempt the restart + return self.restart_nginx() + else: + # Don't have enough bad status codes yet, so no failure condition + return True + + def restart_nginx(self): + if subprocess.call(["systemctl", "restart", "nginx"]) == 0: + return True + else: + return False + +Look at that! We have a pretty complete program there to restart a web server in the event of more than 5 bad requests +sent to a web server. + +3. Testing your Code +===================== + +Testing your code is very important, especially when you are engineering reliability! So GREASE helps with that too! Using +the built in test class found here: :ref:`commandtesttype`. Lets continue our series by writing a test for our command. +First We will write a test using the original command without Variable Storage:: + + from tgt_grease.core.Types import AutomationTest + from tgt_grease.core import Configuration + from my_demo_package import webserver_check_alive + + + class TestLocalWSRestart(AutomationTest): + + def __init__(self, *args, **kwargs): + AutomationTest.__init__(self, *args, **kwargs) + # For example our file name is basic.config.json so after install it will be here + self.configuration = "fs://{0}".format(Configuration.greaseDir + 'etc/basic.config.json') + # Mock data we expect to see from the web server + self.mock_data = { + 'url': 'localhost:8000', + 'status_code': 500 + } + # What we expect detection to tell us + self.expected_data = { + 'status_code': 500 + } + # Enable testing + self.enabled = True + + def test_command(self): + d = webserver_check_alive() + # Overload the restart_nginx function for testing purposes + d.restart_nginx = lambda: True + self.assertTrue(d.execute({'status_code': 500})) + +Now when running `python setup.py test` on your plugin your commands will be tested for their ability to detect +correctly and execute the way you would like. Since we use standard tooling you can also use tools to extract code +coverage and other statistics. + +Testing with Variable Storage +-------------------------------- + +Testing commands that use Variable Storage is just as simple. We just need to refactor our test a little bit to +arrange state around the command a bit:: + + from tgt_grease.core.Types import AutomationTest + from tgt_grease.core import Configuration + from my_demo_package import webserver_check_alive + import datetime + + + class TestLocalWSRestart(AutomationTest): + + def __init__(self, *args, **kwargs): + AutomationTest.__init__(self, *args, **kwargs) + # For example our file name is basic.config.json so after install it will be here + self.configuration = "fs://{0}".format(Configuration.greaseDir + 'etc/basic.config.json') + # Mock data we expect to see from the web server + self.mock_data = { + 'url': 'localhost:8000', + 'status_code': 500 + } + # What we expect detection to tell us + self.expected_data = { + 'status_code': 500 + } + # Enable testing + self.enabled = True + + def test_command(self): + d = webserver_check_alive() + # Create some state in the database + for i in range(0, 5): + d.variable_storage.insert_one( + { + 'createTime': (datetime.datetime.utcnow() + datetime.timedelta(hours=6)), + 'docType': 'statusCode', + 'status_code': context.get('status_code') + } + ) + # Overload the restart_nginx function for testing purposes + d.restart_nginx = lambda: True + self.assertTrue(d.execute({'status_code': 500})) + # Now just clean up state + d.variable_storage.drop() diff --git a/docs/conf.py b/docs/conf.py index 704c55a3..a0a5619d 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -69,7 +69,7 @@ # The short X.Y version. version = u'2.0' # The full version, including alpha/beta/rc tags. -release = u'2.0.1' +release = u'2.0.2' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/index.rst b/docs/index.rst index 2eb8a2b5..056b04a2 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,7 +7,7 @@ Welcome to GREASE's documentation! ****************************************** Welcome to GREASE. Target's operations automation platform! Here lies all the documentation on the project! Make sure -to checkout the :doc:`GREASE_Guide` for more information! +to checkout the :doc:`GREASE_Guide` for more information! Be sure if you're a new user to checkout the User Guide: :ref:`userguide` .. toctree:: :maxdepth: 3 diff --git a/docs/tgt_grease.core.Types.rst b/docs/tgt_grease.core.Types.rst index c9928e2a..4e022716 100644 --- a/docs/tgt_grease.core.Types.rst +++ b/docs/tgt_grease.core.Types.rst @@ -1,6 +1,8 @@ GREASE Core Types ================================ +.. _commandtype: + The GREASE Command ---------------------------------------- @@ -9,6 +11,8 @@ The GREASE Command :undoc-members: :show-inheritance: +.. _commandtesttype: + The Automation Tester ---------------------------------------- diff --git a/readme.md b/readme.md index c7859559..47597a0f 100755 --- a/readme.md +++ b/readme.md @@ -4,9 +4,11 @@ ###### Automation as a Service -[![Build Status](https://travis-ci.org/target/grease.svg?branch=master)](https://travis-ci.org/target/grease) +[![Travis](https://img.shields.io/travis/rust-lang/rust.svg)](https://travis-ci.org/target/grease) +[![AppVeyor](https://img.shields.io/appveyor/ci/lemoney/grease.svg)](https://ci.appveyor.com/project/lemoney/grease) [![Current Version](https://badge.fury.io/py/tgt-grease.svg)](https://pypi.python.org/pypi/tgt-grease) -[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/target/grease/blob/master/LICENSE) +[![Read the Docs](https://img.shields.io/readthedocs/pip.svg)](https://grease.readthedocs.io/) +[![License](https://img.shields.io/badge/license-MIT-brightgreen.svg)](https://github.com/target/grease/blob/master/LICENSE) ## GREASE diff --git a/setup.py b/setup.py index 40511650..6bf3d305 100755 --- a/setup.py +++ b/setup.py @@ -4,7 +4,7 @@ setup( name='tgt_grease', - version='2.0.1', + version='2.0.2', license="MIT", description='GRE Application Service Engine', long_description="""